JANUARY 18, 2013 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Janet Gruber, Iain McLellan, John Kadzandira and with inputs from Zione Themba through the GH Tech Bridge II Project. EVALUATION USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation Cover Photo from USAID Photo Gallery USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation i EVALUATION USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation FEBRUARY 2013 Global Health Technical Assistance Bridge II Project (GH Tech) USAID Contract No. AID￾OAA-C-12-00027 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. This document (Report No. 12-02-017) is available in printed or online versions. Online documents can be located in the GH Tech website at www.ghtechproject.com. Documents are also made available through the Development Experience Clearinghouse (http://dec.usaid.gov). Additional information can be obtained from: GH Tech Bridge II Project 1725 Eye Street NW, Suite 300 Washington, DC 20006 Phone: (202) 349-3900 Fax: (202) 349-3915 www.ghtechproject.com This document was submitted by Development and Training Services, Inc., with CAMRIS International to the United States Agency for International Development under USAID Contract No. AID-OAA-C-12-00027. ACKNOWLEDGEMENTS The EBT Prev Mid-Term Performance Evaluation Team wishes to thank all those who supported and contributed to meetings, discussions and fieldwork arrangements. First and foremost, we should like to thank the project beneficiaries who gave their time and effort to participate in focus group discussions and key informant interviews. We are grateful to the men and women among the target populations whom we had the opportunity to meet in the six priority prevention areas. We wish also to thank the representatives of the project partners, volunteers and staff members, the representatives of Malawi Human Rights Youth Network (MHRYN), Centre for the Development of People (CEDEP), Community Partnership for Relief and Development (COPRED), Christian Community Church (CCC), Namwera AIDS Coordinating Committee (NACC), Nkhotakota AIDS Society Organization (NASO), Society for Women with AIDS in Malawi (SWAM), Malawi AIDS Counseling and Resource Organisation (MACRO) and Theatre For a Change (TFaC). We should like to express our gratitude as well to USAID/Malawi, PSI Malawi and Pact staff members, without whose information and support the evaluation would not have been possible. We are additionally indebted to the representatives of the National AIDS Commission (NAC), BRIDGE II, UNAIDS and UNFPA—all of whom provided valuable insights. Our translator, Zione Themba, provided exemplary translation and invaluable help above and beyond her allotted tasks. We were also fortunate to have an excellent driver, Ahllie, who made our fieldwork travels safe and enjoyable. iii USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation CONTENTS ACKNOWLEDGEMENTS..................................................................................... I CONTENTS ..........................................................................................................III ACRONYM LIST....................................................................................................V EXECUTIVE SUMMARY .................................................................................... IX 1. INTRODUCTION ........................................................................................... 1 1.1 BACKGROUND TO THE HIV EPIDEMIC IN MALAWI ..................................................1 1.2 ALIGNMENT OF THE PROJECT OBJECTIVES AND ACTIVITIES WITH GoM HIV PRIORITIES ..........................................................................................................................................2 2. EVALUATION APPROACH AND METHODS............................................. 5 3. EVALUATION FINDINGS ............................................................................... 9 3.1 AN OVERVIEW OF PROJECT ACHIEVEMENTS AND CHALLENGES .......................9 3.3 EVALUATION QUESTIONS 1–6 ..........................................................................................16 3.4 EMERGING THEMES................................................................................................................54 4. CEDEP ISSUES ................................................................................................ 57 FINANCIAL MANAGEMENT ISSUES .........................................................................................57 OTHER ORGNIZATIONAL MANAGEMENT ISSUES ...........................................................57 CEDEP ENGAGEMENT WITH MSM PEER EDUCATORS ...................................................57 LACK OF MSM-SPECIFIC SUPPORT MATERIALS..................................................................57 RECOMMENDATIONS..................................................................................................................58 5. GENDER PERSPECTIVES AND EBT PREV FOCUS .................................. 59 1. LACK OF GENDER CAPACITY WITHIN EBT...................................................................59 2. SPECIFIC EXAMPLES OF LACK OF GENDER FOCUS.....................................................60 3. RECOMMENDATIONS .............................................................................................................62 APPENDIX A. SCOPE OF WORK................................................................... 65 APPENDIX B. PERSONS CONTACTED........................................................ 81 APPENDIX C. EVALUATION METHODOLOGY (THE EVALUATION PLAN, DRAFT 2)................................................................................................. 91 APPENDIX D. EBT PREV MID-TERM PERFORMANCE EVALUATION QUALITATIVE AND QUANTITATIVE DATA COLLECTION INSTRUMENTS 115 APPENDIX E: EBT PREV EVALUATION—FIELDWORK APPROACH AND METHODS (INCLUDING SAMPLE DETAILS)............................................. 127 iv USAID/MALAWI: EBT Prev Mid-Term Evaluation APPENDIX F: FURTHER DISCUSSION OF THE 13 EVALUATION QUESTIONS ...................................................................................................... 135 APPENDIX G: OVERVIEW OF PROJECT ACTIVITIES (OVERALL AND BY YEAR).................................................................................................................. 139 APPENDIX H: EBT PREV GANTT CHART (AS PROVIDED BY PSI/M AND PACT) ................................................................................................................. 145 APPENDIX I: PROJECT DATA: 2010–2012 CUMULATIVE AND FINANCIAL YEAR 2011-2012.......................................................................... 153 APPENDIX J: NEW PROJECT REFERAL SYSTEM DIAGRAM................... 165 APPENDIX K: REFERENCES........................................................................... 167 v USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation ACRONYM LIST AIDS Acquired Immunodeficiency Syndrome ART Antiretroviral treatment/therapy BBSS Biological and Behavioral Surveillance Survey BCC Behavior change communication CBDA Community-based distribution agent (of FP commodities) CBO Community-based organization CCC Christian Community Church CEDEP Centre for the Development of People COPRED Community Partnership for Relief and Development CPRD Convergent Mixed Methods Parallel Research Design CSW Commercial sex worker DQA Data quality assessment EBT Prev Evidence-Based, Targeted HIV Prevention Project FGD Focus group discussion FP Family planning FPAM Family Planning Association of Malawi FY Fiscal year GBV Gender-based violence GoM Government of Malawi HIV Human immunodeficiency virus HSA Health surveillance assistant HTC HIV testing and counseling IPC Interpersonal communications IPC/A Interpersonal communications assistant IPC/V Interpersonal communications volunteer KII Key informant interview MACRO Malawi AIDS Counseling and Resource Organisation MARP/sMost-at-risk population/s MDHS Malawi Demographic and Health Survey MHRYN Malawi Human Rights Youth Network vi USAID/MALAWI: EBT Prev Mid-Term Evaluation MoH Ministry of Health MSM Men who have sex with men NAC National AIDS Commission NACC Namwera AIDS Coordinating Committee NASO Nkhotakota AIDS Society Organization NGO Non-governmental organization OPI Organizational performance index PEP Post-exposure prophylaxis PEPFARPresident’s Emergency Plan for AIDS Relief PFIP Partnership Framework Implementation Plan PLACE Priorities for Local AIDS Control Efforts PMP Project Monitoring Plan PMTCT Prevention of mother-to-child transmission PMTCT B+ Prevention of mother-to-child transmission, with lifelong ART option for the mother PPA Priority prevention area PSI Population Services International PSI/M Population Services International/Malawi RFA Request for applications RH Reproductive health SBCC Social behavior change communication SOW Scope of work STI Sexually transmitted infection SWAM Society for Women with AIDS, Malawi TFaC Theatre For a Change TOC Targeted outreach communication TRaC Tracking Results Continuously UNAIDS Joint United Nations Programme on HIV/AIDS UNFPA United Nations Population Fund USAID United States Agency for International Development VSU (Police) Victim support unit vii USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation viii USAID/MALAWI: EBT Prev Mid-Term Evaluation ix USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation EXECUTIVE SUMMARY BACKGROUND: HIV IN MALAWI AND THE EVIDENCE-BASED, TARGETED HIV PREVENTION PROJECT In 2011, there were 50,000 new HIV infections in Malawi. Close to 1 million Malawians (out of a population of around 15 million) are living with HIV. Key drivers of the epidemic include high rates of unprotected heterosexual sex, multiple and concurrent sexual partners and couple discordance. An estimated 80% of new infections occur among partners in stable relationships. Surveillance studies have showed higher HIV prevalence rates among certain most-at-risk populations (MARPs), including prevalence as high as 70% among commercial sex workers (CSWs) and 21% among men who have sex with men (MSM). The USAID/Malawi-supported Evidence-Based, Targeted HIV Prevention (EBT Prev) Project, implemented by Population Services International/Malawi (PSI/M) with significant inputs by Pact, has the potential to deliver key information on a number of dynamics underpinning the HIV epidemic. The purpose and objectives of EBT Prev are aligned with the President’s Emergency Plan for AIDS Relief (PEPFAR) Partnership Framework and the Partnership Framework Implementation Plan (PFIP), both of which are aligned with the National HIV Prevention Strategy 2009–2013 of the Government of Malawi (GoM). The Prevention Strategy prioritizes evidence￾based, engendered, complementary prevention interventions and harmonizes biomedical and behavioral interventions. EBT Prev is PEPFAR/Malawi’s flagship prevention project focusing on MARPs and high-risk settings. Project interventions focus on CSWs, MSM, male and female plantation workers, fishermen and women in fishing communities and male and female vendors. The project development hypothesis is that to support reduction of HIV incidence, combination prevention activities in 20 priority prevention areas (PPAs) will result in reduction of reported high-risk behaviors, including multiple concurrent partnerships, as well as result in an increased uptake of condoms and responsive HIV-related services among MARPs. It is also expected that implementation of these activities will provide the GoM with effective models for scale-up of targeted prevention efforts for MARPs and vulnerable populations in other areas. The primary aim of EBT Prev has been to drill down into MARPs, conduct social behavior change communications (SBCCs) and inspire changes in behaviors. EBT Prev was designed in 2008, with a degree of adaptation since, e.g., greater attention to gender-based violence (GBV) and voluntary medical male circumcision (VMMC). MID-TERM EBT PREV EVALUATION A mid-term performance evaluation of the EBT Prev Project was undertaken between late September and late November 2012. The primary purpose of the evaluation was to determine the extent to which the project’s behavior change and communications interventions have resulted in reduction in risky behavior among MARPs. A secondary objective was to understand the extent to which the program approach and multiple components have proven to be an effective strategy. The evaluation findings and recommendations will be used to inform decisions on the strategic direction of future project implementation and USAID/Malawi’s investment in HIV prevention among MARPs, and will also serve as the basis for planning the remaining 18 months of the project. Evaluation Approach and Methodology x USAID/MALAWI: EBT Prev Mid-Term Evaluation This evaluation used primarily qualitative methodology, with some limited quantitative collection of data through two surveys administered to project partner outreach workers and project partner officers. Fieldwork data collection comprised focus group discussions (FGDs) with project interpersonal communications volunteers and assistants (IPC/Vs, IPC/As) and MARP representatives; key informant interviews (KIIs) with project partners and local health workers; questionnaires administered to project partners and outreach workers; and meetings with USAID, the National AIDS Commission (NAC), PSI, Pact, USAID’s BRIDGE II project, the Joint United Nations Programme on HIV/AIDS and the United Nations Population Fund. The data collected were analyzed and triangulated with project data and the analysis and findings served to strengthen responses to the evaluation questions set out in the scope of work (SOW). The evaluation team applied the Convergent Mixed Methods Parallel Research Design. OVERVIEW OF PROJECT ACHIEVEMENTS AND CHALLENGES Achievements The EBT Prev Project is acknowledged to have largely achieved its performance targets to date and to have supported a degree of behavior change among MARPs, especially with regard to HIV testing and counseling (HTC), condom use and enhanced couple communication. EBT Prev has achieved or exceeded most of its indicator targets and has contributed toward behavior change in members of target MARPs. There are trends to indicate more regular and sustained condom use, greater willingness to go for HTC, greater couple communication and potential reduction of multiple concurrent partnerships. The project has developed a valuable national resource in terms of committed, trained volunteers able to support HIV prevention and project partners have further developed their capacity in outreach, managing large numbers of volunteers and engaging with the health system. PSI/M is considered a key strategic player in Malawi in the provision of socially marketed condoms, and as a leader in introducing VMMC. PSI/M is considered innovative in its work with neglected MARPs. PSI/M and Pact have made valuable contributions to the national HIV prevention strategy, community development and support, informed national debate and non￾governmental organization partner development. Challenges Despite being described in the project name as “evidence-based,” there is a perception that the project has provided limited data. Purpose/outcome indicators are only to be measured twice during the project lifetime, which is likely to be insufficient for tracking trends and influences on behavior change. Concern has also been expressed about “reach” and numbers and what these actually mean in terms of supporting sustained behavior change and project attribution for having an impact on the epidemic. Other data use challenges include evidence and discussion of lessons learned and emerging best practices; feedback on data collected by IPC/Vs, IPC/As and peer educators; and measurement of demand-side (i.e., MARPs’) views of quality of services or appropriateness of SBCC interventions. SBCC challenges include insufficient support materials for communications interventions; insufficient focus on service uptake/HTC as entry points to health and other support services; lack of a comprehensive, linked approach to quarterly messaging themes; and insufficient targeted attention to CSWs and MSM. Difficulties with distribution of free condoms and commodities such as HIV test kits have also been significant barriers. xi USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation At the national level, PSI/M has a role to play in informing Malawi’s approach to addressing MARPs and future policy development in the context of a “mixed epidemic.” PSI/M also needs to strengthen its technical expertise on gender in order to help address societal norms and power structures that underpin GBV. The organization should be more consultative and informative about its activities at the district level and below. ACHIEVEMENT TO DATE OF EBT PREV OBJECTIVES Objective (1): Identify, segment and profile priority populations at risk. This objective has been largely achieved through Priorities for Local AIDS Control Efforts (PLACE), condom-mapping and service delivery research. In 2010, PSI/M undertook a cross￾sectional baseline survey designed to monitor trends in condom use and concurrent partnerships among male and female vendors, plantation workers and members of fishing communities. Findings from this survey assisted development of purpose/output indicators. Objective (2): Deliver integrated, behavior change communication programs targeted to high-risk populations in priority prevention areas. EBT Prev has succeeded in part in implementing Objective 2 in terms of reaching agreed targets for reaching MARPs in PPAs, as measured for male and female plantation workers, fishing community members and vendors. The one MARP that has seen consistent underperformance in terms of project reach and SBCC engagement is MSM. CSWs represent another challenging MARP, while all groups present both unique characteristics and shared ones such as mobility. Objective (3): Distribute and promote condoms for use by the general population and for high-risk groups. PSI/M has a well-established social marketing network that enjoys consistent annual growth in sales and sales outlets, especially in the PPAs, where there has been good collaboration between condom social marketers and EBT Prev partners. There have been considerably more difficulties in distributing free condoms to the general population and MARPs. It is unclear what exactly has contributed to the frequent stock-outs in the distribution system, which have made it difficult for EBT Prev outreach workers to provide sufficient condoms to meet the needs of the target populations with whom they work. Objective (4): Enhance the network of existing providers for greater accessibility and services to high-risk populations. Between 2009 and January 2012, the project implemented a referral system that focused on community-based organizations (CBOs) in the PPAs. This led to a proliferation of potential and actual referral agents and organizations, service delivery facilities and records that resulted in an over-intricate and burdensome system that was not adequately delivering the objective. EBT Prev requested and USAID granted, an opportunity to revise the referral system. Since June 2012, project volunteers have been the sole referral agents and refer MARPs to a more limited number of “hub” health facilities; most PPAs have one hub, while a few have two or three. Objective (5): Voluntary medical male circumcision (VMMC) service delivery in Thyolo and Blantyre districts and associated demand creation in Blantyre district. This objective was added in 2012. Five performance indicators are to be measured. The VMMC component has not yet become operational in terms of clinical interventions. Planning and recruitment have been initiated and clinical activities are likely to start in February 2013. xii USAID/MALAWI: EBT Prev Mid-Term Evaluation EVALUATION QUESTIONS 1–6: MAIN FINDINGS, CONCLUSIONS AND RECOMMENDATIONS QUESTION 1: How effective have PSI’s EBT Prev communications approach and targeted outreach communication (TOC) been in supporting adoption of safer sexual behaviors among project target groups? Main Findings and Conclusions: EBT Prev has exceeded its annual targets for reaching all target populations, except one—MSM. Interventions have focused on target populations in 18 PPAs throughout Malawi. Four mobile TOC units are used to visit PPAs to conduct “community-wide events” in the form of dramas and interactive video screenings and to complement the work of IPC/Vs. Half of the MARPs who participated in the evaluation said they had seen either a local drama or had had some participation in a TOC event. Improvement in couple communication and ability to talk about sexual issues was cited in the majority of MARP focus group discussions. Awareness of the impact of multiple concurrent partnerships, behavior choices, HIV testing, partner reduction, GBV, sexual web and condom use were also mentioned. Interventions are handicapped by a lack of support materials. Underuse of radio is a missed opportunity for expanding the reach of interpersonal communication (IPC), while the lack of support for repairing project-donated bicycles reduces IPC/V and IPC/A mobility. Recommendations: The project should (1) create targeted support materials for use by IPC outreach workers so that they can increase participation, stimulate discussion and stay on message; (2) enhance negotiation skills of target populations through role-playing and other approaches that increase skills in couple communication; and (3) provide support for IPC/V and IPC/A bicycle maintenance. QUESTION 1A: Are communications activities adequately tailored to the different categories of populations among the target groups, including CSWs and MSM—i.e., are they responsive to the specific barriers faced by these groups and are they sufficiently skills-oriented to support behavior change? Main Findings and Conclusions: Communications activities are responsive to some target groups (plantation workers, fishing communities, market vendors) but less so to CSWs and MSM. Few support materials have been developed specifically for MSM. IPC/Vs and IPC/As are able to relate messages to the lives of most target populations and engage in participatory discussion of issues. Systems are in place to support IPC/As to encourage IPC/Vs to stimulate discussion. In all FGDs with all MARPs, the ability of IPC/Vs and IPC/As to focus discussions on specific problems without being judgmental or moralistic was much appreciated. Links between IPC/Vs and IPC/As and targeted outreach communication, local dramas and mobile testing could be improved to enhance access to target populations and strengthen follow-up. Recommendations: The project should consider opportunities for targeting discordant couples and for complementing messages on reducing concurrent sexual partners, condom use and negotiation with messages on discordance, disclosure to partners and positive prevention. The project should also involve IPC/Vs and IPC/As more in animating TOCs, local dramas and mobile testing. Methods to consider include identifying outreach workers with brightly colored T-shirts and scattering the workers among audiences to answer questions or hand out materials/ condoms. IPC/Vs and IPC/As could facilitate discussions following dramas. QUESTION 1B: To what extent has the frequency in which all target populations have been reached with messages supported the adoption of safer behaviors, including increased demand for and use of appropriate HIV and reproductive health (RH) services and commodities? Main Findings and Conclusions: No evidence is available to link the frequency and reach of project messages with the adoption of safer sex behaviors and the creation of demand for HIV xiii USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation and RH services. Qualitative data indicate that all MARPs have been influenced by project activities to seek HTC. According to outreach workers and to fishermen, market vendors, plantation workers and CSWs themselves, mobility is a handicap to achieving the desired frequency of contact. Insufficient use of mass media or collaboration with partners also greatly reduces reach and frequency. There is a tendency to conduct outreach in areas close to the residences of IPC/Vs and thus fail to reach MARPs farther afield. Recommendations: The project should encourage IPC/Vs and IPC/As to conduct more small-group sessions and expand mobile HTC linked to existing outreach activities. Data should guide selection of beneficiaries by age and sex; there should be better tracking of the proportion of referrals that translate into uptake of services. The project should record and broadcast TOC community sessions and dramas along with in-studio commentary and community discussion. QUESTION 1C: To what extent has the project supported synergies between different communication interventions and [been] responsive to service priorities likely faced by populations, such as integrated prevention of mother-to-child transmission/antiretroviral treatment (PMTCT/ART) and related GBV services? Main Findings and Conclusions: Prevention of PMTCT activities and ART integration were not underway at the time of the evaluation. Since the beginning of 2012, the project has placed an increased focus on PMTCT and family planning (FP) through alignment of PSI-branded FP commodities with relevant messages. A coherent and cohesive overarching SBCC strategy is absent, however and the lack of HTC kits in public health facilities is also a handicap. There has been moderately effective GBV messaging and the combination of IPC, TOC, drama and community-level advocacy has resulted in increased GBV awareness. SBCC created a demand for services, but uptake was handicapped by insufficient service delivery, especially related to free condom distribution, HTC and sexually transmitted infection (STI) treatment. Recommendations: The project should develop a comprehensive SBCC strategy for future quarterly themes and strengthen measurement of SBCC outputs on service uptake and adherence within indicators on program quality and coverage. The communication strategy should be updated and all future quarterly messages pretested with IPC/Vs, IPC/As and MARP representatives. QUESTION 2: What effect has the addition of new program areas, specifically voluntary medical male circumcision and gender-based violence, had on PSI’s operational management and capacity to implement quality assurance? Preliminary Conclusions Regarding VMMC: It is not possible at the time of this evaluation to measure progress against the new VMMC Objective 5 or to discuss findings. Indications are that PSI has not yet fully estimated operational management inputs specific to implementation of VMMC. In doing so, due consideration should be given to the overall context of public health in Malawi, linkages to other HIV prevention interventions, the HIV prevention needs of women and girls and appropriate and adequate counseling of wives and partners. Preliminary Recommendations: PSI/M should review its internal operational capacity to ensure optimal management and quality assurance of the VMMC component and evaluate the VMMC workload at three-month intervals. PSI/M and other VMMC actors should develop an integrated action plan before implementing VMMC, making use of international best practices. The plan should include adequate attention to gender aspects of VMMC and the impacts of potential behavior change on partners of circumcised men. PSI/M should consider contracting two independent experts to monitor and evaluate the VMMC component in terms of quality assurance and appropriateness of messaging and behavior change focus. xiv USAID/MALAWI: EBT Prev Mid-Term Evaluation Main Findings and Conclusions Regarding GBV: The evaluation team found no evidence that the introduction of the GBV component has in itself taxed PSI operational management capacity. However, indications are that PSI/M has insufficient in-house gender expertise and has undertaken insufficient planning informed by gender-aware approaches. The EBT Prev communications strategy/approach does not adequately address issues of gender and there appears to be no staff member at PSI/M or Pact tasked with primary responsibility. The implementation of the Quarter 4 GBV messaging and activities has been hampered by operational capacity challenges connected to referral system changes. A lack of a systematic, strategically guided and technically sound gender focus has had a significant impact on the quality of delivery of project inputs and on overall quality assurance. A major aspect of this is the lack of a technically sound engendered communications strategy, in which each successive quarter’s messaging builds on earlier ones, incorporating gender as intrinsic and standard. Given that proposed Quarters 5–7 messages also have significant gender perspectives, it is important that PSI/M rectify this deficiency. Recommendations: The project should address as a priority the recruitment of a full-time gender and health expert to provide strategic gender focus and guide project activities to 2014 (including message development, training, research planning and review of project indicators to allow more engendered, disaggregated measurement of progress and performance and gender aspects of the new referral system). All pilot testing of future quarters’ messaging needs to include attention to gender perspectives. Engagement with police victim support units should be reviewed and opportunities for collaboration explored. Outreach workers, hub health workers and HTC counselors would all benefit from a refresher course on gender. The project should consider the addition of a gender-specific indicator linked to SBCC. QUESTION 3: To what extent is the current project monitoring and evaluation framework measuring indicators that are appropriate and sufficient to demonstrate the value of the social behavior change communications approach? Main Findings and Conclusions: The reluctance of some MARPs to sign for the second of the two sessions required for the project to count people receiving interventions presents a measurement challenge. A number of reasons for this reluctance were given: insufficient assertiveness and/or negotiation skills on the part of the IPC/Vs and IPC/As to encourage MARPs to sign twice; a lack of understanding by MARPs of why it is important to sign twice; an apparently strongly held view among some members of MARPs that signing twice should be accompanied by payment for time spent receiving IPC. In addition, it proved impossible to gauge fully how many individual members of MARPs are met only twice per quarter, how many may be met more frequently; and how many entirely new MARPs are provided with EBT Prev IPC per quarter or month. The project monitoring and evaluation (M&E) framework does not adequately address SBCC objectives; exposure to and outcomes of SBCC; effect of SBCC on target groups over time; attribution of SBCC; client/MARP satisfaction; and gender sensitivity and disaggregation. Project indicators do not sufficiently track SBCC processes. It will be challenging to measure project contributions to sustained behavior change among target populations. There is also limited feedback and discussion of project data and limited tracking of the evidence base. Recommendations: PSI/M, Pact and USAID should discuss how best to gauge the impact of project approaches and interventions in endline and other studies (while bearing in mind that EBT Prev was not structured to provide such data). Focus should be given to tighter and more disaggregated measurement of reach; tightening the counting of MARPS reached and how many times; and how often each individual MARP is reached per message. Expert attention should also be given to strengthening SBCC processes and outcomes in project indicators and M&E; to including a demand-side MARP indicator to measure quality of service delivery; and to undertaking a gender-focused review of objectives and indicators to strengthen disaggregated xv USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation and gender-sensitive project tracking. Opportunities should be explored for data feedback to outreach workers. A planned 2013/2014 “Exposure Survey” to examine MARPs’ contacts with project activities should be expanded to include IPC/Vs and IPC/As. QUESTION 4: To what extent have the community-based organizations and IPC/Vs successfully assimilated the capacities introduced through the capacity￾building efforts? Main Findings and Conclusions: Project activities have built the capacity of project partners. Individual project partners demonstrate organizational capacity in terms of management of outreach workers, engagement with MARPs, progress toward implementing the new referral system and financial management. A minority of CBOs enhanced their capacity to engage in community HIV and RH activities due to challenges under the old referral system, but CBOs will likely continue to play an important role in project activities with MARPs. IPC/Vs and IPC/As felt that they had increased their capacity through engagement with the project. The majority of all categories of project outreach workers stated that they are confident they can provide services, make referrals and teach people about condom use. Recommendations: Effective inputs for developing project partners’ capacity should be further focused to strengthen data management and reporting. Organizational performance indexes (OPIs) and data quality assessments (DQAs) should be speedily brought to scale and reported on and any gaps addressed. The project should review its support to outreach workers and develop mechanisms to mitigate potential for burnout. This should be done as the new referral system becomes fully operational and before EBT Prev introduces Quarter 5–7 messaging. The project should review its training approach and make it less didactic through more role play and other participatory methods. Issues of and options for MSM and CSW peer educator support should be speedily addressed, as should particular questions concerning project partner relationships with the Malawi AIDS Counseling and Resource Organisation and the Centre for the Development of People. QUESTION 4a: What elements of these new capacities will CBOs and IPC volunteers sustainably continue to implement without the support of EBT Prev? Main Findings and Conclusions: Project partners expressed concerns about their capacity to sustain project activities once funding ends and to maintain a complex outreach structure dependent on the continued loyalty and commitment of IPC/Vs and also on individual project partner organizations’ capacity to continue to pay staff once EBT Prev funding ceases. Respondents were more positive about the sustainability of individual organizations’ skills in HIV work. The project has provided opportunities to sustain IPC activities once the project ends through the establishment of local partner-managed networks of outreach workers. Recommendations: As part of its exit strategy, the project should actively explore opportunities for scaling up project activities and outreach and referral networks and systems. The model of IPC/Vs, IPC/As and peer educators engaging in HIV prevention should be considered for scale-up. PSI/M and Pact should expand support for project partner resource mobilization. QUESTION 5: How effective have EBT Prev activities been in strengthening the network of community-based HIV and reproductive health providers? Main Findings and Conclusions: Health surveillance assistants and community-based distribution agents were not involved in project activities under the old referral system and specific relationships with these community-based health workers were not cultivated by project stakeholders. The role of community-based health workers in engaging with the new project referral system requires further clarification. The planned project focus in Quarters 5–7 on xvi USAID/MALAWI: EBT Prev Mid-Term Evaluation PMTCT with the lifelong ART option (PMTCT B+), FP and HTC provides a powerful potential entry point for the participation of community-based health providers. Recommendations: All relevant project actors (PSI/M, Pact, USAID, the Malawian Ministry of Health, NAC and project partners) should discuss opportunities to work more closely with community-based HIV and RH providers. These discussions should address the role of IPC/Vs in particular, to ensure any additional work to be undertaken by these outreach workers is effectively managed and supported. Consideration should be given to including an indicator to track involvement of community-based health providers. QUESTION 5a: To what extent has EBT Prev’s support to the provider network improved referral systems for the priority target groups? Main Findings and Conclusions: The first referral system built and maintained the capacity of IPC/Vs and IPC/As to refer clients to services. The project has been responsive to the failures of the first system, using lessons learned to develop the new system, implemented since July 2012. It is not possible to provide findings for the new system because it had only been operational for three months at the time of the mid-term evaluation. A new site visit would be required to see the new referral system in action. However, it received praise for making it easier for MARPs to seek services and for potentially strengthening the links between communities and the health system. There is an apparent lack of consensus among project planners regarding the number of MARPs who are likely to be referred under the new system that is being established. There is a need for further discussion and clarification with project partners and outreach workers. Recommendations for New Referral System: The project needs to remain vigilant against abuse of the new referral system and of clients, with an absolute guarantee for all project￾referred MARPs that no private information will be provided to health workers without an individual’s express consent. The project should double-check that none of its private/faith￾based hubs is charging inappropriate fees for health services. Issues of service delivery quality from the client perspective should be included in all future referral work, including training and refresher sessions. Tracking beyond the first referral; increasing the number of MARPs who go for HTC and receive their test results; and increasing the low numbers of MARPs referred for PMTCT, FP, STI treatment and GBV survivor services are referral-related issues for the project to address using sex-disaggregated data. The project’s relationship with effective CBOs should be maintained and monitored for overall service delivery quality and effective engagement with hub health facilities. The project, USAID and GoM partners should explore the potential for scale-up of the links between communities and the health system in light of eventual lessons to be learned from the new referral system. QUESTION 6: How effectively has PSI maintained a balance between social marketing of condoms and free distribution of male and female condoms for the priority target groups? QUESTION 6a: How have the dynamics of this balance affected access to condoms for the priority target groups? Main Findings and Conclusions: Malawi lacks a coherent distribution system of free condoms, which are necessary for remote rural areas, lower-income segments of the population and young people. There is little evidence that government and international stakeholders systematically coordinate condom procurement and free condom distribution, which contributes to the breakdown of the distribution system and widespread stock-outs. IPC/Vs and IPC/As are well-positioned to promote condom use and deliver free condoms to MARPs and the male and female condoms supplied by project outreach workers are appreciated by the target populations. Female condoms are popular but under promoted. xvii USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation Social marketing is a cost-effective intervention and the consistent availability of socially marketed condoms allows for condom availability during stock-outs of free condoms. EBT Prev target populations interviewed in FGDs stated that they will use socially marketed condoms if free ones are not available. The erratic supply of free condoms has little or no effect on sold condoms. Some employers buy socially marketed condoms and distribute them free to employees. An estimated one-third of free condoms end up being sold, especially by owners of bars and bottle stores later in the evening when demand increases and after the public health distributors of free condoms have closed. Additional outlets developed in PPAs include groceries and kiosks that are also open throughout the evening. Mass media promotion of socially marketed condoms results in increased use of free condoms as well. Recommendations: Mass media advertising of socially marketed condoms should continue, with the goal of increasing overall use of condoms. Obstacles to condom use should be identified and overcome. The project should help provide leadership for reactivating the condom committee of donor and government representatives and for rationalizing the condom procurement and distribution systems. If difficulties persist with frequent stock-outs of free condoms to be distributed by project IPC/Vs and IPC/As, alternative distribution methods should be considered in collaboration with wholesalers and PSI/M distribution mechanisms. Adequate resources should be ensured for promoting female condoms, which should be added to socially marketed product lines as demand grows. xviii USAID/MALAWI: EBT Prev Mid-Term Evaluation USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 1 1. INTRODUCTION 1.1 BACKGROUND TO THE HIV EPIDEMIC IN MALAWI Malawi has a population of around 15 million; in 2011, there were 50,000 new HIV infections. Close to 1 million Malawians are living with HIV. It is therefore essential to understand the drivers of the epidemic and cut rates of transmission. Key factors include high percentages of unprotected heterosexual sex, multiple and concurrent sexual partners and couple discordance. An estimated 80% of new infections in Malawi currently occur among partners in stable relationships, which would represent 40,000 each year. The HIV epidemic in Malawi continues to evolve, requiring dynamic responses to changing situations. The Evidence-Based, Targeted HIV Prevention (EBT Prev) Project has the potential to deliver key information on a number of issues underpinning the epidemic. A study on modes of HIV transmission is supposed to be undertaken soon in Malawi; its intended focus will be on the questions: “Where did the last 1,000 HIV infections occur and where will the next 1,000 come from?” Findings from the 2006 Biological and Behavioral Surveillance Survey (BBSS) showed higher HIV prevalence among certain groups engaging in high-risk behaviors when compared to the general population. This has not changed. Information provided to the evaluation team by UNFPA indicates that 70% of commercial sex workers may be HIV-positive; the corresponding percentage for men who have sex with men is estimated at 21%. A 2007 survey conducted among MSM in Blantyre indicates an HIV prevalence of 21.4%. The 2010 study by Beyrer et al. provides information on the complexity of MSM sexual relations in Malawi, e.g., through “bisexual concurrency” and the fact that many MSM are married. Many grave challenges exist with regard to providing support to MARPs to prevent HIV infection. Among these are legal barriers, unwillingness to address sex work and MSM and a paucity of detailed and reliable studies to guide interventions through an evidence base. The 2012 Government of Malawi’s Global AIDS Response Country Report lists the key sociocultural and economic drivers of the HIV epidemic in Malawi. Among these are: • Low socioeconomic status of women and gender inequalities, e.g., barriers to access to services, adverse cultural practices, gender-based violence, and weak bargaining power for condom use and fidelity. • Low and inconsistent use of condoms. • Significant levels of transactional sex. • Poverty and its many ramifications for sociocultural, economic and health vulnerability. • (Self) stigma and discrimination and other socioeconomic factors, which often result in people living with HIV delaying or dropping out of treatment. • Difficulty in reaching members of vulnerable populations and MARPs. • Discriminatory legislation against MARPs, which stops effective prevention and treatment programs from being implemented. • Low rates of medical male circumcision. • High level of knowledge on modes of infection not reflected in prevalence data, suggestive of inadequate targeted interventions 2 USAID/MALAWI: EBT Prev Mid-Term Evaluation • Sexually transmitted infections and their link to HIV infection (which have not been sufficiently addressed). The Malawi Demographic and Health Survey (MDHS) 2010 shows an estimated reduction in prevalence in the general population over the past seven years, down to 10.6% from the 12.0% estimated in MDHS 2004. Women continue to be especially affected, with 2010 prevalence among all women at 12.9%, compared to all men at 8.1%. Women aged 35–39 represent the group with the highest HIV prevalence, at 24.0%. MDHS 2010 data indicate that HIV prevalence in urban areas is twice that of rural areas. Seventeen percent of women and men aged 15–49 in urban areas are infected with HIV compared to 9% in rural areas, with women in urban areas experiencing a prevalence rate of 22.7%. The prevalence rate for rural women in the same age group stands at 10.5%. HIV prevalence in the southern region is twice that of the central and northern regions. However, overall HIV prevalence has decreased significantly in the southern region—from 17.6% in 2004 to 14.5% in 2010. By contrast, the prevalence rate in the central region has increased since 2004, from 6.5% to 7.6%. HIV prevalence among men and women aged 15–19 shows an increase in MDHS 2010, to 2.7% overall. This percentage when disaggregated reveals a 4.2% rate for young females and 1.3% for young males. In 2010, the overall HIV prevalence among young people aged 15–24 stood at 3.6%. As in 2004, HIV prevalence was higher among women (5.2%) compared to men (1.9%). Both the 2004 and 2010 surveys demonstrate that HIV prevalence among young people increases with multiple sexual partners; for those with multiple concurrent partners, HIV prevalence is higher still. Antenatal care data indicate that prevalence of HIV in pregnant women has declined from 16.9% in 2001 to 10.6% in 2010. As discussed in documents such as the 2012 GoM report, the 2011 Family Planning Association of Malawi (FPAM) report on sex work in Malawi and Watkins’ 2010 study on HIV determinants in Malawi, a number of sociocultural factors significantly influence and continue to shape the HIV epidemic in Malawi. Two key ones are gender inequality and lack of empowerment for women and girls and women’s relatively low socioeconomic status. Traditional cultural practices such as widow cleansing; perceptions of male and female sexuality and what constitutes normative and appropriate behavior for men and women; widespread acceptance of sexual and gender-based violence; a lack of recognition of marital rape; and lack of access to information and services are among the issues that continue to act as gender barriers to equitable prevention, care, treatment and mitigation of HIV and AIDS. In addition, there is evidence of women testing positive, experiencing GBV and subsequently being abandoned by their husbands or partners. Poverty compounds such gender inequalities, for instance, by limiting women’s ability to negotiate for safe sex and by being unable to purchase condoms when free ones are unavailable. 1.2 ALIGNMENT OF THE PROJECT OBJECTIVES AND ACTIVITIES WITH GOM HIV PRIORITIES The purpose and objectives of EBT Prev are aligned with the 2009 GoM and PEPFAR Partnership Framework and the Partnership Framework Implementation Plan, both of which instruments are aligned with the GoM National HIV Prevention Strategy 2009–2013. The HIV Prevention Strategy prioritizes evidence-based, engendered, complementary prevention interventions and harmonizes biomedical and behavioral interventions. EBT Prev objectives reflect these core national strategic priorities, while project interventions are focused on the USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 3 following MARPs: CSWs, MSM, male and female plantation workers, fishermen and women in fishing communities and male and female vendors. Socially marketed condoms are made available to the general population. Project focus on MARPs has been informed by evidence from key studies such as the 2006 BBSS. The project development hypothesis is that to support reduction of HIV incidence, implementation of PSI’s EBT Prev combination prevention activities within the 20 project priority prevention areas (currently there are 18, with two to be added in 2012/13) will help reduce reported high-risk behaviors, including multiple concurrent partnerships; increase the use of condoms; and support responsive HIV-related services among MARPs. It is also expected that implementation of these activities will provide the GoM with effective models for scale-up of targeted prevention efforts for MARPs and vulnerable populations in other areas. The primary aim of EBT Prev has been to drill down into MARPs, conduct SBCC and inspire changes in behaviors. As such, the project must be evaluated in the context of the entire portfolio of HIV projects and programs in Malawi, e.g., the USAID-funded BRIDGE II project, the main focus of which is on general population coverage. EBT Prev works with the most vulnerable and marginalized groups, while its focus on enhancing MARPs’ choices for prevention and safe sex builds on the BRIDGE II-supported research by Watkins and others. In addition, the HIV epidemic in Malawi, as elsewhere, continues to evolve, which requires responsiveness to the realities of epidemiological trends. Since EBT Prev’s design in 2008, the project has been adapted to some degree, for example, by increasing attention on GBV and voluntary medical male circumcision. In August 2011, EBT Prev conducted a population size estimate exercise among male and female plantation workers, vendors and fishermen and women in fishing communities throughout seven of the project PPAs. The FPAM was then conducting its survey of CSWs (that report has since been disseminated); MSM population size is in the process of being estimated and has been completed in Blantyre. The population size estimate report states that findings will be used to calculate annual target percentages for each of the surveyed MARPs. The 2011 FPAM study estimated that there are 19,295 sex workers in Malawi, although it did not specify how this category was defined for the purposes of the study. It does not appear that more informal and transactional sex workers were included in the overall figure, which is, therefore, likely to be an underestimate. The project target for reaching commercial sex workers (Indicator P8.3.D) in FY 2011/2012, as set out in the Project Monitoring Plan, is 7,000, which undoubtedly represents a significant percentage of active sex workers in Malawi (and the project has exceeded that target). It is uncertain as to what number would represent a critical mass of sex workers that would have an impact on this important target population. The high turnover in this work and the mobility of this target population present significant challenges. The purpose of the Evidence-Based, Targeted HIV Prevention Project is to support the GoM’s National AIDS Framework goal of prevention for populations and settings in high-risk areas. NAF Objective 1.1 is to reduce new HIV infections in Malawi through sexual transmission. There are six intervention areas within this overall objective, which EBT Prev addresses through its four project objectives. The GoM-U.S. Government’s five-year Partnership Framework Implementation Plan (PFIP) supports the Partnership Framework 2009–2013. The PFIP sets out the key intervention areas 4 USAID/MALAWI: EBT Prev Mid-Term Evaluation and U.S. Government support and lists the new intervention priorities that have emerged since the finalization of the PFIP, e.g., prioritization of care for women and children and the strengthening of Malawi’s health delivery systems in both clinic and community settings. (For detailed discussion of project activities from inception in March 2009 to date (overall and year on year), please see Appendix G. Appendix H contains a Gantt chart, provided by PSI/M and Pact, which sets out a comprehensive schedule of activities, again from inception to date.) USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 5 2. EVALUATION APPROACH AND METHODS This section of the final report is drawn in part from the second and final iteration of the evaluation plan, itself informed by meetings held with USAID, PSI/M, Pact and the Malawi National AIDS Commission during the week of October 15, 2012; by USAID comments; and by a number of team planning meetings. This section is also based on other work in country by the evaluation team in preparation for fieldwork and analysis. (Please see Appendix E for a full, detailed discussion of the EBT Prev evaluation fieldwork approach and method (including sample details such as the numbers of focus group discussions and key informant interviews held, quantitative surveys administered and respondents interviewed). Moreover, see Appendix C for the final version of the evaluation plan. That appendix contains a map of the six PPAs visited in the course of evaluation fieldwork and information on the rationale for choosing the PPAs. This section of the report provides a brief summary of approach and methods.) The Evaluation SOW, Objectives and Questions As set out in the scope of work, the primary purpose of the EBT Prev mid-term evaluation was “to determine the extent to which the project’s behavior change and communications’ interventions have resulted in reduction of risky behavior among the Most-At-Risk￾Populations.” A secondary objective was to understand the extent to which the program approach and multiple components have been effective. To do so, the evaluation would address the thirteen evaluation questions, grouped under six main questions and by so doing evaluate the extent and efficacy of the project objectives through a performance evaluation approach. The mid-term evaluation was intended to: • Determine if the objectives as defined in the cooperative agreement and in relation to planned activities are being achieved and assess the likelihood of achieving them upon project completion, taking into account the perspectives of the stakeholders and beneficiaries. • Determine the strengths and weaknesses of the existing program and approach, explicitly determining why certain program components are working or not. • Provide concrete recommendations on any program adjustments to be made for the remainder of the project agreement. The Evaluation Questions [See Appendix F, as well as Section 3 below, for further discussion of the evaluation questions. See Appendix A (the scope of work) for background information on the evaluation questions and Appendix G for additional background information and an overview of EBT Prev activities (overall and year by year).] No additional evaluation questions were added during assignment negotiations. However, in the course of in-country evaluation work, USAID requested that particular attention be given to CEDEP management, both by the project and by its own internal processes. One key issue to be addressed was whether CEDEP internal processes and procedures are adequate to support MSM to undertake peer education. See 3.5 below for further consideration. 6 USAID/MALAWI: EBT Prev Mid-Term Evaluation The Evaluation Team • Team Leader: Janet Gruber • SBCC and Social Marketing Consultant: Iain McLellan • Local/HIV Consultant: John Kadzandira • Translator: Zione Themba • Logistics’ Coordinator: Tennyson Banda Evaluation Approach The overall objective of this mid-term performance evaluation was to provide an overview of the EBT Prev Project to date, using primarily qualitative methodology, with some limited quantitative collection of data through two surveys, administered to project partner outreach workers and project partner officers. Analysis of project data and reports was undertaken systematically and the evaluation team made full use of the PSI/M database, its ongoing surveys and continuous M&E activities, in order to validate and triangulate the team’s own findings. The team also and reviewed other relevant national and international documentation, e.g., on VMMC and PMTCT B+. The 2011 USAID evaluation policy definition of “performance evaluation” was used to shape and guide the EBT Prev mid-term evaluation. The evaluation team applied the following conceptual methods and approaches throughout its fieldwork, analysis and report-writing. (See Appendix E for full theoretical and methodological details of all approaches mentioned below (and see also Appendix C for the evaluation plan).) • An optimal balance of qualitative and quantitative methods • Triangulation • Participatory approaches • Gender-sensitive data collection and analysis Evaluation Methodology Data Collection Methods The evaluation team finalized focus group discussion guides and key informant interview guides, as well as two short quantitative surveys for use in undertaking fieldwork data collection. Tools were pre-tested in Lilongwe Old Town PPA on October 20, 2012. (For the full set of evaluation tools, see Appendix D.) Focus group discussions were held with IPC/Vs and at least two different project MARPs in each PPA visited by the evaluation team. In￾depth individual interviews and questionnaires were administered systematically in each PPA. USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 7 Qualitative Data Collection in the Six PPAs District PPA FGDs and KIIs Lilongwe Old Town MSM (KIIs) CSWs (FGDs) IPC/Vs (FGDs) IPC/As (KIIs) Outreach workers and other project stakeholders Mwanza Town and border CSWs (FGDs) Vendors (FGDs) Outreach workers and other project stakeholders Thyolo Satemwa Tea Estate Plantation workers (FGD) IPC/Vs (FGDs) IPC/As (KIIs) Outreach workers and other project stakeholders Zomba Songani Market Vendors (FGDs) IPC/Vs (KIIs) IPC/As (KIIs) Outreach workers and other project stakeholders Mangochi MALDECO Fishing community (FGDs) CSWs (FGDs) Outreach workers and other project stakeholders Nkhotakota Dwangwa Plantation workers (FGDs) Fishing community (FGDs) MSM (KIIs) Outreach workers and other project stakeholders Meetings were conducted with USAID, NAC, PSI, Pact, Johns Hopkins University BRIDGE II, UNAIDS and UNFPA, at which KII guides were used. It was not possible to have meetings with Ministry of Health representatives. The following discussions were conducted with individuals and groups across the six fieldwork PPAs. (For further disaggregation and details, see Appendices B, C, D and E. Appendix E contains two tables, the first of which provides full information on qualitative and quantitative data collection through FGDs, KIIs and the two quantitative surveys. Table 1 also lists where and with whom each FGD, KII and quantitative survey instrument was applied. Table 2 in Appendix E provides full details of the description/occupation and total number of EBT Prev mid-term evaluation respondents (there were 279 respondents—158 8 USAID/MALAWI: EBT Prev Mid-Term Evaluation men and 121 women).) • Eight key informant interviews with the following project partners: Malawi Human Rights Youth Network (MHRYN), CEDEP, Community Partnership for Relief and Development (COPRED), Christian Community Church (CCC), Namwera AIDS Coordinating Committee (NACC), Malawi AIDS Counseling and Resource Organisation (MACRO), Nkhotakota AIDS Society Organization (NASO), Society for Women with AIDS in Malawi (SWAM) and Theatre For a Change (TFaC). • One FGD with MACRO representatives. • Eight project partner quantitative questionnaires administered. • Seven FGDs with IPC/Vs; six FGDs with IPC/As; two FGDs with MSM peer educators (15 total FGDs). • Sixty-three project outreach worker quantitative questionnaires administered (33 to IPC/Vs, 15 to IPC/As and 15 to MSM peer educators). • Eleven KIIs held with hub health workers. • Fifteen FGDs conducted with MARP representatives: four with CSWs; four with plantation workers (two each with men and women); three with vendors (one each with men and women and one with a mixed group); four FGDs with fishermen and women living in fishing communities (two each with men and women). • Meetings with seven organizations in Lilongwe and Blantyre: USAID, PSI/M, Pact, NAC, BRIDGE II, UNAIDS, UNFPA. Data Analysis The evaluation data collected through the evaluation FGDs and KIIs have been analyzed and triangulated with project data (see Appendices C, D and E). Analysis and findings have served to strengthen responses to the evaluation questions as listed in the scope of work (see Appendix A). The evaluation team has applied the Convergent Mixed Methods Parallel Research Design (CPRD). Evaluation Limitations The evaluation team sought clarification in the week of October 22, 2012, from USAID/Malawi regarding Evaluation Questions 4, 4a and 5. Clarification was provided on November 5 and again during review of the draft report, i.e., in both instances after evaluation fieldwork had been completed. (Please see Section 3.3 below for a brief discussion of all evaluation question findings, conclusions and recommendations, including 4, 4a and 5. See Appendix F for further discussion of evaluation questions, including responses and clarifications provided by USAID/Malawi regarding 4, 4a and 5 and the approaches taken by the evaluation team in answering those questions.) As previously noted, it was not possible to meet MoH representatives during the evaluation. Despite several attempts, direct/participant observation of IPC/V or IPC/A sessions, whether with individuals or in small groups, could not be arranged. In addition, an FGD or KII with queen CSWs did not take place. These were all limitations of the evaluation. USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 9 3. EVALUATION FINDINGS This section discusses three aspects of project findings: Section 3.1 provides a strategic overview of project achievements and challenges/issues to date; Section 3.2 addresses achievement of each of the four currently operational project objectives; Section 3.3 briefly considers each of the thirteen evaluation questions set out in the SOW; Section 3.4 briefly addresses emerging, crosscutting themes; and Section 3.5 provides an additional review of CEDEP issues, as requested during in-country fieldwork by USAID. Note: Appendices It is strongly recommended that this section be read in conjunction with the appendices. In terms of planning and methodological approaches to fieldwork and analysis, Appendix C contains the full and final version of the evaluation plan, including details of methodology, Appendix D the qualitative and quantitative tools used during fieldwork, and Appendix E the evaluation fieldwork approach and methods (including sample details and rationale). A considerable amount of information on findings for the evaluation questions can be found in Appendices F and G. Appendix F contains further discussion of background to a number of the evaluation questions and consideration of limitations to Questions 4, 4a and 5. Appendix G covers additional information on project activities to date, including a year-on￾year overview of project work and background information for a number of the evaluation questions—intended to provide a historical perspective of project activities to date and contextualize ongoing activities. Appendix H is a Gantt chart created by PSI/M and Pact, with detailed, time- information on project interventions. Appendix I contains more project data for PEPFAR and other project performance indicators, as provided by PSI/M and Pact. The data cover progress toward indicators for FY 2009/2010 (in part), FY 2010/2011, FY 2011/2012 and cumulatively for 2010–2012. Appendix J is a schematic representation from PSI/M of how the newly instituted referral system is intended to work. 3.1 AN OVERVIEW OF PROJECT ACHIEVEMENTS AND CHALLENGES This overview of EBT Prev achievements to date, and existing and future challenges, is based on comments made during the course of the evaluation—primarily information and views gathered from meetings with government, donor and project actors. To a lesser extent, the overview includes points that emerged from fieldwork qualitative and quantitative data analysis and from triangulation with project and other documentation. Section 3.1 in large part represents a more strategic and general overview of project achievements, challenges and issues as perceived by those working in the HIV field in Malawi, who may not always have been directly involved in EBT Prev implementation. National-Level Views These are views expressed to the mid-term evaluation team in the course of its meetings and fieldwork. Evaluation team members have analyzed all such data and have also triangulated comments where possible, in light of mid-term evaluation findings. Achievements 10 USAID/MALAWI: EBT Prev Mid-Term Evaluation • The project is acknowledged by development partners to have: o Largely achieved its performance targets to date. o Supported a degree of behavior change among MARPs, especially with regard to the desires to seek HTC, use condoms and improve couple communication. • PSI/M is considered by government and development partners to be a key strategic player in Malawi in terms of provision of socially marketed condoms. • PSI/M is also seen by development partners as one of the leaders in the introduction of VMMC to Malawi. • PSI/M is a member of the Technical Working Group on HIV and it was a member of the steering committee that developed the 2009 National HIV Prevention Strategy. All such contributions are valued. • The role of PSI/M in supporting evaluation effectiveness, e.g., in terms of developing outcome indicators, is being considered at the GoM level. • PSI/M is considered to be innovative in terms of working with otherwise neglected MARPs such as CSWs and MSM and its active engagement in plans for mobile VMMC. • Pact is acknowledged as a national leader in community development and support. • The project is considered to have helped inform the national debate on and approach to HIV prevention. • PSI/M’s technical expertise in outreach and interpersonal communication has worked well with Pact’s experience and skills in supporting the network of NGO partners. • Socially marketed condoms have had a gradual and regular growth in sales and have allowed for continued availability during times of stock-outs of free condoms, especially in PPAs. Challenges/Issues • The national strategy draws attention to the importance of addressing MARPs in the context of a “mixed epidemic” in Malawi. The question is: How best will the project inform that debate and future policy development? • PSI should be more consultative and informative about its activities at the district level and below. • There is concern about “reach” and numbers and what these actually mean in terms of support for sustained behavior change and having an impact on the epidemic. • Despite the project being entitled “evidence-based,” there is a perception that only limited data have been provided by the project. Given the project’s focus on targeting MARPs, it is important that data, including lessons learned and emerging best practices, are received on an ongoing basis, in order to inform scale-up of prevention activities. • With regard to issues of gender-based violence, some project activities go beyond the provisions contained in governmental and legal. It also is important not only to address the open manifestations of violence (primarily physical abuse) that constitute GBV, but also to address the societal norms and power structures underpinning it. • Problems with distribution of free condoms and other commodities such as HIV test kits have been significant barriers to achieving project interventions. USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 11 Project Level Discussion at the project level on achievements and challenges has similarly been developed through meetings and fieldwork—in this context primarily with project actors (PSI/M, Pact, project partners such as CCC and project outreach workers). It is also derived from review of project and other documentation and analysis of evaluation data and, to a lesser extent, discussion with development and government partners. The points that follow are informed by insiders’ perspectives and describe honest reflection on activities to date. Achievements • EBT Prev has achieved—and surpassed—most of its indicator targets that are measured annually. • The project has contributed to behavior change among its target MARPs: there are trends indicating more regular and sustained condom use, greater willingness to seek HTC andgreater couple communication and potential reduction of multiple concurrent partnerships (MCP). • EBT Prev has developed a potentially critical national resource in terms of committed, trained volunteers able to provide support to HIV prevention and the network of non￾governmental organizations (project partners) and community links that has resulted. • Project partners have further developed capacity (e.g., in outreach, managing large numbers of volunteers and engaging with the health system) and have leveraged resource mobilization for additional national and international funding. Challenges/Issues • Despite the project being evidence-based, its purpose/outcome indicators are only to be measured twice during the project lifetime, which is likely to be insufficient in terms of tracking trends and influences on behavior change. • The project does not sufficiently provide feedback on data collected by its IPC/Vs, IPC/As and peer educators. This is widely considered to reduce outreach workers’ efficacy and fine-tuning of messages, as well as limiting opportunities for PPA and MARP￾specific messaging. • How best to measure “reach,” and the degree to which the reach inspired positive changes in behavior, need examination. For example, the impact of the reluctance of many MARPs to sign for second IPC sessions, the mobility of many MARPs and the degree of efficacy and recall of TOC and drama on target populations is at present unclear. • There has been no project measurement of the degree to which individuals reached may have initiated and sustained HIV behavior change, or of the trigger/s that led them to do so (e.g., thanks to well-maintained contact with IPC/Vs, effective HTC experience, etc.). • There has been no project measurement of demand-side (i.e., MARPs’) views of the quality of services or the fitness for purpose of SBCC interventions. • SBCC interventions as presented through IPC and quarterly messaging have not to date sufficiently prioritized and integrated focus on service uptake/HTC as key entry points to portfolio of health and other support services. • PSI/M staffing dedicated to SBCC has been insufficient to ensure that strategies were 12 USAID/MALAWI: EBT Prev Mid-Term Evaluation developed that call for the development of interactive support materials to be used by outreach workers and other communication channels. • Quarter 4 messaging on GBV was insufficiently promoted as an element of a comprehensive, combination (i.e., linking biomedical, behavioral and structural interventions) HIV prevention approach. This has implications for Quarter 5–7 SBCC messaging and also for VMMC (e.g., ensuring links between VMMC and uptake of HTC). • There is insufficient expertise on gender and HIV within the project. Opportunities have been missed regarding GBV, technical inputs required with respect to VMMC and Quarterly Messages 5–7. • The project has insufficiently targeted CSWs and MSM. 3.2 ACHIEVEMENT TO DATE OF EBT PREV OBJECTIVES This section of the report responds to the requirement in the SOW that states: Determine if the objectives as defined in the cooperative agreement and in relation to planned activities are being achieved and assess the likelihood of achieving them upon project completion, taking into account the perspectives of the stakeholders and beneficiaries. This part of the evaluation report is informed by review of project, USAID, PEPFAR and national documentation, e.g., the 2008 PSI and Pact RFA application, project annual workplans, quarterly and annual reports, the PFIP, the PEPFAR 2011 guidance on MSM and the 2011 UNICEF, UNAIDS and PEPFAR Prevention Partners’ Trip Report (for details of all documents, see Appendix L). It is relevant here to mention that the EBT Prev Project has become further developed since its first planning stages in 2008 and its actual implementation in March 2009. Such shifts are not unique to this project, but are common and demonstrate responsiveness, especially in projects with a lifetime of five years in the context of an evolving epidemic. Changes to EBT Prev generally reflect shifts in emphasis and increased prioritization of certain HIV prevention approaches, such as VMMC (which has come to the fore since 2008/2009) and PMTCT B+ (in which Malawi is an acknowledged leader), as well as an enhanced focus on gender issues in the context of the epidemic, increased attention to ART and opportunities to build on evidence from increasing numbers of national and regional HIV research studies, including those conducted by PSI/M and its project partners. The EBT Prev Project description focused particularly on concurrency and condom use. Inclusion of the new issues and development of messaging to support SBCC is said to have required considerable change of emphasis and approach by PSI/M and Pact. PSI/M and Pact have also brought changes to EBT Prev by requesting major alterations to the project referral system (see Section 3 specific to Evaluation Questions 5 and 5a for detailed consideration), which were approved in early 2012. The new referral system has been operational since June 2012. An overarching objective of EBT Prev from inception has been to reduce concurrent and casual partnerships and to increase condom use among most-at-risk and other vulnerable populations in selected PPAs. Objective 1: Identify, Segment and Profile Priority Populations at Risk USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 13 (See also discussion of Evaluation Question 3 in Section 3.3 and elsewhere in that section.) This objective has been largely achieved, with Priorities for Local AIDS Control Efforts (PLACE), condom-mapping and service delivery research being conducted. In 2010, PSI/M undertook a cross-sectional baseline survey designed to monitor trends in condom use and concurrent partnerships among male and female vendors, plantation workers and members of fishing communities; findings from this survey assisted development of purpose/output indicators. All 2009/2010 baseline findings have informed EBT Prev activities. However, the baseline took longer than expected. As a result, the project pre-empted its findings and began pilot activities in Dwangwa PPA in 2009. Tracking Results Continuously (TraC) research was undertaken in FY 2010, with the objective of shaping development of targeted messages to be presented through IPC to MARPs. In 2011, PSI/M and USAID conducted a “population size estimate exercise” in selected EBT Prev PPAs (seven in total) to calculate the number of vendor, plantation worker and fishing community MARPs. Purpose/outcome level indicators are to be measured twice during the lifetime of the project—at baseline and endline. In addition, a number of Objective 2 and 3 indicators are to have their baselines set through a 2012 TRaC general population study, e.g., Number and percentage of people that have been exposed to at least one radio spot, episode or program or through other means, such as baseline denominator information. All Objective 5 targets are to be set in 2012/2013. An exposure study will be conducted in 2013 to assess how MARPs may have benefited from project SBCC, condom and referral activities. The same five PPAs that participated in the baseline study have been selected, in order to ensure fidelity of research. Objective 2: Deliver integrated, behavior change communication programs targeted to high-risk populations in priority prevention areas [See also discussions in Section 1 regarding the degree of integration of SBCC messaging. Additionally see Section 3.3 for consideration of the following SBCC-relevant evaluation questions: 1, 1a, 1b and 1c (for overall SBCC); 2 (for VMMC and GBV); 3 (for M&E, e.g., definitions and measurement of “reach”); 4 and 4a (for project partner and volunteer contributions); and 5 and 5a (community and referral systems).] EBT Prev has succeeded in part in implementation of Objective 2 in terms of reaching agreed targets for reaching MARPs in its PPAs, as measured under P8.3.D for male and female plantation workers, fishing community members and vendors. The one MARP that has seen consistent underperformance in terms of project reach and SBCC engagement is MSM (in this context see also Section 3.5). CSWs represent another challenging MARP, while all groups present both unique characteristics and shared ones such as mobility. Achievement of targets for PEPFAR Indicator P8.1.D did not occur in FY 2010 and FY 2011, although targets have been achieved for FY 2012 and the aggregate project target to date has now been overtaken. This indicator addresses preventive interventions to be undertaken on an individual or small-group basis and through TOC and drama with MARPs. This core project activity depends on the work of volunteer outreach workers—the IPC/Vs—and on support from waged, full-time IPC/As, project managers and officers. Of 560 outreach workers trained since the start of the project, 506 are currently active. IPC/Vs receive both basic, quarterly messaging and refresher training, representing a significant investment of PSI/M efforts. There are ten project partners—eight local NGOs such as NACC and MHRYN and two national NGOs, MACRO and CEDEP. TFaC has completed its 14 USAID/MALAWI: EBT Prev Mid-Term Evaluation engagement with the project. IPC/Vs work on average two days a week on behalf of the project, delivering individual and small-group IPC based on the four quarterly messages to date. These messages as stated by PSI are: Quarter 1 (Q1)—introduction of choices; Quarter 2 (Q2)—choices in terms of HIV prevention; Quarter 3 (Q3)—skills training; and Quarter 4 (Q4)—GBV. It is relevant to point out that IPC/Vs, other outreach workers and MARP respondents seldom gave the same answer when asked to describe the four quarters’ messages. The intention is that the Quarter 5 (Q5) message will be PMTCT, Quarter 6 (Q6) will be family planning and Quarter 7 (Q7) will be HTC. The overarching branding and message for project SBCC is Lingalira sankha wekha. Targeted outreach communication, drama and other behavior change communication (BCC) activities complement the core quarterly messaging approach. One challenge has been the staggered introduction of the quarterly messages, due to the fact that the project added new PPAs year on year (the final two are to be added in 2012/2013). Another challenge has undoubtedly been the shift in project SBCC emphasis toward a more service delivery focus. A third and major, challenge has been the inadequate supply of support materials to IPC/Vs. Questions remain regarding the capacity of outreach workers to continue to absorb new intervention content, especially if resources dedicated to training in new quarterly themes are limited and support materials are not produced. There is evidence that some confusion remains among outreach workers concerning the messages for the first four themes that were introduced—this is particularly problematic considering that EBT Prev now intends to introduce additional content (PMTCT, FP and further promotion of HTC), all with their own inherent complexities. The intention is also to build on previous quarters’ themes. It remains to be seen if there is a limit to the capacity of the outreach systems in introducing complex new content as well as effectively communicating existing themes. An overarching SBCC strategy and accompanying reference guide and support materials covering all themes would improve the quality of communications. Objective 3: Distribute and promote condoms for use by the general population and for high-risk groups (See also Section 3.3 for consideration of Evaluation Questions 6 and 6a, both of which address condom issues.) PSI/M has a well-established social marketing network that enjoys consistent annual growth in terms of sales and is constantly increasing sales outlets, especially in the PPAs where there has been good collaboration between those socially marketing condoms and EBT Prev Project partners. There have been considerably more difficulties in the distribution of free condoms to the general population and to MARPs. It is unclear what exactly has contributed to the frequent stock-outs in the distribution system, but they have made it difficult for EBT Prev outreach workers to provide enough condoms to meet the needs of the target populations they serve. Furthermore, some might argue that even if a more efficient distribution system existed, the number of condoms coming into Malawi do not come remotely near to meeting the condom needs of sexually active Malawians. Objective 4: Enhance the network of existing providers for greater accessibility and services to high-risk populations (See also discussion in Section 3.3 of Evaluation Questions 4, 4a and especially 5 and 5a. See USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 15 Appendix J for a schematic representation of the new referral system.) Between 2009 and January 2012, the project implemented a referral system that focused on the role of community-based organizations (CBOs) within each of the PPAs in operation. Each PPA referral network numbered upwards of 30 partners, including CBOs, public health facilities and non-state health providers (Christian Health Association, Banja La Mtsogolo, etc.)—a total of nearly 500 CBOs. A comprehensive referral directory was developed by the project, as were referral tools. Members of MARPs (and other community members) could choose which health facility they attended and for which service. This proliferation of potential and actual referral agents and organizations, service delivery facilities and records resulted in an over-intricate and burdensome system, which was not adequately delivering the objective. Project data show that under the old referral system uptake of the first service to which an individual was referred was less than ideal at 51% (Indicator 4.5), representing 398 of 781 individuals referred. The aggregate percentage for referral specific to HTC stood at 71.2% for FY 2011/2012 (P11.1D). It does not appear that the project was able to follow up on why such a relatively small percentage of clients took up referrals. EBT Prev requested and was granted by USAID an opportunity to revise the referral system, as from January 2012. For the next six months, PSI/M in particular focused on developing the new system, training health workers, project partners and project outreach workers and starting actual implementation. (See Section 3.3, Question 2 for consideration of the ramifications of this focus in terms of PSI/M’s overall management capacity.) Since June 2012, project volunteers are the sole referral agents and refer MARPs to a more limited number of “hub” health facilities (most PPAs have one such hub, while a few have two and Dwangwa has three). CBOs retain a degree of engagement with the new referral system, in that hubs retain the referral directory and can point MARP clients in the direction of appropriate CBOs, e.g., those offering psychosocial and HIV support. Project verbal information is that since the introduction of the new referral system in July 2012, 773 people have been referred across all 18 PPAs. The information that has been made available on how many individuals may have taken up their referrals has been insufficient, considering that it has not been disaggregated (e.g., the sex and category of each person, such as a vendor, the type of service to which the person was referred, or the PPA where the person is resident.) Objective 5: Voluntary male medical circumcision (VMCC) service delivery in Thyolo and Blantyre districts and associated demand creation in Blantyre district This objective was added in 2012. Five performance indicators are to be measured, including two to be addressed through PEPFAR Indicator P5.1.D. VMMC is costed at $4,128,026, 16.6% of the total project budget over its lifetime. The project VMMC component has not yet become operational in terms of clinical interventions, although planning and recruitment have been initiated. Clinical activities are now likely to start in February 2013. Training of health workers is scheduled to begin in November 2012 and will include the participatory development of quality assurance, standard operating procedures and a comprehensive implementation plan. PSI/Malawi will be 16 USAID/MALAWI: EBT Prev Mid-Term Evaluation supported in this training by its head office staff and by PSI/Zimbabwe, where VMMC has already been implemented. Completed activities include identification of and recruitment for the twelve health worker and support staff positions for each team that will conduct VMMC funded under EBT Prev. There are four such teams. 3.3 EVALUATION QUESTIONS 1–6 This section of the report is structured as follows: each of the thirteen evaluation questions and sub-questions is addressed in turn. Findings, conclusions and recommendations are provided individually for eleven of the thirteen questions, so as to provide optimal detail and a clear overview of evaluation processes.1 The exceptions are Question 2a (covering VMCC, which is not yet operational), Question 5 (because it is primarily historical) and Question 5a (partially). This section of the report (and also Appendix F) provides descriptive and normative responses to the thirteen evaluation questions: what EBT Prev has achieved by this mid￾term evaluation since its work began in March 2009; how the project is being implemented and for whose benefit; how it is perceived and valued by project stakeholders (MARPs, project outreach workers, project partners, health workers connected to project referral activities, PSI/M and Pact, GoM and donor partners); whether and how expected results have been achieved; issues of project design, operational management, decision-making and capacity and quality assurance. Section 3.5 addresses an additional question—that of CEDEP participation in EBT Prev. This was added at the request of USAID/Malawi during in-country evaluation work. As previously mentioned, this section of the report is amplified by a number of other appendices, particularly Appendices C, D, F, G and H and review of these is recommended. Please note that a number of the evaluation questions gauge progress against indicators; these were selected during evaluation planning. As a result of fieldwork and discussion in Malawi and a few of the indicators (e.g., for GBV in Question 2b) have been removed as inappropriate for measuring progress against the question. Appendix I contains much additional raw data on project PEPFAR and other indicators, provided to the evaluation team by PSI/M and Pact. Some 2009 data are included, as are both aggregate and disaggregated annual and cumulative data for FY 2010–2012. QUESTION 1: How effective have PSI’s EBT Prev Communications approach and Targeted Outreach Communication (TOC) been in supporting adoption of safer sexual behaviors among project target groups? Question 1. Indicator 5 (1st)—new PEPFAR Indicator P8.3.D: Number of MARPs reached with individual and/or small group level interventions that are based on evidence. See also discussion under Evaluation Question 3, below, with regard to definition of reach and linked challenges. Findings All findings for Evaluation Questions 1, 1a, 1b and 1c were collected through analysis of the 16 FGDs conducted with MARP representatives, the FGDs and KIIs with project outreach workers and KIIs with project partner representatives. In addition, the 63 project outreach 1 The evaluation team was guided in its structuring of this section by the USAID TIPS document entitled “Constructing an Evaluation Report.” USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 17 worker quantitative questionnaires were reviewed and there was opportunity for direct observation of drama and TOC and for examination of project support materials. Moreover, these four evaluation questions were discussed in meetings with PSI/M, Pact, USAID and other organizations. 1. P8.3D Targets for Reaching All Target Populations Exceeded, Except for MSM With regard to P8.3.D, EBT Prev has exceeded its annual targets for reaching all but one target population, according to project reporting. The exception is MSM. For vendors, fishing communities and plantation workers, a total of 145,883 people were reached with project messages at least twice in individual or small-group settings in FY 2010, 2011 and 2012—well above the target of 102,250. The cumulative achievement rate for FY 2010–2012 for these three target populations is 141.3% (see Table 3). A total of 12,785 CSWs were reached, compared to the target of 11,550, for a cumulative achievement rate of 110.7%. MSM proved to be a more challenging target population for the project—1,100 MSM were reached, below the target of 1,800, representing a cumulative achievement rate of 63.9%. Table 3: EBT Prev FY 2010–2012 Cumulative Achievements Against Targets (PEPFAR Indicator P8.3.D) INDICATOR P8.3.D TOTAL CLIENTS REACHED Target Cumulative Achievement Cumulative Percentage Vendors + fishing community + plantation workers (male and female, aggregate) 103,250 145,882 141.3% CSWs 11,550 12,785 110.7% MSM 1,800 1,150 63.9% Overall 116,600 159,817 137.1% 2. Target Population and PPA Selections Based on Solid Baseline Data and Mapping Interventions have been focused on target populations in PPAs that were chosen based on research conducted in five PPAs in 2009, during the pilot phase of the project. The PSI PLACE (Priorities for Local AIDS Control Efforts) methodology was applied in order to identify places where people meet new sexual partners. The mapping of condom outlets and HIV service delivery points was also undertaken and condom sales outlets have been expanded during the project lifetime into 18 functioning PPAs. The PPAs are distributed throughout Malawi and include border areas, plantations, fishing communities and urban and rural market areas. The document entitled Report of the Population Size Estimate Exercise in Selected PPAs analyzes the size and location of three project target populations: vendors, fishing communities and plantation workers. CSWs and MSM were not included. This report, produced in 2011, was conducted in seven out of 17 active PPAs. While the methodology is solid and the populations identified clearly defined and located, questions remain concerning the actual 18 USAID/MALAWI: EBT Prev Mid-Term Evaluation percentages of each target population that are to be reached by interventions. The estimated percentage of vendors to be reached was almost 100%, but the estimated percentages of plantation workers (46–62%) and fishing communities (14–20%) were much lower in the PPAs where they were targeted. 3. Project Partners Have Established a Network of IPC Volunteers Project partners contracted to develop and implement EBT Prev were selected because the majority were already active in PPA districts. Eight of the eleven project partners involved to date had previously been vetted and trained by Pact under the earlier REACH project. This accelerated the start-up time for PPA community IPC interventions. With technical support from Pact and PSI/M, project partners have established a network across the 18 PPAs, comprising project managers, project officers, IPC/As and IPC/Vs. The retention rate of project outreach workers is high: of the 560 who underwent training (332 males and 228 females), 506 remain active (see Table 4). A drop-out rate of only 3.9% is remarkable, considering that only project managers, officers and IPC/As are compensated for their project work (See further discussion below on Evaluation Questions 4 and 4a regarding outreach workers’ views on their work.). A common theme to emerge during discussions with project partners, as well as with outreach workers, was a deep sense of satisfaction in participating in HIV activities. It was also noteworthy that very few (perhaps 5%) of IPC/Vs made any mention of per diem, seating allowance, or other such incentives. This is unusual in that evidence from many other projects, however well-managed and fulfilling these might be for outreach workers, often reveals far more reference from volunteers to their need for some type of compensation. In the FGDs held with IPC/Vs, the two most common reasons cited for volunteering were contributing to their community and helping others and improving their professional skills through project training and work experience. The bicycles provided to the IPC/Vs and IPC/As were appreciated according to those interviewed. However, all of those interviewed reported their bicycles were in need of repairs and were not usable at present. Table 4: Number of Outreach Workers Trained and Retained (H2.3.D) Total number of IPC/Vs, IPC/As and project officers trained since inception of the project 560 Males 332 Females 228 Total IPC/Vs, IPC/As and project officers trained since inception of the project who are currently active 506 Male 307 Female 199 4. IPC/Vs Complemented by Targeted Outreach Communication and Local Dramas Four mobile targeted outreach communication (TOC) units are used by EBT Prev to visit PPAs to conduct “community-wide events,” in the form of stage shows and interactive video screenings. Local drama groups were engaged or created in each of the PPAs and trained USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 19 along with selected IPC/Vs to prepare and perform dramas related to the quarterly themes (the current quarterly message (Q4) is GBV). These community-wide events were viewed by 282,313 people in FY 2011/2012, which is slightly below the target of 308,000. The drama groups performed before 96,286 people, against a target of 38,700 for the same period. There are indications that the TOC events have exceeded targets more recently—43,200 people were reached with TOC events against a quarterly target of 40,000, as reported in the EBT Prev Quarterly Progress Report for Quarter 3, FY 2011/2012. The TOCs and dramas complement sessions conducted by IPC/Vs and IPC/As. However, beyond the TOC and drama teams introducing IPC/Vs at the events and performances, outreach workers tend to be underused. The IPC/Vs and IPC/As present at TOC events and dramas could have been put to better use mingling with the public and stimulating discussion and dialogue, particularly during or after dramas. The TOCs have been able to engage community participation by interviewing local community members, including victim support unit (VSU) police officers speaking about GBV and answering questions from the public. Gender-based violence is the Quarter 4 message theme and was the focus of the TOCs and dramas during the time the evaluation was conducted (late October–early November 2012). Half of the MARPs who participated in the evaluation FGDs said they had seen either the local drama or had had some participation in a TOC. Comments made included, “Wanted more,” “Enjoyed the big crowd,” and “Finally talking about GBV.” Respondents noted that the dramas and events “started a discussion” and that community members were asked to “come up afterwards to ask questions.” 5. Interventions Handicapped by Lack of Support Materials IPC/Vs, IPC/As and MSM peer educators (and CSW queen peer educators, although none was interviewed during the evaluation) have been given various materials to guide discussion on the quarterly themed messages (four to date). They include guides on how to conduct IPC for each quarterly theme; a flannelogram featuring drawings of different men and women to illustrate multiple concurrent partnership webs; a problem or decision tree (described variously); and a Bao traditional board game, which was used to illustrate behavior choices. Almost all members of the MARPs interviewed during evaluation fieldwork recalled seeing the flannelogram. Comments included “We find it important to see real people.” Respondents were able to see the impact of sex with “combinations of people.” The IPC/Vs found the flannelogram “attractive,” “appropriate,” and said that it “stimulated dialogue.” The Bao game was appreciated particularly by men. The guides include illustrative stories that are related to the lives of men and women in fishing communities, vendors and plantation workers; these have also been used to stimulate discussion. However, a detailed analysis of the quarterly guides by the evaluation team found few suggestions of questions to be asked to stimulate discussion. The scenarios for different MARPs are well crafted but little in number. The majority of the IPC volunteers (96%), assistants (94%) and MSM peer educators (72%) who completed the outreach worker quantitative survey felt the existing materials were “easy to understand and use.” The total number of respondents was 63; of these 33 were IPC/Vs, 15 were IPC/As and 15 were MSM peer educators. 20 USAID/MALAWI: EBT Prev Mid-Term Evaluation Despite these successes, outreach workers also felt strongly (IPC/V 56%, 64% IPC/A, 52% MSM peer educators) that project materials have never been “sufficient in number for me to conduct my work.” This view was also reflected in the qualitative fieldwork done with outreach workers including volunteers, assistants and peer educators. In each of the FGDs held with outreach workers there were requests for additional support materials to “make our work easier” and “photos to stimulate a dialogue.” Support materials in the form of flip charts and picture codes serve to guide outreach workers, keep them on message and increase participation. PSI/M could have adapted IPC support materials produced by its sister organization, Society for Family Health in Nigeria. Picture codes produced in Botswana by Pact and replicated across southern Africa could also have been easily adapted, given that these materials address the same issues and target populations and were developed in a participatory fashion with members of MARPs. 6. Progress Made with Enhanced Couple Communication Life choices, prevention options, skills-building and GBV are the four quarterly themes communicated to date by the project. When FGD respondents (MARPs, IPCV/A, MSM peer educators) were asked to list the quarterly themes, none was able to do so correctly and in the right order. The only theme that was clearly identified was the most recent one on GBV. One love, the problem tree, Linga lira (choices), multiple concurrent partnerships, condom use, partner reduction and sexual webs—these were all topics cited by the respondents as quarterly messages discussed in IPC sessions. The lack of distinction between the different themes could be attributable in part to the project goal that each successive theme build on earlier ones. Despite the confusion regarding the different themes, FGDs conducted with plantation workers, fishing community members and vendors in particular revealed some distinctive patterns of positive behavior changes. Improvement in couple communication was cited in the majority of the MARP FGDs (the total number of MARP FGDs was 16, the total number of female MARP respondents was 71 and the total number of male respondents was 72). When asked to cite EBT Prev influences on their lives, 25% of all male and female respondents mentioned increased openness in their couple communication and being more able to talk about sexual issues. Male comments included “Better friends with my wife,” “We didn’t talk about how we feel and now we do,” and “I have an appetite for my wife and vice versa now.” Female comments included “I have made positive changes because I don’t want my husband to cheat,” “I experience less GBV now,” and “Things change with communication.” These positive changes were confirmed by IPC/Vs and IPC/As, who found that being in a position to talk with husbands and wives inspired the changes. Awareness of the impact of multiple concurrent partnerships was mentioned 18% of the time in the FGDs. “That was an eye opener,” “I stopped having a girlfriend in each market,” and “I try to control sexual urges” were typical comments. Behavior choices were mentioned 16% of the time across the seven FGDs, followed by HIV testing and condom use (5% each). Conclusions Insufficient interactive support materials on all themes to be used in IPC made message delivery less effective. • Underuse of radio was a missed opportunity for expanding reach and support for IPC. • Lack of support for the repair of project-donated bicycles reduced IPC/V/A mobility. USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 21 • The role of police VSUs should be encouraged through greater integration into project activities. Recommendations Create targeted support materials for use by IPC outreach workers in order to increase participation, stimulate discussion and keep on message. • Enhance negotiation skills of target populations through role-playing and other approaches that increase skills in couple communication. • Provide support for project IPC/V/A bicycle maintenance. QUESTION 1A: Are communications activities adequately tailored to the different categories of populations among the target groups—including commercial sex workers and MSM (i.e., responsive to specific barriers faced and sufficiently skills-oriented to support behavior change)? Question 1a. Indicator from Objective 2.5 (1st)—new PEPFAR Indicator P8.1.D: Number of the targeted population reached with individual and/or small group level preventive interventions that are based on evidence and/or meet the minimum standards required. (As with P8.3.D, see also discussion under Evaluation Question 3 below, with regard to definition of reach and linked challenges.) Findings 1. Targets for P8.1.D Now Being Achieved Achievement of targets for PEPFAR Indicator P8.1.D did not occur in FY 2010 and FY 2011, although targets have been achieved for FY 2012 and the aggregate project target to date has now been overtaken. This positive change in achievement is noteworthy, given that the indicator addresses preventive interventions to be undertaken on an individual or small￾group basis and through TOC and drama with MARPs. Table 5 shows the cumulative achievement of P8.1.D since 2010, while Table 6 shows more disaggregated (and limited) progress in FY 2010/2011. 22 USAID/MALAWI: EBT Prev Mid-Term Evaluation Table 5: FY 2010, 2011 and 2012 Aggregate Achievement of P8.1.D Target Result P8.1.D FY 2010 25,000 12,353 FY 2011 30,000 25,447 FY 2012 33,500 21,293 Total 88,500 59,093 Table 6: FY 2010/2011 P8.1D Percentage of Clients Reached Note: MSM were not included in the 2010/11 analysis of clients reached, because mapping had not been completed. 2. Communications’ Activities Responsive to Plantation Workers, Fishing Communities, Market Vendors Data from FGDs with male and female plantation workers, fishing community members and market vendors revealed an appreciation of the content delivered during the four quarters. All groups said they found the messages relevant and appropriate to their lives. The messages on couple communication were particularly appreciated and a considerable number of respondents found the messaging on multiple concurrent partnerships and how sexual networks increase the risk of HIV infection informative and helpful. 3. Strong Identification with Drama, TOC Videos, IPC Scenarios and Flannelogram Images Again, the male and female plantation workers, fishing communities and market vendors who had been exposed to the content of the dramas, the TOC videos and the IPC scenarios found them to reflect their own situations and and described the scenarios as realistic. When asked in FGDs if the individuals depicted were like them or like people living around FY 2010/2011 Achievement Rate Fishing communities 76.2% Plantation workers 69.5% CSWs 31.8% Overall 60.8% USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 23 them, the answer was overwhelmingly positive in both rural and urban areas and among both sexes. 4. Communication Activities Not Sufficiently Reflective of CSW and MSM Realities The 23 women participating in the four CSW FGDs found the content regarding condoms, HTC, behavior choices and problem resolution to be useful. Messaging on multiple concurrent partnerships and couple communication messages was not considered relevant to this group. The GBV content was not adjusted to the challenges the women face from their male clients, although the topic was felt to be relevant. The majority of MSM peer educators in the two FGDs said that messages and materials must be better tailored to the realities of their own lives and those of the men they target. A nine-page guide designed to encourage MSM to undergo HTC was created by the project. Its primary objective is to get MSM to conduct a personal risk assessment by having peer educators ask them 17 questions regarding risky behavior. Regrettably, no other support materials to date have been developed specifically for MSM. 5. IPC/Vs and IPC/As Are Able to Relate Messages to Lives of Target Populations and Engage in Participatory Discussion of Issues The project has built participatory methodologies into its IPC work, targeted outreach communication and dramas in order to engage with and involve target populations. Systems are in place to support IPC/As to encourage IPC/Vs to stimulate more discussion, should their sessions be considered insufficiently participatory. IPC/Vs and IPC/As were able to focus discussions effectively on specific problems without being judgmental or moralistic in all FGDs with all the MARPs (including those with CSWs). Conclusions Some progress has been made in using discreet networking by peer educators to reach MSM, but many obstacles, including legal questions and stigma, make achieving targets difficult. • Project support materials were appreciated by target populations, but the contents covered were inadequate to support interpersonal communications. • Despite a lack of support materials, the IPC outreach workers communicated well with target populations and were appreciated by those they reached. • Links between IPC/Vs and IPC/As and TOC, local drama and mobile testing could be improved to benefit from the enhanced access to the target populations and to strengthen local follow-up. Recommendations Evaluation findings and recent country data indicate that discordant couples are an increasingly important MARP. The project should consider opportunities for targeting this group, as was originally planned in the project proposal. • Complement messages on reducing concurrent sexual partners, condom use and negotiation with messages on discordance, disclosure to partners and positive prevention. • Involve IPC/Vs and IPC/As more in the production of the TOC, local drama and mobile testing to create a greater synergy between all such events and local outreach. Outreach workers could be identified with brightly colored T-shirts and scattered among audiences to answer questions or hand out materials or condoms. IPC/Vs and IPC\As 24 USAID/MALAWI: EBT Prev Mid-Term Evaluation could facilitate discussions following drama presentations. QUESTION 1B: To what extent has the frequency in which all target populations have been reached with messages supported the adoption of safer behaviors, including increased demand for and use of appropriate HIV and reproductive health services and commodities? Question 1b indicator. For target MARPs generally, Indicator 1a (Purpose/Outcome level): Reported condom use at last sex with non-spousal, non-cohabiting partner among general population. This indicator cannot be discussed here, because it is to be measured only twice during the lifetime of the project, at baseline and endline. For RH services, Indicator 4 (1): Number of people referred to health services and/or HIV-related services by Pact partner. Data not provided against this indicator. For health commodities, Indicator 3 (1a): Increased penetration of male and female condoms in high-risk sales outlets within targeted PPAs. Findings 1. Evidence of the Creation of Demand for Services by IPC, TOC, Dramas No evidence is currently available linking the frequency and reach of project messages with the adoption of safer sex behaviors and the creation of demand for HIV and reproductive health services. There is evidence of 21,370 individuals who received HTC and received their test results in 2011/2012. This number was 71.2% of the target of 30,000. The evaluation qualitative data reveal that members of all MARPs have been influenced by project activities to seek HTC. Learning about the advantages of early detection and access to ARVs from outreach workers was a motivating factor that was mentioned in four FGDs. The convenience of the mobile testing services provided by the project partner MACRO was brought up in three FGDs. In one FGD with women from a fishing community, six said they had undergone HTC at the MACRO mobile clinic. 2. Lack of HIV Testing Kits and STI Treatment Drugs and Condom Stock-Outs, Resulted in Low Uptake Rate There is evidence that in FY 2011/2012, 781 referrals were made by project outreach workers, but only 398 individuals received a service—a general referral uptake rate of 51%. Most of the referrals were made for HTC and STI services and family planning. The absence of HIV testing kits and STI treatment drugs accounted in large part for the low uptake rate. There is no project evidence of the impact of stock-outs of free condoms on condom use, but there were reports during MARP FGDs in four of the six PPAs visited of difficulties in getting free condoms and not being able to afford available socially marketed condoms. 3. Increased Penetration of Male and Female Condoms in High-Risk Sales Outlets Within Targeted PPAs Through PSI/M’s method of increasing sales points (its Rapid Outlet Creation), a total of 113,756 condoms have been sold in the EBT Prev PPAs. This approach allows any commercial outlet, including small kiosks and bottle stores, to sell socially marketed condoms. The bottle stores and kiosks permit the sales of condoms in the evening when demand is highest and free condoms are not available. Female condoms are not presently being socially marketed beyond hair dressing salons on a small scale. 4. The Mobility of Fishermen, Market Vendors, Plantation Workers and Commercial Sex Workers Represents a Challenge According to outreach workers and to fishermen, market vendors, plantation workers and CSWs themselves, their mobility is a handicap to achieving the desired frequency of contact USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 25 (reach—see also discussion of points 5, 6 and 7 below). The fishermen spend long periods of time on the lake, at fishing camps, or at distant ports; the market vendors travel to different market towns to sell their goods; plantations employ many seasonal workers; and CSWs frequently change locations. (See further discussion under evaluation question 3.) 5. Insufficient Use of Mass Media or Collaboration with Partners Greatly Reduces Reach and Frequency When the EBT Prev and BRIDGE II projects were designed, the intention was that EBT Prev would focus on MARPs and BRIDGE II on the general population. In reality, as the epidemic evolves, the differences between the two projects lessen. With the exception of MSM and CSWs, the target population made up largely of married heterosexual couples is essentially the same for both projects. However, there has been limited collaboration between the two projects in terms of developing common strategies and approaches. This is particularly the case with the use of radio. In retrospect, EBT Prev would have been well served by collaborating with BRIDGE II and benefiting from its network of partner radio stations and purchased air time. BRIDGE II staff did participate in the message development workshops organized by EBT Prev, but there is little evidence of collaboration beyond that. 6. One-on-One Communication Effective for Inspiring Behavior Change, But Limits Frequency and Reach Through IPC According to the evaluation qualitative data, half the time spent undertaking outreach by the IPC/Vs and IPC/As was spent conducting one-on-one intervention with target populations. The other half was spent organizing group sessions with groups ranging in size from five to 20 members. Individual market vendors who cannot leave their stalls or fishermen sitting alone repairing their nets can be reached effectively. There was even evidence that outreach workers personally escorted individuals to service delivery points, e.g., for HTC. However, considering that IPC/Vs rarely work more than an average of six hours a week, the large portion of time spent conducting one-on-one interventions limits frequency of contact and reach to a critical mass of each target population. 7. Coverage and Reach in a Generalized Epidemic Although most targets in terms of reaching MARPs were surpassed, with the exception of MSM, questions remain regarding to what degree the design of the project was sufficiently comprehensive to reach a significant percentage of each of the MARPs, so as to have a noticeable impact on the epidemic in Malawi. For example, an effective intervention was developed collaboratively with Satemwa tea plantation in Thyolo PPA, but there are 10 other tea plantations in the same district. According to IPC/V and IPC/A estimates, a third of the fishermen have not been reached in the two fishing community PPAs visited by the evaluation team (Maldeco and Dwangwa); this does not factor in the many other fishing communities along the lakeshore that were not selected to be PPAs. A person is defined as reached by the project if he or she is reached twice through individual or group IPC sessions. Nonetheless, it is possible that a single individual could be counted more times in a year if s/he is reached twice with each of the quarterly theme messages. Conclusions • When individuals of target populations refuse to provide their names for a second time, it is difficult to count contacts with them. In general, underachieving targets are caused by the lack of mobility of outreach workers, the mobility of some target populations and the limited number of hours spent volunteering each week. 26 USAID/MALAWI: EBT Prev Mid-Term Evaluation • There is a tendency to pick only the low-hanging fruit, i.e., to conduct outreach in areas close to the residences of IPC/Vs and not to reach those slightly further afield. • Free condom stock-outs have been a major problem, but good progress was seen in ensuring that project socially marketed condoms sales outlets are concentrated in PPAs and stocks are widely available for sale. Recommendations Encourage IPC/Vs and IPC/As to conduct more IPC sessions with small groups and to make better use of their time, using support materials that encourage interactivity such as flip charts and picture codes. • Expand mobile HTC that is linked to existing outreach activities, so as to bridge the gap between HTC referral and service uptake. Tailored approaches may also be needed to improve access to STI services, e.g., through expanding work with private health providers. • Data should guide selection of beneficiaries regarding age and sex—there should be better tracking of the proportion of referrals that translate into uptake of services, in order to guide strategies and outreach. • Record TOC community sessions and broadcast them along with in-studio commentary explaining the context on local radio stations. • Record local dramas and post drama discussions and broadcast selected dramas, followed by community discussions with in-studio comment. QUESTION 1C: To what extent has the project supported synergies between different communication interventions and [been] responsive to service priorities likely faced by populations, such as integrated PMTCT/ART and related GBV services? Question 1c indicator. For integrated PMTCT/ART, new PEPFAR Indicator H2.3.D: Number of health workers who successfully completed an in-service training program. For action on GBV, PEPFAR Indicator P12.2.D: Number of people reached by an individual, small group, or community-level intervention or service that explicitly addressed gender-based violence and coercion linked to HIV. Definition: the evaluation team interprets “populations” in Question 1c to mean members of MARPs. See Evaluation Question 2 below for discussion of issues of PSI operational management and capacity to implement quality assurance with regard to GBV. Again, see discussion under Evaluation Question 3 regarding definition of reach and attendant challenges. It appears that EBT Prev activities on PMTCT and its integration with ART (especially lifelong ART to the mother, the “B+” option) had not really gotten underway at the time of the mid-term evaluation (October–November 2012). Since the beginning of 2012, the project has placed an increased focus on PMTCT and family planning (FP) through alignment of PSI-branded FP commodities with relevant messages. Findings These findings are based on FGDs with MARPs and IPC/Vs and IPC/As; an FGD with the Mwanza VSU; KIIs with hub health workers; direct observation; and review of project SBCC materials and other documentation. USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 27 1. Relative Achievement of P12.2D During FY 2011/2012 and H2.3.D The 2012 Project Implementation Plan states that as of June 2012, a total of 876 MARPs had been reached with GBV messages, representing an 8.2% achievement rate (see Evaluation Question 2 [b] for further, detailed discussion on the reasons for failure to achieve targets). Table 7 sets out project P12.2.D data since June, which clearly show that improvements have been achieved. Project data indicate that over the full FY 2011/2012 there has been an aggregate achievement of 263% (men and women aged 15 and upwards) against the target (28,164 individuals against a target of 10,710). Table 7: P12.2.D Data on Males and Females Reached Across all 18 PPAs June– Sept 2012 Males 15–24 Females 15– 24 Males 25+ Females 25+ June 2012 145 225 192 314 July 2012 1,580 1,838 3,272 3,332 August 2012 1,635 1,919 3,293 3,180 September 2012 1,020 1,256 2,408 2,555 Grand Total 4,380 5,238 9,165 9,381 Table 8: Health Workers Trained FY 2010, 2011 and 2012 Target Result H2.3.D FY 2010 240 56 FY 2011 400 498 FY 2012 460 482 Total 1,100 1,036 2. Absence of a Coherent and Cohesive Overarching SBCC Strategy PSI/M’s communication department was tasked with developing an “integrated SBCC strategy” for EBT Prev. It was also tasked with conducting IPC training of outreach workers, including orientation on quarterly messages. According to the project, there were many difficulties with implementing the communication strategy, including delays in the creation of IPC tools containing the quarterly messages, lack of available communication channels in the PPAs and problems obtaining fuel for transporting staff intending to supervise communication activities. PSI/M also cited insufficient project staff to train outreach workers and to develop support materials and SBCC strategies. It could be argued that a fundamental difficulty has been the absence of a coherent strategy 28 USAID/MALAWI: EBT Prev Mid-Term Evaluation to start with. A seven-page creative brief was developed in November 2010 for the “Make a Choice—Live a Better Life” campaign, but it did not provide details regarding specific channels and plans for the quarterly messaging. A one-page “Communication Strategy Summary for Partner Reduction and Condom Use” was produced in 2010. It describes fictional male and female characters who were created to help understand the target populations. The absence of a coherent and comprehensive overarching SBCC strategy has contributed to an ad hoc approach to project communication strategic planning and messaging. 3. Lack of HTC Kits in Public Health Facilities a Severe Handicap As has been discussed earlier, there is evidence that project outreach workers created demand for HTC, in terms of the number of referral coupons given out. Stock-out of HIV test kits (especially between June/July and September 2012, according to four of the 11 health workers interviewed) has severely reduced opportunities for synergy to be developed between project communication interventions and clinic-based testing. The project has seen more success with demand creation for HTC by outreach workers when linked to MACRO mobile testing. 4. Moderately Effective GBV Messaging Considerable GBV messaging has been communicated since June–July 2012, as corroborated by evaluation team field findings. One hundred percent of IPC/Vs and IPC/As interviewed described in detail their communication of GBV messages and more than 90% of all male and female MARP respondents in all groups described recent focus on such issues. The one MARP not to discuss GBV was MSM peer educators, who do not appear to have been oriented in this Q4 communication (despite GBV being a significant issue for many MSM as revealed by programs elsewhere, e.g., in Kenya and Uganda). The reasons for this are not clear. (See also discussion of Evaluation Question 2 [b], on GBV.) 5. Combination of IPC, TOC, Drama and Community-Level Advocacy Has Resulted in Increased GBV Awareness The qualitative data collected by the evaluation team revealed a high level of awareness of GBV among all MARPs (except MSM) as a result of the Quarter 4 themed message. The involvement of the police VSU and village chiefs has created a synergy around the topic, although the evaluation fieldwork shows very limited evidence of concrete changes such as health service visits related to GBV or administration of post-exposure prophylaxis (PEP). 6. Overemphasis on Female-to-Male GBV and Limited Attention to Sexual Violence The evaluation fieldwork provided evidence of missed opportunities for benefiting from the synergies that have been created, in large part due to the overemphasis in project drama on female GBV toward males (which accounts for a very small percentage of all GBV). This messaging has undoubtedly reduced attention to male GBV. Another missed opportunity is the lack of adequate attention to sexual violence as an aspect of gender-based violence. A TOC video drama produced by PSI does draw attention to sexual violence and seeks to solicit a community debate. Despite this, synergies are limited, as community drama and IPC outreach primarily focus on non-sexual physical violence within marriage. The physical and sexual violence inflicted upon CSWs by their clients has not been addressed in any project GBV messaging. 7. Insufficient Attention to Linking GBV and Quarter 1–3 Messaging Evaluation fieldwork indicates there has been insufficient linking of the Quarter 1, 2 and 3 USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 29 messages with the Quarter 4 GBV message. The view was expressed, e.g., by IPC/Vs and IPC/As, that GBV represents a move away from project core messaging on choices and couple communication. The intention appears to have been that GBV should build on and incorporate such core messages and also underline issues of comprehensive prevention, as should all future quarterly messages, e.g., the planned Quarter 5 PMTCT. However, this approach does not seem to have been fully implemented. Opportunities appear to have been missed for emphasizing the links existing between GBV (including sexual violence) and HIV and for providing clear information on post-exposure prophylaxis, etc. The project communication strategy has not been updated to reflect Quarter 4 messaging on GBV, or to address future direction. Thus, issues linked to PMTCT—such as pregnancy and HIV, male partner engagement and access to ART—have not been considered. Conclusions • Demand for services was created by SBCC, but uptake was handicapped by insufficient service delivery, especially related to free condom distribution, HTC and STI treatment. • There are indications of the project’s support of key behavior change, e.g., desire for HTC and increased empowerment with regard GBV. • Insufficient coherent attention has been paid to gender and equity issues in project outreach training and messaging (especially in the case of CSWs and MSM). Recommendations Develop a comprehensive SBCC strategy for all future quarterly themes that is based on an analysis of existing behavioral data, identifies obstacles and includes messages with key benefits and clear recommendations of positive behaviors to be adopted. • Strengthen measurement of SBCC outputs on service uptake and adherence within indicators on program quality and coverage and by monitoring target population comprehension of and response to interventions, especially measuring service uptake attributable to communications efforts. • The communication strategy should be updated to provide information on the process of quarterly message development, the technical correctness of all messages (in light of GBV message weaknesses) and the ways in which all future quarterly messages will address comprehensive prevention, with links to project core messages. • All future quarterly messages should be pre-tested with IPC/Vs, IPC/As and representatives from all project MARPs. All such pre-testing should be reported on in order to ensure that quality assurance issues from the perspectives of MARPs are addressed. QUESTION 2: What effect has the addition of new program areas, specifically voluntary medical male circumcision and gender-based violence, had on PSI’s operational management and capacity to implement quality assurance? 2 [a]: Voluntary Medical Male Circumcision A new Objective 5 was recently added to the project: Voluntary Medical Male Circumcision (VMMC) service delivery in Thyolo and Blantyre districts and associated demand creation in Blantyre 30 USAID/MALAWI: EBT Prev Mid-Term Evaluation district. Five performance indicators are to be measured, including two to be addressed through PEPFAR Indicator P5.1.D. VMMC is costed at $4,128,026, ca 16.6% of the total project budget over its lifetime. It is not possible at the time of this mid-term evaluation to measure progress against the new Objective 5 or to discuss findings. Furthermore, it is only possible to estimate the impacts the introduction of VMMC activities will have on PSI’s operational management and capacity to implement quality assurance. The preliminary conclusions and recommendations that follow are based on meetings with PSI staff members; review of the PSI/M VMMC presentation (itself soon to be updated); discussions with other HIV stakeholders; the evaluation team leader’s experience of VMMC research and implementation elsewhere in Africa; and review of the Malawi VMMC Communication Strategy 2012–2016 and linked documents (e.g., draft leaflets for couples and young men). Preliminary Conclusions • Indications are that PSI has not yet fully estimated operational management inputs specific to implementation of VMMC. While a clinical officer will be employed by PSI/M, it does not appear that any other new staff members will be recruited with expertise in the SBCC and gender and health aspects of VMMC. • It is important that quality assurance be covered for the remainder of the project in terms of both biomedical and sociocultural/gender perspectives, in conjunction with M&E of demand-side (client) perceptions of quality of service. • PSI/M should give careful consideration to the sustainability and potential for scale￾up of the current VMMC approach in terms of human resources for health in the overall context of public health in Malawi and specifically linked to other HIV interventions, e.g., ART provision. A 12-person health and support team represents a significant investment of time, expertise and clinic space, as well as individual commitment. • While VMMC can undoubtedly play a significant role in the “getting to zero” HIV strategy (including in terms of cost-effectiveness), it should always be implemented only as part of a comprehensive HIV prevention package of services and should be used in conjunction with other methods of prevention, such as female and male condoms. • It is essential that any VMMC intervention undertaken by PSI/M always address the HIV prevention needs of women and girls from the outset and engage with gender issues. VMMC is not solely a biomedical intervention, but one that should be introduced and implemented as a behavior change strategy to protect both men and women from infection. The VMMC Communication Strategy 2012–2016 and other draft VMMC materials do not adequately address these key issues. • There are indications, e.g., from South Africa, that circumcised men believe they no longer need to use condoms. Concerns have also emerged in terms of VMMC of married men/men in stable partnerships. Their wives and partners need to be appropriately and adequately counseled to ensure that they understand the mutual benefits. This too must be addressed by PSI/M in planning and implementation. Preliminary Recommendations USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 31 • PSI/M should review its internal operational capacity so as to ensure optimal effective management and quality assurance of the VMMC component. • PSI/M should evaluate VMMC team workload at three-month intervals in order to monitor potential excessive duties and to allow speedy opportunities to consider alternative models of service delivery, should the need arise. • PSI/M (and other VMMC actors, e.g., NAC, BRIDGE II) should develop an integrated action plan before the start of VMMC implementation that addresses the intervention as related to other project HIV-focused IPC activities, making use of international best practices (biomedical and sociocultural). The plan should include adequate attention to gender aspects of VMMC and the impact of potential behavior change on partners of circumcised men. The plan should also include discussion of organizational arrangements to ensure that adequate time and effort have been allocated to the VMMC component. • PSI/M should consider call-down contracts for two independent experts to monitor and evaluate the VMMC component in terms of quality assurance and appropriateness of messaging and behavior change focus. One of these experts should have biomedical expertise in quality assurance, while the other should have social development and gender expertise in the HIV field and experience of quality of service issues from the client perspective. In addition and beyond solely EBT Prev considerations, the VMMC Communication Strategy and linked materials should be reviewed by a gender and health expert to ensure adequate inclusion of such perspectives. None of the documents reviewed adequately addresses barriers to VMMC, whether from the potential client’s perspective, that of his female partner/s, or indeed those of the wider community. Nor do the documents effectively address issues of behavior change subsequent to circumcision (see emerging findings from South Africa, as mentioned above). 2 [b]: Gender-Based Violence Findings These findings are based on an FGD with members of the Mwanza VSU; FGDs with MARPs, IPC/Vs and IPC/As; and KIIs with hub health workers. In addition, the evaluation team directly observed four project GBV dramas (in Lilongwe Old Town, Maldeco, Mwanza and Zomba) and one TOC in Maldeco. Discussions were held with PSI/M and Pact staff members and KIIs were conducted with project partner staff members. Further meetings were conducted with USAID and other HIV stakeholders, e.g., UNAIDS. Findings from all activities were triangulated with review of project and other documentation (the latter category including the March 2012 USAID Gender Equality and Female Empowerment Policy) and informed by team member experience and expertise. Please see discussion above on Evaluation Question 1c with regard to MARPs’ engagement with GBV issues. While this evaluation question specifically requires attention to PSI/M operational management and capacity to implement quality assurance, it should be noted that Pact has also played its part in the development and introduction of activities to address GBV, e.g., in supporting development of drama scripts. 1. Impacts of GBV Component on PSI/M Operational Capacity The evaluation team found no evidence that the introduction of the GBV component has in 32 USAID/MALAWI: EBT Prev Mid-Term Evaluation itself taxed PSI operational management capacity. Indications are that the relevant staff members worked on the development of Q4 messaging in the same way as for previous quarters’ messages. However, there are indications that PSI/M has insufficient in-house gender expertise and has undertaken insufficient planning informed by gender-aware approaches—not only in the case of the Q4 GBV messaging, but overall in terms of project planning, implementation, M&E and reporting. Therefore, it can be argued (in a sense counterfactually) that had PSI/M addressed EBT Prev gender issues more strategically and systematically from the outset, its operational capacity would have been guided by gender-aware planning and implementation processes. As a result, a staff member with dedicated gender expertise might have been tasked with ensuring that such focus was intrinsic to all project activities. Thus, impacts of the GBV component on PSI/M operational capacity must be considered in light of limited gender focus. Had there been more such focus, impacts might well have been different and differently addressed and managed. The insufficient PSI/M (and project) gender expertise can be seen in Q4’s overemphasis on female physical violence, insufficient attention to male sexual violence, absence of tailored messaging for CSWs and MSM and inadequate tailoring of messages for men and women, including sexual coercion of girls. Training of health workers has also not been adequate: the 2102 EBT Prev Trainers’ Guide for Health Workers: GBV is not sufficiently engendered (or indeed tailored to Malawian circumstances). Opportunities appear to have been lost for optimal synergies; for example, MACRO HTC counselors have not been trained in GBV issues, apparently because hub health workers are seen as key to the new referral system. The overall result has been sub-optimal GBV message development, training and coordination—all of which have resulted in less than totally effective operational capacity. 2. Inadequate PSI Capacity for Gender Focus and Implications for Quality Assurance There appears to be no staff member at PSI/M or Pact tasked with primary responsibility for gender and health issues and adequately trained in such matters. This seems to have had an impact on implementation of quality assurance, not only of the Q4 messaging on GBV, but in a wider context. It has not been possible to discover if any member of staff at PSI/M or Pact has had training in gender and health/HIV issues and gender analysis and strategic, technical approaches linked to program/project management. In this context, concerns may be raised with regard to the quality of IPC/V and IPC/A training on gender, project training of health workers, the level of gender awareness used in the development of SBCC support materials (including those scheduled for Q5–7) and how gender focus may or may not have informed project engagement with community leaders and health workers in terms of the referral system. For instance, no mention was made by any IPC/V, IPC/A, or MARP respondent in any of the FGDs of different message presentation for male and female plantation workers, fishermen and women in fishing communities, or male and female vendors. 3. Negative Impacts of Focus on Referral System Changes Specific to Action on GBV The project experienced delays between January and June 2012 in implementing Q4 GBV messages, predominantly due to the major changes introduced in the new referral system implemented in January. These changes required all EBT Prev team members (communications and referral teams) to be involved in orientating project partners, PPA officers, IPC/Vs and IPC/As and health workers at the newly designated hubs in the new system. Community-based organizations also had to be informed of the changes to their USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 33 project involvement. These changes had a negative impact on operational delivery of the GBV component and other interventions. This focus meant that significant delays occurred in the introduction and rollout of the Q4 message and the training of IPC/Vs and IPC/As. This negative impact can be seen in the lack of progress in the first half of 2012 against the annual target for PEPFAR Indicator 12.2.D of 10,710. As of June 2012, a total of 876 MARPs had been reached with GBV messages, representing an 8.2% achievement rate. In contrast, Table 7 (in 1c, above) shows that by the end of September 2012, a total of 9,381 MARPs had been reached with the Q4 IPC messages on GBV—showing that, by this time, the new referral system had begun to be operational. 4. Project Links to the Police Victim Support Units The project has engaged since 2011 with the Malawi Police Service Victim Support Units (VSU), which address issues of gender-based violence and family disputes. A key joint activity is community sensitization, managed by the project partner active in a PPA. VSU officers and outreach workers visit community leaders and VSU officers also participate in TOC events on GBV. Links have been established between VSUs, project partner representatives and outreach workers—all of whom are expected to coordinate referrals. The VSU has been provided with referral registers and is also expected to report back to the project partner. Perceived gaps are the lack of feedback after referrals to health services are made and the lack of support materials being provided by the project. The Mwanza VSU FGD revealed a lack of training on gender and HIV. Mention was made in the FGD of a slight feeling of being detached from project activities, although there is eagerness to be more engaged. (See also discussion of VSU activities under 1c.) 5. Implications for Future Project Attention to Gender Issues A lack of systematic, strategically guided and technically sound gender focus has had a significant impact on the quality of delivery of project inputs and on overall quality assurance. One major aspect of this is the lack of a technically sound, engendered communications strategy, in which each successive quarter’s messaging builds on earlier ones, incorporating gender focus as intrinsic and standard. Given that proposed Q5–7 messages also have significant gender perspectives, it is important that PSI/M remedy this lack. It appears that current staff numbers may be insufficient, especially in light of the introduction of the VMMC intervention (representing 16.6% of the total EBT Prev budget) and the expansion to 20 PPAs. In the context of gender focus, it is important that PSI/M and the project closely consider how best to ensure effective attention for the remainder of EBT Prev. Conclusions • The implementation of the Q4 GBV messaging and linked EBT Prev activities has been hampered by operational capacity challenges connected to the referral system changes. • The EBT Prev communications strategy/approach does not adequately address issues of gender, which shortfall is reflected in insufficiently engendered GBV and other messaging, training and overall project approaches. • GBV messaging has been inappropriate overall in its too-great focus on violence against women. It has not sufficiently addressed sexual violence against women and action that victims can take (e.g., having access to the VSU and referrals for PEP), as well as having failed to develop support within the community. 34 USAID/MALAWI: EBT Prev Mid-Term Evaluation • There are opportunities for the project to maintain and enhance its relationship with VSUs, beyond the Q4 messaging on GBV. • The GBV messaging has not sufficiently built upon earlier quarters’ core messages of couple communication and comprehensive prevention from a gender perspective. • Beyond solely GBV messaging, the fact that there is no PSI/M or Pact staff member tasked with primary responsibility for attention to gender and health issues and with appropriate technical expertise, has affected capacity to develop, as well as implement, quality-assured interventions on gender. • There is insufficient focus within PSI/M and the project on wider aspects of gender and HIV, most importantly the tracking of referrals and uptake, links to CBOs and other community inputs, research and routine M&E. • Attention might also usefully be given to the gender balance of project partner management, which appears to be significantly tilted to male project managers and officers. Is there sufficient opportunity for all project outreach workers to have input on discussions of gender issues? Recommendations • The project should make it a priority to recruit a full-time gender and health expert to provide strategic gender focus and guide project activities to 2014 (including message development, training, research planning and review of project indicators to allow more engendered, disaggregated measurement of progress and performance and gender aspects of the new referral system). • The introduction of VMMC and intended greater focus on PMTCT B+, FP and HTC (Q5–7 messages) require dedicated gender and health expertise, as does continued attention to GBV. • All pilot testing of future quarters’ messaging needs should pay attention to gender perspectives and the gender appropriateness of approaches, support materials, language and scripts. • Engagement with police VSUs should be reviewed and opportunities for closer work together mutually explored. • Project outreach workers would all benefit from a refresher course on gender, facilitated by gender and HIV experts. • Hub health workers and HTC counselors should similarly be offered (refresher) training on gender and HIV that emphasizes service delivery, quality assurance from a client perspective and the human and gender and health rights of all. • Issues of overall project operational capacity and quality assurance should now be addressed through an effective gender and equity strategy, which must include quality assurance from a client/demand-side perspective (e.g., client satisfaction regularly measured through disaggregated, open-ended health facility exit surveys). • The project should consider the addition of a gender-specific indicator linked to SBCC. • In addition, EBT Prev should engage more systematically with existing sources of gender expertise, e.g., within BRIDGE II, Dignitas (for PMTCT B+) and with other organizations in Malawi that are working on issues of health, HIV and gender. USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 35 QUESTION 3: To what extent is the current project monitoring and evaluation framework measuring indicators that are appropriate and sufficient to demonstrate the value of the social behavior change communications’ approach? Findings These findings are based on analysis of the FGDs and KIIs undertaken with project outreach workers; KIIs conducted with project partner representatives; meetings with PSI/M, Pact and USAID; review of the Project Monitoring Plan and other EBT Prev M&E and research documentation and reports; and on triangulation of analysis. It should be noted that all project indicators must have been discussed and agreed upon between PSI/M, Pact and USAID in 2009. Similarly, there must have been agreement on the frequency of measurement of individual indicators. The development of baseline vis-à-vis endline target (10% increase) for purpose/outcome indicators through Objective 1 activities for three of the five MARPs addressed by the project (plantation workers, fishermen and women in fishing communities and vendors) would also have been considered. In addition, the degree to which individual indicators are disaggregated would have been discussed. 1. Challenges Regarding “Reach” It should be noted that the 2011 PSI/M Population Size Estimate Exercise report emphasizes the mobility of fishermen, seasonal plantation workers and itinerant vendors, as do other project and national documents. CSW mobility has long been understood to present difficulties of sustained contact (few data appear to be available regarding MSM, but peer educators in Dwangwa spoke of frequent trips to and from Lilongwe, to Cape Maclear, etc.). Such factors should be borne in mind when considering reach challenges. The 2009 project baseline study informed the definition of “reached” (central to PEPFAR Indicators P8.3.D: Number of MARP/CSW MARP/MSM MARP/targeted population reached with individual and/or small group level interventions that are based on evidence, P8.1.D: Number of the targeted population reached with individual and/or small group level preventive interventions [TOC/drama] that are based on evidence and/or meet the minimum standards required and P12.2.D: Number of people reached by an individual, small group, or community-level intervention or service that explicitly addressed gender-based violence and coercion linked to HIV). It was determined that for P8.3.D, each individual MARP is to be contacted twice for each project quarterly message by an IPC/V, IPC/A, or MSM peer educator, either through IPC individual or group sessions. Each contact is to be validated through the IPC/V or MSM registers. This was stated in evaluation FGDs and KIIs with IPC/Vs, IPC/As, project officers and managers to be partly the case also for P12.2.D. For P8.1.D and for the community-level activities under P12.2.D, it appears that looser criteria are applied (basically, head count estimates), due to TOC events and dramas being performed in communities where viewers and participants cannot be pre-selected, counting may be difficult and no records are kept. The mid-term evaluation fieldwork FGDs and KIIs show challenges regarding measurement of P8.3.D and individual/small-group interventions under P12.2.D. There was mention in all the IPC/V and IPC/A FGDs of MARPs reached by IPC being reluctant to sign for the second of the two sessions required in order for the project to count people receiving interventions. A number of reasons were given: insufficient assertiveness and/or negotiation skills on the part of the IPC/Vs and IPC/As to encourage MARPs to sign twice; a lack of understanding by MARPs as to why it is important to sign twice; and an apparently strongly held view among some members of MARPs that signing twice should be accompanied by 36 USAID/MALAWI: EBT Prev Mid-Term Evaluation payment for time spent receiving the IPC. In addition, it proved impossible to gauge fully how many individual members of MARPs are met only twice per quarter, how many may be met more frequently and how many entirely new MARPs are provided with EBT Prev IPC per quarter or month. There was reference to receiving more than two quarterly IPC sessions in eight of the 15 MARP FGDs (53%); in a few instances upwards of 10–15 IPC sessions were mentioned. It was not possible to review IPC/V reporting forms in detail, so it is not clear how detailed all project partners’ methods are for recording and tracking which individual MARPs have been met, how frequently and how many are added each quarter or month. IPC/As in one FGD spoke of having a monthly target for their organization of 750 MARP contacts—there was uncertainty as to how many should be repeat contacts, how many new and whether and how old contacts might be retained quarter by quarter. In this context, the aggregate Pact P8.3.D data for FY 2012 on MARP contacts per project partner (achievements vs. targets) are helpful, but require further unpacking, e.g., by type of MARP, sex, number of contacts, etc. All such issues are relevant when reviewing project data specific to P8.3.D and P12.2.D on numbers of MARPs reached. 2. How Will the Project Measure Impact at Endline? This refers in part to the baseline for three of the five MARPs (plantation workers, fishing folk and vendors) and the relatively modest 10% increase to be achieved through the five purpose/outcome indicators by the end of the project. In addition, the Project Monitoring Plan does not contain similar focus on CSWs and MSM. It is relevant to point out that the project has not applied a counterfactual in order to track relative levels of impact among MARPs. Therefore, impact and direct attribution would be challenging at endline. Nonetheless and in light of the fact that EBT Prev has just over one year to run (until end of February 2014), planning for the endline and other 2013 studies might usefully include discussion of how the project could more closely gauge impact of its approaches and interventions, not least in order to inform potential scale-up, dissemination of best practices and future project/program planning. (See also Point 7 below.) 3. Insufficient Attention to Tracking SBCC The project M&E framework does not adequately address the following core SBCC indicator criteria: attention to SBCC objectives; demonstration of the measuring of exposure to and outcomes of SBCC; specific demonstration of the effect of SBCC on target groups over time; measurement of attribution of SBCC; gender sensitivity; and appropriate disaggregation. 4. Insufficient Attention to Beneficiary (MARP) Perspectives The project M&E framework does not include a single performance indicator that seeks to measure MARPs’ views on the quality of project interventions, or that tracks the potential for sustained behavior change during (and beyond) the lifetime of the project. There is no indicator for client/MARP satisfaction or for quality of service delivery (e.g., relative client￾friendliness of health workers) as perceived by the demand-side/clients. A project document entitled Quality Assurance Document Table discusses two key issues: “Currently message dissemination does not link client to referral services” and “No feedback is given on the message satisfaction from the clients reached and those serviced under referrals.” These issues are listed to be addressed by end October 2011 and end November 2011. They appear not to have been fully dealt with at the time of writing (late USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 37 November 2012). There is reference in the document to planned exit surveys at health facilities to gauge client satisfaction; information from PSI/M is that these and other client￾focused quality assurance activities are to be conducted as part of the new referral system framework. 5. Limited Disaggregation of Indicators and Lack of Gender Sensitivity A number of the performance indicators as set out in the Project Monitoring Plan, e.g., in Objective 2, do not require sex disaggregation, even where such data would be relevant. One such example is: Increase perception among target population that male condoms are a safe and effective method of STI prevention (Non-PEPFAR); another is Referral uptake rate. The M&E framework lacks the opportunity to track and disaggregate any differences or similarities in male and female responses to project activities, or behavior change. In addition, none of the five indicators in the new Objective 5 (VMMC) includes any attention to gender aspects of the intervention. 6. Limited Feedback and Discussion of Project Data More than 80% of IPC/V and IPC/A FGDs and KIIs mentioned the fact that the project does not provide feedback on analysis of the raw data collected by volunteers, collated monthly by IPC/As (including collection of referral vouchers from hubs) and then reported on by project partners to PSI and Pact. This absence was seen as lacking in courtesy and, more importantly, as missing opportunities for fine-tuning messaging and interventions. IPC/As in Maldeco PPA noted as a hypothetical example that were raw data to indicate a spike in HTC, feedback on this might allow project partners to try to build on such positive behavior. Project partner outreach workers with experience of the earlier REACH project bemoaned the fact that monthly/quarterly review meetings were not included in EBT Prev. This lack of feedback relates also to the data quality assurance (DQA) exercises. 7. Limited Tracking of Evidence Base Despite being the Evidence-Based, Targeted HIV Prevention Project, implementation relies too heavily on the 2009 baseline data, with purpose/outcome level indicators not to be measured until 2014. Objective 1 was achieved in 2010/2011; there is no project indicator to measure quality of data collection, whether through routine M&E or through research. In addition, the PMP and other project documents do not have scope to review indicators, e.g., in the light of quality assurance from a client perspective, or to include more gender-focused measurements of project activities and progress. Objective 2 contains 14 indicators (some of which are “sub-indicators,” i.e., disaggregated to specific MARPs); of these, seven are to be measured solely through baseline and endline. Three of the six Objective 3 indicators are to be measured solely through a general population survey to be undertaken in 2012/2013. Of the five indicators in Objective 4, the four non-PEPFAR indicators are only to be lightly measured, e.g., once an (undefined) baseline has been set. All the above factors have implications for project measurement of impact at endline. (See also Point 2 above.) Conclusions • Challenges linked to reach should be addressed through support to IPC/Vs and other project outreach workers. • Issues of impact measurement should also be addressed. • Project indicators do not sufficiently track SBCC processes and it will be challenging 38 USAID/MALAWI: EBT Prev Mid-Term Evaluation to measure project contributions to sustained behavior change among target populations. o The project M&E framework does not sufficiently meet the requirements of SBCC evidence-based planning and implementation. o The absence of a thorough (and gender-sensitive) project communications strategy has hampered efficacy of planning and implementation, including M&E. • There is no project indicator that allows for measurement of client/demand-side perceptions of quality of service delivery, whether through IPC/other SBCC interventions or through the referral system. • The project has not yet fully addressed how best to incorporate client and outreach worker inputs in routine M&E (or indeed review of indicators). • Project indicators are insufficiently gender-sensitive and disaggregated. • A lack of scheduled data feedback to project partners reduces opportunities for coherence and fine-tuning of messaging and approaches. • EBT Prev does not adequately track its evidence base year-on-year. Recommendations • Focus should be placed on tighter and more disaggregated measurement of reach, including tightening of means to track how many times each MARP reached is counted and how often each individual MARP is reached per message. • PSI/M, Pact and USAID should discuss how best to gauge impact of project approaches and interventions in endline and other studies (while bearing in mind that EBT Prev was not structured to provide such data). • Expert attention should be given to strengthening SBCC processes and outcomes in project indicators and M&E. • A demand-side/MARP indicator to measure quality of service delivery should be included. • A gender-focused review of objectives and indicators would help strengthen disaggregated and gender-sensitive project tracking of SBCC and referral uptake activities. • Opportunities should be explored for data feedback to outreach workers, which should be linked to ongoing fine-tuning of interventions in order to optimize opportunities for behavior change and uptake of referrals. • EBT Prev intends to conduct an “Exposure Survey” in 2013/2014, which will apparently examine MARPs’ relative contacts with project activities. This represents an opportunity to undertake gender-sensitive, disaggregated research that closely examines the strengths and weaknesses of project interventions from a beneficiary perspective and addresses the degree to which the project may or may not have supported positive behavior change. Issues of quality of service delivery need also to be addressed. The Exposure Survey should be expanded to include IPC/Vs and IPC/As. QUESTION 4: To what extent have the CBOs and IPC/Vs successfully assimilated the capacities introduced through the capacity-building efforts? Please see Section 2 (the discussion on evaluation limitations) for the two USAID clarifications as to the precise meaning of this question (and for 4a). Both clarifications were received after conclusion of fieldwork. The evaluation team sought during fieldwork to cover both aspects of Questions 4 and 4a in terms of definition of “CBO.” Consideration USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 39 was given to project partner organizations, e.g., Namwera AIDS Coordinating Committee (NACC) and Society for Women with AIDS (SWAM) and also to actual community-based organizations participating in the first referral system and their potential or actual role in the new system, instituted since June 2012. It should be noted that there was limited opportunity to meet representatives of CBOs during fieldwork. Findings Information for Evaluation Questions 4 and 4a was gathered through the two quantitative surveys administered to project partner representatives (eight in total—for Question 4a in particular) and outreach workers (63 in total, of which 33 were undertaken with IPC/Vs and 15 with both IPC/As and MSM peer educators); the KIIs additionally conducted with project partner staff members; the FGDs and KIIs held with outreach workers; and the 16 FGDs conducted with MARPs. Relatively limited attention was given in a number of outreach worker discussions and MARP FGDs to consideration of the role of CBOs, while there was more detailed discussion of the part played by project partner outreach workers and their organizations. Discussion was held on this subject with PSI/M, Pact and USAID. Project documentation was also reviewed. It should be mentioned that project partner capacity development has not been a major component of EBT Prev in comparison with earlier projects such as REACH. One finding is that EBT Prev has recently focused more closely on assessing project partner capacities and gaps, e.g., through the OPI exercise as described below. Findings for Project Partners 1. Project Partners Have Had Capacity Built Through Project Activities Individual project partners met during fieldwork in the six PPAs were MHRYN, COPRED, CCC, NACC, MACRO, NASO and SWAM (and also CEDEP—for a separate discussion of its capacities and other issues, see 3.4 below). In addition, a discussion was held with TFaC in Lilongwe. It was apparent during evaluation fieldwork that all project partners met demonstrated capacity. They also showed confidence in delivering project activities to a good standard. (See Section 1.3 above for discussion of the role of project partners.) While the project partner survey focused on sustainability (see Evaluation Question 4a), three of the six questions in particular addressed current capacity as well as potential future sustainability. Thus Question 2 (We have developed the technical skills and personnel to conduct IPC and outreach well into the future) scored 90%, while Question 3 (We are confident the EBT Prev Project model of linking BCC directly with service provision works well and should be brought to scale) received a score of 92%. Question 6 (The emphasis on encouraging use of services like condoms, HTC, PMTCT, ART, has resulted in a substantial increase in the use of these services by at-risk populations) received a score of 90%. Individual project partners demonstrated organizational capacity in terms of management of outreach workers, engagement with MARPs and work toward implementation of the new referral system, both through FGDs and KIIs and also through direct observation by evaluation team members of project partner activities. Financial management represents one area in which project partner capacity has been strengthened (or perhaps further developed for those organizations that also worked on REACH). Each project partner receives $25-30,000 per annum for EBT Prev activities. A 40 USAID/MALAWI: EBT Prev Mid-Term Evaluation costed SOW is a core requirement; all expenditures are tracked by Pact (again, see 3.4 for separate consideration of CEDEP issues). In addition, discussion of such issues in FGDs and KIIs with IPC/As and IPC/Vs (obviously confidential) resulted in predominantly positive views being expressed. Thus, all project partners were described as supportive of their outreach workers (despite paucity of support materials and other limitations). MSM peer educators in Dwangwa were appreciative of the support they receive from NASO. The system of project managers and PPA project officers appears to provide a solid, practical framework for engagement on project activities. A significant number of such staff members had a track record of previous work with NGOs at the community level. The role of Pact in supporting project partners was noted, e.g., through the DQA exercises and mentoring of M&E. Mention of data collection and reporting challenges was made by representatives of three of the eight project partners (37.5%). The organizational performance index has been rolled out in 2011/2012, conducted to date with six project partners. The OPI is a tool to measure progress in management; it also captures gaps in capacity. 2. Project Engagement with MACRO It appears that the relationship between the project and MACRO may have experienced recent challenges. MACRO respondents stated that the organization had not received any training, even orientation in the mapping of the overall project and information on what the other partners do. As has been discussed in Evaluation Question 2 [b] findings on GBV, MACRO HTC counselors had not received any training on GBV and specific referral. Findings for CBOs 1. CBO Capacities A number of project partner respondents mentioned that a minority of CBOs had definitely enhanced their capacity to engage in community HIV and RH activities due to involvement in the old referral system. Mention was made of home-based care and HIV support groups that had been listed in the project service directory and had subsequently strengthened links with health facilities. Four of the eleven health workers interviewed for the evaluation (36.3%) stated that they had previously and would continue to refer positive clients to CBOs. A minority of IPC/Vs (some 15%) mentioned that they were also connected to CBOs (e.g., as members), a link they all considered to be positive and effective in enabling MARPs to feel supported in the community. 2. CBO Challenges Under the Old Referral System These included the low capacity of many CBOs, the amount of work required of CBOs in making and tracking referrals and, from the project partner perspective, the sheer number of partners (upwards of 500) to be managed across the PPAs by the project and the resulting proliferation of referral forms. 3. Future Roles for CBOs Within Future Project Interventions It was envisaged that CBOs will continue to play a diminished but important role in project activities with MARPs, despite no longer directly referring clients. Health workers at hub health facilities under the new system will be trained to continue to use the existing service directory, which contains details of CBOs working on services such as home-based care, USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 41 psychosocial support and as HIV support groups. Health workers will make those referrals, while CBO members will be trained to record any referrals their organization receives. This should reduce the workload on individual CBOs, streamline referrals and enable CBOs to concentrate on their comparative advantage of engagement at the community level. This approach will be followed in both existing PPAs and new ones. Findings for Outreach Workers (IPC/Vs, IPC/As and MSM Peer Educators) See also discussion under Evaluation Questions 1 and 4a; see in addition Table 4 above for the retention rate of all outreach workers. It was not possible to interview CSW “queen” peer educators during evaluation fieldwork. 1. Outreach Worker Capacity Development The outreach worker survey analysis showed that the great majority of IPC/Vs, IPC/As and MSM peer educators agreed with the statement that the EBT Prev Project increased their skills and ability (98%, 100% and 92% respectively). IPC/Vs and IPC/As felt they had increased their capacity through engagement with the project. An aggregate 97.6% of the 33 IPC/V and 15 IPC/A respondents to the evaluation outreach worker quantitative study said that their ability to conduct IPC with target populations has increased and 97% of respondents would like to continue in the future. Figures for the 15 MSM peer educator responses were somewhat lower, at 74% and 92% respectively. The majority project outreach workers across all categories stated that they are confident they can provide services, make referrals and teach people about condom use. 2. A High Level of Commitment and Motivation One major finding was the high level of professionalism and commitment shown by all outreach workers (even MSM peer educators, despite lack of support from CEDEP), indicative of effective support not only from PSI/M and Pact, but also from individual project partners. The overall high level of job satisfaction among IPC/Vs and IPC/As was reflected in the qualitative data collected by the evaluation team. “We are motivated by saving lives,” “We are dedicated to helping our people,” and “We are also changing our own lives” were typical comments made when motivation was discussed in the 12 IPC/V and IPC/A FGDs. It was also noteworthy that very few (perhaps 5%) of IPC/Vs made any mention of per diem, seating allowance, or other such incentives. This is unusual in that evidence from many other projects, however well-managed and fulfilling these might be for outreach workers, often reveals far more reference from volunteers to their need for some type of compensation. 3. Mixed Results with the Quality of IPC/V and IPC/A Training Despite such capacity development, the evaluation team found varying degrees of satisfaction expressed in the outreach worker FGDs regarding different kinds of project training (all provided by PSI/M). There was general satisfaction with the week-long initial training. Remarks such as “basically good” and “I learnt Interpersonal Communication methodology” were typical of the comments made in six of the 10 FGDs. There was more dissatisfaction expressed by both IPC/Vs and IPC/As in terms of the day (or half-day, as described by a minority of respondents) quarterly theme training. The consensus among both IPC/Vs and IPC/As was that the length of the quarterly theme training is insufficient. “The volunteers didn’t grasp the contents” and “There’s a need for a day of theory and a day of practice, not half a day of each” were comments made by IPC/As. IPC/Vs’ views were that it was “confusing” and “too much, too fast.” The third largest number of comments across both 42 USAID/MALAWI: EBT Prev Mid-Term Evaluation the IPC/V and IPC/A FGDs concerned the need for more refresher training, especially to enhance IPC skills. “We need more training. It is easy to forget details” and “Less didactic and more participatory, like role-playing” were typical comments. 4. IPC/Vs and IPC/As Much Appreciated by MARPs MARP respondents in every single FGD expressed their appreciation of the work undertaken by IPC/Vs and IPC/As. Mention was made of the ways in which individuals had supported behavior change. One frequent remark was that IPC/Vs in particular would often accompany people to HTC or other services and would then work to help individuals gain the access they needed, e.g., to HIV support groups. There were no negative comments expressed about the work of outreach workers by MARPs in any of the FGDs. In general, the MARPs felt that their issues were well understood and discussions were relevant to their needs and lifestyles. Most noteworthy, the CSWs considered the IPC/Vs and IPC/As, who were not sex workers themselves, to be non-judgmental and very understanding of the challenges faced by CSWs. 5. Possible Overwhelming of Outreach Workers This refers to Question 5 in the project partner survey: The EBT Prev Project has to be careful not to overwhelm its field and outreach workers with too many different types of approaches and content. This scored 74%—the response reflects a widely held concern (shared by more than 60% of project outreach workers) that the project must take care in particular not to overburden its volunteers. Mention was also made in three of the eight KIIs with project partner representatives (37.5%) that the project appears to be rapidly expanding its remit and its messaging, without sufficient attention to the demands thereby placed on outreach workers. 6. MSM Peer Educator Challenges CEDEP challenges are discussed in 3.4. However, it is noted here that all 15 MSM interviewed in the two FGDs in Lilongwe Old Town and Dwangwa PPAs voiced concern over the lack of support they feel they have received from CEDEP and the negative implications for their efficacy as peer educators (e.g., inappropriate support materials and an inadequate supply of these; no payment of transport allowances for several months; etc.). In addition, the comment was made by more than 50% of the respondents that their peer education work is dangerous; as such, there was a strong feeling of being taken for granted by CEDEP. By contrast, there was satisfaction with the increased supply of lube and condoms from PSI/M and support received from NASO in Dwangwa and Pact. Mention was made in Dwangwa PPA of improving attitudes and behaviors among health workers, including a number at one of the new hubs (the respondents did not know if those health workers had received any specific “MSM-friendly” training from the project). 7. Potential CSW Queen challenges As previously noted, it was not possible to meet any CSW queen peer educators during the mid-term performance evaluation. Therefore, points made here are comments rather than findings. Queen peer educators tend to be older and less mobile CSWs. Queens in a number of PPAs (e.g., about 20 in Lilongwe Old Town PPA) have been trained by the project on safe sex negotiation skills, on STIs, FP and referral. However, queens’ referrals do not link to overall EBT Prev referral mechanisms and it does not appear that any have been sensitized to the new system—there is no tracking of queens’ referrals of CSWs. There has also been USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 43 no refresher training of any queens. Queen peer educators have also been trained in use and sale of female condoms through hair salons and other outlets, with funding provided by UNFPA (this funding was described as “erratic”). Discussion with UNFPA representatives did not elicit information on engagement with CSW queens, which may be due to change of personnel. Conclusions • Project partners have developed existing capacities and/or had these enhanced by EBT Prev inputs, most notably from Pact. • The increased focus on tracking project partner capacities, through OPI and DQAs, enables speedier attention to gaps and challenges. • Data collection and reporting remain weak areas for a number of project partners. • The new referral system represents an entry point for continued inclusion of CBOs in project activities and for attention to be given to further linking of community and health systems strengthening (as addressed by the Global Fund and other organizations). • IPC/Vs, IPC/As, peer educators and project officers demonstrate strong loyalty and commitment to the project, showing the strength of support from project partners, Pact and PSI/M. • Outreach workers, especially the IPC/Vs, are the backbone of the project—and the closest of care needs to be taken not to overwhelm their commitment or to take it in any way for granted. • Training of outreach workers (including on quarterly messaging) would repay attention to approach and content. • Issues remain with MSM and CSW peer educators in terms of appropriate support and optimal linking to project activities. Recommendations • The effective inputs to development of project partners’ capacity should be further focused, so as to strengthen data management and reporting. • OPI and DQA activities should be speedily brought to scale, reported on, any gaps addressed and again reported on. • The potential MACRO issues should be followed up, clarified and addressed if necessary. • Continued CBO engagement with the project should be encouraged and opportunities for enhanced links between community and health systems explored. • The project should review its support to outreach workers (IPC/Vs and peer educators) and develop mechanisms to mitigate potential for burnout. This should be done speedily, as the new referral system becomes fully operational and before EBT Prev introduces Quarter 5–7 messaging. • Training should be reviewed: make the approach less didactic, introduce role play and other more participatory methods and anchor all quarterly messaging in comprehensive prevention where referral and behavior change are more effectively and consistently addressed. • Issues of MSM peer educator support should be quickly addressed. If CEDEP 44 USAID/MALAWI: EBT Prev Mid-Term Evaluation engagement remains problematic and slow, other options should be explored (see also 3.4). • The project should consider how best to continue its work with CSW peer educators, e.g., through further discussion with UNFPA. QUESTION 4a: What elements of these new capacities will CBOs and IPC volunteers sustainably continue to implement without the support of EBT Prev? Because this question addresses how CBOs and IPC/Vs will sustainably continue to implement activities once EBT Prev support ends, it cannot be evaluated through an existing project performance indicator. It has been considered through participant observation, FGDs andKIIs and the administration of two quantitative surveys—one for project partners and the other for IPC/Vs, IPC/As and peer educators. No representative of a CBO (i.e., an actual community-based organization in any of the six PPAs visited) participated in the evaluation quantitative data collection. (See Appendix D and Tool I for full details of the project partner survey.) The survey was administered to a total of eight staff members among the eight project partners met in the course of fieldwork. These partners were CEDEP, MHRYN, COPRED, CCC, NACC, NASO, SWAM and MACRO. Respondents included executive directors, project managers and project officers. (See Appendix D and Tool I for full details of the project outreach worker survey.) A total of 33 IPC/Vs, 15 IPC/As and 15 MSM peer educators were sampled (a grand total of 63 respondents), across the six mid-term evaluation PPA field sites. Findings 1. Project Partners’ Views on Sustainability Project partner respondents were asked to evaluate six questions on a scale of 1 to 5, one meaning they did not agree at all with the statement and five meaning they agreed very much. Analysis of the Project Partner Survey Questions on Sustainability Two survey questions in particular focused on sustainability (but see also discussion under Evaluation Question 4 for findings for the other four questions). Aggregate scores were: Q1: We have increased our capacity to sustain EBT Prev Project work beyond the end of current funding. Score: 72% Q4: The EBT Prev Project has allowed us to increase our skills for working on HIV in the future. Score: 98% The aggregate score is low (72%) for project partners’ confidence in having developed capacity to sustain project activities once funding ends, despite evidence of a number of partners (e.g., NACC and COPRED) having been successful in resource mobilization from other donors. Probing revealed concerns over maintenance of what is a complex outreach structure, dependent on the continued loyalty and commitment of IPC/Vs and also on individual project partner organizations’ capacity to continue to pay waged staff (IPC/As, project managers and project officers, etc.) once EBT Prev funding ceases. As one respondent said, “However much our organization might wish to continue, without funding it will be impossible to do so other than on a very small scale and on a voluntary basis.”. Half of respondents stated that they would like the project to engage in detailed discussion in 2013 with the GoM, USAID and other partners on opportunities for scale–up and retention of project approaches and partnerships once funding ceases. USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 45 All eight respondents were positive about the sustainability of individual organizations’ skills in HIV work, as can be seen by the very high aggregate score of 98%. Views such as “Pact has ably supported our capacity development and technical training on HIV” were expressed. 2. IPC/V’s, IPC/A’s and Peer Educators’ Views on Sustainability Two questions on the outreach worker quantitative survey dealt specifically with sustainability issues: Q2: As a result of EBT Prev Project, I will be able to continue the work I am doing after the project ends. Q6: In the future, I would like to continue in my IPC/PE work. The quantitative data collected by the evaluation team confirmed a strong desire among the IPC/As and IPC/Vs to continue IPC work on HIV beyond the lifetime of EBT Prev. When asked if they feel able to continue their IPC work as a result of the EBT Prev Project after it ends, 94% of the IPC/Vs and 88% of the IPC/As agreed (Q2). Among the 15 MSM peer educators surveyed, 92% agreed with the statement that they would like to continue their peer education work in the future, while a lower figure of 74% felt that they would be able to continue. Ninety-six percent of the 33 IPC/Vs interviewed and 100% of the 15 IPC/As interviewed agreed with the statement that they would like to continue their IPC work in the future (Q6). Conclusions • Concerns over sustainability of project activities are universal among project partners. • Local project partners have expanded their capacity to leverage funding from other donors; NACC has been especially successful in resource mobilization. • The project has provided opportunities to sustain IPC activities once the project ends, through the establishment of a local project partner-managed network of outreach workers. However, once again issues of funding sustainability are central to discussion. • The model developed of IPC/Vs, IPC/As and peer educators engaging in HIV prevention should be considered for scale-up. Recommendations As part of its exit strategy, the project should actively explore opportunities for scale-up of project activities and outreach and referral networks and systems. • Support to project partner resource mobilization should be expanded by PSI/M and Pact. QUESTION 5: How effective have EBT Prev activities been in strengthening the network of community-based HIV and reproductive health providers? Question 5 indicator. EBT Prev performance Indicator 4.3: Number of health service workers/providers and volunteers trained in order to participate in user-friendly referral system and/or provider network for HIV-related [services]. Based on clarifications received, this indicator is not relevant to Question 5; it is considered instead in Question 5a. It should be noted that there is no project indicator that directly addresses community-based health providers. 46 USAID/MALAWI: EBT Prev Mid-Term Evaluation Please see Section 2 (evaluation limitations) for discussion of the precise meaning of this question. The question refers to both the old system—in which community-based organizations played a larger part (e.g., in referrals) than they now do under the newly instituted referral system—and to this new system, operational since June/July 2012. Findings Information for Evaluation Questions 5 and 5a was gathered through 11 KIIs with health workers; discussion with five HTC counselors; MARP FGDs; FGDs and KIIs with outreach workers and project partner staff members; discussions with PSI/M, Pact, USAID and other stakeholders; and review of project documentation. Due primarily to the delay in clarification, it was not possible to interview community-based HIV and reproductive health providers, such as MoH health surveillance assistants (HSAs) and community-based distributors (CBDAs) of FP commodities. Community-based Health Worker Engagement Information from PSI/M stated that HSAs and CBDAs had not been involved in project activities under the old referral system and that specific relationships with such community-based health workers had not been cultivated by project stakeholders, other than informally. For instance, PSI/M staff working on reproductive health issues engaged to a limited extent with community-based RH providers in Liwonde and Mchinji PPAs. This information was corroborated through discussion with project outreach workers and project partners. The role of community-based health workers with regard to their engagement with the new project referral system requires further clarification. Conclusions • The old referral system did not engage with community-based HIV and RH providers. • The planned project focus in Quarters 5–7 on PMTCT B+, FP and HTC provides a powerful potential entry point for the participation of such community-based health providers. Recommendations • All relevant project actors (PSI/M, Pact, USAID, MoH, NAC and project partners) should discuss opportunities to work more closely with community-based HIV and RH providers. • These discussions should address the role of IPC/Vs in particular, to ensure that any additional work to be undertaken by these outreach workers is effectively managed and supported so as to avoid overwhelming them. • Consideration should be given to including an indicator to track involvement of community-based health providers. QUESTION 5a: To what extent has EBT Prev’s support to the provider network improved referral systems for the priority target groups? Question 5a indicators: PEPFAR Indicator (P11.1.D): Number of individuals who received testing and counseling services for HIV and received their test results and EBT Prev Performance Indicator 4.5: Referral uptake rate. As described above for Question 5, EBT Prev Performance Indicator 4.3: Number of health service workers/providers and volunteers trained in order to participate in user-friendly referral system and/or provider network for HIV-related [services] is also addressed here. See Section 2 (evaluation limitations) for discussion of the meaning of this question. This USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 47 question refers to the first referral system instituted by the project; as such, findings refer to this system. The new system has been operational only since July 2012. Therefore, preliminary comments have been provided, while recommendations for implementation of the new referral system are given. Findings for the Old Referral System 1. Relative Achievement of Indicators P11.1.D and 4.5 Project data for FY 2011/2012 indicate the following number of HTC referrals under the old system, in response to Indicator P11.1.D. Table 9: P11.1.D Data for FY 2011/2012 P11.1.D Number of individuals who received HTC and received their test results Target Achievement Achievement % 30,000 21,370 71.2% Disaggregated Male <15 years 220 1.0% Male 15+ years 11,605 38.7% Female <15 years 285 1.0% Female 15+ years 9,260 30.7% By test result: positive 1,330 6.2% By test result: negative 20,040 93.8% Project aggregate data for FY 2011/2012 indicate the number of referrals under the old system, in response to the EBT Prev Performance Indicator 4.5: Referral uptake rate. Disaggregated data per type of service indicate relatively high referral rates for a number of services, yet low uptake (e.g., for STIs), while family planning scores more highly in terms of uptake, albeit from a lower base. Referrals for PMTCT, GBV support and PEP are noticeably low in number. Table 10: General Referrals Made FY 2011/2012 and Per Service Total # of clients (MARPs and general population) referred 781 Total # of clients who received services (any) 398 General referral uptake rate 51.0% Type of service Total number of clients referred Total number of clients who received the service 48 USAID/MALAWI: EBT Prev Mid-Term Evaluation HTC 334 165 (49.4%) STI 214 75 (35.4%) FP 93 70 (75.3%) Medical Care 83 60 (72.3%) TB Screening 44 23 (52.3%) PMTCT 7 1 (14.3%) GBV 4 3 (75%) PEP 2 1 (50%) It can be seen that under the old system, uptake of the first service to which an individual was referred was less than ideal at 51% and for HTC stood at 71.2%. Project verbal information is that since the introduction of the new referral system in July 2012, 773 people have been referred across all 18 PPAs. The information that has been made available on how many individuals may have taken up their referrals has been insufficient, considering that it has not been disaggregated (e.g., the sex and category of each person, such as a vendor, the type of service to which the person was referred, or the PPA where the person is resident.) 2. Achievement of Indicator 4.3 A total of 1,036 health workers have been trained by the project since inception. The definition of “user-friendly” as set out in the indicator requires further unpacking. A significant number of those health workers will presumably not still be working with the project, given the reduction in linked health facilities under the new referral system. It is not clear how many, if any, health workers have as yet been trained under the new system. More details will be required to better understand the systems. Review of project training documents, such as the 2102 EBT Prev Trainers’ Guide for Health Workers: GBV, does not discuss the most client-friendly approaches that are being used and allow for their replication. 3. IPC/Vs’ and IPC/As’ Capacity Developed A majority of all IPC/Vs and IPC/As (ca 80%) interviewed stated that they had been well￾trained under the old referral system to refer and support MARPs. 4. Difficulties in Tracking Referrals Despite its community-level approach, the project found it challenging to track referrals under the old system and to collect data on the degree to which community behavior change activities led to uptake of HIV-related services. Six of the eight project partners (75%) interviewed during evaluation fieldwork reported a degree of difficulty for IPC/Vs and IPC/As in tracking referrals under the old system. 5. Project Recording of First Referral Only This point refers to the fact that project partners and outreach workers recorded only the first referral made—and this is set to continue in the new system. This is often made by the USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 49 IPC/V or IPC/A for HTC, or perhaps by the HTC counselor to the VSU as a result of reported GBV. However, project recording and reporting ceases at that first referral. Therefore, if a MARP member is referred onwards by a health worker, e.g., for pre-ART or ART, or by the VSU for post-exposure prophylaxis, this second referral is not recorded by the project. It should be noted that tracking onward referrals is especially important when there is time sensitivity such as the case in PEP and EC. 6. Human Resources for Health Challenges The frequently rapid turnover of government health staff resulted in a lack of institutional anchoring of the old project referral system, exacerbated by the proliferation of linked health facilities. 7. Client Abuse There were reports of client abuse under the old and new systems in three of the six PPAs, described by both CSWs and vendors. There was reference to inappropriate touching of CSWs and verbal abuse (over and above what might be characterized as “normal” health worker behavior). Perhaps 15% of the MARPs who mentioned this issue were concerned that they might be identified by IPC/Vs as a CSW or an MSM on referral forms, or that this might be deduced by receiving health workers at hub facilities. Comments on the New Referral System While it is not possible to provide findings on the new referral system, given that it had only been operational for three months at the time of the mid-term evaluation, the evaluation team has provided the following comments. 1. Greater Ease of Referral and a More Streamlined System When discussed during evaluation FGDs, a high proportion of project outreach workers said that they like the new system in terms of streamlining systems and making life easier for MARPs who seek services. Mention was also made in three KIIs with project partner representatives of the ways in which links between the community and health systems might be strengthened and made more direct under the new system. 2. Perception of Potential Reduction in Numbers Referred Under the New System Upwards of 50% of all project partner respondents and 40–45% of IPV/As mentioned that the new system will result in reduced numbers of people referred. The frequent comment made was that under the old system, any person could be and was referred; the intention with the new system is to refer only MARPs (the exception being the MACRO mobile HTC clinic, which provides its service to all comers). This apparent confusion on the part of outreach workers and other project stakeholders should be quickly addressed. It is important for measuring project achievements that there be a differentiation between MARPs and others who are referred. 3. Concerns over Fee-Paying Hub Health Facilities Approximately one quarter of KIIs and FGDs raised concerns about the ability to afford fees for services and a small number of MARPs, including MSM in Dwangwa, also mentioned this issue. The old system included more fee-paying hub health facilities, causing this concern over fees for services. There is some anecdotal evidence that there are now fewer partner health facilities (hubs) that charge fees. While it is known that clients should not pay for HTC or ART at any health facility, it is unclear to what degree fees are demanded for other services. 50 USAID/MALAWI: EBT Prev Mid-Term Evaluation 4. Stock-Out Challenges Reduce Referral Effectiveness Hub health workers in nearly 50% of health facilities visited during evaluation fieldwork reported virtually 100% stock-out of HIV test kits between June/July and September 2012. This information was echoed by the five MACRO HTC counselors met in Blantyre and the one counselor interviewed at the mobile HTC clinic in Maldeco PPA. Such stock-outs (also reported for STI drugs and family planning commodities) will have a negative impact on client uptake of services, whichever referral system might be in operation and however effective it might be. Conclusions • The first referral system built and maintained the capacity of IPC/Vs and IPC/As to refer clients to services. • The project has been responsive to the failures of the first referral system, using lessons learned to develop the new system, which has been operational since July 2012. • There is apparent confusion among project planners regarding the number of MARPs likely to be referred under the new system. There is a need for further discussion and clarification on this topic with project partners and outreach workers. Recommendations for the New Referral System • The project must continue to be absolutely vigilant to guard against any possible abuse of the new referral system and of clients. • There has to be an absolute guarantee for all project-referred MARPs that no private information will be provided to health workers without individuals’ express consent. • The project should double-check that none of its private/faith-based hubs is charging inappropriate fees for health services. • Issues of quality of service delivery from the client perspective should be included in all future referral work, including training and refresher sessions. • Consideration should be given to whether it is at all possible to track beyond the first referral (using sex-disaggregated data, as also for the two recommendations immediately below). • Attention should be paid to how best to increase the number of MARPs who go for HTC and also receive their test results (FY 2011/2012 aggregate 71.2% for P11.1.D). • The project should address the low numbers of MARPs being referred for GBV survivor services (including PEP), PMTCT, FP and STI treatment. • The project relationship with effective and active CBOs should be maintained and monitored for overall quality of service delivery and degree of effective engagement with hub health facilities. • The project, USAID and GoM partners should explore potential for scaling up the links between community and health systems’ strengthening (e.g., very much a focus of Global Fund health systems’ support) in the light of eventual lessons to be learned from the new referral system. QUESTION 6: How effectively has PSI maintained a balance between social marketing of condoms and free distribution of male and female condoms for the priority target groups? Question 6 indicator: EBT Prev Performance Indicator 2.6: Percentage of sexually active USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 51 adults (20–49) that perceives condoms as being a safe and effective method for preventing HIV/STI Findings Mid-term evaluation data on Questions 6 and 6a were collected through discussions with PSI/M staff members; FGDs and KIIs with IPC/As, IPC/Vs and MSM peer educators; and through a condom outlet survey conducted in three of the six PPAs. 1. Ample but Erratic Supply of Free Condoms Has Little or No Effect on Sold Condoms There has been 10% growth in sales of the Chishango male condom, which was launched in Malawi in 1994, with annual sales of $8 million. In 2011, subsidized sales of this condom represented 22% of the total condom market in the country, including both socially marketed condoms and those distributed for free. According to PSI/M, the ample though erratic supplies of free condoms in Malawi have no impact on the sales of socially marketed condoms. Chishango is viewed as a quality product and has a niche market of customers who can afford it. Chishango sales represent 61% of the market of sold condoms, according to recent market research from 2012. The other socially marketed condom, “ManYuchi,” has the second biggest share of the market. PSI has no control over free condoms in Malawi or how they are distributed. 2. Socially Marketed Condoms Bought by Employers and Distributed Free to Employees PSI/M has agreements with employers who buy and distribute free Chishango condoms within the workplace. This includes tea and sugar plantations that are within PPAs. Sixty-two institutions buy the branded condoms, such as banks, mobile phone companies and tobacco companies. Other purchasers include parastatal groups, e.g., water and dairy boards and also several NGOs. 3. Female Condoms Are Popular, but Under promoted CSWs and other target population women participating in the evaluation FGDs who had tried female condoms liked them and wanted more. The CSWs liked them because they could charge clients more to have sex without a condom and use the female condom without clients’ knowledge. Married women said that they liked being able to wash female condoms and reuse them. Female condom sales in locations such as hair dressing salons have been disappointing, in part due to limited resources that were dedicated to promoting the product. Information from UNFPA is that it was set to support promotion of the female condom, but a funding request was not received in time from PSI for it to be included in UNFPA’s current annual budget. 4. New Strategies to Increase Condom Use In response to complaints about the latex smell associated with the Chishango condom, PSI plans to launch a “new and improved” product that will have a vanilla scent and come in a four-pack. To increase the attractiveness of free condoms to youths, PSI has assisted NAC in the creation of the “Silver Touch” brand that will remain a free condom. The theory is that a branded free condom will increase demand. 5. Systematic Opening of Sales Outlets in PPAs at Locations and Hours Convenient to Users Multiple outlets have been developed in PPAs, including bars and bottle stores, groceries and kiosks, which have the advantage of being open throughout the evening when demand for condoms increases. The Rapid Outlet Creation approach used by PSI during the pilot phase 52 USAID/MALAWI: EBT Prev Mid-Term Evaluation of EBT Prev matched sales outlets to PPAs and hot spots. Sales agents working with three or four wholesalers cover each PPA, to allow anyone who so wishes to sell condoms. An informal survey conducted by the evaluation team found multiple sale outlets of Chishango condoms in five of the PPAs visited and no reports of stock-outs lasting longer than a few days. The one exception was a market in Mwanza that was dominated by Jehovah’s Witness market vendors opposed to condom sales. 6. Consistent Availability of Socially Marketed Condoms Allowed for Condom Availability during Stock-Outs of Free Condoms EBT Prev target populations interviewed in FGDs stated that they will use socially marketed condoms if free ones are not available. PSI/M sees the same steady growth in sales. In fact, the erratic but ample supply of free condoms on the market has no effect on sales of socially marketed condoms. 7. IPC/Vs and IPC/As Are Well-Positioned to Promote Condom Use and Deliver Free Condoms to MARPs According to the outreach workers who participated in the evaluation FGDs, the male and female condoms they supply are appreciated by the target populations. Outreach workers’ dependence on public health services for their supply of condoms limited the number they could give to any individual—none is distributed during long periods of stock-outs. 8. Mass Media Promotion of Branded Socially Marketed Condoms Represents the Only Condom Promotion, Usually Resulting in Increased Use of Free Condoms The only mass media promotion conducted during 2012 was the promotion of the branded socially marketed condom. Ideally, condoms should be regularly promoted because there is a link between promotion and use. There is evidence from other settings that the promotion of socially marketed condoms has a positive impact on demand for condoms from other sources. QUESTION 6a: How have the dynamics of this balance affected access to condoms for the priority target groups? Question 6a indicator. EBT Prev performance Indicator 1a: Increased penetration of male and female condoms in high-risk sales outlets within targeted PPAs Table 11: Condom-Selling Outlets: Cumulative Data FY 2010–2012 (P8.4.D) Comments Number of condom selling outlets in general 938 Outlets visited only by PSI sales reps. NB: there are many outlets selling condoms which sales reps are not directly reaching. Number of condom selling outlets in approved PPAs 1928 Marketing team visited all PPAs and took census of all outlets selling condoms. USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 53 Findings 1. Evidence of Increased Distribution of Free Male and Female Condoms in PPAs In the six PPAs visited by the evaluation team, 38,244 condoms were distributed, of which 30,601 were male condoms and 7,823 were female condoms. The largest numbers were distributed in Dwangwa (22,297) and the least in Thyolo (1,364). 2. Free Condoms Sold Increase Availability An estimated third of free condoms end up being sold, especially by owners of bars and bottle stores later in the evening when demand increases and after the public health distributors of free condoms have closed. 3. Free Condoms Are Necessary for Remote Rural Areas, Lower-Income Segments of the Population and Young People The EBT Prev target populations who have a cash income from fishing, plantation work, market sales, or sex work may prefer free condoms, but have a greater chance of being able to afford to buy socially marketed condoms. However, there are segments of the population, including rural populations and youths, who are less likely to be able to afford to buy condoms and need access to free ones. 4. GoM and Donor Coordination of Condom Services Wanting Despite major problems with the procurement and distribution of free condoms in Malawi, there is little evidence of systematic coordination among the government and international stakeholders. This lack of coordination and collaboration has contributed to the breakdown of the distribution system and widespread stock-outs. According to evaluation respondents, insufficient numbers of condoms are available within the country to come even remotely close to meeting need. An estimated three condoms per sexually active adult are imported annually for both free and socially marketed condoms. A CSW interviewed in Lilongwe pointed out that she would need more than that number each night. Conclusions for Evaluation Questions 6 and 6A • Malawi lacks a coherent distribution system of free condoms. • There are insufficient free condoms in-country to meet the needs of the sexually active population. • Condom social marketing is a cost-effective intervention. • Free condoms will be necessary for priority target groups in remote rural areas, lower-income segments of the population and young people. • Positive response among female target populations to female condoms shows potential for market expansion and needs increased promotion. Recommendations for Evaluation Questions 6 and 6a • Continue to use mass media advertising of socially marketed condoms, with the goal of increasing use of condoms overall and to identify and overcome obstacles to condom use whenever possible. • Provide leadership within the donor community responsible for bringing condoms into Malawi, by working to reactivate the condom committee made up of representatives of donors and the government, in order to rationalize the procurement and distribution systems. • If difficulties persist with frequent stock-outs of free condoms to be distributed by IPC/Vs and IPC/As, explore alternative distribution methods in collaboration with wholesalers and PSI/M distribution mechanisms. 54 USAID/MALAWI: EBT Prev Mid-Term Evaluation • Ensure that adequate resources are available to promote the female condom and consider adding it to socially marketed product lines as demand grows. 3.4 EMERGING THEMES This brief section of the report pulls together a number of themes that have emerged through evaluation data collection, analysis and report writing, as perceived by the evaluation team, i.e., not based on further discussion with any project stakeholders. The themes are crosscutting in that each is informed by a number of the evaluation questions. They are presented here as a potential resource for consideration during the remainder of the project. Behavior Change There is some evidence of behavior change through project activities, e.g., enhanced couple communication, desire to know HIV status and greater interest in using condoms. However, other aspects of potential behavior change, such as the desire to know one’s status, have been hampered by a number of project and systemic weaknesses. Main project weaknesses: • Absence of a social and behavioral change communication strategy. • Insufficient coherence of core messaging in all SBCC on referral and uptake of services. • Insufficient interactive materials to support outreach. • Inadequate targeted SBCC messaging to MSM and CSWs. • Training (including refresher) too didactic and insufficiently rooted in outreach workers’ own life experiences. Systemic weaknesses: • Frequent stock-outs, e.g., of HTC kits. • Insufficient supply of male and female condoms. • Referral system weaknesses, e.g., very limited referrals for many services such as GBV, PEP, FP. • Project training of health workers too biomedical, insufficiently client-friendly and very limited in gender awareness. Outreach Workers and Volunteer Network The project has developed an extremely valuable resource in outreach workers, both waged and (especially) volunteer, whose loyalty and commitment are proven. In the final 15 months of EBT Prev, all stakeholders should prioritize debate on opportunities for sustaining and scaling up the project model of outreach and voluntarism.2 2 In addition, during those 15 months, the project should enhance its appreciation of volunteers, e.g., through public thanks at TOC events and certificates of service, to build toward sustainability of community interventions. USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 55 Project Partners A number of the project partners have demonstrably strengthened capacity to establish networks of outreach workers and support interpersonal communications. Several, e.g., NACC, have successfully built on EBT Prev (and earlier REACH project) capacity development and support to leverage funds for expansion of “EBT Prev-type” activities. MSM represent a perhaps uniquely challenging MARP. CEDEP represents the sole current organization active in their support; opportunities should be sought to widen that engagement, e.g., through other NGOs. Project Data Collection and Analysis The project monitoring plan as it currently stands would benefit from review, so as to include the following: • An indicator or indicators more focused on SBCC. • At least one indicator that measures quality of service delivery from a MARP perspective. • Closer attention to capturing and analyzing gender issues linked to project interventions. The project needs to address, as a matter of urgency, how its research work, e.g., its planned Exposure Survey in 2013, can most effectively contribute toward development of best practices and provide achievable inputs to future scale-up. Gender Gender has received insufficient focus and technical expertise to date within the project. More dedicated technical attention is now essential, given intended future quarterly messaging on PMTCT B+, FP and HTC, all of which have significant, embedded gender dimensions. The introduction of VMMC represents a further major component (both within and beyond the project) where gender focus is essential. Issues of sexual and reproductive health and rights would benefit from greater attention by the project (while acknowledging its vanguard activities with MSM): • Ensuring occupational anonymity of all MARPs referred. • Clamping down on any reported abuse by medical staff. • An overall more engendered and life stage approach to SBCC that incorporates nuanced messaging for men and women, young and old. The New Referral System and Potential of Complementary Health and Community Systems Strengthening There is a need for the new referral system to ensure that higher numbers of MARPs are referred and tracked and that uptake of all services is recorded, reported and analyzed; numbers referred currently represent a weak element of EBT Prev. Benefiting from best practices from other settings is recommended. The project has worked toward developing and strengthening community systems. Opportunities should be sought to foster increased coherence of community and health systems strengthening, as promoted by the Global Fund among others (e.g., by monitoring 56 USAID/MALAWI: EBT Prev Mid-Term Evaluation the quality of community-based services and the number of referrals made between health facilities and CBOs under the new system). Project Exit Strategy and Legacy With 15 months to run at the time of writing this report, EBT Prev and project stakeholders should examine the development of both lessons learned and any existing or emerging best practices, in order to explore the potential for development of a model for going to scale with MARPs. With regard to project reach, while the rationale for project attention to MARPs was grounded in epidemiological realities and subsequent project data, the time is ripe for consideration of how best to capitalize on the EBT Prev model of drilling down into such populations and whether such approaches can be revised or expanded in order to best support interventions that address the continuously evolving HIV epidemic in Malawi. For instance, recent data indicate that upwards of 80% of all new infections annually occur in couples in stable partnerships. Finally, the EBT Prev model of working with local NGOs as project partners and with volunteer structures should be considered in terms of future opportunities—how best to sustain such structures? USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 57 4. CEDEP ISSUES During in-country work on the mid-term evaluation the team was asked to consider Centre for the Development of People (CEDEP) management issues linked to its role as a project partner, and any resultant challenges related to the support given to MSM peer educators by the organization and the project. It should be noted that this represents an addition to the SOW; as such, there was limited time to address the topic. The points discussed below build on discussions with USAID, PSI/M, and Pact as well as on field findings. One overall comment is that project engagement with CEDEP has been challenging. It appears that certain pre-conditions otherwise applied to all project partners were waived, due to CEDEP being the only NGO working with MSM in Malawi; evaluation interviews indicated that financial pre-conditions such as a stand-alone bank account were not required of CEDEP. Another challenge is that CEDEP does not currently seem adequately to manage MSM project interventions or to report on activities in a timely or sufficiently detailed fashion. The general opinion among project stakeholders is that CEDEP should continue its relationship with EBT Prev, as it is the gatekeeper for a key MARP with considerable impacts on the current and future epidemiological profile of HIV in Malawi. However, the relationship does require strengthening. FINANCIAL MANAGEMENT ISSUES There have been ongoing challenges with reconciliation and liquidation of CEDEP monies received from the EBT Prev Project—meaning that Pact has not disbursed funds. It appears that CEDEP is being responsive to PSI/M and Pact concerns. A CEDEP finance officer has been appointed, and Pact is orienting him in project financial management structures. Pact will henceforward directly finance CEDEP MSM project activities. OTHER ORGNIZATIONAL MANAGEMENT ISSUES At present, CEDEP does not report on its project activities in a timely fashion. Data on referrals are also apparently incomplete or lacking (it is not clear if this is across all five PPAs where CEDEP works on behalf of the project). CEDEP ENGAGEMENT WITH MSM PEER EDUCATORS CEDEP identifies MSM and supports their recruitment as peer educators; the men are then trained by PSI. Two of the five CEDEP PPAs were visited during the mid-term evaluation (Lilongwe Old Town and Dwangwa). There are said to be CEDEP project officers for each of the five PPAs, all based in Lilongwe. The information gleaned in Dwangwa PPA was that the CEDEP project officer visits seldom, if at all. None of the seven MSM peer educators who participated in the focus group discussion could recall meeting him. As a result, the Namwera AIDS Coordinating Committee (NASO) project manager and other staff and volunteers have taken on what they can in terms of providing support to the MSM peer educators. This is clearly an ad hoc and inadequate arrangement, despite the best efforts of NASO. LACK OF MSM-SPECIFIC SUPPORT MATERIALS 58 USAID/MALAWI: EBT Prev Mid-Term Evaluation MSM peer educators in both Lilongwe Old Town and Dwangwa PPAs stated that they often lack MSM-specific support materials, which they see as symptomatic of inadequate attention and support from the project. A particular point was that the peer educators faced difficulties in recording referrals and reporting because no carbon papers had been made available. While this could be characterized as a minor matter, it was considered by the MSM peer educators to be symptomatic of general neglect. Another sore point was that while IPC/Vs and IPC/As working with other project partners had received bicycles, this was not so for MSM. MSM transport allowances have not been disbursed for at least several months. While these issues may be an effect of the financial management difficulties (which may soon be redressed), the outcome is that MSM peer educators feel that they are not supported and that their potentially dangerous work is not appreciated. This situation needs to be rectified. RECOMMENDATIONS Two options appear to exist for future CEDEP project engagement: • EBT Prev could institute a (new) Memorandum of Understanding with CEDEP, to be operational until the end of the project, with roles and responsibilities for all partners set out, and in which support to MSM peer educators is prioritized. In this scenario CEDEP would retain project management of MSM prevention interventions, and PSI and Pact would provide further organizational capacity development. • The second option would be to devolve management of MSM prevention interventions and referrals to another project partner, while retaining CEDEP engagement in identifying MSM as potential peer educators and enabling CEDEP also to play a part in all project research, dissemination of findings, and development of best practice studies. USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 59 5. GENDER PERSPECTIVES AND EBT PREV FOCUS This section captures certain issues and points that emerged during fieldwork. However, the section does not represent a detailed, systematic overview of all potential gender-linked project issues. Throughout discussion in this report, including here, reference to gender encompasses female and male perspectives, socialization, roles and responsibilities, as well as attitudes and behaviors—it does not solely address women’s and girls’ situations and needs. This section concludes with a few recommendations for potential consideration by PSI/M, Pact and USAID. 1. LACK OF GENDER CAPACITY WITHIN EBT It appears from the mid-term evaluation that there is insufficient gender capacity within the project, despite EBT Prev receiving more than $400,000 from the Gender Challenge Fund. This is evident in project training as well as in project documentation. There seems not to have been any coherent, systematic attention to gender issues from the outset of planning for the EBT Prev Project, by PSI/M, Pact, or indeed USAID. The project does not have a gender strategy; no gender analysis appears to have been undertaken of any project activity, M&E, or research study; and there is no mention in any of the quarterly or annual reports or annual implementation plans of internal gender mainstreaming. “Internal gender mainstreaming” refers to training of PSI/M and Pact staff in systematic and coherent gender-sensitive planning, document preparation, budgeting, project management, M&E, analysis, and reporting. Gender mainstreaming requires a foundation of gender analysis. The March 2012 USAID Gender Equality and Female Empowerment Policy defines gender analysis as “a tool for examining the differences between the roles that women and men play in communities and societies, the different levels of power they hold, their differing needs, constraints and opportunities, and the impact of these differences on their lives.” The Policy further states that in terms specifically of requirements for USAID: “Gender analysis is one of only two mandatory analysis requirements that are to be integrated in strategic planning, project design and approval, procurement processes, and measurement and evaluation.” While it is acknowledged that the Policy post-dates EBT Prev, gender foci have increasingly been embedded within HIV interventions over the past decade. It is important to note that gender mainstreaming also refers to the fact that “gender” does not solely address challenges faced by women and girls. It seeks to consider equally the ways in which men and boys may also have to deal with barriers and inequalities, based on their gender; on the means by which male socialization can result in men and boys engaging in unequal, sometimes violent, acts against women and girls; and how girls’ socialization can lead to women placing limits on their own capacities and opportunities. This apparent absence of structured, consistent attention to gender is significant in the project—not least because it works with MSM, is increasing its activities on VMMC, and has supported action against GBV in Quarter 4. In addition, the project intention is to focus on 60 USAID/MALAWI: EBT Prev Mid-Term Evaluation PMTCT B+ in Quarter 5, and on FP and HTC in Quarters 6 and 7, all of which should be planned, implemented, and monitored in an appropriately engendered fashion. Internal gender mainstreaming would enable project activities to be more firmly grounded in systematic approaches to HIV prevention that encompass issues of gender and health, including health-seeking behavior, roles, relations of power within sexual partnerships (whether with one or multiple partners), and the ways in which individuals address HIV and related issues. Internal gender mainstreaming is very often matched with external gender mainstreaming. In the context of EBT Prev this would have meant that project partners (NGOs active in working with MARPs, such as CCC and NASO), CBOs supporting referral, and health workers would have been trained on gender, health, and HIV in a systematic and structured manner and would also have been required to document and report on activities while applying a gender perspective. It should be noted here that one finding of the mid-term performance evaluation (again beyond the SOW) is that project partner managers and officers are overwhelmingly male, at least in those organizations whose representatives were met, e.g., NACC, NASO, MACRO. While this absolutely does not preclude gender sensitivity and attention to such matters, and while a preponderance of male managers is likely in large part to reflect the current realities of Malawian sociocultural norms and girls’ relative access to secondary and higher education, it is nonetheless worthy of mention. The apparent lack of a PSI/M or Pact staff member tasked with primary responsibility for attention to gender, equity, and health issues may have had an impact on the capacity to develop, as well as implement, quality assurance specific to ensuring appropriate gender attention. 2. SPECIFIC EXAMPLES OF LACK OF GENDER FOCUS Project Outreach Workers’ Gender Training Some 45% of the IPC/Vs and IPC/As and 100% of the MSM peer educators interviewed could not recall receiving any gender-specific information as part of the one-week basic training (no queen CSWs were interviewed during the mid-term evaluation). This statement should be viewed in light of the fact that the 2010 Draft EBT Prev Project Manual for Training IPC Outreach Workers does indeed include a session of gender and also considers how to achieve gender-sensitive facilitation as an IPC/V. None of the 63 project outreach workers interviewed during the evaluation could recall any refresher training on gender. The gender session provided in the basic training for outreach workers does not include CSWs and MSM under the definition of high-risk groups. Furthermore, the activities covered in the one hour and 45-minute session are not closely contextualized to a Malawian environment and make no reference to any MARP. There is also a lack of contextualization to outreach workers’ own lives and the ways in which gender norms, expectations, and behaviors may have had an impact on their own sexual attitudes and practices, their health￾seeking behavior, and their attitudes and behavior toward the opposite sex. There should also have been an opportunity for outreach workers and health workers to share aspects of their training (not solely gender), so as to create mutual understanding and provide further opportunities for close collaboration. Health Worker Training on Gender USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 61 None of the 11 hub health workers interviewed mentioned receiving project training on gender and equity aspects of health service delivery in the context of referrals. Yet there is abundant evidence to demonstrate the significance of gender-related barriers to access to services, gender issues linked to health worker attitudes and behaviors, etc. The 2012 EBT Prev Trainers’ Guide for Health Workers: GBV is insufficiently engendered (much of it also appears to have been cut and pasted from a PSI/India document, e.g., there is reference to Bangalore). In addition, the trainers’ guide is too biomedical in focus: it does not address the individual health worker’s own attitudes and behaviors with regard to gender; it does not encompass discussion of client-friendly approaches; and it does not discuss the issue of girls experiencing sexual violence and the need for tailored support and treatment, or indeed that of men who experience such violence. Sub-Optimal Approaches to GBV Messaging A significant example of insufficient gender focus is the Quarter 4 GBV dramas, whose scripts focus on physical violence and have an over-emphasis on female violence against men (in terms of the frequency of such violence vis-à-vis male physical and sexual violence against women and girls). There is too little emphasis on sexual violence (beyond the rape film shown in TOC community events). In addition, the engagement of the police VSU with project activities and referrals, while a positive development, appears not to have been fully realized beyond TOC and other community sensitization. The Concept Note for Reshaping of the Project Referral System The EBT Prev referral system has experienced considerable reshaping and refocus in the past six months, with outreach workers becoming the sole referral agents and the number of health facilities reduced, in the interest of streamlining systems and enabling easier access for MARPs. EBT Prev submitted a referral system change Concept Note, in which there are two references to gender. The first states: The [EBT Prev] Service Provision Specialists will also work with the . . . [Hub] in each PPA to implement gender activities, specifically training healthcare workers to screen clients for experience of gender‐based violence (GBV) and to appropriately refer survivors of GBV to needed services, e.g. legal, police or psychosocial. In most PPAs, the ‘Hub’ health facility is the government facility, which typically provides PEP services . . . [Therefore] the GBV survivor could access PEP, if required. A number of points arise: Have the six project Service Provision Specialists had dedicated training in gender, GBV, and counseling for PEP? Have other project actors (e.g., project partners, outreach workers) had such training? Project data for Indicator 4.5 (Referral uptake rate) show that two women had been referred for PEP in FY 2011/2012, of whom one had taken up the service). Given that the project intends to focus on PMTCT B+ in Quarter 5, and on FP and HTC in Quarters 6 and 7, it is important that there be rapid consideration of how best to ensure solid, appropriate gender training for all project actors. 62 USAID/MALAWI: EBT Prev Mid-Term Evaluation It is not clear whether project provisions under the new referral system have provided additional training on these subjects (or will do so), and, in which case, who will provide that training for health workers and community members. The second Concept Note gender reference is as follows: Community-level Gender Sensitization Activities. To fulfill PSI’s obligations regarding Gender Challenge Funds and mindful that changing gender norms is an ongoing process that requires community leadership, the Service Provision Team will work with community elders, Traditional Authorities, Village Headmen, etc. to undertake gender sensitization activities within PPAs. A number of questions could be asked, including: Is this the first time any such gender focus is being envisaged by the project? Or were community leaders and/or CBO representatives ever trained on gender aspects of GBV and HIV more widely—in terms perhaps of being involved in psychosocial care? Who undertook or will undertake such sensitization, and backed up by what sort of gender expertise? Have any impacts ever been monitored and/or evaluated? Have any potential challenges and lessons learned been documented? Project Indicators and Gender Issues The project PMP contains only one indicator that explicitly addresses gender and equity in the context of HIV linked to project activities, in Objective 4: Gender Based Violence and Coercion: Number of people reached by an individual, small group, or community-level intervention or service that explicitly addresses gender-based violence and coercion related to HIV/AIDS (PEPFAR Indicator P12.2.D). However, it can be argued that this indicator does not adequately track whether, and if so how, individuals and communities may have introduced, let alone maintained, action to change behaviors to be more gender-equal. The 2012 USAID Gender Policy includes an example of a more proactive, behavior-change￾focused indicator: Percentage of target population that views gender-based violence as less acceptable after participating in or being exposed to USG programming. Other project indicators are insufficiently engendered and disaggregated, e.g., the majority of those under Objective 2, many of which are disaggregated solely by target group. There does not appear to have been any attention to M&E of any gender inputs into the project to date. In addition, information could be sought from the project regarding the extent to which it may be considering a gender analytic approach in planned research, e.g., the Exposure Study. 3. RECOMMENDATIONS As previously mentioned, these recommendations are not part of the mid-term performance evaluation SOW; they are provided here as a potential additional input to future project activities. • The mid-term performance evaluation presents an opportunity for EBT Prev and USAID to review project gender focus in light of the March 2012 USAID Gender Equality and Female Empowerment Policy approaches and requirements. • Planned future project messaging requires gender expertise: issues of PMTCT, FP, and HTC all need gender-sensitive strategizing, planning, implementation, and M&E. As an example, the paper by Kaler and Watkins, “Asking God About the Date You USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 63 Will Die: HIV Testing as a Zone of Uncertainty in Rural Malawi,” indicates gender differences in attitudes toward uptake of HTC. • The increasing PSI/M engagement with VMMC should be supported by close gender attention and expert inputs. • Project indicators could be reviewed through a gender analytic approach, with a view to strengthening appropriate gender focus and more fine-tuned disaggregation. • Training (including refresher training) of outreach workers, health workers, and indeed PSI/M and Pact staff members might usefully be reviewed in terms of appropriate gender inputs. • The new referral system provides opportunities for closer, gender-focused tracking of service uptake (including PEP, FP, PMTCT, GBV), which should not be ignored. • The final evaluation of EBT Prev should include gender expertise, and this should be set out in the SOW. 64 USAID/MALAWI: EBT Prev Mid-Term Evaluation USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 65 APPENDIX A. SCOPE OF WORK Global Health Technical Assistance Bridge Project GH Tech Contract No. AID-OAA-C-12-00027 SCOPE OF WORK (9.14.2012) I.TITLE: USAID/MALAWI: MID TERM EVALUATION OF POPULATION SERVICES INTERNATIONA’S EVIDENCE BASED TARGETED PREVENTION PROJECT (EBT PREV) Contract: Global Health Technical Assistance Bridge II Project (GH Tech) II. PERFORMANCE PERIOD Evaluation preparations should begin as soon as possible depending on the availability of the selected consultants. Work is to be carried out over a period of approximately 10 weeks, beginning on or about (o/a) September, 2012, with final report and close out concluding by December 2012. III. FUNDING SOURCE Mission-funded IV. PURPOSE OF ASSIGNMENT This mid-term evaluation is a performance evaluation to assist USAID/Malawi and the Government of Malawi in assessing the effectiveness of Population Services International’s Evidence-Based, Targeted HIV Prevention (PSI EBT-Prev) project. The primary purpose of the evaluation is to determine the extent to which the project’s behavior change and communications interventions have resulted in reduction in risky behavior among the Most￾At-Risk-Populations (MARPS). A secondary objective is to understand the extent to which the program approach and multiple components have been effective. The findings and recommendations will inform decisions on the strategic direction of future project implementation and USAID/Malawi’s investment in HIV prevention among MARPS. Furthermore, this evaluation is a required action under the results based agreement signed in March 2009 and will serve as the basis to inform the remaining one and a half years of the program3. V. BACKGROUND The purpose of the EBT Prev is to support the Government of Malawi (GOM)’s National AIDS Framework’s goal of prevention for populations and settings in high risk areas. The agreement was awarded to Population Services International (PSI) in March 2009. The total estimated funding for the original agreement was $20,400,000 which includes 400,000 of Gender Challenge Funds to strengthen programming on gender based violence. PSI EBT Prev is responsible for implementing targeted combination prevention activities for fishing communities, estate workers, vendors, female sex workers and Men who have Sex with Men 66 USAID/MALAWI: EBT Prev Mid-Term Evaluation (MSM) in 20 Priority Prevention Areas within 15 target districts of Mwanza, Mulanje, Thyolo, Zomba, Blantyre, Machinga, Chikwawa, Mangochi, Nkhotat Kota, Lilongwe, Salima, Mchinji, Nkhata Bay, Muzuzu, and Karonga. In response to Malawi Government’s uptake of voluntary medical male circumcision (VMMC) as a important evidence based prevention intervention, PSI received additional PEPFAR funds in 2012 to provide VMMC services in two targeted districts where they already had a strong presence. These additional funds increased PSI’s award ceiling to 24, 498,000 (4,580,000 for dedicated VMMC activities). In May 2009, PEPFAR Malawi and the GOM signed the Partnership Framework (PF) – a document which outlines PEPFAR’s strategic five year implementation plan to support the GOM respond to the impact of the HIV/AIDS epidemic. Recognizing the pivotal role that HIV prevention plays in the sustainability of the National HIV/AIDS Response, the PF devotes 30 percent of all resources to prevention. Under the National Action Framework’s Objective 1: Preventing sexual transmission, the Partnership Framework Implementation Plan (PFIP) agreement is designed to reduce new HIV infections and increase the quality and access of prevention, treatment and care programs. These prevention efforts, are expected to utilize approaches that maximize sustainability including working within local systems, changing social and gender norms, building capacity of local organizations in HIV prevention, and training Malawian providers to deliver safe male circumcision, a highly-effective intervention that once performed greatly reduces an individual’s lifetime risk of acquiring HIV. The PF and PFIP is also aligned with Malawi’s National HIV Prevention Strategy, which shifts the emphasis of previous prevention efforts in Malawi to prioritise evidence-based engendered complementary approaches to reduce multiple and concurrent partnerships and reduce transmission among existing discordant couples. The strategy also harmonises behavioral and biomedical prevention interventions, including timely initiation of HIV treatment, prevention-with-positives interventions provided in HIV testing and counselling, prevention of mother to child transmission (PMTCT) pre￾ART and ART clinics, blood and injection safety, and safe medical male circumcision. PSI Intervention Areas EBT Prev is PEPFAR’s flagship prevention program focusing on high risk populations and settings under the first NAF Objective 1:1 to reduce new HIV infections in Malawi through sexual transmission (NAF Objective 1:1). Within this objective, interventions focus on reducing sexual transmission through the following strategic priorities 1) behavior change communication directed towards partner reduction, 2) condom social marketing to high risk populations , 3) expanded HTC (particularly couples counseling) to achieve high coverage, 4) Timely ART, condoms for discordant couples, and positive prevention, 5) voluntary medical male circumcision and 6) capacity building of indigenous organizations. EBT Prev addresses these key priorities through the following main project objectives: Identify, segment and profile priority populations at risk Using innovative research methodologies, including Priorities for Local AIDS Control Efforts (PLACE) and Tracking Results Continuously (TRaC), the project will locate and enumerate individual groups at highest risk of contracting and transmitting HIV, understand the complex dynamics of risk groups, and isolate priority barriers to adoption of safer behavior (audience segmentation). PLACE will identify, map and describe venues where individuals meet new sexual partners. TRaC will help establish baseline and monitor changes in behavior and determinants of behavior over time and measure the impact of interventions. Regular monitoring methodologies feed into project re-design to increase effectiveness. Priority locations and key populations are identified and surveyed, and targets set. Key outputs include identification of priority prevention areas (PPAs), size estimations of male and female most at risk populations (MARPS), segmentation analysis aimed at identifying constraints and opportunities for promotion of the identified safe behaviors, and baseline indicators set for each group and PPA. Throughout the life of the agreement, research would be conducted USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 67 to better inform communications priorities and interventions, and monitor any changes in target population knowledge, attitudes and behaviors. Deliver integrated, behavior change communication programs targeted to high-risk populations in priority prevention areas Using research evidence for segmented audiences, the project will develop communications materials for target group interventions and design integrated Behavior Change Communication (iBBC) interventions to be delivered through multiple channels to ensure target audiences receive consistent and appropriate messages. Under a branded multi-media campaign – Lingalira Wekha Sangkha, a combination of Targeted Outreach Communication (TOC) events, including interactive drama, hot spot events, wall paintings and radio messages are delivered. At the heart of their communications approach is the use of community based volunteers. Interpersonal communication volunteers (IPC volunteers) are identified by targeted communities and receive quarterly training on new messages for delivery. Peer education approaches are used to reach traditional MARPS such as female sex workers and MSM. Using the “hub and spokes” model, PSI, as communication lead, partners with PACT Malawi, an organization supporting a network of Community Based Organizations (CBOs), to provide technical training and supportive supervision of CBO partners in areas such as peer education or community based product distribution. EBT￾Prev partners will develop and provide a multi-level, multi-media package of behavior change interventions that build and reinforce the opportunity, ability and motivation of target group members to promote adoption of safer sexual health behaviour—using an integrated flexible menu of approaches. The project incorporates messages and activities, focusing on key societal drivers of the epidemic, including gender inequality, stigma and patron-client power dynamics, risk reduction and alcohol reduction counseling. Messages to support HIV positive individuals and discordant couples also include promotion of HIV testing for couples, PMTCT, family planning (FP), and positive prevention. Linkages to condoms and local health care services are strengthened. Distribute and promote condoms for use by the general population and for high-risk groups Facilitation of free and socially marketed male and female condoms is a critical component for comprehensive prevention programs. The project seeks to increase consistent condom use by enhancing consumer choice through targeted, evidence based product development. PSI branded CHISHANGO male condom is marketed nationally and within PPAs, with deliberate efforts made to expand distribution outlets for increased condom availability. Program activities include market research to better understand and respond to segmented audience needs, branding and marketing of USAID procured male condoms (through USAID Commodity Fund), condom messaging through interpersonal communication, radio spots, wall signs, and TOC events. The branded female condom, CARE, is marketed through “queen” commercial sex workers within high risk settings, and through salons, etc. PSI has also supported MSMs through condom and lube distribution (through USAID’s Commodity Fund), to ensure that interventions with this key population address this important behavior. Enhance the network of existing providers for greater accessibility and services to high risk populations Combination prevention efforts require that prevention messages delivered are complemented by access to responsive services. Within Priority Prevention Areas, the implementing partner has worked on strengthening the network of CBOs and health facilities for timely referral to a range of related health and social services. Activities include organization network assessments, delivered through their main sub-partner PACT, development of referral directories based on mapping, training of providers in the delivery of Most-At-Risk-Population (MARP)-friendly health care services, client referrals using 68 USAID/MALAWI: EBT Prev Mid-Term Evaluation referral tools developed, and tracking of referrals made. Referral efforts have been complemented by the delivery of community-based HIV testing and counseling (HTC) services, through MACRO and other local HTC providers. With Gender Challenge Funds, this activity was enhanced to include gender-based violence training of HTC counselors, network meetings to screen for victims of gender based violence (GBV) and timely referrals for Post-Exposure Prophylaxis (PEP) and support. In mid-2012, PSI changed its approach to further streamline referral efforts through one main hub and utilization of frontline IPC volunteers to support referrals directly from targeted communities. Similarly, since 2012, the project has placed an increased focus on PMTCT and FP through alignment of PSI branded FP commodities with relevant messages. Deliver VMMC services in targeted districts Since 2012, four dedicated teams of Voluntary Male Medical Circumcision (VMMC) providers have been established to deliver outreach/mobile tented VMMC services in Thyolo and Blantyre districts. Activities for the VMMC component of the project include supply chain management, HTC, clinical care, condoms, behavior change communications, waste management and monitoring and evaluation. These activities are in line with current HTC outreach services delivered and provide further support to comprehensive prevention messages. Nevertheless, it will be important to gauge the effect of the addition and complexity of the VMMC component on PSI’s management and quality assurance capacities. PSI Sub-Recipients PSI works in close partnership with PACT which has extensive expertise in subgranting and capacity building of local organizations in Malawi. Using a “hub and spokes” model, PACT manages subcontracts to deliver interventions within the 20 PPAs with PSI technical oversight and management support. PACT sub-recipients include NACC, SWAM, CEDEP, Theatre for Change, and MACRO. Strategic partnerships have been envisioned for other partners including BRIDGE II, for VMMC communications support, and BLM. Existing Information Program and project information and data are available for reference and use for the evaluation and have been included as attachments. These include: 1. Partnership Framework Implementation Plan (2009) 2. PSI Cooperative Agreement and modifications 3. PSI Performance Monitoring Plan (PMP) 4. PSI annual implementation plans and quarterly reports 5. PSI monitoring and evaluation tools 6. PSI conducted studies (PLACE, TRaC, condoms for couples) 7. PSI developed IPC and referral tools and training manuals 8. VMMC work plan 9. Key Government of Malawi strategic documents including the HIV/AIDS National Strategic Plan (2012), National Prevention Strategy (2009) and Operational Plans (2010, 2011). 8. 2010 MDHS 9. MICS 2006 • All relevant data collected by the project will be made available to the evaluation team by PSI. USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 69 VI. DEVELOPMENT HYPOTHESIS To reduce HIV incidence, implementation of PSI’s EBT Prev combination prevention activities within the 20 PPAs will result in reduction of reported high risk behaviors, including multiple concurrent partnerships, increased uptake of condoms, and responsive HIV related services among most at risk populations. It is also expected that implementation of these activities will provide the GOM with effective models for scale up of targeted prevention efforts for MARPS and other vulnerable populations in other areas. VII. SCOPE OF WORK This mid-term evaluation is intended to: • Determine if the objectives as defined in the cooperative agreement and in relation to planned activities are being achieved, and assess the likelihood of achieving them upon project completion taking into account the perspectives of the stakeholders and beneficiaries; • Determine the strengths and weaknesses of the existing program and approach explicitly determining why certain program components are working or not working; and • Provide concrete recommendations on any program adjustments to be made for the remainder of the project agreement. Audience and Intended Uses This mid-term evaluation will provide information to be utilized by PSI and USAID/Malawi in determining any changes or modifications to the program and its various components and will subsequently assist in direct future program activities accordingly to improve quality and enhance success. It will also inform new directions that the project is taking to address gender-based violence and the scale up of Voluntary Medical Male Circumcision services, including potential consequences (positive and negative) that these two activities may have on existing program activities. This mid-term evaluation will focus on process and implementation fidelity as it relates to achieving project goals, objectives and activities as they were envisioned in addition to looking at some intermediate outcomes. Evaluation Questions 1. How effective have PSI’s EBT-Prev Communications approach and Targeted Outreach Communication (TOC) activities been in supporting adoption of safer sexual behaviors among project target groups? a. Are communication activities adequately tailored to the different categories of populations amongst the target groups – including commercial sex workers and MSM? (ie. responsive to specific barriers faced and sufficiently skills oriented to support behavior change?)_ b. To what extent has the frequency in which all target populations have been reached with messages supported adoption of safer behaviors including increased demand for and utilization of appropriate HIV and reproductive health services and health commodities? c. To what extent has the project supported synergies between different communication interventions and responsive to service priorities likely faced by populations such as integrated PMTCT/ART and related GBV services? 2. What effect has the addition of new program areas, specifically Voluntary Medical Male Circumcision and Gender-Based Violence had on PSI’s organizational management and capacity to implement quality assurance. 70 USAID/MALAWI: EBT Prev Mid-Term Evaluation 3. To what extent is the current project monitoring and evaluation framework measuring indicators that are appropriate and sufficient to demonstrate the value of the Social Behavior Change Communications approach? 4. To what extent have the CBOs and interpersonal communication (IPC)volunteers successfully assimilated the capacities introduced through the capacity building efforts? a. What elements of these new capacities will CBOs and IPC volunteers sustainably continue to implement without the support of EBT-Prev. 5. How effective have EBT-Prev activities been in strengthening the network of community-based HIV and reproductive health providers? a. To what extent has EBT-Prev’s support to the provider network improved referral systems for the priority target groups? 6. How effectively has PSI maintained a balance between social marketing of condoms and free distribution of male and female condoms for the priority target groups? a. How have the dynamics of this balance affected access to condoms for the priority target groups? VII. EVALUATION DESIGN AND METHODOLOGY The evaluation team should design the evaluation methodology which combines a mix of the most appropriate methods including: desk review, interviews, focus groups, data abstraction and analysis from monitoring data, and other methods that may be appropriate to answer the agreed upon evaluation questions. The evaluation team should develop a matrix listing each of the evaluation questions, the type of answer needed (descriptive, normative, comparative), the data collection method, the data source, sampling method, and analysis method. In addition, data collection tools must be developed for each method. Both the matrix and data collection tools must be shared with the AOTR and HPN team members who will then have a chance to review and provide comments for revision/enhancement of the matrix and tools before commencement of data collection. Technical Requirements 1. A full description of methodology (or methodologies) to answer each evaluation question should be provided by the team. Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be required in an Annex in the final report. 2. The evaluation report should include the scope of work as an annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline need to be agreed upon in writing by the USAID technical officer and cleared by the Program Office. 3. Limitations to the evaluation shall be disclosed in the report, with particular attention to limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparison groups, etc.). Sufficient information should be provided so that a reader can make an informed judgment as to the reliability, validity and generalizability of the findings. 4. Disclosure of conflict of interest: All evaluation team members will provide a signed statement attesting to a lack of conflict of interest, or describing an existing conflict of interest relative to the EBT-PREV project being evaluated. 5. Statement of differences: If a difference arises in the interpretation of the results from the various stakeholders, the evaluation report will include a statement USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 71 identifying any significant unresolved differences of opinion on the part of funders, implementers and/or members of the evaluation team. Findings: Empirical Facts Collected During the Evaluation 1. Evaluation findings should be presented as analyzed facts, evidence and data and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings should have sufficient evidence and documentation that a reader of the findings can be confident that the findings are based on actual data. 2. Evaluation findings should highlight any regional variations or discrepancies as well as identify outcomes or impacts that affect male and female teachers and students differently. 3. Findings should be specific, concise and supported by strong quantitative or qualitative evidence. 4. Sources of information need to be properly identified and listed in an annex. Conclusions: Interpretations and Judgments Based on the Findings 1. Evaluation conclusions should be presented for each finding based on the evidence collected by the evaluation team. 2. Conclusions should logically follow from the gathered data and findings. Because conclusions involve interpretation of collected data, they should be explicitly justified. If and when necessary, the evaluator should state his/her assumptions, judgments and value premises so that readers can better understand and assess them. Recommendations: Proposed Actions for Management 1. Recommendations need to be supported by a specific set of findings. 2. Recommendations should be action-oriented, practical and specific, with defined responsibility for the action. 3. Evaluators should take into consideration the economic and political context of the EBT-PREV project, the strengths and weaknesses of MoH institutional capacity and the feasibility of change and innovation while framing recommendations. IX. TEAM COMPOSITION, SKILLS AND LEVEL OF EFFORT (LOE) The Evaluation Team will be composed of no more than four experts in the following areas: Senior Evaluation Specialist/Team Leader (TL); SBCC specialist with expertise in MARPS programs, social marketing and community based referral systems in particular, with possible inclusion of a VMMC person (noting that VMMC activities have not yet begun). One of the consultants should be national to provide context and linkages to the national program a) Roles of the Team Members: Senior Evaluation Specialist/Team Leader. The Team Leader (TL) will be responsible for overseeing the team and ultimately responsible for the submission of the final draft report to the Mission. S/he will provide team leadership, finalize the evaluation design, coordinate activities, plan and coordinate meetings and site visits, and be responsible for payments of local logistical needs and local staff working with the team in coordination with the logistics coordinator. S/he will lead the preparation and presentation of the key evaluation findings and recommendations to the USAID/Malawi team and other major stakeholders and will consolidate reports from other evaluation team members and ensure that a draft report has been left with the Mission on departure. The Team Leader will take specific responsibility for assessing and analyzing the Project’s progress towards targets, 72 USAID/MALAWI: EBT Prev Mid-Term Evaluation factors for such performance, benefits/impact of the strategies, and compare these with other possible options. The Team Leader should have experience in leading teams of international health experts and working with host country personnel with a strong background in comprehensive HIV prevention. S/he should have extensive experience in conducting qualitative evaluations/assessments. Excellent oral and written skills are required as well as experience in preparing high quality documents. The TL should have an advanced degree in public health or a related field with a minimum of five years of experience in management and evaluation of community based and district level health programs, experience in leading teams of experts in health activities, experience in international health specifically dealing with capacity building at district and community levels and should have worked extensively with USAID-supported HIV prevention programs. SBCC HIV/AIDS Specialist with expertise on MARPS and vulnerable populations, and referral networks. Experience with PSI’s social marketing program would be an asset. One team member must be a Social Behaviour Change Communication (SBCC) Specialist. The SBCC Specialist will be responsible for assessing the PSI strategic communications approaches with all target populations. S/he will need to evaluate the appropriateness of messages and intervention design developed to reach distinct target population including likely impact on behaviour change with the frequency of contact and depth of discussion, as well as the balance on the range of key messages delivered over the life of the project including demand for services, gender programming and stigma reduction. S/he will also need to assess the implementation of both community based HIV products and services, and outcomes of activities to improve the referral networks. S/he will also need to assess the balance of condom social marketing with access to free male and female condoms for the priority target groups. This Specialist should have a master’s degree in public health or related field with a minimum of five years’ experience designing and implementing large scale USAID-supported comprehensive HIV prevention programs in Africa, preferably in the Southern Africa region with MARPS and other hard to reach vulnerable populations. Local HIV Consultant: Combination prevention interventions requires strong linkages between behavioral and biomedical prevention interventions, and capacity building for sustainability of the program. One team member should have extensive HIV expertise and experience within Malawi to inform the team on the broader context in which PSI’s program supports national efforts for MARPS and vulnerable populations as well as biomedical HIV prevention priorities. S/he will have wide knowledge of GOM strategic planning processes, MARPS programming and delivery of community based HIV services/referrals. S/he should also be able to assess the quality of capacity building efforts for CBOs, IPC volunteers, and the likely sustainability of the program efforts once the agreement ends. This specialist should have a master’s degree in public health or related field with a minimum of five years’ experience in HIV prevention programs in Malawi. VMMC service expertise: This is a new area of the program which will not have started delivery of activities at the time of the mid-term evaluation. Nevertheless, given the intensive level of efforts put into the setup of this program, it is important for one member of the team to assess the likely impact this new activity will have on staffing and management of the program. • Local Logistics Coordinator (1) will join the team on a part-time basis to provide support to schedule stakeholder meetings, key informant interviews and focus group discussions; and to organize field visits. Required qualifications include: USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 73 • Minimum 2 years of progressively responsible experience within USAID and/or NGO work settings handling complex logistics, such as coordinating business travel and meetings. • Demonstrated: ability to be resourceful and to successfully execute complex logistical coordination; ability to multi-task, work well in stressful environments and perform tasks independently with minimal supervision. • Ability to work collaboratively with a range of professional counterparts at all levels, including those from host country governmental and non-governmental organization, U.S. Government agencies and other donors. • Capacity for effective time management and flexibility. • Must be able to interact effectively with a broad range of internal and external partners, including international organizations, host country government officials, and NGOs counterparts. • Must be fluent in English. • Proven ability to communicate clearly, concisely and effectively both orally and in writing. An illustrative table of the LOE is found below. Dates may be modified based on availability of consultants and key stakeholders, and amount time needed for field work. Activity Team Leader SBCC Nationa l HIV Local Logistics Coord. Mission sends background documents to GH Tech and Team Members 5 days 4 days 4 0 Logistical preparations- arranging site visits, hotel, interviews 8 Preliminary data analysis, draft evaluation plan 6 days 5 days Submit plan to USAID/Malawi for review travel 2 days 2 days In –briefing, team planning meeting 2 days 2 days 2 days Field work 10 10 10 Draft Report Writing, synthesize data 4 days 4 days 3 days Stakeholders Presentation on preliminary findings 1 day 1 day 1 day Debriefing of Mission staff—draft report submitted 1 day 1 day 1 day Team Departs Country 2 day 2 day Mission sends technical feedback/comments on draft to Team 74 USAID/MALAWI: EBT Prev Mid-Term Evaluation Leader within 5 days Draft revised by Team Leader and team. GH Tech submits final report to Mission. 5 days 4 days 3 days Missions approves report Total LOE 38 35 24 8 A six-day work week is approved while in-country. X. LOGISTICS GH Tech will be responsible for all international travel and consultant logistics. XI. DELIVERABLES AND PRODUCTS The evaluation team shall be responsible for the following Deliverables: Detailed Evaluation Plan Prior to the commencement of fieldwork, the evaluation team will provide in writing a detailed evaluation plan that will be submitted to USAID prior to arrival and will be discussed during the in-briefing. The detailed Evaluation plan states the objectives of the evaluation, the questions that will be answered, sampling frameworks, the specific methods and instruments to answer each question, sample data analysis tables, timelines with data collection start and end dates to ensure the team document not only their understanding of what they are expected to do, but also how they plan to achieve it... The evaluation report should describe the conceptual framework the evaluator will use in undertaking the evaluation. It should set out in detail the evaluation methodology, i.e. how each question will be answered by way of data collection methods, data sources, sampling and indicators. The report must also contain a work plan, which indicates the phases in the evaluation with their key deliverables and milestones. The inception report will be reviewed and approved by the Program Office and the Health Team Leader, with consultation from PSI, before the evaluation team can begin field work. It should contain the following elements: 1. Describe the EBT-PREV program, drawing from relevant documentation, particularly the desk study. 2. State the purpose and scope of the evaluation as set out in the Scope of Work. Include the complete set of evaluation questions and elaborate on them as necessary. Any questions added during the contract negotiations must be clearly indicated and any deleted questions must be mentioned with a reason as to their exclusion. 3. Evaluation Framework: Discuss the overall approach of the evaluation, highlighting the conceptual model(s) adopted. This should incorporate an analysis of the intervention logic of the program. 4. Discuss risks and limitations that may undermine the reliability and validity of the evaluation results. 5. Evaluation methodology: Specify an indicator or indicators for each question that will be used as a guide in answering the question. Many indicators will serve as a measure of success. USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 75 6. Discuss the data collection and data analysis methods that will be used for each question. State the limitations for each method. Include the level of precision required for quantitative methods and value scales or coding used for qualitative methods. Standard data collection methods for USAID evaluations are: surveys, questionnaires, interviews, focus groups, document review and observations. 7. Present the key data sources that will be selected to answer each of the evaluation questions posed. Common sources include: program recipients, program deliverers, persons with knowledge of program recipients, program documents, records, and databases. The evaluation team should also identify how existing data will be incorporated and used to answer the evaluation questions. 8. Discuss the sampling methods and details. Include area and population to be represented, rationale for selection, mechanics of selection, sample size, sample precision and confidence and limitations. 9. Summarize the evaluation methodology in an evaluation planning matrix containing the following column headings: evaluation question, indicator(s), data collection method(s), data source, sampling and comments. Evaluation Question Indicator(s) Data Collection method(s) Data Source Sampling Comments 10. Develop a timeline which shows the evaluation phases (data collection, data analysis and reporting) with their key deliverables and milestones. 11. Specify responsibility for each evaluation phase. Include any changes in the evaluation team. 12. Logistics: Discuss the logistics of carrying out the evaluation. Include specific assistance required from USAID such as providing arrangements for visiting particular field sites or key contacts amongst the key EBT-PREV stakeholders, including MoH. 13. Appendices: Append the Scope of Work. Append relevant draft data collection instruments, such as questionnaires and interview guides. 14. The evaluation team should also propose a table of contents for the evaluation report that will be approved by USAID/Malawi prior to the drafting of the report in the Appendix of the Inception report. The evaluation plan should also contain the following information: 1. Types of information needed. The types of data needed above and beyond the existing data to completely answer the evaluation questions and identify any limitations of the existing data to effectively answer the evaluation questions. 2. Sources of information. The detailed evaluation plan should identify how existing data will be incorporated and what additional information will be required to accurately and sufficiently answer the evaluation questions. The sources of information that will be used in the evaluation should be described in enough detail that a reader can have confidence that the information will be sufficient to meet the 76 USAID/MALAWI: EBT Prev Mid-Term Evaluation evaluation's purposes, given the scope, context, and resources available for the evaluation. a. The evaluation plan should also discuss how confidentiality of information will be maintained. A sample consent form for all primary data collection should be attached as an appendix and referenced in this section. The consent form should include a description of the evaluation objectives and how the information will be used. 3. Criteria for sampling and selecting participants. The evaluation team should identify the sampling methods that will be used to answer the evaluation questions and to articulate any limitations that the method will have on ability to generalize from the findings, conclusions, and recommendations. 4. Methods for collecting information. For each evaluation question, the evaluation plan should specify the methods by which information will be collected (for example: questionnaires, surveys, observations) and the procedures to collect the data. The plan should describe the proposed methodological approaches and how, within the constraints of time and cost, they will yield data that help answer the evaluation questions. The proposed approaches should be grounded in respected methodological frameworks and best-practice literature. Significant or important constraints on the evaluation design should also be identified. 5. Instruments - The evaluation plan should include all instruments that will be used to collect data to answer the evaluation questions with descriptions of how the instruments were/will be piloted and used. 6. Timeframe for collecting information. The evaluation plan should include a detailed timeframe for the evaluation, including instrument development, piloting, and training of enumerators, fieldwork, and data analysis. The evaluation plan should discuss how instruments will be piloted/pre-tested before using them in fieldwork.. 7. Methods for analyzing information. The evaluation plan should detail the practices and procedures that the evaluation team will use to analyzing the data to answer the evaluation questions. For each evaluation question, at least one blank analysis table graphically displaying the data analysis outputs for each question should be presented. A. Periodic Briefings and Reports The evaluation team will provide progress briefings and reports to USAID/Malawi and GH Tech Bridge II on a biweekly basis. Team Leader of the evaluation team is mandated to routinely communicate updates to the USAID M&E Specialist throughout the evaluation process. B. Oral Briefings of Findings Conclusions and recommendations for each evaluation question shall be provided by the evaluation team to USAID/Malawi prior to drafting the evaluation report. The oral briefing should be presented as a PowerPoint presentation. C. Draft Report The evaluation team shall provide a draft report to USAID/Malawi (5 hard copies and an electronic copy) before departing the country. D. Findings Workshop USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 77 After incorporating comments from USAID into the draft report, the evaluation team shall hold a half day workshop to present key findings, conclusions and recommendations. The workshop shall attract at least 30 strategic stakeholders and shall be held in Lilongwe. The evaluation team shall be responsible for costs, logistics and managing invitations (with the assistance of USAID), to this workshop. The evaluation team shall produce a summary one page briefer highlighting (electronic copy) key findings, conclusions and recommendations which USAID can distribute to stakeholders E. Final Report The evaluation team shall submit 50 hard copies and 1 electronic copy of the USAID￾approved final report. The report shall be in English should not exceed 30 pages, excluding relevant annexes, (e.g. SOW, interview transcripts/notes, photos and success stories), and shall include matrices and other visuals to consolidate and summarize data. Upon submission of final draft report, the evaluation team will submit one flash drive comprising of all electronic products of the evaluation, including instruments and data in formats suitable for replication of the analysis, final report and summary brochure. NOTE: In regard to the final report, GH Tech Bridge contract comes to an end December 24, 2012 and all work must be completed and invoiced by that date. The process for final editing and formatting of approved final report content can take up to 30 business days to complete, depending on the length of the report, and the extent of editing required. Our standard process involves a thorough professional edit, internal review of the editing, professional formatting in the USAID branding-compliant template, and conversion of the document to a 508-compliant PDF for web posting. Due to the time required to complete this process, we cannot guarantee with 100% certainty that any report content approved for editing after November 15, 2012 will be completed by the end of the project, although we will do everything we can in order to speed the process along as rapidly as we possibly can. We will work with you to ensure that you have the most complete and polished product we can provide before our contract’s end date. There are various options for editing and formatting the final draft report that we can consider, depending on the timing of fieldwork completion. (1) If the fieldwork ends quite close to our project end date, the consultant could provide a working draft of the report to the mission, to be finalized by another mechanism. (2) If the final draft report has been finished and approved by USAID with less than 30 days left in our contract, we can work with you and the editors to determine what can be completed within our timeframe. XI. RELATIONSHIPS AND RESPONSIBILITIES GH Tech will coordinate and manage the evaluation team and will undertake the following specific responsibilities throughout the assignment: • Recruit and hire the evaluation team. • Make logistical arrangements for the consultants, including travel and transportation, country travel clearance, lodging, and communications. USAID/COUNTRY will provide overall technical leadership and direction for the evaluation team throughout the assignment and will provide assistance with the following tasks: Before In-Country Work • SOW. Respond to queries about the SOW and/or the assignment at large. 78 USAID/MALAWI: EBT Prev Mid-Term Evaluation • Consultant Conflict of Interest (COI). To avoid conflicts of interest or the appearance of a COI, review previous employers listed on the CV’s for proposed consultants and provide additional information regarding potential COI with the project contractors evaluated/assessed and information regarding their affiliates. • Documents. Identify and prioritize background materials for the consultants and provide them to GH Tech, preferably in electronic form, at least one week prior to the inception of the assignment. • Local Consultants. Assist with identification of potential local consultants, including contact information. • Site Visit Preparations. Provide a list of site visit locations, key contacts, and suggested length of visit for use in planning in-country travel and accurate estimation of country travel line items costs. Assist with invitations for the stakeholder presentation. • Lodgings and Travel. Provide guidance on recommended secure hotels and methods of in-country travel (i.e., car rental companies and other means of transportation) and if necessary, identify a person to assist with logistics (i.e., visa letters of invitation etc.). During In-Country Work • Mission Point of Contact. Throughout the in-country work, ensure constant availability of the Point of Contact person and provide technical leadership and direction for the team’s work. • Meeting Space. Provide guidance on the team’s selection of a meeting space for interviews and/or focus group discussions (i.e. USAID space if available, or other known office/hotel meeting space). • Meeting Arrangements. Assist the team in arranging and coordinating meetings with stakeholders. • Facilitate Contact with Implementing Partners. Introduce the evaluation team to implementing partners and other stakeholders, and where applicable and appropriate prepare and send out an introduction letter for team’s arrival and/or anticipated meetings. The AOR for the project will arrange for an initial introductory meeting with appropriate staff at the MOH and PSI. A list of relevant stakeholders and key partners will be provided to the evaluation team by the AOR at the time of arrival but the evaluation team will be responsible for expanding this list as appropriate and arranging the meetings. The AOR and other mission personnel will be available to the team for consultations regarding technical issues, before and during the evaluation. After In-Country Work • Timely Reviews. Provide timely review of draft/final reports and approval of deliverables. XII. MISSION CONTACT PERSON Beth Deutsch USAID/Malawi bdeutsch@usaid.gov Phone Number: +265 999986079 USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 79 Chimwemwe Chitsulo Monitoring, Evaluation and Learning Specialist cchitsulo@usaid.gov Phone Number: +265 999978364 XIII. COST ESTIMATE GH Tech will provide a cost estimate for this activity. 80 USAID/MALAWI: EBT Prev Mid-Term Evaluation USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 81 APPENDIX B. PERSONS CONTACTED COUNTRY: MALAWI USAID Country Office Beth Deutsch, Senior HIV and AIDS Prevention Advisor Ritu Singh, HIV Cluster Lead Ndisale Chitute, HIV and AIDS Program Specialist Chimwemwe Chitsulo, Monitoring, Evaluation and Learning Specialist Anteneh Worku, Strategic Information Advisor Population Services International (PSI) Sarah Gibson Makunganya, Chief of Party EBT Prev Project & Acting Country Director Chiwawa Nkhoma, Deputy Chief of Party, EBT Prev Project Dyson Likomwa, M&E Manager Pamela Msukwa, Program Manager, Condoms (now VMMC) Elton Edward, Service Provision Coordinator Victor Gama, EBT Prev Project Manager Levie Nkhoma, Senior Research Officer Phillip Mkandawire, Research Manager Richwell Tambuli, Interpersonal Communications Specialist Elizabeth Kashoko, Interpersonal Communications Specialist Charles Gona, Service Provision Specialist Brenda Kamanga, Program Manager, Condoms PACT Malawi Kate Musimwa, Program Manager Rolex Tolani, Senior Monitoring, Evaluation, Research & Learning Manager Wezi Msungama, Senior Program Officer National AIDS Commission (NAC) Dr. Linje Manyozo, Head of Social and Behaviour Change A.B. Chidumu, Planning Officer Wellington Kafakalawa, M&E Officer Lucresia Kuchande, HIV Prevention Officer Blackson Matatiyo, Research Officer Christopher C. Teleka, Communications Officer Johns Hopkins University Malawi BRIDGE II project 82 USAID/MALAWI: EBT Prev Mid-Term Evaluation Glory Mkandawire, Chief of Party and Project Director UNAIDS Patrick Brenny, Country Coordinator UNFPA Humphreys Shumba, HIV Prevention Programme Manager Milika Mdala, RHCS Officer Theatre for a Change Claire X Malawi Human Rights Youth Network (MHRYN): KIIs with project partner representatives Weston Msowoya, Programme Manager Peter Mwandira, Project Officer Winnie Kavalo, M&E Officer Elida Numeri, Project Accountant (MHRYN) Likuni PPA: CSW FGD Elina, CSW Hawa, CSW Monica, CSW Maggie, CSW Pilirani, CSW Bernadette, CSW (MHRYN) Lilongwe Old Town PPA: FGD with IPC/Vs (1) Elizabeth Mnemba, IPC/V Potiphar Mwaiwala, IPC/V Brenda Kapalamula, IPC/V MacDonald Namponya, IPC/V (MHRYN) Lilongwe Old Town PPA: FGD with IPC/Vs (2) Grey Palanjela Chricy Mbewe Collin Maseko Kaim (MHRYN) Lilongwe Old Town PPA: FGD with male vendors Yassin, Vendor Eliyasi, Vendor Aaron, Vendor USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 83 Sikauten, Vendor Issa, Vendor Alidin, Vendor Alli, Vendor Bubakari, Vendor Saidi, Vendor James, Vendor Twaibu, Vendor Bwaila Hospital, Lilongwe: KII with hub health worker Winston Chauta, Medical Assistant (CEDEP) Lilongwe: FGD with MSM Peer Educators Names available on request KII with CEDEP project partner representative Gift Trapence, CEDEP Executive Director (COPRED) Mwanza PPA: KIIs with project partner representatives Reuben Billiat, Project Coordinator Lawrence Phiri, Project Officer Jimmy Katuma, Executive Director (COPRED) Mwanza PPA: KII with IPC/Vs Michael Benard, IPC/V Albert Munduka, IPC/V Doreen Valani, IPC/V Rose Makungwa, IPC/V (COPRED) Mwanza PPA: FGD with male and female vendors Zex Mpapa, Vendor Maureen Banda, Vendor Rose Gama, Vendor Esther Chidothi, Vendor Bertha Dinessi, Vendor (COPRED) Mwanza PPA: FGD with CSWs Sunganani, CSW Idesi, CSW Ruth, CSW 84 USAID/MALAWI: EBT Prev Mid-Term Evaluation Anne, CSW Coretta, CSW Mwanza Police: FGD with VSU police officers Constable Bokosi, Crime Prevention Desk Officer Constable Zonasi, Community Policing Desk Officer Constable Rebecca Kayuwa, Victim Support Unit (VSU) Jones Bauleni, VSU Coordinator Mwanza PPA Border Roadside Wellness Clinic: KIIs with hub health workers Audriech Kadyakapita, Site Coordinator & Clinical Officer Vellece Chanunkha, Lay Counselor Trisizio Mlupwa, Clinical Officer Christian Church Community (CCC), Thyolo PPA: KIIs with project partner representatives Dalitso Mcheka, Project Manager Blessings Jim, Project Officer Harrison Kanyemba, Project Officer (CCC) Thyolo PPA: mini-FGD with IPC/As Bright Mayenda, IPC/A Veronica Kamoto, IPC/A Tamandani Likalowa, IPC/A (CCC) Satemwa Tea Estate Thyolo PPA: FGD with IPC/V Edward Paulo, IPC/V Zione Saidi, IPC/V Flora Chaswa, IPC/V Malango Mpoya, IPC/V Bertha Chagwada, IPC/V Charity Sulani, IPC/V Ganizani Matiasi, IPC/V Bengo Mukowa, IPC/V (CCC) Satemwa Tea Estate Thyolo PPA: FGD with female plantation workers Beatrice Moffat, Female Plantation Worker Mary Somnaje, Female Plantation Worker USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 85 Margret Phinifolo, Female Plantation Worker Alice Chitaukali, Female Plantation Worker Agnes James, Female Plantation Worker Fibe John, Female Plantation Worker Margret Raphael, Female Plantation Worker Kellita Kalungu, Female Plantation Worker Enifa Gandali, Female Plantation Worker Magret Paulo, Female Plantation Worker Berita Derrick, Female Plantation Worker Zione Chitseko, Female Plantation Worker Maria Luchenza, Female Plantation Worker (CCC) Satemwa Tea Estate Thyolo PPA: FGD male plantation workers Frank, Male Plantation Worker Rajab, Male Plantation Worker Goliat, Male Plantation Worker Bodza, Male Plantation Worker Duncan, Male Plantation Worker Gawani, Male Plantation Worker Laitoni, Male Plantation Worker Jones, Male Plantation Worker Elias, Male Plantation Worker Chikondano, Male Plantation Worker Givasoni, Male Plantation Worker Julio, Male Plantation Worker Moses, Male Plantation Worker Isaac, Male Plantation Worker Pakhoti, Male Plantation Worker Ronald, Male Plantation Worker Kabichi, Male Plantation Worker Patrick, Male Plantation Worker Satemwa Tea Estate Clinic James Phiri, Clinical Officer and Hub Focal Point Malawi AIDS Counseling and Resource Organisation (MACRO) Wellington Limbe, Executive Director 86 USAID/MALAWI: EBT Prev Mid-Term Evaluation Dezio Banda, M&E Manager Bruno Nathuru, Branch Manager (Lilongwe) Richard Chilongosi, Branch Manager (Mzuzu) Julius Malewezi, Director of Programmes Katawa Msowoya, Branch Manager (Blantyre) Lincy Misoya, Senior Supervisor Alufeyo Patrick Liyaya, HTC Counselor (Blantyre) Patrick Makuluni, HTC Counselor (Blantyre) Grace Chikuse, HTC Counselor (Blantyre) Zerif Kampangire, Senior HTC Counselor Namwera AIDS Coordinating Committee (NACC): KIIs with project partner representatives Saeed Wame, Executive Director Osman Saidi, M&E Officer Josephy Mngongonda, Accountant Hassan Chinunga, Project Officer (NACC) Songani PPA: FGD with IPC/A Zione Alekadala, IPC/A Anthony Phoso, IPC/A Doreen Kapeni, IPC/A Tabu Makwakwa, IPC/A Maureen Mponda, IPC/A (NACC) Songani PPA: FGD with IPC/V Hellen Wyle, IPC/V Marriam Kankutu, IPC/V Rose Kachembere, IPC/V Christina Teki, IPC/V Lawrence Makatu, IPC/V Jeffrey Sumbuleta, IPC/V Dickson Kapalamula, IPC/V (NACC) Songani PPA: FGD with male and female vendors William Mzembe, Vendor Gift Mandala, Vendor Laston Njikho, Vendor USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 87 Elineti Nyirenda, Vendor Lenzo Makiyi, Vendor Elizabeth, Vendor Harold, Vendor Fredrick, Vendor Antony, Vendor Enock, Vendor Ruth, Vendor Marriam, Vendor (NACC) Songani PPA: FGD with CSW Mercy, CSW Atupele, CSW Chipo, CSW Georgina, CSW Janet, CSW Domasi Rural Hospital, Zomba: KII with hub health worker Alex Nakhaonga, Officer In-Charge and Clinical Officer NACC Maldeco PPA, Mangochi District: KII with project partner representative Muyopi Tchewa, Project Officer, Maldeco PPA (NACC) Maldeco PPA: mini-FGD with IPC/A Omar Chingomanje, IPC/A Mirriam Kassim, IPC/A (NACC) Maldeco PPA: FGD with IPC/V Arnold Bendala, IPC/V Lucy Master, IPC/V Monica Mpali, IPC/V Rose Julius, IPC/V Ireen Twairi, IPC/V Jeniffer Chiwaya, IPC/V Katete Ajusu, IPC/V Dave Sanudi, IPC/V (NACC) Maldeco PPA: FGD with CSW Marriam, CSW 88 USAID/MALAWI: EBT Prev Mid-Term Evaluation Catherine, CSW Favour, CSW Lucy, CSW Fanny, CSW Anne, CSW Marriam, CSW Faida, CSW (NACC) Maldeco PPA: FGD with fishermen Imani Male, Fisherman Allie Daudi, Fisherman Kaliba, Fisherman Allie Kaiwa, Fisherman Anafi Saidi, Fisherman Kennedy Maulidi, Fisherman Amos Loga, Fisherman (NACC) Maldeco PPA: FGD with female members of fishing community Janet Amidu, Fisherwoman Joyce Rajab, Fisherwoman Marriam Donald, Fisherwoman Janet Saidi, Fisherwoman Esther Kainja, Fisherwoman Laisa Kawanga, Fisherwoman Tawaba Husseni, Fisherwoman Daina Yasini, Fisherwoman Edah Mulenga, Fisherwoman Maldeco PPA: Malawi Development Corporation clinic: KIIs with hub health workers Stenkam Msonkho, Officer In-Charge and Clinical Officer Sophie Phiri, Nurse Midwife Chrissie Chikwamba, Nurse Midwife Technician Nkhotakota AIDS Support Organisation (NASO) Dwangwa PPA: FGD with IPC/As Kennie Malamula, IPC/A Martha Katole, IPC/A Odrick Nkhata, IPC/A USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 89 Brian Munthali, IPC/A Martha Kaso, IPC/A SWAM Dwangwa PPA: mini-FGD with IPC/As Eliza Ndimbwa, IPC/A Leonard K. Phiri, IPC/A Dwangwa PPA: mini-FGD with NASO and SWAM project partner representatives Lawrence Chiwaya, Project Officer Mphatso Chiwaya, SWAM Field Officer (Nkhotakota) (NASO/SWAM) Dwangwa PPA: FGD with male plantation workers Fiskani Chavula, Male Sugar Plantation Worker Benson Mchenga, Male Sugar Plantation Worker Leonard Shadreck Banda, Male Sugar Plantation Worker Peter Chipeta, Male Sugar Plantation Worker John Banda, Male Sugar Plantation Worker Emmanuel Mwale, Male Sugar Plantation Worker Thom Matiyasi, Male Sugar Plantation Worker Kondwani Mwakibinga, Male Sugar Plantation Worker Austin Musongolo, Male Sugar Plantation Worker Kinex Anguleti, Male Sugar Plantation Worker Zex Limpu, Male Sugar Plantation Worker Kopa Robert, Male Sugar Plantation Worker Jones Paile, Male Sugar Plantation Worker Patrick Chimwaza, Male Sugar Plantation Worker Kinnason Manda, Male Sugar Plantation Worker Geofred Davie, Male Sugar Plantation Worker Steve Howa, Male Sugar Plantation Worker Marko Boxer, Male Sugar Plantation Worker John Katete, Male Sugar Plantation Worker (NASO/SWAM) Dwangwa PPA: FGD with female plantation workers Ethel Mhone, Female Sugar Plantation Worker Stella Saka, Female Sugar Plantation Worker Madalitso Phiri, Female Sugar Plantation Worker Farileni Kamanga, Female Sugar Plantation Worker 90 USAID/MALAWI: EBT Prev Mid-Term Evaluation Florence Manda, Female Sugar Plantation Worker Dalitso Paliani, Female Sugar Plantation Worker Ireen Chisunkha, Female Sugar Plantation Worker Anastasia Konde, Female Sugar Plantation Worker Enelesi Lende, Female Sugar Plantation Worker Alice Yohane, Female Sugar Plantation Worker Jessy Banda, Female Sugar Plantation Worker Joice Mwafulirwa, Female Sugar Plantation Worker Lucy Byson, Female Sugar Plantation Worker (NASO/SWAM) Dwangwa PPA: FGD with female members of fishing community Mary Chiwaya, Fish Woman Royce Kamphangwe, Fish Woman Martha Muyereka, Fish Woman Julia Phiri, Fish Woman Mary Chimphero, Fish Woman (NASO/SWAM) Dwangwa PPA: FGD with fishermen Oswald Chitanje, Fisherman Yotamu Manda, Fisherman Emmanuel Sankhani, Fisherman Simon Kankhomba, Fisherman Amuli Umandi, Fisherman Daudi Mbewe, Fisherman Charles Noah, Fisherman Dwangwa PPA: FGD with MSM Peer Educators Names available on request Dwangwa PPA Matiki Health Centre, Illovo Sugar Company: KII with hub health worker Damson Msiska, HIV and AIDS Coordinator and Clinician Dwangwa PPA Msenjere Health Centre: KII with hub health worker Thomas Nampuntha, Clinical Officer USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 91 APPENDIX C. EVALUATION METHODOLOGY (THE EVALUATION PLAN, DRAFT 2) This Appendix comprises the Evaluation Plan in its final iteration, i.e. after review from USAID Malawi; as such it responds to the points raised by Chimwemwe Chitsulo, USAID Malawi Monitoring, Evaluation and Learning Specialist, in his e-mail of October 24, 2012. It has also been updated to reflect the final fieldwork schedule and sample. However, it remains in large part the document submitted to USAID Malawi on October 21, 2012. Map of Malawi, showing the six PPAs visited by the EBT Prev mid-term performance evaluation team EVALUATION PLAN: TABLE OF CONTENTS 1. INTRODUCTION 2. DESCRIPTION OF THE EBT PREV PROJECT 3. THE PURPOSE AND SCOPE OF THE EVALUATION 4. THE EVALUATION FRAMEWORK 5. RISKS AND LIMITATIONS 6. EVALUATION METHODOLOGY 7. DATA COLLECTION, DATA ANALYSIS METHODS AND DATA SOURCES 8. SAMPLING METHODS 9. THE EVALUATION PLAN MATRIX 92 USAID/MALAWI: EBT Prev Mid-Term Evaluation 10. THE EVALUATION SCHEDULE 11. EVALUATION TEAM MEMBER RESPONSIBILITIES 12. LOGISTICS 1. INTRODUCTION This second draft of the evaluation plan has been informed by the meetings held with USAID, PSI/M, PACT and NAC during the week of October 15, 2012, as well as by a number of Team Planning Meetings. Because the evaluation will be conducted as a participatory process, this evaluation plan has been revised, based on those key initial meetings and team opportunities to work together. While the evaluation team will obviously maintain its independence of data collection, analysis and findings throughout the assignment, the limited time available requires optimal use of in-country resources. These include PSI/M’s respected and comprehensive strategic information and monitoring and evaluation (M&E) database, which will be examined by the team wherever appropriate in order to respond to evaluation questions and to triangulate information sources. The evaluation team has finalized focus group discussion (FGD) guides and key informant interview (KII) guides, as well as a short quantitative management survey for use in undertaking fieldwork data collection (see sections 6, 7 and 9 and Appendix D of the evaluation report). Logistical arrangements, the full evaluation schedule and field tools have been finalized. Tools were pre-tested in Lilongwe Old Town PPA on October 20, 2012. The evaluation data collected through the evaluation FGDs and KIIs will be analyzed and triangulated with project data; analysis and findings will serve to strengthen responses to the evaluation questions as set out in the Scope of Work. 2. DESCRIPTION OF THE EBT PREV PROJECT The start date for EBT Prev was March 1, 2009, and the projected end date is February 28, 2014. Current funding is to a total of USD 24,498,000, of which USD 400,000 is dedicated to activities addressing gender-based violence (GBV) and USD 4,580,000 for Voluntary Male Medical Circumcision (VMMC) activities. The purpose and objectives of EBT Prev are aligned with the 2009 Government of Malawi (GoM) and PEPFAR Partnership Framework and the Partnership Framework Implementation Plan, both of which instruments are aligned with the GoM National HIV Prevention Strategy 2009–2013 (and also with the Malawi National HIV and AIDS Strategic Plan 2012–2016). The HIV Prevention Strategy prioritizes evidence-based, engendered, complementary prevention interventions and harmonizes biomedical and behavioral interventions. EBT Prev objectives reflect these core national strategic priorities. The purpose of the Evidence-based Targeted Prevention project is to support the Government of Malawi’s National AIDS Framework (NAF) goal of prevention for populations and settings in high-risk areas. NAF Objective 1.1 is to reduce new HIV infections in Malawi through sexual transmission; there are six intervention areas within this overall objective, which EBT Prev addresses through the following main project objectives: • Identify, segment and profile priority populations at risk; • Deliver integrated, behavior change communication programs targeted to high-risk populations in priority prevention areas; • Distribute and promote condoms for use by the general population and for high-risk groups; USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 93 • Enhance the network of existing providers for greater accessibility and services to high￾risk populations. The year 2012 saw the start of activities to deliver VMMC in Thyolo and Blantyre districts. A new objective has been added (as per the EBT Prev Project Monitoring Plan, in the draft version dated August 22, 2012): • VMMC service delivery in Thyolo and Blantyre districts and associated demand creation in Blantyre district. As from 2010, EBT Prev has deepened its attention to action against gender-based violence and its role in transmission of HIV. The EBT Prev manual has been updated, health providers have been trained and communication activities implemented. The February 2011 Malawi HIV Prevention Partners’ trip report appears to have informed the enhanced focus on these two intervention areas, and additional funding was received from the Gender Challenge Fund in 2010. EBT Prev seeks to promote normative change and to increase preventative and safer sex behaviors among high-risk populations. To this end, EBT Prev implements targeted combination prevention activities in 18 Priority Prevention Areas (PPAs), working with members of fishing communities, market vendors, commercial sex workers, men who have sex with men (MSM) and estate (plantation) workers. The pilot PPA was Dwangwa, where activities began in October 2009; since then a further 17 PPAs have become operational, with two more to be added as from October 2012. PSI/M has worked in partnership with PACT from the outset of EBT Prev, to deliver PPA interventions using the “hub and spoke” model. PACT works with a number of sub￾recipients, such as MACRO, NACC and the Society for Women with AIDS, Malawi (SWAM). Community-based organizations (CBOs) have had capacity strengthened and health systems have been supported so as to develop, implement and sustain effective client registration and referral mechanisms. 3. THE PURPOSE AND SCOPE OF THE EVALUATION 3.1 The Purpose and Scope of the Mid-Term Evaluation of EBT Prev As set out in the Scope of Work, the primary purpose of the mid-term evaluation of EBT Prev is “to determine the extent to which the project’s behavior change and communications’ interventions have resulted in reduction of risky behavior among the Most￾At-Risk-Populations (MARPs).” A secondary objective is to understand the extent to which the program approach and multiple components have been effective. The scope of the evaluation will be to address the thirteen evaluation questions, grouped under six main questions, as set out in 3.2. By so doing the evaluation will assess the extent and efficacy of project objectives, through a performance evaluation approach, as described in section 4 of this evaluation plan. 3.2 The Evaluation Questions 1. How effective have PSI’s EBT Prev Communications approach and Targeted Outreach Communication (TOC) been in supporting adoption of safer sexual behaviors among project target groups? 1a. Are communications activities adequately tailored to the different categories of populations amongst the target groups—including commercial sex workers and MSM? (i.e. responsive to specific barriers faced and sufficiently skills-oriented to support behavior change?) 1b. To what extent has the frequency in which all target populations have been reached with messages supported the adoption of safer behaviors, including increased demand for and utilization of appropriate HIV and reproductive health services and commodities? 94 USAID/MALAWI: EBT Prev Mid-Term Evaluation 1c. To what extent has the project supported synergies between different communication interventions and [been] responsive to service priorities likely faced by populations, such as integrated PMTCT/ART and related GBV services? 2. What effect has the addition of new program areas, specifically Voluntary Medical Male Circumcision and Gender-based Violence, had on PSI’s operational management and capacity to implement quality assurance? (This question is answered in two parts, because PSI action on VMMC and its responses to GBV require separate attention.) 3. To what extent is the current project monitoring and evaluation framework measuring indicators that are appropriate and sufficient to demonstrate the value of the Social Behavior Change Communications’ approach? 4. To what extent have the CBOs and interpersonal communication (IPC) volunteers successfully assimilated the capacities introduced through the capacity building efforts? 4a. What elements of these new capacities will CBOs and IPC volunteers sustainably continue to implement without the support of EBT Prev? 5. How effective have EBT Prev activities been in strengthening the network of community￾based HIV and reproductive health providers? 5a. To what extent has EBT Prev’s support to the provider network improved referral systems for the priority target groups? 6. How effectively has PSI maintained a balance between social marketing of condoms and free distribution of male and female condoms for the priority target groups? 6a. How have the dynamics of this balance affected access to condoms for the priority target groups? No additional evaluation questions were added during assignment negotiations. 4. THE EVALUATION FRAMEWORK 4.1 Performance Evaluation: definitions and scope The EBT Prev mid-term evaluation will be a performance evaluation. Its overall objective will be to provide a detailed examination of EBT Prev project activities, achievements and challenges to date, using primarily qualitative methodology, with some limited quantitative collection of data through a management survey and analysis of project data where appropriate. The evaluation will additionally provide recommendations as to the way forward for the remainder of the project, to end February 214. The 2011 USAID Evaluation Policy defines performance evaluations as focusing on “descriptive and normative questions: what a particular project or program has achieved (either at an intermediate point in execution or at the conclusions of an implementation period); how it is being implemented; how it is perceived and valued; whether expected results are occurring and other questions that are pertinent to program design, management and operational decision-making. Performance evaluations often incorporate before-after comparisons, but generally lack a rigorously defined counterfactual [as would be the case for an impact evaluation].” The UNAIDS M&E Reference Group (MERG) definition of performance is also relevant: the degree to which an intervention has operated according to specific criteria, standards or guidelines and/or achieves results in accordance with stated plans. The Scope of Work notes that the mid-term evaluation “will focus on process and implementation fidelity as it relates to achieving project goals, objectives and activities as they were envisioned, in addition to looking at some intermediate outcomes.” USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 95 The understanding of the EBT Prev evaluation team is that because this is a mid-term performance evaluation, findings and recommendations cannot consider the higher-level impact of the project. Moreover, issues of direct impact attribution will not be addressed. 4.2 Additional conceptual methods and approaches The evaluation team will also apply the following conceptual methods and approaches. 4.2.1 Optimal balance of qualitative and quantitative methods The evaluation team is mindful of the fact that there are two weeks for evaluation activities, including one week for visits to a number of PPAs and one week for analysis. Focus will be on achieving the best data collection and analysis in the limited time available. The evaluation questions require a mix of quantitative, numerical data analysis and more in￾depth descriptive and normative questions to be addressed through qualitative data collection and analysis. As an example, it is proposed in this draft evaluation plan that management questions (e.g. 5, 5a, 6 and 6a) be addressed through primarily qualitative key informant interviews (KII) and a short quantitative survey. The intention is to conduct a small number of focus group discussions and potentially also KII with members of the MARPs targeted by EBT Prev. A number of the evaluation questions (e.g. 1, 1a, 1b—and 2, to the extent the GBV actions have begun) would repay closer and more small-scale discussion and are probably best served by KII. The evaluation team will additionally make full use of the PSI/M database, its ongoing surveys and its continuous M&E activities, in order to validate and triangulate its own findings. Routine project data, collected by project IPC Volunteers and Assistants, by project partners such as COPRED and by hub health facility workers and by police VSUs, represent a rich source of information for tracking, disaggregating and analyzing progress against performance indicators. The evaluation team will triangulate its own findings against project data. (See Appendix I for raw project data provided by PSI/M and Pact.) 4.2.2 Triangulation The approach of this evaluation will be to triangulate findings and analysis wherever possible, so as to minimize bias and provide as robust an evidence base as is feasible given the limited time available. Triangulation will be addressed through document review (including project data and PSI data sets), qualitative and/or quantitative data collection methods and participant observation. In addition, the half-day project stakeholder workshop scheduled for November 5 will present further opportunity to discuss preliminary evaluation findings, analysis and recommendations. The OECD-DAC definition of triangulation is: “The use of three or more sources or types of information, or types of analysis, to verify and substantiate an evaluation.” 4.2.3 Participatory approaches The evaluation team will apply participatory approaches wherever possible. All data collection tools will ensure the optimal degree of independent input from respondents, including members of MARPs, interviewed during fieldwork. Appendix D contains a sample consent form, to ensure that each and every evaluation respondent gives fully informed consent. The team will additionally seek to ensure the evaluation workshop, scheduled for November 5 in Lilongwe, is as inclusive as is feasible of EBT Prev project stakeholder groups. Due to the participatory approach being central to the evaluation, all data collection instruments (Key Informant Interviews, Focus Group Discussions and a small quantitative management survey) have been finalized by the team members in the first week of fieldwork (week of October 15), when they were able to work together in country. 96 USAID/MALAWI: EBT Prev Mid-Term Evaluation The team leader has considerable experience of developing and applying participatory approaches and methodologies, e.g. during her 4 four years’ work as Social Development Advisor on the DFID-funded HIV and STI Management project in Nigeria. The other team members have similarly applied participatory approaches. 4.2.4 Gender-sensitive data collection and analysis The evaluation team recognizes the absolute centrality of applying a gendered perspective throughout its work. All evaluation activities will be sensitive to gender issues, barriers and challenges that may apply to women and men in the various target populations and will address the ways in which EBT Prev may have addressed these through project activities. It is noted that evaluation questions 1c and 2 specifically address gender and equity issues: the former considers communication synergies and sensitivities to demand-side (target population) service delivery needs in the context of, e.g. integrated PMTCT/ART and action against GBV; the latter addresses PSI management and operational capacities in the light of the addition of VMMC and actions against GBV. The evaluation team proposes that it will seek to review gender and equity issues relevant to all project activities. Such review will of course be dependent on available project data as well as on individuals’ and organizations’ perceptions as revealed in data collection, and also participant observation. The team leader has extensive expertise in gender analysis and in designing and implementing gender and equity-focused programmatic components and evaluations, most recently in South Africa for USAID/PEPFAR and as the gender and equity specialist member of the GAVI Alliance Independent Review Committee. 5. RISKS AND LIMITATIONS This section is written in advance of the international consultants’ arrival in Malawi and before the in-briefing with USAID Malawi, the Team Planning Meeting and the start of fieldwork (as was the case for draft 1 of this Evaluation Plan). Therefore and as often happens during an evaluation, additional risks and limitations may emerge. Should this happen, all such issues will be fully described in the final evaluation report. A potential limitation is noted in advance of fieldwork: the short period of time available for logistical arrangements after the submission of the evaluation plan. All indicators used in the Evaluation Plan (e.g. in the matrix in section 9) are drawn from the version of the Project Monitoring Plan dated August 2012. A few documents sent in advance of in-country work appear to be corrupted or cannot be opened, e.g. that entitled “EBT Prev: Monitoring.” No condom data have been seen in advance of the arrival of the international consultants in Malawi. Some documents were received only on October 12; there was not time to review them and to incorporate information into data collection tools and the matrix. These document issues have inevitably had an impact on full preview of project records and development of the evaluation plan matrix and data collection tools (section 9 and Appendix D). 6. EVALUATION METHODOLOGY In response to the SOW requirements, this section sets out the proposed indicators for each of the evaluation questions and sub-questions. The EBT Prev Project Monitoring Plan (August 2012) has been used as the document from which project indicators have been taken. It is noted by the evaluation team that official USAID and PEPFAR indicators are usually the ones used to measure intervention outputs; these have not been seen (at least in USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 97 their entirety) by the evaluation team. (Please see the body of the report for any changes made to choice of indicator during fieldwork.) Question 1. Indicator 5 (1st)—new PEPFAR indicator P8.3.D: Number of MARPs reached with individual and/or small group level interventions that are based on evidence. Question 1a. Indicator from Objective 2.5 (1st)—new PEPFAR indicator P8.1.D: Number of the targeted population reached with individual and/or small group level preventive interventions that are based on evidence and/or meet the minimum standards required. Question 1b indicator. For target MARPs generally, indicator 1a (Purpose/ Outcome level): Reported condom use at last sex with non-spousal, non-cohabiting partner among general population. For RH services, indicator 4 (1): Number of people referred to health services and/or HIV-related services by PACT partner. For health commodities, indicator 3 (1a): Increased penetration of male and female condoms in high-risk sales outlets within targeted PPAs. Question 1c indicator. For integrated PMTCT/ART, new PEPFAR indicator H2.3.D: Number of health workers who successfully completed an in-service training program. For action on GBV, PEPFAR indicator P1.2.2.D: Number of people reached by an individual, small group, or community-level intervention or service that explicitly addressed gender-based violence and coercion linked to HIV. Question 2. Indicator 1, for GBV—EBT Prev performance indicator (4.6): Number of people reached by an individual, small group or community-level intervention or service that explicitly addresses gender-based violence and coercion related to HIV/AIDS. Question 2. Indicator 2, for VMMC—new EBT Prev indicator (5.3): # of health care workers who successfully completed a male circumcision in-service training program Question 3 indicator. This question is an over-arching examination of whether the current project M&E framework contains appropriate and adequate indicators; as such, it is a meta￾analytical issue, one that will need to be addressed through attention to the other evaluation questions and performance indicators. Question 4 indicator. For CBOs: EBT Prev performance indicator 4.1: Number of people referred to health services and/or HIV-related services by PACT partner. For IPC: EBT Prev indicator 2.5: Number of MARP reached with individual and/or small group level interventions that are based on evidence Question 4a indicator. This question addresses how the CBOs and IPC/Vs will sustainably continue to implement activities once EBT Prev support ends. As such, it cannot be evaluated through an existing project performance indicator, but will be considered through participant observation and interviews. Question 5 indicator. EBT Prev performance indicator 4.3: Number of health service workers/providers and volunteers trained in order to participate in user-friendly referral system and/or provider network Question 5a indicator. EBT Prev performance indicator 4.5: Referral uptake rate Question 6 indicator: EBT Prev performance indicator 2.6: Percentage of sexually active adults (20–49) that perceives condoms as being a safe and effective method for preventing HIV/STI Question 6a indicator. EBT Prev performance indicator 1a: Increased penetration of male and female condoms in high-risk sales outlets within targeted PPAs 7. DATA COLLECTION, DATA ANALYSIS METHODS AND DATA SOURCES This part of the Evaluation Plan combines points 6 and 7 as described in the SOW, in order to provide a more coherent overview of data collection, analysis methods and sources of 98 USAID/MALAWI: EBT Prev Mid-Term Evaluation information. The matrix in section 9 contains all the information in more detail and such information will not be repeated here. There is considerable expertise on the evaluation team in the development and use of qualitative and quantitative instruments. Please see Appendix D for the full set of data collection KII and FGD guides, the quantitative surveys and question lists for meetings with a number of key project partners. 7.1 Data collection 7.1.1 Overview A mixed method approach will be used to gather data to respond to the SOW evaluation questions. The key methods to be used throughout will be: • Key Informant Interviews with one person at a time (qualitative) • Focus Group Discussions (qualitative) • Two small-scale surveys, the first with representatives of project NGO partners and the second with IPC volunteers, IPC Assistants and Peer Educators (quantitative) • Participant Observation—at PPA sites, in health facilities, etc. (of drama, IPC sessions, etc.) • Meetings with key project partners and other stakeholders (qualitative) • Review of PSI data • Review of other relevant documentation, e.g. GoM strategies and statistics, other projects’ reports Because this is a mid-term, performance evaluation (see also 4.1, above), with the USAID required focus on descriptive and normative approaches, the emphasis in data collection will be on qualitative methods. Qualitative FGDs and KIIs allow for detailed consideration of issues such as behavior change triggers, understanding of choice, issues and barriers, through the use of open-ended questions and enabling all respondents to have sufficient time to consider and discuss. Open-ended questions and probing can result in-depth responses about people’s experiences, perceptions, opinions, feelings, and knowledge. Therefore, semi￾structured interview guides have been developed by the evaluation team: these allow respondents to reply individually, with probing questions as follow-up to ascertain detailed understanding and opinions. Semi-structured interviews also allow respondents to shape discussion as well as the moderator, which can result in richer data. See section 8 below for a discussion of the six project PPAs selected for field visits and the rationale underpinning the choice of those PPAs. 7.1.2 Qualitative data collection methods Meetings with key project partners and other stakeholders The following meetings were conducted in week 1 of the mid-term evaluation (October 16– 19), with the following key project partners: 1. USAID/Malawi 2. PSI/Malawi 3. PACT Malawi 4. National AIDS Commission 5. UNAIDS 6. UNFPA 7. MACRO 8. CEDEP 9. JHU Bridge II project USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 99 The over-arching objective of all the week 1 meetings was to obtain project partners’ and other stakeholders’ views on the relative efficacy of the EBT Prev project (all the above stakeholders), including in the context of the overall national HIV environment and the evolving nature of the epidemic in Malawi. In addition, discussions were held with USAID, PSI/M and Pact regarding the evaluation methodology and data collection. USAID also reviewed draft 1 of this evaluation plan and made recommendations for review and addition; these have all been addressed in this final draft of the evaluation plan. Focus Group Discussions and Key Informant Interviews FGD and KII guides have been developed for all respondent groups, based on the evaluation questions as set out in the SOW and focused on assessing different groups’ perspectives, achievements, requirements, etc. (be these CSWs, female vendors, project partner IPC/Vs, other project partner representatives such as project officers, or NAC members of staff). During week 1, the evaluation team finalized the overall number of MARP KII and FGD to be conducted, based on the USAID in-briefing and a follow-up discussion and separate meetings with PSI/Malawi and PACT. Furthermore, a number of KII are to be held across the six PPAs with outreach workers (PPA Project Officers, IPC/V and IPC/A, Queens and MSM Peer Educators and with other stakeholders, such as Hub Focal Points, HCT Counselors, condom distributors and representatives of the police Victim Support Unit. At the time of writing this second draft of the Evaluation Plan, the data collection tools have been pre-tested in Lilongwe Old Town PPA, on October 20, 2012. 100 USAID/MALAWI: EBT Prev Mid-Term Evaluation Qualitative data collection in the six PPAs District PPA FGD and KII Lilongwe Old Town MSM (KII) CSW (FGD) IPC/V (FGD) IPC/A (KII) Outreach workers and other project stakeholders Mwanza Town and border CSW (FGD) Vendors (FGD Outreach workers and other project stakeholders Thyolo Satemwa Tea Estate Plantation workers (FGD) IPCV (FGD) IPCA (KII) Outreach workers and other project stakeholders Zomba Songani Market Vendors (FGD) IPCV (KII) IPCA (KII) Outreach workers and other project stakeholders Mangochi MALDECO Fishing community (FGD) CSW (FGD) Outreach workers and other project stakeholders Nkhotakota Dwangwa Plantation workers (FGD) Fishing community (FGD) MSM (KII) Outreach workers and other project stakeholders In total, the evaluation team plans to conduct 10 focus group discussions (FGDs) with two groups each of CSW, vendors, plantation workers, fishing communities and also with IPC volunteers. The evaluation team will conduct at a minimum six KII with Interpersonal Communication Assistants (IPC/A), IPC volunteers (IPC/V) and men having sex with men (MSM). The rationale for choosing the sample is an attempt to achieve appropriate engagement with all project MARP stakeholders and partners, within the limited time available for fieldwork (six days). Each of the three evaluation team members will undertake a number of FGDs and KIIs, always with the intention to allow respondents sufficient time to address questions in the semi-structured interview guides. None of the FGDs or KIIs will be recorded and confidentiality will be assured (see the Consent Form in Appendix D). Evaluation team members will take detailed notes; these will form the basis of analysis. Since many of the MARP FGDs and KIIs will be conducted in the Chichewa language, an expert translator has been hired to assist. Quantitative data collection Two quantitative surveys will be administered, one with a minimum of eight representatives of the project partners (CEDEP, SWAM, etc.) and one with a minimum of five from each of USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 101 the following groups of project outreach workers: IPC/V, IPC/A and Peer Educators. The sample size has been determined by a wish to achieve a representative overview of project partners’ engagement with the evaluation questions, within the time available. As with the qualitative data collection, the quantitative surveys address evaluation questions as set out in the SOW: sustainability, quality assurance, engagement with MARPs, etc. In addition, the evaluation team will review the quantitative data routinely collected by PSI/M and Pact, in order to track progress over the lifetime to date of the project, in terms of performance indicators, as set out in the Project Monitoring Plan. The team will review project quarterly and annual reports, its baseline and other research findings, so as to triangulate field findings and points raised in other interviews. 7.2 Data Analysis methods The following needs to be emphasized: qualitative analysis cannot ever be totally prescriptive, i.e. it has to allow for a potentially wide range of responses to descriptive and normative questions by individuals whose needs, views and knowledge may be considerably different, as may their world views, based on many variables, e.g. sex, occupation, marital status, socio-economic status, education, life stage and life experience. In addition, data collection for the mid-term evaluation will span two weeks, with six days in the six PPAs chosen as field sites; dedicated time for analysis will be limited to 3.5 days in country. Therefore, elaborate qualitative coding approaches are not feasible in the time available. Thus the over-arching approach to the mid-term evaluation data analysis is what has been termed a “general realist approach,” while applying the primarily inductive methods set out below. The evaluation team will apply the Convergent Mixed Methods Parallel Research Design (CPRD), which consists of four distinct steps: 1. the simultaneous design and implementation of data collection approaches; 2. separate analysis of qualitative and quantitative data; 3. Triangulation of quantitative and qualitative results; and, 4. interpretation of the merged results (see Creswell & Clark 2007)4 The evaluation team proposes the use of the CPRD because of its comparative advantage in team members with skills both in qualitative and quantitative methods of data collection and analysis. This is a performance evaluation: as is described in the 2011 USAID Evaluation Policy, such evaluations are primarily qualitative and address descriptive and normative questions. Bearing this in mind, and with due attention to the realities of the time available for fieldwork and analysis, both qualitative and quantitative data collection and analytical approaches will be applied, in parallel. Furthermore, the use of both qualitative and quantitative methods will enable validation and corroboration of the results. The chart below illustrates this approach diagrammatically. 4 Creswell, John W. and Vicki L. Plano Clark. Designing and Conducting Mixed Methods Research. (2d ed.) Sage Publications: 2010. 102 USAID/MALAWI: EBT Prev Mid-Term Evaluation Proposed steps in evaluation data analysis using the CPRD design Adapted from Creswell & Clark (2006) (1st ed.). Interpret the merged results Summarize and interpret the separate results • Discuss to what extent and in what ways results converge, diverge, relate to each other and/or produce a more complete understanding Merge and Triangulate the two sets of results • We will identify content areas represented in both the qualitative and quantitative sets and compare, contrast and/or synthesize the results in a discussion or table • We will also identify differences within one set of results based on dimensions within the other set and examine the differences within a Analyze the quantitative data • We will analyze the quantitative data, to establish any frequencies, commonalities, divergences, using descriptive t ti ti Analyze the qualitative data • We will analyze the qualitative data using thematic and parallel analysis procedures of “Th ti A l i ” The quantitative strand • Quantitative evaluation questions have been developed by USAID/PSI (SOW) • We have developed questionnaires for IPC/A, IPC/Vs and other project partners • We will develop data analysis templates for summarizing project The qualitative strand • Qualitative evaluation questions have been developed by USAID/PSI (SOW) • We will collect open-ended data using FGD and KII guides and participant observation Step 1 Step 2 Step 3 Step 4 USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 103 The CPRD design will include use of parallel analysis, with a small degree of what is called multi-level analysis. ANALYTICAL METHOD DESCRIPTION APPLICATION Parallel analysis Several data sets (e.g. data from KII and the management survey) are analyzed separately, reviewed jointly and findings, conclusions and recommendations developed Triangulation approach key in this regard Qualitative data lead in parallel analysis, with quant data used to back up/substantiate or indeed qualify other analysis Multi-level analysis Both qualitative and quantitative analytical methods are applied, in the case of the EBT Prev evaluation through the management survey and KII, FGD, PO This approach allows deeper emphasis on quality of interventions, through two-level data collection and analysis; in this case, management perceptions and those of primary stakeholders and other actors The qualitative data from the various FGD and KII sessions will also be analyzed by applying a thematic approach.5 In adopting this approach, the evaluation team will develop a list of potential themes during the fieldwork phase, corresponding to the evaluation questions and linked indicators. The FGD and KII data will be analyzed according to emerging themes. We will code and analyze the data manually to identify trends (and frequencies) in the appearance of themes across the different FGD/KII sessions, always bearing in mind variables (e.g. type of respondent, sex, location). Such thematic analysis allows for the identification of potential crosscutting themes, e.g. changing risk perceptions, The following is a theoretical example of the analytical processes that will be applied. Step 1: Qualitative analysis of evaluation question 1a—Are communications activities adequately tailored to the different categories of populations amongst the target groups – including commercial sex workers and MSM? (i.e. responsive to specific barriers faced and sufficiently skills-oriented to support behavior change?). MARP analysis. 5 For a detailed description of thematic analysis, see Braun, V. and V. Clarke. “Using Thematic Analysis in Psychology.” Qualitative Research in Psychology, 3(2):77–101, 2006. 104 USAID/MALAWI: EBT Prev Mid-Term Evaluation Sample variables Initial response (incl. recall + mode of delivery, e.g. IPC) Do project comms speak to me? If yes, why and how? If no, why not? Is there any link between project comms + any individual BC? Probed discussion re BC, e.g. sustainability Any proposed improvements to messages + delivery? MARP category Sex Location Whether IPC/V Crosscuttin g, e.g. sex, life stage Step 2: Thematic Analysis (theoretical example for evaluation question 1a)—again MARPs Major BC change: going for HTC Prompt/s, e.g. IPC/V support, availability of MACRO mobile clinic Had test? If yes, any sustained BC? If no, why? If no, what next? MARP category Sex Location Whether IPC/V Crosscutting Step 3: Parallel and Multi-level Analysis of qualitative findings from steps 1 and 2 and quantitative findings from surveys Step 4: Triangulation with project data and other documentation Step 5: Interpretation of combined findings There will be a daily team meeting after fieldwork during which key issues and emerging themes will be discussed. The team will work to the report structure in its top line, daily analysis, focusing on the evaluation questions and the indicators linked to each, as set out in the Evaluation Matrix in section 9 of this Evaluation Plan. The evaluation team will finalize its analysis after fieldwork, during the week of October 29, 2012. 7.3 Data sources Please see the matrix in section 9 for information. It is noted here that the evaluation team could not be aware of all data sources in advance of meetings with EBT Prev project stakeholders and other actors. Information on this is still being collected and collated. A further set of meetings are to be held at PSI in Blantyre on October 23, after which a full overview of data sources will be undertaken. The final evaluation report will contain full details of all data sources used. USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 105 8. SAMPLING METHODS As mentioned elsewhere in this Evaluation Plan, the main approach of this evaluation will be to gather qualitative data, through meetings with key partners (PSI, Pact, USAID, etc.), focus group discussions, key informant interviews and participant observation. Two quantitative surveys will also be applied. The evaluation team proposes to visit six PPAs for data collection from the implementing partners and beneficiary groups. The table below sets out the finalized list of PPAs to be visited by the EBT Prev mid-term evaluation team. See below for the rationale behind the choice of each. PPAs to be visited during the EBT Prev mid-term evaluation PPA Region PPA type Target population/s PPA start date 1. Lilongwe Old Town Central Market (urban) Vendors CSW MSM Sept 2010 2. Mwanza Southern Border Vendors CSW Sept 2010 3. Thyolo Southern Plantation Plantation workers Vendors July 2010 4. Zomba Southern Market (rural) Vendors CSW Sept 2010 5. Maldeco Southern Fishing Men and women in fishing communities MSM CSW Vendors Sept 2010 6. Dwangwa Central Plantation Fishing Plantation workers Men and women in fishing communities Vendors MSM Oct 2009 (pilot, i.e. oldest PPA) 8.1 Rationale for the choice of PPAs The intention is to visit as representative a sample of PPAs as possible during the seven days scheduled for fieldwork. Six PPAs will be visited; this represents 33.3% of the total number (18) operational at the time of the mid-term evaluation. The choice of PPA has been extensively discussed with USAID Malawi, PSI/M and Pact; the final decision was taken by the evaluation team. It is emphasized here that while six of the eighteen PPAs will be visited, the mid-term performance evaluation cannot claim to provide a total overview of the EBT Prev project; there may be variables in other PPAs that cannot be covered or addressed. No mid-term performance evaluation can represent a complete review of a project. 106 USAID/MALAWI: EBT Prev Mid-Term Evaluation There are five MARP target populations: Commercial Sex Workers, Market Vendors, Men who have Sex with Men, men and women in fishing communities and plantation workers. The EBT Prev project works across Malawi, with a preponderance of PPAs in the southern region and fewest in the north. The table below sets out the criteria used to determine the mid-term evaluation PPA sample. PPA Selection Criteria PPA Type Plantation 29% of the PPAs can be described as plantation The evaluation will visit two: Thyolo and Dwangwa Fishing 25% of the PPAs can be described as fishing. The evaluation will visit two: Maldeco and Dwangwa Urban market 25% of the PPAs can be described as urban market. The evaluation will visit one: Lilongwe Old Town Rural market 13% of the PPAs can be described as rural market. The evaluation will visit one: Zomba Border 8% of the PPAs can be described as border. The evaluation will visit one: Mwanza PPA geographical location Southern region: eleven of the eighteen currently operational PPAs are in the southern region (61%); the evaluation team will visit four southern PPAs Central region: four of the currently operational PPAs are in the central region (22%) and the team will visit two Target populations Vendors: present in all PPAs; urban and rural market—Lilongwe Old Town and Zomba Plantation workers: Thyolo and Dwangwa Fishing communities: Maldeco and Dwangwa CSW: Lilongwe Old Town, Mwanza, Zomba, Maldeco MSM: Lilongwe Old Town, Dwangwa The issue of sample size and confidence is really not relevant for this mid-term performance evaluation, specifically in terms of the qualitative and quantitative field data collection and analysis. The evaluation is primarily a qualitative exercise to examine descriptive and normative issues. Sample precision and confidence levels would be prone to error and over￾simplification. The evaluation team has determined the overall number of KII and FGD to be conducted, based on the in-briefing and separate meetings with PSI/M and PACT. In addition, guidance will be sought from the MoH and NAC, in order to ensure the fullest possible GoM engagement. See section 7. USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 107 9. THE EVALUATION PLAN MATRIX Please note that this matrix will be discussed by the entire evaluation team, initially on Monday, October 15, and finalized during the Team Planning Meeting on October 17. Inputs will be requested from USAID and PSI (also PACT). In addition, the sole M&E document available in advance of arrival in Malawi was the PSI Project Monitoring Plan, dated August 2012. All indicators used here and elsewhere in the Evaluation Plan are drawn from the PMP. This matrix is provisional. Evaluation Question Indicator Data Collection Method/s Data Source Sampling Comments 1. How effective have PSI’s EBT Prev Communications approach and Targeted Outreach Communication (TOC) been in supporting adoption of safer sexual behaviors among project target groups? Indicator 5 (1st) (New PEPFAR Indicator P8.3.D, Essential/Reported ) KII with relevant PSI and PACT staff member/s Analysis of PSI M&E and research data KII and FGD with MARPs KII and FGD with IPC/V KII with e.g. MoH NACC, JHU-Bridge Management survey Evaluation field work data collection PSI/M M&E data, research and other documentation PSI and PACT staff members MARP members IPC/V Other actors, e.g. MOH Team discussion and analysis will play a major part in consideration of findings and development of conclusions and recommendations (this applies to all evaluation questions, but is mentioned only here) In addition, PSI M&E and other data will represent key sources for all evaluation questions, to be triangulated with field data collection, participant observation and other research documentation. 1a. Are communications activities adequately tailored to the different categories of populations amongst the target groups—including commercial sex workers and MSM? (I.e. responsive to specific barriers faced and sufficiently skills￾oriented to support behavior change?) Objective 2 5 (1st)—new PEPFAR indicator P8.1.D KII with relevant PSI and PACT staff member/s Analysis of PSI M&E and research data KII and FGD with MARPs KII and FGD with IPC/V KII with e.g. MoH Evaluation field work data collection PSI/M M&E data, research and other documentation PSI and PACT staff members MARP members IPC/V Other actors, e.g. MOH 108 USAID/MALAWI: EBT Prev Mid-Term Evaluation NACC, JHU-Bridge 1b. To what extent has the frequency in which all target populations have been reached with messages supported the adoption of safer behaviors, including increased demand for and utilization of appropriate HIV and reproductive health services and commodities? For target populations generally: Purpose/outcome indicator 1c For RH services: objective 4 (1) For health commodities: objective 3 (1a) KII and FGD with MARPs KII and FGD with IPC/V KII with health providers KII with PSI staff members resp. for condom distribution Evaluation field work data collection PSI/M M&E data, research and other documentation MARP members (all target populations) IPC/V Health providers PSI staff 1c. To what extent has the project supported synergies between different communication interventions and [been] responsive to service priorities likely faced by populations, such as integrated PMTCT/ART and related GBV services? For integrated PMTCT/ART: new PEPFAR indicator H2.3.D For action on GBV: PEPFAR indicator P12.2.D KII and FGD with MARPs KII and FGD with IPC/V KII with health providers Evaluation field work data collection PSI/M M&E data, research and other documentation MARP members (all target populations) IPC/V Health providers 2. What effect has the addition of new program areas, specifically Voluntary Medical Male Circumcision and Gender-based Violence, had on PSI’s operational management and capacity to implement quality assurance? GBV: performance indicator 4.6 VMMC: performance indicator 5.3 KII with relevant PSI and PACT and PACT subs’ staff KII with HCT counselors (GBV) KII with health workers trained in VMMC Evaluation field work data collection and analysis PSI/M M&E data, research and other documentation GBV and VMMC training reports GBV data from HCT records PSI, PACT and CBO staff HCT counselors Health workers trained on VMMC As per the question, the focus will be on organizational capacity in terms of ensuring quality assurance, at PSI office level and at the service delivery points 3. To what extent is the current project monitoring and evaluation framework See discussion in section 6 KII with relevant PSI staff Review of database: KII Evaluation field work data collection and PSI staff As noted elsewhere, this is an over￾arching question, one that will emerge through close examination of USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 109 measuring indicators that are appropriate and sufficient to demonstrate the value of the Social Behavior Change Communications’ approach? recurrent M&E and research studies analysis PSI/M M&E data, research and other documentation operations on the ground. The evaluation team will seek to garner the views of other projects’ stakeholders, e.g. JHU Bridge II 4. To what extent have the CBOs and interpersonal communication (IPC) volunteers successfully assimilated the capacities introduced through the capacity building efforts? For CBOs: performance indicator 4.1. For IPC: performance indicator 2.5 KII with CBO reps FGD with IPC/V Participant observation KII and FGD data analysis Training reports CBO representatives IPC volunteers Participant observation will be central to evaluation of this question 4a. What elements of these new capacities will CBOs and IPC volunteers sustainably continue to implement without the support of EBT Prev? See discussion in section 6 KII with CBO reps KII with PSI and PACT staff FGD with IPC/V Participant observation KII and FGD data analysis CBO reps PSI and PACT staff IPC/V This question requires projection based on current observation, i.e. currently CBOs and IPC/V are supported by EBT Prev 5. How effective have EBT Prev activities been in strengthening the network of community￾based HIV and reproductive health providers? Performance indicator 4.3 KII with CBO reps KII with PSI and PACT staff FGD with IPC/V FGD with MARPs KII with health providers (including e.g. DMO/DACC rep) Participant observation KII and FGD data analysis CBO reps PSI and PACT staff IPC/V Health providers MARP members 5a. To what extent has EBT Prev’s support to the provider network improved referral systems for the priority target groups? Performance indicator 4.5 FGD with MARP members KII with health providers (including e.g. KII and FGD data analysis Training reports PSI M&E data MARP members Health providers 110 USAID/MALAWI: EBT Prev Mid-Term Evaluation DMO/DACC rep) Participant observation 6. How effectively has PSI maintained a balance between social marketing of condoms and free distribution of male and female condoms for the priority target groups? Performance indicator 2.6 KII with PSI staff KII with condom outlet stakeholders KII data analysis PSI condom data PSI staff Condom outlet stakeholders 6a. How have the dynamics of this balance affected access to condoms for the priority target groups? Performance indicator 1a FGD with MARP members KII with PSI staff KII with condom outlet stakeholders KII and FGD data analysis PSI condom data PSI staff MARP members USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 111 10. EVALUATION TIMELINE This is the original evaluation schedule as agreed by GH-Tech Bridge II and USAID Malawi, in consultation with the evaluation team. It has been amended to reflect the Malawian national holiday on October 15. At the time of writing (October 12, 2102) it has not been finalized. A full schedule will be included in the final evaluation report. SUNDAY MONDAY TUESDAY WEDNESDAY THURSDAY FRIDAY SATURDAY OCTOBER 1 2 3 4 5 6 + two days’ Sept reading for TL Background reading Background reading Background reading Draft evaluation plan (TL) 7 8 9 10 11 12 13 OFF Draft evaluation plan Draft evaluation plan Draft evaluation plan Draft evaluation plan Draft evaluation plan Depart/Submit draft evaluation plan to USAID 14 15 16 17 18 19 20 Arrive Malawi (TL and SBCC) National holiday in Malawi In-briefing TPM Fieldwork Fieldwork Fieldwork 21 22 23 24 25 26 27 OFF Fieldwork Fieldwork Fieldwork Fieldwork Fieldwork Fieldwork 28 29 30 31 OFF Draft Report Writing, synthesize data Draft Report Writing, synthesize data Draft Report Writing, synthesize data 112 USAID/MALAWI: EBT Prev Mid-Term Evaluation SUNDAY MONDAY TUESDAY WEDNESDAY THURSDAY FRIDAY SATURDAY NOVEMBER 1 2 3 Draft Report Writing, synthesize data (SBCC and TL) Stakeholders’ Presentation on preliminary findings 4 5 6 7 8 9 10 OFF De-brief and submit report/mission review Depart country/mission review Arrive home/mission review Mission review Mission review 11 12 13 14 15 16 17 OFF Holiday Mission review/submit comments to team Draft revised by Team Leader and team Draft revised by Team Leader and team Draft revised by Team Leader and team. Draft revised by Team Leader and SBCC 18 19 20 21 22 23 24 OFF Draft revised by Team Leader SUBMIT TO GH￾TECH AND MISSION for final SIGN OFF 25 26 27 28 29 30 USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 113 11. EVALUATION RESPONSIBILITIES The details of team member responsibilities will be finalized during the Team Planning Meeting, scheduled for October 17, 2012, in Lilongwe. It will be the first opportunity for the three team members (John Kadzanwira, Iain McLellan and Janet Gruber) to discuss and plan the evaluation together. Tennyson Banda, the local Logistics Coordinator, will also have input. A description of core responsibilities for each of the team members follows. 1. Janet Gruber: Senior Evaluation Specialist and Team Leader Key responsibilities: • Overseeing the team • Finalization of evaluation design (the evaluation plan) • Planning and co-ordination • Presentation of findings and recommendations at in-country workshop • Submission of final draft report Specific technical responsibilities: • Assessment of the project’s progress towards targets and factors influencing performance • Benefits of the project strategies • Comparison of strategies with other options • Janet will also take lead responsibility for assessing the likely impact of the new VMMC component • Janet will also lead on GBV action and all other gender issues that may arise 2. John Kadzanwira: Local HIV Consultant • Extensive HIV expertise and experience in the Malawian context, e.g. GoM strategic HIV planning • Understanding and detailed knowledge of how EBT Prev supports national work with MARPs and other vulnerable populations • Understanding of the biomedical context and linkages between behavioral and biomedical HIV prevention interventions • Knowledge of community-based HIV services and referral systems • Knowledge of CBO and IPC/V capacity-building activities and likelihood of sustainability of such capacities 3. Iain McLellan: SBCC HIV Specialist • Key expertise: Social Behavior Change Communication; assessment of PSI/M’s strategic communication approaches with all target populations • Expertise on MARPs and vulnerable populations • Expertise on referral networks • Experience of community-based HIV services 114 USAID/MALAWI: EBT Prev Mid-Term Evaluation • Ideally, experience of PSI’s social marketing program, including the balance of male and female condom social marketing 4. Tennyson Banda: Local Logistics’ Coordinator Support to scheduling stakeholder meetings, KII and FGD and to organize field visits. Also to organize transport and accommodation for team members. 12. LOGISTICS The following meetings have been held in the first week of the evaluation (October 15–20): • USAID Malawi: October 16 and 19 • PSI/Malawi: October 16 • Pact Malawi: October 16 (plus a Skype call with Kate Musimwa October 18) • National AIDS Commission: October 17 • UNAIDS: October 18 • UNFPA: October 18 • JHU Bridge II project: October 18 • MACRO: October 19 • CEDEP: October 20 Fieldwork The following PPAs are to be visited; arrangements have been made by the evaluation team in partnership with PSI/M and Pact. PPA Location Date Lilongwe Old Town Central October 20 Mwanza Southern October 22 Thyolo Southern October 23 Zomba (Songani) Southern October 23 Maldeco Southern October 25 Dwangwa Central October 27 Fieldwork ends on October 27; thereafter the evaluation team returns to Lilongwe to undertake analysis and draft report writing in the week of October 29. A debrief will be given to USAID on the morning of November 2 and a half-day partners’ meeting is to be convened on November 5. USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 115 00 APPENDIX D: EBT PREV MID-TERM PERFORMANCE EVALUATION QUALITATIVE AND QUANTITATIVE DATA COLLECTION INSTRUMENTS TOOL A: FDG AND KII GUIDE FOR VENDORS, COMMERCIAL SEX WORKERS, PLANTATION WORKERS, FISHING COMMUNITIES AND MEN WHO HAVE SEX WITH MEN (QUALITATIVE) Message frequency and demand creation (1b) 1) What messages about HIV have you heard in your community? What did they suggest you do about preventing HIV? 2) What messages about HIV have you heard about from people in person (IPCV peer educators or other outreach like drama)? What did they suggest that you do about preventing HIV? 3) After hearing the messages in the mass media or from people in person, what actions, if any, did you specifically take to prevent HIV? (Probe condom use, HTC, GBV and couple communication.) Adequately tailored (1a) 4) To what degree has this communication on HIV addressed you directly or people like you? 5) How would you describe the language used, in terms of being clear and easy to understand? 6) To what extent is the content relevant to you and people like you in your community? 7) To what degree are interpersonal communications (by IPCVs and Peer Educators) on HIV communicating directly to you or people like you? 8) For example, when considering communication on the theme of Gender-Based Violence, to what extent was it reflective of realities in your communities and easy to understand and act upon? 9) How do you perceive your risk of being infected with HIV today, compared to what it was two years ago? If it has changed, what caused the change? 10) In what ways are you better prepared to prevent HIV infection today (by using condoms, reducing partners, communicating with partners and accessing HTC) compared to before being exposed to recent HIV messages? 11) What means are you employing today to keep from becoming infected with HIV or re￾infected (explain re-infected)? Balance of free vs. socially marketed condoms (6) 12) Where do you currently obtain condoms, and what is your experience with getting them for free or paying for them? 13) What are the obstacles you face in using condoms regularly? 14) How affordable and easy to obtain are condoms? 15) What is your experience of being referred to HIV-related services and/or Family Planning and Reproductive Health Services? 116 USAID/MALAWI: EBT Prev Mid-Term Evaluation 16) What has been your experience with mobile HIV Counseling and Testing (underwent testing, barriers to testing)? What or who influenced you to go? How did you hear about it and from what sources (IPCV, TOC, drama, video)? 17) Elaborate on your exposure, if any, to the topic of making choices regarding HIV (Lingalira Sankha Wekha through IPCV, TOC, drama and video)? 18) Elaborate on your exposure, if any, to the theme of Gender-Based Violence (IPCV, TOC, drama, video)? 19) What was the impact in your community, if any, of the Gender-Based Violence communications? 20) Elaborate on your exposure, if any, to the theme of Preventing Mother to Child Transmission and Life-Long ART? 21) How did you find the approach used to communicate on these topics, and how useful were they? 22) What could be done to improve the way these and other themes are communicated to you and people like you in the future? 23) What is your experience, if any, with peer educators or people conducting outreach who are like you or do the same work as you? 24) What support materials have you seen such as flannelographs, game boards, picture cards or others? 25) What has been your experience, if any, with HTC mobilization events, TOC videos and drama performances on HIV? TOOL B: FGD FOR IPC/V (QUALITATIVE) 1. What are your current IPCV tasks? 2. What is your involvement with Targeted Outreach Communication? 3. Describe your training (probe on issues such as gender and HIV, access, barriers to treatment). 4. Describe the utility of the IPCV guide. 5. What target populations have you worked with? 6. What are the challenges in working with those populations (probe on individual MARP members’ challenges, e.g. gender aspects, life stage, level of education, etc. as well as implementation challenges, e.g. finding MARPs twice for each quarterly message, signing twice, etc.)? 7. What support materials do you have to work with (flannel boards, games, posters, etc.)? 8. What additional support materials would be useful? 9. What encourages you most to continue your work as a IPC/V? 10. How do the IPCAs help you? 11. Describe the last time you were visited when conducting IPC and got help. 12. What different themes have you communicated to target populations? 13. Compare the level of difficulty in communicating the different themes. 14. How do you go about making referrals to services? 15. What usually happens after you make referrals to services, and how do you know? 16. Describe your relationship with those who conduct services. 17. What additional help could you receive to make it easier to do your IPCV work? 18. What other forms of communication have supported you in your IPCV work (radio, posters, billboards)? 19. What could be done to increase the number of people from target populations who are reached and are inspired to take up HIV-related services? 20. What evidence have you seen of the target population making positive changes in behavior like condom use, reducing the number of partners, accessing HTC? USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 117 TOOL C: PPA PROJECT MANAGERS/OFFICERS AND HEALTH SERVICE HUB FOCAL POINTS (QUALITATIVE) Strengthened service providers (5) 1) How has the response of the health service hub improved its ability to refer most-at-risk populations to HIV and FP services as a result of the EBT Prevention Project? Improved referrals (5a) 2) In what ways has the system of referrals of most-at-risk populations to HIV services been improved by the EBT Prevention Project? 3) What are the main obstacles to successful use of services by most-at-risk populations? 4) Crosscutting: gender training? Stigma? Quality assurance of services offered to MARPs in new system? TOOL D: HEALTH WORKER KII GUIDE (QUALITATIVE) 1. Briefly explain what you know about the EBT Prev Project. (Check if PSI, PACT, USAID are mentioned, IF not, probe if the respondent just forgot to mention them.) 2. What is your (facility) involvement in the EBT Prev project? 3. How has the response of the catchment population of this health facility (service hub) changed since the start of the EBT Prev project? (CHECK : client numbers for HTC, condoms, FP, PMTCT, general health care etc…) 4. Which population groups are you serving most at this facility for the following services: HTC?, condoms?, FP?, STI?, general health care? (CHECK if MARPs are mentioned, IF NOT, please PROBE) 5. How have services in general at this facility changed as a result of the EBT Prevention Project? 6. In what way has HIV prevention work changed in your community because of the EBT Prev project, if at all? 7. In what ways has the system of referrals of most-at-risk populations to HIV services been improved by the EBT Prevention Project? 8. What are the main obstacles to successful use of services by most-at-risk populations? 9. Would you provide statistics for the following services (if possible, comparing the period before EBT and after) : HTC, STI, FP, PMTCT, condoms. 10. Crosscutting: gender training? Stigma? Quality assurance of services offered to MARPs in new system? TOOL E: HTC COUNSELOR KII GUIDE (QUALITATIVE) Evaluation question 5a. To what extent has EBT Prev’s support to the provider network improved referral systems for the priority target groups? 1. Please describe your work as an HTC Counselor (probe on mobile clinics, PPAs and sites visited, frequency of visits). 2. What is your involvement with the EBT Prev project (probe on links within PPAs with PPA Project Officers, IPC/V Assistants)? 3. (How) has engagement with the project affected the individual HTC Counselor’s service delivery and its quality? 4. Training by the project (including any refresher). 118 USAID/MALAWI: EBT Prev Mid-Term Evaluation 5. Management arrangements (supportive supervision, reporting, etc.). 6. Views on reporting: format of forms (user-friendliness, etc.), frequency of reporting, appropriateness, use of data, feedback, etc. 7. To what extent has EBT Prev been successful in improving the referral system linking MARPs who come for HTC to relevant services? 8. GBV: probe in depth on project links on this, individual’s training, feedback to IPC/Vs and IPC/As. 9. What is your view on the quality of the referral system to get clients to your clinic? 10. What is your view on the timeliness and quality of the services accessed by your HTC clients if you refer them on (probe on whether there is adequate feedback to HTC counselors on onward referrals and actions)? 11. Crosscutting: gender training? Stigma? Quality assurance of services offered to MARPs in new system? TOOL F: KII EVALUATION TOOL FOR PROJECT PARTNERS (LOCAL NGOS) - QUALITATIVE Synergies (1c) 1) To what degree have EBT Prevention Project communications promoted specific HIV￾related services (condoms, HTC, ART, and GBV)? 2) In what ways has the EBT Prevention Project enabled your organization to better link with or provide target populations with specific HIV-related services (condoms, HTC, PMTCT, ART, and GBV)? Improved IPC capacity (4) 3) How has the EBT Prevention Project improved the capacity of your organization to implement HIV prevention programming? What could be done to improve this capacity? 4) How has the EBT Prevention Project improved the capacity of Interpersonal Communication Volunteers? What could be done to improve this capacity? Sustaining CBO and IPC capacities (4a) 5) What capacities developed by your organization and the IPC volunteers during the EBT Prevention Project are likely to be sustained into the future after it ends? Strengthened service providers (5) 6) To what degree has the EBT Prevention Project been successful in improving the referral system linking MARPs to relevant services and the quality of those services? Improved BCC and IPC capacity (4) 7) To what degree has your organization improved its capacity for conducting Behavior Change Communication and Interpersonal Communications interventions? 8) What are the most important skills your organization has developed for conducting BCC and IPC in the future? TOOL G: FOR PSI AND PACT (QUALITATIVE) 1) Management of the project: relative effectiveness of partnership, strengths and weaknesses. Lessons learned to date about partnership arrangements and any perceived implications for project components. USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 119 What is the value added of both PSI and Pact in working together? What is the role of the other partner (PSI or Pact) in the EBT Prev Project? How did the collaboration come about? VMMC and GBV impact on operational management and quality insurance (2) 2) What impact, if any, have new project areas like VMMC and GBV had on PSI’s organizational management and the provision of quality interventions (probe on potential positives, e.g. integration into referral network, enhanced in-house capacity, effectiveness of IPC/Vs. Probe also on potential negatives, e.g. insufficient capacity to deliver interventions effectively and with quality assured, dilution of focus, etc.)? M&E and indicators (3) 3) To what degree has the project monitoring and evaluation framework made it possible to keep track of deliverables including changes in target population behavior? Are current indicators most appropriate and effective for measuring SBBC? 4) What difficulties have been encountered in tracking behavior changes of target populations? 5) How has information on changes in target population behaviors been used to improve the quality of programming? Free vs. Socially Marketed Condoms (6) 6) How successfully have PSI and its partners been at promoting both free and socially marketed condoms? What are the challenges and benefits in promoting both at the same time? Success in promoting free and socially marketed condoms (6a) 7) How has the promotion of free and socially marketed condoms had an impact on access to condoms by priority target populations? Specifically for Pact 1. How would you characterize Pact’s collaboration with PSI? 2. (How) were NGO partners selected before the EBT Prev Project? 3. What was the previous experience of the NGO partners in HIV/AIDS programming? 4. What were the greatest needs of the NGO partners for technical assistance under the EBT Prev Project? 5. What has been Pact’s involvement in training the NGO partners and the IPCVs? 6. What has been Pact’s involvement in developing support materials like guides, flip charts, and flannelographs? 7. What is the biggest accomplishment in the conducting of IPC, peer education and POC? 8. How would you assess the EBT Prev Project Monitoring and Evaluation tools? TOOL H: NAC (QUALITATIVE) 1. NAC’s engagement with PSI? 2. NAC’s engagement with EBT Prev? 120 USAID/MALAWI: EBT Prev Mid-Term Evaluation 3. Relationship over time, since project inception in 2009—including responsiveness to the evolving epidemic and changes in approach, e.g. PMTCT B+ and VMMC? 4. How does EBT Prev’s work support the work of NAC? 5. How is its work aligned/has it been aligned with the key GoM HIV instruments: the National HIV Prevention Strategy 2009–2013, the National HIV and AIDS Strategic Plan 2012–2016 (and its Operational Plan 2009–2011)? 6. Has the work of EBT Prev, its research and data, informed the development of national instruments (e.g. the 9 Strategic Themes in the NHSP 2012–2016) and the national response? 7. EBT Prev/PSI links to the national M&E Framework and recent work towards evaluation effectiveness? 8. Gender focus of the project and links to NAC position and activities. 9. Future sustainability of EBT Prev activities, within the national response? 10. Any further discussion points as arise. TOOL I: QUANTITATIVE SURVEYS 1. EBT Prev project partners Sample size: minimum of eight representatives of NGO partner Introduction: we are doing an assessment of the EBT Prev project. We would appreciate if you can help us with the assessment. We will read you several statements. We would like you to tell us to how much you agree with the statements on a scale of one to five. Five means that you agree very much, and one means that you don’t agree at all. Sustainability 1. We have increased our capacity to sustain EBT Prev Project work beyond the end of current funding. 1 2 3 4 5 2. We have developed the technical skills and personnel to continue well into the future. 1 2 3 4 5 3. We are confident that we can sustain EBT Prev Project in the long term through collaboration with existing and new partners. 1 2 3 4 5 4. The EBT Prev Project has allowed us to increase our skills for working in HIV/AIDS in the future. 1 2 3 4 5 5. The emphasis of the EBT Prev Project on reaching out to high-risk groups has resulted in decreasing new HIV infections. 1 2 3 4 5 USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 121 6. The emphasis on encouraging use of services like condoms, HCT, Prevention of Mother to Child Transmission, ART, has resulted in a substantial increase in the use of these services by at-risk populations. 1 2 3 4 5 2. EBT Prev Project IPC Volunteers, IPC Assistants, Peer Educators Sample size: EBT Prev Project IPC Volunteers, IPC Assistants and Peer Educators Introduction: We are doing an assessment of the EBT Prev Project. We would appreciate if you can help us with the assessment. PART A: Please tell us which of the following prevention activities you are involved with: 1. Male condom promotion 2. Female condom promotion 3. HIV Counseling and Testing 4. Use of Reproductive Health Services 5. PMTCT/ART 6. Partner reduction 7. Gender-based violence 8. .Alcohol abuse reduction PART B: Please tell us which of the target populations you have worked with: 1. Female Sex Workers 2. Fishing communities 3. Plantation workers 4. Vendors 5. Men who have sex with men 6. Others (Please list:___________________________________) PART C: Please tell us how much you agree with the following statements on a scale of one to five. Five means that you agree very much, and one means that you don’t agree at all. 1) As a result of the EBT Prev project my ability to conduct Interpersonal Communication and peer education with target populations has increased. 1 2 3 4 5 2) As a result of participation in the EBT Prev project I will be able to continue the work I am doing after the project ends. 1 2 3 4 5 122 USAID/MALAWI: EBT Prev Mid-Term Evaluation 3) I found the support materials like guides, flip charts, flannelographs to be easy to understand and use. 1 2 3 4 5 4) Support materials were sufficient in number for me to conduct my work. 1 2 3 4 5 5) My EBT Prev project training has increased my skills and ability. 1 2 3 4 5 6) In the future, I would like continue in my Interpersonal Communication or Peer Education work. 1 2 3 4 5 7) Interpersonal Communication or Peer Education work is relatively easy to do and few problems have been encountered. 1 2 3 4 5 USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 123 Sample Consent Form EBT PREV Project mid-term evaluation October 2012 INFORMED CONSENT FORM for................. (group of respondents, e.g. vendors) NAME OF INTERVIEWER Purpose of discussion for which consent is being sought I am undertaking an evaluation of the EBT PREV project, which is working in this area with a number of groups of people and health workers on activities to prevent HIV transmission, to practice safe sex and to support people to be tested. Please feel free to ask any questions; if you would like someone to explain the information sheet to you, please ask. Guarantee of confidentiality Please note that all your responses and comments are entirely confidential and anonymous. Your name and any other identifying information will not be used. You are also free to stop the discussion at any time, or to ask additional questions. Reimbursement There will be no reimbursement or remuneration for participation in this discussion. Type of discussion I would like to invite you to participate in a discussion on aspects of the project. This is a focus group discussion/a key informant interview/a checklist interview. 124 USAID/MALAWI: EBT Prev Mid-Term Evaluation Photography Subject Consent and Release Form By signing this form, I hereby grant to The Global Health Technical Assistance Project the right to create a photograph of me, and to display that photograph in a report on the subject of ___________________________. I understand that my photograph may be used in a document that may be viewed worldwide on the Internet. I understand that I have the right to refuse being photographed. I confirm that these photographs were taken with my knowledge and consent. Photo Subject 1 (Name of Person in Photo) Child Under 18 (Signature) Parent/Guardian (for child under 18) (Date) Photo Subject 2 etc. (Name of Person in Photo) Child Under 18 (Signature) Parent/Guardian (for child under 18) (Date) THIS BOX FOR PHOTOGRAPHER'S USE Photographer Name: Project: File Name/Frame #: Folder Name/Film Roll ID: USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 125 Caption/Notes: Photography Subject Verbal Consent and Release Record Photographer’s Name: Assignment: I (photographer) confirm that I informed the subject(s) below that their image(s) will be used in a report that could be publicly viewable on the Internet. I informed them of the topic of the report, the way their photograph will be used and described, and their right to refuse. I confirm that the subject(s) provided their consent to be photographed and to have their photograph used in the manner described. Instructions: Please record the following information for every subject: Name or other identifier of person in photo File name/Frame # Date 126 USAID/MALAWI: EBT Prev Mid-Term Evaluation USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 127 APPENDIX E: EBT PREV EVALUATION—FIELDWORK APPROACH AND METHODS (INCLUDING SAMPLE DETAILS) THE EVALUATION SOW, OBJECTIVES AND QUESTIONS As set out in the Scope of Work, the primary purpose of the mid-term evaluation of EBT Prev was “to determine the extent to which the project’s behavior change and communications’ interventions have resulted in reduction of risky behavior among the Most￾At-Risk-Populations.” A secondary objective was to understand the extent to which the program approach and multiple components have been effective. The scope of the evaluation was to address the thirteen evaluation questions, grouped under six main questions and by so doing evaluate the extent and efficacy of the project objectives, through a performance evaluation approach. The mid-term evaluation was intended to: • Determine if the objectives as defined in the cooperative agreement and in relation to planned activities are being achieved, and assess the likelihood of achieving them upon project completion taking into account the perspectives of the stakeholders and beneficiaries; • Determine the strengths and weaknesses of the existing program and approach explicitly determining why certain program components are working or not working; and • Provide concrete recommendations on any program adjustments to be made for the remainder of the project agreement. The Evaluation Questions 1. How effective have PSI’s EBT Prev Communications approach and Targeted Outreach Communication (TOC) been in supporting adoption of safer sexual behaviors among project target groups? 1a. Are communications activities adequately tailored to the different categories of populations amongst the target groups—including commercial sex workers and MSM? (i.e. responsive to specific barriers faced and sufficiently skills-oriented to support behavior change?) 1b. To what extent has the frequency in which all target populations have been reached with messages supported the adoption of safer behaviors, including increased demand for and utilization of appropriate HIV and reproductive health services and commodities? 1c. To what extent has the project supported synergies between different communication interventions and [been] responsive to service priorities likely faced by populations, such as integrated Prevention of Mother-to-Child Transmission and antiretroviral treatment (PMTCT/ ART) and related GBV services? 2. What effect has the addition of new program areas, specifically [a] Voluntary Medical Male Circumcision and [b] Gender-based Violence, had on PSI’s operational management and capacity to implement quality assurance? 3. To what extent is the current project monitoring and evaluation framework measuring indicators that are appropriate and sufficient to demonstrate the value of the Social Behavior Change Communications’ approach? 4. To what extent have the CBOs and IPC volunteers successfully assimilated the capacities introduced through the capacity-building efforts? 128 USAID/MALAWI: EBT Prev Mid-Term Evaluation 4a. What elements of these new capacities will CBOs and IPC volunteers sustainably continue to implement without the support of EBT Prev? 5. How effective have EBT Prev activities been in strengthening the network of community￾based HIV and reproductive health providers? 5a. To what extent has EBT Prev’s support to the provider network improved referral systems for the priority target groups? 6. How effectively has PSI maintained a balance between social marketing of condoms and free distribution of male and female condoms for the priority target groups? 6a. How have the dynamics of this balance affected access to condoms for the priority target groups? No additional evaluation questions were added during assignment negotiations. However, in the course of in-country evaluation work, USAID requested that particular attention be given to CEDEP management, both by the project and its own internal processes. One key issue to be addressed was whether CEDEP internal processes and procedures are adequate to support MSM to undertake peer education. See 3.4 in the main body of the report for further consideration. The Evaluation Team Team Leader: Janet Gruber SBCC and Social Marketing Consultant: Iain McLellan Local/HIV Consultant: John Kadzandira Translator: Zione Themba Logistics’ Coordinator: Tennyson Banda Evaluation approach The overall objective of this mid-term performance evaluation was to provide an overview of the EBT Prev project to date, using primarily qualitative methodology, with some limited quantitative collection of data through two surveys, administered to project partner outreach workers and project partner officers. Analysis of project data and reports was undertaken systematically; the evaluation team made full use of the PSI/M database, its ongoing surveys and its continuous M&E activities, in order to validate and triangulate its own findings and reviewed other relevant national and international documentation, e.g. on VMMC and PMTCT B+. The 2011 USAID Evaluation Policy defines performance evaluations as focusing on “descriptive and normative questions: what a particular project or program has achieved (either at an intermediate point in execution or at the conclusions of an implementation period); how it is being implemented; how it is perceived and valued; whether expected results are occurring and other questions that are pertinent to program design, management and operational decision-making. Performance evaluations often incorporate before-after comparisons, but generally lack a rigorously defined counterfactual [as would be the case for an impact evaluation].” The Scope of Work notes that the mid-term evaluation “will focus on process and implementation fidelity as it relates to achieving project goals, objectives and activities as they were envisioned, in addition to looking at some intermediate outcomes.” The evaluation team applied the following conceptual methods and approaches throughout its fieldwork, analysis and report writing. An optimal balance of qualitative and quantitative methods: the evaluation team was mindful of the fact that there were two weeks for evaluation activities, including visits to six PPAs and less than one week for in-country analysis. Focus was on achieving the best data collection and analysis in the limited time available. The evaluation questions required a mix of USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 129 quantitative, numerical data analysis and more in-depth qualitative analysis descriptive and normative questions, as well as participant/direct observation and review of project and other documentation. Triangulation: this evaluation triangulated findings and analysis throughout, so as to minimize bias and provide a robust evidence base. Triangulation was addressed through document review (including project data and PSI data sets), qualitative and/or quantitative data collection methods and participant observation. Participatory approaches: the evaluation team applied these wherever possible. All data collection tools sought the optimal degree of independent input from respondents, including members of MARPs, interviewed during fieldwork. Gender-sensitive data collection and analysis: the evaluation team recognized the absolute centrality of applying a gendered perspective throughout its work. All evaluation activities were sensitive to gender issues, barriers and challenges that may apply to women and men in the various target populations and has considered the ways in which EBT Prev may have addressed these through project activities. The evaluation team sought to review gender and equity issues relevant to all project activities. Therefore, FGD and KII guides included questions on issues of gender and discussed project gender inputs; wherever possible, project documents reviewed were examined using gender analytic approaches. Analysis of findings included gender-focused review, e.g. of the potentially different approaches to health-seeking behavior of women and men and responses to project IPC messaging. However, it should be borne in mind that such focus was not requested as part of the SOW; it represented an additional input by the evaluation team. Evaluation Methodology 1. Data collection methods The evaluation team finalized focus group discussion (FGD) guides and key informant interview (KII) guides, as well as two short quantitative surveys for use in undertaking fieldwork data collection. Tools were pre-tested in Lilongwe Old Town PPA on October 20, 2012. For the full set of evaluation tools, see Appendix D. Meetings were conducted with: USAID, NAC, PSI, Pact, JHU BRIDGE II, UNAIDS, UNFPA. KII guides were used. It was not possible to have meetings with Ministry of Health representatives. The following discussions were conducted with individuals and groups across the six fieldwork PPAs. For further disaggregation and details, see Appendices B, C and D. • Eight Key Informant Interviews (KII) with the following project partners: MHRYN, CEDEP, COPRED, CCC, NACC, MACRO, NASO, SWAM, TFaC • One FGD with MACRO representatives • Eight project partner quantitative questionnaires were administered • A total of fifteen FGDs were held with IPC/Vs (seven FGDs); IPC/As (six FGDs); MSM Peer Educators (two FGDs) • A total of 63 project outreach worker quantitative questionnaires were administered (33 with IPC/Vs, 15 with IPC/As and 15 with MSM Peer Educators) • Eleven KII were held with Hub health workers • Fifteen FGDs were conducted with MARP representatives: four with CSWs; four with Plantation workers (two each with men and women); three with Vendors (one each with men and women and one with a mixed group); four FGDs with fishermen and women living in fishing communities (two each with men and women) • Meetings with seven organizations in Lilongwe and Blantyre: USAID, PSI/M, Pact, NAC, BRIDGE II, UNAIDS, UNFPA. 130 USAID/MALAWI: EBT Prev Mid-Term Evaluation Table 1: Qualitative and quantitative data collection during evaluation fieldwork PPA Qualitative and quantitative fieldwork Lilongwe Old Town Qualitative MSM: FGD with 7 Peer Educators CSW: FGD with 5 respondents Male Vendors: FGD with 11 respondents IPC/V: 2 mini FGDs; 1 with 4 respondents, 1 with 3 MHRYN: 2 KII Hub Health Worker: 1 KII CEDEP: 1 KII Quantitative questionnaires 1. Project partner CEDEP + MHRYN 2. Outreach workers: 7 IPC/Vs Plus FGD with 6 MACRO staff members (6 men) Drama on GBV Mwanza (Town and border) Qualitative CSW: FGD with 5 respondents Vendors: mixed FGD with 5 respondents (4 women) IPC/Vs: 4 KII VSU: FGD with 4 respondents (3 men) Hub Health Workers: 3 KII COPRED: 3 KII Quantitative questionnaires 1. Project partner: COPRED 2. Outreach workers: 4 IPC/Vs Drama on GBV Condom outlet survey Thyolo Satemwa Tea Estate Qualitative Male plantation workers: FGD with 18 respondents Female plantation workers: FGD with 13 respondents IPC/V: mixed FGD with 8 respondents IPC/A: mini- FGD with 3 respondents Hub Health Worker: 1 KII CCC: 3 KII Quantitative questionnaires 1. Project partner: CCC 2. Outreach workers: 3 IPC/As; 8 IPC/Vs USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 131 Blantyre (MACRO) Qualitative HTC Counselors: mini FGD with 3 respondents Senior HTC Counselor: 1 KII Zomba Songani Market Qualitative Vendors: mixed FGD with 12 respondents (8 men) CSW: FGD with 5 respondents IPC/V: FGD with 7 respondents (4 male) IPC/A: FGD with 5 respondents (4 female) Hub Health Worker: 1 KII NACC: 4 KII Quantitative questionnaires 1. Project partner: NACC 2. Outreach workers: 5 IPC/As; 7 IPC/Vs Drama on GBV Condom outlet survey Maldeco Qualitative Fishermen: FGD with 8 respondents Women in fishing community: FGD with 9 respondents CSW: FGD with 8 respondents IPC/A: mini FGD with 2 respondents (1+1) IPC/Vs: FGD with 7 respondents Hub Health Workers: 3 KII MACRO: 1 KII NACC: 1 KII Quantitative questionnaires 1. Project partner: NACC 2. Outreach workers: 2 IPC/As; 7 IPC/Vs Drama on GBV TOC event on GBV MACRO mobile HTC clinic Dwangwa Qualitative Male plantation workers: FGD with 19 respondents Female plantation workers: FGD with 13 respondents Fishermen: FGD with 7 respondents Women in fishing community: FGD with 5 respondents MSM: FGD with 7 respondents IPC/As: FGD with 5 respondents (3 male) Hub Health Workers: 2 KII 132 USAID/MALAWI: EBT Prev Mid-Term Evaluation NACC: mini FGD with 2 respondents (1+1) Quantitative questionnaires 1. Project partner: SWAM + NASO 2. Outreach workers: 7 with MSM; 5 with IPC/As Condom outlet survey Table 2: Total number of evaluation qualitative respondents (sex-disaggregated) MSM 15 CSW 23 Male Vendors 20 Female Vendors 8 Male Plantation Workers 37 Female Plantation Workers 26 Fishermen 15 Women in fishing communities 14 Hub Health Workers 11 (7 men) HTC Counselors 5 (4 women) VSU staff members 4 (3 men) IPC/V 33 (18 female + 15 male) IPC/A 15 (8 female + 7 male) Project partner staff members + MACRO FGD 18 (15 male) 6 (all male) Representatives of: USAID PSI/M Pact NAC UNAIDS UNFPA 3 F, 2 M 5 F, 7 M 2 F, 1M 1 F, 5 M 1 M 1 F, 1 M Total number of respondents 279 (158 men, 121 women) 2. Data analysis The evaluation data collected through the evaluation FGDs and KIIs have been analyzed and triangulated with project data; analysis and findings have served to strengthen responses to the evaluation questions as set out in the Scope of Work. USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 133 The evaluation team has applied the Convergent Mixed Methods Parallel Research Design (CPRD), which consists of four distinct steps: 1. The simultaneous design and implementation of data collection approaches; 2. Separate analysis of qualitative and quantitative data; 3. Triangulation of quantitative and qualitative results; and 4. Interpretation of the merged results. The evaluation team decided to use the CPRD because of its comparative advantage in having team members who have skills both in qualitative and quantitative methods of data collection and analysis. See Appendix C for further discussion of analytical processes applied by the evaluation team. It was not possible to meet MoH representatives during the evaluation; despite attempts, direct/participant observation of IPC/V or IPC/A sessions, whether with individuals or in small groups, could not be arranged; in addition, an FGD or KII with Queen CSWs did not take place. These were all limitations of the evaluation. 134 USAID/MALAWI: EBT Prev Mid-Term Evaluation USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 135 APPENDIX F: FURTHER DISCUSSION OF THE 13 EVALUATION QUESTIONS 1. BACKGROUND INFORMATION ON A NUMBER OF EVALUATION QUESTIONS This additional information is provided for a number of the evaluation questions, in order to give background contextualization. 1.1 Project activities to address gender-based violence—relevant to evaluation questions 1c and 2b EBT Prev submitted an additional Program Description (PD) to USAID in 2010; that document stated: “The primary purpose of this additional activity is to strengthen the quality and impact of HIV/AIDS programming in Malawi, by improving the quality of care and support provided to program beneficiaries in HIV testing and counseling settings and community dialogue sessions, through intensifying gender integration interventions.” The objectives are to strengthen gender integration at service delivery level (e.g. HTC) and to train counselors and other health workers on gender and health issues. The PD mentions the training of 500 counselors. Funds to the value of $400,000 from the Gender Challenge Fund are being used to implement project activities. PEPFAR indicator P12.2.D was added to the Project Monitoring Plan. The project has engaged since 2011 with the Malawi Police Service Victim Support Unit (VSU), which addresses issues of gender-based violence and family disputes. The evaluation team was able to visit one VSU, In Mwanza PPA. VSU involvement with EBT Prev in that location is through collaboration with COPRED, the Mwanza PPA project partner. A key activity is community sensitization, jointly undertaken by the VSU and COPRED (all VSU activities are targeted at the general population). The COPRED Project Manager in Mwanza serves as a pivotal referral person; if a member of the public comes into the VSU and requests information on HIV or referral to HCT, the VSU refers to COPRED. There is also collaboration with IPC/Vs, in the context of community sensitization and as sources of correct information of HIV, GBV, etc. for VSU police officers and complainants. The VSU has been provided with referral registers but no project materials, according to police respondents. One perceived gap is the lack of feedback after referrals to health services are made; another is the lack of materials. The project Quarter 4 message is on gender-based violence. Project partners were oriented on GBV messages towards the end of Q3; the Q4 message is still being communicated at the time of writing (early November 2012), according to IPC/Vs and IPC/As. Specific to 1c regarding PMTCT and ART: It appears that EBT Prev activities on PMTCT and its integration with ART (especially lifelong ART to the mother, the “B+” option) have not really got underway at the time of this mid-term evaluation (October–November 2012). Since the beginning of 2012, the project has placed an increased focus on PMTCT and Family Planning (FP) through alignment of PSI-branded FP commodities with relevant messages. 1.2 Project activities related to VMMC—relevant to evaluation question 2a The project VMMC component has not yet become operational: its original start date was to be 2012; this is now likely to be February 2013. Training of health workers is scheduled to begin in November 2012; this will include the participatory development of quality assurance Standard Operating Procedures and a comprehensive implementation plan. PSI/Malawi will be supported in the training by its head office staff and PSI/Zimbabwe, where 136 USAID/MALAWI: EBT Prev Mid-Term Evaluation VMMC has already been implemented. Completed activities include identification and recruitment for the twelve health worker and support staff positions for each team that will conduct VMMC funded under EBT Prev. There are five such teams. These staff members will be dedicated to VMMC, i.e. there will be no task shifting; the intention has been to recruit primarily from retired health workers. Current project work focuses the procurement of supplies and equipment. PSI will manage the Blantyre VMMC activities, to be conducted at a static clinic. BLM will commence VMMC in Thyolo, with the intention of eventual management by PSI. A number of the teams will be mobile, visiting other health facilities. The intention is that having dedicated VMMC teams will allow for greater flexibility and responsiveness. The target population will be men aged 15–49; the project considers it possible that younger men in particular might be positive about circumcision. The key benefit to be promoted will be reduced risk of HIV transmission (WHO indicates that MMC reduces the risk of female￾to-male sexual transmission of HIV by approximately 60%, based on Kenya, Uganda and South Africa research findings); the project strategy will be to counsel continued male condom use. The Malawi VMMC Communication Strategy 2012–2016 has been reviewed in connection with the evaluation, linked documents developed by BRIDGE II and currently being pre￾tested (e.g. leaflets for young men and couples and the community mobilization kit), have also been very briefly reviewed. 1.3 Project focus on M&E—relevant to evaluation question 3 All project indicators, targets and definitions were agreed between PSI/M, Pact and USAID. This is relevant in terms of all discussion on indicators and M&E. The most recent version of the Project Monitoring Plan, dated August 2012, shows that all five purpose/outcome level indicators are to be measured twice: at baseline in 2009 and during an endline study in 2013/2014. The three indicators that address MARP safe sex practices have all had baselines defined, all are to be measured in 2013/2014, at the end of the project and all are to achieve 10% increase against the baseline data. These are: Reported condom use at last sex with non-spousal, non-cohabitating partner among target groups; Reported always using condom with non-spousal partner in the last 12 months among target group and Decrease in percentage of people who report having more than one sexual (multiple) partners in the last 12 months among identified priority groups. The fourth and fifth purpose/outcome indicators refer to general population condom use at last sex with non-spousal partner and always using a condom in the last 12 months. The project baseline study sample included male and female representatives of fishing communities, plantation workers and vendors; it did not include MSM and CSW. 1.4 Project referral systems: relevant to evaluation questions 5 and 5a Between 2009 and January 2012, the project implemented a referral system that focused on community-based organizations’ (CBO) role within each of the 18 PPAs in operation. Each PPA referral network numbered upwards of 30 partners, including CBOs, public health facilities and non-state health providers (CHAM, BLM, etc.). A comprehensive referral directory was developed by the project, as were referral tools. Members of MARPs (and other community members) could choose which health facility they attended, and for which service. Due to these challenges, the project submitted a Concept Note to USAID in early 2012. This has resulted in a significant change to the referral system, with a new “hub and spoke” model being applied from July 2012. IPC/Vs and IPC/As have become the sole referral agents and a maximum of three health facility hubs per PPA now partner the project—these can be public or non-state, e.g. BLM or CHAM facilities. CBOs continue to play a role, albeit reduced: hub clients can be referred to a CBO for psychosocial or spiritual guidance and USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 137 support, for home-based care or to an HIV Support Group. Please see Appendix J for the project diagram that sets out the new approach. 2. LIMITATIONS LINKED TO EVALUATION QUESTIONS The evaluation team sought clarification in the week of October 22, 2012 from USAID Malawi regarding evaluation questions 4, 4a and 5; clarification was provided on November 5 and again during review of the draft report, i.e. in both instances after evaluation fieldwork had been completed (see section 2 in the main body of the report). Please see section 3.3 in the main body of the report for brief discussion of all evaluation question findings, conclusions and recommendations, including 4, 4a and 5. This Appendix provides full and detailed consideration of all thirteen evaluation questions, including responses and clarifications given by USAID Malawi regarding 4, 4a and 5 and the approaches taken to by the evaluation team in answering those questions. Q4. To what extent have the CBOs and IPC/Vs successfully assimilated the capacities introduced through the capacity building efforts? The evaluation team requests for clarification were: Does this mean the CBOs that were far more engaged in the old approach to referrals, or does it refer to the new approach (instituted in January this year and implemented from June–July), where the chief role for CBOs is that they receive referrals for psychosocial and home-based care? Or does “CBO” actually mean the project implementing partners, e.g. CEDEP, CCC? Q4a. What elements of these new capacities will CBOs and IPC/Vs sustainably continue to implement without the support of EBT Prev. Evaluation team request for clarification: As above for 4. The following response was given by USAID Malawi on November 5, 2012: For evaluation questions 4 and 4a, “CBO” does not refer to the project partners, e.g. NACC. USAID would like an overview of the first/old referral system, to record its achievements, challenges and processes. In addition, USAID would be interested to receive information on how likely it is that CBOs will sustain any activities supported by the project under the first referral system. A further response was received from USAID as part of its review of the draft evaluation report, submitted on November 5, 2012: The latter—not just referrals. Under Pact, local partner organizations and IPC/Vs are the main vehicle for delivery of the program. We would assess the referral system through CBO networks versus streamlined version under a separate question. Q5. How effective have EBT Prev activities been in strengthening the network of community-based HIV and reproductive health providers? Evaluation team request for clarification: Does this network refer to the old approach, and to those CBOs and other health facilities that were involved in the referral system? Or does it mean the hubs—as community-based health facilities? In other words, is this question and also Q4 primarily addressing a historical perspective, or the current situation? The following response was provided by USAID Malawi: Both. Under the previous referral system, networks were assessed and strengthened to deliver a range of services. It is unclear if or how reproductive health providers were involved (including Health Surveillance Assistants as MoH frontline health workers, and/or Community-based Distribution Agents that were trained). Under the new referral system, the role of HSAs is still not that clear. It is also not clear if, or how, PSI is linking its other FP activities planned to be implemented within PPAs. Please note also that discussion in section 3.3 regarding evaluation question 2 specific to VMMC (What effect has the addition of new program areas, specifically Voluntary Medical Male Circumcision and Gender-based Violence, had on PSI’s operational management and capacity to implement quality assurance?) reflects the fact that PSI clinic-based interventions VMMC have 138 USAID/MALAWI: EBT Prev Mid-Term Evaluation not yet become operational, while planning and recruitment were underway at the time of the evaluation, with training to begin in November 2012. USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 139 APPENDIX G: OVERVIEW OF PROJECT ACTIVITIES (OVERALL AND BY YEAR) 1. AN OVERVIEW OF THE EBT PREV PROJECT Please see Appendix H for a Gantt chart prepared by PSI/M at the request of the evaluation team, which sets outs a detailed timeline of all project activities. The start date for the Evidence-based Targeted Prevention (EBT Prev) project was March 1, 2009, and the end date is February 28, 2014. The theory of change informing the project is that project activities will result in reduction of reported high risk behaviors, including multiple concurrent partnerships, promote increased uptake of condoms, and facilitate responsive HIV-related services among MARPs. Thus EBT Prev seeks to promote normative change and to increase preventative and safer sex behaviors among MARPs. EBT Prev implements targeted, combination prevention activities in 18 (soon to be 20) Priority Prevention Areas (PPAs) throughout Malawi, working with male and female members of fishing communities, male and female market vendors, CSW, MSM and male and female plantation workers. The pilot PPA was Dwangwa, where community sensitization activities began in October 2009; since then a further 17 PPAs have become operational, with two more to be added as from October 2012. The project purpose is to: Increase adoption of safer sexual behaviors among target populations and venues with high prevalence of risky sexual behavior. Project targets were developed through the 2009 quantitative baseline survey and are to be measured in the end line study in 2013/2014. A significant number of core SBCC indicators are only to be measured twice during the lifetime of the project: at baseline and at endline. To date EBT Prev has been implemented through activities under four objectives: Objective 1: Identify, segment and profile priority populations at risk. Objective 2: Deliver integrated, behavior change communication programs targeted to high￾risk populations in priority prevention areas. Objective 3: Distribute and promote condoms for use by the general population and for high-risk groups. Objective 4: Enhance the network of existing providers for greater accessibility and services to high-risk populations. A new objective 5 has been added: VMMC service delivery in Thyolo and Blantyre districts and associated demand creation in Blantyre district scheduled to start in 2012. Pact Project Inputs A core activity for Pact focus as a partner in EBT Prev is to provide management of the 10 project partners, including financial oversight. Each project partner sends an annual, costed Scope of Work to Pact, which then monitors activity expenditure. Pact’s comparative advantage is in community development, grants’ management and support to project partners to implement EBT Prev activities. Pact works on objectives 2, 3 and 4, in support of training and in the rollout of the former and new referral systems. Another core Pact input is Data Quality Assessments, which have to date been conducted in 14 PPAs. Pact undertook the Organizational Network Analysis (ONA) exercise in the pilot phase of the project. 140 USAID/MALAWI: EBT Prev Mid-Term Evaluation Pact has recently completed an Organizational Performance Index (OPI) with six of the project partners; this activity focuses on the level of resource mobilization and potential sustainability of project partner activities after the end of EBT Prev. Capacity-building of project partners is contextualized to project-specific activities. Pact is also a partner in BRIDGE II, supporting adult prevention interventions in 11 Southern Region districts. The role of project partners See also 3.4 in the main body of the report, for discussion of CEDEP issues. PSI/M has worked in partnership with Pact from the outset of EBT Prev to deliver PPA-level interventions. Both PSI and Pact work in different areas with 10 current sub-recipients (in this report defined as project partners): Malawi AIDS Counseling and Relief Organization (MACRO), Centre for the Development of People (CEDEP), Christian Community Church (CCC), Community Partnership for Relief and Development (COPRED), Nkhotakota AIDS Society Organization (NASO), Namwera AIDS Coordinating Committee (NACC), the Society for Women with AIDS, Malawi (SWAM), Malawi Human Rights and Youth Network (MHRYN), Foundation for Community Services (FOCUS) and Towvirane. Eight of the project partners work in one or more of the 18 currently active PPAs, with various MARP groups. The exceptions are: CEDEP, which works only with MSM in five PPAs; and MACRO, whose mobile HTC clinics visit all PPAs on a three-month schedule and which provides testing to all who request the service. The project contract with Theatre for a Change (TFaC) has ended; its inputs were provided to drama groups who perform for the general population in almost all of the 18 PPAs, currently on issues of GBV. Each of the eight project partners active with MARPs other than MSM works with Interpersonal Communication Volunteers (IPC/V), who on average provide two days a week support on project activities, and IPC Assistants (IPC/As), who work full-time and are salaried. Project Managers and Officers are salaried project partner staff members. The Peer Educator approach is used to provide IPC to MSM and CSW (thus through CEDEP for MSM and for CSWs through e.g. NACC and MHRYN, working with “Queen” CSWs). The project works with MSM in five PPAs (Lilongwe Old Town, Dwangwa, Cape Maclear, Maldeco/Makawa and Mchinji); there is engagement with CSWs in 16 of the 18 PPAs, the exceptions being Dwangwa and Mulanje. The PPAs were selected for work with MSM based on assessments and previous project work with the target population. The evaluation team as no information on why two PPAs had no interventions with CSW. PSI has throughout provided training to IPC/Vs, IPC/As and Peer Educators. All IPC/Vs and IPC/As encountered during evaluation fieldwork (48 in total) described their initial one week’s training as adequate, informative and positive in terms of being able to deliver IPC. However, it should be noted that 45% of all IPC/Vs and IPC/As did not recall receiving gender training as part of that initial one week, despite this component being addressed.6 A number of project partners have been able to build on prior validation and capacity development provided under EBT Prev (and by REACH, a former HIV project on which Pact worked, which had a far larger capacity development component). The only current or former EBT Prev partners not to have participated in REACH are MHRYN, TFaC and CEDEP. However, it is relevant to point out that Pact considers MHRYN to be particularly strong in its financial management capacities, strengthened at least in part by EBT Prev Pact inputs and support. EBT Prev Budget Allocations The two pie charts below set out the original project budget (to a total of $20,400,000) and the proposed budget (to a total of $24,528,070), which includes the new VMMC component 6 Please see Gender section of the main document for discussion of EBT Prev gender issues. USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 141 (costed at $4,128,026, ca 16.6% of the total project budget from inception) and activities addressing gender-based violence (to a total of $400,000). The proposed budget was submitted to USAID for approval in September 2012. The charts have been provided to the evaluation team by PSI/M. Chart 1 provides insight into the relative proportions of project resources spent on each objective. Significantly, the largest allocations were made to the BCC interventions and outreach (objective 2) followed by greater accessibility and services for high risk populations (objective 4). Condom distribution and promotion also was allocated a significant portion of resources. Finally, the initial identification, segmentation and establishment of profiles of priority populations received the smallest portion of project resources. Chart 1: Total Budget for EBT Prev as approved in April 2011 Chart 2: Proposed budget for EBT Prev, including VMMC costs, submitted for approval September 2012 In Chart 2, the relative proportions allocated to the different objective elements have remained the same. The addition of the VMMC intervention has significantly changed the profile of the project in terms of allocated funds. 142 USAID/MALAWI: EBT Prev Mid-Term Evaluation 2. OVERVIEW OF PROJECT ACTIVITIES BY YEAR 2.1 Project Year 1 activities (March 2009–September 2009) Discussion of all activities from inception to date is based on project annual work and implementation plans and quarterly and annual project reports. Financial years (FY) in terms of the project run from October to September; this is why project years end in September of each year. The pilot phase of the project was between September 2009 and March 2010; this was when sensitization activities were done in Dwangwa with community leaders, key health workers (e.g. the District Health Officer), committees such as the District Executive Committee and community members were undertaken. During the pilot phase baseline research was conducted in the following five PPAs: Lilongwe Old Town, Zomba, Thyolo, Mwanza and Maldeco. One early focus of the project was to strengthen community-based organizations (CBOs), so that these might effectively participate in community HIV prevention activities. Health service providers were supported so as to develop, implement and sustain effective client registration and referral mechanisms, using the original version of the “hub and spoke” model, as envisaged in the 2008 PSI and Pact EBT Prev application in response to the USAID RFA. PLACE, condom mapping and service delivery research were conducted; findings have informed EBT Prev activities. The PLACE methodology systematically identified concentrations of the project MARPS and ensured that the PPAs selected corresponded with those concentrations. This allowed for a highly concentrated and focused use of project resources. The condom mapping also allowed for the creation of a clear picture of existing condom sales and distribution patterns and the systematic increase of distribution outlets to match the PPA intervention zones. 2.2 Project Year 2 activities (October 2009–September 2010) Actual expenses, $14,276,194 , 58% Objective 1, $804,279 , 3% Objective 2, $2,066,107 , 8% Objective 3, $1,591,371 , 7% Objective 4, $1,662,093 , 7% VMMC, $4,128,026 , 17% Actual expenses Objective 1 Objective 2 Objective 3 Objective 4 VMMC USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 143 As from 2010 EBT Prev has deepened its attention to action against gender-based violence and its role in transmission of HIV. The EBT Prev manual has been updated, health providers have been trained and communication activities implemented by IPC/Vs, through Targeted Outreach Communication (TOC) and through development of project links to the Malawi Police Service Victim Support Unit. The National Strategic Plan is one of several national documents and reports that highlight gender as a weak component of the national response. The evaluation team did not gain insights into the role of health providers in GBV in year 2 since that element of the project has significantly changed. TRaC research was completed; its objective was to inform development of targeted messages to be presented through IPC to MARPs. In year 2, 230 individuals (aggregate data only) were trained as IPC/Vs and IPC/As; in total the 230 reached over 20,000 MARPs (P8.3.D) in Dwangwa, Mwanza, Mulanje, Thyolo and Zomba PPAs. Chishango condom distribution (7.3 M) achieved 92% of sales target, while CARE female condoms achieved a 71.2% distribution target (79,214). Referral network activities were instituted in Dwangwa PPA in July 2010 and in Mulanje, Thyolo and Mwanza PPAs in October 2010. The FY 2 annual report describes the role of the Ministry of Health HIV Unit in the development of the project referral system and its help in reviewing referral tools. MACRO mobile HTC activities were delayed to an extent, due it appears to management issues and fuel shortages. In year 2 project activities were scaled up to eight PPAs. See Appendix H for the Gantt chart that sets out the start dates for all PPAs and other details of project activities since inception in March 2009. A project review of pilot phase lessons learned, challenges and achievements was conducted in March 2010; its conclusions informed future project development, e.g. on Pact’s working arrangements with project partners and refinement of IPC/V training approaches. 2.3 Project Year 3 activities (October 2010–September 2011) In project year 3 (FY 3) EBT Prev activities were scaled up from the 13 existing PPAs to an additional four PPAs. Project research findings (from studies in 2009) were presented to stakeholders in August 2011: findings of the PLACE study, the Condom Outlet and Health Service Delivery Point mapping and the Venue-based, Targeted Quantitative Survey. In November 2010, EBT Prev conducted its Communications Strategy Development. The key issues to be presented through quarterly messages (as then envisaged; these have since changed, e.g. the Q4 message has been GBV). The over-arching message is: Lingalira sankha wekha (Think about it; it’s your choice); the Q1 message was to introduce Lingalira sankha wekha; for Q2 Condom use and/or partner reduction for a better life; for Q3 What is the best choice for me; and for Q4 How do I live my choice? One other activity in quarter 4 of FY 2010/2011 was discussion with MSM Peer Educators about the relevance of project messaging. It appears that the plan was to develop MSM￾specific messages for Qs 3 and 4; this seems not to have happened. The development of the training curriculum for service providers on GBV was delayed. A large number of IPC/V refresher trainings were undertaken, as was training of health workers in e.g. Nkhatabay PPA. Supervisory visits were conducted in connection with referral. The project report for Quarter 4 2011 (July–September) sets out the problems emerging and recurring under the then referral system. 2.4 Project Year 4 activities (October 2011–September 2012) 144 USAID/MALAWI: EBT Prev Mid-Term Evaluation The evaluation team has reviewed project activities to June 2012; the Quarter 4 report (July–September) had not been signed off by USAID at the time of the assignment. A major activity across all three quarters was the reshaping of the referral system (see section 3, evaluation question 5a for detailed discussion). A Concept Note was submitted to USAID in early 2012 and subsequently agreed. The new referral system became operational in June/July; work towards that goal resulted in a degree of delay on other project activities, such as the rollout of Quarter 4 messaging on gender-based violence, with IPC/Vs and IPC/As being oriented only as from April. See section 3.3, evaluation question 2 for consideration of issues of PSI operational capacity. The project streamlined its quarterly messaging across all PPAs in the first three quarters of year 4; previously these had been somewhat staggered according to start date of individual PPAs. By end March 2012 IPC/Vs and IPC/As had been oriented in Quarter 3 messaging (“making choices”) in all active PPAs. In January 2012 a message development workshop was conducted; core messaging for quarters 4–7 was discussed. The intention is that the Quarter 5 message will be on PMTCT, Quarter 6 on Family Planning and Quarter 7 on HTC. The workshop report has not been seen by the evaluation team; therefore, it is not possible to gauge the extent to which the messaging shift towards more service delivery focused activities will be reflected in the communication strategy, IPC/V and IPC/A training, hub health worker training, etc. The project has suffered from the absence of a coherent and systematically developed communication strategic plan. As a result messages and materials have tended to be developed piecemeal. PSI/M has found that its SBCC staff found it difficult to both conduct strategic planning and lead the outreach interventions. Inadequate project resources were available for communication staffs to handle both tasks. The most glaring inadequacy was the absence of sufficient support materials to be used by outreach workers covering the theme topics. Those that existed tended to be simple, lacked detail and offered little or no support in terms of making interventions interactive and participatory. USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 145 APPENDIX H: EBT PREV GANTT CHART (AS PROVIDED BY PSI/M AND PACT) FY 2009 (Mar 09 to Sept 09) FY 2010 (Oct 09 to Sept 10) FY 2011 (Oct 10 to Sept 11) FY 2012 (Oct 11 to Sept 12) Project Implementation Signing of Cooperative Agreement No: 674-A-00-09- 00031-00 Prevention for Populations and Settings with High Risk Behaviors on March 13, 2009. n/a n/a n/a Objective 1: Identify, segment and profile priority populations at risk. n/a Project signed in March 2009. Quarter 1: Oct-Dec 2009 • Data Collection for PLACE Step 3 (venue verification) completed in 5 PPAs in Nov 2009. • Submitted study design for targeted, quantitative research study National Health Science Research Committee (NHSRC) for Ethical approval. • Revised Project Monitoring Plan and project log frame. Quarter 1: Oct–Dec 2010 • Data analysis by the research team. Quarter 1: Oct–Dec 2011 • Developed a quality assurance plan. • Developed an HTC tool to assist data collection for target populations for MACRO. Quarter 2: Mar 2009 only • Conducted meeting with MoH and other stakeholders in order to identify research priority prevention areas (PPAs) for Priorities for Local AIDS Control Efforts (PLACE) research. • Commenced recruitment process for a new research advisor and officers at PSI/Malawi. Quarter 2: Jan–Mar 2010 • Undertook PDA form development training in April 2010. • Conducted mapping process for HIV services delivery in Dwangwa. Quarter 2: Jan–Mar 2011 • Pretesting of IEC materials for the Lingalira brand. • M&E team provided training among implementing partners. Quarter 2: Jan–Mar 2012 • Implemented a qualitative study assessing need for a male condom targeting couples. • Submitted two abstracts 2012 International Aids Conference 146 USAID/MALAWI: EBT Prev Mid-Term Evaluation • Prepared draft strategy document and data collection forms delineating methodology for PLACE research. Quarter 3: Apr–June 2009 • Piloted data collection forms and converted into Personal Digital Assistance (PDA) format. • Notified DHOs, DCs and other stakeholders for; Mwanza, Zomba, Mulanje and Thyolo and Mangochi about implementation of PLACE study. • Recruited and trained 20 Research Assistants in PLACE data collection for step 2 and use of PDAs. • Implemented and completed data collection in the five pilot PPAs. Quarter 3: Apr–June 2010 • Developed data collection tools, pretesting, PDA form development and data collection for targeted, quantitative research. Quarter 3: Apr–June 2011 Undertook population size estimations among target populations. Pre-tested three quarterly messages for “Lingalira Sankha Wekha” campaign. Quarter 3: Apr–June 2012 • Developed a study design for general population and submitted for ethical review. • Pretesting of IEC materials for GBV messages. USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 147 Quarter 4: July–Sept 2009 • Completed PLACE Step 2 data analysis in preparation for PLACE Step 3 (venue verification) by end August 2009. • Undertook GIS data analysis training in August 2009. • Confirmed the target groups with the donor, government and other stakeholders • Undertook preparatory activities for step 3 data collection. Quarter 4: July–Sept 2010 • Completed data collection in all the PPAs by end August 2010. • The research team implemented an M&E back-up system for PACT/Malawi for capturing data for the project. • Developed a plan to conduct monthly back-up for data collected by PACT/Malawi. Quarter 4: July–Sept 2011 • Dissemination of EBT Prev research results in August 2011. • Undertook Condom Audit in the pilot PPAs. Quarter 4: July–Sept 2012 • Report writing in progress and not submitted/ approved by the donor. Objective 2: Deliver integrated, behavior change communication program targeted to high￾risk populations in priority prevention areas n/a Project signed in March 2009 Quarter 1: Oct–Dec 2010 • TOC continue implementation of activities in the hot zones. Quarter 1: Oct–Dec 2010 • IPC training for Q1 messages in the Pilot PPAs • Undertook a Communications Strategy Development process for the program. • Development IEC materials continued. • FSW Queens Training in the Pilot PPAs Quarter 1: Oct–Dec 2011 • The program rolled out to 14 PPAs. • Q3 Message Orientation Meetings. • Initial training in Likuni. • Dissemination of IPC Messages among MSM. • Training of Drama Groups to disseminate HIV prevention & EBT Prev Messages. • Supportive supervision visit among the IPCV/As . Quarter 2: Mar 2009 only No program activities. Quarter 2: Jan–Mar 2010 • Study visit to PSI/Mozambique undertaken to learn implementation of iBCC activities for MARPs. • Held a 4-day Message Design Quarter 2: Jan–Mar 2011 • Refresher training among implementing partners in all pilot PPAs. • MSM Message Design Workshop. Quarter 2: Jan–Mar 2012 • Program rolled out in 18 PPAs. • Conducted orientation and mentoring visits for Q2 and Q3 messages. • IPC Refresher Training in 148 USAID/MALAWI: EBT Prev Mid-Term Evaluation Workshop focusing on concurrency as a key driver of HIV infection. • IPC Workers Refresher Training was conducted in Dwangwa pilot PPA. • Distribution of IEC materials in the pilot PPAs. Mzuzu, Mangochi and Zomba PPAs. • The EBT Prev Message Development Workshop from Jan 17–20, 2012. • Undertook streamlining of TOC activities with more focus in the PPAs. Quarter 3: Apr–June 2009 • From April 20–24, four TOC teams, and representatives from seven community drama groups, participated in a refresher training for EBT Prev. • TOC started implementing messages addressing Multiple Concurrent partnership activities in the hot zones. • Drafting of IEC materials for the project. • Setting up meeting with CEDEP for proposal to work with MSM. • Getting approval from Kayerekera Coal mine for the TOC to undertake its activities in the mine. Quarter 3: Apr–June 2010 • In-house message refining meetings. • DEC meeting in the Pilot PPAs in preparation for program roll out. • Sub-contracting production of communication materials. Quarter 3: Apr–June 2011 • Orientation of EBT Prev Implementing partners in Pilot PPAs on Q2 messages • Orientation for Makanjira PPA on Q1 messages • Implemented large scale TOC shows for “Lingalira Sakha Wekha” to increase Lingalira brand awareness among the target groups. • Conducted two District Executive Committee meetings in Liwonde and Mangochi Quarter 3: Apr–June 2012 • Continued disseminating Lingalira Sankha Wekha decision-making messages in all PPAs through IPC and drama. • IPC Team prioritize streamlining of activities for Objective 4 delayed orientation of GBV messages. • IPC orientation training on GBV in all 18 PPAs. Quarter 4: July–Sept 2009 • The Video Production Unit produced two videos on MCP and GBV. • TOC teams conducted “Chishango Nights” at various “hot spots” with the aim of increase sales of the Quarter 4: July–Sept 2010 • Five IPC trainings were conducted in Mwanza, Mulanje, Thyolo, Zomba and Mangochi. • TOC teams continued to deliver a range of communications activities at Quarter 4: July–Sept 2011 • The project scaled up to 10 PPAs • Continued implementing refresher and couching session for IPCA/Vs. Quarter 4: July–Sept 2012 • Report writing in progress and not submitted/ approved by the donor. USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 149 Chishango condoms. PPA level Objective 3: Distribute and promote condoms for use by the general population and for high-risk groups Quarter 2: Mar 2009 only • Condom distribution sales activities undertaken. Quarter 1: Oct–Dec 2010 • New look Chishango pack was launched in December 2009. • PSI Malawi secured additional UNFPA funding for a CARE Male Involvement campaign, which will commence in the next quarter. • Training was conducted for BLM central region clinic providers on CARE and a total of 30 providers were trained. Quarter 1: Oct–Dec 2010 • Improved Chishango Condoms procurement process implemented. • In collaboration with UNFPA, PSI trained 150 men on the benefits of CARE female Condoms and Gender issues related to Sexual and Reproductive Health, as part of the male involvement campaign. • PSI Malawi received additional funding from UNFPA for the procurement of CARE promotional materials, production of radio adverts and development of a CARE short video. • The CARE Promoters End of Year meeting took place in December to draw up work plans for 2011. Quarter 1: Oct–Dec 2011 • PSI Malawi launched the “Hello Sweetie” promotion for Chishango “Limited Edition” in November 2011 was undertaken to prevent a potential condom stock-out. • In line with PSI Malawi’s marketing plan for Chishango that responds to consumer needs, consultations were conducted with USAID regarding the procurement of an improved, better smelling condom. • No ROC activities were implemented due to fuel shortage currently being experienced nationwide. Quarter 2: Mar 2009 only • Continued the targeted sale and distribution of Chishango male condoms in our existing 18 “hot zones.” • Initiated discussions with UNFPA about their plans to continue funding the female condom social marketing Quarter 2: Jan–Mar 2010 • Rapid Outlet Creation (ROC) activities took place in Dwangwa Mulanje and Thyolo PPAs. • CARE Male Involvement campaign was launched. Quarter 2: Jan–Mar 2011 • Rapid Outlet Creation (ROC) Activities in Makawa/Maldeco and Makanjira PPAs. • CARE Community Open Days: To increase knowledge of the CARE female condom. • CARE Motivational Talks: in five health clinics in Embangweni Community Quarter 2: Jan–Mar 2012 • Samples of apple scented and vanilla scented condoms were pretested during this quarter. The pretesting findings showed that many of the participants preferred vanilla scent as compared to apple. • Review of the Chishango Marketing Plan took place on 150 USAID/MALAWI: EBT Prev Mid-Term Evaluation program in Malawi. Hospital in Mzimba and Kasalika Clinic in Kasungu. 29th March 2012. Quarter 3: Apr–June 2009 • Continued the targeted sale and distribution of Chishango male condoms. • Plans to re-launch Chishango with Kfw funds. • Continued co-funding for female condom social marketing from UNFPA. Quarter 3: Apr–June 2010 • ROC follow up in Mulanje and Thyolo PPAs. • Chishango advertising & promotion activities launched after donor approval. Quarter 3: Apr–June 2011 • Three million Blue/Gold (B&G) male condoms were ordered as an interim measure to avoid a potential condom stock out. • Blue & Gold Condom were not met due to the limited supply of condoms to key suppliers. • An additional order for 4 million Chishango-branded foil condoms was placed and is expected to be received in country by February 2012. • ROC activities were undertaken in Lilongwe, Mchinji, Mzuzu, Nkhata-Bay, Zomba, Liwonde, Cape Maclear and Dwangwa. Quarter 3: Apr–June 2012 • An order of 8 million vanilla￾scented condoms was placed through USAID with an expected date of arrival on October 31, 2012. • Pretesting of new packaging materials was developed and will be pretested in Q4. Quarter 4: July–Sept 2009 • 4,082,556 male condoms were distributed during the period representing 84% achievement against target. • Held discussions with NASFAM and Land O’ Lakes to social market CARE. Quarter 4: July–Sept 2010 • Chishango artwork design finalized and new packaging on market. • TOC Teams conducted one-day training sessions with Commercial Sex Workers (CSW) in Lilongwe, Mwanza, Mulanje, Thyolo, Mangochi and Zomba districts on general HIV Quarter 4: July–Sept 2011 • Chishango Limited Edition packaging materials was completed. • Preparations to undertaken formative research to determine the need to launch a male condom targeting stable couples. • Consultations on the Quarter 4: July–Sept 2012 • Report writing in progress and not submitted/approved by the donor. USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 151 prevention, condom use and negotiation skills and related issues. feasibility for the procurement of an improved physical product • ROC activities were undertaken in Ndirande, Lunzu, Bangwe, Chinsapo/Likuni and Salima PPAs and in Dwangwa PPA for the second time Objective 4: Enhance the network of existing providers for greater accessibility and service to high￾risk. n/a Project signed in March 2009 Quarter 1: Oct–Dec 2010 • Elton Edward, Service Provision Coordinator took up position with PSI Mw on 1st Nov 2009. • Orientation and meetings with EBT Prev project staff at both PSI and Pact. • Meetings with the DHO and District AIDS Coordinator for Nkhotakota. • Implemented Dwangwa Pilot Activities i.e. Organizational Network Analysis (ONA) and sensitization meetings in Dwangwa. • PSI Malawi contracted the CEDEP to work with MSM under EBT Prev. Quarter 1: Oct–Dec 2010 • HTC Assessments in Zomba and Mangochi PPAs. • Supervision of Referral Networks. • HTC Mobilization in Dwangwa and Mwanza. • PPA Entry Meetings: conducted in Mzuzu, Nkhatabay and Mchinji. Quarter 1: Oct–Dec 2011 • Referral System training for Health Personnel Targeting Nurses and Clinicians in Cape Maclear and Liwonde PPAs- 31 trained. • Referral System training for Community health workers from NGOs/CBOs in Cape Maclear and Liwonde PPAs- 35 trained. • District hospital staff referral system orientations in 7 PPAs￾198 trained • HTC Assessments in Cape Maclear, Liwonde, Karonga and Likuni PPAs – 18 facilities assessed. Quarter 2: Mar 2009 only No program activities. Quarter 2: Jan–Mar 2010 • Study tour to PSI/Zim to learn about managing and sustaining sustain referral networks. • HTC quality assessments carried out in Dwangwa. • Referral tools developed in Quarter 2: Jan–Mar 2011 • HTC Assessments in Nkhatabay, Mzuzu and Mchinji PPAs. • Supervision of Referral Networks in Mwanza, Thyolo and Mulanje. Quarter 2: Jan–Mar 2012 • Streamlining planning meeting conducted in early January 2012 with the aim of planning implementation. • DACC and DHMT Orientated on Streamlining approach of 152 USAID/MALAWI: EBT Prev Mid-Term Evaluation partnership with Dwangwa Network Task Force. • PPA Entry Meetings in Liwonde and Mangochi (Cape Maclear). referrals in all PPAs in February and March 2012. • Development of an integrated training manual in March 2012 for IPCVs, to promote streamlining approach. Quarter 3: Apr–June 2009 • During the USAID post-award technical meeting, it was agreed that a pilot PPA would be established in Dwangwa to inform the roll-out of EBT Prev activities. Quarter 3: Apr–June 2010 • Meeting with Managers of Service Providers in Dwangwa • Training for HTC Counselors, Clinicians and CBO leaders. • Undertook the first Organizational Network Analysis (ONA) Meeting in Mulanje and Mwanza Quarter 3: Apr–June 2012 • Referral System Training for Mzuzu and Mchinji PPAs. • Supervision to network members in Zomba, Makanjira, Mulanje, Mwanza and Thyolo PPAs with EBT Prev local implementing partners. • HTC Assessments in Liwonde and Cape Maclear PPAs in 15 facilities. • Recruitment of three Service Provision Specialists. • HTC Provision: a total of 9,107 clients were tested for HIV in Dwangwa, Lilongwe, Mchinji, Mangochi and Mulanje PPAs. Quarter 3: Apr–June 2012 • Integrated training in all 18 PPAs. • District quarterly meetings in Mzuzu, Mchinji and Lilongwe. • Supervisory visits in 11 Health facilities aimed at checking implementation of referral services among IPCV/As. USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 153 APPENDIX I: PROJECT DATA: 2010–2012 CUMULATIVE AND FINANCIAL YEAR 2011-2012 This Appendix represents team collation of raw data provided by PSI/M and Pact; it sets out both FY 2011/2012 data and cumulative data sets 2010–2012 (where available) in order to track in more detail progress against PEPFAR and other project performance indicators. There are P8.3.D data for FY 2009/2010, but data for other indicators for that initial year of project interventions have not been made available to the evaluation team. Headings and other descriptors are as provided by PSI/M and Pact; therefore, any gaps in information are to be referred to EBT Prev. The intention of this Appendix is not to analyze data in any further detail; it is rather to set out the PSI/M and Pact data year-on-year and cumulatively, so as to provide an overview for evaluation report readers of achievements (and otherwise) against project targets. Insights on issues of what the current coverage means in terms of reaching a significant critical mass of target populations and whether or not the annual target estimates were underestimated or not can be found in the body of the evaluation report. 1. COMBINED DATA SETS (FOR SEVERAL PROJECT PEPFAR INDICATORS) FY 2010/2011 (October 2010–September 2011) Indicator Indicator Description FY 2010 Target Achievement Achievement % P8.3D Number of MARPs reached with individual and/or small group level interventions that are based on evidence and/or meet the minimum standards required 45,550 69,135 151.8% Dis￾aggregated by type of MARP and sex Vendors 27,750 41,237 148.6% Plantation Workers 7,100 14,304 201.5% Fishing communities 5,150 9,177 178.2% CSWs 4,550 3,949 86.8% MSM 1,000 468 46.8% Male 37,755 Female 31,380 P8.1D Number of the targeted population reached with individual and/or small group level preventive interventions that are based on evidence and/or meet the minimum standards required 30,000 22,869 76.2% Dis￾aggregated by type of MARP and sex Fishing communities 7,872 Plantation Workers 8,431 CSWs 6,566 Male 10,830 Female 12,039 154 USAID/MALAWI: EBT Prev Mid-Term Evaluation FY 2011/2012 (October 2011–September 2012) Indicator Indicator Description FY 2010 Target Achievement Achievement % P8.3D Number of MARPs reached with individual and/or small group level interventions that are based on evidence and/or meet the minimum standards required 70,300 90,582 128.9% Disaggregated by type of MARP and by sex Vendors 45,000 52,782 117.3% Plantation Workers 10,000 13,219 132.2% Fishing communities 7,500 15,163 202.2% CSWs 7,000 8,836 126.2% MSM 800 582 72.8% Male 47,492 Female 43,090 P8.1D Number of the targeted population reached with individual and/or small group level preventive interventions that are based on evidence and/or meet the minimum standards required 35,000 21,293 60.8% Disaggregated by type of MARP and by sex Fishing communities 11,900 9,063 76.2% Plantation Workers 12,950 9,000 69.5% CSWs 10,150 2,330 23.0% Male 11,357 Female 9,936 P8.5D Number of individuals from target audience who participated in a community wide event 78,700 139,486 177.2% P11.1D Number of individuals who received testing and counseling (T&C) services for HIV and received their test results 30,000 21,370 71.2% P12.2D Gender-Based Violence and Coercion: Number of people 10,710 28,164 263.0% P8.5D Number of individuals from target audience who participated in a community￾wide event 268,000 238,313 88.9% P11.1D Number of individuals who received testing and counseling (T&C) services for HIV and received their test results 21,000 30,005 142.9% USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 155 reached by an individual, small group, or community-level intervention or service that explicitly addresses gender￾based violence and coercion related to HIV/AIDS H2.3D Number of health service workers/providers and Volunteers trained in order to participate in user-friendly referral system and/or provider network for HIV related [services] 306 2. P8.3.D NUMBER OF MARPS REACHED WITH INDIVIDUAL AND/OR SMALL-GROUP￾LEVEL INTERVENTIONS THAT ARE BASED ON EVIDENCE (NEW PEPFAR INDICATOR P8.3.D) Project Pilot Data FY 2009/2010 (October 2009–March 2010) Indicator Code Indicator Description FY 2009 Target Achievement P8.3D Number of MARPs reached with individual and/or small group level interventions that are based on evidence and/or meet the minimum standards required No target—base line data 10,251 Disaggregated by type of MARP and sex Vendors These are pilot data from two partners (NASO & SWAM) in Dwangwa, the 1st project PPA 4,127 Plantation Workers 3,720 Fishing communities 2,404 Male 5,241 Female 5,010 Six Months’ Data After Pilot in Dwangwa (CCM & COPRED Included) FY 2009/2010 (April– September 2010) Indicator Code Indicator Description FY 2010 Target Achievement Achievement % P8.3D Number of MARPs reached with individual and/or small group level interventions that are based on evidence and/or meet the minimum standards required 1,750 10,580 604.6% Disaggregated by type of Vendors 600 4,371 728.5% Plantation Workers 450 4,490 997.8% 156 USAID/MALAWI: EBT Prev Mid-Term Evaluation MARP and sex Fishing communities 200 1,524 762.0% CSWs 200 195 97.5% MSM 300 0 0.0% Male 5,991 Female 4,589 Annual achievement—2011/2012 EBT PREV 2010–2012 FY CUMULATIVE ACHIEVEMENTS AGAINST TARGETS INDICATOR P8.3D TOTAL CLIENTS REACHED Target Achievement Vendors + fishing comm + plantation workers 103,250 145,882 CSWs 11,550 12,785 MSM 1,800 1,150 Overall 116,600 159,817 Annual achievement—cumulative P8.3D Proportion of clients reached Annual Achievement Rate Vendors + Fishing comms + Plantation workers 141.3% CSWs 110.7% MSM 63.9% Overall 137.1% P8.3D Proportion of clients reached Annual Achievement Rate Vendors 117.3% Fishing communities 202.2% Plantation workers 132.2% CSWs 126.2% MSM 72.8% Overall 128.9% USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 157 P8.3.D MARPs reached 2011 (disaggregated) PPA (All) P8.3D MSM ANALYSIS District (Multiple Items) R_Year (Multiple Items) Values Row Labels MSM reached MSM Cond dist Lube dist October 21 57 November 8 29 35 December 40 124 196 Month Male vendors Female vendors Fisher men Females in fishing community Male plantation workers Female plantation workers CSWs’ clients CSWs Jan 5,076 4,239 1,398 851 1,115 897 202 1,539 Feb 6,365 4,342 1,562 823 1,310 1,138 132 1,498 March 6,019 4,499 1,914 847 1,292 1,113 122 1,396 April 5,851 4,040 1,267 877 1,492 1,252 154 1,193 May 5,150 3,633 1,274 798 1,296 1,023 167 1,027 June 4,422 3,767 1,334 778 1,282 1,092 146 1,090 July 1,710 1,106 455 329 519 483 12 173 Aug 1,986 1,754 523 366 704 663 61 329 Sept 2,616 3,135 1,423 912 1,062 898 137 303 Oct 3,702 3,425 1,347 710 1,451 1,260 808 Nov 4,866 3,630 1,700 850 1,735 1,460 113 878 Dec 4,992 3,694 1,310 692 1,669 1,317 161 1,075 Grand Total 52,755 41,264 15,507 8,833 14,927 12,596 1,476 11,309 EBT PREV 2011/2012 FY ANNUAL ACHIEVEMENTS AGAINST ANNUAL TARGETS INDICATOR P8.3D TOTAL CLIENTS REACHED Annual Target Annual Achievement Vendors 45,000 52,782 Fishing communities 7,500 15,163 Plantation workers 10,000 13,219 CSWs 7,000 8,836 MSM 800 582 Overall 70,300 90,582 158 USAID/MALAWI: EBT Prev Mid-Term Evaluation March 60 100,000 100,000 April 65 May 70 June 61 July 53 January 46 February 65 August 52 September 41 Grand Total 582 100,153 100,288 NB: see also the PACT Excel spreadsheet that disaggregates P8.3.D by NGO partner and MARP 3. P11.1.D (NB: only for 2011/2012, not cumulative) Number of individuals who received testing and counseling services for HIV and received their test results (PEPFAR Indicator P11.1.D) P11.1.D Number of individuals who received HTC and received their test results TARGET ACHIEVEMENT Achievement % 30,000 21,370 71.2% Disaggregated Male <15 years 220 1.0% Male 15+ years 11,605 38.7% Female <15 years 285 1.0% Female 15+ years 9,260 30.7% By test result: positive 1,330 6.2% By test result: negative 20,040 93.8% USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 159 4. P8.1.D Number of the targeted population reached with individual and/or small-group-level preventive interventions that are based on evidence and/or meet the minimum standards required (New PEPFAR Indicator P8.1.D, Essential/Reported) EBT PREV 2011/2012 FY ANNUAL ACHIEVEMENTS AGAINST ANNUAL TARGETS INDICATOR P8.1D TOTAL CLIENTS REACHED Annual Target Annual achievement Fishing communities 11,900 9,063 Plantation workers 12,950 9,000 CSWs 10,150 3,230 Overall 35,000 21,293 EBT PREV 2010–2012 FY CUMULATIVE ACHIEVEMENTS AGAINST TARGETS P8.1D 2010–2012 FY Cumulative Number of clients reached (sex-disaggregated) Cumulative achievement Male 26,129 Female 18,033 Achievement 44,162 Target 65,000 2010/2011 FY P8.1D Proportion of clients reached Annual Achievement rate Fishing communities 76.2% Plantation workers 69.5% CSWs 31.8% Overall 60.8% (It should be noted that it is not possible to establish a proportion of MSM clients reached since the assessment of MSM to establish estimates of population size has not been completed.) 160 USAID/MALAWI: EBT Prev Mid-Term Evaluation TOC. 2011/2012 FY P8.1D number of clients reached against annual target 11,900 12,950 10,150 35,000 9,063 9,000 3,230 21,293 - 10,000 20,000 30,000 40,000 Fishing communities Plantation workers CSW Overall Annual achievement Annual Target 2010-2012FY P8.1D Cumulative achievement against Target 26,129 18,033 44,162 65,000 - 20,000 40,000 60,000 80,000 Male Female Achievement Target Cumulative achievement USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 161 5. P8.1.D and P8.5D (TOC event)—no FY or year given Fisher￾men Women in fishing communities Male plantation workers Female plantation workers CS Ws CSWs’ clients # of GBV male (comm unity event) # of GBV female (commun ity event) Jan 54 48 44 32 98 48 Feb 339 110 399 375 214 55 Mar 248 75 234 115 142 112 Apr 271 259 28 42 3,400 2,700 May 292 313 239 311 148 105 9,625 9,675 Jun 221 45 347 121 145 90 10,660 7,140 Jul 180 294 466 372 201 98 Aug 1,605 1,483 570 832 486 438 Sep 2,161 1,658 2,075 2,410 594 287 Oct 1,010 373 455 250 50 22 Nov 172 137 232 60 189 99 Dec 25 20 575 331 81 14 Tot 6,578 4,815 5,636 5,209 2,37 6 1,410 23,685 19,51 5 6. P8.5.D Number of individuals from target audience who participated in a community-wide event 2011/2012 FY achievement against target Annual Target Annual Achievement Achievement % Drama 38,700 96,286 248.8% Community￾Wide Events 40,000 43,000 107.5% Annual Target Annual achievement 2010–2012 cumulative Drama 38,700 96,286 Community￾Wide Events 308,000 281,313 162 USAID/MALAWI: EBT Prev Mid-Term Evaluation 7. P12.2.D Gender-Based Violence and Coercion: Number of people reached by an individual, small group, or community-level intervention or service that explicitly addresses gender-based violence and coercion related to HIV/AIDS (PEPFAR Indicator P12.2.D) 2011/2012 achievement against target Annual achievement 15–24 yrs 9,618 25 yrs + 18,546 Annual Achievement 28,164 Annual Target 10,710 Achievement % 263.0% From the PSI “Pivot Table” Name of Organization (All) P12.2D GBV DATA ANALYSIS PPA (All) District (All) Year 2012 TOC & DRAMA: P8.5D 2010-2012FY cumulative achievement against target 38,700 308,000 96,286 281,313 - 50,000 100,000 150,000 200,000 250,000 300,000 350,000 Drama Community Wide Events Annual Target Annual achievement USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 163 Data Month Males 15–24 yrs reached Females 15–24 yrs reached Males 25+ yrs reached Females 25+ yrs reached June 145 225 192 314 July 1,580 1,838 3,272 3,332 August 1,635 1,919 3,293 3,180 September 1,020 1,256 2,408 2,555 Grand Total 4,380 5,238 9,165 9,381 8. Objective 4, performance indicator 5: Referral uptake rate (non-PEPFAR) 2011/2012 FY aggregate (no cumulative data provided) General Referrals made Total # of clients referred 781 Total # of clients received the service 398 General referral uptake rate 51.0% Disaggregated by type of service (FY 2011/2012) Type of service Total number of clients referred Total number of clients who received the service HTC 334 165 STI 214 75 FP 93 70 Medical Care 83 60 TB Screening 44 23 PMTCT 7 1 GBV 4 3 PEP 2 1 TOTAL 781 398 164 USAID/MALAWI: EBT Prev Mid-Term Evaluation 9. PEPFAR indicator H2.3.D: Number of health care workers who successfully completed an in-service training program. NB: no data provided on health workers trained H2.3D Number of IPCV/A & Project Officers trained (sex-disaggregated) Total IPC/V and A and Project Officers trained since inception of the project 560 Males 332 Females 228 Total IPC/V and A and Project Officers trained since inception of the project who are currently active 506 Male 307 Female 199 10. P8.4.D 2a. Number of targeted condom service outlets (PEPFAR Indicator P8.4.D, Recommended) and 2b. Number of targeted condom service outlets in approved PPAs. Cumulative data P8.4D CONDOM SELLING OUTLETS: CUMULATIVE DATA Comments Number of condom selling outlets in general 938 Outlets visited only by PSI Sales Reps. NB: there are many outlets selling condoms which Sales Reps are not directly reaching Number of condom selling outlets in approved PPAs 1,928 Marketing team visited all PPAs and took census of all outlets selling condoms USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 165 APPENDIX J: NEW PROJECT REFERAL SYSTEM DIAGRAM STREAMLINED REFERRAL NETWORK ACTIVITIES Streamlined Referral Network Activities ÔHubÕ Health facility COMMUNITY LEVEL IPCA/V IPCA/V IPCA/V IPCA/V Provides: ART- services PMTCT services FP services STI services HTC services PEP services IPCV/A use referral tools & to make referrals to hub Health facility in PPA CBO CBO CBO CBO Psychosocial support provided by CBOs Police VSU Legal Service Undertakes: Screening of GBV survivors Branding: With approval from MoH/DHO Community-Level GBV sensitization including TAs, VHM, etc 166 USAID/MALAWI: EBT Prev Mid-Term Evaluation USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 167 APPENDIX K: REFERENCES Beyrer, Chris, Gift Trapence, Felistus Motimedi et al. “Bisexual Concurrency, Bisexual Partnerships, and HIV Among Southern African Men Who Have Sex with Men (MSM).” Sex Transm. Infect. April 21, 2010. Available at http://sti.bmj.com/content/early/2010/ 04/16/sti.2009.040162.abstract. Chizimba, Robert M. and Grace T. Malera. Counting the Uncatchables! Report of the Situation Analysis of the Magnitude, Behavioural Patterns, Contributory Factors, Current Interventions and Impact of Sex Work on HIV Prevention in Malawi. Lilongwe: Family Planning Association of Malawi, International Planned Parenthood Federation and United Nations Population Fund, November 2011. Desmond Tutu HIV Foundation. Men Who Have Sex with Men: An Introductory Guide for Health Care Workers in Africa. Edited by Benjamin Brown, Zoe Duby, Andrew Scheibe, and Eduard Sanders. Cape Town: Desmond Tutu HIV Foundation. Revised edition 2011. Available at http://www.esther.eu/wp-content/files_mf/13206639469.MSM_Guidefor HealthCareWorkers.pdf. EBT Prev Project. “A Guide for the IPC Session on HCT.” Lilongwe and Blantyre: EBT Prev Project. (Undated). EBT Prev Project. “A Guide for the Targeted Outreach Communication Session for Commercial Sex Workers.” Lilongwe and Blantyre: EBT Prev Project. (Undated). EBT Prev Project. “A Summary of Research Activities Among Selected Targeted Populations.” Lilongwe and Blantyre: EBT Prev Project, 2012. EBT Prev Project. “Application in Response to USAID-Supported RFA 674-08-0024. Sector II: Prevention for Populations and Settings with High-Risk Behaviors. Revised December 2008.” Blantyre and Lilongwe: PSI Malawi and Pact, December 2008. EBT Prev Project. “Concept Note. Linking MARPs to Key HIV Services by Community-Based Referral Processes.” Lilongwe and Blantyre: EBT Prev Project, 2011. EBT Prev Project. “Condom Social Marketing” (PowerPoint presentation). Lilongwe and Blantyre: EBT Prev Project, October 2012. EBT Prev Project. “Draft Manual for Training IPC Outreach Workers.” Lilongwe and Blantyre: EBT Prev Project. (Undated). EBT Prev Project. “Drama Checklist.” Lilongwe and Blantyre: EBT Prev Project. (Undated). EBT Prev Project. “EBT Prev Messages Disseminated by Partner by Quarters.” Lilongwe and Blantyre: EBT Prev Project. (Undated). EBT Prev Project. “EBT Prev Minimum Quality Standards Checklist.” Lilongwe and Blantyre: EBT Prev Project. (Undated). EBT Prev Project. “EBT Prev Trainers’ Guide for Health Workers: GBV.” Lilongwe and Blantyre: EBT Prev Project, 2012. EBT Prev Project. “EBT Prev: Update on Branded Lingalira Sankha Wekha Campaign Communication Deliverables to Date.” Lilongwe and Blantyre: EBT Prev Project. (Undated). EBT Prev Project. “General Referral Form.” Lilongwe and Blantyre: EBT Prev Project. (Undated). EBT Prev Project. “HIV Prevention Training for EBT Prev 2010” (PowerPoint presentation). Lilongwe and Blantyre: EBT Prev Project, 2010. EBT Prev Project. “HTC Referral Form.” Lilongwe and Blantyre: EBT Prev Project. (Undated). EBT Prev Project. “HTC Register.” Lilongwe and Blantyre: EBT Prev Project. (Undated). EBT Prev Project. “Hub Checklist.” Lilongwe and Blantyre: EBT Prev Project. (Undated). 168 USAID/MALAWI: EBT Prev Mid-Term Evaluation EBT Prev Project. “Indicator Calculation Table: 2011/12 FY Data Achievements Against Annual Target.” Lilongwe and Blantyre: EBT Prev Project, 2012. EBT Prev Project. “IPC Guides for Quarter 2 Messages.” Lilongwe and Blantyre: EBT Prev Project. (Undated). EBT Prev Project. “IPC Guides for Quarter 2 Scenarios.” Lilongwe and Blantyre: EBT Prev Project. (Undated). EBT Prev Project. “IPC Guides for Quarter 3 Messages.” Lilongwe and Blantyre: EBT Prev Project. (Undated). EBT Prev Project. “IPC Guides on Choices: Quarter 1.” Lilongwe and Blantyre: EBT Prev Project. (Undated). EBT Prev Project. “IPC Mentoring/IPC Supervision Checklist.” Lilongwe and Blantyre: EBT Prev Project. (Undated). EBT Prev Project. “IPCV/IPCA Checklist.” Lilongwe and Blantyre: EBT Prev Project. (Undated). EBT Prev Project. “IPCV/IPCA Refresher Reporting Form.” Lilongwe and Blantyre: EBT Prev Project. (Undated). EBT Prev Project. “Main Data Model.” Lilongwe and Blantyre: EBT Prev Project. (Undated). EBT Prev Project. “Monitoring and Evaluation Section” (PowerPoint presentation). Lilongwe and Blantyre: EBT Prev Project. (Undated). EBT Prev Project. “MSM Activities” (PowerPoint presentation). Lilongwe and Blantyre: EBT Prev Project, October 2012. EBT Prev Project. “Objective 2: Communications” (PowerPoint presentation). Lilongwe and Blantyre: EBT Prev Project, October 2012. EBT Prev Project. “Objective 4: Referral Activities” (PowerPoint presentation). Lilongwe and Blantyre: EBT Prev Project, October 2012. EBT Prev Project. “Project Monitoring Plan (Updated August 22, 2012).” Lilongwe and Blantyre: EBT Prev Project, 2012. EBT Prev Project. “Project Monitoring Plan (Updated June, 2010).” Lilongwe and Blantyre: EBT Prev Project, 2010. EBT Prev Project. “Quarter 3 Case Study for Jijo.” Lilongwe and Blantyre: EBT Prev Project. (Undated). EBT Prev Project. “Quarter 3 Case Study for Madalo.” Lilongwe and Blantyre: EBT Prev Project. (Undated). EBT Prev Project. “Quarterly Progress Report. Quarter 1, Fiscal Year 2010 (October–December 2009).” Lilongwe and Blantyre: EBT Prev Project, 2009. EBT Prev Project. “Quarterly Progress Report. Quarter 1, Fiscal Year 2011 (October–December 2010).” Lilongwe and Blantyre: EBT Prev Project, 2010. EBT Prev Project. “Quarterly Progress Report. Quarter 1, Fiscal Year 2012 (October–December 2011).” Lilongwe and Blantyre: EBT Prev Project, 2011. EBT Prev Project. “Quarterly Progress Report. Quarter 2, Fiscal Year 2010 (January–March 2010).” Lilongwe and Blantyre: EBT Prev Project, 2010. EBT Prev Project. “Quarterly Progress Report. Quarter 2, Fiscal Year 2011 (January–March 2011).” Lilongwe and Blantyre: EBT Prev Project, 2011. EBT Prev Project. “Quarterly Progress Report. Quarter 2, Fiscal Year 2012 (January–March 2012).” Lilongwe and Blantyre: EBT Prev Project, 2012. EBT Prev Project. “Quarterly Progress Report. Quarter 3, Fiscal Year 2009 (April–June 2009).” Lilongwe and Blantyre: EBT Prev Project, 2009. EBT Prev Project. “Quarterly Progress Report. Quarter 3, Fiscal Year 2010 (April–June 2010).” Lilongwe and Blantyre: EBT Prev Project, 2010. EBT Prev Project. “Quarterly Progress Report. Quarter 3, Fiscal Year 2011 (April–June 2011).” Lilongwe and Blantyre: EBT Prev Project, 2011. EBT Prev Project. “Quarterly Progress Report. Quarter 3, Fiscal Year 2012 (April–June 2012).” Lilongwe and Blantyre: EBT Prev Project, 2012. USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 169 EBT Prev Project. “Quarterly Progress Report. Quarter 4, Fiscal Year 2009 (July–September 2009) and Annual Report FY 2009.” Lilongwe and Blantyre: EBT Prev Project, 2009. EBT Prev Project. “Quarterly Progress Report. Quarter 4, Fiscal Year 2010 (July–September 2010) and Annual Report FY 2010.” Lilongwe and Blantyre: EBT Prev Project, 2010. EBT Prev Project. “Quarterly Progress Report. Quarter 4, Fiscal Year 2011 (July–September 2011) and Annual Report FY 2011.” Lilongwe and Blantyre: EBT Prev Project, 2011. EBT Prev Project. “Receiving Referral Register (RRR) for Receiving Service Providers.” Lilongwe and Blantyre: EBT Prev Project. (Undated). EBT Prev Project. “Referral Organizations’ Register.” Lilongwe and Blantyre: EBT Prev Project. (Undated). EBT Prev Project. “Referral System: Dwangwa Referral Network.” Lilongwe and Blantyre: EBT Prev Project. (Undated). EBT Prev Project. “Tentative Workplan (Year 1).” Blantyre and Lilongwe: PSI/M and Pact, 2009. EBT Prev Project. “Trainers’ Guide: Gender-Based Violence (for Health Workers).” Lilongwe and Blantyre: EBT Prev Project. (Undated–2011/2012). EBT Prev Project. “Updated List of IPC/V and IPC/A by PPA: EBT Prev Project.” Lilongwe and Blantyre: EBT Prev Project, February 2012. EBT Prev Project. “Workplan and Implementation Plan (FY 20112).” Blantyre and Lilongwe: PSI/M and Pact, 2011. EBT Prev Project. “Workplan and Implementation Plan (Year 2).” Blantyre and Lilongwe: PSI/M and Pact, 2010. Government of Malawi (GoM). 2012 Global AIDS Response Progress Report: Malawi Country Report for 2010 and 2011. Lilongwe: GoM, March 31, 2012. Available at http://www.unaids.org/en/dataanalysis/knowyourresponse/countryprogressreports/2012countries/ce_MW_Narrative_Re port[1].pdf. Kaler, Amy and Susan Watkins. “Asking God About the Date You Will Die: HIV Testing as a Zone of Uncertainty in Rural Malawi.” Demographic Research, 23: 905–932. November 9, 2010. Available at http://www.demographic￾research.org/Volumes/Vol23/32/. National AIDS Commission (NAC). Malawi Biological and Behavioral Surveillance Survey 2006 and Comparative Analysis of 2004 BSS and 2006 BBSS. Lilongwe: NAC, NSO, USAID, CDC, and FHI, 2007. National AIDS Commission. “Malawi National HIV and AIDS Strategic Plan January 2012–December 2016: Final Draft: December 31, 2011.” Lilongwe: NAC, December 2011. National AIDS Commission. “Operation Plan of the National HIV Prevention Strategy. Phase I: October 2009–June 2011.” Lilongwe: NAC, January 2010. National AIDS Commission, World Health Organization, University of California San Francisco, UNAIDS, Centers for Disease Control and Prevention. Report of the Malawi Triangulation Project: Synthesis of Data on Trends in the National and Local HIV Epidemics and the Reach and Intensity of Prevention Efforts. Process, Key Findings and Recommendations. Lilongwe: NAC, 2006. National Statistical Office (Malawi) and ICF Macro. Malawi Demographic and Health Survey 2010. Zomba, Malawi and Calverton, Maryland: NSO and Macro, September 2011. Available at http://www.measuredhs.com/publications/publication-fr247-dhs-final-reports.cfm. Pact Malawi. “EBT Prev Project: Data Quality Assessment Report. October 2011–August 2012.” Lilongwe: Pact, August 2012. Pact Malawi. “EBT Prev Project: Data Quality Assessment Report. October–December 2011.” Lilongwe: Pact, 2012. Pact Malawi. “Pact’s Role in EBT Prev” (PowerPoint presentation). Lilongwe: Pact, 2012. President’s Emergency Plan for AIDS Relief (PEPFAR). “Technical Guidance on Combination HIV Prevention (MSM).” Washington DC: PEPFAR, May 2011. 170 USAID/MALAWI: EBT Prev Mid-Term Evaluation PSI Malawi. “Program Description: PSI Malawi Cooperative Agreement No. 674-A-00-09-00031-00. Evidence Based Targeted HIV Prevention Project (EBT Prev).” Blantyre and Lilongwe: PSI/M, 2010. PSI Malawi. “Start-Up Plan: The Evidence-Based Targeted HIV Prevention (EBT Prev) Project Prevention for Populations and Settings with High Risk Behaviors (Cooperative Agreement Number: 690-A-00-09-00031-00).” Blantyre: PSI/Malawi, March 2009. PSI/Malawi and USAID. “Report: Population Size Estimate Exercise in Selected PPAs.” Lilongwe: PSI/M and USAID, 2011. Republic of Malawi. Malawi Voluntary Medical Male Circumcision Communication Strategy 2012–2016. Lilongwe: NAC. (Undated–2011/2012). Republic of Malawi and National AIDS Commission. National HIV Prevention Strategy 2009 to 2013. Lilongwe: NAC, 2009. Available at http://www.k4health.org/sites/default/files/_ National%20HIV%20Prevention%20Strategy.pdf. Schouten, Erik, Andreas Jahn, Dalitso Midiani, Simon D. Makombe et al. “Prevention of Mother-to-Child Transmission of HIV and the Health-Related Millennium Development Goals: Time for a Public Health Approach.” Lancet, 378(9787): 282–284, 2011. UNICEF, UNAIDS, PEPFAR. “Malawi HIV Prevention Partners’ Visit, February 2011: Trip Report.” Lilongwe: UNICEF, UNAIDS, PEPFAR, USAID, MoH, NAC, OPC, February 2011. USAID. Checklist for Assessing USAID Evaluation Reports. Washington DC: USAID, 2011. Available at http://transition.usaid.gov/policy/evalweb/evaluation_resources.html. USAID. Evaluation: Learning from Experience—USAID Evaluation Policy. Washington DC: USAID, January 2011. Available at http://transition.usaid.gov/evaluation/USAIDEvaluation Policy.pdf. USAID. Gender Equality and Female Empowerment Policy—USAID Policy. Washington DC: USAID, March 2012. Available at http://transition.usaid.gov/our_work/policy_planning_ and_learning/documents/GenderEqualityPolicy.pdf. USAID. Performance Monitoring & Evaluation: TIPS: Conducting Mixed-Method Evaluations. Washington DC: USAID TIPS No. 16, 2010. Available at http://transition.usaid. gov/policy/evalweb/documents/TIPS-ConductingMixedMethodEvaluations.pdf. Watkins, Susan. Desk Review of HIV Determinants in Malawi. Lilongwe: BRIDGE II Project, March 2010. World Health Organization. Programmatic Update. Use of Antiretroviral Drugs for Treating Pregnant Women and Preventing HIV Infection in Infants: Executive Summary. Geneva: WHO HIV/AIDS Programme, April 2012. Available at http://www.unaids.org.cn/en/index/ Document_view.asp?id=594. USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 171 172 USAID/MALAWI: EBT Prev Mid-Term Evaluation For more information, please visit http://www.ghtechproject.com/resources USAID/MALAWI: EBT PREV Mid-Term Performance Evaluation 173 GH Tech Bridge II Project 1725 Eye Street NW, Suite 300 Washington, DC 20006 Phone: (202) 349-3900 Fax: (202) 349-3915 www.ghtechproject.com