1 October 2014 This publication was produced at the request of the United States Agency for International Development and independently prepared by International Business & Technical Consultants, Inc. (IBTCI). USAID/Zambia: Zambia-Led Prevention Initiative (ZPI) End-of-Project Performance Evaluation 2 ACKNOWLEDGEMENTS The ZPI End-of-Project Performance Evaluation was carried out with the help of two research assistants, Mwanza Bisoloma and Cynthia Lungu, in addition to assistance from national, provincial, district and community stakeholders, volunteers and beneficiaries. Many institutions and stakeholders gave freely of their time and opinions during the evaluation. The IBTCI evaluation team would like to thank the following institutions and individuals for their support and the information they provided:  USAID/Zambia  Government of the Republic of Zambia  ZPI consortium and subgrantees  Chiefs, headmen, volunteers and beneficiaries in Luapula, Western, Southern, Eastern, Lusaka and Copperbelt Provinces  USAID implementing partners We would like to give special thanks to the USAID staff members who supported this evaluation, in particular Ms. Patricia Sitimela, Senior M&E Specialist. The team would also like to thank all the individual staff members at FHI360 who provided their support to make this a participatory evaluation. 3 USAID/ZAMBIA-LED PREVENTION INITIATIVE (ZPI) Authors: Dr. Rosemary Barber-Madden, Dr. William Sambisa, Milka Juma, and Dr. Moses Simuyemba. October 21, 2014 AID-RAN-I-00-09-00016/AID-611-TO-14-00001 Cover Photo Credit: Dr. Moses Simuyemba Caption: An outreach site for ZPI service delivery DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development (USAID) or the United States Government. 4 TABLE OF CONTENTS EXECUTIVE SUMMARY .................................................................................................................................. 8 EVALUATION PURPOSE ............................................................................................................................................................ 8 PROJECT BACKGROUND ......................................................................................................................................................... 8 EVALUATION QUESTIONS, METHODS, AND LIMITATIONS .................................................................................................. 8 FINDINGS.................................................................................................................................................................................... 9 CONCLUSIONS ........................................................................................................................................................................ 11 RECOMMENDATIONS ............................................................................................................................................................. 11 1. EVALUATION PURPOSE & EVALUATION QUESTIONS ...................................................................13 2. PROJECT BACKGROUND ........................................................................................................................14 3. EVALUATION METHODS & LIMITATIONS ..........................................................................................15 4. FINDINGS ....................................................................................................................................................16 5. CONCLUSIONS..........................................................................................................................................48 6. RECOMMENDATIONS ..............................................................................................................................50 ANNEX I: EVALUATION SCOPE OF WORK ............................................................................................51 ANNEX II: DATA COLLECTION TOOLS ...................................................................................................62 ANNEX III: EVALUATION PLAN AND ANALYSIS MATRIX ..................................................................70 ANNEX IV: LIST OF DOCUMENTS REVIEWED .......................................................................................82 ANNEX V: LIST OF PERSONS INTERVIEWED .........................................................................................84 ANNEX VI: NUMBERS OF RESPONDENTS FOR KIIS AND FGDS .........................................................90 ANNEX VII: PROVINCIAL DATA ANALYSIS CHARTS...........................................................................91 5 ACRONYMS AED Academy for Educational Development AIDS Acquired Immuno-Deficiency Syndrome ANC Antenatal Care ART Antiretroviral therapy ARV Antiretroviral BCC Behavior Change Communication CAT Community Alcohol Teams CBD Community-based distribution CBO Community-based organization CBV Community-based volunteer CHAMP Comprehensive HIV/AIDS Management Program CMMB Catholic Medical Mission Board COMPACT Community Mobilization for Preventive Action CSH Communications Support for Health CPO Community Purchasing Order CSW Commercial Sex Worker DAFT District AIDS Task Force DHO District Health Officer DHS Demographic and Health Survey DMO District Medical Officer DMPA Depot Medroxy-progesterone Acetate DQA Data Quality Assessment EE Economic Empowerment (GROW) EOP End-of-Project Performance ESP Economic Strengthening Pathway FP Family Planning FGD Focus Group Discussion FHI-360 Family Health International 360 GBV Gender-based Violence GDA Global Development Alliance GROW Grass Roots Building Our Wealth GRZ Government of Republic of Zambia HIV Human Immunodeficiency Virus HTC HIV Testing and Counseling IDA Individual Development Accounts IEC Information, Education and Communication IP Implementing Partner IR Intermediate Result 6 JRT Job Readiness Training KAP Knowledge, Attitude, and Practice KII Key Informant Interview KP Key Population LOP Life of Project M&E Monitoring and Evaluation MARP Most-at-risk Populations MC Male Circumcision MCDMCH Ministry of Community Development, Mother and Child Health MCP Multiple and Concurrent Partnerships MFI Microfinance Institutions MIO Male Involvement Officer MOH Ministry of Health MOHA Ministry of Home Affairs MOT Modes of Transmission MOU Memorandum of Understanding MTA Men Taking Action NAC National AIDS Council NGI New Generation Indicators NGO Non-governmental Organization NHC Neighborhood Health Committees OVC Orphans and Vulnerable Children OYDC Olympic Youth and Development Center P3 Public-Private Partnerships PATF Provincial AIDS Task Force PC Population Council PCI Project Concern International PEG Psycho-Education Program Group PEPFAR President’s Emergency Plan for AIDS Relief PLA Participatory Learning Action PLWHA People Living with HIV/AIDS PMP Performance Monitoring Plan PMTCT Prevention of Mother-to-Child Transmission PRA Participatory Rural Appraisal PRISM Partnership for Integrated Social Marketing R-MARS Rapid - Monitoring of AIDS Referral System REFLECT Regenerated Freirean Literacy through Empowering Community SES Socioeconomic Status SHARPZ Serenity Harm Reduction Program Zambia SOW Scope of Work 7 STEPS Economic Strengthening, Prevention and Support to Orphans and Vulnerable Children (STEPS OVC) STI Sexually Transmitted Infection VCT Voluntary Counseling and Testing VSU Victims’ Support Unit ZPCTII Zambia Prevention, Care and Treatment Partnership Project II ZDHS Zambia Demographic and Health Survey ZHECT Zambia Health Education and Communication Trust ZPI Zambia-led Prevention Initiative ZSBS Zambia Sexual Behavior Survey 8 EXECUTIVE SUMMARY EVALUATION PURPOSE International Business and Technical Consultants, Inc. (IBTCI) is pleased to present the end-of-project (EOP) performance evaluation report of the USAID/Zambia’s Zambia-led Prevention Initiative (ZPI), implemented by Family Health International 360 (FHI360) and its partners. The EOP’s report aimed to answer the following four key questions: 1) to what extent did the project achieve the planned objectives and results in HIV prevention among vulnerable and most-at-risk populations; 2) to what extent were the project design, implementation, and management effective, and why; 3) what progress was made toward ensuring the sustainability of ZPI’s approaches; and 4) to what extent did the strategic information activities serve to inform the planning, implementation and monitoring of the project? PROJECT BACKGROUND The ZPI project aimed to increase the use of community-level interventions through a targeted approach, and to provide technical leadership on comprehensive, community-based prevention efforts aimed at reducing new HIV transmission in Zambia. The ZPI life-of-project budget was USD $39,726,852; of which $29,213,733 was allocated for the duration of the project. FHI360 was the lead implementing partner in the project, in collaboration with the Population Council (PC), Catholic Medical Mission Board (CMMB), Afya Mzuri, the Comprehensive HIV and AIDS Management Program (CHAMP), Grass Roots Soccer, HODI, Project Concern International, and the Zambia Health Education and Communications Trust (ZHECT). The project was implemented in 44 districts in nine provinces. ZPI’s activities were guided by the theoretical construct of interrelated ‘risk and vulnerability lenses’. ZPI used the Stages of Change Model that encompasses both cognitive and behavioural approaches to develop its behavior change framework. The model permitted the project to focus on individual change, such as condom use, with the understanding that the effects move beyond individual change and result in changing social norms, such as gender norms. EVALUATION METHODS AND LIMITATIONS The evaluation used both quantitative and qualitative methods including a document review, with primary collection through 97 key informant interviews (KIIs) and 37 focus group discussions (FGDs), with 187 community volunteers and 225 beneficiaries in nine districts across six provinces. In all, 509 people were interviewed. To minimize interviewer bias the team collected qualitative data in pairs and conducted thematic analyses as a group. Evaluators used secondary data sources for quantitative methods, including a comparative analysis of baseline (2011) and midline surveys (2013), and comprehensive analysis of program performance data against selected targets to proximate end-line data. The evaluation triangulated the qualitative and quantitative data to respond to the evaluation questions. The qualitative data supported the surveys’ findings and the routine data. The evaluation had the following limitations: The evaluation team was charged with collecting and analyzing qualitative data to validate effects of ZPI community based interventions. It was not feasible to undertake a study similar to the sample size, timeline and geographic coverage of the baseline and midline house hold surveys. The evaluation team opted to conduct Focus Group Discussions (FGDs), having selected purposefully nine (9) communities within six (6) provinces where programmatic interventions were carried out. More than 400 tribal chiefs, headmen, community volunteers and beneficiaries were interviewed in 37 FGDs. However, these data provided adequate evidence to substantiate the effect of community based interventions. Interviewers conducted data collection and thematic analysis in sub- teams of three (3) to minimize interviewer bias and facilitate ongoing analysis. In addition, these qualitative findings provide the contextual information about how ZPI and its partners carried out interventions, uptake at community level, and actions taken to prevent HIV/AIDS and its transmission, within a year after the midline household survey. 9 FINDINGS Project achievement of the planned objectives and results. Overall, outreach improved as a result of the ZPI model and the project exceeded most of its targets, achieving most behavioral outcomes. The qualitative data corroborates the survey’s findings as well as the increase in some performance indicators as shown by routine data; particularly the change in perception toward adopting new behaviors and acceptability and support for intervention implementations by traditional leaders and beneficiaries. The adoption of new behaviors largely explains the improvements in most performance indicators. This picture is consistent across all the provinces visited by evaluators. The ZPI project implemented community-based HIV prevention education, and promoted HIV counseling and testing (HTC), for individuals and couples. The interventions were implemented in an interconnected manner, with all implementing partners applying a ‘one lens’ integrated approach. They used both traditional socio￾cultural structures including neighborhood health committees (NHCs), and groups set up to mitigate specific health and social behavioral risks and vulnerabilities. CBVs successfully undertook outreach activities to prevent HIV transmission, gender-based violence (GBV), and alcohol abuse in addition to promoting uptake of HTC services. The ZPI project aimed to reach at least 572,000 individuals from the targeted population with individual and/or small group level HIV preventive interventions. Overall, the routine data showed that the project had reached 625,410 individuals (109 percent of its life of project (LOP) target) with prevention interventions (315,287 males and 310,123 females). Provincial variations in this performance indicator were observed. Lusaka had the highest number of individuals reached with prevention interventions (170,374) followed by Northwest (94,754) and Southern (93,319) provinces. Zambia’s Northern Province saw the least number of individuals reached because the program only began implementing its interventions there in 2013 (i.e., phase three of the rollout of the program based on the project design). The qualitative findings show that due to ZPI’s community-based volunteers (CBVs) demand for creation activities, individuals were increasingly willing to use available HIV services in the communities and health facilities, including mobile facilities offering HTC and medical male circumcision (MC) services. Health facility service providers were also considered to be the main channels for HIV prevention messages. Quantitative analysis shows that the reach of HTC and medical MC services increased significantly during the project. Overall, the project has reached 216,408 clients (173 percent of its LOP target of 125,000 clients: 101,452 males and 114,956 females) with HTC services. Trend analysis shows the number of individuals receiving HTC services increased five-fold from 24,374 in 2010 to 117,896 in 2013. Qualitative information obtained from beneficiaries, community leaders and volunteers in rural and urban FGDs and KIIs indicated that ZPI conducted both gender-sensitive individual and couple HTC promotion. This increased awareness and acceptance of HTC services. Qualitative data for both male and female KIIs and FGDs shows acceptance of male circumcision as a preventive measure against HIV infection across all the provinces visited. 72% of FDG respondents, both community members and traditional leaders, stated that male circumcision was positively correlated with the prevention of HIV infection. ZPI created community-based interventions such as the community alcohol teams (CATs) and psycho￾education program groups (PEGs) to tackle growing alcohol abuse. Both groups worked to mitigate the contributory influence of alcohol abuse and other drugs on HIV infection, providing support to adult males and females, as well as promoting delayed alcohol consumption among youths aged 15-24. The project created a total of 28 village anti-GBV committees with 361 members in seven provinces. The anti￾GBV committees were established with trained community chiefs, village headmen, and other resource persons. These individuals sensitized and mobilized people to GBV prevention. As a village headman said during a KII: “violence has reduced in the village, insults have [been] reduced and fights in marriage have reduced. Even now when we teach people [they] see the benefits.” Both quantitative and qualitative findings showed that the economic empowerment program (i.e., Grass Roots building Our Wealth - GROW) had a positive impact on its members earning capacity. The GROW Model was successful in reaching almost 5,000 beneficiaries with loans and demonstrated success as a structural 10 intervention supporting the HIV prevention effort, with considerable uptake at community level by women and later by men. The midline survey showed that individuals exposed to income-generating activities were more likely to be earning money than those who were not. The ZPI project’s economic empowerment program was most active in the Eastern Province with about 41 percent of all GROW members (2,025/4,977). The exposure effect seen in the midline survey findings is supported by the program performance data and the qualitative findings. By July 2014, the GROW program had reached 16,021 adults and children with economic strengthening services (153 percent of the LOP target). Trend analysis shows the number of adults and children provided with economic strengthening services increased about six-fold from 1,308 in 2010 to 7,243 in 2013. However, the rate of expansion of those reached declined in 2014, although an additional 4,617 people were reached. 61% percent of FGDs (21/36) reported that the GROW program benefited individual group members and their families tremendously. Project design, implementation, and management effectiveness and reasons. The ‘risk and vulnerability lens’ approach brought about innovation by integrating components that predispose individuals and communities to HIV infection, constructing a multifaceted platform for change based on the theory of change. ‘Lenses’ provided a range of entry and implementation points to HIV prevention and options to address other social and health problems. Involvement of national, provincial, and district public sector structures lent buy￾in and fostered participatory implementation. Anchoring interventions in communities through the grantee approach and community purchasing orders (CPOs) engaged local communities and promoted sustainability. The incorporation of a strong gender element into the programming at all levels of the project, including staff and volunteers, brought about changes in gender norms as reported in the midline survey and substantiated by the qualitative data. The qualitative data showed that male involvement catalyzed changes toward gender equity and brought about couple counseling and reductions in alcohol and substance abuse. Involving traditional leaders was a key entry point as they became role models for communities, and were partners in intervention implementation. These findings could be used to inform programming in the next generation, particularly at provincial and district levels. The project reached underserved areas, improving access to critical HIV prevention information and services where they did not exist, chiefly in Western and Luapula provinces. The project built strong partnerships, with provincial, district, and community leaders, including other USAID implementing partners. The project engaged in joint planning and implementation activities with all of these groups. However, the rollout of major activities during project closure in Western and Northern Provinces was below standard. Also, subcontracting grantees using annual contracts discouraged long-term planning and optimal performance. The implementation of the mental health lens was more challenging than had been anticipated due to a lack of appropriate mental healthcare facilities that can be linked to the outreach program. During the course of the project activities, the Government of the Republic of Zambia (GRZ) realigned the Ministry of Community Development, Mother and Child Health (MCDMCH) and Ministry of Health in a way that involved redefining key functions of each Ministry relating to the activities of ZPI at national, provincial and district levels. The project had to put in extra effort to bring the MCDMCH up to speed with its strategies and coordination of activities with the newly formed ministry. In addition, changes in the President’s Emergency Plan for AIDS Relief (PEPFAR) indicators resulted in changes in data collection and tracking process with limited time for re-trainings of staff and volunteers, thus affecting data quality. Progress made toward ensuring the sustainability of ZPI’s approaches. In spite of the lack of a clearly defined sustainability plan, the project achieved a high level of ownership at the national, provincial and district levels. The strongest buy-in was at the provincial and district levels. Community involvement at every stage of the project created ownership and led to collective actions to create change. Communities bought into the concept of the lenses and adopted positive social and sexual behaviors, namely male involvement, counseling and testing, gender equity, economic empowerment (GROW approach), and alcohol reduction. Qualitative findings indicate community activities will continue with support from the adoption of chiefdom by-laws, alcohol support groups, GROW groups, and community volunteers. Community volunteers and beneficiaries 11 have gained knowledge and experience that will remain at the local levels; assuring that ZPI’s approaches will continue, albeit not at the current scale. Strategic information systems to inform the planning, implementation and monitoring of the project. ZPI established a Monitoring and Evaluation (M&E) system and Performance Monitoring Plan (PMP) as a basis for strategic information framework that functioned well at the project’s central level. Here, the project used routine M&E data and survey findings to inform annual work plan development and budgeting. The project also conducted quarterly internal review meetings with provincial and senior management teams to assess progress toward targets and plans for the next quarter. However, there was no deliberate effort to use the data at the lower levels for decision-making. Where the project met or exceeded targets, such as for HTC and GBV, ZPI re-focused its annual work plan on areas where targets were not met. Volunteers in some communities had reporting forms, while others did not. Some used the wrong reporting forms. ZPI’s provincial offices had no capacity to analyze and use generated strategic information. Inconsistencies in reporting to national systems were identified in Southern, Luapula and Western Provinces. The baseline and midline studies provided population-based, cross-sectional data on “lens” indicators. If conducted as planned, an end line survey would have provided quantitative evidence on changes in “lens” indicators by the end of the project. However, data from this end-of-project evaluation was used to proximate endline data and corroborate findings in the midline study and trends shown by the PMP data such as positive attitudes toward HIV testing, male condom use, income earning among females, and improved gender equity attitudes. Behavioral change progresses cannot be solely attributed to the project’s HIV activities because other HIV interventions were being implemented in the ZPI intervention regions. The evaluation collected sufficient evidence through both qualitative methods and data triangulation to conclude that ZPI contributed to both social and risky health behavior changes in the communities visited. CONCLUSIONS Overall, ZPI exceeded most of its planned objectives and targets and demonstrated the success of the ZPI model. ZPI’s main comparative advantage was its design of a theoretical construct of ‘risk and vulnerability’ lenses, based on the Stages of Change Model. The ‘lenses’ approach for project design was effective, and should be able to inform programming in the next generation. The structures were used to reach out to and support communities to adopt new social, cultural, and sexual behaviors, reducing ‘risks and vulnerability,’ and promoting healthy behavioral changes. The momentum generated by the project needs to continue to sustain the project‘s gains. Furthermore, the project demonstrated expertise in transferring a new theoretical construct that generated coordinated HIV prevention efforts at community level. Some provincial and district level grantees had the capacity to undertake these approaches from the outset. At the same time, these organizations had the necessary local credibility to enter hard-to-reach, vulnerable communities, and provide continued support. The economic empowerment approach enabled communities to be engaged in micro-savings and micro￾lending. The project carried out several studies that were used to monitor the success of the project’s implementation and to assess what amendments would be needed in mid-course. Survey findings structured according to the ‘lenses’ were used for work plan development. ZPI established an M&E system to track project performance, assuring timely submission of progress data each year, as a basis for a strategic information framework. RECOMMENDATIONS The following recommendations are for USAID/Zambia to strengthen and capitalize on the achievements and best practices of Zambia ZPI project and to make improvement in those areas where performance is low. Important lessons are drawn to inform the design of future HIV Prevention interventions under USAID/Zambia’s Country Development Cooperation Strategy. 12  Continue to support and expand coverage of effective and proven community-based, community-led interventions to reduce HIV risk and vulnerability.  Disseminate the best practices and lessons learned among program planners and policy makers of the application of the theoretical construct of “risk and vulnerability lenses” and the theory of change to promote adaptation and replication.  For future generation of programs, build program design, management, and monitoring capacity at provincial, district and community levels, and decentralize technical and management functions responsibility from central to provincial and lower levels to support sustainability at these important levels of programming. o Upgrade the organizational and management capacity of district and community structures, especially local NGOs and community leaders to enable them to prioritize, manage, monitor, and be accountable for primary prevention and strengthening connections with health delivery system for diagnosis, care and treatment, and support. 13 1. EVALUATION PURPOSE & EVALUATION QUESTIONS 1.1 EVALUATION PURPOSE This end-of-project performance (EOP) evaluation aimed to analyze the Zambia-Led Prevention Initiative (ZPI) and the success of its implementation in the relevant areas to date. This analysis includes relevance to identified needs; ability to achieve critical results; efficiency in achieving those results; and steps made toward sustainability. This evaluation aimed to provide USAID/Zambia, the United States Government (USG)/ the President’s Emergency Plan for AIDS Relief (PEPFAR), the Government of the Republic of Zambia (GRZ), and Family Health International 360 (FHI360) with objective data demonstrating the effectiveness of community-led behavioral and structural HIV prevention interventions, and identify promising practices to inform future HIV Prevention intervention methods under USAID/Zambia’s Country Development Cooperation Strategy. The evaluation involved an assessment of ZPI’s activities in a representative sample of the nine provinces where implementation took place. The evaluation analyzed gender implications in order to assess the degree to which gender equity was advanced in HIV prevention interventions as indicated in the overall purpose. The evaluation conclusions, recommendations and lessons learned are to be used to inform HIV prevention strategy development, identify promising practices in community-led HIV prevention and strengthen HIV combination prevention programming in the future. 1.2 EVALUATION QUESTIONS Four key questions, stated in the Scope of Work (Annex I), guided the EOP evaluation: 1. To what extent did the project achieve the planned objectives and results in HIV prevention among vulnerable and most-at-risk populations? 2. To what extent were the project design, implementation, and management effective and why? 3. What progress has been made toward ensuring the sustainability of ZPI’s approaches? 4. To what extent are the strategic information activities serving to inform the planning, implementation and monitoring of the project? According to the scope of work (SOW), the main aim of the evaluation is to analyze project performance and achievements related to the:  Relevance of ZPI’s approaches to identified needs at the community, district, provincial and national levels, and the ability to achieve critical results;  Effectiveness of ZPI’s project approaches and main activities in HIV prevention, including: o HIV/AIDS prevention, treatment, care and support interventions: male circumcision (MC), HIV testing and counseling (HTC), prevention of mother-to-child transmission (PMTCT), stigma and discrimination reduction; o Alcohol/substance abuse interventions; o Intervention for sexual abuse in homes; o Economic vulnerability; o Sexual and gender norm, and gender-based violence (GBV) interventions; and o Reproductive health/family planning.  Efficiency with which financial resources/inputs (funds, expertise, time etc.) are used to achieve results; and  Sustainability or degree to which ZPI project services or processes continue over the medium and long-term, once financial, material and training inputs decrease or discontinue. 14 2. PROJECT BACKGROUND With an estimated HIV prevalence of 14.3 percent among men and women aged between 15-49, Zambia is one of the most affected Sub-Saharan African countries hit by the HIV/AIDS pandemic. Of the 13.1 million people living in Zambia in 2010 (Census of Population and Housing Report, 2010), an estimated 1.1 million people were infected with HIV, with less than 15 percent aware of their status. Key social and behavioral factors contribute to the high prevalence of HIV in Zambia, including gender-based violence, low levels of male circumcision, multiple and concurrent partnerships (MCP), and low levels of condom use. ZPI aims to use community-level interventions through a targeted approach and to provide technical leadership and expertise on comprehensive, effective, community-based prevention efforts aimed at reducing HIV transmission in Zambia. A USAID Task Order for the ZPI was awarded to the former Academy for Educational Development (AED) (now under FHI360). The ZPI life-of-project budget was $39,726,852.00. Up to year 4, a total of $29,000,000 was obligated. FHI360 is the lead implementing partner for the USAID-funded ZPI, in collaboration with the Population Council (PC), Catholic Medical Mission Board (CMMB), Afya Mzuri, Comprehensive HIV and AIDS Management Program (CHAMP), and Zambia Health Education and Communications Trust (ZHECT). The ZPI project’s main aims are as follows: 1.Build capacity in communities affected by HIV/AIDS to access more effective, gender-sensitive, higher￾quality HIV prevention programs, including HTC, MC, and PMTCT. 2.Strengthen the continuity and coordination of, as well as commitment to, effective, efficient, and sustainable HIV prevention, including HTC, MC, and PMTCT. 3.Design efficient, sustainable, and locally owned responses to HIV/AIDS, including increased engagement with the private sector. 4.Provide community-based family planning and reproductive health services as an adjunct to effective prevention of HIV/AIDS. The ZPI project targets vulnerable and most-at-risk populations, including: youth; orphans and vulnerable children (OVC); persons living with HIV and AIDS (PLHA); people engaging in MCP; discordant couples, especially in cases of undisclosed zero-positive status or risky sexual behavior; and at-risk, HIV-negative adults. The ZPI project also targets "better off and better educated" Zambians for whom the 2007 Zambia Demographic and Health Survey (ZDHS) shows higher HIV prevalence. The ZPI project is present in all of Zambia’s provinces, having used a staggered/phased approach to set up operations throughout the country. Since rollout, the ZPI project has undergone a number of modifications. The first significant modification was done in December 2010; modifying the Task Order by replacing the Branding Strategy and Marking Plan. The second modification changed AED to FHI360. On April 26, 2012, another modification was made to adjust the life-of-project (LOP) targets for PMTCT and HIV prevention, discontinue partnership with Project Concern International (PCI) as a sub-partner, and to realign the budget. ZPI’s modification was to exercise USAID’s unilateral right to exercise the family planning option activity and to effect key personnel changes, in addition to incorporating mandatory PEPFAR reporting requirements. ZPI originally planned for an endline household survey at the end of Year 3, later cancelled due to budget constraints. ZPI was planned as a four-and-a-half-year project to be expanded in three phases. In Year 1, the project was initiated in five provinces, namely Eastern, Southern, North-Western, Lusaka and the Copperbelt, covering four to seven districts in each province for a total of 28 districts. In Year 2, the project was expanded to Luapula and Central provinces. In Year 3, the project initiated activities in the remaining two provinces - 15 Northern and Western. The project was implemented in at least 50 percent of all districts nationwide, carrying out activities in 44 districts in nine provinces. ZPI’s activities were guided by the theoretical construct of ‘risk and vulnerability lenses’ and the theory of change. ZPI used the Stages of Change model that encompasses both cognitive and behavioural approaches to develop its behavioral change framework. This model was chosen because it allowed the project to focus on individual change, such as condom use, with the understanding that the effects move beyond individual change and include changing social norms, such as gender norms, in target communities. ZPI’s community mobilization methodologies used this concept of change as a process to design the Lens interventions. The ZPI targeted nine ‘lenses’, that are often not recognized or addressed through traditional HIV prevention interventions and programs: most-at-risk behaviors; gender-based-violence; economic vulnerability; alcohol and other drug abuse; mental health; people with disabilities; violence against children; uptake of biomedical services; and stigma & discrimination. 3. EVALUATION METHODS & LIMITATIONS 3.1 METHODOLOGY The evaluation used a mix of quantitative and qualitative methods including a document review. The document review included an assessment of ZPI’s Work plans, Quarterly Reports, PMP data, Contract and Modifications, and studies conducted by the ZPI project (Annex IV: List of documents reviewed). The evaluation triangulated the qualitative and quantitative data to respond to the evaluation questions. The qualitative data provided a unique perspective on the project’s performance, and also collaborated findings from the project’s surveys with the routine M&E data. The evaluation analyzed gender implications related to gender equity in HIV prevention interventions as an overarching element. Qualitative methods were used for primary data collection through key informant interviews (KIIs) and focus group discussions (FGDs). Evaluators purposively selected sites for data collection in six of the nine provinces where ZPI was implemented, namely Copperbelt, Eastern, Luapula, Lusaka, Southern and Western. Nine districts were selected for the evaluation due to the high concentration of project interventions in each district. Respondents for KIIs and FGDs were selected due to their key roles in the project as either major stakeholders, facilitators, community volunteers or beneficiaries. The team interviewed a total of 509 respondents at the national, provincial, district and community levels. A total of 96 KIIs were carried out with national, provincial, district and community stakeholders, including 28 chiefs and headmen (Annex V: List of persons interviewed). Thirty-six FGDs were held with 187 facilitators and community volunteers and 225 beneficiaries in nine districts (Annex VI: Number of KIIs and FGDs, by gender and type of respondents). Each FGD was comprised of 12 community volunteers or beneficiaries. National stakeholders interviewed included representatives of the National AIDS Council (NAC), Ministry of Community Development Maternal and Child Health (MCDMCH), and representatives of District AIDS Task Forces (DATFs); and Provincial AIDS Task Forces (PATFs), health officers, Victim Service Units, local non-government organizations (NGOs), sub￾grantees, and traditional leaders including Chiefs and Headmen. Quantitative methods adopted included secondary analysis of household surveys and routine monitoring data. Secondary data sources used for quantitative methods included a comparative analysis of baseline (2011) and midline household surveys (2013), and comprehensive analysis of program performance data against selected targets to proximate end-line data. Trend analysis was carried out using routine performance monitoring data framed by the PMP indicators. The Population Council (PC) conducted the baseline and midline household surveys while routine PMP data was collected and managed by FHI360. The evaluation was conducted between July and September 2014. A team of three international and one national consultant, two research assistants (RAs), and one logistician undertook evaluation activities beginning 16 on July 7, 2014 with document review and development of methodology plans. The evaluation team convened in Lusaka on July 21 to develop data collection tools (Annex II: Data Collection Tools), pilot testing, training of RAs and finalize detailed methodology and analysis plans (Annex III: Evaluation Plan and Analysis Matrix). The team spent three weeks collecting data from July 25 to August 15. The final evaluation report was prepared and submitted to USAID/Zambia on October 21, 2014. 3.2 LIMITATIONS The evaluation had the following limitations: The evaluation team was charged with collecting and analyzing qualitative data to validate effects achieved from baseline to midline household surveys. It was not feasible to undertake a study similar to the sample size, timeline and geographic coverage of the baseline and the midline house hold surveys. The evaluation team opted to conduct Focus Group Discussions (FGDs), having selected purposefully nine (9) communities within six (6) provinces where programmatic interventions were carried out. More than 400 tribal chiefs, headmen, community volunteers and beneficiaries were interviewed in 37 FGDs. However, these data provided adequate evidence to substantiate the effect of community based interventions. Interviewers conducted data collection and thematic analysis in sub- teams of three (3) to minimize interviewer bias and facilitate ongoing analysis. In addition, these qualitative findings provide the contextual information about how ZPI and its partners carried out interventions, uptake at community level, and actions taken to prevent HIV/AIDS and its transmission, within a year after the midline household survey. To minimize any interviewer bias associated with qualitative data collection, the team conducted data collection in pairs and thematic analysis as a group. 4. FINDINGS The findings for the four evaluation questions are derived from data triangulation following the document reviews, secondary data analysis, FGDs and KIIs. The qualitative data gave a unique perspective into the project’s performance, and also collaborated findings of the project’s surveys with the routine M&E data. The most frequent, dominant or significant themes arising from the KIIs and FGDs responses related to how ZPI’s design, implementation and management is perceived and valued, rather than the full range of respondents’ responses on their perceptions of ZPI. Question 1. To what extent did the project achieve the planned objectives and result in HIV prevention among vulnerable and most-at-risk populations? A summary of ZPI’s progress toward results to date under each intervention lens and activities is provided in Table 1. Overall, ZPI has achieved seven out of its eleven targets. The LOP targets were achieved through community-led behavioral mobilization HIV prevention activities implemented in nine intervention provinces. Furthermore, the adoption of the small grants process and community purchase orders (CPOs) increased the number of organizations implementing various facets of the lenses within provinces. This approach contributed to increased geographic coverage and reach of the targeted populations. Furthermore, with regard to economic empowerment (EE) interventions, the approach received an overwhelming response as the project implementation progressed with its community-based HIV prevention efforts. There was higher demand across the intervention provinces for EE activities than previously anticipated. 17 A. HIV KNOWLEDGE, TESTING AND RECEIVING RESULTS 1. Summary Of Key Activities ZPI has been implementing community-based HIV prevention education, including the promotion of HTC for individuals and couples through door-to-door (i.e., household-to-household) outreach, and PMTCT approaches. ZPI’s HIV prevention interventions have been guided by the concept of interrelated ‘risk and vulnerability lenses’. ZPI has implemented these lenses in an interconnected and interwoven manner with no one sub partner or sub-grantee implementing just one lens. ZPI’s community-based volunteers (CBVs) have been trained to educate and promote HIV prevention services in selected communities through the lenses approach, as well as to effectively and efficiently refer potential beneficiaries to different HIV prevention and treatment service providers within the intervention communities, districts and provinces. Using traditional community socio-cultural structures, including neighborhood health committees (NHCs), and groups set up to mitigate specific health and social behavioral risks and vulnerabilities, CBVs undertook the promotion and management of community-led HIV prevention, including the promotion and provision of the HTC services. Community-based HTC outreach services were done in collaboration and support of the provincial and district-level Ministry of Health (MOH), and NAC structures, such as health facilities; the District Health Office (DHO); District AIDS Task Forces (DATFs); and Provincial AIDS Task Forces (PATFs). The MOH, through its health facilities, provided the HIV testing services as part of meeting the demand that was generated by the community‘s various implementing partners’ (IPs) care and support programs including ZPI. 18 Table 1: Summary of ZPI’s performance towards attaining results and objectives: indicator performance against targets Indicato r # Indicator (Years tracked) LOP Targets Actual Achieved (Number ) % Achieved Against Target Achievem ent of Indicator Explanation for Indicator Performance 1 Number of the targeted population reached with individual and/or small group level preventive interventions that are based on evidence and/or meet the minimum standards required (Years 1- 4) 572,00 0 625,410 109% Achieved The LOP target was achieved because of the expansion in community mobilizations activities. Small grants process and CPOs also helped toward achieving the target. 2 Number of the targeted population reached with individual and/or small group level preventive interventions that are primarily focused on abstinence and/or being faithful, and are based on evidence and/or meet the minimum standards required (Years 1-3) 174,00 0 143,375 82% Not Achieved The LOP target could not be 100% achieved because the indicator was retired by PEPFAR end of Year 3 3 Number of key populations reached with individual and/or small group level HIV preventive interventions that are based on evidence and/or meet the minimum standards required (Year 4 only) 2,000 675 34% Not Achieved This indicator was introduced in Year 4. 34% achievement represents data for six months (Jan-June 2014). The target is expected to be reached before the end of 2014. 4 Number of individuals who received Testing and Counseling (T&C) services for HIV and received their test results. (Years 1- 4) 125,00 0 216,408 173% Achieved The LOP target was achieved because of the expansion in community mobilizations activities lead to more people accessing (T&C) services. The other reason is that the project focus was not (T&C) services hence the under estimation in setting the original target number. 5 Number of pregnant women with known HIV status (includes women who were tested for HIV and received their results) (Years 1-3) 27,500 9,796 36% Discontin ued The indicator was discontinued 6 Number of People Living with HIV/AIDS (PLHIV) reached with a minimum package of Prevention with PLHIV (PwP) interventions (Years 1-3) 17,000 20,090 118% Achieved The LOP target was achieved because of the expansion in community mobilizations activities. Small grants process and CPOs also helped in achieving the target. 7 Number of MARP reached with individual and/or small group level interventions, based on evidence and/or meet the minimum standards required. (Years 1-3) 75,000 113,374 151% Achieved The LOP target was achieved because of the expansion in community mobilizations activities. Small grants process and CPOs also helped in reaching the target. 19 Indicato r # Indicator (Years tracked) LOP Targets Actual Achieved (Number ) % Achieved Against Target Achievem ent of Indicator Explanation for Indicator Performance 8 Gender Based Violence and Coercion: Number of people reached by an individual, small group or community-level intervention or service that explicitly addresses gender-based violence and coercion related to HIV/AIDS. (Years 1-3) 92,500 96,500 104% Achieved The LOP target was achieved because of the expansion in community mobilizations activities. Small grants process and CPOs also helped in reaching the target. 9 Gender Norms within the Context of HIV/AIDS: Number of people completing an intervention pertaining to gender norms, that meets minimum criteria (Year 4 only) 2,900 2,972 102% Achieved This indicator was achieved by using community and small group level interventions via established groups such as men's and GROW groups as well as Young Women Christian Associations (YWCA) 10 C5.7.D - Number of eligible adults and children provided with economic strengthening services ((Years 1- 4) 10,500 15,849 150% Achieved The Economic Empowerment (EE) activities received an overwhelming response as the project progressed. There was therefore higher demand across the intervention provinces for EE activities than previously estimated. 11 Number of community health and para-social workers who successfully completed a pre-service training program (Years 1-3) 30,000 10,171 34% Not Achieved The indicator was retired by PEPFAR in Year 3 hence we were not able to measure the achievement on this indicator through comparing the life of the project targets and what has been achieved Source: PMP Routine Data, ZPI Project 2014 20 2. Key Findings and Analysis The evaluation’s analysis triangulated the qualitative and quantitative data to respond to the first evaluation question. The qualitative data gave a unique perspective into the project’s performance, and also collaborated findings of the project’s surveys with routine M&E data. Quantitative findings of the baseline and midline household surveys1 indicate that across the four provinces evaluated, namely Copperbelt, Eastern, Luapula and Western (four of the six provinces included in the evaluation sample), the proportion with comprehensive knowledge increased significantly from baseline to midline (males: 47 percent to 53 percent; females: 35 percent to 45 percent). However, comprehensive HIV knowledge remained low in Western at midline (19– 22 percent) (ZPI, Midline Household Survey 2013 Report). ZPI aimed to reach at least 572,000 clients from the targeted populations with individual and/or small group level HIV preventive interventions that are evidence-based and/or meet minimum standards required (Indicator #1; Table 1). Overall, the findings indicate that the target has been met. The PMP data shows that the project has reached 625,410 clients, (315,287 males and 310,123 females), (109 percent of its life of project target; see Table 1 and Figure 1) with community-led behavioral HIV preventive interventions. The lower numbers observed in Year 4 (see Figure 1) are due to the scaling down of interventions and programs at the end of Year 3. For example, ZPI’s project consortium partners such as ZHECT and CHAMP were discontinued at the end of Year 3. It is also important to note that Year 4 data does not cover a complete year (12 months) but only covers data for the period of the six months (January to June) prior to the EOP evaluation. Provincial variations in this performance indicator are also observed (Indicator #1). Cumulatively, Lusaka has the highest number of individuals reached with prevention interventions (170,374), followed by Northwest (94,754), and Southern (93,319) provinces, with the Northern (17, 554) and Western (93,319) province having the least number of individuals reached with community-led behavioral HIV preventive interventions (see Annex VII: Provincial Data Analysis Charts). ZPI began implementing community-led behavioral HIV prevention interventions in the Northern and Western provinces in 2013 (i.e., phase three of rollout of the program based on the project design), while the Copperbelt, Eastern, Lusaka, Northwestern, and Southern provinces began implementing community-led behavioral HIV prevention interventions in the first phase of the program (i.e., 2011). Central and Luapula provinces began implementing their community-led behavioral HIV prevention interventions in 2012 (i.e., phase two of rollout of the program based on the project design). 1 The evaluation team conducted secondary analysis on PMP, baseline and mid-line survey data to observe trends in performance of indicators. The Population Council (PC) conducted the baseline and midline surveys while routine PMP data was collected and managed by FHI-360. 21 Figure 1. Number of targeted population reached with individual and/or small group level preventive interventions that are based on evidence and/or meet the minimum standards required (Indicator #1) Source: PMP Routine Data, ZPI Project 2014 The above findings are corroborated by the FGD data. Beneficiaries and volunteers in almost all FGDs (83.3 percent; 30/36 FGDs) and KIIs (82.5 percent; 33/37 community KIIs) demonstrated high HIV and/or AIDS risk and prevention knowledge. Target beneficiaries (i.e., women and men of reproductive aged between 15- 49; in-school and out-of-school youth aged 10-24) in rural and urban FGDs also confirmed high health seeking behaviors. As a female in a PMTCT FGD in Choma District in Southern Province reported: “before the project people had no knowledge about HIV and HIV prevention…. people are now accepting to ‘stick’ to one partner and stigma and discrimination is no longer there.” The beneficiaries also reported that, due to the sensitization by the CBVs, individuals were increasingly willing to use the various available HIV/AIDS services in the communities and health facilities, including mobile facilities offering testing and counseling, and medical male circumcision. The main channels for HIV prevention messages for community were service providers of local health facilities, and sub-grantees and sub-partners CBVs, who conducted individual and/or small group level HIV prevention education. One of ZPI’s aims was to provide a targeted population with individual and/or small group level HIV preventive interventions that are primarily focused on abstinence and/or being faithful, and are based on evidence, and/or meet minimum standards requirements. A total of 174,000 clients were targeted for the community-led behavioral HIV prevention interventions (Indicator # 2; Table 1). The project has reached 143,375 clients (82 percent of its life of the project target) with abstinence and/or being faithful prevention interventions (73,643 males and 69,732 females). The findings indicate that the target has been met. The pattern of achievement for this intervention is similar to the one observed for the prevention interventions targeting the general population (Indicator #1). Well-structured community-based youth HIV prevention programs focusing on 85,556 124,342 256,828 119,126 625,410 572,000 0 100,000 200,000 300,000 400,000 500,000 600,000 700,000 Year 1 Year 2 Year 3 Year 4 Total Reach LOP Target Target/Reach P8.1.D- Number of the targeted population reached with individual and/or small group level preventive interventions that are based on evidence and/or meet the minimum standards required 109% of LOP target 22 abstinence and/or being faithful were mainly implemented in Lusaka province. Youth programs in the rural provinces were still at the early stage of development with respect to program design, implementation and geographic coverage. However, wherever youth programs were being implemented, the targeted population reported benefiting from the programs and activities, as a male youth participant indicated in an FGD in Chipata District in Eastern Province: “as youth we have been sensitized on HIV prevention through condom use and are free to use condoms. We also go round distributing condoms and have learnt how we can protect ourselves from STIs and unwanted pregnancy.” KIIs with traditional leaders, and FGDs with both young men and women, indicate that youth HIV prevention services are lacking in rural areas and young people lack the information and services they need to prevent HIV infection. As one female KII project partner said: “The role of the project was to create demand for the clinical services. However we had a challenge when we created demand then the health Centre staff would tell us that they don’t have enough HIV test kits… when it came to provision of male circumcision, again you find that you create a lot of demand for male circumcision, but male circumcision providers were not there. We used to meet with MOH to discuss this but the cause of the shortages was not known since the medical stores in Lusaka reported availability of the kits while the MOH staff would say these medical stores does not supply us with the commodity.” The majority of respondents (both men and women, including community and traditional leaders) during FGDs indicated their full support of youth HIV prevention programs, including the reduction of early pregnancy and marriage, as well as the reduction of alcohol and drug abuse. As a female youth FGD participant in Chipata District in Eastern Province said: “[peer educators] went to schools to teach fellow youth to guard themselves from [exploitation] by adults. They taught them on early marriages, abstinence, condom use, and substance abuse [prevention]. Mainly looking at the dangers…” B. HIV testing and receiving results Results of the baseline and midline household surveys indicate that HIV testing increased significantly from baseline to midline for both males and females. Only Eastern, (males: 65 percent to 82 percent; females: 76 percent to 85 percent) and Western (males: 40 percent to 71 percent; females: 63 percent to 89 percent) provinces showed significant differences in testing in both males and females. Additionally, males who were exposed to HIV prevention interventions were more likely to have been tested, compared to those who were not exposed (72 percent versus 54 percent: p<0.01) (see Figure 2). The midline study showed that there was no difference by exposure among females (ZPI, Midline Household Survey 2013 Report). The comparison of survey findings is limited to only the baseline and midline household surveys. 23 Figure 2. Percent of males and females ever tested for HIV and received results at midline, by exposure Source: ZPI Project Midline (2013) Household Surveys ZPI’s target was to provide community-based HIV counseling and testing services (HTC) to 125,000 clients (Indicator # 4). Based on ZPI 4th quarter PMP report, overall, the project reached 216,408 (173 percent of its life-of-project target; see Table 1) clients with HTC services (101,452 males and 114,956 females) disaggregation is based on the 3rd quarter ZPI PMP report. Trend analysis shows the number of individuals receiving HTC services increased five-fold from 24,374 in 2011 to 117,896 in 2013. The findings indicate the target has been met. HTC varied across the provinces. Luapula province had the highest number of individuals (47,432) counseled and tested for HIV, followed by Northwest (44,582) and Lusaka (39,461), followed by Southern (25,886) and Eastern (25,891) provinces (see Annex VII). The Northern (5,482) and Western (1,624) provinces recorded the lowest performance on this indicator, similar to other indicators. ZPI began implementing HIV prevention interventions in the Northern and Western provinces in 2013 (i.e., phase three of rollout of the program based on the project design). As stated above, these two provinces had a shorter period of implementing the community-led behavioral and structural HIV prevention interventions compared to the provinces which began implementing the HIV prevention interventions during the first phase of the program (i.e., in 2011; Copperbelt, Eastern, Lusaka, Northwestern, and Southern provinces), and second phase of the program (i.e., 2012; Central and Luapula provinces). The PMP data are corroborated by the FGDs and KIIs findings. Target beneficiaries, including traditional leaders and volunteers in rural and urban FGDs and KIIs, indicated that ZPI conducted gender-sensitive, individual and couple HIV counseling and testing promotion. Subsequently, awareness and acceptance of counseling and testing is very high. Therefore, increased awareness of counseling and testing might be due to the effects of ZPI’s activities in educating people about and making referrals for HTC services (i.e., the door￾to-door HCT and couple testing through PMTCT approaches). “Counseling and testing [including] couple counseling were a challenge because in our culture, men have the say. If men feel left out things won’t go anywhere. And, there were some wives that didn’t want to be tested with their husbands. We gradually convinced them.” (Male Headman interviewed in Southern Province). Throughout the sampled provinces and districts, qualitative findings also indicate that community door-to-door counseling and testing was acceptable and adopted. As a KII with a female Nurse-in-Charge in Chipata District revealed: “The door-to-door testing is acceptable by the people. It lessened the burden of people coming to the health Centre for such a service and also it reduced the burden to people not to wait for a long time at the facility for counseling and testing especially on days when we are busy with other clients.” 54 67 72 76 0 20 40 60 80 Males Females Percentage Unexposed Exposed 24 Qualitative findings also show that traditional leaders such as chiefs and headmen served as role models for community-based HIV prevention. ZPI strategically engaged and increased the involvement of members of this influential group in the design and implementation of community-based HIV prevention efforts. Traditional leaders, especially chiefs and headmen, recognized as the community gatekeepers by ZPI, were the first individuals contacted when the HIV prevention interventions were introduced in selected communities. Most of these leaders agreed to be tested for HIV and spoke openly about their experiences and encouraged their community members to be tested. “Since the project started, Chiefs meet with their people every Friday and teach them about these issues, distribute condoms, teach people ‘how they can behave in society.' We discuss so our people will pay attention. We talk about using condoms to protect themselves, and they do use them, [we discuss that] men [should be] treating women with respect, understanding that GBV is the law, that women have rights and are human beings, and encourage people to send their daughters to schools. We work with the police who are now coming to the villages, and women go to Victims Service Units to report violence and rape.” (Male Chief interviewed in Western Province). Men, women and youth were aware of the importance of knowing their status so that they can protect themselves or get treatment, since antiretroviral therapy (ART) is now more available and accessible. As a male community leader said “since the project [was introduced in my community] everyone wants to be tested for HIV, before it was not possible to convince people to go for testing, people would feel uncomfortable.” (Chief interviewed in Choma District, Southern Province). Furthermore, as a male youth said during a FGD in Chipata District in Eastern Province: “testing and counselling worked well. Many youths were free to test. Door-to￾door HTC sensitization helped a lot to sensitize people [toward testing and counselling].” Both men and women in FGDs indicated that they believe it is important to know their status, so that they can take the appropriate measures and avoid transmission to their new-born children. One married woman in a PMTCT FGD in Choma District in Southern Province said: “there is now openness in going for voluntary counseling and testing.” However, FGDs held with both male and female youth revealed that they lacked the appropriate information and strategic social support to seek counseling and testing services in particular. While some respondents revealed a fear of knowing their status and stigma around HIV, most indicated that due to outreach and education activities on the importance of counseling and testing, people in their communities felt positive about the need for the services. The qualitative data indicated that testing of pregnant women via PMTCT service was mandatory in few communities. This mandatory clause is obligated by community by-laws, enacted by chiefs and enforced by community headmen and health service providers in these communities. The community by-laws mandated that husbands accompany their wives for their first antenatal care (ANC) visit, and also undergo HTC with their spouses. As one chief said during a KII in Samfya District in Luapula Province: “To encourage mothers to attend clinics, it is mandatory for every pregnant mother to attend ANC sessions, to test for HIV infection accompanied by their husband, and also for every birth to be delivered at the clinic.” CBV sensitized men on PMTCT and gender￾related issues and encouraged them to voluntarily participate in the healthcare of their pregnant wives. As one woman in a PMTCT FGD in Choma District in Southern Province said: “men and women go to antenatal clinics together. Husbands are more caring to their pregnant wives. They do not abuse them or overload them with work.” Counseling and testing is widely available through referrals to static sites, both private and public health facilities. Mobile HTC is also provided through private and public health facilities to reach rural communities. KIIs with nurses-in-charge, and with leaders and program officers of civil society organizations (CSOs) across the selected provinces attested to the availability of HIV testing kits to meet the demand created by the CBVs. One service provider in a KII in Samfya District in Luapula Province said: “from time to time, DHOs supplied health facilities with HIV test kits and reagents when shortages were experienced.” However, limited human resources and transport for mobile HTC pose challenges to reaching rural areas without access to static sites. 25 C. Medical male circumcision (MC) The use of medical male circumcision as a preventive measure against HIV infection increased significantly between the baseline and midline surveys in all provinces except Eastern, where it remained at six percent (ZPI, Midline Household 2013 Report). The traditional male circumcision practice is prevalent in Eastern province. Traditional leaders and cultural gatekeepers have resisted the replacement of traditional male circumcision with medical male circumcision. They argue that medical male circumcision will contribute to the destruction of their traditional practices of male and female youth rites of passage into manhood and womanhood (these two initiation practices are intricately related). The household surveys showed that medical male circumcision rates varied greatly across provinces. The Western province had the highest circumcision prevalence (63 percent at midline); the greatest increase from baseline to midline was seen in Luapula, where circumcision prevalence went from five percent to 33 percent (see Figure 3). Figure 3. Percent of males reporting that they are circumcised at baseline and midline household surveys Source: ZPI project Baseline (2011) and Midline (2013) Household Surveys ZPI and its partners have undertaken community-based behavioral change communications (BCC) campaigns and door-to-door HTC outreach to increase awareness about the benefits of male circumcision. The involvement of community and traditional leaders in the promotion of medical male circumcision has encouraged men in their communities to seek out these services, particularly where circumcision is not traditionally practiced or culturally accepted. Furthermore, after community sensitization and education on the benefits of male circumcision by CBVs, male community members were referred to health facilities that provided such services or mobile units that came at designated schedules and health facilities. Qualitative data shows mixed responses with respect to the availability of MC services at both static and mobile facilities. Some community members indicated that mobile units did not always turn up when they said they would, indicating that even when demand exists, services may not be readily available. Some 33% of FGD respondents reported that where these services were available, the distance to facilities was a barrier to accessing them. In addition, respondents felt that few referral services were available in case of male circumcision-related complications that developed after surgery. “… One problem is the mobile clinic comes only once a month and if there is a problem after a man is circumcised, we don’t have a way to take care of them.” (Headman in Southern Province). Qualitative findings, collaborating the quantitative survey data, indicated significant acceptance of male circumcision as a preventive measure against HIV infection across all the evaluated provinces. Medical male circumcision was discussed in 72 percent of the FGDs (26/37) conducted during the evaluation. A vast 14 6 5 47 18 40 6 33 63 39 0 20 40 60 80 100 Copperbelt*** Eastern Luapula*** Western** All provinces*** Percentage Baseline Midline 26 majority of respondents, both community members and community and traditional leaders, reported that male circumcision (referred to as mudulidwe in the local language) was positively correlated with the prevention of HIV infection. Furthermore, some respondents reported that young boys were now being circumcised, with male circumcision becoming an accepted practice and custom in their communities. In Chipata District, Eastern Province, a FGD with headmen led to the following summation by one of the male participants: “male circumcision [“mudulidwe”] is very important and is helping us the elderly and young people in HIV prevention. We are happy about this program and it should be continued.” Also attesting to the acceptability and adoption of the medical male circumcision, a key informant reported, “Men, young and old, are freely being circumcised. One thousand-plus men have been circumcised in my chiefdom.” (Male Chief in Choma District, Southern Province). ZPI staff in Eastern province reported that male circumcision is not a traditional practice in some communities, for example, among the Ngoni people in Eastern Province. This affected the initial uptake of MC services. However, the BCC campaigns and involvement of community and traditional leaders in promoting male circumcision began to wear down resistance toward accepting medical male circumcision. “In 2011 there was resistance to male circumcision but after a year or so, things changed.” (Headman interviewed in Southern Province). According to respondents, medical male circumcision has become more accepted among the younger than the older age groups, with older men saying they felt it was too late to get circumcised, that they would have little benefit from it, and/or take too long to heal. Parents want their male children to get circumcised, especially during school holidays, making demand for medical male circumcision very high during those periods. D. Alcohol and other drug abuse ZPI’s intervention provinces experienced varied alcohol problems. During the baseline household survey, clinical alcohol problems were observed to be highest among males in Copperbelt and Eastern provinces. Improvements were observed in the Copperbelt province during the midline survey, while in the Eastern province alcohol problems remained extremely high (ZPI, Midline Household Survey 2013 Report). ZPI created community structures and interventions, such as the community alcohol teams (CATs) and psycho￾education programs (PEGs), to tackle growing alcohol abuse, perceived by the community as a social problem. The teams and groups were able to mitigate the contributory influence of alcohol and other drug abuse on HIV infection. As shown by the PMP data, ZPI helped create 278 community alcohol teams with 8,474 members across the nine intervention provinces. Alcohol and other drug abuse was discussed and mentioned as a growing problem in 86 percent of the FGDs (31/36) conducted during the evaluation. However, 42 percent of these FGDs reported that men and women had begun to cut down on alcohol and other drug abuse. According to Serenity Harm Reduction Programme Zambia (SHARPZ), a ZPI consortium partner, the CAT and PEG teams provided individual and group counseling support to community members, both adult males and females. The support included promoting delayed and reduced alcohol consumption among adolescents and youths aged between 15-24, including among adult men and women. “Response on alcohol and drug abuse [among the youth] has been good. [There has been a] lot are thinking [and action] about how they can change their behavior, (e.g. to reduce smoking or drinking)” (FGD with youth coaches in Lusaka District in Lusaka Province). In almost all FGDs and KIIs, respondents reported the link between alcohol abuse and gender-based violence, including its association with highly risky sexual behaviors. As a comment from a male youth participant in a FGD in Chipata District in Eastern Province demonstrated: “excessive drinking also put youth at risk of contracting 27 HIV.” Therefore, in some provinces such as Eastern and Southern, CATs and PEGs also included youth as members, in addition to targeting youth with community-based behavioral interventions. Given the connections between substance abuse and GBV and HIV infection, CAT and PEG community volunteers were trained to concurrently support anti-alcohol abuse, promote GBV prevention, and promote HIV prevention among both male and female community members including youth. As one female PEG beneficiary in Chipata District in Eastern Province said: “Before they came to teach us [anti￾alcohol abuse] there was a lot of GBV in homes, we never had peace. Our partners were drinking a lot and we were being beaten, and development was lacking. So when they came to teach us [anti-substance abuse] there is development in the homes now, even GBV has somehow reduced.” Furthermore, community and traditional leaders interviewed in different chiefdoms acknowledged that alcohol abuse was becoming a danger to society and took steps to restrict its sale and consumption. Some traditional and community leaders gave up drinking alcohol to be good role models in their communities. For example, a chief in the Southern province publicly announced that he would stop drinking beer, which he subsequently did. Also, as a chief during a KII in Samfya District in Luapula Province said: “to reduce drunkenness in the community, we have mandated that [beer] bars be open at 10am and close at 10pm.” Furthermore, punitive measures were put in place for individuals who were revealed to be upsetting the social order when drunk. For example, in Samfya District in Luapula Province, these people were asked to pay a goat or chicken(s) to the community leaders. In most of the FGDs and KIIs, respondents reported that the alcohol and substance abuse measures adopted by the communities have begun to contribute to the reduction of alcohol abuse and smoking of marijuana (the most populous drug being abused in Zambia) among both adult men and youth. Zambia is the world’s third largest marijuana, weed-smoking country by population (World Drug Report 2014)2. As a woman in a PMTCT FGD in Choma District in Southern Province said: “people have accepted to reduce taking alcohol and men do not squander money on alcohol anymore.” E. Gender-based violence Since 2010, ZPI has been promoting community-based change in gender norms and GBV prevention to decrease women’s vulnerability to HIV. ZPI assisted its target villages to form anti-GBV committees to address GBV at the chiefdom level; these structures were linked to the district and provincial anti-GBV committees. A total of 28 village anti-GBV committees with 361 members in seven provinces were formed by the project. The anti-GBV committees were established with trained community chiefs, village headmen, and other resource persons in communities who sensitized and mobilized people on GBV prevention and were informed on the different types of GBVs, including social, economic, physical and sexual abuse. “Traditional leaders become facilitators and volunteers on community mobilization on anti-GBV activities. The community members who were gure wamkhulu [persons involved in male youth rite of passage ceremonies] became facilitators together with the headmen, and the alongozi [persons involved in female youth rite of passage ceremonies] also became facilitators.” (Male Key Informant interviewed in Katete District, Eastern Province). Additionally, in recent years, Zambia has made progress in protecting women and young girls at a national level. In 2011, illustrating the Zambian government’s long-standing commitment to responding to sexual and gender-based violence, the Anti-GBV Act was passed. Furthermore, communities were informed about the Anti-GBV Act and one of its important provisions, the importance of writing a will that includes both women 2 United Nations Office on Drugs and Crime, World Drug Report 2014 (United Nations Publication, Sales No. E.14.XI.7). 28 and men as beneficiaries. Some community members were trained on some aspects of the Act to function as paralegals. These individuals sensitized their communities on laws related to GBV. “The GBV trainings and community sensitization activities have contributed to increased knowledge on GBV and associated factors among stakeholders.” (FGD participant, Chipata District, Eastern Province) In addition, in 2012, the Gender and Development Division under the Office of the President released national guidelines for the management of violence cases. The guidelines integrate medical, legal, and psychosocial responses to GBV. Multi-sectorial GBV committees—at the national and decentralized levels—facilitated the formation and resuscitation of anti-GBV committees (where they had not been formed or had become dormant). ZPI aimed to provide individual, small group, or community level interventions or services that specifically addressed GBV and coercion related to HIV/AIDS to about 92,500 clients (Indicator # 8; Table 1). Overall, the project cumulatively reached 96,500 (104 percent of its life of project target; see Figure 4) clients with GBV prevention messages and/or services (43,425 males and 53,075 females). Figure 4 shows that in Year 4, the indicator was not tracked since it was discontinued by PEPFAR in year 3. Trend analysis of the PMP data shows the number of individuals reached with GBV prevention information and services increased just below five-fold from 12,723 in 2010 to 62,804 in 2013. The findings indicate the target has been met. Furthermore, the GBV prevention reach varied across the targeted nine provinces. Eastern Province had the highest number of individuals (22,696) reached with GBV prevention, followed by Southern (21,966) and Northwest (15,238), and then Lusaka (13,013) and Copperbelt (12,767) provinces (see Annex VII). Once again, the Northern (335) and Western (74) provinces recorded the lowest performance on this indicator, as with other indicators. The reason for this low performance indicator remains the same as the one stated above. However, the number of people completing an intervention pertaining to gender norms that meet the minimum criteria reached 2,972 (102 percent of its life of project target; see Table 1). The findings indicate the target will be surpassed significantly, as the indicator is being tracked for the following six months. Figure 4. Number of people reached by an individual, small group or community-level intervention or service that specifically addressed GBV and coercion related to HIV/AIDS (Indicator # 8) Source: PMP Routine Data, ZPI Project 2014 FGDs with beneficiaries and KIIs reported that the anti-GBV committees and actions taken thereof by the anti-GBV community volunteers, made more people recognize acts of GBV and report them through the community courts, including civil and criminal courts. GBV prevention was discussed and mentioned in 72 percent of the FGDs (26/36) and 67 percent of KIIs conducted during the evaluation. Both the beneficiaries 12,723 20,973 62,804 0 96,500 92,500 0 20,000 40,000 60,000 80,000 100,000 120,000 Year 1 Year 2 Year 3 Year 4 Total Reach LOP Target People Reached 104% of LOP target achieved 29 and volunteers, including community and traditional leaders, recognized that this intervention was interwoven and an integral part of most of the HIV prevention program implemented by the project. Based on the qualitative findings of couples that indicated experiencing GBV, many have started opening up to the problems they are facing in their marriages. As one Village Headman said during a KII: “violence has reduced in the village, insults have [been] reduced and fights in marriage have reduced. Even now when we teach people [they] see the benefits.” More men were mobilized by sensitizing them about the dangers of GBV and, more importantly, men became the drivers of GBV prevention in most communities visited. Community members who have experienced any form of GBV have begun to seek help and support from anti-GBV committees, the Police Victim Support Unit (VSU), and other partners dealing with domestic violence, such as the USAID-funded World Vision One-Stop Centers in Chipata District, Luapula Province. Several benefits were also listed as a result of the anti-GBV and gender norm efforts “perceived effects of the intervention included - reduction in early marriages, reduction in early pregnancy among adolescents and youth; reduction in rape cases or incessant. Communities have moved away from traditional punishment for incessant or rape to reporting such cases the police. [Further] to us the GBV program was the best since we engaged the traditional leaders and GBV was considered as a non-go area.” (Male Key Informant, Katete District, Eastern Province) F. Economic Empowerment It is widely known that economic disparities between women and men in high-prevalence settings increases women’s vulnerability to HIV through various channels, such as increased high-risk behaviors, vulnerability to GBV, limitations on women’s ability to negotiate safer sex, and increased dependence on transactional sex. There is evidence that programs promoting economic empowerment combined with HIV prevention offer strategic opportunities to get people involved in HIV prevention programs and may contribute to reductions in HIV risk behaviors (Kim et al. 2008, Pronyk et al. 2008). A key part of ZPI’s mandate was to economically empower Zambian women, while also recognizing that economic approaches alone will not necessarily decrease the risk of HIV. Therefore ZPI adopted an economic model called Grass Roots building Our Wealth (GROW). The GROW model was used as the platform for the formation of community-based micro-saving and/or micro-loan groups. The GROW approach links economic empowerment activities with community￾led HIV prevention. The PMP data showed that through the GROW program, 468 GROW groups for economic empowerment were created with a membership of 4,977 (4,198 females and 802 males). In Lusaka province, 19 GROW Girls Groups3 with 339 members had been created by August 2014. The GROW groups together have savings of 532,487.00 Kwacha (approximately US$87,008.00) and have provided 1,064,666.00 Kwacha (approximately USD$74,079) in loans. Both the quantitative and qualitative findings showed that micro-saving and micro-loaning activities appear to be having a positive impact on people’s earning capacity as well as encouraging women to rely on income they are generating through GROW activities. According to the midline household survey, those who were exposed to income-generating activities were significantly more likely to be earning money compared to those not exposed (ZPI, Midline Household Survey 2013 Report). This was most pronounced in Eastern province, where those exposed were two times more likely to be earning money compared to those not exposed. The Eastern province is where ZPI’s GROW program has been the most active, attracting 2,025 members, or 40.7 percent of all GROW members (2,025/4,977). The exposure effect seen in the midline household survey findings is supported by the PMP data, as well as the qualitative findings. By July 2014, the GROW program reached 15,849 adults and children with economic strengthening services (151 percent of the life-of-project target; Indicator # 10, Table 1; see Figure 5). Trend analysis using the PMP data shows the number of eligible 3 In Lusaka, as from 2012, ZPI partnered with the Olympic Youth Development Centre (OYDC). The partnership was mainly focused on working with girls (ages 15-24 years old) empowering them with HIV prevention knowledge, lifesaving skills and positive and progressive gender norms. OYDC mobilized 370 girls. Sexuality and life skills groups have up to 25 members. Furthermore, the girls formed money saving groups. 15-20 girls participate in each savings group; they choose their own leadership, they register activities in books, and agree on mandatory saving amount (e.g. K2/weekly). During their meetings they cover both savings and HIV related activities. Currently, 10 groups at operating from the OYDC and 4 are school-based saving groups. 30 adults and children provided with economic strengthening services increased about six-fold from 1,308 in 2010 to 7,243 in 2013, although the number reached dropped to 4.445 in 2014. Although the target has been achieved, there is an observed drop in people reached in Year 4. The drop in the performance indicator in Year 4 is due to the scaling down of activities by ZPI. For example, at the end of Year 3, ZHECT and CHAMP consortium partnership with ZPI was discontinued. Also, Year 4 data does not cover a complete year (12 months) but only covers data for the period of six months (January to June) prior to the EOP evaluation. Additionally, the economic strengthening services intervention achievement varied across the targeted nine provinces. Eastern Province had the highest number of adults and children (4,322) provided with economic strengthening services, followed by Northern (3,418) and Lusaka (2,700), and then Copperbelt (1,774) and Luapula (1,273) provinces (see Annex VII). Once again, the Western (79) province recorded the lowest performance indicator as per other indicators. The explanation for differential provincial indicator performance remains the same as mentioned above. Qualitative findings corroborated the midline household survey and the PMP data. Economic empowerment was discussed and mentioned in 61 percent of the FGDs (21/36) conducted during the evaluation. In Lusaka, the GROW Girl program is viewed as an empowerment platform for young girls aged 15-24, with some of them currently enrolled in schools. As a KII with a GROW Girls Group participant in Lusaka Province demonstrated: “The project has been very helpful in the lives of young people. It may not be visible, we can’t see it physically, but inside they have a new spirit. They have different approaches to the way they used to think in their lives, which will indirectly and directly impact on their communities, on their education and in many other ways in their lives. So the benefits of this project are huge and life-changing.” Furthermore, a female participant in a FGD in Katete District, Eastern Province reported, “ZPI trained us to save money through our groups so that we reduce poverty. This would help us take our children to school. As a result [we] would reduce on promiscuity and prevent HIV infections.” Furthermore, across all the provinces visited during evaluation, most FGD participants reported that members of GROW groups had benefited from the micro-saving and/or micro-lending approach adopted by the groups. For example, female FGD participants in Kitwe District in the Copperbelt Province, mentioned that some GROW members were able to (i) finish building their houses and pay individuals contracted to build the houses using monies loaned from the group; (ii) furnish their homes with sofas and one woman mentioned having bought a mattress with the money she loaned from the group; and (iii) use the money to pay school fees for their children. “My husband does not work so I use the money to do business and support my family” (Female FGD participant, Kitwe District, Copperbelt Province). Additionally, monies loaned from the GROW groups were used to fund small businesses by individual members of the groups: “I have bought two sewing machines with the money I loaned. I want to buy another one” (Female FGD participant, Kitwe District, Copperbelt Province). 31 Figure 5. Number of eligible adults and children provided with economic strengthening services (Indicator # 10) Source: PMP Routine Data, ZPI Project 2014 In summary, the triangulated end-of project qualitative data substantiated most of the findings of the quantitative data derived from the baseline and midline surveys as well as the trends observed from the routine PMP data. Furthermore, the qualitative data obtained from the FGDs and KIIs with beneficiaries and program staff including community grantee and community leaders gave a unique perspective into the project’s performance. 1,308 2,853 7,243 4,445 15,849 10,500 0 2,000 4,000 6,000 8,000 10,000 12,000 14,000 16,000 18,000 Year 1 Year 2 Year 3 Year 4 Total Reach LOP Target People Reached 151% of LOP target achieved 32 Question 2. To what extent were the project design, implementation, and management effective and why? A. Best practices used during project design, implementation and management. The evaluation documented creative and constructive actions undertaken by ZPI and its partners, showing the potential to improve health outcomes for ‘at-risk and vulnerable populations,’ and are worth replicating in current and future programming of this nature. The following are key to ZPI’s success: a. Design The ZPI project Lenses Approach: ZPI’s activities were guided by nine independent and interconnected ‘risk and vulnerability lenses,’ which the project defined as factors that increase an individual’s vulnerability to HIV transmission, in addition to being prevalent issues in the lives of Persons Living with HIV (ZPI project Work plan, 2012). These lenses are: most-at-risk behaviors; gender-based violence; economic vulnerability; alcohol and other drug abuse; mental health issues; disability; violence against children; low uptake of biomedical services; and stigma and discrimination. This multifaceted approach to address ‘the lenses’ enabled the project to effectively tackle HIV prevention from several angles in a manner that addressed many community social and health problems. The ‘lenses’ were found to have been inter-related and very well integrated, offering a broad range of approaches that met diverse community needs. ZPI used the Stages of Change Model that encompasses both cognitive and behavioural approaches to develop its behavior change framework. The model was selected because it permitted the project to focus on individual change, such as condom use, with the understanding that the effects move beyond individual change and include changing socio-cultural norms, such as gender norms and rites of passage to adulthood, in target communities. ZPI’s community mobilization methodologies used this concept of change as a process to design the ‘lens’ interventions. All ZPI’s technical strategies incorporate a minimum package of activities that promotes this process. The project based its steps of change on the participatory learning action (PLA) methodology, using different tools in community discussions for the different stages of change. All ZPI’s lens￾based activities used this process and the tools. Identifying and utilizing each partner’s technical strengths: Each of the major partners brought specific technical expertise that avoided the duplication of effort as much as possible and optimized each organization’s strengths. Table 2 gives an overview of each partner’s role. FHI360, as the lead organization, was responsible for more activities than the other organizations and was involved in all technical areas. There was little duplication among the partners other than with the Comprehensive HIV/AIDS Management Program (CHAMP) and the Zambia Health Education and Communication Trust (ZHECT), which appeared to have had some overlap in targeting workplace programs. Subgrantee organizations expressed satisfaction with technical expertise provided by ZPI’s partners and described it as timely and responsive to their needs. Table 2: ZPI Consortium partners and roles Partner Main Role FHI360 (Family Health International) Overall leadership and management Monitoring and evaluation Community mobilization Economic empowerment activities Private sector engagement in economic empowerment Oversight for all technical activities and materials 33 CMMB (Catholic Medical Missions Board) Technical lead for PMTCT and MC Implement Men Taking Action (MTA) Approach Support to ZPI project Office CHAMP Expand the Global Development Alliance (GDA) Providing of HTC to workers and their families Conducting couple counselling Supporting economic empowerment activities HODI Zambia Lead community mobilization based activities Formation of community-based solidarity groups Facilitating formation of community-based referral systems Facilitating trainings on GBV Grassroots Soccer Engaging young people in HIV prevention Promoting HIV risk reduction behaviours Promoting gender awareness, human and children’s rights Population Council Leading operations research Designing and implementing baseline, midline and end line surveys Leading gender and BCC technical area Afya Mzuri Lead the knowledge management and dissemination Collect, collate and catalogue documentation Facilitating creation of database Facilitating literature reviews ZHECT Work with small and medium enterprises Engaging professional institutions Engaging private sector leadership Facilitating interventions for Zambian affluent Anchoring interventions in communities: The use of sub-grantees, who were community-based organizations (CBOs) or faith-based organizations (FBOs) with local credibility in the communities where the activities were being implemented, proved a good strategy for reaching out to the communities, securing community engagement, and promoting sustainability. This was done in two ways: a) through small grants of USD$25,000 annually and b) through Community Purchase Orders (CPOs) of less than USD$1,000. A total of 14 sub-grants were awarded (three in Northern province, three in Western Province, two in Central province, one in Lusaka province, two in Southern province, and two in Eastern province), amounting to approximately USD$84,000, of the total project budget. This approach enabled the project to engage community members and groups in its activity implementation. CPOs allowed ZPI’s provincial teams to purchase specific community-based services for community groups and/or individuals through a mechanism that was easy to administer and allowed the completion of defined tasks within a short time frame of no more than three months. Thirty-five CPOs were made in the course of the project, totaling approximately USD$25,000 (sixteen CPOs in Lusaka province, nine CPOs in Southern province, eight CPOs in Eastern province, and two CPOs in Central province). In addition, ZPI used participatory community mobilization approaches that allowed the community to engage and be active in the interventions, such as the Participatory Rural Appraisal (PRA) based methodologies (e.g. Regenerated Freirean Literacy through Empowering Community (REFLECT), Men Taking Action (MTA), and Stepping Stones). Each approach supported local ownership of the interventions and created a platform for intervention sustainability. Engagement of key public sector entities. From the outset, the project consulted and worked closely with key public sector entities, such as the Ministry of Community Development, Mother and Child Health (MCDMCH), MOH and NAC. This engagement was strongest with NAC at all levels (national, 34 provincial and district), whereas with the MCDMCH it was stronger at the district level than at the national level. Interaction with the MOH was largely restricted to the national level due to the current set-up of the health sector system, where the MOH is responsible for national and provincial level health care and the MCDMCH is responsible for district and community level care. It is not surprising that connections with the MCDMCH were strongest at the district level since the project was focused on a community-led approach to the HIV prevention interventions. The engagement with these bodies at these levels promoted modest ownership by them in addition to both technical and managerial partnership. Incorporation of a strong gender element into programming. Gender was a common theme that cut across all lenses in the program design. It was operationalized through the training of staff involved in the project at the national, provincial, and district levels. The project used and referred extensively to Zambia’s anti-GBV law. Initially there was no male involvement officer on the project; this position was later created in order to strengthen the gender component. ZPI staff reported how the gender training positively affected them; not only changing their perceptions on gender but affecting their work in their technical role. “Each staff member had to deal with their own risks and vulnerabilities in the training, we learned a lot about our own situations” (Key Informant, ZPI staff). The project also integrated gender into the training of community leaders and volunteers. Once sensitized to gender issues, traditional leaders became gender ‘champions’ and role models. All communities visited during the evaluation reported changes in gender norms, such as men escorting women for antenatal visits, influenced in part by policy change in the health sector encouraging men to do so. Men reported helping with household chores. In FGDs, volunteers and beneficiaries reported a decrease in GBV, and generally reported more understanding of the complexities of gender issues in the communities. “The GBV committee taught about not fighting, that a wife is part of men’s lives, and they are taught to respect and beating has reduced.” (Male Beneficiary, FGD Choma District, Southern Province). The midline study also reported these positive changes in gender norms, although attribution was a challenge. b. Implementation Involving traditional leaders. Traditional leaders were involved in various aspects of the program. Firstly, in communities where ZPI was implemented, they were approached and informed about the project and what it intended to do around specific lenses, and then engaged as champions (i.e., chiefs and headmen, who were a port of entry to communities). This ensured the acceptability of the planned behavioral and structural HIV prevention interventions. Secondly, the community leaders were used as role models in various lenses, such as GBV and alcohol and substance abuse, thereby promoting positive messages and appropriate behaviors by community members. Lastly, in some cases the local traditional leadership, mainly headmen, also served as volunteers for ZPI activities. This provided greater access to volunteers by the community and also contributed to more people being reached, as reported by 19 headmen interviewed in Choma District, Southern Province. Male involvement. In some interventions (i.e., those involving gender norms, couple counseling, and alcohol and substance abuse), male involvement was highly prioritized. The engagement of a Male Involvement Specialist in 2012 showed great commitment to the project’s emphasis on male involvement to challenge harmful gender norms, question traditional norms and encourage positive cultural norms. Men Taking Action (MTA) groups were organized, at the outset, to involve men in HIV prevention activities. Members of MTA groups interviewed by evaluators reported supporting their spouses to attend antenatal care (ANC) and have HIV testing via PMTCT, getting involved in the gender norm transformation in communities such as anti-GBV activities, as well as in couple HTC - done via door-to-door approaches -, and spearheading the anti-alcohol and drug abuse activities in their villages. MTA groups were also involved in program implementation as volunteers and members of GROW groups. More importantly, however, the chiefs and headmen championed male involvement in ZPI’s activities. They acted as role models and leaders of community initiatives that encouraged more men to be involved in community-led HIV prevention efforts. 35 Utilizing economic strengthening (i.e. GROW Groups) as an entry point for other HIV interventions. GROW Groups were initially set up as community micro-savings and micro-lending groups. Due to the multifaceted approaches used in the ZPI lenses, GROW Groups became channels for disseminating HIV prevention messages, encouraging community dialogue and addressing social and cultural norms. The groups interviewed by evaluators were very motivated to engage in economic empowerment activities. They reported being able to meet regularly and contribute their savings or borrow money, for example, to expand their small businesses or pay for their children’s schooling. “I used savings to buy more fish to sell in the markets in Mongu and, even in Lusaka” (A female GROW Group respondent, Mongu District, Western Province). This approach required little support beyond the initial training, a safe place to keep their money, and monthly supervision from a facilitator. The GROW Groups are self-governing and relatively independent, a key factor for sustainability and ensuring that HIV prevention messages and peer support on behavior change is ensured. Reaching out to underserved areas. ZPI worked to reach underserved areas. For example, the project selected Western Province as a target in Year 3 of the project. This is one of the most underserved provinces in the country, with regard to many health indicators, and relatively few NGOs provide community health services and outreach. Through local NGOs, ZPI was able to reach out to 16 fishing camps and organize 16 groups of MTAs with 275 members. Another example was the selection of sites through consultation with the MCDMCH for family planning interventions in areas that were identified as some of the most underserved (e.g. Chama, Milenge, Samfya and Serenje districts), having no partners working in family planning. Building strong partnerships. Establishing and fostering partnerships has been at the heart of ZPI’s approach. This is reflected in the number of national and provincial level partners (12), small grants recipients (14), and CPO beneficiaries (35). Other than these groups there is evidence of partnerships at various levels with government agencies (e.g., the MOH, MCDMCH, and NAC), other USAID implementing partners (e.g., Economic Strengthening, Prevention and Support to Orphans and Vulnerable Children or STEPS OVC, Zambia Prevention care and Treatment or ZPCTII, and Communications Support for Health or CSH), and with the main community key stakeholders and gatekeepers (28 Chiefs and Headmen). These partnerships allowed ZPI to foster acceptability and ownership of the community-led behavioral and structural HIV prevention interventions across different levels of HIV prevention implementation in Zambia. The partnership allowed these organizations to work more closely, not only with ZPI, but also with one another, and in some cases, encouraged collaboration outside and beyond ZPI. C. Lessons learned during project design, implementation and management. The evaluation documented a number of lessons learned by ZPI and its partners: a. Design Late rollout to some provinces and late initiation of some activities. The project launched activities in five provinces in Year 1 (2011), namely, Copperbelt, Eastern, Lusaka, North-Western and Southern provinces. In Year 2 (2012), it established activities in Central and Luapula Provinces, with Western and Northern Province activities finally added in Year 3 (2013). PMP data shows that Western and Northern provinces struggled to reach many of their targets at this late stage of the project, a fact that may have been exacerbated by the budget cuts near the end of the project. Nevertheless, it is clear that starting activities in these provinces at this late stage of the project was far from ideal. There was little time to make real gains in their interventions and inadequate time for the community to own and sustain the prevention efforts. Given the fact that many of the activities were directly related to behavior change, a substantial amount of time would need to pass to consolidate and assess real progress. Another example was the Boys for Change initiative launched in 2014. The project provided youth training in April 2014. At the time of this evaluation, 36 this initiative had had just over two months of activities and was closing out, limiting the assessment of these activities. The overall gain seems limited to trainings and capacity building at this time. The complexities of mental health. The mental health lens is the least developed, particularly at the community level. Qualitative findings through KIIs and FGDs confirm this. Mental health is a complex issue that needs more strategic and technical designing on a national level to be adequately addressed, especially in its connection with HIV risk and vulnerability. Services for mental health are still quite limited in most parts of the country, making it impossible to link community needs with organized governmental and non￾governmental structures. It was therefore a challenge to sensitize communities on mental health when there are no services for referral for further support. ZPI’s 2014 work plan stated that since the establishment of the MCDMCH, there has been a slow realignment of mental health. Policy development and higher-level psychiatric service delivery fall under the jurisdiction of the MOH, while community-based Mental Health Programing is now under the MCDMCH. ZPI has, however, advocated for the re-establishment of a National Mental Health Advocacy Committee through the MOH Public Health Division. For the most part, mental health was only partially covered in the work under the alcohol and other drug abuse lens, mainly by the Serenity Harm Reduction Programme in Zambia (SHARPZ). Internal evaluation design and attribution of findings. The attribution analysis of the internal evaluations findings (the baseline and midline household surveys) is a challenge for reasons outlined earlier. The lessons learned by the ZPI project in evaluation design have been transferred to other projects, such as Community Mobilization for Preventive Action (COMPACT), and the evaluation design was strengthened to better integrate the findings to inform the project interventions. Another lesson relating to implementing the studies was the period taken to obtain ethical approval. It took nearly six months for ethical approval to be given. More time needs to be taken into account when planning such studies. b. Implementation Annual contracts can be a hindrance to long-term planning. The annual contracting procedure prevented sub-grantees from doing long-term program planning, due to lack of continued funding guarantees in the coming year. The continuation of funding was based on performance, according to the ZPI project’s documentation. However, this still created uncertainty and planning was limited to annual cycles, while the some budget reductions affected the project’s success. Having dedicated M&E staff at all levels is critical for data quality and effective reporting. One of the weaknesses observed in the M&E system was the need for dedicated and capable lower-level M&E staff (i.e., at provincial and district levels) to assure data quality and complete reporting. HIV technical advisors in all but two provinces handled the M&E responsibilities. Further training is required for economic empowerment groups. GROW Groups interviewed reported a need for further training related to entrepreneurship if they are to increase their assets and utilize their savings to greater advantage. They also reported that other community savings groups were receiving additional technical assistance to accelerate growth. In one district, one group decided to leave the ZPI project and go with another project that offered entrepreneurship training and practical training in skills such as “tie and dye”. Private sector engagement on HIV programs, cost sharing, and sustainability are persistent challenges. Private sector engagement and monetary contribution to HIV prevention 37 programs remains a challenge, though a lot of progress has been made in the project. Thirty-four organizations were involved in the Public-Private Partnerships (P3) component of the ZPI project’s HIV prevention intervention effort. The evaluation shows that not all private partners managed to cost-share, although activities were still carried out with them. One sub-grantee reported they decided to stop focusing on working with private sector organizations, in order to focus more on schools. More work is needed to establish more sustainable corporate social responsibility programs in the private sector. The ZPI project has been meeting with the P3 unit at the Ministry of Finance to amend the P3 Act in this regard. D. Changes occurring during implementation, both external and internal that may have had a bearing on the activity outputs and outcomes. a. External Changes Realignment of the MCDMCH and MOH. The realignment of these ministries took place during the course of the project’s implementation. This has proved to be a challenge for various reasons. For instance, the FP focal person was at the MOH under the Department of Public Health, but when the MCDMCH was established, this role was shifted to the MCDMCH without a specified focal point person for FP. The mental health lens was also affected in the realignment as outlined earlier, whereby the MOH became responsible for higher-level psychiatric service delivery and the MCDMCH for community-based Mental Health Programing, which negatively affected the mental health activities of the project. The fact that the MCDMCH organized the handling of its roles from within might explain why linkages with the ZPI project seem to be weaker at the national level than at the district and community levels. Changes in PEPFAR indicators. One of the biggest challenges was the change in indicators and reporting templates during the course of the project. During the implementation period, PEPFAR changed from New Generation Indicators (NGI) to Rapid-Monitoring of AIDS Referral System (R-MARS) indicators. Of the 10 major PEPFAR indicators that were tracked during the project period, five were retired (P7.1.D, P8.2.D, P12.2.D, C5.7.D and H2.2.D) and one was dropped (P1.1.D; a health facility level indicator). Only three indicators were tracked during the project’s four-year implementation period (P8.1.D, P8.3.D and P11.1.D), and one was only tracked in the final year (gender norms). This tracking of indicators was determined by PEPFAR, and was therefore beyond the control of the project. However, it is clear these changes had adverse effects. Changing indicators halfway through the project deviated from achieving the project targets. Ultimately, these changes affected the data quality, and affected the effectiveness and the efficiency of the M&E activities; it also made tracking progress a challenge. In some cases, the organizations had to go back and compile numbers retrospectively to match the new indicators in their own time and at their own expense. The changes also led to lower data quality, as DQAs revealed that some organizations had not moved on from using old forms for data collection to new ones as changes occurred. KIIs also revealed some level of confusion and frustration regarding the discontinuation in indicator tracking and reporting. This was further exacerbated by the fact that everyone in the project was new to the indicators, including the people who were supposed to train others and provide technical support, and as such, answers were not always forthcoming when organizations sought help. b. Internal Changes Handling of budget cutbacks. One of the most important challenges to the project’s advancement was budget cutbacks and the consequent ending of subcontractor and sub-grantee activities. This was largely beyond the project’s control and determined by funders. However, stakeholders were not given ample notice 38 regarding these cuts, nor were they invited to participate in any discussions on how to handle the cuts. While ZPI was not responsible for the cuts, the situation was not handled well. D. Effectiveness of tools used to track changes in HIV prevention among vulnerable and most at risk-populations. PMP development process. The PMP development process was consultative, though a workshop, and involved all partners except ZHECT (ZPI PMP 2013). The M&E strategy had three aims: a) identify effective and efficient prevention interventions that can be scaled up; b) indicate the program’s impact on the intended target populations; and c) determine improvements in the capacity and commitment of the GRZ, and the private sector. The ZPI project’s M&E system was intended to be aligned with existing government and local partners’ monitoring systems with use of the National AIDS Council Management Information System (NACMIS) and the Health Management Information System (HMIS), where possible (ZPI PMP 2013). However, this was not the case. It was project specific, was not aligned and did not feed into the national HMIS system as its indicators were community-based rather than facility-based. In addition to routine monitoring, several other strategies were planned to track progress, including baseline, midline and endline household studies, and operations research studies. Two operations research studies per year were initially planned (ZPI PMP 2013). However, challenges with ethics approval and the unforeseen complexities of designing such studies by the Population Council, made it difficult to carry out these studies. In the end, only the baseline, midline and four other studies were completed. There was a denial of ethics approval for the Adolescent Reproductive Health study because some questions were too sensitive for the targeted youth. The other studies carried out were the Caregiver Study with STEPS OVC, the Network Analysis Study; GROW Girls Study, and Boys for Change (discussed under Question 4). With regard to routine project monitoring, data was collected by subgrantees, aggregated and submitted to the ZPI project’s Provincial Offices, then further aggregated by provincial offices and submitted to the ZPI National Office. The ZPI project’s consortium implementing partners collected data and submitted quarterly progress reports directly to the ZPI project’s National Office in Lusaka (ZPI PMP 2013). This process was reported to have mostly run smoothly, with a few delays in the submission of reports from various levels. One challenge was the belated dissemination of the PMP by the ZPI project to provincial staff, delaying their awareness of the reporting requirements. Review of DQA reports. The evaluation team analyzed reports from 11 DQAs carried out by ZPI over the course of the project. It was unclear whether any standard tools were used for the data quality assessment, as the DQA reports varied as to what was reported, making comparison between provinces and within the same province at different periods difficult. Table 3 below provides a summary of findings related to data quality from the 11 DQAs. Table 3: Summary of findings related to data quality from the DQA reports DQA area No. of orgs reported on No. Complying % complying % not complying Accuracy 13 3 23 77 Completeness 6 0 0 100 39 Filing of reports 7 1 14 86 Timeliness 2 1 0 100 The data quality elements reported wer not consistent. Both over-reporting and under-reporting were common errors that affected data quality. There were inconsistencies in data collection and reporting forms used, with some organizations using forms that were incompatible with the ZPI project’s requirement and some using outdated reporting forms. Many of the reporting tools used for ZPI’s reporting had to be updated several times during the project due to changes in indicator types. Furthermore, new reporting forms were not always used in a timely manner. Of seven provinces, whose record keeping was reported on, only one kept its program records in a competent manner. The rest of the parameters were not consistently reported on to conduct any meaningful secondary analysis on data quality. This, however, points to inconsistencies in reporting DQA results and in the tools used for the DQA, as detailed earlier. Qualitative data also showed reporting inconsistencies, including volunteers not having reporting forms, and having to report by word of mouth with no supporting documentation. Effectiveness of M&E system and indicators to measure ZPI’s success or weakness Overall, the ZPI project PMP is well documented and conceptualized. The weaknesses arise from a lack of implementation of certain aspects, such as providing feedback to be used for decision making to lower levels. Based on findings from the document review, including data quality assessment reports and qualitative data, the M&E system is strongest at the national and provincial levels and weakest at the sub-grantee and community levels. This is largely as a result of the weaker capacity of the community-level organizations and individuals both in human resources, human capacity, and enabling systems. These are the sources of the information for the majority of the indicators captured and reported to USAID/Zambia, representing a weakness in the system that was not adequately addressed. More effort and resources should have been allocated to improving the data collection and reporting capacity of the sub-grantees and community volunteers. This might have contributed to strengthening the data quality and use at the lower levels. Effectiveness and efficiency of ZPI in HIV prevention and areas of comparative advantage Overall, the ZPI project was effective in achieving its HIV prevention results and objectives, having achieved most of its targets, despite an extensive reduction in its budget. The ZPI project lenses approach detailed earlier played a major role in its success and was a key comparative advantage. Other key factors that greatly contributed to this success include building strong partnerships with all of the stakeholders involved, anchoring activities in the communities, involving traditional and community leaders, incorporating a strong gender element into the programming, promoting male involvement, and targeting underserved populations and areas. All these factors gave the project a comparative advantage compared to similar programs currently or previously implemented in Zambia. In terms of efficiency, the project faced notable challenges related to project design, implementation and management. In particular the late start of some project areas and activities that were outlined above, as well as the limitations of having annual contracts with subcontractors, which was further complicated by the late distribution of funds to some of them. Internal and external changes such as budget cuts, changes in PEPFAR indicators, and the realignment of the MOH and MCDMCH all had some negative impacts on the project’s ability to implement its community-based HIV prevention interventions Question 3. What progress has been made toward ensuring the sustainability of ZPI’s approaches? The evaluation documented the potential for sustainability based on the assumption of ownership by the participants, consolidation of collaborative arrangements, increased knowledge and skills transfer, and evidence of technical, financial, and institutional capacity building. 40 A. The influence of the project on strategy, programming, and policy at the national, sub-national and community levels. ZPI’s participation in strategic national task forces and working groups was substantial. The project’s consortium partners facilitated dialogue, leadership, and decision-making among multiple stakeholders at the national level. The ZPI project provided technical and financial support, primarily for the work of the National HIV/AIDS/STI/TB Council, and the Ministry of Community Development and MCDMCH at the national, provincial and district levels. The ZPI consortium partners played an important role in the organization of the Third National HIV Prevention Convention, providing assistance in the call for papers and reviews of abstracts for presentation and support for international participants supported by the Gates Foundation Bridge Project. The ZPI project collaborated with the MOH on family planning. The project also collaborated with the Ministries of Gender and Child Development, Youth and Sport, and Home Affairs. The project pursued several opportunities to provide input into policy and programming processes and promoted the understanding of the theoretical constructs of ‘risk and vulnerability’ related to HIV prevention at all levels, as shown in Table 4 below. Table 4: ZPI Participation in National Programming and Policy Development Government Entity Participation in National Technical Work Groups and Task Forces National HIV and AIDS/STI/TB Council  Technical and financial support to the Third National HIV Prevention Convention (2013)  Prevention Technical Working Group  PMTCT Technical Working Group  Key Populations Technical Working Group Ministry of Community Development and Maternal and Child Health (MCDMCH)  Safe Motherhood Technical Working Group  GBV Technical Working Group  Family Planning Technical Working Group (FP TWG)  Community-Based Family Planning Task Force  Community-Based Distributors (CBD) Task Force Other  Prisons AIDS Advisory Committee (PAAC)  Zambia Network against the Harmful Use of Alcohol (ZNAHUA), Men Engage in Country Network At the provincial and district levels, the ZPI project brought together representatives of the public and private sectors, NGOs, and community leaders, especially Chiefs and Headmen, to introduce new theoretical constructs, and programmatic approaches and drive program implementation, ranging from developing collaborative arrangements with district authorities, to engaging chiefs, chiefdom councilors, and headmen, some of whom were trained as volunteers. At the district level, the program fostered ownership by engaging a broad range of key stakeholders, including District Commissioners, District Health Officers, District AIDS Coordinating Advisors, District AIDS Task Forces, small community-based groups funded by sub-grantees, and community leaders. Building support and buy-in among this array of disparate entities with differing roles and functions, knowledge and experience levels, and distinct points of view was a challenge, considering the multifaceted nature of the ZPI project HIV prevention lens approach. Provincial and district officials described how the ZPI project and its partners integrated the prevention lenses into district and community planning and programming and helped build their technical and management capacities. DATF members in one district described ZPI’s contribution to DATF coordination of the four pillars of community-led HIV prevention interventions. As one DATF member said: “ZPI (sub-grantee) chaired 41 the Prevention Pillar, and participated in the Impact Mitigation Pillar. They helped with technical training in prevention, mapping district wards, assisted with transport, fuel, and provided volunteers’ allowances to facilitate prevention activities in the district. At the last quarterly provincial meeting, our district had the highest number of women in PMTCT who attended with their husbands and had joint testing. ZPI was the glue that supported us to reach our targets. Even though the project ended, our activities at community level will continue but with less technical support and funding.” In addition, the ZPI project selected sub-grantees with recognized experience at the district and community levels to spearhead district community-led behavioral and structural HIV prevention efforts. At the community level, the project engaged chiefs and headmen whose support and engagement was vital and assured a high degree of buy-in and access to community volunteers and beneficiaries from the outset. This was the entry point for community volunteers and beneficiaries to actively engage, and this assured a high degree of buy-in across the various community stakeholders. Sub-grantees and other stakeholders interviewed lauded the project’s support, which enabled the pursuit of joint goals and collaboration in HIV prevention in working through challenges and gaps together. One sub-grantee summarized the support, saying: “ZPI gave us a solid series of trainings, and had a flexible but reliable M&E system. The framework was easy for our staff to follow. The ZPI provincial office allowed us to talk directly with the Lusaka office if we had a question. They were always very willing and open to modifications. Of course, the provincial office was here (in the district) so we didn’t have to wait.” However, this experience was not universal across provinces, as discussed under Question 2 above. In addition, the project worked with other USAID implementing partners, such as, the STEPS OVC, ZPCTII, and Corridors of Hope (COH III), in joint planning and implementation activities, and provided support in areas related to community outreach and demand generation. The ZPI project and COH III collaborated on the formation of FHI360 Gender Technical group that examined how to share gender expertise and methodologies for mainstreaming gender in all HIV prevention efforts, in addition to the training of key staff on gender analysis. At the provincial level, in Luapula and North Western provinces, the project referred men taking part in its community mobilization activities to ZPCT facilities for male circumcision. With regard to collaboration with Partnership for Integrated Social Marketing (PRISM), the ZPI project made referrals to PRISM partners for HTC and MC services, and conducted community mobilization for MC for PRISM in Eastern and Northwestern provinces. The ZPI project was a member of the Communication Support for Health technical working group, and involved in meetings for reviewing materials communication materials that were being developed. STEPS-OVC was an early member of the ZPI consortium for the first two years of the project. The ZPI project provided technical assistance on the use of gender lens to design interventions, and gender-based violence. The ZPI project conducted the Caregivers Assessment Study that examined the motivations of volunteers that provide support to orphans and vulnerable children, and provided training in safe from harm parent-to-child communication tools. B. The adoption of ZPI approaches by external stakeholders (such as government counterparts and the Global Fund to Fight AIDS, TB and Malaria). According to stakeholder reports, adoption of the ZPI project’s approaches was more likely to occur at the provincial, district and community levels. This is shown by the results achieved through ZPI’s collaboration at these levels. Provincial and district public and private sector representatives, and subgrantees key informants reported that ZPI-led interventions implemented at these levels and platforms increased the understanding 42 of the drivers of HIV transmission. They also reported that the project provided technical support to district and community stakeholders and built capacity within communities. The ZPI project’s sub-grantees strengthened collaborative relationships at the district levels, particularly with District AIDS Coordinating Advisors, District AIDS Task Forces, District Health Officers, Office of District Commissioner, and Victim Services Units in eight of nine districts visited. A DATF member interviewed during the evaluation explained how the ZPI project introduced the ‘risk and vulnerability’ lenses, saying: “ZPI empowered the village headmen and trained them and the community volunteers. The number of men being circumcised has increased, and the culture is changing.” A representative of the Police Victim Support Units described the project’s work on GBV, saying the “ZPI (sub-grantee) worked with communities, especially in cases of rape, assault, property grabbing, wife beating and sexual abuse. These cases were reported to us for investigation and referred to the magistrate for adjudication where [appropriate].” DATF members emphasized how the ZPI project improved their own understanding of ‘risk and vulnerability,’ commenting that in the case of GBV: “We thought GBV was wife-beating and fighting. Now we know that there are other issues like alcohol and economic violence. The chief has gone after young girls married off by their parents and ordered them back to their families and to stay in school.” (KII DATF member in Western Province). One chief reported having received technical support from the project to analyze issues of HIV prevention, alcohol reduction, and GBV, and which prevention measures and efforts to be included in the Chiefdom’s Strategic Plan in Southern Province. Other Chiefdom Councils adopted by-laws related to the aforementioned topics and monitored by headmen in Luapula Province. In one district visited in Western Province, community leaders said that the Chief and District Commissioner collaborated on HIV prevention and served as role models. Both were tested and counseled with their wives at a public event to demonstrate the importance and acceptability of counseling and testing. C. Evidence of organizational and technical capacity built among local implementing partners’ communities. Stakeholders, community leaders, volunteers, and beneficiaries reported having increased knowledge about HIV prevention, gender norms, male involvement, testing and counseling, male circumcision, gender-based violence, alcohol and substance abuse, and economic empowerment. Furthermore, they reported that they undertook proactive actions, such as community mobilization, community meetings on related topics, and conducted door-to-door visits where problems were identified. The ZPI project helped chiefs and headmen to organize village committees to promote HIV prevention. In all nine districts assessed, headmen organized community volunteers to conduct door-to-door sensitization for male circumcision, alcohol reduction, GBV prevention, prevention of early marriage, and economic empowerment. The exact number of committees and groups and their memberships is discussed under Question 1. Integrating the gender dimension through each lens catalyzed changes in social, cultural, gender and sexual norms and practices related to the ‘risk and vulnerability’ lenses. Traditional leaders, volunteers and beneficiaries emphasized, for the most part, that the ZPI project increased their knowledge and understanding of the drivers of HIV and prevention measures, and produced changes in the behavior in the community. One chief announced in public that he “stopped drinking, as of that day.” (KII Western Province) The evaluation found community volunteers and beneficiaries to be quite knowledgeable, and conversant about technical and legal issues as a result of the project training. For example, in more than 80 percent of 43 FGDs across the provinces visited, the following subjects were discussed with regard to the drivers of HIV; appropriate HIV prevention measures to be adopted to reduce risk and vulnerability, their availability within a five to 15km distance, and the role that alcohol and cannabis use plays in HIV transmission. Similarly, appropriate measures for testing and counseling for men and women, couples counseling, and GBV were discussed in 72 percent of the FGDs. Economic empowerment was discussed in 61 percent of the FGDs, with male involvement, PMTCT and family planning discussed in more than 50 percent of FGDs. Both volunteers and beneficiaries reported that when local committees could not resolve problems related to alcohol abuse, GBV, early marriage, and issues of access to service, they sought the assistance of headmen and the chief to enforce by-laws. Problems within families, such as wife-beating, neglecting children’s needs - such as buying uniforms and sending them to school, male involvement in PMTCT, or complicated marital relations were reported to headmen, and, if necessary to the chief and the Victim Support Unit for investigation and to magistrates for adjudication, where appropriate. A group discussion with 19 headmen recounted a number of successes brought about by the ZPI project’s technical support. A number of headmen commented:  “We accepted the messages. Yes, we had difficulties and naturally there was some resistance, some people took a long time.”  “But, we, village headmen, took an active role and formed local committees and groups. If someone misbehaves, drinking, taking drugs, or beat their wives, they are reported to us.”  “In this project, we learned it is worthwhile to have a trusting relationship within the family.”  “If a girl child gets pregnant, she must go back to school.”  “We appreciate this project. Before if a person was affected they were considered condemned and now we are able to prevent and treat them.” Economic empowerment was another important topic emphasized in the KIIs and FGDs. Across all of the FGDs with the GROW Group members, the most common response was that this system helped them solve financial problems in the community and within their own families. For example, money earned and loaned from the GROW Groups helped solve problems during funerals, pay for school uniforms, and also helped expand local groceries, fish-selling and tailoring businesses, and increased family incomes. Prior to GROW, beneficiaries borrowed locally with high interest rates (kaloba). However, as noted under Question 2, GROW Group members did report challenges in building assets beyond a certain threshold, indicating a need for assistance in seeking loans and other technical support. In addition, beneficiaries reported that community facilitators and some health care providers sensitized GROW group members about HIV prevention, HTC, PMTCT and ART. KII and FGD respondents pointed to organizational capacity building, including building organizational systems and the providing equipment, as weaker than technical capacity building. Sub-grantees reported variable quality in capacity building as discussed under Question 2. 44 D. Evidence that ZPI models will continue to be implemented beyond the life of the project. The ZPI project strategy for sustainability was to engage with government and civic leaders at the national, provincial, district, and community levels to embed sustainability in community-led behavioral and structural HIV prevention interventions. This was to be achieved through community mobilization, the establishment of referral systems and technical assistance to PATFs, DATFs, NGOs and CBOs. Community platforms created by the ZPI project and its partners promoted ownership and sustainability of community-based HIV prevention efforts. However, continuity on a large scale is probably not feasible for most interventions. Programmatic level community activities will continue, and knowledge and understanding of the drivers of HIV transmission and related ‘risks and vulnerabilities’ are well embedded at community levels and supported by the adoption of chiefdom by-laws. Furthermore, Alcohol Support Groups and GROW Groups were consolidated to a greater extent, although they will need additional technical support to expand their efforts and increase assets. Community volunteers will continue to be a critical resource in the implementation of community-based intervention. They have the knowledge and experience that will remain at local levels to ensure that some approaches will continue, albeit perhaps not at the current scale. As a facilitator in a Boys for Change Group commented: “The guys asked me: ‘This program ended?’ I said ‘yes, it has come to an end’. They said ‘Tell them that THEIR program came to an end, but OUR program is still going on. We will continue’.” Training of “champions” is another strategy that was adopted to promote ownership and sustainability of the key elements of the program. Influential people in the community such as chiefs and headmen were identified as champions. Community-based HIV prevention, counseling and testing, anti-GBV efforts, and anti-alcohol and substance abuse are lenses that were accepted in the communities, and were adopted by traditional leaders, both chiefs and headmen. These influential people or champions have adopted these preventive measures into the local socio-cultural and economic development structures, and will continue implementing them in the medium term. At the same time, the project rolled out activities in the later stage of programming in Western and Northern provinces where limitations in capacity and structures in these regions were known. As the project experienced budget cuts in some aspect, the activities in these two provinces were discontinued at a very early stage, meaning there was insufficient time to assure ownership and continuity. “We made policy for our people, and want to continue this support. Our subjects are very poor. They have to walk long distances for PMTCT, [and to] get their medications and CD4 results at health clinics. We don’t have transport or a way to communicate with our people and the clinics. Our young people don’t have access to jobs when they finish school. We want to expand the program … we are just getting started!” (Chief, Western Province). Other challenges to sustainability were reported in the KIIs and FGDs. These are related to the availability and accessibility of health information and services for hard-to-reach communities and populations. These included distances between villages and from villages to health facilities for HTC, PMTCT and MC ranging from five to 15km; mobile clinics circulating just once a month, stock outs of reagents for HTC; and a lack of a means to report needs and otherwise communicate with district authorities and health facilities about health needs, results of testing, CD4 counts, and other exams. As one Chief in Southern Province emphasized, “The clinics are far apart so not easily accessed by other members of the community. CD4 machines are fewer in the province. People need to go to Choma for CD4. The clinics in Sikalongo and Batoka only offer testing and ARV drugs.” 45 This concern was echoed by two Chiefs in Mukututu, Western Province, “The problem is that testing is only done in Namsheke or Muoyo, and there are no mobile services for testing. Also, PMTCT is only provided in Muoyo. This is a problem for women to walk that far. We have a saying- ‘it’s bad to carry something you can’t carry.’ We need help to get the government to set up a health center in the region.” These challenges will limit efforts to connect hard-to-reach populations to health information and services and will interfere with the continuity of care. Overall, the ZPI project built support and buy-in among disparate entities with differing roles and functions, knowledge and experience levels, and distinct points of view, and assembled a united response. The project did not have a well-articulated and thoroughly developed sustainability plan and had not planned for contingencies other than the usual, such as the budget cutbacks experienced. E. Evidence of ability to leverage support for the project. The ZPI project consortium members have extensive experience in leveraging resources. One sub-grantee in Choma district reported having received funds from other USAID-funded projects, the Global Fund (CHAZ), bilateral donors such as DFID and Government of Germany, and from the Ministry of Gender and Child Development, and Ministry of Youth and Sport. Another sub-grantee in Katete District reported that they are leveraging funds from the International Labor Organization to advance the GROW concept. Organizations that already had the capacity to secure funding from other sources, and had an adequate organizational structure may continue activities. Others with a more limited organizational structure may secure additional funding, but could have limited capacity to manage program activities. Overall, the ZPI project provided training to sub-grantees focused on HIV prevention and the theoretical constructs of ‘risk and vulnerability’ lenses. The project did not engage in any other organizational and managerial development capacity building with sub-grantees beyond training in M&E and financial reporting functions. Question 4. To what extent are the strategic information activities serving to inform the planning, implementation and monitoring of the project? A. Extent to which strategic information was used in programming for results and decision-making. The ZPI project established an M&E system and PMP as a basis for its Strategic Information Framework, and recruited an M&E advisor with the skills to develop a system to track the community’s progress to program level, and assure timely submission of semi-annual and annual progress report each year. When some PEPFAR indicators were retired, the project received new indicators and adapted project-generated data to the new PEPFAR indicators for reporting. In cases where the project had met or exceeded targets, such as for HCT and GBV, the project re-focused its emphasis on areas where targets were not being met. Retiring PMP indicators limited the project management system’s efficiency, particularly the relevance of the data collected under the “old” indicators for decision making. The ZPI project held quarterly review meetings at the central level in which the project senior management team, M&E advisor, and senior technical advisors analyzed the M&E data for quality as well as assessed performance of its HIV prevention efforts in meeting the targets. When sub-grantee performance was not satisfactory, such as where performance targets were not adequately met, the project’s staff were deployed to provinces to assess problems and assist with improving implementation. Beyond the quarterly review meetings, there was no deliberate effort to review and use the data at lower levels for decision-making. The project did not define management indicators to assess performance on project management at the central, 46 provincial, and sub-grantee levels, which would have enabled the project to assess the management and implementation capacity of the sub-grantees. Sub-grantees reported the numbers of persons reached through the different project interventions to the ZPI project Provincial Offices and to the National AIDS Council through DATFs on a quarterly basis. Sub￾grantees submitted reports to DATFs directly in each district. However, the project provincial offices did not submit reports to the PATFs. A DATF member reported having received technical support to improve capacity to collect and use service reports submitted to them by all implementing partners operating in the district. This information was used in planning, implementation, and management of HIV prevention interventions at the district level. However, this type of sub-grantee technical support to DATFs does not appear to have been uniform across the districts visited. The project trained provincial, sub-grantee staff and community volunteers in data collection and reporting and supplied them with standardized sheets for reporting. Sub-grantee reports in Southern, Luapula, and Western Provinces showed inconsistencies in reporting to the national systems. In Eastern, the DATF reported that three sub-grantees submitted quarterly data, but the ZPI project’s Provincial Office did not submit quarterly reports. At the project activity levels, evaluators found that community volunteers in some communities had reporting forms, while others had no forms and some used the wrong forms. Since most ZPI project’s Provincial Offices had already been phased out, the evaluation team was unable to assess how these offices used the program data in management and decision-making. M&E team data quality audits found that record keeping was poor. Efforts were made to address this weakness by changing reporting formats several times and building the capacity of provincial and sub-grantee teams to help improve reporting. The ZPI project relied on one of its consortium partners, the Population Council, for expertise in capacity building on reporting system aimed at sub-grantee staff. B. Criteria for selection of studies to be undertaken and the quality of study designs and results. The administrative data lists several studies that were due to be conducted during the ZPI project’s duration. The Population Council was responsible for conducting the baseline, midline, and endline household surveys to evaluate the project results, and conduct the operations research studies (two proposed per year). The Population Council also designed and conducted smaller studies. It was not feasible to conduct two operations research studies per year. The delays occurred because the University of Zambia, Biomedical Research Ethics Committee meets only once a month with some reviews taking up to 6 months for approval. Secondly, the ZPI project may have underestimated the complexity of organizing these studies during the planning phase. The project eventually conducted four studies, including the Caregiver Study with STEPS OVC, the Partnership Study, the GROW Girls Study, and the Boys for Change Study. The baseline and midline evaluations were conducted in Copperbelt, Eastern, Luapula, and Western provinces using a cross-sectional community-based approach. The survey targeted randomly selected males (15–59 years old) and females (15–49 years old) at community and household level. At baseline, a total of 845 males and 1,594 females completed the interviewer-administered survey, and at midline 750 males and 1,437 females completed the survey. Positive changes were found with regard to HIV testing, male condom use, income earning among females, and gender equity attitudes in both males and females. However, as the midline report emphasizes, attributing improvement in the outcome indicators to the ZPI project’s community-based HIV prevention interventions was not possible, for reasons discussed under Question 2. These included lack of control groups, difficulty matching intervention areas to the study areas, and a lack of adequate exposure to the interventions being assessed in the survey. These limitations are not unique to the ZPI project and point to the general challenges of attributing findings to one project in a setting that has so many players 47 addressing HIV/AIDS issues simultaneously. It should be noted, however, that the ZPI project did contribute to social and risky health behavioral changes in the communities, as attested by the qualitative findings of the evaluation, and to a limited extent the comparison between the baseline and midline household surveys. Lastly, more rigorous study for attribution would likely be too costly and not an efficient use of limited resources. Lessons learned from ZPI’s design of studies and subsequent potential use of the findings has been adopted in other projects such as COMPACT. The GROW Girls Study was designed to identify approaches to improve both their economic and health status and understand the challenges facing girls and young women affected by both poverty and HIV. The study sought to assess the link between HIV outcomes and economic empowerment. The Grow Girls intervention was evaluated upon completion of a one-year pilot phase. A critical conversations methodology was used to elicit ideas, opinions, and experiences from the girls (18-31 years old, average age 22) who were part of these groups in Namwala and Lusaka Districts. The study assessed integrated activities of Population Council Safe Spaces, Grass Roots Soccer activities, and Economic Empowerment. The girls interviewed were very interested in these interventions and, as a result, the dropout rate was low. Several girls started businesses through this activity. Girls who participated in Grow Girls had a positive experience, and demonstrated behavioral changes that reduced their risk for transmission and acquisition of HIV, and indicated improved economic and social empowerment that led to risk reduction, improved self-esteem, and social connectedness. The combination of several interventions for the same target group was effective. The study was limited by the fact that only girls aged 18 and over were interviewed, which resulted in a lack of information on the program’s impact on younger girls. The Caregiver Study was designed to examine the motivations of people volunteering as STEPS OVC caregivers and to understand how best to recruit and retain volunteer workers for the purpose of informing future programs that required the use of volunteers. The study utilized a multi-staged, mixed-method survey of 758 active caregivers, selected using a quota-purposive sampling frame in four phases: design stage, initial instrument development, and validation study, and main study. The two main findings were that community and religious values were virtually universal in the study population with a majority of the volunteers reporting economic and material interests and needs. The findings showed a high level of commitment to people with HIV; volunteers are motivated by compassion for others and a desire to help the community. Monetary compensation is still helpful. The main purpose of the HIV/AIDS Organizational Network Analysis study was to measure community participation around HIV prevention in ZPI-supported districts. Social network analysis procedures were used to systematically assess the extent and quality of collaboration between key HIV advocacy and service organizations through a short survey to determine organizational characteristics followed by KIIs. Study results from four districts showed that network connections among organizations involved in HIV prevention existed, but that they were not ‘high functioning’, in that there was a low density of active collaboration and a near absence of formal inter-organizational ties. Boys for Change was an assessment carried out as an outgrowth of the GROW Girl study. The assessment was used to design a 12-week intervention. Results showed that very few boys understood girls’ sexual development or alcohol use as a drug. Results were used to develop the Boys for Change intervention. Obtaining ethical approval was the main challenge identified in conducting all of these studies. Other challenges were that it took time to plan the studies and required a considerable amount of time to collect the data, analyze it, and report out. Subcontracting data collection in the baseline study did not work well. The Population Council decided to undertake all stages of the subsequent studies themselves. ZPI and USAID made a strategic decision to do the endline study themselves due to budget constraints. 48 Most of the studies undertaken in this project were under-budgeted, causing restrictions in completing and reporting out. The ZPI project’s M&E team was involved in the initial planning process, provided direction in selecting ZPI lenses, and facilitated transport. Ethics approval for the studies was lengthy. Synchronization of time and resources was a challenge due to distribution issues and the time taken to agree on frameworks. Budget approval sometimes took time but did not delay the study activities once approved. Overall, the studies carried out were scientifically sound with rigorous methodologies that ensured quality. However, it is unclear how some of the studies were used to ultimately inform programming, or if any significant changes arose in the project. As a result, study results were not disseminated widely. C. Strategies used to disseminate strategic information by project/implementing partners, and the use of this information at different levels. The dissemination of the studies was limited to the immediate ZPI project’s key stakeholders. There were no specific resources to disseminate the study results to the communities, which meant that dissemination remained at a national level where results were discussed in workshops without feedback to the communities. Furthermore, no actions were developed in response to the study findings and recommendations. In addition, no discernable use of the study results can be seen in the annual planning processes of the project. D. Extent to which the monitoring and evaluation system was utilized for program improvement. The ZPI project’s M&E system is discussed under Question 4 sub-section A, which documented that the M&E system was mainly used for program monitoring and improvement at the project’s central level, with limited use where the community-level activities were carried out. 5. CONCLUSIONS Extent to which the project achieved planned objectives and results in HIV prevention among vulnerable and most-at-risk populations. Overall, ZPI achieved and exceeded most of its planned objectives and targets. Of the targets that were not met, some of them may not have been within the project’s control. The project’s main area of comparative advantage was the design of a theoretical construct of ‘risk and vulnerability’ lenses that used the theory of change. ZPI created an enabling environment in which key HIV prevention messages were accepted and lodged within traditional and community development structures, supported by the adoption of Chiefdom by-laws and role modeling by community leaders. Extent to which the project design, implementation and management were effective. The ‘lenses’ approach for project design was effective. The project encouraged the formation of structures to reach out to and support communities to adopt new social, cultural and sexual behaviors, reducing ‘risks and vulnerability,’ and promoting healthy behavioral changes. ZPI used a unique approach to empower and train community leaders that brought about changes in social, cultural, and sexual practices. The ZPI consortium of both international and national level NGOs demonstrated knowledge and expertise in developing and transferring a new theoretical construct that generated coordinated HIV prevention efforts at the district and community levels. Some provincial and district-level sub-grantees had the capacity to undertake these approaches from the outset, with many tracking, monitoring and reporting results. At the same time, these are the organizations with the local credibility to enter hard-to-reach, vulnerable communities, and can provide continued support. 49 Progress made toward ensuring sustainability of ZPI approaches. There is a high level of ownership at the national, provincial and district levels, with the strongest buy-in at the provincial and district levels. Community involvement at every stage of the project created ownership and led to collective actions to create change. The project made huge strides in increasing knowledge about the drivers of HIV transmission and how to adopt healthy behavioral changes among key communities. However, much work remains to be done. If the momentum is lost, communities may become disenfranchised and project gains may recede, threatening sustainability. The economic empowerment approach (e.g. GROW Groups) provided an important opportunity for communities to build savings. Communities, however, recognized that they need to develop capacity and be connected to other structures to enhance the growth of their assets through loans and other means. Extent that strategic information activities informed planning, implementation and monitoring of the project. ZPI established an M&E system and PMP to track progress from the community to program level and assure the timely submission of Semi Annual and Annual Progress data each year as a basis for a Strategic Information Framework. When the PEPFAR indicators were retired, ZPI reviewed new indicators and adapted project-generated data to the new PEPFAR indicators for reporting. In cases where the project had met targets, the project re-focused its emphasis on areas where targets were not being met. ZPI reviewed M&E data quality and performance in meeting PEPFAR targets on a quarterly basis at the central level and took action to improve tracking and reporting where performance was inadequate. Evaluators found no evidence of deliberate efforts to use the data at the lower levels for decision￾making. The project did not define management indicators to assess performance on project management at the central, provincial and sub-grantee levels, which would have enabled the project to assess the management and implementation capacity of sub-grantees. The project undertook surveys and operational research studies. The baseline and midline studies were conducted in the provinces. Results of the midline study showed positive changes in HIV testing, male condom use, income earning among females, and gender equity attitudes in both males and females. As emphasized in the midline study report, improvements should not be attributed to ZPI interventions due to a number of other HIV interventions operating in these regions of the country. However, ZPI did contribute to social and risky health behavior changes in the communities visited by evaluators. The original plan was to conduct two operation research studies per year. Due to a time lag with ethics approval and the complexity of organizing these studies, which was underestimated in the planning phase, four studies were eventually carried out including the Caregiver Study with STEPS OVC, Partnership Study, GROW Girls Study, and Boys for Change Study. All studies reviewed by evaluators were scientifically sound with rigorous methodologies that guaranteed quality. Limitations identified were the lengthy process of obtaining ethics approval for the studies through the University of Zambia, Biomedical Research Ethics Committee, which interfered with the timing. In addition to ethics approval, delays in the release of funding resulted in delays initiating the study activities. The budgets did not provide for the dissemination of findings to provincial and district levels. Therefore, it is unclear how the studies were used to ultimately inform programming. 50 6. RECOMMENDATIONS 1. Continue to support and expand coverage of effective and proven community-based, community-led interventions to reduce HIV ‘risk and vulnerability.’ 2. Widely disseminate the best practices and lessons learned among program planners and policy makers of the application of the theoretical construct of “risk and vulnerability lenses” and the theory of change to promote adaptation and replication. Dissemination budgets should be set aside during planning of studies for the dissemination activities at all levels, including provincial and district levels. 3. Build program design, management, and monitoring capacity at provincial, district and community levels, and decentralize technical, management and monitoring functions responsibility from central to provincial and district levels. a. Upgrade and reinforce the organizational and management capacity of district and community structures, especially local NGOs (sub-grantees) and community leaders to enable them to prioritize, lead, manage, monitor, and be accountable for primary prevention and strengthening linkages with health delivery system for diagnosis, care and treatment, and support. b. Improve M&E systems at the community level, by appointing dedicated M&E staff; train for community level M&E responsibilities and provide them with adequate tools including reporting forms, computers and Internet to facilitate monitoring and adequate report. 4. Ensure an integrated package of the ‘lenses’ approach in primary prevention and assure the availability of and access to biomedical services, particularly for vulnerable groups considered to be ‘negative’ - youth and People Living with HIV/AIDS. 5. Further training for economic empowerment groups, particularly on entrepreneurship to promote assets to grow and capacity for management to advance. 6. Explore mechanisms to reduce community isolation and improve access to communication by: a. Promoting innovative uses of low cost and free mobile technology to improve M&E systems at the community level (e.g. using mobile phones for data collection, reporting and receiving feedback from the supervising health structures). b. Providing bicycles and other means of transportation to ease the burden on community volunteers in the most isolated communities. c. Promoting the use of SMS or text messaging to disseminate health messages to the community on HIV prevention including testing and treatment. 7. Continue to provide technical support for capacity building at the policy and implementation levels to the National AIDS Council and Ministry of Community Development and Maternal and Child Health. 8. Support advocacy for the re-establishment of a National Mental Health Advocacy Committee through the Ministry of Health Public Health Division. 9. Involve the private sector through advocacy and technical assistance in developing their corporate social responsibility for HIV prevention programs for sustainability, and support advocacy with the Ministry of Finance for amending the Public-Private Partnerships (P3) Act. 51 ANNEX I: EVALUATION SCOPE OF WORK SECTION C - DESCRIPTION / SPECIFICATIONS/ STATEMENT OF WORK C.1 BACKGROUND With an estimated HIV prevalence of 14.3 percent among men and women aged 15-49, Zambia is one of the Sub-Saharan African countries most affected by the HIV and AIDS pandemic. Of the 12.9 million people living in Zambia in 2010, an estimated 1.1 million people were infected with HIV, and less than 15 percent knew their status. Key social and behavioral factors contribute to the high prevalence of HIV in Zambia, including gender-based violence (GBV), low levels of male circumcision (MC), multiple and concurrent partnerships (MCP), and low levels of condom use. The Zambia Led Prevention Initiative (ZPI) Project targets vulnerable and most-at-risk populations, including: youth; orphans and vulnerable children (OVC); persons living with HIV and AIDS (PLHA); people engaging in MCP; discordant couples, especially in cases of undisclosed zero-positive status or risky sexual behavior; and at-risk HIV-negative adults. ZPI also targets "better off and better educated" Zambians for whom the 2007 Zambia Demographic and Health Survey (ZDHS) shows higher HIV prevalence. ZPI has a presence in all the provinces of Zambia, having used a staggered approach to set up operations throughout Zambia. Since inception, the ZPI project has undergone a number of modifications. The first significant modification was done in December 2010 to modify the Task Order by replacing the Branding Strategy and Marking Plan. The second modification, which was signed on November 22, 2011, was to record a novation from Academy for Education Development (AED) to FHI Development 360 LLC (FHI360). On April 26, 2012, another modification was made to adjust the life of Project (LOP) targets for prevention of mother-to-child transmission (PMTCT) and HIV prevention, remove Project Concern International (PCI) as a sub-partner and to realign the budget. ZPI’s modification number five was to exercise USAID’s unilateral right to exercise the family planning option activity and to effect key personnel change as well as to incorporate mandatory PEPFAR reporting requirements. C.2 PROJECT DESCRIPTION In August 2010 Zambia-Led Prevention Initiative, a Task Order was awarded to the former Academy for Educational Development (AED) (now under FHI360). The purpose of the ZPI project is to increase utilization of community-level interventions through a targeted approach and to provide technical leadership and expertise on comprehensive, effective, community-based prevention efforts aimed at reducing new HIV transmission in Zambia. ZPI has a life-of-project amount of $39,726,852.00. The main objectives of the Project are to: 1. Build capacity in communities affected by HIV/AIDS to access more effective, gender-sensitive, higher-quality HIV prevention programs, including HIV testing and counseling (HTC), MC, and PMTCT; 2. Strengthen the continuity and coordination of, as well as commitment to, effective, efficient, and sustainable HIV prevention, including HTC, MC, and PMTCT; 52 3. Design efficient, sustainable, and locally owned responses to HIV/AIDS, including increased engagement with the private sector; and 4. Provide community-based family planning and reproductive health services as an adjunct to effective prevention of HIV/AIDS. For the full project description, please refer to Section J.2. C.3 PURPOSE AND USE OF THE EVALUATION By December 2014, the ZPI project will have been implemented for just over four years (August 2010 to December 2014). The purpose of this end-of-project performance evaluation is to analyze the ZPI project achievements to date in areas related to its performance, including relevance to identified needs; ability to achieve critical results; efficiency in achieving those results; and steps made towards sustainability. The evaluation will involve assessment of the ZPI activities in a representative sample of the nine provinces where implementation takes place. These provinces must include Lusaka, Central, North-Western, Copperbelt, Southern, and Luapula provinces. The other three provinces will be identified in the Work plan. As part of this overarching purpose, the evaluation should analyze gender implications so that gender equity can be achieved in HIV prevention interventions. The evaluation conclusions, recommendations and lessons learned will be used to inform HIV prevention strategy development, identify promising practices in community-led HIV prevention and strengthen HIV combination prevention programming. The evaluation should point out areas that require greater attention in future programming. The target audiences for this evaluation include: • USAID/Zambia: to identify promising practices and areas for improvement and to inform the design of future HIV Prevention interventions under USAID/Zambia’s Country Development Cooperation Strategy. • United States Government (USG)/PEPFAR: to demonstrate effectiveness of community led behavioral and structural HIV prevention interventions in order to strengthen combination prevention strategies. • Government of the Republic of Zambia (GRZ): to demonstrate the effectiveness of community led HIV prevention interventions as a potential methodology for a sustainable HIV response by the government. • FHI360: to inform the strengthening of its HIV prevention approaches. The report will be disseminated widely with relevant stakeholders and project beneficiaries, as well as submitted to the Development Exchange Clearing House (DEC). C.4 EVALUATION QUESTIONS The following key questions will guide the end-of-project performance evaluation: 1. To what extent did the project achieve the planned objectives and results in HIV 53 prevention among vulnerable and most at risk populations? In order to address the above question the Contractor must assess whether the activity managed to achieve the planned results. The assessment must focus on quality and quantity of outputs for the activity. Assess the factors that facilitated or inhibited the achievement of these results. 2. To what extent were the project design, implementation, and management effective and why? In order to address the above question the Contractor must assess the best practices and lessons learned during project design, implementation, and management. Indicate any changes that occurred during implementation of this activity, both the external environment and or internal to the activity, in the evaluation report especially where they may have had a bearing on activity outputs and outcomes. Assess the effectiveness of the tools used to track changes in HIV prevention among vulnerable and most at risk￾populations and whether these were good predictors of organizational success. Assess the effectiveness and efficiency of ZPI in HIV prevention and areas of comparative advantage. 3. What progress has been made towards ensuring the sustainability of ZPI’s approaches? In order to address the above question the Contractor must assess the influence of the project on strategy, programming, and policy at the national, sub-national and community levels. Assess the adoption of ZPI approaches by external stakeholders (such as government counterparts and the Global Fund to Fight AIDS, TB and Malaria, etc.). Assess the evidence of organizational and technical capacity built among local implementing partners, communities. Assess the evidence that ZPI models will continue to be implemented beyond the life of the project. Assess the evidence of ability to leverage support for the project. 4. To what extent are the strategic information activities serving to inform the planning, implementation and monitoring of the project? In order to address the above question the Contractor must assess the extent to which strategic information was used in programming for results and decision-making. Analyze the criteria selection of studies to be undertaken and the quality of study designs and results. Assess the strategies used to disseminate strategic information by project/implementing partners, and the use of this information at different levels. Analyze the extent to which the monitoring and evaluation system was utilized for program improvement. C.5 EVALUATION DESIGN AND METHODOLOGY C.5.1 Evaluation Design The evaluation will be carried out in Zambia by an independent evaluation team using a combination of qualitative and quantitative methods. Given that a counterfactual (control or comparison group) group was not established at the beginning of the project, USAID anticipates use of a non-experimental evaluation design for this evaluation. Baseline and midline surveys were however, completed and the contractor shall endeavor to make comparisons to assess changes in behaviors and practices. Contractors are required to come up with creative ways to assess the effectiveness of prevention interventions, which may require reconstructing the baseline and designing new tools to assess in a more rigorous way the outcomes of the HIV prevention interventions. Contractors are required to elaborate a detailed evaluation design and methodology as part of their Work plan. The evaluation design document shall include a series of data collection instruments. C.5.2 Data Collection Methodology The methodology must include an analysis of the results to date in the Performance Monitoring system and 54 program reports (i.e. trend analysis of results and on progress made on desired behavior change based on results of available studies, including the baseline and midline surveys. The contractor shall also analyze current approaches and challenges in key areas and perceptions of the ZPI project from interviews with local implementing NGOs, National AIDS Council and Ministries of Health and Community Development, Mother and Child health officials at the national, district and community levels, community workers and volunteers and project beneficiaries. The analysis must lead to specific recommendations on approaches for more effective implementation and operations for future HIV prevention programming. The assessment must also include a thorough review of the uptake data and analyze trends of the service utilization/uptake rates at the relevant service delivery sites. The analysis shall lead to recommendations for strengthening approaches to ensure program sustainability for the government and other key local and international stakeholders at project end. The evaluation team shall provide a more detailed description of the proposed methodology for carrying out the work and how data quality will be ensured as part of their Work plan. The methodology shall comprise a mix of approaches and tools appropriate to the evaluation’s research questions. These tools may include a combination of the following: • Review of relevant ZPI project documents. USAID will avail to the evaluation team the following project documents and monitoring reports: a) Contract, annual work plans and PMP b) Quarterly progress reports as submitted to the Contracting Officer’s Representative (COR) PEPFAR Annual and Semi Annual reports c) Project quarterly financial information d) Field Visit reports available from the COR e) ZPI Baseline and mid line reports f) Any other reports pertaining to project performance that is available and required by the evaluation team • Quantitative analyses (e.g. results against targets annually and over the life of the project by partner; cost-benefit or return on investment analysis, as appropriate) • Case studies of successful HIV Prevention activities • Key informant interviews and focus group discussions with a wide range of stakeholders including, but not limited to: a) USAID Zambia HIV/AIDS Multisectoral Office b) USAID Zambia Office of Financial Management c) ZPI sub-partners d) ZPI Beneficiary organizations (Grants Under Contract) 55 e) Community beneficiaries and leaders f) Other USAID and/or PEPFAR funded projects in Zambia g) National HIV/AIDS/STI/TB Council and Ministry of Community Development, Mother and Child Health h) Other donors funding HIV prevention activities The evaluation team shall present to USAID/Zambia for review and approval a detailed data collection plan that details how and where data will be collected within and as part of the Work plan. C.5.3 Data Analysis Prior to the start of data collection, the evaluation team shall develop and present, for USAID review and approval, a data analysis plan that details how: (1) qualitative data such as key informant, stakeholder, and beneficiary interviews and/or focus group discussions will be transcribed and analyzed; (2) quantitative data will be analyzed and presented to determine trends over time, including dummy tables; and (3) the evaluation will weigh and integrate qualitative data from these sources with data from project capacity assessments, service delivery data, and project monitoring records to reach conclusions and recommendations. Where needed, data will be disaggregated and analyzed by gender. Data collection and analysis shall emphasize, but is not limited to, design and use of strategic information in programming for results, criteria for selection of studies to be undertaken, quality of study designs and results, dissemination and use of strategic information by project/implementing partners for programming/decision-making and advocacy. Data collection and analysis shall also emphasize use of strategic information/data to inform scale￾up and decision-making, gaps in strategic information/data for strategic decision-making, and monitoring and evaluation system and utilization of routine monitoring data for program improvement. C.5.4 Challenges Associated with the Required Evaluation There is a lack of a rigorous design and tools to assess the outcomes of the HIV prevention interventions. As part of the Work plan, the contractor is required to propose a rigorous design and evaluation methodology despite all the potential data limitations in order to increase the rigor and credibility of the evaluation results. C.6 DELIVERABLES 1. Final Evaluation Design and Methodology: The contractor shall submit and obtain approval from USAID/Zambia for a detailed evaluation design, including a data analysis plan, prior to initiating any in￾country work. 2. Work plan: The contractor shall submit a detailed work plan aligned to the approved evaluation design within six days of arrival in the country. 3. Briefings: The Evaluation Team Leader shall brief the USAID Contracting Officer’s Representative (COR) at the onset of the assignment, weekly during the course of the evaluation as schedule permits, and at the end of the assignment (before leaving the country). The evaluation team shall organize and provide entry, mid-term, and final briefings for USAID/Zambia staff, other USG agencies and staff, implementing partners, select local partners, and host government officials. 4. Interview Notes and List of Resource Documents: The Evaluation Team shall provide 56 USAID/Zambia summaries of all key meetings, workshops, and discussions conducted during the course of the evaluation and copies of any relevant documents and reports gathered during the evaluation. 5. Summary Presentation of Findings to USAID/Zambia and Stakeholders: Two business days prior to departing Zambia, the evaluation team shall present initial findings to USAID/Zambia for review, comment and feedback. A PowerPoint presentation and handout (maximum of two pages) shall be prepared for the presentation. The team shall also present major findings of the evaluation to stakeholders. The team shall consider USAID/Zambia and stakeholder comments and revise the draft report as appropriate. 6. Evaluation Report: A draft evaluation report is due five business days after the field visit is completed. Within 10 business days of receiving USAID/Zambia’s feedback to the draft report, two hard copies and one electronic (MS Word) copy of the final evaluation report are due to USAID/Zambia. The evaluation report shall include the following: a) Executive Summary (3 – 5 pages) b) Background; c) Introduction; d) Methodology; e) Findings, including Lessons Learned; f) Recommendations; g) Conclusions; and h) Annexes, including: i. Scope of Work ii. Data collection tools iii. Key data sets, including interview transcripts iv. List of key informants v. Documents consulted Note: The main report i.e. from Introduction to Conclusions, shall be 40 – 50 pages and the contractor is free to include as much as is needed in the Annex. The evaluation report shall meet the criteria for quality evaluation reports specified in Annex I of the Evaluation Policy (http://transition.usaid.gov/evaluation/USAIDEvaluationPolicy.pdf). If USAID/Zambia disagrees with any aspects of the report, the evaluation team shall include a section in the report describing the points of disagreement. C.7 EVALUATION TEAM COMPOSITION Evaluation Team Leader/Senior Evaluation Specialist, Ms. Rosemary Barber-Madden. Ms. Rosemary Barber-Madden is a Global Health Expert with over 30 years of experience in international health programs and 8 years of experience as an evaluation Team Leader. Ms. Barber-Madden has over 15 years of experience in project evaluations involving data collection, analysis, implementation and design. Her extensive expertise and experience in public health and community health programs includes serving as Team Leader for USAID on evaluating the capacity of community based organizations to design and implement HIV strategies for the most at-risk populations in Honduras. She has also served as a Senior Health Consultant for AIDS, Tuberculosis, and Malaria to develop capacity of governments and civil society in design and management in Tanzania, Uganda and Brazil. She has extensive professional experience in Mozambique where she served as Senior Health Expert and has led the National Evaluation of maternal and child health. Ms. Barber-Madden has demonstrated strong assessment, technical and managerial skills; particularly in a cross￾cultural context. She has excellent writing, analytical and leadership skills. Ms. Rosemary Barber-Madden is 57 an EdD in Health and Public Administration, and also has Masters in Public Health. Team Members William Sambisa has over 15 years of professional experience in international development and a strong expertise in quantitative and qualitative research modeling, statistical techniques, monitoring and evaluation, implementation and quality assurance. He is a Manager in leading comprehensive Monitoring and Evaluation systems for international development programs, particularly in public health. He has experience working with unified and rigorous approach to program effectiveness, capacity building for performance, design, analysis and has field experience in countries including Nigeria, Zimbabwe, Botswana, Kenya, Zambia and Tanzania. Mr. Sambisa is a PhD in Rural Sociology and Demography and he also has M.Sc in Population Studies. Moses Simuyemba is a Medical Doctor and a Public Health Specialist with over 11 years of experience in managing HIV/AIDS and other related health programs. He serves as the Monitoring and Evaluation (M&E) Specialist for the Medical Education Partnership Initiative in Zambia, where he specialized in research design and M&E. He has successfully worked on 13 development projects in Zambia, and facilitated the development of several NGOs’ strategic plans, performed research and conducted case studies for evaluation; where he used both qualitative and quantitative data collection and research methods. He has successfully led project design, planning, implementation, M&E and reporting in a timely manner. Milka Juma has over 12 years of experience in monitoring and evaluation, implementation, design, research, and data collection in the health sector. He has particular expertise and experience in health interventions for HIV/AIDS, sexual reproductive health and children health within Africa, including Kenya, Ethiopia, Tanzania, Uganda, Zambia and Zimbabwe. Mr. Juma is a PhD in Health Promotion Research and Interventions, he also has MA in Population Studies and Demography. C.8 SCHEDULING, LOGISTICS AND SUPPORT The evaluation will be carried out over a period of approximately eight to nine weeks. The work plan must include a timeline based on the parameters described in the Deliverables section and above, which includes the contractor’s suggested time for fieldwork and writing the draft report. The evaluation team shall submit a detailed timeline as part of the work plan to USAID/Zambia. Extensive travel throughout Zambia is anticipated. The evaluation team shall not receive logistical support for travel, other than visa letters. To facilitate field visits, USAID will provide introductions to key stakeholders, including: ZPI partners, government counterparts, and other stakeholders. USAID/Zambia personnel will be made available to the team for consultations regarding sources and technical issues, before and during the evaluation process. 58 C.9 ILLUSTRATIVE ACTIVITY AND LEVEL OF EFFORT (LOE) MATRIX Description Team Leader HIV/Technical Specialists Local HIV/ Technical Specialist Period (No. of days) (No. of days) (No. of days) During Preparatory phase: Reading of background documents & preparation & submission of first deliverable & revisions 5 5 5 Week 1 Travel to Lusaka, Zambia 2 2 N/A Week 2 Planning meeting with USAID & external meetings with ZPI project staff & implementing partners in Lusaka 3 3 3 Week 2 Field work in selected provinces (includes data collection, analysis and initial draft of findings) 20 20 20 Week 3-5 Further Data Analysis and Writing including synthesis of evaluation findings 5 5 5 Week 6 Debriefing with USAID/Zambia 1 1 1 Week 7 Depart Zambia 2 2 N/A Write first draft report 5 4 4 Week 8-9 Prepare response to USAID comments & revise draft report and resubmit to USAID 3 3 3 Week 9- 10 Finalization and submission of final report 5 3 3 Week 10- 13 Total 51 48 44 END OF SECTION C 59 SECTION J – LIST OF DOCUMENTS EXHIBITS AND OTHER ATTACHEMENTS J.1 USAID/ZAMBIA HIV/AIDS MULTISECTORAL RESULTS FRAMEWORK, 2004-2010 HIV/AIDS MULTISECTOR RESULTS FRAMEWORK SO9. Reduced Impact OF HIV/AIDS Through Multi-sectoral Response Indicators: • % of OVCs receiving care and support • % of people 15-24 that are HIV infected • % of PLWHA receiving basic care and psychosocial services IR9.1 Reduced HIV/AIDS Transmission IR9.2 Improved Care and Support for People living/affected by HIV/AIDS IR9.3 Strengthened capacity of key sectors to mitigate the HIV/AIDS Impact IR9.4 Improved Policy and Regulatory Environment Illustrative Indicators: SO9 Specific • # of high-risk people receiving counseling and testing services • # of people reached with abstinence/being faithful community outreach programs • % of never married young men and women aged 15-24 who have never had sex Illustrative Indicators: SO9 Specific •# of OVCs receiving care and support • # of individuals provided with HIV-related palliative care Illustrative Indicators: SO9 Specific • #of HIV service outlets/programs provided with technical assistance or implementing programs related to policy and/or capacity building, including stigma and discrimination reduction programs • # of individuals trained in implementing programs related to policy and/or capacity building, including stigma and discrimination reduction programs Illustrative Indicators: SO9 Specific • # of organizations with HIV/AIDS workplace programs • % of general population with accepting attitudes toward PLWHA April 10, 2003 60 J.2 PROJECT DESCRIPTION The purpose of the ZPI project is to increase utilization of community-level interventions through a targeted approach and to provide technical leadership and expertise on comprehensive, effective, community-based prevention efforts aimed at reducing HIV transmission in Zambia. The main objectives of the Project are to: 1. Build capacity in communities affected by HIV/AIDS to access more effective, gender- sensitive, higher-quality HIV prevention programs, including HIV testing and counseling (HTC), MC, and PMTCT; 2. Strengthen the continuity and coordination of, as well as commitment to, effective, efficient, and sustainable HIV prevention, including HTC, MC, and PMTCT; 3. Design efficient, sustainable, and locally owned responses to HIV/AIDS, including increased engagement with the private sector; and 4. Provide community-based family planning and reproductive health services as an adjunct to effective prevention of HIV/AIDS. Programmatic Scope Sustainability is a key tenet of the reauthorization language for US HIV/AIDS funding. Whereas, the USG will continue to provide PEPFAR support in line with US policy commitments, all efforts will support the Government of the Republic of Zambia (GRZ) strategy and build the base for sustainable programs. Further, the USG will assist the Zambians to develop, supply, and support the expertise, leadership, and institutions necessary to win the fight against HIV/AIDS. Accordingly, Zambian ownership and leadership from design through implementation is key. The project will focus on establishing a more effective community-based combination prevention effort, as opposed to a clinical or media-based prevention strategy. However, this project will also create and maintain linkages to prevention via clinical sites, and via community-based and media-based efforts of the GRZ, USG, other donors and partners. Community-based activities will promote structural, behavioral, and biomedical prevention. The primary aim is to foster community participation to create commitments to reduce risky sexual and social behaviors. Activities will be implemented from national down to community levels, and will promote balanced, age￾appropriate, HIV prevention messages and services, ensuring that these messages are filtered to the community caregivers through HTC, and PMTCT service sites. The project will focus on the major drivers of the epidemic and continue to support abstinence and behavior change activities as defined by PEPFAR (including the reduction of multiple concurrent partners and delay of sexual debut). Although the project will not provide clinical services, it will promote MC, in line with GRZ strategy and policy. Gender equity efforts will promote positive norms for males and females, and discourage harmful norms. 61 This project, with its multi-faceted, combination prevention approach, will not only include behavioral and basic biomedical prevention, but also structural prevention efforts. It will require a variety of holistic, effective, efficient, appropriate, and evidence-based approaches. In the absence of efficient means to count infections prevented, the contractor will focus primarily on promoting proven-effective interventions for which reliable data exists. Though PEPFAR will still require projects to count clients who receive services, and people trained, this project will nevertheless seek to be more outcome-oriented. Target Groups The project will provide prevention services while balancing support among the more than 85% of Zambians who are HIV-negative, as well as the nearly 15% who are HIV-positive. The USG Zambia seeks to provide individual, household and community assistance equitably to OVC, at- risk youth, PLWHA, and to HIV negative, at-risk adults. This balance will include the provision of services, as well as access to economic growth opportunities. Working closely with other USG prevention related projects, especially the COPI-OVC, SHARE, and relevant follow on projects, this project will develop Zambian prevention leaders at the community, district and national level, especially positive prevention leadership by PLWHA. The Project is set out to meet the following targets: • P8.1.D: Number of targeted population reached with individual and/or small group level preventive interventions that are based on evidence and/or meet the minimum standards required (600,000). • P8.2.D: Number of the targeted population reached with individual and/or small group interventions that are primarily focused on abstinence and/or being faithful, and are based on evidence and/or meet the minimum standards required: subset of 8.1.D (174,000). • P11.1.D: Number of individuals who received testing and counseling services for HIV and received their test results (125,000). • P12.2.D: Number of people reached by an individual, small group or community-level intervention or service that explicitly addresses gender-based violence and coercion related to HIV/AIDS. (92,500). • C5.7.D: Number of eligible adults and children provided with economic strengthening services (10,500). • P1.1.D: Number of pregnant women with known HIV status, including women who were tested for HIV and received their results (Target 30,000). END OF SECTION J 62 ANNEX II: DATA COLLECTION TOOLS END-OF-PROJECT EVALUATION OF THE USAID FUNDED ZAMBIA PREVENTION LED INITIATIVE (ZPI) PROJECT Key Informant Interview (KII) Schedule for National, Provincial and District Stakeholders Province: District: Organization: Date: Sex: Position: Venue: Time Started: Time Ended: Interviewer: Introductory remarks Thank you for agreeing to talk with us. International Business & Technical Consultants, Inc. (IBTCI) has been contracted by the United States Government through the United States Agency for International Development (USAID) to carry out this end-of-project evaluation of the Zambia Led Prevention Initiative (ZPI) project. The purpose of this evaluation is to assess the achievements of the ZPI project to date, its successes and challenges. We believe that you are in a good position to tell us about your organization and what it is doing in relation to the ZPI project, hence this interview. We anticipate the interview will last about an hour or less and appreciate any information you can provide. Your answers to the questions we will ask are completely confidential and the information you give will be reported without names. Your participation is voluntary and you can refuse to answer any or all of the questions with no penalty. Similarly, the nature of your responses positive or negative will not lead to any benefit or consequence. Do you mind if we record this discussion as backup in case we miss any important points? Do you have any questions? Can we begin now? Introduction 1. Please give us a background of your organization and what it does. 2. How long has your organization been involved with the ZPI project? Probe: Start and end date of the involvement. 3. Please specify in what activities was your organization involved with the ZPI project. Key Evaluation Question 1: To what extent did the project achieve the planned objectives and results in HIV prevention among vulnerable and most at risk populations? 1. In your opinion, what have been the major achievements of the ZPI project? Probe: (i) To what extent were the planned activities and interventions achieved? (ii) Explain how service(s) provided under these activities met community needs. (iii) How accessible were these services? 63 2. To what extent did the interventions address gender issues that expose women to HIV/AIDS? Probe: (i) What activities worked best? How? (ii) Explain the quality of gender activities provided by the program. (iii) How accessible were these services? 3. To what extent did ZPI meet the needs of the beneficiaries in the communities in which the project operated? Probe for each category: (i) Adolescent/Youth; (ii) PMTCT – Mothers; (iii) Men; (iii) Women and (v) Local Leaders 4. What factors facilitated the achievement of results? What were their effects on project achievement? 5. What factors inhibited the achievement of results? What were their effects on project implementation? Key Evaluation Question 2: To what extent were the project design, implementation, and management effective and why? 1. In your opinion, what components of the project design were the most effective and why? Probe: (i) How relevant were the interventions to the achievement of the planned outcomes? (ii) How relevant was the project partnership to the achievement of the planned outcomes? Explain. 2. In your opinion, what components of the project implementation were the most effective and why? Probe: (i) What have been the effects of these components on the project outcomes? (ii) How adequate was the implementation of the planned interventions? Explain. 3. In your opinion, what components of the project management were the most effective and why? Probe:What have been the effects of these components on the project outcomes? 4. Overall, what are the lessons learned on project design, implementation, and management and their effects in accomplishing the projects targets and outcomes? 5. In your opinion, which performance monitoring methods and tools were effective in tracking changes in HIV prevention among vulnerable and most at risk-populations? 6. In your opinion, what were the innovative activities implemented by ZPI? Explain. Key Evaluation Question 3: What progress has been made towards ensuring the sustainability of ZPI’s approaches? 1. To what extent did the project influence strategy, programming, and policy at the national, sub￾national and community levels? Explain. 2. To what extent did external stakeholders (e.g. government counterparts, the Global Fund to Fight AIDS, TB and Malaria, etc.) adopt ZPI approaches? 3. To what extent did the project build the organizational and technical capacity of local implementing partners and communities? Explain. 4. What interventions have been successfully incorporated into the exiting service delivery at the community and health facility levels? Probe: Which intervention and activities are likely to continue beyond the life of the project? Explain how you see these intervention and activities being continued. 64 Key Evaluation Question 4: To what extent are the strategic information (SI) activities serving to inform the planning, implementation and monitoring of the project? 1. What data were used to inform ZPI project in: a) planning; b) implementation; and c) monitoring for results and decision-making? Probe: (i) How did you use the data? (ii) How often did you use the data? 2. What were the key data sources? Probe: (i) How often did you collect the data? (ii) To what extent was the monitoring and evaluation system used for program improvement? 3. Was the data collected, transmitted, collated and interpreted in a timely manner to inform project decision-making? Explain. 4. How are data and information generated by the project used by its consortium partners, other NGOs, CBOs, GRZ, USAID, or donors? Probe: How useful and appropriate was the data and information to ZPI consortium managers, NGOs, government, CBOs, or project managers? In what ways? 5. What was the quality of study designs, data collection and findings? Probe: How were study findings used for planning and implementation? 65 ZPI End-line-Performance Evaluation FGD Guide for Beneficiaries Province: _____________________________________________________________________________ District: ______________________________________________________________________________ Intervention site: ______________________________________________________________________ FGD venue: __________________________________________________________________________ Date: ______________________________________________________________________________ 1. What type of services have you received from the ZPI (will describe ZPI in context) at: a. Community level: Probe: (i) Who provides the services at community level? (ii) In your opinion, explain how the service(s) provided by these cadres meet your needs. (iii) How accessible were they? b. Health facility level: If not mentioned spontaneously. Probe: For HTC, PMTCT, MC, RH/FP? 2. From your observation, to what extent did the project activities address: a. Men in HIV prevention: To what extent were men involved in project HIV prevention activities? Explain your answer. Probe: (i) What activities worked best? How? (ii) In your opinion, explain the service(s) received/provided under these activities meet your needs. (iii) How accessible were these services? b. Women in HIV prevention, including PMTCT: To what extent were FP services were FP services provided to PMTCT mothers? To what extent were men involved in PMTCT activities? Explain your answers. Probe: (i) What activities worked best? How? (ii) In your opinion, explain the service(s) received/provided under these activities meet your needs. c. Couple in HIV prevention within marriage or partnerships: To what extent were couples involved in project activities? Explain your answer. Probe: (i) What activities worked best? How? (ii) In your opinion, explain the service(s) received/provided under these activities meet your needs. (iii) To what extent were men in these relationships involved in couple HIV prevention activities? d. Adolescents/Youth in HIV prevention: Probe: How were adolescents or youth involved in HIV prevention? What activities were used to engage youth in prevention? Probe: (i) What specific activities were targeted at girls? (ii) Overall, what activities worked best? How? (iii) In your opinion, explain the service(s) received/provided under these activities meet your needs. e. Alcohol and substance abuse in HIV prevention: Probe: Who was targeted in these activities? Probe: (i) What activities worked best? How? (ii) In your opinion, explain the service(s) received/provided under these activities meet your needs. f. Gender-based violence in HIV prevention: Probe: Who were targeted in these activities? Probe: (i) What activities worked best? How? (ii) In your opinion, explain the service(s) received/provided under these activities meet your needs. (iii) To what extent were men involved in gender-based violence prevention activities in this project? 66 g. Economic vulnerability in HIV prevention: Probe: Who was targeted in these activities? Probe: (i) What activities worked best? How? (ii) Explain how the service(s) received/provided under these activities meet your needs. h. Other populations: Who else was targeted by the project? 3. From your observation, to what extent did the project activities address workplace HIV prevention program? Explain your answer. Probe: (i) What activities worked best? How? (ii) In your opinion, explain how the service(s) received/provided under these activities meet your needs. (iii) How accessible were these services? 4. From your observation/ experiences, what changes have you noticed as a result of project services/activities: a. At community level: Probe: Explain and specify the changes. b. At individual level: Probe: Explain and specify the changes. 5. From your observation/ experiences, what activities have been successfully incorporated into the exiting service delivery at the community level? Probe: Which activities are likely to continue beyond the life of the project? Explain how you see these activities being continued. 6. Based on your experience and/or observations, how can HIV prevention program be strengthened to increase: a. Access to different target groups b. Uptake of services c. Consistent access and use d. Expansion to other areas 7. What suggestions would you recommend to strengthen the ZPI project components? Please explain. 67 FGD Guide for Community Volunteers/Leaders Province: _____________________________________________________________________________ District: ______________________________________________________________________________ Intervention site: ______________________________________________________________________ FGD venue: __________________________________________________________________________ Date: ______________________________________________________________________________ 1. What HIV prevention services do you provide under the ZPI project at the community level? 2. a. Who are the main beneficiaries/users of each of the HIV prevention services? For each service, probe for gender & age of beneficiaries. b. To what extent does the project adequately address the HIV prevention needs of both men and women? Explain your answer. c. To what extent were men involved in the HIV prevention intervention implementation? Probe: What aspect of men involvement worked well? Explain. d. To what extent were women involved in the HIV prevention intervention implementation? Probe: What aspect of women involvement worked well? Explain. e. How can future projects effectively involve both men and women as implementers of community level HIV prevention activities? 3. Under the ZPI project: a. What were your achievements for each service/activity? Probe: Explain these achievements. To what extent did you achieve the targets? b. What factors facilitated the achievements of the project targets? Explain. c. What factors hindered achievement of project targets? Explain. 4. Based on your experiences, what suggestions do you have to strengthen service provision? 5. Describe any observed changes in each beneficiary category (as listed in 2a) as a result of ZPI activities since you started providing the services? Probe: (i) Service uptake, (ii) Preventive behaviors targeted by ZPI? 6. a. What type of records/reports of services provided did you keep? b. Describe your experiences with data capture at your level of service delivery. c. How have you used the data to inform decision-making in relation to the services you provide? d. Where do you send the record/report of services provided? Probe: What type of feedback do you receive for the records/services you submitted? e. How can data capture at your level be strengthened? Explain. f. How can data use at your level be strengthened? Explain. 7. From your observation/ experiences, what activities have been successfully incorporated into the exiting service delivery at the community level? Probe: Which activities are likely to continue beyond the life of the project? Explain how you see these activities being continued. 8. Overall: a. What aspects of the ZPI project worked well? Explain your answer. b. What did not work well and why? c. How can aspects that did not work well be improved? d. What suggestions would you recommend to strengthen the ZPI project components? Please explain. 68 END-OF-PROJECT EVALUATION OF THE USAID FUNDED ZAMBIA PREVENTION LED INITIATIVE (ZPI) PROJECT Introduction and Consent Form Good day. My name is ___________________, and we are conducting an evaluation of the Zambia Led Prevention Initiative Project in collaboration with the Government of Zambia, USAID and other stakeholders. The purpose of the mid-term performance evaluation of USAID/Zambia Led Prevention Initiative Project (ZPI) implemented by FHI360 is to determine the effectiveness of the interventions of ZPI and document what has worked well and what has not. Lessons from this evaluation will be integrated in future programming of USAID, and the Ministry of Health (MOH) to support national efforts in strengthening the HIV prevention in Zambia. You were selected as a Key Informant to provide information for this evaluation. The information collected will only be used for the evaluation. All the information is strictly confidential. [Interviewer collects signed consent forms]. I would also like to clarify that this interview is voluntary and that you have the right to withdraw from interview at any point without consequence. Thank you very much. At this time, do you have any questions? Are you willing to participate in this study? Yes 1) Proceed. No 2) Thank the KI and STOP HERE. May I begin the discussion now? Yes 1) Continue with the Key Informant Interview. No 2) STOP HERE. Start Time: ____:____ Interviewee signature _____________________________ Date ___________________ Interviewer signature _____________________________ Date ____________________ Thank you 69 END-OF-PROJECT EVALUATION OF THE USAID FUNDED ZAMBIA PREVENTION LED INITIATIVE (ZPI) PROJECT Person’s name: As a member of this evaluation team I understand that I may have access to confidential information about study sites and participants. By signing this statement, I am indicating my understanding of my responsibilities to maintain confidentiality and agree to the following: I understand that names and any other identifying information about the study sites and participants are completely confidential. I agree not to divulge, publish, or otherwise make known to unauthorized persons or to the public any information obtained in the course of this evaluation project that could identify the persons who participated in the study. I understand that all information about the study sites or participants obtained or accessed by me in the course of my work is confidential. I agree not to divulge or otherwise make known to unauthorized persons any of this information. I understand that I am not to read information about study sites or participants, or any other confidential documents, nor ask questions of the study participants for my own personal information but only to the extent and for the purpose of performing my assigned duties on this evaluation project. I agree to notify the Project Director/Designee immediately should I become aware of an actual breach of confidentiality or a situation that could potentially result in a breach, whether on my part or on the part of another person. ______________________________ ________________ _____________________ Signature Date Printed name 70 ANNEX III: EVALUATION PLAN AND ANALYSIS MATRIX END-OF-PROJECT EVALUATION OF THE USAID FUNDED ZAMBIA PREVENTION LED INITIATIVE (ZPI) PROJECT Key Evaluation Questions Sub Questions Methods Sampling Selection Data Analysis Document Review/ Secondary Data Review KII FGD 1. To what extent did the project achieve the planned objectives and results in HIV prevention among vulnerable and most at risk populations? a. What were the main achievements of the project? b. Did the planned activities/ interventions achieve the planned results? c. What was the quality of the project outputs (or activity outputs? d. To what extent did the interventions address gender issues that expose women to HIV/AIDS?  Desk review of ZPI Work plans, quarterly and annual reports  Secondary data sources for health outcomes such as baseline & midline studies; operations research study; Zambia DHS 2007 and 2013 preliminary data.  Pre-existing data including PMP data since 2010.  M&E plans & analysis of ZPI national and provincial program data including consortium member program data.  PEPFAR Report. KII implementin g partners (IPs) managers and staff; USAID/Zam bia managers; MOH. - All intervention provinces and selection of one or more districts in each province. Selective national level stakeholders, provincial and district teams. Performance monitoring system data (2010-2014). Secondary data sources: HMIS, DHS and special and operations research studies. Comparative analysis: observed and reported outputs and reported outcomes with ZPI and PEPFAR indicators disaggregated by gender and location. Baseline and midline data comparison; compare with DHS on similar indicators. Use of a three-point rating system (positive change, negative change and unchanged) for outcomes and (achieved, not achieved, partially achieved) for output indicators. Gender Assessment Tool. Qualitative analysis: content and thematic analysis of KII data in relation to capacity developed and beneficiary outcomes. 71 Data triangulation e. To what extent did ZPI meet the needs of communities? f. What factors facilitated or inhibited the achievement of results? Document review: ZPI Quarterly & Annual Reports; budget allocations; portfolio management documents. KII IP managers and staff; USAID/Zam bia managers: MOH at central, province, district and community levels. FGDs with beneficiaries at the community level: men and women of reproductive age (in separate groups & equal gender presence); pregnant women; youth aged 12-24 years; PLHA. Community volunteers. ZPI quarterly and annual reports. Selective national level stakeholders, provincial and district teams. Selection of participants for FGDs per community (6-8 participants; equal for gender). Purposive selection of district and health facilities: population size and equal representation of urban/rural districts. Content analysis of reports using a structured checklist. Qualitative analysis: content and thematic analysis of KII and FGD interview data in relation to capacity developed and beneficiary outcomes (using ATLAS.ti). Quantitative data analysis using STATA. Use of Attributes of Sustainable Health Programs Framework (Bongiovanni A. et al. 2012). Data triangulation g. How has the implementing partner dealt with those challenges? ZPI Quarterly & Annual Reports. KII IP managers and staff; USAID/Zam bia managers: MOH. Community volunteers. Selective national level stakeholders, provincial and district teams. Purposive selection of district and health facilities. Systematic content analysis of reports using a structured checklist. Qualitative analysis: content and thematic analysis of KII and FGD interview data in relation to capacity developed and beneficiary outcomes. Data triangulation. 2. To what a. What are the best practices Field Visit reports. Project performance reports. KIIs with FHI360 and FGDs with Community Key personnel interviewed 72 extent were the project design, implementation, and management effective and why? used during project design, implementation and management? These reports will be reviewed for best practices and lessons learned during project design, implementation and management. Best practice documentation, where available, will be reviewed. Emphasis will also be placed on whether lessons learned where effectively documented, communicated and replicated at various levels (national, subnational and community). its sub￾partners. (Population Council, CHAMP, CMMB, ZHECT and Afya Mzuri). KIIs with key stakeholders (NAC, MCDMCH, MOH, Other USAID/PEPF AR funded projects). These KIIs will seek to get these respondents knowledge of the key project lessons have been and any best practices identified through ZPI and what the results of these best practices and lessons learned are at national beneficiaries and volunteers will be conducted to get their perceptions and views on what they feel the unique contribution s of the program have been and what best practices have been at the community level. The FGDs will also solicit what lessons have been learned at community level from ZPI. Gender perspectives will be sought for best practices. during KIIs from FHI, sub-partners and key stakeholders. Content and theme analysis of qualitative findings. KIIs and FGDs. Quantitative analyses by partner to compare different approaches and their results. b. What are the lessons learned? c. During project design, implementation and management? 73 and subnational levels. d. What changes occurred during this activity, both in the external environment and /or internal to the activity, and where they may have had a bearing on activity outputs and outcomes? Quarterly progress reports. Project pe rfo r man ce. Quan t i t a t i ve analy s es by partner. KIIs with FHI360 and its sub - partners. (Population Council, CHAMP, CMMB, ZHECT and Afya Mzuri ) . KIIs with key stakeholders (NAC, MCDMCH, MOH, Other USAID/PEPF AR funded projects). FHI360, sub - partners and stakeholders will be able to inform on changes they are aware of, the implications of the changes on the project and how these changes FGDs with community beneficiaries and volunteers will elicit whether they noticed any changes during the course of project and what effects these changes had at the community level. Content and theme analysis of qualitative findings . KII s and FGD s . 74 were addressed. e. How effective were the tools used to track changes in HIV prevention among vulnerable and most at risk￾populations. f. Were these were good predictors of organizational success? M&E tools, Progress Reports, Baseline and Midline survey reports, Surveys tools, PMP The DQA shall be based on structures and systems kept in place to ensure effective data collection, analysis, storage and reporting. KIIS with key program staff of FHI360 and sub-partners involved in reporting and M&E. Questions will seek to understand the M&E systems with reference to validity, reliability, precision, integrity, timeliness and accessibility. The DQA survey will be administered (scored) concurrently during KIIs. - DQA will be done for FHI360 and its sub￾partners (Population Council, CHAMP, CMMB, ZHECT and Afya Mzuri). Content and theme analysis of qualitative findings. KIIs and FGDs. Scoring of DQA and qualitative analysis of major system features and characteristics detailing strengths and weaknesses. To assess the effectiveness of the M&E system as means of capturing and sharing data across FHI360 and its sub-partners. g. What was the effectiveness and efficiency of ZPI in HIV prevention and areas of ZPI baseline and midline reports, project performance. Quantitative analyses by partner. KIIs with FHI360 & sub-partners. (Population Council, CHAMP, CMMB, FGDs with community beneficiaries and volunteers will elicit how ZPI is Content and theme analysis of qualitative findings. KIIs and FGDs. Quantitative analysis of baseline and midline data. 75 comparative advantage? ZHECT and Afya Mzuri). KIIs with key stakeholders (NAC, MCDMCH, MOH, Other USAID/PEPF AR funded￾projects). The stakeholders will address these from national and subnational perspectives. perceived to differ from other HIV programs that have been implemented in the community and its unique strengths and contribution. Gender perspectives will be sought for comparative advantage. 3. What progress has been made towards ensuring the sustainability of ZPI’s approaches? a. Did the project influence strategy, programming, and policy at the national, sub-national and community levels? Document review of ZPI project quarterly reports. Data abstraction ZPI results against PMP & PEPFAR indicators PEPFAR reports. KIIs with FHI360 and its sub￾partners (Population Council, CHAMP, CMMB, ZHECT and Azuri Mzuri to obtain in￾depth information and knowledge of ZPI plans, FGDs with beneficiaries & community leaders to obtain perceptions about how ZPI assisted and influenced health activities aimed at HIV prevention, HCT, gender KIIs targeted to national, provincial, district, & community leaders. FHI and sub￾partners, all key stakeholders at national/provincia l/district levels. Content and theme analysis of KII and FGD data. 76 experience, and results at all levels. KIIs with stakeholders (at all levels) to obtain opinions, perspectives and reports about ZPI influence on policy and programmin g at all levels, and effects of such influence. equity and GBV. alcohol/ substance abuse, sexual abuse, mental health, PMTCT. Community volunteers. b. To what extent did external stakeholders (such as government counterparts and the Global Fund to Fight AIDS, TB and Malaria, etc.) adopt ZPI approaches? Document review of ZPI quarterly reports. KIIs with national (government and civil society) and international (Global Fund and other donors) stakeholders to obtain information and opinions about approaches, and cross over or adoption of . KIIs targeted to key national-level stakeholders as outlined under Questions 1 & 3 to obtain perceptions and opinions about how ZPI collaborated with government counterparts (NAC/MOH/MC DMCH) and the Global Fund (PR and CCM). KIIs targeted to the Global Fund (CCM & PR) and Content and theme analysis of KIIs. 77 ZPI approaches. other donors to determine if and how ZPI approaches were adopted. c. What is the evidence that organizational and technical capacity was built among local implementing partners and communities? Document review of ZPI quarterly progress reports, field visit reports, and M&E measures of capacity built. KIIs with stakeholders (all levels) to obtain opinions, perspectives and reports about extent that ZPI built capacity among local implementin g partners and communities . FGDs with local IPs and community leaders to obtain perceptions about how ZPI built their capacity and what type of capacity was built. KIIs with selected national and provincial stakeholders. FGDs with district and community partners and leaders in selected districts and communities. Content and theme analysis of KIIs and FGDs. Data triangulation. d. What is the evidence that ZPI models will continue to be implemented beyond the life of the project? KIIs with stakeholders (all levels) to obtain opinions, perspectives continuity of ZPI approaches beyond the life of the project. FGDs with local IPs and community leaders to obtain perceptions about if and how the continuity of the activities will be assured. KIIs with selected national and provincial stakeholders. FGDs with district and community partners and leaders in selected districts and communities. Content and theme analysis of KIIs and FGDs. Data triangulation. e. What evidence of the ability to leverage Document review of ZPI quarterly progress reports, field visit reports, and financial reports. KIIs with stakeholders (all levels) to obtain FGDs with local implementin g partners KIIs with selected national and provincial stakeholders, Content and theme analysis of KIIs and FGDs. Data triangulation. 78 support for the project? information about support leveraged (financial, political, new partnerships etc.). and community leaders to obtain information about support leveraged at community levels. FGDs with district and community partners and leaders in selected districts and communities. 4. To what extent are the strategic information (SI) activities serving to inform the planning, implementat ion and monitoring of the project? 4.1 What SI were used to inform ZPI: - Planning - Implementation - Monitoring for results and decision-making? Project design document review to explore the extent to which SI informed the design (e.g. ZDHS, other national data, reports from other projects implementing similar activities). M&E plan review to explore if the activities are implemented on time & all planned indicators are captured and reported in the ZPI quarterly/ annual reports. Exploration of how the information captured/reported in the quarterly reports is used by the ZPI project. Data from other projects implementing related activities. M&E Plan review to explore if project activities are implemented on time & all planned indicators are KIIs with ZPI consortium and development partners: FHI360 (prime), Population Council (PC), CHAMP, CMMB, ZHECT and Afya Mzuri (subs). KIIs with GRZ NAC, MOH/ HIV/AIDS, MCDMCH, USAID/ Zambia. KIIs with ZPI consortium (NAC, MOH, FGDs conducted with health facility & community – based service providers to explore their experiences with capturing the data, including: - Whether the data at the service delivery level is used at that level & if yes, how? - Where they send the data? -Do they receive feedback? KII respondents will be purposively selected to ensure that the views of key project implementers and stakeholders are captured. Data collected for each of the sub-questions will be analyzed by each of three methods, then reviewed and coded by emerging themes to find out the extent to which strategic information was used to inform project planning, implementation including monitoring activities. The analysis will also focus on identifying SI areas that may require strengthening for better quality SI activities and project utilization. 4.2 To what extent did the ZPI project use relevant and adequate SI to inform project: - Design - Implementation 79 - Monitoring of activities and evaluation plans captured and reported in the ZPI quarterly/ annual reports MCDMCH, USAID/ Zambia) to explore their views on the extent to which national, provincial, district, and community￾level data has been used to inform ZPI project activities. KII with ZPI consortium to see how SI use can be improved. 4.3 What was the quality of study designs, implementation and results? Project proposal/design document including M&E Plan, PMP documents. KIIs with ZPI consortium: FHI360 (prime), Population Council (PC), CHAMP, CMMB, ZHECT and Afya Mzuri,(subs) Analysis of the M&E plan implementation. Assessment of PMP reports. KII data. Review/analysis of study 4.5 Is the ZPI designs and results. project SI generated in a timely manner to inform project decision-making? M&E plan. M&E database at the national & provincial levels. PMP reports. Project quarterly & annual reports. 80 M&E plan review to if the project activities were implemented on time & all planned indicators were captured and reported in the ZPI quarterly/ annual reports. KIIs with GRZ-NAC, MOH/ HIV/ AIDS, MCDMCH, USAID/ Zambia. The KIIs will: -Assess the extent to which the study adopted best practices for project designs, implementati on M&E in informing project decision￾making & achieving the planned activities and expected outcomes. -Explore extent to which ZPI SI was adopted by the ZPI consortium & key stakeholders including the 4.4 To what extent was the M&E system used for program improvement? M&E plan. M&E database at the national & provincial levels. Quarterly & annual reports. Baseline & midline reports. 4.5 How are data and information generated by ZPI used by the project consortium, other NGOs, CBOs, government, USAID, and donors? 4.6 How useful and appropriate was the strategic information to the ZPI consortium managers, NGOs, government, CBOs, and project M&E plan. M&E database at the national & provincial levels. PMP reports. Project quarterly and annual reports. 81 managers? In what ways? MOH & MCDMCH. 82 ANNEX IV: LIST OF DOCUMENTS REVIEWED Central Statistical Office (CSO), Ministry of Health (MOH), Tropical Diseases Research Centre (TDRC), University of Zambia and Macro International Inc. 2009. Zambia Demographic and Health Survey 2007. Calverton, Maryland, USA: CSO and Macro International Inc. FHI360. 2012. Gender Integration Framework. How to Integrate Gender in Every Aspect of Our Work. FHI360. Kim J. et al., 2008. Exploring the role of economic empowerment in HIV prevention. AIDS 22 Suppl 4: S57-71. Ministry of Health. 2009. Zambia Demographic and Health Survey 2007 revised 2009. Central Statistical Office. National HIV/AIDS, STI, TB Council (NAC). 2010. National Aids Strategic Framework (NASF). 2011- 2015. Towards improving the quality of life of the Zambian People. National HIV/AIDS, STI, TB Council (NAC). National HIV/AIDS, STI, TB Council (NAC). 2010. Zambia Global Fund Round 10-HA Proposal. National HIV/AIDS, STI, TB Council (NAC). 2014. Zambia Country Report Monitoring the declaration of commitment on HIV/AIDS and the University Access. Government of the Republic of Zambia. President’s Emergency Plan for AIDS Relief (PEPFAR). 2009. Next Generation Indicators Reference Guide. US Government. Pronyk, A.E. et al. 2008. A combined microfinance and training intervention can reduce HIV risk behavior in young female participants. AIDS 22(13): 1659-1665. Population Council. 2012. Critical Conversations with Girls Adolescent Girls in Lusaka, Zambia: Learning from Girls about Social, Health, and Economic Empowerment. Zambia Led Prevention Initiative (ZPI). Population Council. 2014. Critical Conversations #2: Learning from Grow Girls about Social, Health, and Economic Empowerment. Zambia Led Prevention Initiative (ZPI). Population Council. 2014. HIV/AIDS organisational networks analysis study. Zambia Led Prevention Initiative. Republic of Zambia. 2010. Global Fund Grant Proposal for Round 10. August 20, 2010 Republic of Zambia. 2011. National AIDS Strategic Framework 2011-2015. November 25, 2010. Republic of Zambia. 2014. Zambia Country Report: Monitoring the Declaration of Commitment on HIV and AIDS and the Universal Access. Biennial Report. March 31, 2014. United Nations Development Programme (UNDP). 2013. Millennium Development Goals Progress Report Zambia 2013. United Nations Development Programme (UNDP). United Nations Office on Drugs and Crime, World Drug Report 2014 (United Nations Publication, Sales No. E.14.XI.7). 83 Unknown. 2006. HIV/AIDS Indicators Country Report Zambia 1992-2007. USAID. 2010. Modification of Contract. 12/03/2010. US Government. USAID. 2010. Task Order, Contract No: GHS-I-00-07-00008-00, Order No: GHS-I-02-07-00008-00, Effective date Aug 16 2010-Dec 13 2014. US Government. USAID. 2011. Modification of Contract. 11/22/2011. US Government. USAID. 2012. Modification of Contract. 10/23/2012. US Government. USAID. 2013. Modification of Contract. 04/26/2013. US Government. USAID. 2013. Modification of Contract. 10/01/2013. US Government. USAID. 2013. Modification of Contract. 10/02/2013. US Government. USAID. 2014. Modification of Contract. 03/27/2014. US Government. Zambia Led Prevention Initiative. 2010 -201_. Work plans #1-5: August 16-December 31, 2010 – January-December 2014. USAID. Zambia Led Prevention Initiative. 2010. Quarterly Reports # 1-15: October 1-December 31, 2012 to January-March 2014. USAID. Zambia Led Prevention Initiative. 2012. Household survey to evaluate the Zambia Led Prevention Initiative: Baseline report. Lusaka, Zambia: USAID. Zambia Led Prevention Initiative. 2013. Motivations for entering volunteer service and factors affecting productivity: A mixed method survey of STEPS-OVC volunteer HIV caregivers in Zambia. Lusaka, Zambia: USAID. Zambia Led Prevention Initiative. 2010. Performance Monitoring Plan (PMP), A tool for monitoring and evaluating ZPI Interventions. USAID. Zambia Led Prevention Initiative. 2013. Performance Monitoring Plan (PMP), A tool for monitoring and evaluating ZPI Interventions. USAID. Zambia Led Prevention Initiative. 2010-2014. Work plans #1-5 August 16 to December 31, 2010 - January 1-December 31, 2014. USAID. Zambia-led Prevention Initiative. 2014. Midline evaluation of the Zambia Led Prevention Initiative (ZPI). Lusaka, Zambia: USAID. 84 ANNEX V: LIST OF PERSONS INTERVIEWED KEY INFORMANT INTERVIEWS PROVINCE LUSAKA Name Se x Organization/Contact Details Position Date Interview ed Batuke Walusiku Mwewa F Catholic Medical Missions Board(CMMB), Mobile: 0978773067 Country Director 12/08/2014 Alice Chitofwa F AATAZ – Anti Aids Teachers Association of Zambia Executive Director 12/08/2014 Joseph Matafwali M AATAZ – Anti Aids Teachers Association of Zambia Accountant 12/08/2014 Mavis Banda F AATAZ – Anti Aids Teachers Association of Zambia Project Manager 12/08/2014 Philip Chimponda M SHARPZ, Mobile: 099323418 Coordinator 13/08/2014 O’Jay Mwenya, M SHARPZ Counselor 13/08/2014 Kennedy Mutale M SHARPZ Counselor 13/08/2014 Chisha Mwambazi M SHARPZ M&E Advisor 13/08/2014 Dr. Caroline Phiri F Ministry of Community Development, Mother and Child Health Director 13/08/2014 Dr Nambao F Ministry of Community Development, Mother and Child Health 13/08/2014 Linda Nonde F Community Support for Health –CSH Chief of Party 13/08/2014 Ethel Kopulande F Community Support for Health –CSH FP/RH Specialist 13/08/2014 Answell Chipukuma M Community Support for Health –CSH BCC Advisor 13/08/2014 Tina Moyo F Population Council, Mobile: 0974001440, tmoyo@popcouncil.org Operations research and dissemination Officer, 11/08/2014 Chabu Kabanga M Population Council,Mobile: 0969272387, ckangale@popcouncil.org 11/08/2014 Clement Chileshe M Olympic Youth Development Centre(OYDC) Centre Director 14/08/2014 Mildred Siabeenzu F Olympic Youth Development Centre(OYDC), Mobile: 0962248163 ZPI Coordinator 14/08/2014 Sombwa Musunsa M Olympic Youth Development Centre(OYDC),Mobile:09557549048 Programme Manager 14/08/2014 Deogratius Chileshe F Olympic Youth Development Centre(OYDC),Mobile:0977278504 Outreach worker for GROW girls based at ZPI 14/08/2014 85 Lyson Zulu M Olympic Youth Development Centre(OYDC),Mobile:0975145069 Health &Education Officer 14/08/2014 Rosanna Nyendwa Sammon F CHAMP, Mobile: 0966761571/0950 263634 Managing Director 18/08/2014 Michael Welsh M ZPCTII, Mobile: 0961 438599 Country Director 18/08/2014 Chileshe Chilangwa F Zambia-led Prevention Initiative (ZPI) Mobile: 0974778315, cchilangwa@fhi360.org Deputy Country Director, Chief of Party, 19/08/2014 Felly Nkweto Simmonds F Zambia-led Prevention Initiative (ZPI) Mobile: 0966 700 352 fsimmonds@fhi360.org Senior Technical Advisor 19/08/2014 Baron Banda M Zambia-led Prevention Initiative (ZPI) Mobile: 0974778315, babanda@fhi360.org Director Finance and Administration 23/07/2014 Kennedy Chipampe M Zambia-led Prevention Initiative (ZPI) Mobile: 0974778315, kchipampe@fhi360.org M&E Officer, Programme Quality 23/07/2014 Arlene Phiri F USAID/Zambia HIV Prevention Advisor 18/08/2014 86 KEY INFORMANT INTERVIEWS PROVINCE: SOUTHERN Name Sex Organization/Contact Details Position Date Interviewed Cornwell Handema M Brethren in Christ Church (BICC) Executive Director 05/08/2014 Veronica Mweemba, F District Aids Task Force, DATF, Mobile:0955837545 05/08/2014 Munyansa Siatindwa F District Aids Task Force, DATF, Mobile:0977781548 05/08/2014 Vincent Sikanyeela M District Commissioner’s Office, Mobile:0974463263 05/08/2014 Keizluck Mweemba M District Health Office, Information’s Dept, Mobile: 0977678125 05/08/2014 Florence Beenzu F NZP+, Mobile: 0978058873 05/08/2014 Grace L. Chirwa F NZP+, Mobile: 0977619652 HIV Counselor 05/08/2014 Pondie Mudenda F NZP+, Mobile: 0977876727 Psycho-social Counselor 05/08/2014 Patricia M. Mainza F Choma General Hospital, Mobile: 0974233911 Nurse 05/08/2014 Stephen Lutangu M Victim Support Unit, Police Mobile: 0978223875 Police officer 05/08/2014 Chief Singani M Singani Chiefdom, Mboole Area Chief 05/08/2014 18 Village headmen M Singani Chiefdom, Mboole Head Men 05/08/2014 87 PROVINCE: EASTERN Name Sex Organization/Contact Details Position Date Interviewed Ernest Kabulosando M District Aids Task force(DATF) Chairperson 05/08/2014 Charles Ngala M District Aids Coordination Advisor (DACA) 05/08/2014 Solomon Banda M Katete District Women’s Development Association, Mobile:0977352736 M&E 05/08/2014 Vaines Phiri F Katete District Women’s Development Association Project Coordinator 05/08/2014 KEY INFORMANT INTERVIEWS PROVINCE: WESTERN Name Sex Organization/Contact Details Position Date Interviewed Colins Lilembalemba M District Aids Coordinator Advisor(DACA) HIV Technical Officer 28/07/2014 Eugene Nyambe Mukelabai M Peoples Participation Service(PPS) M&E Officer 28/07/2014 Fine Nasilele M Peoples Participation Service(PPS) M&E Officer 28/07/2014 David Mubita M Peoples Participation Service(PPS) M&E Officer 28/07/2014 Martha Hambwezya F Young Women Christian Association (YWCA) Field officer 28/07/2014 Purity Kalumba F Young Women Christian Association (YWCA) Accounts clerk 28/07/2014 Chanda Bwelele M Treatment, Advocacy & Literacy Campaign (TALC) Coordinator 28/07/2014 Lubosi Mushala M Treatment, Advocacy & Literacy Campaign (TALC) Program officer 28/07/2014 Lutangu Mwilima F Treatment, Advocacy & Literacy Campaign (TALC) Finance officer 28/07/2014 Jennifer Mulemwa F Treatment, Advocacy & Literacy Campaign (TALC) Adherence Support Worker 28/07/2014 Dr Francis Liywali M Provincial Health Office (PHO)/ District Health Office (DHO), Mobile:0977713151 District Health Office 29/08/2014 Chief Sambiana M Area Chief 30/07/2014 Chief Mamuchisana M Area Chief 30/07/2014 Headman Imakonda M Headman 30/07/2014 Parent Hanyama M Victim Support Unit (VSU) 31/07/2014 88 Agnes Gumbo F Vizenge Clinic Nurse in Charge 06/08/2014 Dr Malama M Provincial Medical Office PMO 06/08/2014 Stembile Sakala F Young Happy Healthy and Safe (YHHS) Programme Officer 07/08/2014 Ann Chiseni F Chisomo Community Programme Manager 07/08/2014 Kennedy Tembo M ZPI Acting Provincial Coordinator 08/08/2014 PROVINCE: COPPERBELT Name Sex Organization/Contact Details Position Date Interviewed Prisca Kambole F Pro-Life Advancement and Education Project(PLAEP),Mobile: 0955950184, priscakambole@gmail.com Executive Director 11/08/2014 Alinuswe Mwamuilima M Pro-Life Advancement and Education Project(PLAEP) Finance Officer 11/08/2014 PROVINCE: LUAPULA Name Sex Organization/Contact Details Position Date Interviewed Dr Chibwe M Ministry of Health Provincial Medical Officer 29/07/2014 Dr Mulenga Mavuto M Ministry of Health Community Medical Officer 29/07/2014 Katai Bright M Catholic Medical Missions Board(CMMB) Provincial Coordinator 29/07/2014 Fr. Protazio M. Lungu M Caritas Director 29/07/2014 Alick S. Zulu M Caritas M&E Officer 29/07/2014 Catherine Mwale Shimete F Caritas Program Officer 29/07/2014 89 James C. Mulenga M District Commissioner’s Office District Administrative Officer 29/07/2014 Martin Mulenga M District Aids Task Force (DATF) Chairperson 29/07/2014 Bwalya Paul Nyambe M NZP+ Coordinator 29/07/2014 Senior Chief Mwewa M Mundubi Chieftainship Senior Chief 31/07/2014 Evans Chala M Mundubi Chieftainship Headman, Sambo 31/07/2014 Mwelwa M. Chabu M Mundubi Chieftainship Headman, Chilupula 31/07/2014 Nicholas Kabaso M Mundubi Chieftainship Headman, Kabanga 31/07/2014 James Kabanga M Mundubi Chieftainship Headman, Mwafuli 31/07/2014 Emmanuel Kaluba M Mundubi Chieftainship Headman, Malombela 31/07/2014 90 ANNEX VI: NUMBERS OF RESPONDENTS FOR KIIS AND FGDS Interviewees by Province, Category and Gender 4 The percentage disaggregated by gender is calculated using the total of each group interviewed (i.e. stakeholder, volunteer, and beneficiary). Province Total Stakeholder4 Volunteers Beneficiaries Male Female Male Female Male Female # % # % # % # % # % # % Lusaka 31 10 50.0 10 50.0 11 100 _ _ _ _ _ _ Luapula (Samfya District) 54 14 93.3 1 6.7 11 57.9 8 42.1 6 30.0 14 70.0 Western (Mongu, Muoyo & Senanga Districts) 65 11 78.6 4 21.4 8 50.0 8 50.0 1 0 29.4 24 70.6 Eastern (Chipata & Katete) 90 5 55.5 4 44.4 6 66.7 3 33.3 2 9 40.3 43 59.7 Southern (Choma District) 202 23 79.3 6 0.7 58 49.6 59 50.4 1 6 28.6 40 71.4 Copperbelt (Kitwe District) 60 1 50.0 1 50.0 12 80.0 3 20.0 2 4.6 41 95.4 Total 502 64 71.1 26 28.9 10 6 56.7 81 43.3 6 3 28.0 162 72.0 91 ANNEX VII: PROVINCIAL DATA ANALYSIS CHARTS CB EP SP NWP LSK CP LP NP WP Year 1 16,877 8,208 18,398 26,056 16,017 Year 2 8,687 19,072 23,394 21,151 40,760 2,416 8,862 Year 3 37,583 28,742 42,535 28,323 64,062 11,013 35,196 2,272 7,102 Year 4 9,010 12,015 8,992 19,224 27,253 4,098 17,913 15,282 5,339 Total reach 72,157 68,037 93,319 94,754 148,092 17,527 61,971 17,554 12,441 72,157 68,037 93,319 94,754 148,092 17,527 61,971 17,554 12,441 - 20,000 40,000 60,000 80,000 100,000 120,000 140,000 160,000 Number Number of the targeted population reached with individual and/or small group level preventive interventions that are based on evidence and/or meet the minimum standards required CB EP SP NWP LSK CP LP NP WP Year 1 2,634 1,329 6,283 7,508 5,632 Year 2 3,166 4,121 5,244 7,879 12,407 559 1,286 Year 3 4,465 5,721 8,434 12,080 44,727 1,764 5,911 1,531 694 Year 4 0 0 0 0 0 0 0 0 0 Total reach 10,265 11,171 19,961 27,467 62,766 2,323 7,197 1,531 694 10,265 11,171 19,961 27,467 62,766 2,323 7,197 1,531 694 - 10,000 20,000 30,000 40,000 50,000 60,000 70,000 Number Number of the targeted population reached with individual and/or small group level preventive interventions that are primarily focused on abstinence and/or being faithful, and are based on evidence and/or meet the minimum standards required 92 CB EP SP NWP LSK CP LP NP WP Year 1 2,581 5,353 5,033 6,325 5,082 Year 2 4,838 9,754 6,538 7,245 10,306 1,031 6,650 Year 3 10,391 8,844 11,778 24,070 23,038 4,475 31,920 2,822 558 Year 4 2,118 1,600 2,537 6,942 617 616 8,862 1,789 287 Total reach 19,928 25,551 25,886 44,582 39,043 6,122 47,432 4,611 845 19,928 25,551 25,886 44,582 39,043 6,122 47,432 4,611 845 - 5,000 10,000 15,000 20,000 25,000 30,000 35,000 40,000 45,000 50,000 Number tested and counselled Number of individuals who received testing and counseling services for HIV and received their test results CB EP SP NWP LSK CP LP NP WP Year 1 264 3,235 3,599 3,311 2,314 Year 2 1,296 5,237 5,001 4,500 2,777 239 1,923 Year 3 11,207 14,224 13,366 7,427 7,922 1,169 7,080 335 74 Year 4 0 0 0 0 0 0 0 0 0 Total reach 12,767 22,696 21,966 15,238 13,013 1,408 9,003 335 74 12,767 22,696 21,966 15,238 13,013 1,408 9,003 335 74 - 5,000 10,000 15,000 20,000 25,000 Number reached Number of people reached by an individual, small group or community￾level intervention or service that explicitly addresses gender-based violence and coercion related to HIV/AIDS 93 CB EP SP NWP LSK CP LP NP WP Year 1 384 669 0 255 0 Year 2 195 1,475 183 0 813 187 0 Year 3 722 2,077 99 491 1,887 719 762 467 19 Year 4 473 101 170 12 0 120 511 2,951 0 Total reach 1,774 4,322 452 758 2,700 1,026 1,273 3,418 19 1,774 4,322 452 758 2,700 1,026 1,273 3,418 19 - 500 1,000 1,500 2,000 2,500 3,000 3,500 4,000 4,500 5,000 Number reached Number of eligible adults and children provided with economic strengthening services 94 International Business and Technical Consultants, Inc. 8618 Westwood Center Drive, Suite 400 Vienna, VA 22782 USA