December 2012 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by International Technical & Business Consultants, Inc. (IBTCI). The authors of this evaluation report are: Rose Schneider, Ya-Shin Lin, Robie Siamwiza, and Beyant Kabwe. EVALUATION Zambia Prevention, Care & Treatment Partnership Project II MID-TERM EVALUATION REPORT ZAMBIA PREVENTION, CARE & TREATMENT PARTNERSHIP PROJECT II MID-TERM EVALUATION REPORT USAID Contract Number: AID-RAN-I-00-09-00016 Task Order: AID-611-TO-12-00004 Zambia Prevention, Care and Treatment Partnership Project II (ZPCT II) Implemented by: International Business & Technical Consultants, Inc. (IBTCI) 8618 Westwood Center Drive, Suite 220 Vienna, VA 22182 Phone: +1 703.749.0100 Fax: +1 703.749.0110 http://www.ibtci.com DISCLAIMER: The authors’ views expressed in this document do not necessarily reflect the views of the United States Agency for International Development or the United States Government. ii CONTENTS Acronym List............................................................................................................................ ii Executive Summary ................................................................................................................. 1 Description of the Project Evaluated ............................................................................................... 1 Methodology ................................................................................................................................... 1 A. Evaluation Findings ......................................................................................................... 2 B. Recommendations ............................................................................................................ 7 C. Higher Level Issues ......................................................................................................... 9 INTRODUCTION.................................................................................................................. 10 D. Evaluation Purpose & Questions .................................................................................. 10 E. Project Background ........................................................................................................ 10 F. Methods and Limitations ............................................................................................... 12 G. Findings & Conclusions By Evaluation Task .............................................................. 15 Task I: Effectiveness ....................................................................................................................... 15 Task II. Efficiency .......................................................................................................................... 22 Adequacy and Appropriateness of Planned Activities and Corresponding Budgets ........................ 22 Cost-efficiencies ............................................................................................................................ 23 Modified Mix of Activities ............................................................................................................. 23 Task III. Cross-Cutting Principles ................................................................................................ 23 Practices, products, and tools to replicate ...................................................................................... 24 Task IV. Factors of Success ........................................................................................................... 25 Success in Implementation and Causative Factors ......................................................................... 25 Task V. Obstacles Affecting Progress ........................................................................................... 27 ZPCT II’s Key Obstacles in Program Management ....................................................................... 29 Approaches and Lessons Learned .................................................................................................. 29 Obstacles Requiring Policy Changes or Dialogue ........................................................................... 29 Task VI. Factors Affecting Sustainability ..................................................................................... 30 Contributions to Comprehensive HIV/AIDS Programming ......................................................... 30 Task VII. Beneficiary Relations ..................................................................................................... 32 Task VIII. Linkages, Coordination and Partnerships .................................................................. 34 ZPCT II Effectiveness at Establishing Linkages ............................................................................ 34 ZPCT II and MOH wide Linkages and Coordination with GRZ ................................................... 34 Best Practices for PPPs ................................................................................................................. 35 Task IX. Performance Monitoring ................................................................................................. 36 Task X. Partnerships and Management Systems ......................................................................... 39 Description of CO-Partnership Arrangements ............................................................................... 39 Performance of the Prime, Sub-partners, and Sub-grantees ........................................................... 40 Task XI. USAID Project Management .......................................................................................... 42 USAID Regulations and their Effect on Project Implementation ................................................... 42 Policy Issues .................................................................................................................................. 42 Strategic Approaches to USAID Management to Enhance Implementation and Success ............... 43 Coordination within USG Agencies to Enhance Implementation .................................................. 43 Specific Steps to Enhance USG Management ................................................................................ 43 Pending Management Actions ....................................................................................................... 44 H. Recommendations ...................................................................................................... 44 iii I. Higher-Level Issues ....................................................................................................... 46 Lessons Learned ............................................................................................................................ 46 Alignment ..................................................................................................................................... 80 Partnerships .................................................................................................................................. 81 Continuum of Care ........................................................................................................................ 81 Innovations ................................................................................................................................... 81 Linkages ........................................................................................................................................ 81 Procurements ................................................................................................................................ 82 Zambian Leadership and Human Resources Support .................................................................... 82 Gender Integration ........................................................................................................................ 83 ZPCT II’s Gender Response ......................................................................................................... 83 Gender Indicators ......................................................................................................................... 84 Training and Sensitization Workshops ........................................................................................... 85 Referral Services for Victims of GBV ............................................................................................ 85 Sustainability ................................................................................................................................. 85 Environmental Considerations ...................................................................................................... 86 Annexes Annex A: Evaluation Scope of Work ............................................................................................ 48 Annex B: ZPCT II Supported Facilities (by District) .................................................................. 54 Annex C: Professional Background of the IBTCI evaluation team............................................. 56 Annex D: List of Documents Reviewed ....................................................................................... 59 Annex E: List of Key Informant Interviews ..................................................................................61 Annex F: Client Exit Interview Questionnaire ............................................................................ 64 Annex G: Meetings on quality of HIV/AIDS services in one province ...................................... 66 Annex H: Meetings on Quality of HIV/AIDS Services in One Province ................................... 69 Annex I: Meetings on quality of HIV/AIDS services in one province ....................................... 72 Annex J: Client Exit Interview Summary .................................................................................... 73 Annex K: Selected Notes from FGDs–Service Coverage ............................................................. 74 Annex L: Focus Groups – Beneficiary Perceptions ..................................................................... 75 Annex M: Focus Groups – Services to be Improved at Community Level .................................. 78 Annex N: Training by Course, Sub-Course and Province ............................................................ 79 Annex O: Task III: Cross-Cutting Principles .............................................................................. 80 Annex P: Capacity-Building Training Log .................................................................................. 87 Annex Q: ZPCT II M&E Team: Organogram and M&E Post Descriptions ............................. 90 Annex R: ZPCT II Partners and Performance .............................................................................91 Annex S: USAID Management of ZPCT II Deliverables ............................................................ 95 Annex T: Life of Project & FHI Principles ................................................................................. 97 Annex U: USAID Evaluation Criteria .......................................................................................... 99 Annex V: Key Principles and Requirements ................................................................................ 100 List of Tables Table 1. ZPCT II Supported MoH Facilities and Services by Provinces .............................................. 13 iv Table 2. ZPCT II Supported Private Sector Facilities and Services ...................................................... 13 Table 3. ZPCT II Indicator Tracking Table ............................................................................................. 18 Table 4. Cross-Cutting Principals .......................................................................................................... 24 Table 5. Project Successes ........................................................................................................................ 26 Table 6. Types of Focus Groups ............................................................................................................ 32 Table 7. Status of ZPCT II-supported Referral Networks ..................................................................... 36 Table 8. ZPCT II Methods of data collection per PEPFAR program area ............................................... 37 Table 9. Comparison of Organizational Health Strategies .................................................................... 80 Table 10. ZPCT II alignment with the National HIV/AIDS Strategic Framework ................................... 82 Table 11. Types of Gender Trainings and Categories of Participants ......................................................... 85 List of Figures Figure 1. Potential Logical Framework ................................................................................................. 15 Figure 2. What factors influence your choice to use or not to use services? 33 Zambia Prevention, Care and Treatment Partnership Project II (ZPCT II) Mid-Term Evaluation Report i ACKNOWLEDGEMENTS The International Business & Technical Consultants, Inc. (IBTCI) evaluation team acknowledges the United States Agency for International Development (USAID); the Zambia Prevention, Care and Treatment Partnership Project II (hereafter referred to as ZPCT II); the Centers for Disease Prevention and Control (CDC); the United States (U.S.) Peace Corps; and the Department of Defense (DOD) for their cooperation and support for this evaluation. The team offers special appreciation the Contracting Officer’s Representative for this evaluation, Patricia Sitimela, and the Alternative Contracting Officer’s Representative, Justus Kamwesigye. IBTCI appreciates their continued guidance and assistance throughout the evaluation process. The IBTCI team is also grateful to Joy Manengu, the Contracting Officer’s Representative of ZPCT II, for being generous in proving information and assistance. A special thank you is also extended to the ZPCT II Chief of Party (COP), Michael Welsh, and the ZPCT II staff for their cooperation. The ZPCT II staff were generous with their time and support in discussing technical points and providing of information during the evaluation. The IBTCI evaluation team appreciates the generous time and support from the staff of the Government of the Republic of Zambia (GRZ), the National HIV/AIDS/STI/TB Council and its provincial and district offices. IBTCI also wishes to thank the Ministry of Health and its Provincial Health Offices in Central, Copperbelt, Luapula, Northern, North Western, and Muchinga Provinces. We also appreciate the cooperation and support of GRZ staff at the District Health Offices in Kabwe, Kapiri, Mposhi, Ndola, Lufwanyama, Mansa, Mwense, Kasama, Mpika, Kasempa and Solwezi, as well as selected hospitals and health centers in these districts. IBTCI would also like to thank Anderson Nkhuwa, who assisted in logistics, coordination, financial administration. The IBTCI evaluation team also appreciates the work of local research Assistants, Stephen Busiku, Chendela Masengu, Chilayi Mayondi, and Sam Kapembwa. These research assistants facilitated focus group discussions and conducted client exit interviews; they ensured that the perspectives of beneficiaries were an integral part of the evaluation analysis. Finally, appreciation is extended to the IBTCI home office team which includes Ajay Kalotra, Jessica Gajarsa, Kris Merschrod, Bryan Shipp, and Elizabeth Dean. The evaluation team appreciates the support of the IBTCI home office team which provided logistical, administrative, and technical support. Zambia Prevention, Care and Treatment Partnership Project II (ZPCT II) Mid-Term Evaluation Report ii ACRONYM LIST AIDS Acquired Immunodeficiency Syndrome ART Antiretroviral Treatment ARV Antiretroviral Drugs CBO Community-Based Organization CC Critical Care CD4 Cluster of Differentiation 4 CDC Centers for Disease Control and Prevention CHAZ Churches Health Association of Zambia CHV Community Health Volunteer CSO Civil Society Organizations CT Counseling and Testing DACA District AIDS Coordination Advisor DATF District AIDS Task Force DHIO District Health Information Office DHMT District Health Management Team DHO District Health Office DMO District Medical Office DOD Department of Defense DQA Data Quality Assessment FBO Faith-based organization FGD Focus group discussion FHI 360 Family Health International 360 FOG Fixed Obligation Grant GBV Gender Based Violence GRZ Government of the Republic of Zambia HAART Highly Active Antiretroviral Treatment HAC Health Advisory Committee HCW Heath Care Worker HIV Human Immunodeficiency Virus HMIS Health Management Information System HRH Human Resources Health IBTCI International Business & Technical Consultants, Inc. KCTT KARA Counseling and Training Trust KII Key informant interview LOP Life of Project M&E Monitoring and evaluation MC Male Circumcision MCP Multiple Concurrent Partners MNCH Maternal, Newborn and Child Health MOH Ministry of Health Zambia Prevention, Care and Treatment Partnership Project II (ZPCT II) Mid-Term Evaluation Report iii MOU Memorandum of Understanding MSH Management Sciences for Health NAC National HIV/AIDS/STI/TB Council NGO Non-Governmental Organization NASF National AIDS Strategic Framework NHSP National Health Strategic Plan NZP+ Network of Zambian People Living with HIV/AIDS OI Opportunistic Infection PATF Provincial AIDS Task Force PEPFAR U.S. President’s Emergency Plan for AIDS Relief PHO Provincial Health Office PLHA People Living with HIV/AIDS PMO Provincial Medical Office PMP Performance Monitoring Plan PMTCT Prevention of mother to child transmission PPP Public-Private Partnership QA/QI Quality Assurance/Quality Improvement RA Recipient Agreement RH Reproductive health SCMS Supply Chain Management Systems SFH Society for Family Health SI Social Impact SMGL Saving Mothers Giving Life SMS Short Message Service SOW Scope of work STI Sexually transmitted infections TB Tuberculosis TOT Training of Trainers TWG Technical Working Group USAID United States Agency for International Development USG United States Government UTH MC University Teaching Hospital Male Circumcision Unit WHO World Health Organization ZISSP Zambia Integrated Systems Strengthening Project ZPCT II Zambia HIV/AIDS Prevention, Care and Treatment Partnership II ZPRS Zambia Partner Reporting System Zambia Prevention, Care and Treatment Partnership Project II (ZPCT II) Mid-Term Evaluation Report iv ZPCT II SUPPORTED FACILITIES MAP ZPCT II serves selected districts in six provinces shown in the map below: Central, Copperbelt, Luapula, Northern, North Western, and Muchinga. 1 EXECUTIVE SUMMARY International Business & Technical Consultants, Inc. (IBTCI) is pleased to present the Mid-Term Evaluation Report of the United States Agency for International Development (USAID) funded Zambia Prevention, Care, and Treatment Partnership Project II (hereafter referred as ZPCT II). ZPCT II is implemented by Family Health International 360 (FHI 360) in Zambia under USAID Contract number GHH-I-1-07-00043-00, an award under the AIDS Support and Technical Resources (AIDSTAR) mechanism is an indefinite quantity contract (IQC). This report includes the findings, conclusions, and recommendations that the IBTCI evaluation team found over the course of the evaluation period, August – October 2012. The Purpose of the Mid-term Evaluation Implementation of the ZPCT II Task Order reached its mid-point in December 2011. USAID Zambia has contracted with IBTCI to perform a mid-term external evaluation to assess the progress of the ZPCT II program towards meeting its intended results. The objectives of this mid-term evaluation are three￾fold:  Part A (Retrospective): To evaluate the progress made toward achieving project objectives, including an assessment of project design (conceptual framework) in light of implementation experience to date;  Part B (Prospective): Based on the above findings, to make recommendations for ZPCT II project implementation through 2014, including the optimal mix of activities and funding for achieving project objectives and sustainability; and  Part C: Using the above findings, frame issues to debate/discuss/resolve at a level higher than the project, e.g. at the level of the Government of the Republic of Zambia (GRZ) and/or other donors. Description of the Project Evaluated ZPCT II is a five-year, $124 million contract supporting Human Immunodeficiency Virus/Acquired Immunodeficiency Syndrome (HIV/AIDS) service delivery in six of Zambia’s ten provinces (Central, Copperbelt, Luapula, Muchinga, Northern, and North Western). To date, $82 million has been obligated into the award, spanning nine program areas. The five main objectives of ZPCT II are the following: 1. Expanding existing HIV/AIDS services and scaling up new services, as part of a comprehensive package that emphasizes prevention, strengthening the health system, and supporting the priorities of the Ministry of Health (MOH) and the National HIV/AIDS/ Sexually Transmitted Infection (STI)/ Tuberculosis (TB) Council (NAC). 2. Increasing the involvement and participation of partners and stakeholders to provide a comprehensive HIV/AIDS service package that emphasizes prevention, strengthens the health system, and supports the priorities of the MOH and the NAC. 3. Increasing the capacity of the Provincial Medical Offices (PMOs) and District Medical Offices (DMOs) to perform technical and program management functions. 4. Building and managing public-private partnerships (PPPs) to expand and strengthen HIV/AIDS service delivery, emphasizing prevention, in private sector health facilities. 5. Integrating service delivery and other activities, emphasizing prevention, at the national, provincial, district, facility, and community levels through joint planning with the GRZ, other United States Government (USG), and non-USG partners. Methodology The evaluation team collected data for this performance evaluation using a variety of quantitative and qualitative methods to answer the questions in the 11 tasks required by the evaluation Scope of Work (SOW). These 11 tasks evaluated the performance of ZPCT II in the following areas: effectiveness; efficiency; cross-cutting principles; factors of success; obstacles; beneficiary relations; linkages, coordination, and leveraging; performance monitoring and management systems; and USAID project management. 2 Data collection methods included a document review, Key Informant Interviews (KIIs), client exit interviews, and Focus Group Discussions (FGDs). Purposively selected districts and facilities were selected in each of the six ZPCT II supported provinces. In each province, the team collected data from one urban and one rural district. Criteria used to select health facilities included representation among different levels of care, preference for sites with more comprehensive services, and sites with static rather than mobile Antiretroviral Therapy (ART) services. FGDs were conducted with beneficiaries of ZPCT II-supported services and non-beneficiaries, separately for men and women. Five client exit interviews at each health facility, across different HIV/AIDS services were conducted. The evaluation team analyzed quantitative and qualitative data from ZPCT II reports. Project monitoring and evaluation (M&E) and financial systems were analyzed for trends and comparison against targets. Client exit interview data were summarized using descriptive statistics. FGD summaries were coded and classified by themes for content analysis. The evaluation team met frequently, interchanged information and analyzed data. The team debriefed with USAID, USG partners, the NAC, the MOH, and ZPCT II at the end of the assignment. In response to USAID’s request for the team to focus most of its effort on a prospective analysis to inform implementation through 2014, the team identified, discussed, and refined a set of feasible recommendations. The team also reviewed findings and defined a series of issues for debate, discussion, and resolution at a level higher than the project. A. EVALUATION FINDINGS As an initial step, the evaluation team considered the project design, the five stated project objectives, and the performance and implementation. The team found that ZPCT II had no conceptual, logical or results frameworks to help guide the design and monitoring of the project. The five ZPCT II objectives were found to be strategies or activities and therefore difficult to evaluate with so few measureable indicators available in the ZPCT II’s current M&E system. In order to assess project achievements, the team rearranged these activities and indicators, so that most of ZPCT II’s current activities and indicators were organized under two overarching objectives: 1) increase the proportion of individuals receiving HIV/AIDS related facility based services. The team found that some 29 indicators were defined and tracked related to facility-based services under the first objective. However, for Objective 2) decrease the proportion of new HIV/AIDS infections, the team found that there were no indicators and related targets for scaling up community based services. The team found that ZPCT II has only three indicators with planned targets for training activities in relation to community based services; ZPCT II has defined one indicator for building the capacity of District Health Offices (DHOs) and PHOs to support community prevention activities. The evaluation team developed a suggested results framework (or a similar framework could be adopted) to more clearly define ZPCT II objectives, indicators, and targets for the second half of the project and to guide decisions on the proper mix of activities. This suggested results framework could also be adapted for the design of future programming. For Objective One, expanding existing and scaling up new services, the team evaluated the performance of ZPCT II performance against the life of project (LOP) targets and found that it is on track or has surpassed many of the targets for HIV service delivery indicators. Many service delivery accomplishments were above 60 percent for the period of August 2009 to June 2012; this is approximately 60 percent of the five year period of performance. The team found that ZPCT II has reached 181.8 percent of the LOP target for the indicator that tracks counseling and testing (CT) activities; the team also found that ZPCT II has reached 100 percent for the LOP target for CT in prevention of mother to child transmission (PMTCT) activities. These achievements far surpass the planned targets for these indicators and are reportedly a result of ZPCT II’s strong support to MOH for scale up of these services. ZPCT II has reached 84 percent for the LOP target for an indicator that tracks facility based services for HIV+ pregnant women who receive ART prophylaxis; this achievement is slightly higher than planned targets over the period of performance. The fact that ZPCT II has surpassed these targets could also indicate that the targets had been set too low at the beginning of the project and should be reassessed. 3 Although ZPCT II has met or surpassed most of the targets for indicators related to HIV/AIDS service activities, the ZPCT II achievement towards the targets for indicators of palliative care for both adult and children were reported at low levels (44 percent and 32 percent respectively). ZPCT II has achieved 55 percent of its target for training of both staff and community/lay persons, which is on track for this indicator ZPCT II stated that the project requested to slow down training activities in the budget realignment. However, the evaluation team considered the need to accelerate training, especially early in the project, of both health workers and community/lay persons. This would prevent the occurrence of untrained persons providing HIV/AIDS services. The team believes that accelerated training in service provision would also reduce the overcrowding in ZPCT II supported facilities and decrease the strain on overburdened staff as revealed during data collection. Discussions on the setting of targets (especially for Objective 1) revealed a multi-step process that included calculations of ZPCT II’s current project targets and intended to build on the performance of the previous project ZPCT I. The evaluation team found there should be further discussion between ZPCT II and USAID about how the project sets targets and measures achievements. In addition, this issue should be clarified in order for ZPCT II to readjust targets; balance priorities and resources; and meet the five objectives during the remaining period of performance. Portfolio reviews, currently focused on service delivery targets, could also be used to present progress on all five objectives that align with the PEPFAR Summary of the HIV/AIDS Framework for Zambia. For Objective 2, increasing involvement and participation of partners, the ZPCT II reported that all targets for laboratory facilities and services in public and private partners tracked at over 70 percent of targets. For Objective 3, increasing the capacity of PHOs and DHOs, the ZPCT II only has one indicator “number of PHOs and DHOs provided with technical assistance.” ZPCT II reported that it achieved 100 percent in proving support to the 47 targeted PHOs and DHOs through its subcontracted partner, Cardno. This single indicator does not support evaluation of the increase in the capacity of PHOs and DHOs. Under Objective 4, build and manage private public sector partnerships (PPPs), ZPCT II reported PPPs with 18 facilities that report on services, and that other PPPs have signed memorandums of understanding (MOUs). The indicator “the number of private facilities that provide HIV/AIDS services,” is general and should be better defined. This program area would benefit from additional indicators that measure the progress of PPPs. This report provides a table with the number of facilities and selected services provided. ZPCT II reported PPPs at 18 facilities reporting out services, and other PPPs reportedly signed memorandums of understanding. The indicator “the number of private facilities that provide HIV/AIDS services,” is vaguely defined. These activities would benefit from additional indicators that measure the progress of PPPs. This report provides a table with the number of facilities and selected services provided. The evaluation team found that currently, there are no indicators to track progress against Objective Five, “integrating service delivery and other activities, emphasizing prevention through joint planning with the GRZ, USG, and other non-USG partners.” The team also found that ZPCT II has made some progress in the integration of services, such as family planning and chronic diseases. The evaluation team notes that it is harder to track the second aspect of Objective Five, “joint planning with the GRZ, USG and non-USG partners.” The team assessed project efficiency as defined in the evaluation SOW which includes evaluating the mix of HIV/AIDS activities and funding allocated to ZPCT II. The team found that ZPCT II’s activities were focused mostly on MOH and private facilities to support health staff and volunteers to provide HIV/AIDS counseling, testing, and treatment. The team found that ZPCT II’s support for, and coordination with, the official district and the NAC at the community level, faith based organizations (FBOs) and other community based organizations (CBOs) to support the mix of HIV/AIDS activities as part of a comprehensive package that emphasizes prevention, strengthening the health system, and supporting the priorities of the MOH and NAC, were considerably lower. The team assessed ZPCT II’s systems, management, and coordination of project resources and found that these project functions were not consistently efficient. The project has multiple and disconnected 4 internal systems for the management of information, as well as the planning, tracking, and management of key project activities and functions (specifically the procurement and logistics functions and training, mentoring, and renovation activities). These inefficiencies limit the availability of data for decision making and have resulted in unnecessary difficulties in tracking the resources for training, renovations, and other key project activities. Budget allocations are based on the U.S. President’s Emergency Plan for AIDS Relief (PEPFAR) funding areas and past experience. Without activity based budget, effectiveness is compromised. This is in spite of the fact that ZPCT II budgets are appropriate and adequate to achieve the targets and project objectives based on the current LOP monthly burn rate of approximately $2 million USD. ZPCT II was tasked with the integration of ten Cross-Cutting Principles and Requirements the project objectives, activities, and management functions. The evaluation team found that the status of integration was stronger for some project areas. In terms of the “Continuum of Care” cross-cutting principle the evaluation team found that the structure was established in the referral system. The team also found that the status of integration under the “Innovations” cross-cutting principle was strong. The team found that SmartCare, Short Message Service (SMS), and Baby Care tracking were initiated. Under the “Leadership Development” cross-cutting principle, ZPCT II staffing has been nationalized. The evaluation team found that the status of integration under the “Environmental Compliance” cross-cutting principle is advanced. However, the team found that project performance is lagging under the “Gender Integration” cross-cutting principle. ZPCT II training on gender is limited. Project indicators that track gender integration have been reduced and do not track quality. Project activities do not have a focus on behavioral change communications or on the prevention of gender based violence (GBV). The evaluation team found that activities to support gender sensitization at the community level and the promotion of gender equity were limited. An extensive discussion of ZPCT performance in terms of the ten Cross￾Cutting Principles and Requirements including how the project has addressed gender issues is in included in Annex O. The evaluation team assessed the multiple successes of the project. Respondents most frequently cited project successes in the areas of CT, PMTCT, and care in all supported sites as well as the provision of ART and lab services in almost one-third of project supported sites. Respondents also stated that ZPCT II’s performance in environmental compliance has also been successful. Respondents discussed that factors of success included project performance in supporting facility renovation, training, mentorship, and allowances, as well as the provision of equipment, materials and supplies. Other factors of success cited by respondents include the lab sample referral system; the use of volunteer lay counselors and adherence support workers, and the use of hired data entry clerks. Practices, products, and tools cited as successful and innovative and useful for replication included, mobile ART services, mentorship, polyvalent health workers, clinical guideline job aids, the use of SMS to increase adherence to ART, referral forms and use of encrypted email to send pediatric diagnostic results. The evaluation team was tasked with investigating obstacles that affected the project performance. The team discussed obstacles with ZPCT II staff, volunteers, and community groups in targeted areas. During these discussions respondents also recognized ZPCT II’s many contributions. Respondents frequently mentioned the issue that GRZ has mandated allowances for its staff and U.S. contractors could not fully pay; this has emerged as an obstacle. This challenge reportedly affected staff motivation, resulting in lower attendance at ZPCT II supported trainings. This issue has also incurred costs to the provincial and district health offices. Another major obstacle mentioned by respondents is the high turnover rate of MOH staff and overall inadequate staffing level. Another obstacle that emerged during data collection is that there is limited joint planning between ZPCT II and the MOH. Respondents also cited the issue of different planning cycles that affect coordination between the provincial and district levels of the MOH and that the project has systems for implementation that are often parallel to those of the MOH. The evaluation team found that the various systems for data management, procurement, maintenance, and planning for capacity-building present challenges for more effective and efficient implementation of ZPCT II activities to support the MOH. Respondents also mentioned that there is limited collaboration between ZPCT II and the NAC, specifically at the provincial and district level. This presents challenges to energizing the broad network of CSOs that could be mobilized for palliative care, prevention and other HIV/AIDS activities. 5 ZPCT II’s approach and planning for sustainability were also assessed by the team. MOH staff interviewed generally felt that ZPCT II has made considerable contributions in developing a comprehensive MOH HIV/AIDS program in the supported provinces. These respondents mentioned that factors that improve sustainability include ZPCT II contributions and support for lab and pharmacy services the provision of supplies, equipment, furniture, and covering of fuel and other operating costs were mentioned. Planning for gradual assumption of these costs will be needed. For example, the new cadre of data entry clerks and the funding support to volunteers was recognized as ZPCT II contributions. However, there was no clear sustainability plan for continuing these services at project end. Data entry clerks are ZPCT II staff and are not yet included in the GRZ civil service establishment. The evaluation team found that ZPCT II training of clinical staff and volunteers has helped increase the ability of the MOH to provide quality HIV services. The evaluation team suggests that more training of trainers within the MOH is needed to build sustainable capacity. Additional training of trainers will need to be negotiated with the MOH, given the issues of allowances and assumption of leadership by the MOH and the recognized staff shortages. ZPCT II has a cascade approach to building the capacity building of MOH staff at the provincial and district levels through mentoring and training activities. This is a structured approach that focuses on four key areas: planning, finance, governance, and human resources. The evaluation team found that this cascade approach has the potential to support and sustain programs, especially if expanded. As ZPCT II began this approach relatively recently, ZPCT II should assess and adjust capacity building plans to assure its contribution to sustaining PHO and DHO capacity. The evaluation team found that ZPCT II’s systems of data collection and analysis are strong, but resource intensive. These data collection systems are not yet fully shared with the PHOs and DHOs. ZPCT II should address the issue of sharing and strengthening data systems that can be sustained within the constraints of the MOH’s infrastructure and human resources if these systems are to be sustained. 1. The team assessed the satisfaction of project beneficiaries during client exit interviews and FGDs. Overall, 76 percent of the 78 clients who participated in the exit interviews at facilities rated services they received on the day of the survey as good or excellent, with the remaining 24 percent rating services as fair. Clients were also asked whether specific issues had been of concern during their visit on the day of the interview and needed improvement. The two areas of major concern were cleanliness (33 percent) and availability of medicine (21 percent). 2. During the FGDS, 15 of the 44 groups reached a consensus that they were not satisfied with their ability to access multiple services; and 10 of the 44 groups did not reach a consensus and had mixed opinions. During FGDs, when asked about their ability to access multiple services, respondents frequently mentioned that challenges to accessing multiple services and products include crowding, insufficient staff, long wait times, and the need for additional HIV/AIDS facilities, particularly citing limited access to ART-related services. The evaluation team noted that groups of respondents from urban areas mentioned longer wait times. In addition, ZPCT II supported facilities tend to attract a larger number of clients and this is likely to result in crowding and longer wait times. 3. When respondents were asked for their opinions on the professionalism of staff at their respective facilities, 11 of the 44 groups reached a consensus that they were satisfied; eight of the 44 group reached a consensus that they were not satisfied; and seven groups did not reach a consensus and had mixed opinions. When asked about their opinions on the quality of care at their respective facilities, eight of the 44 groups reached a consensus that they were satisfied; five of the 44 group reached a consensus that they were not satisfied; and 8 groups did not reach a consensus and had mixed opinions. When asked about their opinions on how confidentiality was handled at their respective facilities, 14 of the 44 groups reached a consensus that they were satisfied; nine of the 44 groups reached a consensus that they were not satisfied; and four groups did not reach a consensus and had mixed opinions. Please see Annex L for a full discussion of the reasons for whether FGD respondents were satisfied with services or not. The team also assessed the development of external linkages and partnerships between ZPCT II and the MOH. These partnerships were developed with formal recipient agreements. During interviews with MOH staff, respondents reported that these agreements were generally not productive for partnership 6 effectiveness. The team reviewed recipient agreements and discussed these with MOH staff. Some DHO and health facility staff members interviewed were not aware of the content of the agreements. Some staff stated that ZPCT II did not meet with them to discuss the content before asking them to sign. There were significantly different perceptions between ZPCT II and MOH staff on the agreements and the process of their negotiation and implementation. There were also significant differences of perceptions between ZPCT II and the PHOs, DMOs, and facility staff on the broader partnership between the provincial and district offices and facilities. Where the ZPCT II perceives the project to be working in close partnership with the MOH at all levels, a number of the PHO, DHO, and facility staff stated that that they felt they were not treated as equal partners and were not involved in joint decision￾making. The evaluation team found that respondents found the partnerships between ZPCT II and the much smaller number of private sector partners to be positive; these respondents said the system for consultation between the private partners and ZPCT II was more consistent. The evaluation team assessed the partnerships between ZPCT II and its sub-partners and grantees as well as the project’s organizational structures. These partnerships encompass direct relationships between each major sub-partner and FHI 360 (the prime); yet there was more limited lateral interchange among sub￾partners. Most of the sub-partners are placed in the Program Unit, even though they are technical experts in areas such as gender, behavior change, and capacity development. The Technical Unit is currently focused almost exclusively on clinical services. As ZPCT II restructures its activities to meet all five project objectives, it will require more integration of the skills of staff. The evaluation team found that ZPCT II would need to be more integrated and the technical experts of sub-partners better used in order to accomplish the following goals: to strengthen its partnerships with the MOH and private sector partners; to provide support to civil society; to incorporate gender across all activities; and to expand prevention services. The evaluation team found that ZPCT II partnerships and interchange between ZPCT II and the NAC provincial and district taskforces are generally poorly developed. Formal MOUs have not been established for the most part. The evaluation team also found that ZPCT II could leverage more of the opportunities to partner with the large, multi-sectoral network of CSOs; this could potentially expand prevention, palliative care, and other community-based activities. The team reviewed ZPCT II’s various performance monitoring systems with an emphasis on the sharing and use of data for management decisions. The evaluation team found that ZPCT II’s M&E system is implemented at national, provincial, and facility levels and it is aligned with the MOH Health Management Information System (HMIS). ZPCT II collects facility data on clinical services delivery for project and MOH use. ZPCT II supported sites also collect data on GBV screening and other community activities. The use of data at the facility level (where data entry clerks are posted) reportedly varied considerably. Where there was an active data entry clerk, she or he collected, analyzed, and provided data, responded to data requests and participated in data analysis discussions. At the other end of the spectrum, the team found that some data entry clerks had not yet been trained. These data entry clerks did not provide analysis for their facilities, and interpreted their role as to only provide data to ZPCT II. The evaluation team also found that data use was also inconsistent across the district. Some of the staff members from DHOs and District Health Information Offices (DHIOs), and other MOH offices reported that they were not included in ZPCT II data analysis sessions and received only hard copies of their district’s data. If ZPCT II had provided electronic copies to district level staff, they would be able to analyze data from their districts’ facilities. DHOs often get summary data, not raw data from their own facilities. ZPCT II and some DHOs reported that data are not regularly shared in quarterly review meetings. The team found no tracking by ZPCT II or the MOH of MOH staff use of data for management decisions. The evaluation team reviewed the SmartCare system and found that there are challenges including outdated software, incomplete data entry, backlogs of data to be entered, and equipment breakdowns. Inadequate digital storage space has affected the merging of files from several districts. Staff reported that “security measures” place significant restrictions on access to SmartCare data. These issues significantly affect SmartCare’s use for the tracking and management of clinical cases. ZPCT II reported that it is installing a new version of the software and will conduct training in using the upgraded system. Since 7 SmartCare is focused on individual client data, it serves a different function than other ZPCT II and MOH monitoring systems, and is not a parallel system. The evaluation team reviewed ZPCT II internal performance monitoring systems. Currently ZPCT II has an internal draft of a Performance Monitoring Plan (PMP) and a database that collects services data from supported facilities. The team’s review found that the draft PMP lacked essential elements, although it is complemented by a number of guides, such as the M&E work plan and procedures manual. The multiple M&E manuals and plans make updates and changes more time consuming. Regarding the PMP, there was conflicting information provided to the evaluation team by USAID, FHI 360, and Cardno. The team was not provided with documentation to show that the PMP had been approved. ZPCT II has provincial and national level databases that can be used to generate pre-programmed automated PEPFAR reporting. In contrast, ZPCT II has not automated its reporting for tracking, analysis, and production of reports on the status of training, procurement, and other technical assistance activities for project management. In addition, access to the ZPCT II database is restricted to a single person. Although ZPCT II is a partnership, the activities of sub-partners are not included in ZPCT II’s database. The evaluation team found that the lack of automated production for reporting and the separation of the sub-partners activity data severely limits ZPCT II to implement an effective data management system. Extensive amounts of data are collected and stored by ZPCT II, however, the organization of, access to, and shared use of these data is not adequately extensive. B. RECOMMENDATIONS Recommendations for ZPCT II to address Objective One, expand and scale up services, include the following:  ZPCT II should reassess and adjust the LOP targets for indicators to be challenging yet achievable. This and further analysis would enable ZPCT II to better balance its resources to achieve service delivery, capacity development, partner participation, integration, and other project objectives; this would also aid in the setting of LOP targets. ZPCT II should use elicit external technical assistance and guidance from USAID to guide these adjustments. Special attention should be paid to prevention and community targets to assure sufficient resources are identified.  ZPCT II should develop a structured approach and coordinate timelines with MOH for planning, data management, procurement and other systems. By integrating and harmonizing plans with MOH, ZPCT II would be able to implement expansion and scale-up more effectively.  ZPCT II should increase its support of M&E Working Groups and other venues in order to assist the MOH to streamline and consolidate the multiple data streams and move toward one national HIV/AIDS M&E Plan. In preparation for this, ZPCT II should develop one project M&E system internally by merging its various M&E guiding documents into one PMP for easy standardization and use.  To develop data driven management in ZPCT II and the MOH, ZPCT II should train and mentor data entry clerks, MOH, DHIO, and other staff to strengthen data analysis and use capacity at district and provincial level facilities. ZPCT II should upgrade computer hardware and software, as it is doing for Smart Care. ZPCT II internal project databases at both the central and provincial levels should be strengthened and automated to produce management and technical reports for data driven decision-making ; this would be especially useful the areas of training, procurement, and ART retention. Since this effort would strengthen data for management decision-making, this step would complement, not duplicate, the Smart Care system. Recommendations for ZPCT II to address Objective Two, increase the involvement and participation of partners, include the following:  ZPCT II should provide consistent support to GRZ leadership to encourage discussions related to the coordination of partners, including USG and other donor partners, the MOH, NAC and other GRZ sectors for more effective comprehensive HIV/AIDS programming. USAID could support this with monitoring of progress.  ZPCT II should work more closely with MOH, the USAID DELIVER Project, and the PEPFAR funded Supply Chain Management System (SCMS) project to coordinate for a health sector-wide approach to procurement and maintenance of drugs, supplies, and equipment. ZPCT II should align 8 and synchronize its procurement and maintenance plan with MOH priorities and timelines as specified in the National Strategic Plan (2011-2015). ZPCT II should also increase its internal capacity to guide and manage its procurement and maintenance activities.  ZPCT II should formally and more intensively work across all supported districts with the DATFs through formal MOUs. Additionally, ZPCT II should establish formal links to the District Development Coordinating Committee to strengthen coordination and participate in district-level HIV/AIDS multi-sectoral approaches.  To increase partnership and comply with GRZ policies on gender, ZPCT II should identify, mentor, and support a broad base of community groups and private sector partners to focus on GBV prevention actions at the community level, and to identify GBV cases and refer for care. ZPCT II should lead the development of a set of community-based indicators for public and private sector actors to track and monitor the impact of GBV interventions.  As a base for the strengthening of community partners, ZPCT II should accelerate technical assistance and other resources to integrate GBV case management into the referral system. This process should involve developing a comprehensive toolkit for GBV sensitization at the facility and community level as a resource to address GBV survivors’ medical, legal, and psychosocial needs. Recommendations for ZPCT II to address Objective three, increase the capacity of PHOs and DHOs technical and program management functions, include the following:  ZPCT II should expand its current interventions led by sub-partner, Cardno to build the management and technical capacity of PHOs and DHOs (e.g., in the areas of governance, finance, planning, and human resources). ZPCT II should do this across all supported districts to build DHMT technical and management capacity more rapidly. This initiative should be coordinated with current MOH technical and management capacity development approaches to ensure harmony.  ZPCT II should redesign its district level graduation criteria and capacity building approach for facilities to include measuring technical and management capacity at the district level. Adjusted criteria should focus on measuring capacity across the following areas: governance, finance, human resources, and technical abilities especially in service quality. This adjusted approach should be jointly planned with PHOs and DHOs.  USAID should emphasize the need for ZPCT II to strengthen and considerably expand its support to PHO and DHO capacity building approaches. To address Objective Four, build and manage public private partnerships, include the following:  USAID should clarify with ZPCT II if PPPs should be spread across provinces to serve as models for expansion.  ZPCT II should continue to strengthen mechanisms to develop peer exchanges of information and lessons learned across the PPPs. ZPCT II should also encourage interchange between PPPs, DHOs, and the MOH on training curricula and protocols.  ZPCT II should assist private partners to address the tendency of clients to move back and forth between public and private services resulting from low insurance coverage or the inability of clients to pay for services. ZPCT II should work in an area such as the Copperbelt, to pilot a service delivery model and referral system that recognizes and accommodates the tendency of clients to move between public and private health care systems. The evaluation team believes that addressing this issue is feasible in the project’s remaining time. Recommendations for ZPCT II to address Objective Five, integrate service delivery though joint planning, include the following:  ZPCT II should consider how to strengthen its strategic approaches to integrate family planning, chronic diseases care, and other services into HIV/AIDS services. If feasible, ZPCT II should consider how to document lessons learned in order to contribute its strong clinical capacity to the MOH national strategy being developed to incorporate the control of chronic diseases (e.g., diabetes, hypertension, etc.) into MOH protocols and training. ZPCT II could assess the feasibility of approaches being considered and help pilot effective integrated services as models. USAID and ZPCT II should discuss this issue to assure agreement on a feasible approach. Recommendations to address USAID oversight and guidance: 9  Given the scale and complexity of ZPCT II, USAID should provide additional level of effort to supply strategic, high level monitoring, and guidance to ZPCT II as it undertakes strategic changes to its project.  USAID should conduct periodic independent visits to MOH and NAC provincial and district sites to discuss project progress and obstacles, and support the strengthening of relationships between ZPCT II, GRZ partners, and civil society. C. HIGHER LEVEL ISSUES The evaluation team identified some issues that require higher level consultation and dialogue between the governments of Zambia, the United States, other donor governments, civil society, and the public at large. These key issues include the following:  There should be a policy dialogue on the issue of GRZ allowances due to its significant impact on the effectiveness of ZPCT II and similar projects on the PHO and DHO in terms of priorities, planning, and the use of GRZ financial resources. This issue affects many USAID and other donor partners while there is little cohesion among partners and donors in the approach and the amounts paid.  The evaluation team recommends that the USG, other donor governments, and the GRZ have a dialogue to increase and/or enhance overall human and financial resources with an emphasis on systems strengthening. This dialogue is necessary to advance the capacity of the Zambian health sector and civil society to provide and expand services, lead effective strategies to prevent new infections, and strengthen public health approaches to address the drivers of the HIV epidemic.  The evaluation team recommends a re-orientation of GRZ agencies, cooperating partners, and civil society to increase understanding of regional patterns of HIV transmission in order to comprehensively address prevention issues. This would enable private and public actors to mitigate potential risks that come with the expansion of roads and trade, and effectively address the needed changes in cultural patterns in the Zambian society that drive the epidemic. Strategic approaches and dialogue are needed to also address the impact of regionalization and economic integration on Zambia due to multiple socio-economic initiatives in Southern Africa.  The evaluation team recommends USG agencies to consider significant adjustments of all USG health sector resource allocations to align them more closely with the USG -Zambia Partnership Framework’s first commitment: “Accelerate and intensify prevention in order to reduce the annual rate of new HIV infections.”  The evaluation team recommends consideration and action on the integration of nurse prescriber, community health worker, and data entry clerk cadres into the civil service establishment with pre￾service training requirements, career progression paths, and remuneration package by the Public Service Division-Cabinet Office.  Strengthening of the donor coordination system to re-enforce the “three ones.” These include: one agreed upon AIDS action framework, one national AIDS coordinating authority, and one agreed country-level M&E system. This would promote strategic direction and resource use, support to country ownership, and would streamline and prioritize M&E systems and resources.  The impact of regionalization and economic integration on Zambia through various socio-economic initiatives in Southern Africa. This includes the need for regional-level dialogue and initiatives to harmonize strategic approaches. 10 INTRODUCTION International Business & Technical Consultants, Inc.(IBTCI) is pleased to present the Mid-Term Evaluation Report of the United States Agency for International Development (USAID) funded Zambia Prevention, Care, and Treatment Partnership Project II (hereafter referred as ZPCT II). ZPCT II is implemented by Family Health International 360 (FHI360) in Zambia under USAID Contract Number GHH-I-1-07-00043-00. This report includes the findings, conclusions, and recommendations of the ZPCT II mid-term performance evaluation, which the team conducted during August – October, 2012. D. EVALUATION PURPOSE & QUESTIONS The purpose of the evaluation was to perform a mid-term external evaluation to assess the progress of the ZPCT II program towards meeting its intended results. A stated in the evaluation scope of work (SOW), the evaluation team is required to provide USAID with information on project progress; to develop recommendations based on evaluation findings for project implementation through 2014; and to frame issues to debate, discuss, and resolve at a level higher than the project. The evaluation SOW required the team to undertake 11 Evaluation Tasks with multiple underlying questions for each task. These questions are included in the evaluation SOW presented in Annex A. To fulfill the evaluation purpose and address the questions the team responded to all the questions that correspond to the 11 required evaluation tasks, which include the following: Task I Effectiveness; Task II Efficiency; Task III Cross-cutting Principles; Task IV Factors of Success; Task V Obstacles; Task VI Sustainability; Task VII Beneficiary Relations; Task VIII Linkages, Coordination, and Leveraging; Task IX Performance Monitoring and Impact; Task X Management Systems within the Project; and Task XI USAID Management. Structure of this Evaluation Report Findings and conclusions are presented in terms of the 11 Evaluation Tasks listed above. The evaluation findings on the progress ZPCT II has made towards the five main project objectives (listed below in Section C. Project Background) are presented under Task I Effectiveness. The findings and conclusions on ZPCT II performance in terms of the Key Principles and Requirements are crosscutting issues that are discussed under Task III Cross-cutting Principles with the exception of the findings and conclusions on ZPCT II performance in terms of Sustainability which is discussed under Task VI Sustainability. Section F presents the evaluation recommendations in relation to the five main ZPCT II objectives that are listed below. E. PROJECT BACKGROUND ZPCT II is a five-year (2009 to 2014) $124 million task order funded by USAID through the U.S. President’s Emergency Plan for AIDS Relief (PEPFAR). ZPCT II supports the Government of the Republic of Zambia (GRZ) goals of reducing prevalence rates and providing antiretroviral therapy (ART). The project serves selected districts in six provinces shown in the map (found before the Executive Summary of this report): Central, Copperbelt, Luapula, Northern, North Western, and Muchinga. The five main objectives of ZPCT II are the following: 1. Expanding existing HIV/AIDS services and scaling up new services, as part of a comprehensive package that emphasizes prevention, strengthening the health system, and supporting the priorities of the MOH and the NAC. 2. Increasing the involvement and participation of partners and stakeholders to provide a comprehensive HIV/AIDS service package that emphasizes prevention, strengthens the health system, and supports the priorities of the MOH and the NAC. 3. Increasing the capacity of the Provincial Medical Offices (PMOs) and District Medical Offices (DMOs) to perform technical and program management functions. 4. Building and managing public-private partnerships (PPPs) to expand and strengthen HIV/AIDS service delivery, emphasizing prevention, in private sector health facilities. 11 5. Integrating service delivery and other activities, emphasizing prevention, at the national, provincial, district, facility, and community levels through joint planning with the GRZ, other United States Government (USG), and non-USG partners. HIV/AIDS Situation in Zambia The prevalence of HIV infection among adults (men and women aged 15-49 years) is 14.3 percent in 2007, representing a decline from the 15.6 percent reported in 2002.1 In particular decreases occurred among women (from 17.8 to 16.1 percent) and urban residents. The prevalence of HIV/AIDS is estimated at 14.6 percent among adults (men and women aged 15-49 years), representing a decline from 15.6 percent reported in 2002.2 The prevalence rates among women is higher (at 16.1 percent) than for men (12.3 percent). Urban areas have higher HIV prevalence rates (20 percent) than rural areas (10 percent). HIV incidence in adults aged 15–49 was halved since 1990 and was estimated to be at a stable level of 1.6 percent in 2009 (2 percent in women, 1.2 percent in men).3 In 2009, an estimated 82,681 adults were newly infected with HIV (59 percent women, 41 percent men). This translated into 226 new adult infections per day. Although HIV incidence has stabilized, the absolute number of new HIV infections increased due to Zambia‘s expanding population. Between 2002 and 2009 HIV prevalence increased in three of the nine provinces of Zambia by an average of 2.1 percent.4 The proportions of multiple concurrent partners (MCP) increased among women (from 13.2 to 16.9 percent) whose condom use with non-regular partners remained low at 37.4 percent. ZPCT II Project Partners The project’s prime implementing partner is FHI 360 and is supported by a number of sub-partners. Sub￾partners include the following organizations:  Management Sciences for Health (MSH) contributes towards strengthening the MOH health system focusing on laboratory and pharmaceutical systems at national, district and the health facility levels through training and technical support.  CARE International (hereafter referred to as CARE) contributes to the provision of comprehensive HIV/AIDS services, including prevention, care and treatment, through training and supporting community volunteers and strengthening the continuum of care through referral networks.  Social Impact (SI) integrating gender in health facility service delivery and community prevention, care and treatment activities.  Cardno Emerging Markets Group (hereafter referred to as “Cardno”), contributes towards building the capacity of PMOs and DMOs to provide technical and program management oversight including enhanced problem solving, mentoring, supervision, and monitoring of HIV/AIDS programs.  The Churches Health Association of Zambia (CHAZ) contributes towards the expansion and scale up and integration of prevention, care and treatment services through ten mission health facilities in three provinces supported by ZPCT II.  The KARA Counseling and Training Trust (KCTT) contributes towards strengthening the MOH health system through training facility and community based health workers counseling and testing (CT) services.  The University Teaching Hospital Male Circumcision Unit (UTH MC Unit) contributes towards implementation of male circumcision services in ZPCT II supported health facilities through training and technical support. No other sub-partners were referred to in documents or discussed as part of ZPCT II agreements. The report makes recommendations were made to ZPCT II (FHI 360) as the prime to maintain a coordinated approach and oversight of individual ZPCT II activities. Although one, or more, of the sub￾partners would implement a specific recommendation, ZPCT II (prime) provides training, technical assistance, equipment, supplies, renovations, and other inputs focusing on HIV/AIDS service delivery and community activities. In Table 1 below entitled “ZPCT II supported MOH Facilities and Sites” shows the number of ZPCT II supported MOH facilities by province. The table also clearly presents the HIV/AIDS services supported by ZPCT II in these provinces (ART, prevention of mother to child transmission -PMTCT, CT, clinical 12 care, lab, specimen referral for Cluster of Differentiation 4 - CD4, and MC). However, the table does not include palliative care, community mobilization, or other services that may be provided in the communities served by these facilities, nor does it present the systems strengthening components of the project. The table does provide an overview by province of clinical services provided. The table demonstrates that PMTCT, CT, and clinical care services are provided in all 371 ZPCT II supported MOH facilities, with the small exception of only nine MOH facilities that do not provide PMTCT services. ART services are provided in 135 of the 371 ZPCT II supported MOH facilities. There are 128 laboratory facilities in the 371 ZPCT II supported MOH; of this number 82 are referral labs with the capacity to do CD4 counts. This signifies ZPCT II support across a considerable number of MOH facilities. In Table 2 the types of services supported by ZPCT II in private facilities are presented. In the ZPCT II supported private sector facilities 14 of the 18 facilities provide ART services. PMTCT is supported in only 10 of the 18. CT services are supported in 18 private facilities, 14 of these have laboratories. The geographic distribution of ZPCT II supported private facilities favors the Copperbelt Province with 14 of the 18 facilities being located within its borders. Please see Annex B for a table that presents the ZPCT II supported facilities by district. F. METHODS AND LIMITATIONS The evaluation team used both quantitative and qualitative methods to answer the questions under the 11 evaluation tasks required by the evaluation SOW. Evaluation Team. The evaluation field team consisting of: an expatriate Team Leader, an expatriate HIV/AIDS Program Management Specialist, a Zambian resident Deputy Team Leader, two Zambian HIV/AIDS Program Experts, and a Zambian Logistics Coordinator. The team divided into three sub￾teams in order to collect data in the six provinces in which ZPCT II implements.5 Sub-team A was led by the Team Leader who was a Health Evaluation Expert and nurse clinical specialist and included a Zambian HIV/AIDS and Health Program Expert who was a clinician. Sub-team B was led by the Deputy Team Leader/ HIV/AIDS and Gender Specialist and included a HIV/AIDS Program Management Specialist. Sub-team C was led by a HIV/AIDS M&E Specialist and included a Research Assistant /HIV/AIDS Communications Expert. Please see Annex C for brief descriptions of the qualifications and experience of the team members. The evaluation team collected data using the following methods: Desk Review. The evaluation team conducted an extensive desk review of data and reports from the MOH and other GRZ agencies, USAID, ZPCT II, as well as materials from other donors and independent research organizations. Please see Annex D for a list of the key documents included in this review. Quantitative data from ZPCT II reports, monitoring and evaluation (M&E) and financial systems were analyzed for trends and were used for the comparison of findings against the reported achievement of targets. Key Informant Interviews. The evaluation team conducted individual and group key informant interviews (KIIs) with GRZ officials, USAID, ZPCT II staff, public and private health facility staff, PEPFAR partners, private sector partners, and Health Advisory Committee (HAC) members. The evaluation team interviewed MOH staff at the central, provincial, and district levels. The team also interviewed members of the NAC structure such as members of District AIDS Task Forces (DATFs) and Provincial AIDS Task Forces (PATFs). Please see Annex E for a full list of key informants. Client Exit Interviews. The evaluation team trained and supported four local Researcher Assistants to conduct five client exit interviews at each of the health facilities selected for site visits. The Research Assistants used an interview tool that contained 14 multiple choice questions to guide the client exit interviews. Client exit interviews were conducted at the facilities to facilitate the identification of respondents during site visits and to ensure that respondents could answer questions with a very short recall time. Please see Annex F for the full Client Exit Interview questionnaire. Data from client exit interviews were entered and coded in Excel, imported and analyzed in STATA, a data and statistical analysis software program, and summarized with descriptive statistics. 13 Table 1. ZPCT II Supported MOH Facilities and Services by Provinces Province Number of Health Facilities Urban Facilities Rural Facilities ART PMTCT CT CC Lab Specimen Referral for CD4 MC Central 74 22 52 24 73 74 74 26 46 8 Copperbelt 81 65 16 43 79 81 81 42 57 8 Luapula 82 5 77 30 76 82 82 20 51 7 Muchinga 32 6 26 9 32 32 32 9 16 4 Northern 45 11 30 17 45 45 45 19 27 5 North-Western 57 6 47 12 56 57 57 14 20 6 TOTALS 371 115 248 135 361 371 371 130 217 38 *ART – Antiretroviral Therapy; CC – Clinical Care; CT – Counseling and Testing; PMTCT – Prevention of Mother to Child Transmission; MC – Male Circumcision Table 2. ZPCT II Supported Private Sector Facilities and Services District Number of Health Facilities Urban Facilities Rural Facilities ART PMTCT CT CC Lab Specimen Referral for CD4 MC Central Province Kabwe/ Mkushi 4 4 0 0 2 4 4 4 0 0 Copperbelt Province Ndola 6 6 0 6 3 6 6 4 5 0 Kitwe 5 5 0 5 4 5 5 4 3 0 Mwense 1 1 0 1 0 1 1 0 0 0 North-Western Province Solwezi 2 2 0 2 1 2 2 2 0 2 TOTALS 18 18 0 14 10 18 18 14 8 2 *ART – Antiretroviral Therapy; CC – Clinical Care; CT – Counseling and Testing; PMTCT – Prevention of Mother to Child Transmission; MC – Male Circumcision Please see Annex B for a table that presents the ZPCT II supported facilities by district. 14 Focus Group Discussions (FGDs). The four Research Assistants facilitated the following FGDs in each district that the team visited: four FGDs with HIV-positive (+) persons (most of these FGDs included six to eight participants); two FGDs with beneficiaries (i.e., clients who received services at ZPCT II supported facilities); and two FGDs with non-beneficiaries. These FGDs were facilitated according to a semi-structured focus group guidelines that are provided in Annex G. FGDs were conducted separately by gender to reduce the potential of inhibition during discussion. The Research Assistants were trained by the evaluation team to verify these eligibility criteria of FGD participants before initiating discussion sessions. FGD participants were identified with assistance from the Network of Zambian People Living with HIV/AIDS (NZP+) as well as staff members from ZPCT II, the PHOs and the DHOs. To decrease the likelihood that clients will give biased responses about the health services they receive from a facility because of the fear of being overheard, Research Assistants scheduled focus groups to take place in the community instead of the health facility being discussed. They also conducted discussions in the language preference of the group. FGDs were recorded, and summaries of the discussions were typed in English and sent to the IBTCI home office team, where they were entered by the Senior Program Coordinator, and coded by the Research Support Manager in Excel. The HIV/AIDS Program Management Specialist and Senior Evaluation Methodologist (from the IBTCI home office) reviewed the coded data sets for validity and reliability. Responses to close-ended questions were coded according to response summary consensus, or lack thereof. For example, answers to the close ended question “Is the group satisfied with the quality of services?” were coded as 1 for “Yes,” 2 for “No,” and mixed opinions were coded as 3. For open-ended questions (e.g., “What factors influence the decision to use or not use services?”). Similar responses were grouped under the same code. FGD results were not summarized statistically; the findings describe where salient results emerged in response to close-ended questions. Sampling. The evaluation team purposively selected districts to visit in each province. Selection criteria for districts included representation among graduated and non-graduated districts, as well as rural and urban areas, and the logistical feasibility for reaching district sites within the evaluation schedule. Two health facilities were purposively selected in each district. The selection of facilities was based on the following criteria: 1) to create a balance of representation among different levels of care (hospital and health center); 2) to collect data from facilities that offered a comprehensive range of HIV/AIDS services, especially ART services; 3) to create a balance of representation between private sector clinics and public facilities; and 4) the size of client attendance population at each facility.6 Focus group participants and clients for exit interviews were purposively selected because drawing statistically representative samples of beneficiaries of targeted health facilities was not feasible within the resource constraints of the evaluation. As a result, focus group and client exit interview findings reflect the views of participants and are valuable, but cannot be generalized to the entire population of beneficiaries and clients for facilities that are supported by ZPCT II and non-project supported facilities. Selection criteria for beneficiary FGD participants focused on HIV+ clients who received HIV/AIDS services (e.g. CT, PMTCT, ART, pre-ART, etc.) selected at the ZPCT II-supported health facilities in each of the visited districts. The evaluation methodology also included at least five client exit interviews to be conducted at each health facility visited, one for each of the five HIV/AIDS services as possible (i.e., CT, PMTCT, ART, clinical care, and MC). Triangulation of findings. To assure the validity and reliability of evaluation findings, the evaluation design included triangulation of information collected from different sources (i.e., desk review, respondents from different levels of care in six provinces, and representatives of different stakeholder groups), methodological triangulation (i.e., application of different data collection methods), and triangulation of findings from different evaluation team members with different areas of expertise accumulated from experience in Zambian and non-Zambian contexts. Client exit interview results, based on interviews conducted at health facilities, were triangulated with focus group findings, based on discussions conducted in the community. Coordination. Interview notes were shared between the sub-teams electronically and discussed in team meetings. The sub-teams communicated by phone and by email when traveling in the field. At the conclusion of field work the sub-teams compared evaluation results and discussed project implications in 15 order to establish a consensus on the main findings, conclusions, and recommendations included in this report. Limitations of evaluation methods. There are some limitations to the evaluation design. The team employed a purposeful sampling strategy for selecting districts and sites to visit, as well as FGD participants and key informants. Geographical accessibility was one of the criteria for the selection of districts in order to reach participating rural districts within a reasonable travel time. This factor has limited the representativeness of respondents, given that more accessible facilities tend to receive more support. In addition, the scheduling of data collection in the field coincided with Child Health Week, which negatively affected the availability of key MOH staff for an almost a two week period. The team triangulated information collected from the document review and data analysis from KIIs, FGDs, and client exit interviews. As these data sources were collected from individuals, information from interview data is subject to personal biases, opinions and recollection. G. FINDINGS & CONCLUSIONS BY EVALUATION TASK TASK I: EFFECTIVENESS Project Design, Performance, Contract Objectives and Implementation The evaluation team has considered the project design, contract objectives, performance and implementation through document reviews, in depth interviews, data analyses and review of project systems. In considering the project design, the team found no conceptual, logical or results frameworks on which the project design and monitoring would have been based. The evaluation team also reviewed the project objectives and found that, from a M&E standpoint, the current “objectives” are actually a combination of strategies and activities. The evaluation team, therefore, used the ZPCT II’s established five project objectives to create a simple logical framework and results framework to guide discussion. In Figure 1 two project objectives and three strategies are presented. Each of the implementation strategies contributes to meeting both objectives. Figure 1. Potential Logical Framework To frame the discussion of an approach for the second half of the project, the evaluation team has created a possible results framework by which USAID and ZPCT II can plan future activities. The suggested results framework, more clearly presents the same information contained in ZPCT II’s original objectives within two overarching project objectives and provides three intermediate results for each objective. Objective 1: Increase the proportion of people receiving integrated HIV/AIDS-related facility based services.  Intermediate Result (IR) 1.1: Improve public and private facility-based quality HIV/AIDS prevention, care and treatment services.  IR 1.2: Scale-up public and private facility-based coverage area of integrated HIV/AIDS prevention, care and treatment services. Objective 1: Increase proportion of individuals receiving facility services Objective 2: Decrease proportion of new HIV infections Strategy 1: HIV/AIDS Service Delivery Strategy 2: HIV/AIDS Planning & Delivery Strategy 1: HIV/AIDS Management Capacity 16  IR 1.3: Increase the capacity of the PMOs and DMOs to perform technical and program management functions. Objective 2: Decrease the proportion of new HIV infections.  IR 2.1: Improve community-based quality HIV/AIDS prevention and care services.  IR 2.2: Scale-up community-based coverage area of integrated HIV/AIDS prevention and care services.  IR 2.3: Increase the capacity of the PMOs and DMOs to perform technical and program management functions of prevention activities. After creating this framework the evaluation team organized ZPCT II’s existing program indicators, aligning them with the new objectives. These are seen in Table 3. Faithfully using the same ZPCT II indicators but rearranged, the evaluation team assessed project progress, to the extent possible, against objectives, intermediate results and targets that were originally established by ZPCT II. The team’s analysis found that for Objective 1, IR 1.1 Improve public and private facility-based quality HIV/AIDS prevention, care and treatment services and IR 1.2 Scale-up public and private facility-based coverage area of integrated HIV/AIDS prevention, care and treatment services, there are considerable output data (29 indicators) that can be tracked for progress against targets. This reflects considerable ZPCT II progress against service delivery targets. For a few targets on new service delivery activities (PMTCT partner participation and couples counseling) no targets were set, and therefore progress toward meeting these targets cannot be assessed. Other than a few similar new activities, most other indicators have targets upon which ZPCT II achievements have been reported. For Objective 1, IR 1.3, Increase the capacity of the PMOs and DMOs to perform technical and program management functions, ZPCT II has defined and reported on only one indicator. No additional indicators are defined or included (e.g., increased level of skill in management or technical capacity building). For Objective 2, IR2.1: Improve community-based quality HIV/AIDS prevention and care services, ZPCT II has defined three targets for training of community/lay persons: 1) CT, 2) PMTCT, and 3) ART adherence in facilities and communities. However, for Objective 2, IR 2.2 Scale-up community-based coverage area of integrated HIV/AIDS prevention and care services and IR 2.3 Increase the capacity of the PMOs and DMOs to perform technical and program management functions of prevention activities, there are no indicators, targets or achievements noted. These indicators and targets have not been defined and therefore, progress cannot be reported. If USAID and ZPCT II choose to adopt this or another conceptual framework for the current project’s remaining time or later for a future project, ZPCT II, with USAID guidance, will need to select indicators by which Objective 2 (prevention and care) activities can be measured. The lack of indicators and tracking of this objective represents a need to define more specific indicators and targets, and to realign and balance the project budget, staffing, and other resources in order to meet this objective. ZPCT II should work with an external consultant to facilitate in-depth discussions to help define and develop this or a similar framework for realigning the project. Strategic guidance from USAID will assist ZPCT II in this process. This process will help guide the second half of the project, as well as future projects to help ensure that the mix of activities and balance of resources are more clearly defined. This process will support the definition of more clearly defined project objectives, intermediate results, indicators, and targets. This process should focus on better alignment of project objectives, intermediate results, indicators, and targets with the Zambia National Strategic Health Plan (2011-2015). The information in Table 3 presents ZPCT II’s established targets and the reported performance against targets for the period from August 2009 to June 2012. These same targets and achievements are presented in the table in Annex H in a slightly different format with more detail. 17 Progress in Achieving the Life of Project Targets Objective 1. Expanding existing HIV/AIDS services and scaling up new services, as part of a comprehensive package that emphasizes prevention, strengthening the health system, and supporting the priorities of MOH and NAC. The evaluation team assessed ZPCT II’s performance against targets for the life of project presented in Table 3 and in the table in Annex D. Overall, ZPCT II has surpassed many of its service delivery and other Life of Project (LOP) targets or is on track to achieve most of the LOP targets, especially for activities related to HIV/AIDS services. Many indicators report performance at over 60 percent of the targets in the period from August 2009 to June 2012, which is over 60 percent of the project’s lifespan. This shows high output level performance by ZPCT II for most services delivery. As presented in Table 3 under IR I.2, during the period of August 2009 to June 2012, ZPCT II has met or exceeded the LOP targets for the service delivery indicators related to CT and CT for PMTCT. This is because ZPCT II has expanded and scaled up the HIV/AIDS services to health facilities and provided these services at higher levels than the targets established at project start-up. ZPCT II reported the achievement of 180.8 of the target for the delivery of CT services to individual clients. ZPCT II reported the achievement of 100 percent of the target for the delivery of CT for PMTCT services to pregnant women. The evaluation team analyzed the high performance against these targets and believes that there are two likely reasons for ZPCT II to have significantly surpassing these two service delivery targets. First, ZPCT II made CT, and CT for PMTCT service delivery a priority for project expansion. Second, the LOP targets may have been set to low at the beginning of the project. ZPCT II reported the achievement of 84 percent of the target for the delivery of ART prophylaxis treatment services to HIV+ pregnant women, which is on target for targets established over the LOP. ZPCT II reported the achievement of 44 percent of the adult palliative care service delivery targets and 32 percent of the pediatric palliative care service delivery targets. These numbers are not on track for the targets established over the LOP. Many ZPCT II LOP targets were set using a process in which the past performance of the ZPCT I project was considered and was not strictly set based on a new plan of ZPCT II activities, budgets and/or existing capacity. While ZPCT II performance is high, it should be noted that for some services, the targets were set too low. This could be because when ZPCT II planned these targets to build on the work of ZPCT I, the predecessor project, the previously given technical support to facilities and trained staff in each facility to continue with ZPCT II was not taken into consideration. For ZPCT I, some services included start-up activities. For example, some clients counted under the indictor “the number of clients started on ART” during the previous project, ZPCT I, may have been counted as “the number of clients on ART” in ZPCT II since their care was continued in ZPCT II with the same budget as in ZPCT I. Under IR 1, in Table 3, the evaluation team noted that ZPCT II has reported the achievement of approximately 55 percent of training targets for health workers; this is on track for the LOP targets set for this point in the period of performance. ZPCT II has reported the achievement of approximately 50 percent of targets for training health workers in PMTCT, ART and for treatment of Pulmonary Tuberculosis (TB) in HIV. The actual numbers of health workers trained appear relatively low relative to the need to provide several million clients with HIV/AIDS services in project supported sites. The evaluation team considers the issue that training targets could have been too high or MOH and facility staff could not have participate in trainings due to staff shortages and the issue of allowances. Accelerating training early in the project would address shortages of MOH staff and avoid the risk of untrained staff providing HIV/AIDS services. The evaluation team also suggests that ZPCT II should adjust the process of training to include more training of trainers (TOT) and ongoing mentoring. These accelerated training activities would likely help to mitigate the long wait times and compromised confidentiality reported by client respondents in FGD and client exit interviews. The evaluation team notes that HIV/AIDS services at ZPCT II supported facilities are in very high demand. Annex G provides a table of ZPCT II training data disaggregated by course, sub-course and province. This table illustrates ZPCT II activities to strengthen human resource support. 18 Table 3. ZPCT II Indicator Tracking Table Objective/Strategy/Intermediate Result Life of Project (LOP) Indicator Target (August 2009 - May 2014) Actual (as of June 2012) % Achieved Objective 1: Increase the proportion of people receiving integrated HIV/AIDS-related facility-based services Intermediate Results 1.1: Improve quality of public and private facility-based HIV/AIDS prevention, care and treatment services HIV+ clients attending HIV care/treatment services that are receiving treatment for TB 17,000 15,829 93.1 Training Individuals trained in CT according to national or international standards 2,316 1,413 61.0 Health workers trained in the provision of PMTCT services according to national or international standards 5,325 3,097 58.2 Individuals trained to provide HIV palliative care (excluding TB/HIV) 3,120 1,759 56.4 Health workers trained to deliver ART services according to national or international standards 3,120 1,759 56.4 Individuals trained to provide treatment for TB to HIV+ individuals (diagnosed or presumed) 3,120 1,759 56.4 Individuals trained to provide MC services 260 291 111.9 Individuals trained in the provision of laboratory-related activities 375 725 193.3 Objective 1: Increase the proportion of people receiving integrated HIV/AIDS-related facility-based services Intermediate Result 1.2: Scale-up public and private facility-based coverage area of integrated HIV/AIDS prevention, care and treatment services Services delivered Individuals who received HIV/AIDS CT and received their test results 728,000 1,316,398 180.8 Individuals who received HIV/AIDS CT and received their test results (including PMTCT) 1,300,000 1,888,458 145.3 Number of pregnant women receiving PMTCT services with partner N/A 44,225 N/A No. of individuals who received testing and counseling services for HIV and received their test results (tested as couples) N/A 92,412 N/A Pregnant women who received HIV/AIDS CT for PMTCT and received their test results 572,000 572,060 100.0 HIV-infected pregnant women who received antiretroviral prophylaxis for PMTCT in a PMTCT setting 72,000 60,987 84.7 Individuals provided with HIV-related palliative care (excluding TB/HIV) (adults and children) 560,000 246,335 44.0 Pediatrics provided with HIV-related palliative care (excluding TB/HIV) 60,000 19,377 32.3 Individuals newly initiating on ART during the reporting period 115,250 89,346 77.5 Pediatrics newly initiating on ART during the reporting period 11,250 6,846 60.9 Individuals receiving ART at the end of the period 146,000 155,473 106.5 Pediatrics receiving ART at the end of the period 11,700 10,764 92.0 Registered TB patients who received HIV/AIDS CT and their test results at a USG-supported TB service outlet 30,400 25,351 83.4 Number of males circumcised as part of the minimum package of MC for HIV prevention services N/A 7,940 N/A 19 Objective/Strategy/Intermediate Result Life of Project (LOP) Indicator Target (August 2009 - May 2014) Actual (as of June 2012) % Achieved Tests performed at USG-supported laboratories during the reporting period: (a) HIV testing, (b) TB diagnostics, (c) syphilis testing, and (d) HIV/AIDS disease monitoring 3,813,000 3,850,122 101.0 Service outlets and laboratories Service outlets providing CT according to national or international standards 370 389 105.1 Service outlets providing the minimum package of PMTCT services 359 376 104.7 Service outlets providing ART 130 148 113.8 Service outlets providing HIV-related palliative care (excluding TB/HIV) 370 389 105.1 Service outlets providing treatment for TB to HIV+ individuals (diagnosed or presumed) in a palliative care setting 370 389 105.1 Service outlets providing MC services 50 44 88.0 Laboratories with capacity to perform: (a) HIV tests and (b) CD4 tests and/or lymphocyte tests 111 113 101.8 Laboratories with capacity to perform clinical laboratory tests N/A 140 N/A Private health facilities providing HIV/AIDS services 30 18 60.0 Objective 1: Increase the proportion of people receiving integrated HIV/AIDS-related facility-based services Intermediate Result 1.3: Increase the capacity of the PMOs and DMOs to perform technical and program management functions Local organizations (PMOs and DMOs) provided with technical assistance for HIV-related institutional capacity building 47 47 100.0 Objective 2: Decrease the proportion of new HIV infections Intermediate Results 2.1: Improve community-based quality HIV/AIDS prevention and care services Community/lay persons trained in counseling and testing according to national or international standards (excluding TB) 2,506 1,373 54.8 Community/lay persons trained in the provision of PMTCT services according to national or international standards 1,425 1,000 70.2 Community/lay persons trained in the provision of ART adherence counseling services according to national or international standards 600 610 101.7 Objective 2: Decrease the proportion of new HIV infections Intermediate Results 2.2: Scale-up community-based coverage area of integrated HIV/AIDS prevention and care services No indicators for this IR Objective 2: Decrease the proportion of new HIV infections Intermediate Results 2.3: Increase the capacity of the PMOs and DMOs to perform technical and program management functions of prevention activities No indicators for this IR 20 Expand Quality Services. The target for the number of HIV+ clients attending HIV care/treatment services that are receiving treatment for TB addresses quality, especially given issues of co-infection. The project has done well in the integration of HIV with TB treatment. Additional indicators to measure quality would help the project assess the degree to which ZPCT II support is successful at improving the quality of prevention, care, and treatment. Scaling-Up Services. ZPCT II achieved the scaling-up of services by introducing HIV/AIDS-related prevention, care and treatment at new sites and by increasing the number of people treated in health facilities. In Annex D the status of scale up activities as of June 2012 are presented. Under IR 1.3, ZPCT II scaled up ART service delivery resulting in 89,346 new individuals initiating ART. ZPCT II reported that the scale up of pediatric services has resulted in reaching over 44 percent of the estimated 350,000 clients countrywide who are currently on free ART. Service Delivery for MC service was also scaled up. Although at start-up, no LOP target was set for MC, a target of 8,000 was set for MC in 2011. MC was scaled up rapidly and exceeded the target, providing 7,940 procedures by June 2011. ZPCT II adjustments of targets for MC services will need to consider the recent large donor funds provided for MC. Scale up of care activities includes HIV palliative care and support. Some 246,335 adults and children received palliative care against a LOP target of 560,000. While reaching this number is impressive, the LOP target of 560,000 far exceeds the number of positive individuals being diagnosed as HIV+. Under Objective 1, the first recommendation addresses the need to promptly readjust and set challenging, but achievable targets in order to better balance resources and to meet all objectives. Under IR 1.2, the team noted that the number of ZPCT II supported service outlets providing CT, PMTCT, and palliative care has slightly exceeded the target of 370 people. ZPCT II has also slightly exceeded the targets for the number of service outlets for ART, TB in HIV, and lab services. The number of outlets counted included ZPCT II supported sites that were supported under ZPCT I as well as new sites. The evaluation team understands the nature of PEPFAR reporting and the pressure for expansion. It would be helpful for management purposes to have additional data that tracks ZPCT I and newly supported sites separately. Objective 2. Increase the involvement and participation of partners and stakeholders to provide a comprehensive HIV/AIDS service package that emphasizes prevention, strengthens the health system and supports the priorities of the MOH and NAC. Under Objective 2, ZPCT II supports laboratories with sub-partner, MSH. ZPCT II is currently on track to reach Objective 2 since all of the LOP targets are over 70 percent or fully achieved. The project supports 140 laboratories in 124 public facilities and 15 private health facilities. ZPCT II reported that 113 of these laboratories have the capacity to provide HIV testing and CD4 count analysis or total lymphocyte count analysis, while the remaining 26 provided basic laboratory support. Objective 3. Increase the capacity of the PMOs and DMOs to perform technical and program management functions. ZPCT II is currently behind in reaching Objective 3 targets, as only 24 out of the 42 districts have graduated, which represents 57 percent of the total targeted districts. With the remaining time to project end date, ZPCT II needs to accelerate capacity building in the remaining districts to order to achieve this objective by the project’s end date. Cardno's management and organizational development inputs are assisting PHOs and DHOs to improve governance, human resources, planning, and financial management practices. At the time of this evaluation 12 of the 24 districts that had graduated across the six provinces were been assessed using capacity building management indicators in the following areas: human resource retention, performance management, financial management, and action plan reviews. This assessment is part of Cardno’s capacity development strategy. ZPCT II reported providing technical assistance to 47 PHOs and DHOs in HIV-related institutional capacity building which is the LOP target. In particular, ZPCT II provided these PHOs and DHOs with support in the integration of HIV/AIDS services into MOH health services for reproductive health 21 (RH); malaria; and maternal, newborn and child health (MNCH). ZPCT II sponsored trainings in these technical areas to strengthen the capacity of PMOs and DHO staff in providing facility mentorships and technical assistance. Objective 4. Build and manage public-private partnerships to expand and strengthen HIV/AIDS service delivery, emphasizing prevention, in private sector health facilities. According to the Performance Monitoring Plan (PMP), Objective 4 will be met if 30 PPPs for HIV/AIDS service delivery are established in all targeted provinces through the implementation of tested technical approaches from the public sector. ZPCT II is somewhat behind in reaching Objective 4. The team noted that ZPCT II renewed the 18 Memorandums of Understanding (MOUs) with private sector facilities that were developed during the ZPCT I, to ensure technical support continued. The ZPCT II LOP target was to engage 30 PPPs; this target was interpreted to include the 18 continuing PPPs initiated by ZPCT I. In consideration of how the budget for these activities was planned, if this interpretation is approved by USAID, ZPCT II has achieved 60 percent of this performance target. If, however, the target of 30 denotes that ZPCT II is required to establish 30 new PPPs, limited progress in this area has been made. Table 2 does not show the reporting of private sector services from other than the previous 18 facilities. ZPCT II reports that the 12 remaining facilities to achieve the LOP target have already been identified and MOUs have been signed. However, it is not clear if the criteria of a signed MOU should be an appropriate performance measurement for this objective, “to build and manage of PPPs to expand and strengthen HIV/AIDS service delivery, emphasizing prevention, in private sector health facilities.” If USAID approves that the signing of an MOU is the only indicator or performance measurement for effectively building and managing PPPs, the LOP target for PPPs will be reached by the end of 2013. ZPCT II developed criteria for selecting private sector sites which ensure that sites with comprehensive HIV/AIDS services were supported in areas of CT, PMTCT, ART, MC, laboratory, and pharmacy services. It is unclear if the intent of this objective was to choose sites where all these services existed previously and for ZPCT II to continue support, or if the intention was to establish new sites and assist these sites to achieve a comprehensive service delivery. The sites that met the selection criteria were mainly found in Copperbelt Province, but some are also located in Central and North-Western Provinces. The target for meeting PPPs does not state that ZPCT II is required to initiate PPPs in each province, nor spread the PPPs across the provinces. A full discussion of activities of PPPs is covered in the Task VIII Section. Objective 5. Integrate service delivery and other activities, emphasizing prevention, at the national, provincial, district, facility, and community levels through joint planning with the GRZ, other USG and non-USG partners. ZPCT II does not have indicators that measure the integration of services and activities at the national, provincial, district, facility, and community levels. In addition, the project has not defined indicators to track coordination with GRZ, other USG agencies and other non-USG donors and partners. Therefore, the evaluation team based the assessment of Objective 5 performance on the definition in the PMP: “ZPCT II activities are incorporated into all PHO and DHMT action plans annually and ZPCT II is participating in all 12 Technical Working Groups (TWGs) with the MOH, NAC and other partners.” The evaluation team found ZPCT II to be behind in achieving Objective 5 as joint planning and participation in working groups with the MOH is weak in many PHOs and DHOs. As for integration of family planning (FP), maternal child health (MCH), and other integrated services for HIV, some progress has been made. There is still a need to develop referral mechanisms and further strengthen established ones. Furthermore, although ZPCT II has collaborated with the MOH, PMOs, DMOs, and health facilities, as well as provided technical support to plan in-service integration, little was done to encourage joint planning at those levels to operationalize it with integrate services. In contrast with ZPCT II performance in the integration of service delivery and other activities with MOH partners, the evaluation team found collaboration was ongoing with other USG partners, including John Snow, Inc. (JSI), the Society for Family Health (SFH), and JHPIEGO at the provincial-level in most provinces. Similarly, at the national level, ZPCT II regularly meets with other USG partners such as JSI (on commodities logistics systems), the SFH, Marie Stopes International (on FP and MCH services), and 22 JHPIEGO (on MC services). Unfortunately, these meetings were not formalized at the provincial level and did not lead to active integration of services. As for collaboration with the civil society, the evaluation team also found that ZPCT II often collaborates with DATFs and PATFs based in the provincial capital, where ZPCT II’s provincial office is located. The team found that the partnerships between ZPCT II and DATFs were essentially non-existent in districts other than that of the provincial capital. The evaluation team reviewed a selection of ZPCT II reports for information on Objective 5. In the ZPCT II October – December 2010 Quarterly Report, ZPCT II refers to the integration of services and the provision of support to ART outreach to two home-based care centers. The July – September 2011 Quarterly Report refers to the same home-based care activity and added, “five mobile counseling and testing episodes were jointly planned and conducted.” The October – December 2011 Quarterly Report refers to the same ART outreach activity. In this same report, under “Meetings and Workshops Attended”, FP commodities meetings were referenced, but ZPCTY II’s role was not discussed. The evaluation team did not find (in the reports reviewed) references to the integration of FP or MCH with HIV. Some reports discussed screening for chronic diseases and activities for Gender-Based Violence (GBV) under the “Palliative Care” section. The team found few activities during field visits that relate to integrating services such as family planning or MCH with HIV. TASK II. EFFICIENCY Mix of Activities and Funding Allocated to ZPCT II Despite the reduction in prevalence from 16.1 percent in 2002 to 14.3 percent in 20077, the number of people that contract the HIV virus is still significant. This issue is related to the limited adoption of prevention activities by the population that will require changes in behavior and in cultural attitudes and increased access to effective prevention information and outreach from not only the MOH but also other sectors and civil society. HIV/AIDS prevalence is also related to the ART scale up that has led to more People Living with HIV/AIDS (PLHA) living longer, and requiring more ARV and other medications and care). The ZPCT II project strengthened and supported various interventions outlined in the National Health Strategic Plan (NHSP) in 2011-2015, to address the HIV/AIDS problem in Zambia in the areas of prevention, treatment, human resource, pharmacy, and medical supplies, medical equipment and infrastructure and ICT. ZPCT II has also contributed greatly to progress toward the national targets. The evaluation team suggests a modified mix of activities to as an optimal combination later in this section. Adequacy and Appropriateness of Planned Activities and Corresponding Budgets The ZPCT II project has an approved ceiling of $124,099,097 for the life of the project. The project’s cumulative obligated amount was $82,818,000, out of which $68,801,115.29 was spent as of June 30, 2012 leaving a total balance of $55,297,981.71. Looking at current expenditure and outputs, the project budgets are more than adequate to achieve the targets and project objectives. Using the calculated/projected life of project monthly burn rate of $2,049,311, the remaining obligation is estimated to last for the remaining duration of the project. This is possible because the project has already met and or exceeded the LOP targets in some areas. This includes the provision of CT services at MOH facilities for which ZPCT II has achieved 181 percent (1,316,398 against the target of 728,000 for the number of adults being tested who got results). ZPCT II has also achieved 107 percent of the target for the number of individuals receiving ART (155,473 against the target of 146,000). The project also doubled the number of MC procedures and graduated over half of supported districts. With additional budget and an emphasis on balancing activities for prevention with those for treatment, ZPCT II can increase prevention activities given the GRZ priority on accelerating prevention. Additional funds for community level prevention activities can be accelerated with increased staff, training, materials, and activities focused on prevention. Expanding ZPCT II capacity building support to PHOs and DMOs to increase their technical and management capacity is also appropriate and would further support the expansion of all HIV/AIDS services. Increased support and coordination with civil society organizations through collaboration with provincial and district AIDS Coordinating Networks could also accelerate prevention, as well as strengthen referral networks that support ART treatment. 23 ZPCT II budgets are based on PEPFAR funding areas; therefore, it is difficult to track budgets by activity. However, a review of ZPCT II expenditures showed that project resources were used according to approved funding areas. The administration budget was over-spent and it has reportedly been negatively affected by increased costs of maintaining project vehicles older than five years. Increased expenses for vehicle maintenance have reduced funds for technical and mentorship field visits. Cost-efficiencies The amount of unspent funds remaining is $55.2 million, up to June 2014. Based on the current LOP burn rate, of approximately $2 million per month, the project is likely to reach its end date and remain with an amount of unspent funds between $6 to over $11 million – unless ZPCT II increases its burn rate to above $2.3 million per month. It appears that ZPCT II is not doing activity based budgeting. If done, it would allow a clearer calculation of the cost of various activities/services, which would, in turn, allow a more accurate planning of expenditures. Modified Mix of Activities The evaluation team was asked to identify a modified mix of activities to accelerate the project impact. The team identified the following opportunities to modify the mix of activities after discussion with a number of project stakeholders.  The direct training of Health Care Workers (HCW) could be adjusted to provide more Training of Trainers (TOT) at the provincial and district levels to carry out the cascade training approach. In combination with the TOT, ZPCT II staff could increase their mentoring as follow-up. This would require some adjustments to allowances, but this step has the potential to increase the technical capacity and strengthen partnerships between ZPCT II and the PHOs and DHOs.  Providing more funding, training, and support to Community Based Organizations (CBOs) to expand community based prevention, GBV sensitization, and support activities could greatly accelerate the mix of activities in behavior change for prevention, as well as care, referral, and adherence. Although many small CBOs have limited capacity, a strategy to identify and strengthen an increased but small number core CBO partners that could meet Fixed Obligation Grants (FOGs) requirements. ZPCT II could then support and build the capacity of CBOs to catalyze the many members of the numerous churches, women’s groups, HIV positive support groups, and other associations. This step would provide cost effective methods to expand community based HIV activities.  Decentralizing ZPCT II staff and recruiting local staff to more directly support districts could be considered. This would allow significantly more staff time to directly support district offices and health facilities. Recruiting local staff from the area could also make retention more likely. The actual cost effectiveness of ZPCT II should be studied. This analysis should determine the likelihood of this model to provide more consistent technical assistance, mentoring, and support to districts and facilities while reducing high transport costs.  A cost effective approach to providing technical assistance could be the training of ZPCT II staff as polyvalent technical advisors. This is a realistic and practical approach to providing technical assistance to more sites. This mix of skills in the advisors would support the cross training of DHO and facility staff who manage multiple HIV/AIDS services and interventions. ZPCT II states that they have already adopted this approach – it could be expanded. TASK III. CROSS-CUTTING PRINCIPLES The ZPCT II key cross-cutting principles represent a set of beliefs articulated by the USG on how it will work with partners within the context of Zambia’s nationally agreed upon HIV/AIDS response. The principles cut across USG supported programs and range from issues of alignment, which supports working within government systems to promote “aid effectiveness” and human resource development, integration of gender to promote equal access to services and equity in utilization, to include issues of sustainability of outcomes of technical, managerial and financial support. These principles are expected to 24 have been incorporated into ZPCT II’s programming. Therefore, evaluation team assessed the application and incorporation of these cross-cutting principles into ZPCT II’s operations. Although the results of the evaluation team’s assessment of cross-cutting principles are integrated into the body of this report, some of the principles are discussed in more detail in Annex O in response to specific questions raised in the SOW. A summarization of the ZPCT II’s achievements in addressing cross￾cutting principles are presented in Table 4, as well as references to sections of the report where the assessment of the principle is embedded. Factors of Successes in Project Systems and Management The evaluation questions regarding factors related to successes in project systems and project management did not elicit many answers from health staff. However, the following factors were discussed as related to the success of the ZPCT II project. Quarterly district technical review meetings. These are meetings with ZPCT II, DHO technical staff, and facility in-charges. Data on HIV/AIDS were reported to be shared and discussed during some of these meetings. While the actual frequency of such meetings appear to vary considerably, where they reportedly occur, they are found to be helpful. Review meetings were also reported to have taken place at some facilities. Allowances, including community volunteer allowances and transportation allowance for staff overtime. Transportation allowances are provided to community volunteers to facilitate their travel to the health center, where they are expected to work two days per week in addition to the work in their community. The evaluation team found that the currently distributed allowances contribute to building the skills and experience of community members as well as extends a modest amount of coverage of services by staff. Practices, products, and tools to replicate Respondents mentioned that the following practices should be replicated. Outreach ART Services. This outreach effort allows HIV services to reach more rural populations. ZPCT II support is provided in the form of fuel and supplies. The same MOH staff and ZPCT II/MOH protocols are used for outreach ART services. Mentorship. Respondents mentioned both formal and informal mentorship as a practice that built the capacity of staff and volunteers. There is potential for expanding mentorship by ZPCT II staff especially if decentralization moves forward. Polyvalent Health Worker Capacity Building. Since facility staff numbers are limited, staff members often provide multiple HIV services and adopt task shifting practices. Respondents discussed the advantages of being cross-trained in HIV/AIDS tasks. Respondents mentioned that the following tools should be replicated. Job aids for clinical guidelines. Staff found ZPCT II job aids helpful and were interested in further support to replicate aids, such as posters for ART treatment and defaulter definitions for use at non￾project supported sites. If these reproductions can be supported by the project, providers would find it valuable and it would support DHO district-wide capacity building efforts. SMS technology. SMS has been developed to contact clients on their cell phones regarding appointments, and other information to increase continuity of care. Referral forms. These were developed and used as a tool to strengthen the referral networks between facilities and communities. Use of encrypted email results to send pediatric diagnostic results. This technology tool is being used is some locations to inform parents and to reduce waiting time. Tracking tools. Such as the “Baby Tool” is being used to track babies of HIV + mothers at Arthur Davidson Hospital to improve follow up after delivery. 25 Table 4. Cross-Cutting Principals Cross Cutting Issues Summary of Achievements Embedded in Section Alignment ZPCT II is aligned with MOH and NAC Strategic Plans/Frameworks and the USG Partnership Framework but is not aligned at operational level at provincial and district levels, e.g. planning and budgeting cycles, and performance monitoring processes. Annex on Cross-cutting Principles Partnership Strength of the partnership varies across provinces with ZPCT II reportedly working more collaboratively with some PHOs and DHOs than others. In some districts ZPCT II was perceived as working outside of the partnership arrangement. Task VIII. Linkages, Coordination and Leveraging; and Task X. Management Systems - Partnerships Continuum of Care Referral system established; data collected on loss to follow-up. Annex on Cross-cutting Principles and Task I. Effectiveness Innovation ZPCT II has instituted SmartCare development of database, Short Message Service (SMS) to notify/remind clients of appointments; facility renovation to promote client & staff satisfaction Task IV. Factors of Success Linkages Formal agreements between MOH and private sector partners with ZPCT II signed. Task VIII. Linkages, Coordination and Leveraging Procurement Although equipment is procured based on the identified needs of facilities and through MOH approved vendors, the procurement system is not totally aligned with MOH’s planning, budgeting, and equipment procurement and maintenance system. Task V. Obstacles; and Task VI. Sustainability Zambian Leadership Development and Human Resource Support 99% of ZPCT staff is Zambian; and MOH partners at provincial and district levels are receiving technical and managerial capacity building, and gender sensitization. Task I. Effectiveness; and Annex on Cross-cutting Principles Gender integration Sensitization and training is taking place, and two gender-related indicators are being tracked in ZPCT II’s PMP. Task IV. Factors of Success; Task V. Obstacles; Task IX. Performance Monitoring and Annex on Cross-cutting Principles Sustainability Respondent’s views varied about the sustainability of ZPCT II. Task VI. Sustainability Environmental Considerations ZPCT II is compliant with the laws and regulatory policies of Zambia. Annex on Cross-cutting Principles TASK IV. FACTORS OF SUCCESS Success in Implementation and Causative Factors The evaluation team interviewed MOH staff at the national, provincial, and district health levels, as well as facility staff to elicit their perspectives on the successes and success factors of ZPCT II support. The frequently cited successes are presented in Table 5, which includes factors that were identified as related to the successes. 26 Table 5. Project Successes Success in implementation Aspects of Successes, Including Quotes from Key Informant Interviews Factors Related to the Successes (from MOH respondents) Health facilities are able to offer comprehensive services that may not have been offered at the site before  Services offered at 371 MOH and 18 private facilities: - Basic package (CT, PMTCT, Care) in 97% of facilities. - Expanded package (ART, clinical lab) in 38 % of facilities.  Comprehensive services: - “They look at infrastructure, empowering Human Resources Health (HRH), community volunteers, rehabilitation, support CT, PMTCT, ART.” Health Facility Staff - “Focus on excellent patient care […] they didn’t just concentrate on our knowledge. ZPCT II looks at the patient like a whole, the whole picture of how the client experiences the visit.” Health Facility Staff - “Enrollment in ART and PMTCT is increasing: this we cannot take away from them.” DHO Staff - “[ZPCT II] contribute[s] and help[s] to implement government guidelines.” MOH Respondent - “[Most of the impact of ZPCT II has been] at the implementation level.” MOH Respondent  Training (ART, MC, PMTCT)  Use of nurse practitioner/s  Ongoing mentorship (NW Province)  ART outreach sites  Lab sample referral system  Renovation of space for service  Providing supporting materials, equipment, supplies  Laboratory support of clinical services Large increase in Voluntary Counseling and Testing (VCT) uptake  181% of Life of Project (LOP) target.  “Before ZPCT II there was low demand and even where there was demand we didn’t have manpower and referral systems set up. We are benefiting.” Health Facility Staff  “Community mobilization for HIV sensitization and promotion of CT has been a resounding success and people get to the facilities on their own including coming in as couples.” Health Facility Staff  Lay volunteers provide counseling  Training and refresher courses for lay counselors  Mobile counseling and testing ART  “Reduction in default rate as a result of Smart-Care which makes cheating by clients impossible as the Smartcard contains all the information on the patient.” Health Facility Staff  “The community component started/ existed before ZPCT II but there were no incentives.” Health Facility Staff  “… empowered with a lot of information on HIV.” Health Facility Staff  “ZPCT II helps us keep accreditation every year in preparation for [Council of Health Professions] accreditation.” Health Facility Staff  “We would not been able to provide services without their support.” MOH Respondent  Adherence support workers  Mobile ART  SmartCare  Clinical training  Good access to ARV  Lab and pharmaceutical network  Data entry clerks  Computers PMTCT  “We’re doing well because people are coming, people are knowledgeable about services, know that they are supposed to do testing and come to ANC.” Health Facility Staff  Clinical training  Lay volunteers provide counseling  PCR lab for infant diagnosis Greater male involvement in PMTCT  “They come with their partners—95%.” Health Facility Staff  ZPCT II numbers reported in the January – March 2012 Quarterly Report was 33.3%.  19,823 men tested as partners of 59,469 pregnant women who received PMTCT services.  Availability of couples counseling Stronger integration of gender  Institutionalization of screening clients for GBV.  Recognition and efforts made to infuse GBV into the referral directory.  TOT in gender carried out for senior staff in PHO and DHOs Stronger adherence to environmental compliance  Staff is more aware of the need for the use of cardboard sharps boxes to protect themselves and others from contaminated needles and other sharps. 27 TASK V. OBSTACLES AFFECTING PROGRESS ZPCT II’s Key Obstacles in Field-Level Implementation Civil service human resource remuneration and benefits packages (allowances). The first and most frequently discussed obstacle to field level implementation was the issue of allowances. MOH and ZPCT II staff frequently mentioned that USG regulations regarding the MOH and other GRZ allowances are an obstacle. Respondents mentioned that these regulations affect MOH budgets at the provincial and district levels. As per USG regulations, ZPCT II (and other partners) provides reimbursement for out of town travel, and reimbursement or allowances for lunch during trainings or meetings that exceed six hours, or a daily subsistence allowance if the training or meeting requires overnight stay. According to GRZ regulations, government employees are entitled to out-of-pocket allowances which cover other incidental costs, but these not allowed by USG regulations. USG contractors usually pay lower than the mandated GRZ allowances. This generally affects the motivation of GRZ employees to attend ZPCT II sponsored trainings. It also incurs debts for the PHOs and DHOs that must make up the difference from their budgets, as the allowances are part of civil servants’ conditions of service. This amounts to costs up to millions of kwacha (KW) that are not accounted for when GRZ agencies plan and allocate their budgets. The PHOs and DHOs do not have control over the schedule ZPCT II supported trainings, nor control over the debt that these trainings incur. There is also donor inconsistency on allowances paid for training and other out-of-pocket payments by implementing partners. After a period of six years, a policy dialogue on this issue has been reactivated with discussions in early October 2012. Community volunteer allowances. Transportation allowances are provided to community volunteers at approximately 150,000 KW/month to facilitate their travel to the health centers, where they are expected to work two days per week in addition to the work in their community. This small amount limits the willingness of some volunteers to work in facilities or to travel to do follow-up on adherence, or to promote MC or other HIV/AIDS services. The amount of volunteer allowances varies by agency affiliation. Some MOH officials and staff have stated that volunteers are to be considered as volunteers and not provided with transport allowances. This constraint does not affect ZPCT II alone, but it is a larger issue that affects current volunteer motivation and long-term sustainability of the use of volunteers. Inadequate MOH human resources. Human resource numbers are significantly low at all levels of the MOH affecting its ability to provide adequate, quality services and to participate fully in ZPCT II training, mentoring and other capacity building activities. The availability of MOH staff at all levels is affected because staffing is currently inadequate and GRZ employees are offered or expected to participate in competing training opportunities from different donor funded projects. This lack of coordination affects MOH programs and motivations of overwhelmed staff at all levels. Turnover of MOH staff is also high. Some PHO staff members stated that they move ZPCT II trained staff from ZPCT II supported districts to non-supported districts in order to expand HIV/AIDS services. The previous Government of Denmark manpower study and the MOH Zambia Integrated Systems Strengthening Project (ZISSP) supported a planned Human Resource Database that should provide basic data for MOH/GRZ human resources planning and decision-making. Constraints on construction. Limited funds and PEPFAR regulations constrain construction in facilities; this presents obstacles to providing adequate space for CT, PMTCT, ART, and other services. This constraint has resulted in overcrowding at established facilities. ZPCT II has funded some renovations to facilities to provide needed space for counseling and other HIV/AIDS services allowed under PEPFAR regulations. Use of data entry clerks. Data entry clerks are identified as ZPCT II employees. The MOH recognizes their valuable services such as data entry, processing, and reporting. However, the supervision of data entry clerks by the MOH and ZPCT II does not always allow these employees to function as needed. Staff in the Central and Northern Provinces noted that they are required ask permission from ZPCT II for data entry clerks to travel with outreach teams. At other times, data entry clerks do not perform services in facilities requested by staff because “it is not included in their job description” – this presents an obstacle to the smooth functioning of facilities. 28 Gender Related Obstacles. Field level implementation is also affected by gender-related obstacles. The gender audit completed by the USAID Office of the Inspector General in 2011 (see Annex O) presents a number of deficiencies in ZPCT II’s gender strategy. In response, ZPCT II initiated a number of activities, such as a Steering Committee, manuals, guidelines, and a data base. The evaluation team found that obstacles remain to adequate gender integration. As indicated in Annex O, ZPCT II, PHO, and DHO staff were prioritized for gender sensitization and GBV training for the purpose of overseeing the effective integration of gender into clinical care. However, most gender-related factors that drive the HIV/AIDS epidemic are socio-cultural and occur at household and community levels where community volunteers and lay counselors operate. Community volunteers and lay counselors have yet to be trained in gender integration as they engage in behavioral change communication when working with households and communities to prevent GBV and to make appropriate referrals as required. Community volunteers and lay counselors are primarily in regular contact with multi-sectoral forums that feed into district development planning mechanisms (such as Area Development Committees). ZPCT II supported gender sensitization and GBV training for community volunteers and lay counselors would enable these partners to provide more effective contributions through these forums. During fieldwork in two provinces which include urban clinics, clinical staff said they used the GBV checklist but had not identified any GBV cases. Considering evidence from the two most recent Zambia Demographic and Health Surveys that indicate GBV is widespread and pervasive, the evaluation team concluded that screening tools and activities are inadequate and need to be supplemented with community level screenings and referrals. The evaluation team did not ascertain the degree of progress made by PHOs in cascading gender training down to DMOs and health workers for the purpose of engendering work plans. However, this approach may encounter exterior obstacles because of recent changes in the health sector, and the separation of roles and responsibilities between the MOH and the Ministry of Community Development, Mother and Child Health (MCDMCH). Team discussions with the SI Advisor and Gender Specialist identified the lack of support services to refer cases from health facilities and the absence of a Steering Committee to coordinate gender mainstreaming as constraints in the program. Gender is a cross-cutting issue in Zambia’s national development strategy, requiring a multi-sectoral approach, as well as harmonization and synchronization with other sector interventions. Since gender is linked to effective behavioral change for HIV prevention, uptake of counseling, treatment, and adherence, ZPCT II should increase mentoring, supervision, and refresher training to address prevailing cultural norms that continue even after sensitization training. Operations research is needed to show the impact of gender on service outcomes. Rigorous analysis of the results of couples counseling and male involvement in PMTCT is also needed so that the information is infused into ZPCT II’s training and mentoring processes. Top management support of the gender component would also increase its full integration within the project. Obstacles to collaboration with NAC and community structures. Collaboration with formal and informal community structures is critical to ZPCT II success. NAC structures at the provincial, district, community, and household levels exist and are charged with coordinating HIV/AIDS activities with community leaders, CBOs, FBOs, youth groups, schools, and many other private and public sector organizations across all sectors. ZPCT II has supported a number of PATFs and DATFs, some even served as chairs, assisted to assess community groups and the development of district level directories of health organizations and actors. However, limited collaboration with PATF’s (in districts among Copperbelt, Northern, and Luapula Provinces) were reported by District AIDS Coordination Advisors (DACA). The requirement that ZPCT II registers with the DATF is not met in some districts. There are opportunities to overcome these obstacles by increasing linkages with CBOs and exchanging lessons learned. These linkages could provide a broad base of community support for ZPCT II prevention activities and community mobilization for CT, MC, etc. Informing the DATF when ZPCT II funds a CBO in their district would increase coordination and DATF guidance. ZPCT II’s Key Obstacles in Operating Systems ZPCT II procurement and maintenance. There are some obstacles to continuous functioning of sophisticated HIV diagnostic equipment procured by ZPCT II, due to limited and erratic power supplies, limited availability of staff trained in their use, inadequate maintenance contracts and other conditions 29 which are often not within ZPCT II’s control. Contracts that include short maintenance clauses and a limited number of commercially trained maintenance technicians in Zambia often results in long repair delays that limit lab tests needed for HIV/AIDS care. ZPCT II’s system for procurement and tracking is not fully integrated with that of the MOH; this negatively affects the MOH’s procurement, tracking and maintenance of equipment. Facility staff reported that the MOH is responsible for the maintenance of ZPCT II procured equipment, however, these costs have not yet been built into the MOH’s procurement system and budget. Joint programming systems obstacles. The difference in the project systems used for ZPCT II planning, supervision, training and monitoring and others affect ZPCT II’s ability to integrate with the MOH. Different planning cycles of the MOH and ZPCT II present obstacles to joint programming. The districts and facilities plan their annual activities from January to December while ZPCT II planning is based on an October to September fiscal year. This affects the joint programming between the MOH and ZPCT II in planning, financial reporting and budgeting, technical support, supervision, and other systems. Obstacles affecting M&E and supervision. In response to the pressures of PEPFAR reporting, ZPCT II supports data entry clerks. ZPCT II intends for these clerks to report to their M&E officers, as well as the District Information Officers (DHIO). However, district staff reported that sometimes data are not routed to DHIOs. DHOs report that they get summaries of the data but often do not get the raw data that is sent to ZPCT II by the data entry clerks. This presents an obstacle to their access and ability to analyze the data. About the supervision of HIV activities, facility staff reported that ZPCT II advisors arrive at the time to perform a “supervisory support” without notice and without seeking permission from DHO staff to visit facilities. This presents obstacles to strong working relationships and affects the continuity of M&E and supervisory systems of the district. Financial systems obstacles. ZPCT II manages all funding for Recipient Agreements (RA) reportedly due to donor requirements for accountability. RAs can provide valuable documentation of ZPCT II and MOH collaboration at various levels. Some District Health Management Teams (DHMTs) report that they are being asked to sign RAs without adequate time to review and discuss with ZPCT II. This limits the ability of DHMTs to make decisions regarding the financial resources to be used in their districts. It can also constrain the ZPCT II mandate to strengthen MOH ownership. RAs, once signed, are filed or sometimes lost; some staff reported that RAs are often not used as a tool for ongoing joint collaboration. The evaluation team found out-of-date RAs in DHMT offices and a number of RAs did not have current annual amendments. ZPCT II’s Key Obstacles in Program Management Planning and M&E systems. The PEPFAR emphasis on achieving target numbers affects the management of ZPCT II staffing. Testing and treatment have driven most ZPCT II’s systems for planning and reporting, data collection, analysis, and supervision. M&E systems focus almost exclusively on reporting on targets while the program management systems that focus on planning, tracking, and reporting on activities such as training, mentoring, supervision, and procurement are less developed. The limited information for management places inadequate attention on other project activities such as building capacity of PHOs and DHOs, improving sustainability and integrating gender and other cross cutting issues into project interventions. Approaches and Lessons Learned Preferred approaches to support. All MOH staff consistently discussed the value of ZPCT II technical assistance and other support. However, they discussed preferred approaches used by other USAID supported projects such as Supply Chain Management Systems (SCMS) and ZISSP. These projects use models that are more fully embedded in the MOH structure and collaborative approaches that more fully respected and aligned with the MOH organizational structure, designated functions, responsibilities, and lines of communications and reporting. Obstacles Requiring Policy Changes or Dialogue GRZ allowances. Policy dialogue on the issue of GRZ allowances should be given high priority due to its significant effect on ZPCT II budgets, as well as on PHO and DHO budgets. This dialogue should 30 prioritize the use of financial resources and the debts being incurred “on behalf” of ZPCT II and other USG funded projects. Increased resources in the health sector. USG dialogue with the GRZ is needed on increasing the resources for human and financial systems in order to advance the capacity of the health sector to provide services and to lead initiatives to strengthen approaches that address the epidemic’s drivers. Alignment with the Partnership Framework. Significant adjustments could be considered for USG resources allocations and dialogue with ZPCT II to align its resources more closely with the USG Zambia Partnership Framework’s first commitment: “Accelerate and intensify prevention in order to reduce the annual rate of new HIV infections.” TASK VI. FACTORS AFFECTING SUSTAINABILITY ZPCT II’s strategic and technical approach states that “sustainability was built in from the start.” All assistance will be provided in collaboration with the MOH within its existing systems and structures and, finally, “a new emphasis will be put on increasing the Ministry’s capacity to manage and maintain improved HIV/AIDS services.” Contributions to Comprehensive HIV/AIDS Programming Facility level staff interviewed during this evaluation generally felt that ZPCT II has made significant contributions in developing a comprehensive MOH HIV/AIDS program in the supported provinces. Facility staff mentioned that positive enhancements for sustainability included ZPCT II support activities in the following areas: facility renovations; staff trainings on ART, PMTCT, CT, M&E, lab, and pharmacy services; the provision of supplies, equipment, furniture; and supporting data entry clerks and volunteers. Respondents also gave ZPCT II credit for mentoring on strategies for closing commonly known service gaps, such as linkages to improve continuity of care and health worker shortages. Such strategies included: the escorting of HIV+ clients from PMTCT and CT sessions to register for pre-ART/ART care; the use of SMS messages and adherence support workers to contact those ART patients who do not pick up their ARV drugs; and the use of lay community counseling volunteers both at facilities and in communities. ZPCT II Efforts to Strengthen GRZ Management, Finance, and Technical Capacity ZPCT II provided technical assistance via its partnership with Cardno, supporting provincial staff in conducting a baseline assessment8 of district staff covering four components: finance, governance, human resources for health (HRH) and planning. This capacity building strategy included cascade training/workshops in each of the four areas with mentorships launched in late 2011. The mentorship component supported PHO staff conducting structured quarterly mentoring visits to DHO staff, beginning with finance and HRH ZPCT II reported that all 47 of the targeted PMOs and DMOs were provided with institutional capacity￾building. PHO respondents were generally aware of these capacity building activities and confirmed that workshops and mentorship visits have taken place. Generally PHO and DHO staff noted changes in how funds are allocated and tracked, and improved HRH systems. Annex P presents the ZPCT II “Capacity Building Training Log 2011.” The evaluation team reviewed the ZPCT II approach to capacity building for management strengthening and found that the assessment, the approach, the logical progression, and the tracking of training were sound. Cardno’s role was defined as “increasing the capacity of PHOs and DHMT’s to manage ZPCT II program activities at the project’s close.” It could be more beneficial to put increased emphasis on training and mentoring PHO and DHO staff to manage HIV activities during the project as partners to ZPCT II, and to sustain MOH HIV/AIDS activities at the project’s close. Since the MOH and ZISSP also have management assessment and strengthening initiatives, the closer the working relationship between their approach and that of ZPCT II the better. Graduation system. ZPCT II’s graduation system identifies higher-performing facilities for reduced technical assistance as they move toward sustainability. Strengthening the DHO capacity to manage and support these graduated facilities is not an explicit part of the process and is not a graduation criterion. 31 Since Districts as a whole are not graduated, staff turnover is high, and staff trained by ZPCT II are sometimes transferred from graduated to non-graduated facilities, graduated facilities are sometimes found to have regressed during post-graduation performance assessments. Operations research is being conducted on this complex issue. If ZPCT II works more closely with the PHOs and DHOs, they could jointly address the potential for a district wide approach to graduation. Comparison of graduated and non-graduated facilities. The evaluation team did not note significant difference between graduated and non-graduated sites. ZPCT II staff at one provincial office stated that because graduated districts continue to experience staff turnover and add new facilities, there might not be a great difference between graduated and non-graduated sites. Data use. ZPCT II has invested significant efforts to increase the availability and timeliness of data especially for reporting data on PEPFAR service delivery and SmartCare. ZPCT II staff, including data entry clerks, help to collect, analyze, and present data. ZPCT II’s data systems are stronger on reporting routine project requirements by PEPFAR. In some provinces and districts ZPCT II present data to DHOs and PHOs that they find valuable. However, to build additional MOH capacity and to sustain data use, District Information Officers and other staff pointed to the need for additional sharing by ZPCT II of raw data and increased district team training, to further prepare them in the use of data/data analysis, with regular joint quarterly meetings that include the use of data for decision making. ZPCT II’s investments in separate staff, vehicles, computers, etc., for the current data system will need to be reassessed to assure sustainability and integration within the MOH. Joint planning for ownership. Both PHOs and DHOs reported formal annual and other meetings to coordinate with ZPCT II in planning of activities. Generally ZPCT II participates when their supported provinces and districts plan activities. However, respondents reported that ZPCT II often prepares plans that are not in synch with province and district plans. This is partly due to different timing of ZPCT II’s annual plans and reporting. It was reported that other USG contractors had adjusted their timelines to match those of the MOH. The extent of joint planning varied across provinces and districts and generally staff stated that it could be improved to be more productive. As ZPCT II moves toward sustaining the advances made; a more collaborative, joint approach would support the MOH to sustain quality planning. Empowering PHOs and DHOs is also vital to the ownership intended. Facility-level sustainability. Respondents expressed the importance of the routine ZPCT II material support in maintaining availability of services: allowances, support for transport of samples and volunteers, meetings, provision of furniture, equipment, materials and supplies and transportation for monthly data collection. Such material needs were cited as challenges before ZPCT II support began. While this support is valuable and appreciated, long-term sustainability is not assured. This is an issue that affects all projects and would benefit from additional dialogue and joint MOH –ZCT II planning to move toward sustainability. Procurement done with sustainability in mind would include decisions on the procurement of major diagnostic and treatment equipment that is most appropriate for MOH conditions with plans for preventive maintenance and rapid repairs. ZPCT II’s planning in close coordination with the MOH, can improve the sustainability of facility equipment and operations. Quarterly meetings and joint technical assistance. RAs for both graduated and non-graduated sites generally require ZPCT II to perform routine quarterly meetings and ongoing ZPCT II mentoring and exchanges during quarterly visits to facilities with the PHO and DHO. During team visits, respondents reported that quarterly meetings did not take place at three of the six PHOs and two of the DHOs; four DHOs confirmed joint quarterly meetings. These joint quarterly meetings are critical to sustainability. The evaluation team recognizes the reality that PHOs and DHOs often have multiple and competing priorities making joint efforts difficult, however, joint technical assistance visits are valuable to MOH partners to enhance supervision skills and reinforce standards of care. Joint technical assistance visits can also make for the best use of project vehicles and other resources that need additional emphasis. GRZ ownership and commitment. At the national level, there are indications of increased commitment by the GRZ to invest in health. The GRZ increased its health sector budget by 45 percent between 2011 and 2012 (ZMK 2,579.9 billion in 2012).9 According to Director General of the NAC, the GRZ has increased its budget for ARVs from USD $5 million in 2011 to USD $12 million in 2012. This 32 is a significant increase, however, it represents 15 percent of the total ARV budget. The health budget constitutes 8 percent of the national budget, still below the 15 percent commitment set in the 2001 Abuja Declaration. There are a number of initiatives that indicate a commitment to increasing human resources to help address needs in HIV/AIDS service delivery. New cadres include: a nurse prescriber cadre, a GRZ community health worker cadre, a PHO-level HIV/AIDS medical specialist cadre, and psychosocial counselors. Some health facilities have employed previously ZPCT II-supported volunteers as data entry clerks. Staff members have also employed creative solutions in splitting the ZPCT II allowances for two volunteers amongst several volunteers, as well as sharing staff transportation allowances. TASK VII. BENEFICIARY RELATIONS Under Evaluation Task VII, the team was required to evaluate the project’s success at delivering quality HIV/AIDS services to beneficiaries. Under this task the team examined whether services coverage (location and range of services offered) represent an efficient use of project resources. The team also examined how program beneficiaries view the following aspects of services: hours of operation, professionalism of staff, quality of care, confidentiality, ease of accessing multiple services, and the availability of products. As part of this examination, the team compared the perspectives of project beneficiaries and beneficiaries who are not currently using ZPCT II supported services to determine what factors influence the decision to use or not use services. The team also evaluated the success of community volunteers in helping the project achieve its objectives. To answer these evaluation questions, the team conducted 78 client exit interviews at ZPCT II supported facilities. The team also conducted 44 FGD with PLWHA. This includes 25 FGDs with beneficiaries of ZPCT II supported sites and 19 FGDs with beneficiaries of facilities not supported by ZPCT II in order to compare perspectives. To prevent biased responses, FGDs took place in the community, not at the health facilities being discussed. Please see Table 6 that summarizes the types of participants in each FGD.10 Table 6. Types of Focus Groups Gender Beneficiary Type Graduated Female Male Yes No Urban Rural Yes No 20 24 25 19 26 18 24 20 Total: 44 Total: 44 Total: 44 Total: 44 Please see Annex J for a summary of responses from the client exit interviews. Please see Annex K for notes from FGDs with ZPCT II beneficiaries on service coverage. Please see Annex L for examples of comments from ZPCT II beneficiaries during FGDs. Please see Annex I for a table that summarizes the opinions of ZPCT II beneficiaries on which services should be improved at the community level. Client perspectives on the quality of HIV/AIDS services. Overall, 76 percent of the 78 clients who participated in exit interviews at facilities rated services they received on the day of the survey as “good” or “excellent,” with the remaining 24 percent rating services as “fair.” Clients were also asked whether specific issues had been of concern during their visit on the day of the interview and what areas needed improvement. The two areas of major concern were cleanliness (33 percent) and availability of medicine (21 percent). Ease of accessing multiple services and products. During the FGDs, 15 of the 44 groups reached a consensus that they were not satisfied with their ability to access multiple services; 10 of the 44 groups did not reach a consensus and had mixed opinions. During FGDs, when asked about their ability to access multiple services, respondents frequently mentioned challenges encountered when trying to accessing multiple services and products, include crowding, insufficient staff, long wait times and the need for additional HIV/AIDS facilities (particularly citing limited access to ART-related services). The evaluation team noted that groups of respondents from urban areas mentioned longer wait times. In 33 addition, ZPCT II supported facilities tended to attract a larger number of clients; this preference likely resulted in crowding and longer periods of waiting. Hours of operation, professionalism of staff, quality of care, and confidentiality. During FGDs, very few groups responded when asked whether they were satisfied with the hours of operation at their facility. One group stated that the hours of operation were satisfactory. When asked for their opinions on the professionalism of staff at their respective facilities, 11 of the 44 groups reached a consensus that they were satisfied; eight of the 44 group reached a consensus that they were not satisfied; and seven groups did not reach a consensus, having mixed opinions. When respondents were asked about their preference regarding service provider gender (receiving health services from a male or female staff member) at their respective facilities, the majority of groups stated that male providers were “more caring.” When asked for their opinions on the quality of care at their respective facilities, eight of the 44 groups reached a consensus that they were satisfied; five of the 44 group reached a consensus that they were not satisfied; and 8 groups did not reach a consensus, having mixed opinions. When asked about their opinions on how confidentiality was handled at their respective facilities, 14 of the 44 groups reached a consensus that they were satisfied; nine of the 44 groups reached a consensus that they were not satisfied; and four groups did not reach a consensus, having mixed opinions. Comparison of beneficiaries to potential beneficiaries who do not use services. FGDs indicated that ZPCT beneficiary groups were less satisfied than non-beneficiary groups in the areas of confidentiality, quality of care and professionalism of staff. Participants were asked, “What should be improved about HIV/AIDS related service delivery in health facilities?” Beneficiary FGDs were more likely to express dissatisfaction with the current level of confidentiality. Beneficiaries cited reasons for this including limited space (“ART patients and those coming for general health care all share the same room and hence there was a complaint of lack of confidentiality”), and the lack of staff discretion (“By and large, they said that there was a problem with the level of confidentiality”). Participants stated that HCWs are not able to uphold patient confidentiality policies (“Everyone knows that these people all have AIDS, it is just along the path and everyone pass through going wherever they are going”). Beneficiary FGDs were four times as likely to mention quality of care as an area of dissatisfaction and are approximately half as likely to be dissatisfied with the professionalism of staff. On the positive side, beneficiary FGDs were twice as likely to mention community health volunteers as an area of satisfaction. As mentioned above, discussions by the team with the Research Assistants identified that perhaps because the facilities and services in ZPCT II supported services had much higher volumes, the issues of space, confidentiality and staff treatment were negatively affected. Additional exploration of these finding and other findings could be done to further explore this issue. Factors influencing decision to use or not to use services. Focus group participants were also asked to rate the factors that influenced their choice of where to receive services. In 35 of the 44 FGDs with ZPCT II beneficiaries and non-ZPCT II beneficiaries, respondents stated that the proximity of the facility is the main factor that affects their decision of whether to use or not use HIV/AIDS services. The second most commonly cited factor (20 out of 44 FGDS) is whether a facility offers ART/CD4 analysis. Figure 2. What factors influence your choice to use or not to use services? (N= 44 beneficiary and non-beneficiary FGDs) 34 Success of community volunteers contributing to ZPCT II objectives. During FGDs with beneficiaries, 14 of the 44 groups reached a consensus that they were satisfied and greatly appreciated the work of the community health volunteers. During KIIs with staff at PHOs, DHOs and ZPCT II supported facilities, the majority of respondents stated they greatly appreciated the work of the community health volunteers, including PMTCT and CT counselors and support volunteers. TASK VIII. LINKAGES, COORDINATION AND PARTNERSHIPS ZPCT II Effectiveness at Establishing Linkages FHI360, as the prime, is responsible for managing the partnership with the MOH at the national and decentralized levels. Provincial MOH offices oversee the partnership at provincial and district levels, and the FHI 360 headquarters is responsible for oversight of all levels of program functioning. ZPCT II sub￾partner CARE, manages the community component and is responsible for all community based activities. These activities include community education and motivation, fixed obligation grants to CBOs and the development and operation of district referral systems. Information in Annex R provides an outline of the roles and responsibilities of ZPCT II partners. ZPCT II supports linkages between health facilities and communities via community HCWs (counselors and adherence support workers) and sub-grants to Civil Society Organizations (CBOs) and FBOs – HIV/AIDS information, education, communication campaigns, counseling, and testing. ZPCT II and MOH wide Linkages and Coordination with GRZ The establishment and management of formal links by ZPCT II to strengthen coordination with the GRZ and the MOH were found to be weak across provinces. This is in contrast to the view held by ZPCT II but PHO and DHO respondents stated the need for stronger coordination with ZPCT II through joint planning and coordination of activities. Recipient Agreements and MOUs represent the formal linkages between ZPCT II and local partners. These MOUs serve as a reference point for discussing respective roles, responsibilities, and progress in achieving a common agenda. To date, ZPCT has signed recipient agreements (RAs) with PHOs and DHOs in the six provinces, and with hospital and other health care facilities. MOUs are also with private sector partners. Typically, recipient agreements did not link ZPCT II and stakeholder coordination. The team did not find minutes of meetings that documented discussions of the content of the RAs before MOH partners signed the agreement, nor did they find evidence of subsequent meetings to periodically evaluate progress in the activities and support stated in the MOUs. Many PHO and DHO staff felt the RAs did not represent partnership and collaboration, rather they were imposed upon them. A DMO Officer in one province stated that his office could not amend or refuse agreements signed between the project in Lusaka and the MOH, “ZPCT II brought us the agreement as a fait accompli and expected us to sign before our office had an opportunity to read it.” In another province, DHMT respondents expressed ignorance about the content of their RA and were unaware of their right to negotiate with ZPCT II on content. ZPCT II held a very different view of how recipient agreements were developed, 35 negotiated, signed, and implemented. However, interviewees were clear on their stated opinions about the process. Best Practices for PPPs Management. ZPCT II’s success in managing private sector partnerships was better defined and documented than those of the MOH. These performance measurements were done through a clearly articulated set of roles and responsibilities, processes for achieving mutually agreed upon milestones, and availability of tools to assess progress. This includes a process for pre-assessment to determine a facility’s eligibility for the program; a meeting to discuss mutual roles and responsibilities and potential project support; the MOU; assessment tools for accreditation; and clearly defined milestones for satisfying requirements of the Medical Council of Zambia. To date, ZPCT II has signed amendments to the contracts initiated under ZPCT II, engaging 18 private sector partners of the 30 expected, over the LOP. ZPCT II reported they had signed new agreements, however, Table 2 displays information on HIV/AIDS services in the 18 private facilities, yet there are no HIV/AIDS activities reported for additional partners coming on board. Over 50 percent of the PPPs are found in Copperbelt Province, where the population is large and more urbanized and, thus more attractive private companies. The best practices for private partners could be studied for lessons learned for engaging the MOH. ZPCT II has only begun to engage parastatal and large private corporations. Previously, CHAMP/COMET (a former ZPCT II partner) worked with mining companies in Northwestern Province to extend mobile services to communities living in close proximity to Lumwana Mines. Unfortunately, a gap in service delivery was created when CHAMP’s relationship with ZPCT II ended, leaving ART clients without a secure supply source. The Solwezi Urban Health Clinic reported that they were in discussion with the Lumwana Mines, ensuring ART services continue in communities in mining areas. The gap in services and the structuring of a rapid response was a missed opportunity and a lesson learned for future programming with the private sector partners. The HIV/AIDS services offered by private sector partners expanded access to prevention and treatment services for clients who could afford, and are willing to pay, for health care. Payment can be made from their own resource pools, or through coverage or subsidies from employers. ZPCT II’s support works closely with providers in these facilities to ensure that appropriate and nationally approved drugs are available and are prescribed according to national and international standards. Lessons learned about reasons for clients shifting from private to public services related to insurance coverage should be further studied for ZPCT II’s work with private partners. Private facilities, however, have constraints to ensuring access and continuity of care. While ARVs are free if private facilities are accredited, not all of these facilities meet accreditation requirements. If accredited they could access drugs from the national supply chain that provides consultations and treatments for opportunistic infections. Patients receiving care in a private facility face the prospect of a cap by their insurance provider for treatment and may have to drop out and/or seek care in a public facility until their insurance plan will reinstitute payments for services. Community Component. ZPCT II sub-partner CARE oversees the management of the community component of linkages and partnerships. They work with partners outside of the health sector and at the community level. They work with the PATFs and DATFs in districts where DATFs are well run and coordinate HIV/AIDS-related work with other public sector ministries, CSOs, and the private sector. However, the number of staff supporting community linkages is small. This component would benefit from additional staffing and resources, incorporating more prevention to balance with current predominate focus on mobilizing for clinical services. Coordination with the PATF/DATF. ZPCT II belongs to the Provincial Treatment & Care Technical Working Group, although this working group is no longer functioning. Some PATFs and DATFs have functioned consistently well, while others have experienced difficulties due to gaps in leadership and support from the NACA. Therefore, ZPCT II’s work with the PATFs and DATFs has been difficult for some provincial offices. With the resurgence of PATFs and DATFs with increased support from NAC, they can provide a multi-sectoral networking function and an opportunity for ZPCT II to expand outreach and community based activities. CBOs and FBOs receiving ZPCT II grants do not always 36 register their activities with the DATF, which has the responsibility of monitoring HIV/AIDS-related activities. The process of registration of activities through the DATF is important because this information provides a holistic picture of HIV/AIDS and related district activities. Referral Network. ZPCT II supports the development of district referral networks to link service providers into a referral system, and to provide an easily accessible source of assistance for clients in need of non-clinical care and support. The referral system is coordinated by the DATF in districts where the DATF is strong and, according to ZPCT II, where DATFs are weak the district health office performs the coordinating function. It is clear that the success of the network is tied to a coordinating mechanism that encompasses a broad range of social services. Table 7 provides summary information on how referral networks are functioning in 44 ZPCT II supported districts. Table 7. Status of ZPCT II-supported Referral Networks (compiled by the evaluation team) District Referral Networks 44 Districts 6 Provinces (Central, Copperbelt, Luapula, Muchinga/Northern, and North-Western) Functional 37 All supported-districts apart from Chinsali, Chilubi, Milenge, Lufwanyama, Chavuma, Ikelenge, and Mafinga Weak 6 Chinsali – NP, Chilubi – LP, Milenge – LP, Lufwanyama-CB Chavuma – NWP, Ikelenge- NWP Inactive 1 Mafinga - Muchinga/NP Quality assurance system: Technical quality. At ZPCT II-supported facilities both MOH and ZPCT II systems were designed to improve quality. ZPCT II has a Quality Assurance/Quality Improvement (QA/QI) tool that is based on the Health Professions Council of Zambia ART accreditation guidelines, as well as on MOH national guidelines for PMTCT and other HIV/AIDS technical areas. The QA/QI tool is administered quarterly, covering the technical areas of M&E, ART/Clinical Care, CT/PMTCT, laboratory, and pharmacy services. ZPCT II implements a quality assurance system through which ZPCT II assesses supported districts with selected facilities that reach a certain level of technical performance. If these facilities pass the criteria, the district is considered a graduated district. Graduated districts receive reduced ZPCT II technical assistance. QA tools consist of checklists in the technical areas of CT/PMTCT, ART/Clinical Care, Laboratory, and Pharmacy. These tools are administered monthly for six months, and the facility must score above 80 percent in these technical areas to graduate. ZPCT II coordinates a graduation ceremony and develops a post-graduation plan to harmonize services with the MOH Performance Assessment Program. Although ZPCT II defines districts as the unit that graduates, the QA approach works primarily with facilities in the above technical areas; strengthening of DHOs is not a direct part of the quality improvement process. A process to more directly involve and strengthen the DHOs to oversee and provide supportive supervision to maintain the quality of facility services after graduation is needed since staff report declines in the indicators after graduation. ZPCT II strengthening of DHOs would allow them to improve the quality of all facilities in a district, thus expanding the QA approach. TASK IX. PERFORMANCE MONITORING Status of ZPCT II’s Program Monitoring System. The ZPCT II M&E system was implemented at national, provincial, district, facility, and community levels. The project developed a PMP, a M&E work plan, and a M&E procedures manual. The PMP summarized key project indicators and targets to be monitored during the LOP. The M&E system also comprised the central database that store data from provinces. ZPCT II put in place a Data Quality Audit (DQA) system and sometimes the project teams participate in the MOH DQAs done quarterly. ZPCT II has also developed and established a QA/QI system with guidelines. The implementation of the M&E system at various levels is as follows: 37 National level: The ZPCT II national level has seven officers managing the M&E system. These are: the Senior Strategic Information Advisor who heads the M&E unit who is supported by two QA/QI Officers, one Database Manager and one Database Officer and two M&E Officers. The Data Manager merges data collected from the project provinces into a central database. The SI unit at the ZPCT II Lusaka office also conducts quarterly DQAs and QA/QI technical visits to selected health facilities. Provincial level: The provincial office SI unit is comprised of three M&E Officer and one QA/QI officer who collect data from the supported health facilities in the five provinces. District level: There are no M&E district structures for ZPCT II. In some districts the project works with the DHOs to collect data from health facilities while in others data is directly collected by the facility data entry clerk(s), bypassing the DHOs. Health facility level: The HCWs update registers regularly at health facilities using MOH registers, or log books, as they attend to clients. For example, for CT they use the daily CT log, while for ART they use both the ART paper register and SmartCare. New clients are logged onto the register and are given IDs/dates of appointments and regimens. The HCWs submit data to data entry clerks on a daily basis at the health facility level. The data entry clerk collects all HIV/AIDS facility-based data and sends it to the ZPCT II provincial office and the MOH DHIO where data are reviewed to ensure that data are complete, valid and accurate. Once the data have been reconciled at the district level, these are entered into a computer database at the ZPCT II provincial office. The resulting database is then electronically transmitted to the ZPCT II Lusaka office for further review, validation, and aggregation. ZPCT II supported sites also use registers for community activities such as a GBV screening tool and referral forms, for chronic diseases, and for TB. Community based organizations send summarized, paper based data reports directly to the ZPCT II provincial office. The data are aggregated at the province and transmitted electronically to the ZPCT II central office in Lusaka where it undergoes the same data cleaning and consolidation process conducted for facility-based data. Reports for USAID are generated quarterly and sent out directly from the Lusaka office. PEPFAR reports are generated semi-annually and annually. Quality, use and timeliness of the PMP Plan. The PMP is a summarized document that outlined annual project indicator targets to be reached by ZPCT II during the LOP. The evaluation team noted that ZPCT II’s PMP is not used to provide high quality data, as it lacked essential elements of a useful PMP, such as plans for routine data collection (including indicator definitions), timelines, plans for data analysis, reporting, review and use, feedback and budgets in line with USAID guidance on how to develop a useful PMP cited in the CDIE’s Tips series no. 7 of 1996. To address this need, ZPCT II has developed an M&E work plan, which includes elements such as, performance indicators, data sources & collection methods, data quality management systems and performance targets. These systems are reviewed and revised routinely as part of the PMP. Though this was seen as a duplication of effort (e.g., when a modification is made, the revision would need to be made in a number of different documents). The evaluation team did not find definitive proof that the PMP was approved; ZPCT II told the team that approval was still outstanding. There was no email provided that confirmed approval. Evidence of the Use of the PMP and M&E Plan. ZPCT II’s M&E work plan clearly outlines data utilization and feedback mechanisms at various levels. It proposes/states that the project would develop annual utilization plans. However, the evaluation team did not find evidence of these plans. To increase utilization, data entry clerks have been trained to produce trend data in the form of graphs, which can be used by HCWs and the DMOs. The evaluation team also found evidence that ZPCT II staff and data entry clerks provided some feedback on the data, and its possible use to ZPCT II supported health facilities. The level of data used at the facility level was reported to vary considerably. At the high data utilization end, some data entry clerks reported that they work closely with facility staff and were knowledgeable about ART and care and reporting. These data entry clerks reported providing data, responding to ad hoc requests from providers regarding individual patients and drug supplies. They also participate monthly in facility-level reviews, for which they prepare data for presentations in aggregate forms. At one site, the 38 data entry clerk reportedly had a very positive relationship with clients, as the patient goes to the data entry clerk for his/her medical record. At the lower end of data utilization, DHO staff reported that data entry clerks were not trained to analyze data, and do not actively present data for use in facilities. A data entry clerk reported that his role was to provide data for ZPCT II reporting, the site conducted no meetings in which data were used and that he did not received hoc requests for queries into the SmartCare database, (not even for individual client information). Table 8. ZPCT II Methods of data collection per PEPFAR program area A data entry clerk working at a private clinic noted that the site did not use the same standards for declaring a patient as “lost to follow-up” as the public facilities, since ART patients regularly received care at both private and public clinics depending on their intermittent availability of insurance funds (e.g., funds are available at the beginning of the calendar year and are depleted over subsequent months). At the district level, data use also appeared to be inconsistent across DHOs. Both ZPCT II and some DMOs interviewed indicated that such data are not regularly shared in quarterly review meetings. District Health Information Officers at more than one district reported not being included in ZPCT II supported review meetings. She reported receiving ZPCT II courtesy calls and hard copies of ZPCT II data in the past. At the provincial level, there was evidence of feedback and discussion between PHO and ZPCT II staff about data use. Provincial level results were included as part of the agenda for most of their meetings. There was strong evidence that service delivery/clinical data were shared internally within ZPCT II at the national level, by top management and technical teams to inform decision-making on project PEPFAR Program Area Data Collection System Adult Care and Support services  Bullets - applies to all ART program areas to the left  Services are captured on SmartCare and SmartCare registers/data collection tools at the facility level.  Patient data are transcribed onto a Monthly ART or Care and Support Summation Sheets and sent to the ZPCT II provincial M&E officer.  This summation sheet output is then entered into the ZPCT II MS Access database and this database is in turn sent electronically to the ZPCT II provincial office. Pediatric Care and Support services Adult HIV/AIDS treatment services Pediatric HIV/AIDS treatment services Male Circumcision  MC data are collected on a client form similar to the SmartCare form: then entered into a basic MS Access database at the facility by a data entry clerk.  This database is sent to the DHIO and ZPCT II provincial office on a monthly basis for further aggregation and reporting. Counseling and Testing Services  CT data are collected using an Integrated CT-PMTCT Register at the facility level or SmartCare for CT/PMTCT where this system is installed.  Health facility staff aggregate and summarize this information onto a CT/PMTCT monthly summary form that is entered into the ZPCT II database and sent to their respective DHIO and ZPCT II provincial office every month.  At the ZPCT II provincial office, the data reviewed and merged with the provincial level ZPCT II database, which is sent to the ZPCT II Lusaka office. PMTCT Services TB/HIV Care  Data are integrated within the Care and Support systems as well as the CT system.  Data on patients receiving TB treatment while on HIV is collected together with all other care and support indicators. On the other hand, data on the linkage between TB and HIV in terms of TB infected clients counseled is collected as part of the CT statistics and system. Laboratory Services  Data for the laboratory are collected from the Laboratory Tests Register and compiled on the Laboratory Monthly Summation Forms.  These forms are sent to the ZPCT II provincial office every month. At the provincial level, these data will be entered into the ZPCT II database and sent to Lusaka office. Gender  Program reports and information systems. Community-based Services Sub-system  Data collected on standard recording and reporting protocols from community service organizations. 39 performance and assess the quality of care and services. Such data sharing informs ZPCT II’s focus of the quarterly DQAs and QA/QI supervisory visits that ZPCT II conducts. The levels of the use of data for decision-making varies considerably across supported sites, district and provincial levels and could be improved. A clear signal that “the value of data is in its sharing and use,” and active monitoring of that sharing between the ZPCT II, MOH and CBOs would do much. There is great potential to accelerate access to data by all parties by the posting of these data on an easily accessible, user friendly web page on ZPCT II/ FHI 360’s web site, assisting the posting of data on a page of the MOH’s website. Appropriateness of methods for data collection. ZPCT II has a robust system of data collection at all levels with M&E manuals outlining procedures of data collection, indicator calculations, data collection tools coupled with trainings at various levels. SmartCare collects clinical data at health facility level, while the provincial level has a Microsoft Access database that collects data from the health facilities in the districts. Data is then sent to the national level Access database which merges provincial data into one database. ZPCT II maintains provincial and national level databases, from which the project can generate pre￾programmed automated reports for PEPFAR reporting. In contrast, the tracking, analysis and reporting of data for project management are not yet automated (such as in the areas of training activities and other areas of technical assistance, as well as project functions such as procurement). ZPCT II also reported that ART retention, a key indicator of ART quality, is not automated and must be calculated by hand. Some of the current restrictions in SmartCare do not allow these data to be generated automatically. This is a significant missed opportunity, as one of the main advantages of implementing an electronic patient information system is to monitor how the client is faring and the care provided over time. Further, CARE, Cardno, and SI supported activities are not captured in the current database. This limits the usefulness of the data-base as a comprehensive management tool. Since only the ZPCT II Data Manager is able to generate and manipulate data, the ability of ZPCT II managers to produce and use data for program management is limited. Because many data sets are not automated for easy access and reporting, managers have to request data reports and cannot produce them in real time for themselves. The evaluation recommendations address the need to increase the use of data for management decisions by increasing access to data by managers. Other factors in health environment. The GRZ is moving towards the integration of services. This means that ZPCT II supported activities can potentially become an entry point for non-communicable diseases-related services, which have recently increased significantly. It will also greatly affect how donor money is spent and resources allocated to project priorities. TASK X. PARTNERSHIPS AND MANAGEMENT SYSTEMS Description of CO-Partnership Arrangements Internal Arrangements. There are various arrangements with sub-partners and grantees under the ZPCT II umbrella. Some sub-partners work closely together with the prime on a day-to-day basis, and have seconded staff to ZPCT II (prime). These personnel share office space and logistical arrangements. Sub-partners in this category are CARE, MSH, Cardno, and SI. This is also replicated at the provincial level where FHI 360 and CARE staff share offices. Cardno and SI do not have officers operating at decentralized levels, but provide guidance and technical assistance support from headquarters in Lusaka. Other ZPCT II partners (such as Churches Health Medical Association, the Salvation Army, and UTH) function as sub-grantees that are contracted to carry out specific activities or functions. They are less integrated into the day-to-day operations of ZPCT II, but focus on their particular area of operations. CHAZ has project positions but are not at ZPCT II Head Quarters (HQ) and are not shown in ZPCT II’s organogram. CHAZ defined their role as receiving and distributing goods provided by ZPCT II. The linkages between CHAZ’s large FBO network of facilities and CBOs, and the FBO capacity building activities of CARE were not discussed by CHAZ or CARE. This represents a missed opportunity for links across ZPCT II partners. Discussions with the Director of Programs on relationships within the internal partnership stated that partners consider FHI 360 as their “client.” This could be an inhibiting 40 factor to strong partnership functioning. Ideally, the ZPCT II partnership should maximize the strength of each partner for a collaboration that is stronger than its individual components and partners. The partnership is managed through an annual meeting of partner directors who discuss common issues. Each sub-partner also manages the officer assigned to ZPCT II. For example, CARE retains a level of supervisory and reporting authority over officers placed in ZPCT II. Therefore, officers report to the Director of Programs ZPCT II as well as the directors in their respective organizations. Performance of the Prime, Sub-partners, and Sub-grantees The effectiveness and performance of ZPCT II as a comprehensive project was presented under Task I. To provide additional information on performance of all tasks, this management section presents the performance of the prime, sub-partners and grantees. The evaluation team reviewed ZPCT II reports, databases and other sources of information. ZPCT II quarterly reports do not routinely or consistently provide information on partner performance contributing to ZPCT II objectives. FHI 360 as prime was contacted for partner specific information. Annex R presents information on partners from ZPCT II sources. In addition, the evaluation team reviewed the ZPCT II’s training database and noted that it does not include all partners’ training. The Program Unit Director stated that the first table on ZPCT II quarterly reports presents all performance/achievements but the team’s review found that it does not disaggregate contributions/performance by partner. Contributing and inhibiting factors. FHI 360 played a strong role in defining roles; managing project partnerships; and defining the approaches, strategies and inputs of the contract partners (MSH, CARE, Cardno, SI, TSA, and local partners, CHAZ and Kara). The ZPCT II Partners’ chart in Annex R documents the prime, international, and local partners, as well as their roles, responsibilities, and reporting structures. This chart demonstrates the relationships between the prime and its partners; during discussions with partner respondents, only the implementation of SI gender strategies were cited to be implemented via beneficial relationships or synergies among partners. The ZPCT II Organogram in Annex Q presents the management structure of the project. Within ZPCT II the Technical and Program units have distinct functions. The placement of the Technical Advisors of the partners in community mobilization, gender, and capacity building in the Program Unit limits the technical input into the comprehensive planning and technical approaches of the ZPCT II Technical Unit. The Project Director states that there are many opportunities for cross communication. However, the evaluation team found that there are missed opportunities for the partners’ technical advisors to provide more valuable input to ZPCT II technical approaches. Challenges in the partnership. The evaluation team’s review determined that the Technical Unit’s planning, training, mentoring and support to HIV/AIDS services could greatly benefit from integration with the community mobilization, gender, and capacity building technical skills of their partners. An additional factor that is seen as inhibiting the effective functioning of ZPCT II’s partnerships, is the tendency of partners to have parallel systems for planning activities, information systems tracking, and reporting. The evaluation team suggests that a comprehensive training plan and information system could benefit ZPCT II partners to building synergies across all technical assistance such as mentoring and training activities. For example, the ZPCT II training data-base tracks primarily FHI 360 clinical training, while the Cardno partner PHO and DHO management training is separately tracked. It was noted that SI also plans and tracks gender training separately. Linking all partners’ planning, training, and tracking systems would contribute to more effective project inputs to support provinces and districts. It could also provide PHOs and DHOs a comprehensive view of the partnership that supports their programs. Another inhibiting factor is that internal partners are funded to carry out activities, but they do not receive budgets for logistics (e.g., their own vehicles, administrative overhead, etc.). They are dependent on ZPCT II decision-making for their budget levels and logistical support. Although ZPCT II partners develop quarterly travel plans, to rationalize their use and explain the sharing of transportation, this process has not resulted in cohesive and internally connected results. The evaluation team’s review of provincial-level field visit reports prepared by partners’ Technical Officers shows a fragmented and disconnected approach to mentoring and technical assistance in the facilities. Each individual report is based on an officer’s observations about his/her HIV component of work for that visit and does not connect the trip to an integrated facility-level support plan of development for an integrated cluster of 41 improvements. In addition, technical assistance for clinical services generally drive decision making priorities for partners’ shares of logistics support. This limits the ability of partners to provide integrated technical support to facilities. Coordination is deliberate and strongest at the central level where formal mechanisms administered by the USG and GRZ facilitate coordination (i.e., through USG country program and chief of party meetings, and MOH and NAC technical committees). At the decentralized level, coordination is informal and dependent on the time, inclination, and operational constraints of various partners. Challenges, improvements and changes to improve partnership functioning. Discussions between the evaluation team and ZPCT II staff at the central and provincial levels assisted the team’s recognition of (at least initially) the benefits of collaboration amongst all partners. However, this collaboration will require a considerable reorientation of FHI 360’s conceptualization of partnership. Since FHI 360 is profoundly clinical and service delivery focused, they would have to reorient themselves in order to more fully share and interact with partners who have technical skills that would help them achieve the other four project objectives, in addition to the service delivery objective. Since the Technical Unit is essentially focused on clinical care, if ZPCT II incorporates more prevention into the second half of the project, the unification of these units’ functioning could improve the partnership’s internal functioning. The ZPCT II improvements in consolidating planning, information and reporting systems across all activities would benefit all partners and could include a wide sharing of project documents, agreements and other information that currently are predominantly within FHI 360’s or ZPCT II internal systems and controls. When the more agile, consolidated information systems are developed, more information on training, services, tools, curricula, recipient agreements and other project information can be widely shared and posted to a ZPCT II’s open access web site for ready access by internal partners and by the MOH, PHO, DHO, and other partners. This major improvement could greatly accelerate the strengthening of MOH ownership, especially by PHOs and DHOs, who currently have limited access as partners to ZPCT II. Inexpensive technology inputs could be used to assist PHOs and DHOs in their ability to download their RAs and refer to them as needed. It can also make available to all partners, ZPCT II’s plans and reports, training curricula, QA/QI tools and results and other information to provinces and districts. External Arrangements. The MOH and NAC interpret the partnership with ZPCT II as the project’s commitment and ability to provide support to the national HIV/AIDS response within the context of the three ones approach. FHI 360 has an MOU with the MOH articulating the role and responsibilities of ZPCT II in supporting the MOH’s HIV/AIDS interventions. ZPCT II participates in technical working groups and sector-wide approaches, which are mechanisms for sharing information and coordinating work. The MOH expressed concern that most of the USG projects were not following the principle of joint planning, but rather tended to develop their own plans and present them to MOH. ZPCT II was included in this group. However, the MOH also acknowledged that ZPCT II tended to be more transparent and collaborative at provincial and district levels (even though the evaluation team encountered varied opinions from PMOs on the level of collaboration across the five provinces). In addition to interactions with the GRZ, ZPCT II has relationships with other USG-supported partners at district and facility levels to coordinate the provision of support to services under USAID guidance. The coordination of joint efforts has not been formalized at these levels and sometimes operates on an ad hoc basis. In districts where the DATF and DACA are strong, coordination is facilitated through the DATF. ZPCT II has interpreted its partnership responsibilities within the GRZ rather narrowly (e.g., as working primarily with the MOH and NAC), and has consequently missed opportunities for having a greater impact in its mandated areas. For example, the evaluation team noted a major roadwork project in Copperbelt Province that would create a direct link to Northwestern Province, opening up previously isolated rural communities to national and international travelers. The new highway runs quite close to some ZPCT II-supported health facilities but there has been no discussion by ZPCT II staff on the potential impact of the new road on risk and vulnerability to HIV infection on local communities. Planning is necessary to prepare local health staff to address HIV/AIDS prevention and other health problems arising from an influx of single men establishing camps alongside the road. In addition, another opportunity is for ZPCT II to contact the MCDMCH about the possible move of PMTCT 42 responsibilities to this newly formed Ministry. ZPCT II should explore the opportunity to coordinate with the MCDMCH, although the role of the MCDMCH is not yet well defined by the Zambian government. CSO partners are situated between internal and external relationships with ZPCT II. The evaluation team has found that ZPCT II’s work to develop community partnerships and the integration of community workers into service delivery (via partner CARE) has contributed to successful partnerships with CSOs and an example of good practice. Community involvement promotes the ability of individuals, households, and community groups to change norms, values, and socio-cultural practices for prevention and treatment of HIV/AIDS. CARE is responsible for managing the partnership between ZPCT II and CSOs via FOGs provided to six organizations to mobilize communities to access HIV/AIDS prevention and treatment services. FOGs are given only to high performing and low risk CSOs that have the capacity to produce results, and can use their own resources to mobilize communities to achieve project targets and claim re-imbursement later. CARE does the first screening of CSOs for FOGs and final approval is given by USAID before a contract is given. FOGS are given on the basis of potential performance rather than need or geographical site. Most FOGs are in Copperbelt Province where the population is high and CSOs numerous. In addition to the FOGs, ZPCT II works with Neighborhood Health Committees and Health Advisory Committees to motivate communities to utilize prevention and treatment services in the local health facilities. TASK XI. USAID PROJECT MANAGEMENT Key Management Actions USAID has provided active management of the ZPCT II project. The Contracting Officer’s Representative (COR) generally interacts with ZPCT II staff on a frequent and regular basis, often weekly. There is an official management meeting between USAID and ZPCT II monthly. USAID has provided official guidance and feedback to ZPCT II on a number of its deliverables, including the annual portfolio review. The COR and others in USAID have monitored the timely submission of the contract deliverables. USAID has done data quality audits, stating that the quality of the data from ZPCT II is high. USAID provided the notes from USAID and ZPCT II meetings to the evaluation team. These documented discussions covered issues such as project indicators, PMP, and M&E Plan. There was conflicting information provided during interviews with staff from USAID, FHI 360, and Cardno. The evaluation team was not provided with documentation to show that the PMP was approved. In addition, the M&E Plan was tentatively approved, although it is not a contract deliverable for this project and further USAID guidance is being provided. ZPCT II has produced an M&E Procedures Manual. The evaluation team reviewed it and found to be well done and complete in terms of indicator definitions, coding, and tools. According to USAID guidelines, the PMP and the M&E Plan should be merged into one document. The COR stated that USAID was going to return to the use of Results Frameworks. USAID is also in the process of reviewing the ZPCT II performance monitoring plans, these include the PMP, the M&E Plan, and the M&E Procedures. The COR has involved M&E specialists at the mission and an M&E specialist at the USAID regional Mission in South Africa in the process. Annex S provides a full list of ZPCT II deliverables with specific USAID management actions. USAID Regulations and their Effect on Project Implementation Please see the section about Civil service human resource remuneration and benefits packages (allowances) under Evaluation TASK V. Obstacles Affecting Progress, for a detailed ex[planation of how USG restrictions on the payment of allowances for GRZ employees to participate in project activities has constrained implementation. Policy Issues USAID and ZPCT II should assess the use of nurse prescribers at MOH and private facilities, which is an accepted policy following the OR study. This study confirmed the effectiveness of nurse prescribers in 43 managing ART treatment regimes for HIV+ clients. More recently, the Health Professionals Council has promoted the realignment of responsibilities of cadres of staff to allow nurses who are specially trained to prescribe ART. Training for prescriber nurses is a one-year course that combines lectures and supervised practice at the University Teaching Hospital; ten were trained the first year. Another issue that should be examined is that data entry clerks placed as ZPCT II employees in MOH facilities frequently have higher salaries than MOH staff. These clerks have played a valuable role in data creation and use, but these positions may not be sustainable because of their higher salaries if and when these positions are transferred to the GRZ. The data entry clerk positions are new and do not have an officially established role within the MOH, although there is a position of Information Officer at the district and provincial levels. The Public Service Management Division in the Cabinet would need to create a position for these staff move to MOH staff. Currently, ZPCT II co–supervises these staff and data entry clerks report to both ZPCT II and the MOH. In some cases, the DHMT or facility staff have to seek ZPCT II’s written permission for data entry clerks to travel with them on mobile teams. USAID regulations on the management of donor funding does not allow USG funds be put into GRZ basket funding. In addition, USAID regulations on the management and timely reporting of project funds have resulted in ZPCT II management of essentially all project funds. This is viewed by some PHO and DHO as a lack of confidence in their ability to handle funding, although they handle MOH finances and reporting. The human resources for health are very limited in Zambia. The use of nurse prescribers, incentives to physicians and nurses to serve in rural areas and the building of GRZ multi-agency offices and housing in rural areas have helped. An agreement between the USG and GRZ that USG contractors cannot employ MOH staff is in effect. ZPCT II actions taken in response to USAID management actions and regulations. USAID support was instrumental in the MOH Directorate of Technical Support (TSS) sponsored MOU between ZISSP/Brite Management and Leadership Academy (MLA) and the ZPCT II Capacity Building Program. This MOU recognizes the synergies and disparities between these two programs in building capacity of PMO and DMO staff to improve performance in management functions. Strategic Approaches to USAID Management to Enhance Implementation and Success USAID’s COR provided the team with brief notes in a ZPCT II-USAID meeting. Minutes on January 28, 2011 show topics such as: current items with action, ongoing issues, non-ZPCT II information, travel, and MOH updates. The ZPCT II, as a PEPFAR implementing partner, is provided notice, guidance, templates and training on the Zambia Partner Reporting System (ZPRS) that supports them to submit their results annually to the system. USAID has also provided strategic guidance to ZPCT II on budget realignment issues. Notes from the December 15, 2011 meeting of the ZPCT II COR and ZPCT II management document key issues and action points about the necessary changes for budget realignment. The guidance covered the pace of utilization of funds; the overall rationale for the budget realignment; the rational for specific changes in staffing and increases in fringe benefits; the Negotiated Indirect Cost Rate Agreements (NICRA); and other direct costs (ODC) such as reduced technical assistance, shifts of funding from sub-contractors and reallocation of staff costs to “service center” category. USAID provided clear guidance to ZPCT II to submit strong justifications for budget realignment in two separate documents. Coordination within USG Agencies to Enhance Implementation USG partners reportedly usually meet biweekly but they meet weekly during Country Operating Plan season. The evaluation team’s discussions with the USG Department of Defense (DOD), the Center for Disease Control and Prevention (CDC), and Peace Corps provided information additional to that from USAID. USG partners discussed a high level of coordination within USG agencies in Zambia under the leadership of the Office of the Global AIDS Coordinator (OGAC). There has been coordination on information systems including SmartCare and on assessing the data quality. Specific Steps to Enhance USG Management 44 Given the scale and complexity of the ZPCT II program, additional support should be given to the COR in monitoring and providing guidance to the project in order to enhance USG management. This support could be in the form of periodic strategic reviews with discussions to provide an outside perspective that balances the day-to-day oversight and guidance to ZPCT II. USG management could also define a series of priority management information that it needs to oversee ZPCT II’s operations, although high level reporting is already provided to the USG. Analysis and simple, clear presentations of information and trends in implementation. would allow the USG to see the integration of programming and reporting by ZPCT II and the potential for sustainability of these systems by the MOH at project’s end. Periodic independent visits by the USG to MOH and PATF provincial and district sites would greatly enhance USG project management. Direct dialogue with GRZ partners and civil society would provide invaluable information on the status of the partnership, linkages across sectors, successes and obstacles and the potential for the sustainability of HIV/AIDS services. Pending Management Actions Only one outstanding action was discussed by the COR as pending. This action is that the COR is to discuss with ZPCT II what guidance and management should be provided on TB in HIV. The COR stated that ZPCT II is moving forward on this and has begun to order microscopes. H. RECOMMENDATIONS To address Objective 1. Expanding existing and scaling up new services:  ZPCT II should reassess and adjust the LOP targets for indicators to be challenging yet achievable. This and further analysis would enable ZPCT II to better balance its resources to achieve service delivery, capacity development, partner participation, integration, and other project objectives; this would also aid in the setting of LOP targets. ZPCT II should use elicit external technical assistance and guidance from USAID to guide these adjustments. Special attention should be paid to prevention and community targets to assure sufficient resources are identified.  ZPCT II should develop a structured approach and coordinate timelines with MOH for planning, data management, procurement and other systems. By integrating and harmonizing plans with MOH, ZPCT II would be able to implement expansion and scale-up more effectively.  ZPCT II should increase its support of M&E Working Groups and other venues in order to assist the MOH to streamline and consolidate the multiple data streams and move toward one national HIV/AIDS M&E Plan. In preparation for this, ZPCT II should develop one project M&E system internally by merging its various M&E guiding documents into one PMP for easy standardization and use.  To develop data driven management in ZPCT II and the MOH, ZPCT II should train and mentor data entry clerks, MOH, DHIO, and other staff to strengthen data analysis and use capacity at district and provincial level facilities. ZPCT II should upgrade computer hardware and software, as it is doing for Smart Care. ZPCT II internal project databases at both the central and provincial levels should be strengthened and automated to produce management and technical reports for data driven decision-making ; this would be especially useful the areas of training, procurement, and ART retention. Since this effort would strengthen data for management decision-making, this step would complement, not duplicate, the Smart Care system. 45 To address Objective 2. Increasing involvement and participation of partners and stakeholders:  ZPCT II should provide consistent support to GRZ leadership to encourage discussions related to the coordination of partners, including USG and other donor partners, the MOH, NAC and other GRZ sectors for more effective comprehensive HIV/AIDS programming. USAID could support this with monitoring of progress.  ZPCT II should work more closely with MOH, the USAID DELIVER Project, and the PEPFAR funded Supply Chain Management System (SCMS) project to coordinate for a health sector-wide approach to procurement and maintenance of drugs, supplies, and equipment. ZPCT II should align and synchronize its procurement and maintenance plan with MOH priorities and timelines as specified in the National Strategic Plan (2011-2015). ZPCT II should also increase its internal capacity to guide and manage its procurement and maintenance activities.  ZPCT II should formally and more intensively work across all supported districts with the DATFs through formal MOUs. Additionally, ZPCT II should establish formal links to the District Development Coordinating Committee to strengthen coordination and participate in district-level HIV/AIDS multi-sectoral approaches.  To increase partnership and comply with GRZ policies on gender, ZPCT II should identify, mentor, and support a broad base of community groups and private sector partners to focus on GBV prevention actions at the community level, and to identify GBV cases and refer for care. ZPCT II should lead the development of a set of community-based indicators for public and private sector actors to track and monitor the impact of GBV interventions.  As a base for the strengthening of community partners, ZPCT II should accelerate technical assistance and other resources to integrate GBV case management into the referral system. This process should involve developing a comprehensive toolkit for GBV sensitization at the facility and community level as a resource to address GBV survivors’ medical, legal, and psychosocial needs.  USAID should provide a consistent venue and support GRZ leadership of discussions that strongly support the coordination of partners, including USG and other donor partners, the MOH, NAC and other GRZ sectors for coordinated strategies to move effectively toward the Three Ones. High level USG monitoring of progress should be included. To address Objective 3. Increase the capacity of PHOs and DHOs technical and program management functions:  ZPCT II should expand its current interventions led by sub-partner, Cardno to build the management and technical capacity of PHOs and DHOs (e.g., in the areas of governance, finance, planning, and human resources). ZPCT II should do this across all supported districts to build DHMT technical and management capacity more rapidly. This initiative should be coordinated with current MOH technical and management capacity development approaches to ensure harmony.  ZPCT II should redesign its district level graduation criteria and capacity building approach for facilities to include measuring technical and management capacity at the district level. Adjusted criteria should focus on measuring capacity across the following areas: governance, finance, human resources, and technical abilities especially in service quality. This adjusted approach should be jointly planned with PHOs and DHOs.  USAID should emphasize the need for ZPCT II to strengthen and considerably expand its support to PHO and DHO capacity building approaches. To address Objective 4. Building and managing public private partnerships:  USAID should clarify with ZPCT II if PPPs should be spread across provinces to serve as models for expansion.  ZPCT II should continue to strengthen mechanisms to develop peer exchanges of information and lessons learned across the PPPs. ZPCT II should also encourage interchange between PPPs, DHOs, and the MOH on training curricula and protocols.  ZPCT II should assist private partners to address the tendency of clients to move back and forth between public and private services resulting from low insurance coverage or the inability of clients to pay for services. ZPCT II should work in an area such as the Copperbelt, to pilot a service delivery 46 model and referral system that recognizes and accommodates the tendency of clients to move between public and private health care systems. The evaluation team believes that addressing this issue is feasible in the project’s remaining time. To address Objective 5. Integrating service delivery though joint planning:  ZPCT II should consider how to strengthen its strategic approaches to integrate family planning, chronic diseases care, and other services into HIV/AIDS services. If feasible, ZPCT II should consider how to document lessons learned in order to contribute its strong clinical capacity to the MOH national strategy being developed to incorporate the control of chronic diseases (e.g., diabetes, hypertension, etc.) into MOH protocols and training. ZPCT II could assess the feasibility of approaches being considered and help pilot effective integrated services as models. USAID and ZPCT II should discuss this issue to assure agreement on a feasible approach. To address the overall USAID oversight function:  Given the scale and complexity of ZPCT II, USAID should provide additional level of effort to supply strategic, high level monitoring, and guidance to ZPCT II as it undertakes strategic changes to its project.  USAID should conduct periodic independent visits to MOH and NAC provincial and district sites to discuss project progress and obstacles, and support the strengthening of relationships between ZPCT II, GRZ partners, and civil society. I. HIGHER-LEVEL ISSUES Policy dialogue on the issue of GRZ allowances should be given high priority due to its significant effect on ZPCT II and on PHO and DHO priorities, planning, and the use of their own financial resources. PHOs and DHO are being “expected” by staff to provide allowances or additional reimbursements to the amount paid by ZPCT II allowances. This results in PHO and DHO financial debts that are being incurred by USG and other donor projects. This issue affects many USAID and other donor partners: there is little cohesion among partners and donors in the approach and the amounts paid. Integrating financial resources would not greatly benefit PHOs or DHOs because the amount of allowances is controlled by USG and other policies. Policy dialogue on prevention is critical for strengthening public health approaches to address the drivers of the HIV epidemic. The USG should prioritize dialogue with the GRZ, other donor governments, and multilaterals to increase overall human and financial resources for systems strengthening for prevention interventions. This dialogue should emphasize advancing the capacity of the health sector and civil society to expand services and lead broad strategies for the prevention of new infections. USAID could use ZPCT II as a venue to leverage existing opportunities to address regional patterns of HIV transmission. The evaluation team found that planning is needed to link HIV prevention activities with the potential risks that can come with the expansion of roads and trade. Other high level issues of concern include the need to openly address HIV prevention in Zambian society and risky cultural patterns that drive the epidemic. USG agencies should consider the realignment of resource allocations and dialogue towards the USG Zambia Partnership Framework’s first commitment: “Accelerate and intensify prevention in order to reduce the annual rate of new HIV infections.” Lessons Learned Integration. ZPCT II has built a somewhat parallel approach in its efforts to accelerate HIV/AIDS interventions against targets. ZPTC II approaches and systems need to be assessed and adjusted to become more fully integrated with the GRZ approach, systems and timelines. ZPCT II can use lessons learned from their HIV/AIDS programs in other countries and from other USG partners in Zambia. ZPCT II should document this integration process to provide valuable lessons learned. Alignment. ZPCT II planning, implementation, and monitoring of services could be better aligned with MOH at all levels. For example, ZPCT II could develop best practices for joint planning at the district 47 level to avoid separate ZPCT II’s plans that are often perceived as parallel. These practices could provide lessons learned for sustainable technical and managerial interventions. Gender integration. Gender integration within ZPCT II and its partners requires strong technical assistance at project start up and throughout the project to assist managers and technical staff to “engender” planning, programming and budgeting and to track gender differentiated outcomes. Environmental consideration. ZPCT II was aligned with Zambian environmental impact and mitigation considerations from project start up and has had focal point persons nationally and in provincial offices to ensure compliance. This has assisted the project to meet national requirements as well as assist partners to adopt environmentally friendly practices that are replicable to non-ZPCT II supported sites to enhance sustainability. 48 ANNEX A EVALUATION SCOPE OF WORK SECTION C – DESCRIPTION / SPECIFICATIONS/STATEMENT OF WORK C.1 IDENTIFICATION OF THE TASK The USAID/Zambia Population, Health, and Nutrition Office seeks an independent team to perform a mid-term evaluation of the Zambia Prevention, Care, and Treatment II Project (ZPCT II). ZPCT II is a five-year, $124 million contract that is tasked with public sector HIV/AIDS service delivery support in five provinces of Zambia. To date, $82 million has been obligated into the award, across nine program areas. The objectives of this mid-term evaluation are three-fold: a) to assess progress made toward achieving project objectives, targets and milestones; b) to recommend modifications to project activities or priorities, as necessary, to address implementation issues, apply lessons learned, or capitalize on new opportunities; and c) identify relevant issues that require discussion and resolution at a level higher than the ZPCT II project. USAID/Zambia requests that the in-country components of this evaluation be fielded on or about June 2012. C.2 BACKGROUND The prevalence of HIV infection among Zambian adults (men and women age 15-49) was 14.3 percent in 2007, representing a decline from the prevalence of 15.6 percent reported in 2002. In particular, decreases occurred among women (from 17.8 to 16.1 percent) and urban residents (from 23.1 to 19.7 percent). These trends, however, belie several patterns of the HIV/AIDS epidemic in Zambia. Between 2001 and 2007, adult HIV prevalence increased in three of the nine provinces by an average of 2.1 percent. The proportion of multiple concurrent partnerships increased among women (from 13.2 to 16.9 percent) whose condom use with their non-spousal/non-regular partner remains low (37.4 percent). These patterns and the continued high prevalence of HIV infection among the general population underscore a need for comprehensive HIV/AIDS programming. The U.S. Mission to Zambia’s HIV/AIDS interagency team under the President’s Emergency Plan for AIDS Relief (PEPFAR), which is composed of the Department of State, the Department of Defense, the U.S. Centers for Disease Control and Prevention (CDC), and the U.S. Agency for International Development (USAID), began HIV/AIDS programming in 2003 under the oversight of the Office of the Global AIDS Coordinator. Peace Corps joined the PEPFAR team in 2004. The U.S. Mission collaborates with the Government of the Republic of Zambia (GRZ) through the Ministry of Health, the National AIDS/HIV/TB/STI Council, and other line ministries through their HIV/AIDS focal points. C.3 Overview of the Zambia Prevention, Care and Treatment (ZPCT) Program ZPCT I (2004-2009), implemented by Family Health International, worked in partnership with the Ministry of Health to improve HIV/AIDS services in five of the country’s nine provinces. ZPCT I assisted the GRZ to initiate, scale up, and strengthen a comprehensive package of HIV/AIDS services – counseling and testing (CT), prevention of mother to child transmission (PMTCT), clinical care and ART – to serve hundreds of thousands of Zambians in the communities where they live. Day-to-day program functions were managed by ZPCT’s provincial offices through recipient agreements directly with the Ministry of Health at provincial, district and facility levels. The recipient agreements provided for joint activity planning with the Ministry of Health while the project managed financial matters. ZPCT I assistance included infrastructure renovations, provision of essential medical and laboratory equipment, M&E support, technical assistance and training and mentoring for health care workers and community volunteers across all technical areas. ZPCT II is a five-year, $124 million contract supporting HIV/AIDS service delivery in five of Zambia’s nine provinces (Central, Copperbelt, Luapula, Northern, and North Western) with five objectives of: Expanding existing HIV/AIDS services and scaling up new services, as part of a comprehensive package that emphasizes prevention, strengthening the health system, and supporting the priorities of the Ministry of Health and National AIDS/HIV/TB/STI Council. 49 Increasing the involvement and participation of partners and stakeholders to provide a comprehensive HIV/AIDS service package that emphasizes prevention, strengthens the health system, and supports the priorities of the Ministry of Health and National AIDS/HIV/TB/STI Council. Increasing the capacity of the Provincial Medical Offices and District Medical Offices to perform technical and program management functions. Building and managing public-private partnerships to expand and strengthen HIV/AIDS service delivery, emphasizing prevention, in private sector health facilities. Integrating service delivery and other activities, emphasizing prevention, at the national, provincial, district, facility, and community levels through joint planning with the GRZ, other USG, and non-USG partners. he life-of-project targets for ZPCT II are contained in Annex L. In addition to the life of project service delivery targets contained in Annex L, FHI360 is responsible for ensuring that key principles and requirements established under the contract are applied and met during the course of implementation. These principles and requirements are summarized in Annex L. C.4 ZPCT II Achievements and Challenges to Date Since the ZPCT II launch in September 2009, following a three month transition from ZPCT I, there have been notable achievements. Services are now available in 41 of the 42 planned districts and 15 participating districts have graduated from intense technical assistance. The numbers of outlets providing Testing and counseling, Prevention of mother to child transmission of HIV, TB/HIV, and male circumcision services and the numbers of individuals accessing services have substantially exceeded targets. For example, the 2014 life of project target for individuals receiving ART is 146,000. Between August 2009 and April 20ll, however, the numbers of individuals receiving ART increased from 78,000 to 129,225, meaning that 75 percent of the life of project target had already been achieved less than two years into implementation. Counseling and testing services are currently reaching 649,000 people, almost 90 percent of the life of project target. Performance monitoring, discussions with staff and observations on site have nonetheless identified a number of ongoing or emerging implementation challenges: Despite the capacity building plan being developed under the leadership of the Ministry of Health and in partnership with other development partners, the implementation of this project component lags behind other service delivery components; Human resource shortages exist at all levels of service delivery, but are especially acute at the health center level; High demand for HIV/AIDS services is outstripping the space currently available in many health facilities; Delayed roll out of the quality assurance instruments and systems planned under the project is slowing the pace of institutionalizing these systems in Ministry of Health facilities; and, The transportation-related costs of maintaining community volunteers, who are the backbone of HIV/AIDS service delivery, threaten program sustainability. C.5 STATEMENT OF NEED/PURPOSE Purpose Implementation of the ZPCT II Task Order has reached its mid-point in December 2011. USAID Zambia plans to conduct a mid-term external evaluation to assess the progress of the ZPCT II program towards meeting its intended results. The objectives of this mid-term evaluation are three-fold: Part A (Retrospective): To evaluate the progress made toward achieving project objectives, including an assessment of project design (conceptual framework) in light of implementation experience to date; Part B (Prospective): Based on the above findings, to make recommendations for ZPCT II project implementation through 2014, including the optimal mix of activities and funding for achieving project objectives and sustainability and, 50 Part C: Using the above findings, frame issues to debate/discuss/resolve at a level higher than the project, e.g. at the level of the GRZ and/or other donor organizations. Statement of work The mid-term evaluation team will undertake the following tasks. For each Task, the evaluation team will assess progress and provide specific recommendations for modifications to project activities or management, as needed, and frame any higher-level issues to be discussed externally. C.5.1 Specific Tasks C.5.1.1 Task I: Effectiveness: Evaluate ZPCT II Project design, performance, contract objectives and implementation to date holistically for each PEPFAR funding area and project objectives. To what extent is the project on course to meet the life of project targets? To what degree do the project outputs represent the maximum possible given the inputs in each program area? Are the life of project targets appropriate, given the implementation environment and the evolution of HIV/AIDS service delivery in Zambia? If not, the evaluation team should make specific recommendations for contractual modifications, such as life of project targets, tasks or other aspects of the contract C.5.1.2 Task II: Efficiency: Evaluate the mix of activities and funding allocated to ZPCT II. To what degree do the program outputs represent the optimal combination to address HIV/AIDS in the program’s country context? To what degree are planned activities and corresponding budgets adequate and appropriate to achieve targets and objectives? How well have the project resources been used? What opportunities exist to accelerate project impact with a modified mix of activities or funding? Are there cost-efficiencies that could be achieved? C.5.1.3 Task III: Cross-cutting principles: Evaluate FHI360’s success at implementing key principles and requirements established under the contract during the course of implementation (see Annex L). To what degree do these principles or requirements need further attention or modification during the second half of ZPCT II implementation? Are they still relevant? Assess the degree to which the project is complying with USAID’s mandatory requirements such as gender integration and environmental compliance. For example, how is the project addressing gender inequalities/inequities in service provision, service delivery and behavior change for the beneficiaries? Assess the adequacy of ZPCT II’s gender strategy? To what degree is the project adhering to the implementation of the environmental compliance and mitigation plan? C.5.1.4 Task IV: Factors of success: Describe the project’s key success factors in terms of field-level implementation, project systems, and project management. What are the promising lessons and approaches for expanding the comprehensive HIV/AIDS service delivery services to additional sites? What are the lessons learned from successful interventions that merit continuation or replication? What practices, products and tools should be considered for dissemination? C.5.1.5 Task V: Obstacles: Describe the project’s key obstacles in terms of field-level implementation, project systems, and project management. What approaches, lessons learned from other settings, or project modifications might address these obstacles? Are there any obstacles that require policy changes or dialogue? Are there obstacles that threaten project performance or sustainability? C.5.1.6 Task VI: Sustainability: Evaluate ZPCT II contributions to building sustainable HIV/AIDS service delivery in Zambia and GRZ ownership of HIV/AIDS programs. What is ZPCT II’s contribution to developing a comprehensive Ministry of Health HIV/AIDS program in the five supported provinces? How do participating Ministry of Health districts and provinces show signs of improving their ability to manage services and operations (managerial, financial, technical etc.)? How does HIV/AIDS service delivery in “graduated” facilities compare to services in facilities that continue to receive technical assistance under ZPCT II? Are these activities likely to continue without further USG investments? What investments or approaches would promote services delivery program sustainability? To what extent is the GRZ demonstrating ownership and commitment to sustaining accomplishments under ZPCT II? 51 C.5.1.7 Task VII: Beneficiary relations: Evaluate the project’s success at delivering quality HIV/AIDS services to beneficiaries. Does services coverage (location and range of services offered) represent an efficient use of project resources? How do program beneficiaries view the following: hours of operation, professionalism of staff, quality of care, confidentiality, ease of accessing multiple services, availability of product? Compare project beneficiaries and potential beneficiaries who do not utilize services. What factors influence the decision to use or not use services? Evaluate the success of community volunteers in helping the project achieve its objectives. C.5.1.8 Task VIII: Linkages, coordination and leveraging: Evaluate the project’s success at establishing linkages between and among stakeholders. How effectively has FHI360 the project coordinated, managed and linked district, facility and community-based services? Has ZPCT II succeeded at meeting its public-private partnership targets? Evaluate the effectiveness of ZPCT II in creating these partnerships? Are there best practices or approaches to developing sector linkages that should receive additional attention or resources during the second half of the project? Assess ZPCT II project’s wider participation in terms of collaboration and coordination within the partnership, other USG programs/projects, and other key donor programs/projects, and stakeholders in HIV/AIDS arena. C.5.1.9 Task IX: Performance monitoring and impact: Evaluate ZPCT II performance monitoring systems. What M&E systems does FHI360 ZPCT II use in monitoring progress and trends? Does the M&E plan facilitate utilization of data for improved performance? Has the project established reasonable methods of gathering the data necessary to monitor and evaluate progress and indicator data? Is the performance management plan developed and used to provide timely and high-quality data? If not, why not? What suggestions does the team have for improving the M&E system? What else is going on in the health sector/environment that might account for the changes or the project achieving its objectives? C.5.1.10 Task X: Management systems within the project: Evaluate the factors that have contributed to or inhibited the functionality and performance of the ZPCT II project partnership, which includes seven sub-partners and the Ministry of Health. Evaluate the performance and effectiveness of the prime, sub￾partners and sub-grantees in the ZPCT II project coalition and their contribution towards realization of the program objectives. C.5.1.11 Task XI: USAID’s project management: Evaluate USAID's management of the ZPCT II project. Are there any management actions or steps that might enhance project implementation and success? C. 6. METHODS AND PROCEDURES: The evaluation team is expected, in its proposal, to describe in detail a methodology for collecting the necessary information and data. The proposal should include a description of how the methodology responds to the above tasks and questions; from whom and how data will be collected; and how the data will be analyzed. The methodology should be collaborative and participatory, including partners and key stakeholders (including beneficiaries, GRZ, USAID, and other USG staff, and other donors) to the extent possible in planning and conducting the evaluation. The evaluation team shall be expected to apply both qualitative and quantitative methods and approaches for collecting and analyzing the information required to assess the evaluation objectives. The methodology will be discussed with and approved by USAID/Zambia prior to implementation. The methodology should include, but not be limited to document reviews of ZPCT II project contract, reports (quarterly, semi-annual and annual reports), Ministry of Health reports and strategic approaches, ZPCT II project reports, key informant interviews, client interviews, and structured site visits and observations. The contractor will be expected to conduct field work in two randomly-selected districts (one urban, one rural) per province in the five provinces supported by the project. The field work shall include key informant interviews including with health facility staff, people living with HIV/AIDS, program beneficiaries, district medical staff, provincial medical staff and Ministry of Health technical officers. Participatory Process: The assessment shall be conducted in a participatory manner. The contractors are expected to host a series of consultative meetings/workshops with the Ministry of Health and implementing partners in an attempt to identify needs or gaps. To ensure ownership of the results and 52 recommendations, the contractors shall present their draft findings to the partner for review and comments prior to finalizing the report. Consultation with USAID and USG, : The contractor shall be expected to consult regularly with the USAID/technical officer responsible for the ZPCT II program, other USG health sector staff, and other donors implementing similar activities in Zambia. In-house review: The draft report will be peer reviewed by a selected team comprising of USAID and other USG staff members. The input from this review will be provided to the evaluation team and will need to be incorporated in the final report. Consultation with beneficiaries and stakeholders: The contractor shall visit and verify program activities in the field and consult widely with beneficiaries and other stakeholders. The input from the beneficiaries and stakeholders shall be used to inform recommendations and future plans for the program. The Contracting Officer’s Representative will provide technical directions during the performance of this Scope of Work (SOW). Additional USAID criteria for a quality evaluation and report are contained in Annex M. C.7 PROPOSED EVALUATION TIMELINE Below is a list of the specific tasks to be accomplished by the evaluation team, with an estimated level of effort for each task. Task Duration Review background documents and prepare draft work plan; discussions with USAID contacts; field logistics planning and extricate or validate the ZPCTII theory of change. 10 days Travel to Zambia 2 days Team planning meetings: finalize evaluation methodology; plan interview and site visit schedule; develop interview and site visit protocols; assign team tasks and writing responsibilities; prepare draft report outline.* 6 days Interviews with Ministry of Health and implementing partner representatives at national level. 6 days Conduct field visits and interviews (three teams in five regions), including data collection, key informant interviews, site visits, observations. 24 days Full team synthesis and analysis of findings, follow-up interviews, and preparation of draft report and recommendations.** 12 days Debrief meeting and presentation of key findings with USAID/Zambia, Ministry of Health and other key stakeholders. 1 day Depart Zambia 2 days USAID/Zambia, Ministry of Health and ZPCT II provide comments on draft report (~4 weeks) Revise report based on feedback. 10 days Submit final report to USAID December 3, 2012 TOTAL 73 days *Draft site visit protocols and draft report outline to be reviewed and cleared by USAID ** Draft evaluation report and recommendations will be submitted to USAID in advance of in-country debriefing and presentation of key findings. Final report will incorporate USAID feedback from debriefing presentation and review of draft report. The mid-term evaluation will take place over a period of approximately 12-weeks beginning in August, 2012, with in-country tasks commencing by August 13, 2012 and final report submitted by December 3, 2012. Total level of effort (LOE)—73 days of LOE for Evaluation Team Leader (including two international travel days); 71 days for HIV/AIDS Program Management Specialist (including two international travel 53 days); up to 63 days each for two of the local HIV/AIDS Program Experts; 65 days for the local specialist Deputy Team Leader, and 65 days for Evaluation Logistics Coordinator. A six-day work week is authorized for work in Zambia. END OF SECTION C 54 Annex B ZPCT II SUPPORTED FACILITIES (BY DISTRICT) ZPCT II Supported Facilities and Services District Number of Health Facilities Urban Facilities Rural Facilities ART PMTCT CT CC Lab Specimen Referral for CD4 Male Circumcision Central province Kabwe 16 15 1 9 16 16 16 11 7 2 Mkushi 8 1 7 3 8 8 8 3 4 1 Serenje 11 1 10 3 11 11 11 3 6 2 Chibombo 15 0 15 4 15 15 15 4 11 1 Kapiri Mposhi 18 3 15 5 18 18 18 4 15 1 Mumbwa 6 2 4 0 5 6 6 1 3 1 Sub-totals 74 22 52 24 73 74 74 26 46 8 Copperbelt Province Ndola 19 19 0 10 18 19 19 10 13 0 Chingola 8 6 2 6 8 8 8 5 5 3 Kitwe 23 23 0 15 22 23 23 13 17 0 Luanshya 6 6 0 3 6 6 6 4 4 1 Mufulira 9 6 3 2 9 9 9 2 7 2 Kalulushi 5 3 2 3 5 5 5 2 2 0 Chililabombwe 2 2 0 2 2 2 2 1 1 1 Lufwanyama 3 0 3 2 3 3 3 1 2 0 Mpongwe 3 0 3 0 3 3 3 3 3 1 Masaiti 3 0 3 0 3 3 3 1 3 0 Sub-totals 81 65 16 43 79 81 81 42 57 8 Luapula Province Chienge 4 0 4 2 4 4 4 2 2 0 Kawambwa 8 1 7 4 8 8 8 3 4 2 Mansa 29 4 25 5 29 29 29 5 19 3 Milenge 3 0 3 1 3 3 3 1 1 0 Mwense 22 0 22 5 16 22 22 4 14 0 Nchelenge 10 0 10 10 10 10 10 2 8 1 Samfya 6 0 6 3 6 6 6 3 3 1 Sub-totals 82 5 77 30 76 82 82 20 51 7 Muchinga Province Nakonde 8 0 8 3 8 8 8 1 5 1 Mpika 11 2 9 2 11 11 11 3 6 1 Chinsali 6 2 4 1 6 6 6 2 2 1 Isoka 5 2 3 2 5 5 5 2 2 1 Mafinga 2 0 2 1 2 2 2 1 1 0 Sub-totals 32 6 26 9 32 32 32 9 16 4 Northern Province Kasama 13 3 10 6 13 13 13 5 10 0 Mbala 11 3 8 3 11 11 11 4 5 1 Mpulungu 3 1 2 1 3 3 3 1 0 1 Mporokoso 2 2 0 2 2 2 2 2 1 1 Luwingu 2 2 0 1 2 2 2 1 1 1 Kaputa 4* 0 2 1 4 4 4 4 3 1 Mungwi 7 0 7 2 7 7 7 1 5 0 Chilubi Island 3 0 3 1 3 3 3 1 2 0 Sub-totals 45 11 30 17 45 45 45 19 27 5 North-Western Province Solwezi 14* 2 9 3 14 14 14 4 8 0 Kabompo 7 1 6 2 7 7 7 2 2 1 Zambezi 9 3 6 2 8 9 9 2 2 1 55 ZPCT II Supported Facilities and Services District Number of Health Facilities Urban Facilities Rural Facilities ART PMTCT CT CC Lab Specimen Referral for CD4 Male Circumcision Mwinilunga 10 1 19 2 10 10 10 2 1 1 Ikelenge/ Mufumbwe 6 0 6 1 6 6 6 1 3 1 Chavuma 5 0 5 1 5 5 5 2 3 1 Kasempa 6 1 5 1 6 6 6 1 1 1 Sub-totals 57 6 47 12 56 57 57 14 20 6 TOTALS 371 115 248 135 361 371 371 130 217 38 ART – Antiretroviral Therapy; CC – Clinical Care; CT – Counseling and Testing; PMTCT – Prevention of Mother to Child Transmission; MC – Male Circumcision. * Not all facilities labeled as urban/rural. ZPCT II Private Sector Facilities and Services District Number of Health Facilities Urban Facilities Rural Facilities ART PMTCT CT CC Lab Specimen Referral for CD4 Male Circumcision Central Province Kabwe/ Mkushi 4 4 0 0 2 4 4 4 0 0 Copperbelt Province Ndola 6 6 0 6 3 6 6 4 5 0 Kitwe 5 5 0 5 4 5 5 4 3 0 Mwense 1 1 0 1 0 1 1 0 0 0 North-Western Province Solwezi 2 2 0 2 1 2 2 2 0 2 TOTALS 18 18 0 14 10 18 18 14 8 2 ART – Antiretroviral Therapy; CC – Clinical Care; CT – Counseling and Testing; PMTCT – Prevention of Mother to Child Transmission; MC – Male Circumcision 56 ANNEX C PROFESSIONAL BACKGROUND OF THE IBTCI EVALUATION TEAM IBTCI Evaluation Field Team Rose Schneider, RN MPH, Team Leader Ms. Rose Schneider is a Senior Health and HIV/AIDS Specialist with over 25 years of public health and HIV/AIDS experience in developing countries in Africa, Asia, Latin American, and the CIS. She is well￾versed in all aspects of HIV/AIDS program development, especially with PEPFAR programs in Africa, and has a specific focus on sustainable and practical prevention solutions, and treatment services at a local level. Ms. Schneider has worked consistently for over 25 years in HIV/AIDs and family health programming. During her seasoned career, Ms. Schneider has worked with PEPFAR, USAID and other organizations including the World Bank, UNFPA, Peace Corps, GHTech, PATH, MSH, JSI, Save the Children, PLAN and others. As a professional with vast international experience, Ms. Schneider has demonstrated her in-depth analytical, managerial, organizational, communication, writing, and interpersonal skills and abilities. While conducting assessments for Child Survival Program Design in Ethiopia for CARE in 2001, she worked consistently in partnership with District Health Team, traditional healers, community leaders and other private health partners. In this collaboration, Ms. Schneider utilized her communication and interpersonal skills to interact successfully with local communities. An example of her ability to interact with international organizations and local communities simultaneously was in Malawi with the BASICS Project, where she conducted an assessment of twelve international PVOs in their technical and management capabilities to partner with the local MOH District Teams for Child Survival service delivery. For Zambia, Uganda and Kenya, in 2004, Ms. Schneider designed a successful OVC program to create a USAID funded multi agency collaborative. As part of her design, Ms. Schneider developed technical program approach, multi-agency management structure, implementation plan, activities and sequencing, as well as defined roles and responsibilities of partners. Ms. Schneider has worked as a clinical nurse, nurse supervisor, and nurse trainer in the U.S. and in Latin American and the Caribbean. She served with CARE/Medico as teaching hospital supervisor and nurse trainer in a 400 bed regional hospital in Honduras and as a nurse trainer in the Dominican Republic. After receiving her BSN/RN from the Catholic University of America in Washington DC in 1975, Ms. Schneider received her MPH from John Hopkins School of Hygiene and Public Health in 1976. Her native language is English, and she has demonstrated her mastery of the language with strong oral and writing skills. Ya-Shin Lin, HIV/AIDS Program Management Specialist Ms. Yashin Lin is an evaluation and HIV/AIDS program specialist with 12 years of experience in international health, including nearly a decade of experience supporting a USAID-funded projects focusing on quality assurance in health service delivery. She has evaluated PEPFAR funded programs in Angola and Vietnam for USAID missions and has also conducted assessments of HIV/AIDS service delivery quality in Cote d’Ivoire and Rwanda. While working for the Healthcare Improvement Project, Ms. Lin provided technical assistance and training to HIV/AIDS quality improvement collaboratives in Rwanda from 2002-2006. In this role, Ms. Lin developed and delivered presentations, flowcharts, and other training tools to teach health care teams to analyze the quality of HIV/AIDs services, identify and resolve quality assurance issues, as well as establish and maintain HIV/AIDS information systems. She also conducted a health systems assessment in Angola and delivered her findings and recommendations to address human resources, quality assurance, and other capacities. Ms. Lin developed a replicable health service delivery assessment module based on this experience. She has also mentored program and clinical staff in Rwanda, Cote d’Ivoire, Mozambique and Benin on how to develop and implement M&E and quality improvement systems for service delivery. Ms. Lin has served as a Research and Evaluation Advisor at the USAID Office of HIV/AIDS from 2009 - 2011 and holds a Master of Public Health (MPH) degree in Epidemiology with a concentration in Maternal and Child Health (MCH) from the University of Washington. Her country experience includes Angola, Benin, Cote d’Ivoire, Democratic Republic of Congo, Ecuador, Eritrea, Guatemala, Honduras, Malawi, Mozambique, Nicaragua, Rwanda, South Africa, Vietnam. 57 Beyant Kabwe, Deputy Team Leader/ Zambia-based HIV/AIDS Program Expert Mr. Kabwe is an international development and monitoring and evaluation specialist with 10 years of experience in conceptualizing, designing and implementing management systems for international projects, in health: specifically related to HIV/AIDS programming. Successfully helped Zambia develop the National HIV/AIDS/STI/TB M&E system in 2002 and National M&E frameworks/plans for 2002 to 2005, 2005 to 2010 and 2006 to 2015. He is well versed in qualitative and quantitative research design, supervision and implementation. Mr. Kabwe has expertise in the areas of PEPFAR, PMTCT, male circumcision, counseling and testing, TB-HIV, 01 Care, and quality assurance in both the public and private sectors. Mr. Kabwe is studying towards a MPH in Health Promotion from Leeds Metropolitan University of UK and holds a BA in Development Studies from University of Zambia. He also holds a certificate from University of Pretoria for Monitoring and Evaluation of HIV/AIDS programs in conjunction with Measure Evaluation/USAID. He is member of the Monitoring and Evaluation Technical working group at National AIDS Council (NAC) since 2003 and has been saving as a chair of this TWG since 2010, he is a member of the Zambia Evaluation Association as well as International AIDS Society. He has attended a myriad of trainings covering subject like M&E; Population, Health and Environment, Research and Ethics, Strategic Planning; and Training of Trainers. Robie Siamwiza, Zambia-based HIV/AIDS Program Expert Robie Siamwiza has been an HIV/AIDS and reproductive health (RH) professional for 20 years. Her strong areas of expertise include HIV and AIDS multisectoral policy analysis and program development; HIV/AIDS strategic planning including coordinating the development of the 2011-2015 National HIV/AIDS Strategic Framework; skills in advocacy; designing and implementing an HIV/AIDS human rights program to combat stigma and discrimination; designing and implementing gender analysis and sensitization training programs; skills in community projects - mobilization, project design and implementation. While working as Country Director of POLICY Project in Zambia, she worked with faith-based groups to pilot the establishment of multisectoral referral centers in all districts in Southern Province. The work was funded under the first round of PEPFAR funding in Zambia and was undertaken to provide support services to people living with HIV. Anderson Nkhuwa, Evaluation Logistics Coordinator Mr. Anderson Nkhuwa is a professional program associate with experience in logistics, budgeting, and project support. As the program associate for Education Quality Improvement Program, he is especially knowledgeable in Zambian program budgeting and design. Most recently, he trained 30 EQUIP2 staff in budgeting, planning and reconciliation of travel advances in accordance with the Academy for Education Development and USAID financial procedures. In preparation for the 2009 Annual Work Plan and Budgets hearing at Mulungushi Conference Centre, Mr. Nkuwa organized and provided logistics to more than 200 Ministry of Education Senior Officers. Mr. Nkhuwa received his B.A. in Development Studies and Economics from the University of Zambia. IBTCI Home Office Support Team Ajay Kalotra, Program Director As CEO of IBTCI, Mr. Ajay Kalotra provides overall leadership and strategic direction to the company. As part of his duties, Mr. Kalotra manages programs within IBTCI’s five practices, which include: Monitoring and Evaluation, Financial Management & Accountability, Banking & Non-Bank Financial Markets, Procurement Reform, and Sector Development and Competitiveness. Mr. Kalotra also provides guidance to IBTCI’s Business Development practice, where he oversees the capture management and proposal development processes, writes and reviews technical and cost proposals, and leads IBTCI’s marketing efforts. Kris Merschrod, Senior Technical Advisor Dr. Merschrod has over 30 years of experience in research design, monitoring, evaluating, and creating Performance Measurement Plans (PMPs) for international development programs, focused in sociological economic development, energy efficiency, and community and sustainable development. He is well versed in evaluation impact and performance evaluation methodologies using “before and after” indicators as well as time series data. For the last three decades, Dr. Merschrod has served as a Chief of Party (COP) and an independent consultant working with USAID Missions and nongovernmental 58 organizations (NGOs) to develop logical and results frameworks and related PMPs to ensure that project indicators meet USAID needs as well as the day to day M&E needs of the implementing partners. His evaluation and design work has included: production chains, marketing, agricultural extension, comparative credit schemes, cooperative development, reducing violence against women, sustainable forestry, irrigation, and alternative development. Dr. Merschrod holds a Masters and Ph.D in the Sociology of Development both with a specialization in research and evaluation methods. Ms. Gajarsa is an international development professional with eight years of experience providing technical, management, and M&E support to programs funded by USAID, the U.S. Department of State, the U.K. Department for International Development, and the Massachusetts Department of Public Health (MDPH). Prior to joining IBTCI, Ms. Gajarsa developed and managed monitoring and evaluation systems for human security and community development programs in Liberia and Afghanistan. As a M&E Consultant for LandMine Action in Liberia, Ms. Gajarsa designed an outcome evaluation framework for assessing the reintegration outcomes of 350 ex-combatant graduates of an agricultural training Program, including work plans, research questions, indicators, coded datasets, and databases. As a Geographic Information Systems (GIS) Analyst at the MDPH, Ms. Gajarsa routinely performed the geocoding of health surveillance and environmental data, as well as created maps for community health investigations, policy decisions, reports, presentations, and the internet. Bryan Shipp, Senior Program Coordinator Mr. Bryan Shipp is a Senior Project Coordinator for IBTCI’s Monitoring and Evaluation practice, where he provides administrative, operational and technical support for Mission-wide M&E programs, performance and impact evaluations and sector assessments. Mr. Shipp has provided support to programs in a variety of sectors including democracy and governance, economic growth and health, for clients including USAID, Department of State and World Bank. He possesses understanding of M&E principles, demonstrated through hands-on experience with the development of PMPs for USAID in Russia and Somalia, and has supported drafting of numerous evaluation and assessment reports. Mr. Shipp received his B.A. in Political Science from Virginia Tech. Elizabeth Dean, Program Coordinator Elizabeth Dean is a Program Coordinator who is well-versed in business development, recruitment, and providing logistical and financial administration support to international programs. Ms. Dean provides business development support for IBTCI's M&E Practice and has contributed to successful proposals under the USAID Evaluation Services Indefinite Quantity Contract. Ms. Dean has provided logistical and financial administration support to a number of international programs and evaluation field teams. She also provides technical support such as cleaning and entering collected data into various databases and spreadsheets as well as copy editing final evaluation reports and assessments to aid in the high quality of project deliverables. Ms. Dean earned her degree in Communication Studies from Christopher Newport University. 59 ANNEX D LIST OF DOCUMENTS REVIEWED Byrne, Christine. Office of Inspector General, Audit of USAID/Zambia's Gender-Related HIV/AIDS Activities. Rep. Vol. 4-611-11-006-P. 2011. Print. Country Operational Plan FY 2012. Country Operational Plan FY 2012. Raw data. Family Health International. 2010 Annual Reports: Sub-Partner-Level Aggregates of Direct Counts. Raw data. Family Health International. 2012 Semi-annual Reports: Sub-Partner-Level Aggregates of Direct Counts. Raw data. Family Health International. Environmental Mitigation and Monitoring Plan/Public Sector HIV/AIDS Service Delivery Support Program in Zambia (Zambia Prevention, Care and Treatment Partnership II. Rep. Print. Family Health International. Marking and Branding Plan Public Sector HIV/AIDS Service Delivery Support Program in Zambia. Rep. Print. Family Health International. Section 3.1 Accomplishments and Constraints. Raw data. Family Health International. Section 3.1 Accomplishments and Constraints. Raw data. Family Health International. Zambia FY2012 Emergency Plan Reporting. Raw data. Family Health International. Zambia Prevention, Care and Treatment Partnership: End of Program Report October 2004 - September 2009. Rep. Print. FHI360 FY12 Workplan Monthly Projection. Raw data. Maryce, Ramsey. Zambia Gender Assessment Report. Rep. 2010. Print. McCord, Judith. Judith Robb-McCord, Consultant's Report. Rep. Print. National AIDS Strategic Framework 2011 - 2015: Towards Improving the Quality of Life of the Zambian People. Rep. 2010. Print. RFTOP: Public Sector HIV/AIDS Service Delivery Support. Rep. Print. Southern Africa HIV and AIDS Information Dissemination Service (SAfAIDS), and Irish Aid Government of Ireland. SAfAIDS Zambia Gender and HIV Policy Brief. Rep. Print. Southern Africa HIV and AIDS Information Dissemination Service (SAfAIDS), and Youth Vision Zambia (YVZ). Situational Analysis on SRH/HIV Integration in Zambia. Tech. 2011. Print. Summary of the U.S. President’s Emergency Plan for AIDS Relief (PEPFAR) HIV/AIDS Partnership Framework with the Government of the Republic of Zambia. Rep. Print. United States Agency for International Development. Task Order: Public Sector HIV/AIDS Service Delivery Support Program. Rep. Print. USAID/Zambia. Country Development Cooperation Strategy 2011-2015. Rep. Print. Zambia Prevention, Care and Treatment Partnership II - ZPCT II Work Plan January 1, 2011 -- December 31, 2011. Raw data. Zambia Prevention, Care and Treatment Partnership II (ZPCT II) Work Plan for Year One Period: January 1, 2011 - December 31, 2011. Rep. 2010. Print. Zambia Prevention, Care and Treatment Partnership II (ZPCT II) Work Plan for Year One Period: June 1 - December 31, 2010. Rep. Print. Zambia Prevention, Care and Treatment Partnership II (ZPCT II) Work Plan for Year One Period: June 1, 2009 - May 31, 2010. Rep. Family Health International, 2009. Print. Zambia Prevention, Care and Treatment Partnership II (ZPCT II) Work Plan January 1, 2011 - December 31, 2011. Rep. 2010. Print. Zambia Prevention, Care and Treatment Partnership II (ZPCT II) Work Plan January 1, 2012 - December 31, 2012. Rep. Print. 60 Zambia Prevention, Care and Treatment Partnership II (ZPCT II) Work Plan January 1, 2012 - December 31, 2012. Rep. Print. ZPCT II COP !! Targeting Setting Final: Indicators. Raw data. ZPCT II Performance Management Plan. Rep. Print. ZPCT II Project Achievements: 1 August 2009 to 31 December 2010. Rep. Print. ZPCT II Quarterly Report (June- Sept 2009). Rep. Print. ZPCT II Supported Sites. Raw data. Copy of ZPCT II SUPPORTED SITES_ Jun 2011_22 07 2011 (2). ZPCT II Zambia Prevention, Care and Treatment Partnership: Monitoring and Evaluation Plan. Rep. 2010. Print. ZPCT II Zambia Prevention, Care and Treatment Partnership Quarterly Progress Report April 1 - June 30, 2010. Rep. 2010. Print. ZPCT II Zambia Prevention, Care and Treatment Partnership Quarterly Progress Report January 1 - March 31, 2010. Rep. 2010. Print. ZPCT II Zambia Prevention, Care and Treatment Partnership Quarterly Progress Report January 1 - March 31, 2011. Rep. Print. ZPCT II Zambia Prevention, Care and Treatment Partnership Quarterly Progress Report January 1- March 31, 2010. Rep. Print. ZPCT II Zambia Prevention, Care and Treatment Partnership Quarterly Progress Report January Progress Report January 1 - March 31, 2011. Rep. 2011. Print. ZPCT II Zambia Prevention, Care and Treatment Partnership Quarterly Progress Report July 1 - September 30, 2011. Rep. Print. ZPCT II Zambia Prevention, Care and Treatment Partnership Quarterly Progress Report October 1 - December 31, 2010. Rep. Print. ZPCT II Zambia Prevention, Care and Treatment Partnership Quarterly Progress Report October 1 - December 31, 2011. Rep. 2012. Print. ZPCT II Zambia Prevention, Care and Treatment Partnership: Work Plan Amendment for Year One, Period: June 1, 2009 - May 31, 2010. Rep. Print. ZPCT Quarterly Report (April - June 2009). Rep. Print. ZPCT/Zambia Prevention, Care and Treatment Partnership: Costing Analysis of HIV Services in ZPCT Supported Provinces. Raw data. 61 ANNEX E LIST OF KEY INFORMANT INTERVIEWS S/n Name Position Facility/Organization District 1 George M. Chigali Programmed Manger ZPCT II Kabwe 2 Mwiche Gideons SCCO ZPCT II Kabwe 3 Francis Mwema PTA (Technical Advisor) ZPCT II Kabwe 4 Ndalaweta Patrick Snr. Finance & Administration Officer ZPCT II Kabwe 5 Kampamba Elias Associate Finance Officer ZPCT II Kabwe 6 Nicky Mapulanga QA/QI Officer ZPCT II Kabwe 7 Alpha Banda Snr. Program Officer ZPCT II Kabwe 8 Monde Sooka Program Officer ZPCT II Kabwe 9 Benson Chisha MCPTO ZPCT II Kabwe 10 Phyllis Kanyembo PMC ZPCT II Kabwe 11 Sitenge Gift Snr M&E Officer ZPCT II Kabwe 12 Mudala James M&E Officer ZPCT II Kabwe 13 Violet W. Kunda Snr. PMTCT/CT ZPCT II Kabwe 14 Bernard Manda Srn. Pharmacy & Lab Officer ZPCT II Kabwe 15 Dr. Peter Mwaba Permanent Secretary MOH Lusaka 16 Dr. G. Syakantu Director of Clinical Services MOH Lusaka 17 Dr. Elizabeth Chizema Director of Public Health MOH Lusaka 18 Dr. Max Bweupe PMTCT Specialist MOH Lusaka 19 Mr. Mike Walsh Chief of Party FHI Lusaka 20 Mr. Andrew Mlewa Deputy Chief of Party FHI Lusaka 21 Mr. Sitwala Mungunda Program Manager Care Lusaka 22 Mr. Jay Goulden Assistant Country Director Program Care Lusaka 22 Mr. Japhet Kakoma HIV Prevention Manager CHAZ Lusaka 23 Dr. Richard Nsakayana Senior Advisor Capacity Building CORDNO-EMG Lusaka 24 Ms. Lumba Kalumba KCTT Lusaka 25 Ms. Gail Bryan-Mofya Senior Advisor Pharmaceutical MSH Lusaka 26 Ms. Silvia G. Bonilla Program Development Specialist Social Impact Arlington, Virgina 27 Dr. W. Mumba, Acting PMO PMO Northwestern 28 Dr. George Liabwa, PMO PMO Northwestern 29 Dr. Harrison N'guni, DMO Northwestern 62 Solwezi 30 Mr. Norbert Saihemba, Project Manager ZPCT II Provincial Team - Northwestern ZPCT II Solwezi 31 Dr. Matthew N'gambi, Health Facility 1 Solwezi General. Hospital Solwezi 32 Mr. Dan Chikonde, ART Specialist Health Facility Solwezi Urban Clinic Solwezi 33 Mr. Christopher Kaole, In chargeHealth Facility 1 &2 - Northwestern Nselauke Clinic, Kasempa 34 Mr. Daman Kiboko, Health Advisory Committees Northwestern Coordinator Solwezi 35 Agnes (?), Health Advisory Committees Member Kasempa 36 Mr. Emmanuel Ndalameta Health Advisory Committees Northwestern Member Kasempa 37 Mr. Brian Sweta PATF/DATF Northern PACA Solwezi 38 Mr. Sawomba Dunstan, PATF/DATF Northwestern DACA Solwezi 39 Mr. Ronald Kapesha, PATF/DATF DACA Kasempa 40 Dr. Jairos Mwami, Mukinge Mission Hospital (NWP) Executive Director Kasempa 41 Mr. Charlton Sulwe Provincial Program Officer ZPCT II Ndola 42 Dr. Dariot Mumba Provincial Technical Advisor ZPCT II Ndola 43 Dr. Harrison Ng’uni District Medical Officer ZPCT II Solwezi S/n Name Position Facility/Organization District 1 George M. Chigali Programmed Manger ZPCT II Kabwe 2 Mwiche Gideons SCCO ZPCT II Kabwe 3 Francis Mwema PTA (Technical Advisor) ZPCT II Kabwe 4 Ndalaweta Patrick Snr. Finance & Administration Officer ZPCT II Kabwe 5 Kampamba Elias Associate Finance Officer ZPCT II Kabwe 6 Nicky Mapulanga QA/QI Officer ZPCT II Kabwe 7 Alpha Banda Snr. Program Officer ZPCT II Kabwe 8 Monde Sooka Program Officer ZPCT II Kabwe 9 Benson Chisha MCPTO ZPCT II Kabwe 10 Phyllis Kanyembo PMC ZPCT II Kabwe 11 Sitenge Gift Snr M&E Officer ZPCT II Kabwe 12 Mudala James M&E Officer ZPCT II Kabwe 13 Violet W. Kunda Snr. PMTCT/CT ZPCT II Kabwe 14 Bernard Manda Srn. Pharmacy & Lab Officer ZPCT II Kabwe 15 Dr. Peter Mwaba Permanent Secretary MOH Lusaka 16 Dr. G. Syakantu Director of Clinical Services MOH Lusaka 17 Dr. Elizabeth Chizema Director of Public Health MOH Lusaka 18 Dr. Max Bweupe PMTCT Specialist MOH Lusaka 63 19 Mr. Mike Walsh Chief of Party FHI Lusaka 20 Mr. Andrew Mlewa Deputy Chief of Party FHI Lusaka 21 Mr. Sitwala Mungunda Program Manager Care Lusaka 22 Mr. Jay Goulden Assistant Country Director Program Care Lusaka 22 Mr. Japhet Kakoma HIV Prevention Manager CHAZ Lusaka 23 Dr. Richard Nsakayana Senior Advisor Capacity Building CORDNO-EMG Lusaka 24 Ms. Lumba Kalumba KCTT Lusaka 25 Ms. Gail Bryan-Mofya Senior Advisor Pharmaceutical MSH Lusaka 26 Ms. Silvia G. Bonilla Program Development Specialist Social Impact Arlington, Virgina 27 Dr. W. Mumba, Acting PMO PMO Northwestern 28 Dr. George Liabwa, PMO PMO Northwestern 29 Dr. Harrison N'guni, Solwezi DMO Northwestern 30 Mr. Norbert Saihemba, Project Manager ZPCT II Provincial Team - Northwestern ZPCT II Solwezi 31 Dr. Matthew N'gambi, Health Facility 1 Solwezi General. Hospital Solwezi 32 Mr. Dan Chikonde, ART Specialist Health Facility Solwezi Urban Clinic Solwezi 33 Mr. Christopher Kaole, In chargeHealth Facility 1 &2 - Northwestern Nselauke Clinic, Kasempa 34 Mr. Daman Kiboko, Health Advisory Committees Northwestern Coordinator Solwezi 35 Agnes (?), Health Advisory Committees Member Kasempa 36 Mr. Emmanuel Ndalameta Health Advisory Committees Northwestern Member Kasempa 37 Mr. Brian Sweta PATF/DATF Northern PACA Solwezi 38 Mr. Sawomba Dunstan, PATF/DATF Northwestern DACA Solwezi 39 Mr. Ronald Kapesha, PATF/DATF DACA Kasempa 40 Dr. Jairos Mwami, Mukinge Mission Hospital (NWP) Executive Director Kasempa 41 Mr. Charlton Sulwe Provincial Program Officer ZPCT II Ndola 42 Dr. Dariot Mumba Provincial Technical Advisor ZPCT II Ndola 43 Dr. Harrison Ng’uni District Medical Officer ZPCT II Solwezi 64 ANNEX F CLIENT EXIT INTERVIEW QUESTIONNAIRE Date: ____/___/2012 District:_______________Province:___________ Interviewer initials: ________ Facility: ___________________________________ A Client Profile Circle one in each category Male..........................................................................1 Female......................................................................2 Age range Below 18...........................................................................1 19 - 29..............................................................................2 30 - 39..............................................................................3 40- 49................................................................................4 50+ ...................................................................................5 B Mark the clinical service area where client was identified for this interview Circle ON COUNSELING AND TESTING …………………… ANTENATAL CARE ……………..…………………… ADULT ART …………………………….……………… ADULT CARE AND SUPPORT … …………………… PEDIATRIC ART ……….……………………………… PEDIATRIC CARE AND SUPPORT ………………… MALE CIRCUMCISION ……………………………… TB/HIV …………….…………………………………… LAB OR PHARMACY SUPPORT …………………… 1 What services have you received here? Please read each of these and circle ALL that ap a) COUNSELING AND TESTING ………………… b) ANTENATAL CARE ……………..………………… c) ADULT ART ………………………………………… d) ADULT CARE AND SUPPORT … ………………… e) PEDIATRIC ART ……….…………………………… f) PEDIATRIC CARE AND SUPPORT ………………… g) MALE CIRCUMCISION …………………………… h) TB/HIV …………….………………………………… i) LAB OR PHARMACY SUPPORT ………………… j) LINK TO COMMUNITY HEALTH SERVICES …..10 2 How long did you wait between the time you arrived at this facility and the time you were able to see a provider for the consultation? Minutes ..................?...........................................................1 Saw provider immediately....................................................2 Don’t know........................................................................ 3 Now I am going to ask about some common concerns clients have at health facilities. As I mention each one, please tell me whether any of these concerns affected you today, and need improvement. 3 Time you waited to receive a service Major Minor No problem Don’t know 1 2 3 8 65 4 Amount of explanation you received about the problem or treatment Major Minor No problem Don’t know 1 2 3 8 5 Privacy during your consultation? Major Minor No problem Don’t know 1 2 3 8 6 Confidentially about your case? Major Minor No problem Don’t know 1 2 3 8 7 Availability of medicines at this facility? Major Minor No problem Don’t know 1 2 3 8 8 The cleanliness of the facility? Major Minor No problem Don’t know 1 2 3 8 9 How the staff treated you? Major Minor No problem Don’t know 1 2 3 8 10 Ease of receiving other services when referred to another department in the health facility or to another facility outside? Major Minor No problem Don’t know 1 2 3 8 11 Overall, how would you rate the quality of the health services you have received here? EXCELLENT .……………………………………..… GOOD ………..……………………………………..… FAIR ..………..……………………………………..… POOR ..………..……………………………………..… UNACCEPTABLE ………………………………..… 66 ANNEX G MEETINGS ON QUALITY OF HIV/AIDS SERVICES IN ONE PROVINCE ZPCT II Mid-Term Evaluation COMMUNITY FOCUS GROUP GUIDE Date: Interviewer initials: District: Province: Group: Group: Community: Venue: Languages used: Instructions: Introduce yourself to the group and provide an overview on the purpose of the session. Ask participants if they wish to continue participating or opt out of the meeting. Facilitate self-introductions—each person can explain how they wish to be addressed. Facilitate ground rules for the meeting and agree on break times if needed. Script: I would like us to discuss where you go for health care and the experiences you have had with service provision. I will use the term health facility and it includes health posts, clinics, and hospitals. 4. 1. What particular health facility services are you receiving that are related to your HIV/AIDS status? From traditional healers? Why? Probe: Ask how many are receiving ART, treatment for TB and/or other OI, CT, MC, PMTCT, laboratory tests, other and from which type of provider. On probe: 5. 2. What factors influence your choice of where to go for services? 3. How satisfied are you with the way you are treated at the health facility? On Probes  As you use the health services, do you feel your rights are respected? Why? Please give examples of instances in which you feel you were not respected. 6.  In your opinion, is there a difference in the way men and women are treated in the HIV/AIDS services?  Have you been involved in decisions about your services? Have the services been well explained?  If there are misunderstandings or questions, how do you get this resolved?  Do the staff listen to your opinions and concerns?  Have you changed facilities for your services? Why? (Probe: location, range of services offered, hours of availability) 4. How do you rate the service(s) and delivery by the staff? 67 a) Are you receiving what you need to receive—such as drugs, information, treatment for infections, laboratory work such as CD 4 count, other? If not, what are those things you are not receiving?  How do you rate the services? Excellent / Good / Fair / Poor / Unacceptable? In the table below please write down the number of people for each rating category: Excellent Good Fair Poor Unacceptable Services 4 3 Delivery by staff 43 10.  Please explain your ratings Opinions on Needed Improvement(s) in the Health Facility We have discussed your experiences at the health facility. Let us now discuss your opinions about how health services can be improved. 5 What should be improved about HIV/AIDS related service delivery in health facilities? 11. On Probe  Are you comfortable with the present level of confidentiality?  Would you feel more comfortable being seen by a health worker of the same sex as yourself?  Would you prefer to be seen by an older or younger health worker?  Other improvements you would like to see? 6 Have you ever been referred for HIV/AIDS services in another health facility? Have you ever been referred for HIV/AIDS service by a community provider? Did you go for this referral? 7 How easy was it for you to identify whom to see? What to do? 8 For those of you on ART, can you describe what care you received and how often before you started on ART? Community based services Let us shift our focus of discussion to look at community-based services. 12. 9 How many of you receive services from community health volunteers? 68 10 For those who have received services from CHV, how satisfied are you with services provided by CHVs? [Probe: Are the hours of operation, professionalism, quality of care, confidentiality, location, ease of interaction? 11 How can the services be improved at the community level? 12 ZPCT has been helping to improve HIV and AIDS services in this district, what would you consider to be new technologies/approaches? SMS messages? Smart cards? 13 Can you tell us how you have used the new approach/innovation? Has it helped you to access services more easily? Sustainability ZPCT currently supports the health sector to improve its services. If the project ends, how can the community support continuation of the services? Thank you for your participation in the focus group discussion. Starting time of discussion: End: 69 ANNEX H MEETINGS ON QUALITY OF HIV/AIDS SERVICES IN ONE PROVINCE Indicator Life of project (LOP) Percentage Achievement Comments (overachieved= over 100%, underachieved= >60%, On target = 60% to 80%, Achieved =100% Targets (Aug 09 - May 14) Status Now (as of Jun 12) 1.1 Counseling and Testing (Projections from ZPCT service statistics) Service outlets providing CT according to national or international standards 370 389 105.1 Overachieved: The project was supposed to reach 222 sites (60% of 370) therefore, the LOP target was low since the project started with 219 sites from ZPCT I. The project reached 170 sites instead of 150 to reach LOP target Individuals who received HIV/AIDS CT and received their test results 728,000 1,316,398 180.8 Overachieved: The project was supposed to reach 436,800 individuals (60% of LOP target). The project overachieved by 588,398 against LOP target. Individuals who received HIV/AIDS CT and received their test results (including PMTCT) 1,300,000 1,888,458 145.3 Overachieved: They were to reach 780,000 individuals (60% of LOP target). They reached 588,398 more than intended LOP target Individuals trained in CT according to national or international standards 2,316 1,413 61.0 On target. Although the training target is on target, this is a process indicator which should be reached at the begging of the project. The target set was too high considering the time shortage of staff in MOH. 1.2 Prevention of Mother-to-Child Transmission (Projections from ZPCT service statistics) Service outlets providing the minimum package of PMTCT services 359 376 104.7 Overachieved: They were to reach 215 sites (60% of LOP target) They overachieved by 17 sites against LOP targets. Pregnant women who received HIV/AIDS CT for PMTCT and received their test results 572,000 572,060 100.0 Target reached: This indicates that the project will be able to overachieve this target by the end of project. They were supposed to reach 343,200 (60% of LOP target). HIV-infected pregnant women who received antiretroviral prophylaxis for PMTCT in a PMTCT setting 72,000 60,987 84.7 On target: The project will be able to reach LOP target in the remaining period to project end Health workers trained in the provision of PMTCT services according to national or international standards 5,325 3,097 58.2 Underachieved: The project is supposed to train over 3,200 HWs. This is a process indicator which should be reached at the beginning of the project. The target set was too high considering the shortage of staff to train in MOH. 1.3 Treatment Services and Basic Health Care and Support (Projections from ZPCT service statistics) Service outlets providing HIV-related palliative care (excluding TB/HIV) 370 389 105.1 Overachieved: The project was supposed to reach 222 sites (60% of 370) therefore, the LOP target was low since the project started with 219 sites from ZPCT I. The project reached 170 sites instead of 150 to reach LOP target Individuals provided with HIV-related palliative care (excluding TB/HIV) (adults and children) 560,000 246,335 44.0 Underachieved: They are supposed to reach 336,000 individuals (60% of LOP target). The target was set too high Pediatrics provided with HIV-related palliative care (excluding TB/HIV) 60,000 19,377 32.3 Underachieved: They are supposed to reach 36,000 children. (60% of LOP 70 Indicator Life of project (LOP) Percentage Achievement Comments (overachieved= over 100%, underachieved= >60%, On target = 60% to 80%, Achieved =100% Targets (Aug 09 - May 14) Status Now (as of Jun 12) target) The target was set too high. Individuals trained to provide HIV palliative care (excluding TB/HIV) 3,120 1,759 56.4 Underachieved: The project is supposed to train over 1,800 individuals. This is a process indicator which should be reached in the first year of the project. The target was set too high. Service outlets providing ART 130 148 113.8 Overachieved. The target set was too low considering that the project was scaling up from where ZPCTI left off. Individuals newly initiating on ART during the reporting period 115,250 89,346 77.5 On target Pediatrics newly initiating on ART during the reporting period 11,250 6,846 60.9 On target Individuals receiving ART at the end of the period 146,000 155,473 106.5 Overachieved: The target set was too low considering that the project was scaling up from ZPCT I. The targets should be revised based on available resources and capacity. Pediatrics receiving ART at the end of the period 11,700 10,764 92.0 Overachieved: The target set was too low considering that the project was scaling up from ZPCT I. The targets should be revised based on available resources and capacity. Health workers trained to deliver ART services according to national or international standards 3,120 1,759 56.4 Underachieved: The project is supposed to train over 1,800 individuals. This is a process indicator which should be reached in the first year of the project. The target was set too high without considering the shortage of staff to train TB/HIV Service outlets providing treatment for TB to HIV+ individuals (diagnosed or presumed) in a palliative care setting 370 389 105.1 Overachieved: The project was supposed to reach 222 sites (60% of 370) therefore, the LOP target was low since the project started with 219 sites from ZPCT I. The project reached 170 sites instead of 150 to reach LOP target HIV+ clients attending HIV care/treatment services that are receiving treatment for TB 17,000 15,829 93.1 Overachieved: The target set was too low considering that the project was scaling up from previous project. Individuals trained to provide treatment for TB to HIV+ individuals (diagnosed or presumed) 3,120 1,759 56.4 Underachieved: The project is supposed to train over 1,800 individuals. This is a process indicator which should be reached early in the project to support quality of activities. The target was set too high without considering the shortage of staff to train Registered TB patients who received HIV/AIDS CT and their test results at a USG-supported TB service outlet 30,400 25,351 83.4 On target 1.4 Male Circumcision (ZPCT II projections) Service outlets providing MC services 50 44 88.0 On target Individuals trained to provide MC services 260 291 111.9 Overachieved. Annual targets needs to be adjusted upwards based on available resources Number of males circumcised as part of the minimum package of MC for HIV prevention services 8,000 (2012 annual target) 7,940 99.2 Overachieved. Annual targets needs to be adjusted upwards based on available resources and also considering that Northwestern province has higher rates of MC 2.1 Laboratory Support (Projections from ZPCT service statistics) Laboratories with capacity to perform: 111 113 101.8 Achieved 71 Indicator Life of project (LOP) Percentage Achievement Comments (overachieved= over 100%, underachieved= >60%, On target = 60% to 80%, Achieved =100% Targets (Aug 09 - May 14) Status Now (as of Jun 12) (a) HIV tests and (b) CD4 tests and/or lymphocyte tests Laboratories with capacity to perform clinical laboratory tests N/A 140 N/A N/.A Individuals trained in the provision of laboratory-related activities 375 725 193.3 Over achieved. The targets set were too low, they need to be adjusted upwards. Tests performed at USG-supported laboratories during the reporting period: (a) HIV testing, (b) TB diagnostics, (c) syphilis testing, and (d) HIV/AIDS disease monitoring 3,813,000 3,850,122 101.0 Achieved 2.2 Capacity Building for Community Volunteers (Projections from ZPCT service statistics) Community/lay persons trained in counseling and testing according to national or international standards (excluding TB) 2,506 1,378 54.8 Underachieved. The training needs to be accelerated for lay persons Community/lay persons trained in the provision of PMTCT services according to national or international standards 1,425 1,002 70.2 On target Community/lay persons trained in the provision of ART adherence counseling services according to national or international standards 600 609 101.7 Achieved 4 Capacity Building for PHOs and DHOs (ZPCT II projections) Local organizations (PMOs and DMOs) provided with technical assistance for HIV-related institutional capacity building 47 47 100.0 Achieved 5 Public-Private Partnerships (ZPCT II projections) Private health facilities providing HIV/AIDS services 30 18 60.0 On target: The signing of MOUs with remaining private facilities should be accelerated to reach the LOP target. Gender Number of pregnant women receiving PMTCT services with partner N/A 44,225 N/A No. of individuals who received testing and counseling services for HIV and received their test results (tested as couples) N/A 92,412 N/A 72 ANNEX I MEETINGS ON QUALITY OF HIV/AIDS SERVICES IN ONE PROVINCE Meeting name Purpose Frequency Participants ZPCT II level Data Review Meetings Review performance of the ZPCT II supported facilities on a monthly basis. These address among other things:  Gaps identified in the data  Trends in services statistics  Performance of the project in relation to specific project strategies  Issues requiring follow-ups at facility level  Priority areas for the next field visits requiring technical support  Any other requested data by technical staff that addresses key performance areas. e.g. uptake of positive clients in ART service and ART retention rate Monthly · ZPCT II staff QA/QI meeting at ZPCT II office To review QA/QI report before it is send to Lusaka: issues discussed  Problem areas identified and prioritized  Specific facilities affected by district  Specific remedial action planned and persons responsible  Action already taken in previous quarter  Targets set to solve problems identified  Time line (deadlines) to remedy the gaps identified Quarterly · ZPCT II staff Facility level ART review meeting ART data review and other issues Monthly · Facility staff, ZPCT II Performance assessment The facility in-charge are supposed to do an internal assessment before they are assessed by the DMO Twice a year · Facility staff, District level Quarterly review meeting Review district performance (may include review of ART , PMTCT, CT data—ZPCT II often presents such data aggregated for district) Quarterly · ZPCT II, DMO, DHO facility in￾charges for the respective district under review Performance assessment The assessment covers the performance of a facility in all the service areas including HIV related services. Specific issues that are covered can be obtained from the DMO or PMO Twice a year · DMO 73 ANNEX J CLIENT EXIT INTERVIEW SUMMARY Client exit interviews were conducted in the five provinces. While the survey protocol indicated that research assistants were to interview one client in each service (i.e. MC, PMTCT, CT, TB/HIV and ART), the execution of this design proved to be more challenging as certain services were offered only on certain days at individual sites, and research assistants were also scheduling these along with focus group appointments. Seventy-eight clients from 16 health centers and hospitals in 10 districts were interviewed. Clients from Central, Northern and Luapula provinces were represented with 19-21 clients each, while there were 6 interviews in NW province and 12 in Copperbelt. The clients were mostly women (63%), 41% were between the ages of 19-29 and 22% were between the ages of 40-49. About one-fifth were identified at counseling and testing, another fifth were identified at ANC, and one out of four were identified at ART. Of these, 96% had received counseling and testing (including through PMTCT) at some point at the facility. The survey asked clients about their experiences at the clinic on that day. Out of the 78 clients, 76% rated the quality of health services excellent or good, and 24% rated the quality as fair. The table below summarizes the main survey findings: Concerns that affected you to-day and need improvement Percent responding (N=78) Confidentiality 10% Availability of medicine at this facility 21% Cleanliness of the facility 33% Treatment by staff 15% Ease of receiving other services when referred to another department in the health facility or to another facility 8% 74 ANNEX K SELECTED NOTES FROM FGDS–SERVICE COVERAGE Services Notes from FGD discussions General lack of access “The other problem why we are not very satisfied is that when its 11:00hrs, they stop giving out files and if you come after 11:00hrs, they ask you to come the following day and they say that on Fridays they don’t give ARVs and if the drugs runs on Thursday you couldn’t go to refill and when you on Friday they will tell you to come and get the drugs on Monday.” “When getting cards or files, they will check the cards and tell you that you should come six days before your drugs runs out, and when you come back for refill, they will start shouting at you saying “why have you come early before your days” and when you try to reason with them to say that this is when I have a chance and these other days I might not be around, that’s why I have come early before six days expires and you told me to come six days before, now what’s the problem? So they take us for fools” ARVs stock outs: “[…] there are times when ART drugs are rationed and not everything is adequate at the facility.” “There is need to support some surrounding clinics in terms of access to ART because Kapiri urban is heavily over crowed and the ratio of the health worker to patient is almost unachievable to provide quality health care. […] a way should be found to give adherence counselors an allowance because this will encourage them to take their work seriously.” “There should some rotation of health workers in clinics and that there should be many ART clinics in the area.” Inadequate CD4s “There is no CD4 count and there is no information that is provided how to use the medicine, the clinical officer just administers the drugs …we would like to see a new infrastructure building meant for patients on ART and there should be an increase in health personnel” “[….] the services are poor because we have to wait for long hours in order to be attended to. For instance only 30 people can access the CD4 machine a day and this makes patients spend nights at the health facility so that they can meet the 30 patient bench mark” 75 ANNEX L. FOCUS GROUPS – BENEFICIARY PERCEPTIONS Positive FGD comments Negative FGD comments Hours of operation “They do work long hours” “The services and their delivery could be said to be excellent and good said the participants. (Despite long waiting time,) participants still appreciated the services and the manner in which they are done which was the most important thing to them.” “The other reason of long queues, health workers report for work late, they come 09:00 or 09:30hrs when they are supposed to report at 08:00hrs” Professionalism of staff “[…] there are some nurses that are good hearted and do their work well” “We are happy because the people we find there are more knowledgeable than us they help us very well. The goodness that I have found is that they check, my scale CD4 count so that help me to know how my health is. The other issue is that they encourage us and give us more information on things we didn’t know.” “In my case am very happy because I was not even expecting to even have a child and I was very sick but now am health and I have a child who is negative. -Even me I was among the first people who were positive and by then, they were not giving ARVS here and I was told to go to Copperbelt and that’s were I started the treatment and I now have a child who is now two years old.” As for me am satisfied I have never find any problem. I am also satisfied the way the staff work, it is good. The respondents said that people are well received when they go for health services. Heath care providers were complimented of being very confidential and helpful. They said that they are satisfied to a greater margin because there is still room for improvement and make their services more satisfying beyond the current state. “Some time people attending to them (nurses) have a bad attitude to HIV/AIDS positive clients to the extent that patients get shouted at and sometimes cards are just thrown at them. Nurses, not male ones as such, say that it was not them that made them to be what they are. There was unanimous consensus that they are not treated like human beings at times-something they desire should be improved. This feeling among the participants made them say they are not satisfied with the way they are treated at their respective health facilities especially Solwezi Urban Clinic.” “The participants all agreed that they treatment they receive is harsh by the medical health workers” “ZPCT should provide their medical staff because the government health workers have bad attitude towards work” The Participants all complained that they health workers are rude and are not properly attended to Participants all said that they felt that there rights were not respected because the health workers do not pay attention to the patients and that they are at the mercy of the Health workers in terms of accessing medical care. “Health workers spend most of the time chatting instead of attending to patients” “Participants all agreed that there has never been a chance for them to ask questions or be head on any query they may have as the medical staff are usually aggressive and hence they just tend to be frustrated. “[…] health care givers are harsh to them especially females.” “[…] felt that their rights were not well respected because patients are not received with care. Others are shouted at and are asked question in the language they do not understand. Nurses do not respect patients and speak about patients’ statuses in public domains.” “Sometimes they are shouting at patients instead of them being happy but they are just shouting how do you take your medicine and you the one who is shouting at you is very young that does not look well” “We go to the clinic happy but our coming back is full of complaints. Since we are on an emergency such that when our supplies finishes, they should find all every means to get us the 76 Positive FGD comments Negative FGD comments drugs so that we can take the drugs because we shouldn’t miss even a day or they are killing me.” “Another issue is favoritism, where a person they know will come, that person will be served first even if that very person found you already on the queue, this also creates conflicts” “There is one, for example you have to collect medicines, I don’t know if they have moods, they will just shouting at you that time has gone and you should go back and come the next day, but when you check the time, it could be around 15:00hrs and its not yet time to knock off and maybe that very day that is the last day your drugs are finishing. You could have also started off from home very early in the morning. -You could come to the clinic as early as 08:00hrs and they will tell you that you will collect your drugs at 14:00hrs and those who were behind you will be given the drugs. “ Quality of care “People who used to be very sick when they first came are now looking better and others are still coming to get treatment “ “There was a generally consensus that the staff try their best and within their means they have tried to care for the patients.” “The services are good, said the participants. Without the services and the good manner in which they are delivered, many could be suffering. They added that at times, food supplements are given to them at no notice. The services are good and their delivery is good and without their work, many people would have died. ZPCTII is really helping in helping people to lead well informed lives.” “They group members agreed that staff delivery is way below expectations and there is need of some urgent intervention.” “It was generally agreed that the services are not good because the Health workers attitudes are bad and they treat the patients as if they were not human beings.” “They participants generally agreed that the delivery of they staff was bad and they attributed the situation to the low staffing levels at the facility. “ “Participant complained that they have never been involved in decision making and that they services they receive are not adequately explained because the medical staff tends to be too busy dealing with various patients” “ZPCT should also try and cater services in the surrounding clinic because there is a lot of congestion at the clinic” Confidentiality “By and large, they said that there was no problem with the level of confidentiality and were of the view that many times, information about their status is leaked by themselves and the people who attend to them in hospitals or clinics.” “It was all agreed that there is no confidentiality because the facility they are using is too congested as patients on ART and general health care are mixed and being attended to by one clinical officer.” “By and large, they said that there was a problem with the level of confidentiality and that at times, individuals are told in public places (road side or market) by health care givers to go to the health facility to get drugs. It was sad to note that at times, they participants felt coerced to by health care giver to undergo reviews concerning their statuses. They clarified that the manner in which they are told to them sounds like it is a forcing matter.” “By and large, they said that there was a problem with the level of confidentiality. Participants said that health care givers do not keep secret the information they share with them.” “They is no confidentiality, I have heard staffs telling people that such a person is suffering from AIDS even going to the extent telling the relatives that has AIDS so don’t waste your time that one is died.” “[…] everyone knows that these people all have AIDS, it is just along the path and everyone pass through going wherever they are 77 Positive FGD comments Negative FGD comments going. So in this area there is no confidentiality” “Participants collectively were of the view that the record cards are too big and as a result, clients are carrying big bags (like lap top bags) to hide their cards in. The arrangement of places where to be attended are too far apart such that you get tired of moving from one place to another for you to finish being attended to. This feeling among the participants made them say they are not satisfied with the way they are treated at their respective health facilities adding that some times, they have to wait up to 3 hours for them to be attended to. 78 ANNEX M FOCUS GROUPS – SERVICES TO BE IMPROVED AT COMMUNITY LEVEL Frequency Support for better community health volunteers Provide allowances to community volunteers – before with allowances they were much more attentive; also there is some corruption by CHV (fees); they provide an important service; they need to buy food/care for families. 8 Provide bicycles for transportation to community volunteers 9 Increase the number of volunteers so that more will come to the community; more volunteers are needed because of the increasing number of cases. Bring back the trained Community Counselors into the communities 8 Refresher courses are needed for CHV and Counselors; how to check CD4 count; health in general, doing community sensitizing 9 Provide CHV with stationary and test kits; computers to print out instructions, referrals 3 Use HIV+ people as CHV. HIV- people tend to talk too much in the community, These volunteers would need training in ART, etc. 3 Community awareness raising Neighborhood Health Committees should be trained in HIV issues; teachings and sensitization in the community; new CHV speakers should come to these meetings, sex education for all, village leaders should be made aware to reduce discrimination 10 Pastors need to be trained and HIV explained – they have faith healing approaches and are ignorant 1 Couples counseling is needed because some spouses do not allow the spouse to participate; to increase awareness in the community; whole family counseling 3 Literature translated to local languages 1 Community support for income generation Business training for income to buy more food 2 Formation of clubs (entertainment) so that we can make money 2 Agricultural and farming training, or milling to produce good food and to generate income to help pay for the transportation of those too poor 3 Improving clinics Provide ARVs to those that are positive somewhere nearer to them like a mobile clinic kind of set up 1 There should be CD4 count machines at clinics administering ART 1 Health centers closer to the community 1 79 ANNEX N TRAINING BY COURSE, SUB-COURSE AND PROVINCE ZPCT II TRAINING ACCOMPLISHMENTS 80 Annex O TASK III: CROSS-CUTTING PRINCIPLES Alignment The MOH is ZPCT II’s main partner, thus the project is expected to work within the context of MOH’s national policy and strategic plan. However, the NAC, through its national policy and strategic framework, articulates national priorities for the HIV and AIDS response. Both MOH and NAC support the “three ones” approach: one national coordinating authority, one strategic framework, and one monitoring and evaluation framework. The principles outlined in the ZPCT II RFTOP SOW are aligned with the guiding principles articulated in NAC’s strategic framework to HIV/AIDS. The SOW for ZPCT II was drafted in 2009, and is based on objectives and priorities identified in the Zambian National HIV/AIDS Strategic Framework (2006-20l0). As Table 9 indicates there is very little difference between the priorities identified in the 2006-2010 National HIV/AIDS Strategic Framework and the more recent MOH National Health Strategic Plan 2011-2015 and NAC National HIV/AIDS Strategic Framework 2011-2015. The USG/GRZ HIV/AIDS Partnership Framework was designed to align with the most recent MOH and NAC national HIV/AIDS strategies. Table 9. Comparison of Organizational Health Strategies Ministry of Health (MoH) Strategic Plan National HIV/AIDS/STI/TB Council Strategic Framework USG/Zambia Partnership Framework 2011-2015 2006 – 2010 2011-2015 2011-2015 Expand access to HIV/AIDS prevention services including male circumcision, PMTCT and provision of safe blood Intensify prevention Reduce the annual rate of new HIV infections Accelerate and intensify prevention in order to reduce the annual rate of new HIV infections Continue expanding ART services for both adults and children in both rural and urban areas Expand treatment, care and support Provide universal access to comprehensive and quality treatment, care and support for people living with HIV/AIDS, their caregivers and their families Mitigate the socio￾economic impact of HIV/AIDS especially among the most vulnerable groups, orphans and vulnerable children (OVC), PLHA and their caregivers and families. Strengthen the decentralized response and mainstreaming HIV/AIDS Mitigate the socio￾economic impacts of HIV/AIDS especially among the most vulnerable groups orphans, and vulnerable children, people living with HIV and their caregivers /families Strengthen the systems which underpin Zambia’s response to HIV and AIDS Improve the monitoring of the response Strengthen the capacity for a well-coordinated and sustainably managed Strengthen the capacity for a well￾coordinated and 81 Ministry of Health (MoH) Strategic Plan National HIV/AIDS/STI/TB Council Strategic Framework USG/Zambia Partnership Framework 2011-2015 2006 – 2010 2011-2015 2011-2015 HIV/AIDS multi￾sectorial response sustainably managed HIV/AIDS multi￾sectorial response Integrate advocacy and coordination of the multi￾sectorial response Partnerships See Tasks VIII and X for discussion on partnerships. Continuum of Care The HIV/AIDS continuum of care is a very broad spectrum of services that links prevention interventions with treatment, and nutritional, financial, legal, material and spiritual support on the care and support side. ZPCT II employs a cadre of community workers (Adherence Support Workers and Lay Community Counselors) to facilitate links between Information, Education and Communication (IEC), advocacy and counseling and testing activities taking place in the community (but only CT is directly supported by the ZPCT II) in facility based services. Clients are often referred to the nearest health facility or escorted by the HCW. Once in the clinical setting, the client is registered, tested, and directed to an appropriate stream of services if she or he is HIV positive. Samples for tests that cannot be completed at a particular health facility are sent to a nearby health clinic, equipped to carry out the required laboratory tests or to a referral hospital, which offers specialized services. A referral form is used to track referral of the sample to ensure that results are returned to the referring facility. Respondents in health facilities stated that the return of counter-referral forms is poor. Clients receiving treatment are followed-up in the community by the Adherence Support Worker (ASW) if they submitted correct information about their place of residence. The staff interviewed in the health facilities admitted that some people do not want their true identity known and will provide false information about their place of residency. This compromises the link between facility and client as it limits the potential for follow-up visits to the client’s home. The ZPCT II offers a range of services that stretch from prevention to treatment, but it is neither comprehensive nor all-inclusive. Prevention activities are limited to encouraging people to know their status and receive treatment if positive. Clients testing negative continue to receive general information about HIV at the community level, but they are not advised or assisted to participate in post-test clubs or receive specific information and motivation to encourage/support positive behavior change. Innovations See Task IV: Factors of success. Linkages The GRZ has embarked on a process of decentralizing decision-making and development initiatives to provincial and district-levels – a process guided by the National Decentralization Policy and National Decentralization Operational Plan. In the context of responding to the Zambian HIV/AIDS epidemic, the decentralization effort is aimed at devolving decision-making power and resources for HIV and AIDS to provincial, district and community-levels to facilitate ownership of the local epidemic and implementation of effective local responses in a sustainable way. 82 The ZPCT II is supporting the decentralization process by strengthening provincial and district health offices to govern, supervise, and mentor health facilities within their mandated areas of responsibility. Provincial level offices have been created to work closely with PHOs, DHOs and community structures to prepare HIV/AIDS action plans. Community plans are submitted to districts, where the local authority/district council combines them to form a district plan. The district plan is then submitted to the MOH and NAC, where they are amalgamated into national plans. MOH’s strategic response to HIV/AIDS is based on the analyses, priorities, targets and recommendations articulated in the National HIV/AIDS Strategic Framework. ZPCT II’s approach to prevention aligns with four of the six drivers to the HIV/AIDS epidemic in Zambia, identified in the national HIV/AIDS strategic framework as presented in Table 10. Table 10. ZPCT II alignment with the National HIV/AIDS Strategic Framework Drivers ZPCT II Interventions Multiple and concurrent sexual partners Counseling and testing Low and inconsistent condom use Counseling and testing Low levels of MC in most provinces MC Vertical transmission from mother to child PMTCT In addition, ZPCT II promotes treatment among positives and adherence to a strict regime as a prevention strategy; and addresses gender inequalities that perpetuate risk and vulnerability based on dominance of male interests and lack of self-assertiveness on the part of women. There is a disconnect between the local authority/council and the ZPCT II-supported DHO. Whereas ZPCT II helps the DHO, health facility, and surrounding communities to analyze what is driving the epidemic in the area, required interventions for addressing the drivers, realistic targets, and technical assistance required to address the problem, such assistance is not given to the local authority/district council. The local authority/district council is tasked to prioritize interventions from various sectors, draft budgets, and cut where they see fit. The local authority/district council could further benefit from technical assistance to see the big picture of HIV/AIDS Procurements See Tasks V and VI: Obstacles and Sustainability. Zambian Leadership and Human Resources Support The Zambian leadership development and human resource support provided by ZPCT II has shown results in the type of the staffing of ZPCT II and in the training and capacity development provided to partners by these staff to strengthen, in a sustainable manner, their projects and programs. Over 99 percent of ZPCT II staffing is composed of Zambians; some have been with ZPCT II since its inception and have a history of interactions with MOH, ZPCT II’s main partner. The staff has benefitted from in-service capacity building/leadership development to improve their knowledge and skills. Leadership development should include gender training to prepare staff to appropriately define and address gender issues that affect the prevention, treatment and care activities in HIV/AIDS. The 2011 gender audit by the USAID Office of Inspector General found the following deficiencies in ZPCT II’S gender strategy: 1) the strategy was incomplete; 2) it did not report progress on gender-related goals; 3) strategies to reduce gender-based biases were not being consistently implemented; 4) the ZPCT II supported sites were not proactively screening for signs of GBV; 5) few services to victims were available; 6) male involvement in PMTCT was not being encouraged and 7) couples counseling was not being done. Following these observations, ZPCT II undertook the following activities:  Formed a gender steering committee within the project (consisting of Unit heads) to promote accountability for gender integration at all levels of project implementation. 83  Revised system for collecting data against gender indicators for the purpose of reporting to USAID versus internal use for advocacy, and future planning and programming.  Developed training manuals for integrating HIV/AIDS into the service delivery areas.  Developed guidelines for screening GBV across all service delivery areas.  Established a database on GBV, which could be used for advocacy campaigns.  Began working on a referral system in health facilities to refer GBV cases for support services. Although some progress has been made, unfortunately, as time has passed the number of gender indicators reported by ZPCT II has been significantly reduced. A more complete discussion of gender initiatives in ZPCT II is included in Annex H. To date, 56 MOH and private sector staff have been trained in planning, 51 in governance, 10 in human resources, 10 in finance and 58 in gender. ZPCT II is working with Zambia Integrated Systems Strengthening Project (ZISSP) in strengthening PHO and DHO capacity to manage health care. Gender Integration HIV/AIDS affects all geographical areas, and ethnic and socio-economic groups in Zambia. However, available evidence shows that gender is a determinant to risk and vulnerability to infection. Gender inequalities that perpetuate the dominance of male interests and lack of self-assertiveness on the part of women in sexual relations put both men and women at risk. Factors influencing gender-related risk and vulnerability include the following:  Differential power relations between men and women at household and community levels that influence access to information, income, and services.  The socialization process predisposes women’s ability to negotiate for safe sex. Socialization processes include traditional and cultural initiation ceremonies where women are taught to accept that men can have multiple partners, and can initiate violence within a marital setting.  Social pressures on women to demonstrate their fertility, so they do not use condoms.  A cultural trend for inter-generational relationships between girls and much older men.  Gender-based violence as a form of social control, particularly for women. The possibility of GBV has been identified as a constraint to women accessing voluntary counseling and testing or obtaining information on their HIV status if tested in an antenatal clinic. The possibility of being physically abused by a husband/partner or other family member has been documented in the 2001 to 2002 DHS, which reports that 53% of the women participating in the sample reported having experienced physical mistreatment. Approximately 47% of women interviewed in the Demographic and Health Survey (2007) said they had been victims of gender-based violence at some point in their life. ZPCT II’s Gender Response ZPCT II incorporated gender into prevention and treatment strategies at the beginning of the project. However, the first gender strategy did not articulate how it would be integrated. An analysis of project documents by SI found that some indicators were not gender specific, difficult to track, too many, and not aligned to PEPFAR indicators. A SI Gender Specialist was engaged in 2010 to conduct an assessment report and develop a revised draft strategy. The revised indicators were aligned with the PEPFAR indicators (e.g. GBV and prevention) through couples counseling and testing. A 2011 gender audit by the USAID Office of Inspector General found the following deficiencies in the implementation of the ZPCT II’S gender strategy:  The strategy was incomplete.  The project did not report progress on gender-related goals. 84  Strategies to reduce gender-based biases and obstacles were not being consistently implemented at ZPCT II-supported facilities.  ZPCT II-supported clinics generally tried to refer patients to gender-based violence services if the patients disclosed abuse but were not proactively screening for signs of GBV  Few services were available to victims of gender-based violence.  Male involvement in prevention of mother to child transmission activities is encouraged but not mandatory, however HCWs at some ZPCT II-supported facilities required patients to bring their male partners as a pre-requisite to service.  Although couples counseling was considered to be a very useful strategy, there were no activities directed toward expanding the service. Following these observations, ZPCT II undertook the following activities:  Formed a gender steering committee within the project (consisting of Unit heads) to promote accountability for gender integration at all levels of project implementation.  Revised system for collecting data against gender indicators for the purpose of reporting to USAID versus internal use for advocacy, and future planning and programming.  Developed training manuals for integrating HIV/AIDS into the service delivery areas.  Developed guidelines for screening GBV across all service delivery areas.  Established a database on GBV, which could be used for advocacy campaigns.  Began working on a referral system in health facilities to refer GBV cases for support services. Gender Indicators ZPCT II collects data on two gender indicators for reporting to USAID are:  GBV: # of clients screened for GBV in CT, PMTCT, ART and Clinical care setting using the engendered CHC checklist; # of GBV survivors treated for their injuries; # of survivors of rape provided with PEP disaggregated by sex  # of GBV survivors provided with Emergency Contraception  Counseling and testing: # of couples who received HIV counseling and testing and received their test results in a CT site supported by ZPCT II Other gender-based indicators are used internally by ZPC II to track progress in integrating gender into project activities. The indicators are:  # of training manuals revised to include gender-based protocol and norms for service delivery  # of HCW and CBV trained on engendered training packages, disaggregated by technical programmatic area (PMTCT, CT, Clinical Care & ART, Capacity Building and GBV) disaggregated by sex  # of couples counseled on FP and accepting a contraceptive method  # of males/ females under 18 seeking HIV counseling and testing services  # of PMOs/ DMOs trained with the use of gender-sensitive Capacity Building training packages  # of individuals referred by community mobilization implementing agencies to HIV services (disaggregated by service and sex)  # of individuals referred from HIV health facility based-services to social support services (legal, shelter, 8`) disaggregated by sex  The Gender Specialist reported that data is being collected on these indicators to build a database for advocacy on GBV issues but among the documents provided to the project, there is no evidence clinical services are being engendered, data generated through SmartCare is consistently disaggregated to aid project planning and management, and gender advocacy is now taking place through community mobilization activities. 85 Training and Sensitization Workshops Budgets for gender training are included in the recipient agreements. Gender sensitization training was conducted for the following cadres during 2011-2012: Table 11. Summarized Information on Types of Gender Trainings and Categories of Participants Type of Training Province Gender Level Year Female Male Gender Sensitization Central, Copperbelt, Luapula, Northern, & Northwestern 11 4 PMOs & DMOs 2011 GBV TOT ? 7 12 ZPCT II Staff 2012 GBV for Health Workers at Facility Level Central, Copperbelt, Luapula, & Northern 33 44 XXX xxxx Gender Mainstreaming Lusaka Staff & Provincial Managers 5 12 ZPCT II Staff & Provincial Managers 2012 Gender Integration ZPCT II Lusaka & Provincial Staff Copperbelt, Luapula, Northern, & Northwestern 5 14 2011 Training in GBV for health facility workers is being carried out in the Northwestern Province during the first week in October 2012. TOT training and preparation of the directory for GBV referrals are provided to PHO staff, which is expected to institute a cascade system for district and facility level training. HCWs are expected to infuse gender into their work plans. Four gender training manuals have been developed by ZPCT II and used in the training process:  Addressing Gender Inequalities in ART Adherence: Adherence Support Workers Facilitator’s Manual  Generic Gender Module  Human Resource Management Workshop For Provincial And District Medical Office Personnel  Guidelines for Male Involvement in PMTCT Service In addition various training manuals developed by GRZ are adapted and gender integrated for gender project use. The TOT trainings are just being implemented mid-way through the project and the process of cascading down to facility level has not yet taken off. Clients are being screened for GBV at a cursory level following a list of questions in a checklist but due to the nature of GBV and associated stigma, few in clinical settings are admitting to being victims even when there is strong evidence to the contrary. In the facilities visited during fieldwork, clinical staff was asked if GBV is a problem in their area. Most said it did not happen in their community. This suggests that there really is no problem in spite of overwhelming evidence contrarily from the DHS (2007) or, alternatively, not enough time is given to identifying victims during the screening process because of time constraints. Referral Services for Victims of GBV The Referral Network was created to support the practice of abstinence, sexual partner reduction, condom use, male circumcision, PMTCT and Counseling and testing on the prevention side to medical care, psychosocial support, nutritional, financial, legal, material and spiritual support on the care and support side. Infusion of GBV into district directories is still at a nascent stage and is dependent on the District AIDS Task Force who coordinates directory development and dissemination. Sustainability 86 See Task VI: Sustainability. Environmental Considerations The ZPCT II has an Environmental Impact Mitigation Plan that is shared with partners. An environmental impact assessment was conducted at project start-up and is conducted at the beginning of support to a facility. The assessment is carried out in conjunction with the District Health Management Team (DHMT), who are also involved in selecting vendors to carry out renovations to facilities. The ZPCT II focal point person for environment is also the Infrastructure Support Program Officers and oversees infrastructure development in refurbishment projects. The dual role enables the Officer to ensure that renovations are consistent with the Environmental Impact Mitigation Plan. A desk review revealed that the status of the state of disposal of medical waste including incineration of sharps, laboratory and other waste did not change in any facility from August 2009 to June 2012. A number of rural facilities still lack running water, incinerators, and septic tanks/soak ways, which would facilitate proper disposal of medical waste. Facilities that use pits for waste disposal are encouraged to ensure appropriate depth, location and lining of pits with impervious polythene sheeting for disposal of lab waste. The evaluation team learnt that the ZPCT II Senior Infrastructure Support Officer attended the USAID environmental compliance training and engaged USAID on construction and provision of incinerators, as bio-medical disposal is becoming increasingly difficult in the ZPCT II supported health facilities. As of June 2012, no clarity or solution was provided so situation remains the same. However, ZPCT II was encouraged to approach their COR and Mission Environmental Officer (MEO) for further guidance. 87 ANNEX P CAPACITY-BUILDING TRAINING LOG ZPCT II Training Accomplishments This table presents the number of people trained for each type of course and sub-course, and is disaggregated by province. ZPCT II has trained a total of 11,850 Health Care Workers and Lay counselors. 1,766 were trained in ART/palliative care, 2,793 were trained in Counseling and testing, 1,042 in Laboratory management, 291 in male circumcision, 4,407 in PMTCT and the remaining in pharmacy. Course Sub Course Province Grand Total Central Copperbelt Luapula North Western Northern ART/Palliative Care ART/OIs (Standard 12days￾Inhouse 6 days) 111 419 108 95 115 848 ART/OIs￾Refresher 50 123 51 64 44 332 ART￾Pediatrics 85 260 77 76 88 586 ART/Palliative Care Total 246 802 236 235 247 1,766 Counseling and Testing Training Basic Child Counseling￾HCWs 30 57 46 45 45 223 Basic Child Counseling￾Lay 40 60 30 45 30 205 Basic CT￾HCWs 80 100 60 79 58 377 Basic CT-Lay 81 61 80 59 76 357 Counseling and Testing￾Refresher￾HCWs 41 60 39 39 68 247 Counseling and Testing￾Refresher-Lay 36 20 35 20 38 149 Couple Counseling￾HCWs 60 40 49 40 50 239 Couple Counseling￾Lay 40 41 49 40 65 235 Supervision counseling￾HCWs 36 25 24 33 37 155 88 Course Sub Course Province Grand Total Central Copperbelt Luapula North Western Northern Supervision Counseling￾Lay 36 23 12 13 12 96 Youth CT￾HCWs 45 48 22 38 21 174 Youth CT-Lay 83 75 74 51 53 336 Counseling and Testing Training Total 608 610 520 502 553 2,793 Laboratory Infrastructure ART Commodity Management A for Lab 26 37 11 20 17 111 ART Commodity Management B for Lab 10 25 14 4 12 65 CD4% count 6 20 13 18 12 69 DNA PCR 6 6 Dry Blood Spot Collection 58 64 50 25 50 247 Equipment Use & Maintenance 98 197 104 74 71 544 Laboratory Infrastructure Total 198 349 192 141 162 1,042 Male Circumcision Male Circumcision 74 61 37 48 63 283 Male Circumcision TOT 22 2 2 8 Male Circumcision Total 76 61 39 50 65 291 Other Trainings Training Adherence Counseling HCWs 51 116 51 32 52 302 Adherence Support Workers 30 60 34 7 41 172 Adherence Support Workers￾Refresher 70 121 77 73 96 437 ART Pediatrics Mentorship￾HCWs 9 9 Clinical Training Skills 5 3 3 4 5 20 89 Course Sub Course Province Grand Total Central Copperbelt Luapula North Western Northern Model sites provincial mentorship 52 49 62 60 53 276 Stigma Reduction TOT 9 9 18 Other Trainings Training Total 217 358 227 185 247 1,234 Pharmacy Training ART Commodity Management A for Pharmacy 41 55 25 40 29 190 ART Commodity Management B for Pharmacy 9 48 50 8 12 127 Pharmacy Training Total 50 103 75 48 41 317 Prevention of Mother to Child Transmission of HIV Training Family Planning 77 77 22 51 50 277 PMTCT Refresher￾HCWs 346 346 182 118 229 1,221 PMTCT Refresher-Lay 50 50 50 61 48 259 PMTCT￾HCWs 495 497 284 247 384 1,907 PMTCT-Lay 151 177 125 150 140 743 Prevention of Mother to Child Transmission of HIV Training Total 1,119 1,147 663 627 851 4,407 Grand Total 2,514 3,430 1,952 1,788 2,166 11,850 90 ANNEX Q ZPCT II M&E TEAM: ORGANOGRAM AND M&E POST DESCRIPTIONS 91 ANNEX R ZPCT II PARTNERS AND PERFORMANCE Partner Responsibilities Performance FHI – Prime  Overall program, technical and financial leadership; Responsible for all program indicators and M&E system  Liaises with USAID and manage relationships with the MOH, NAC, and private and all project partners  Coordinate with other USG partners to ensure uniformity of activities across the country  Provide oversight and guidance to all partners in the consortium.  UnderObjective1-Lead implementation (with the MOH) in scaling up HIV/AIDS services in the five provinces.  Institute a biannual review with the MOH, NAC, USAID and partners to ensure program results are in line with MOH and NAC goals. FHI has developed the capacity of health care workers in 371 facilities using the Ministry of Health’s, Health Management Information System (HMIS) through training and mentorship. FHI has developed the capacity of health care workers in 371 MOH and Private Sector facilities in all the technical areas including Counseling and Testing, PMTCT, ART, Laboratory and Pharmacy (numbers in ZPCT II targets table provided by ZPCT II) Successfully scaled up HIV and AIDS prevention, care and treatment services to 371 Ministry of Health facilities and CHAZ facilities and 18 private sector facilities in Northern, Luapula, Copperbelt, North Western and Central provinces Successfully coordinated and implemented ZPCT II partnership activities in all the HIV technical areas (Counseling and Testing, PMTCT, ART, Male Circumcision, laboratory and pharmacy), including community mobilization and referral, gender and capacity building), resulting in the project meeting and surpassing some of the LOP targets (Please see ZPCT II project targets table provided by ZPCT II) Successfully initiated innovations that improved the quality and access of services: web to SMS, CHC checklist, Nurse Prescriber program, sample referral. Successfully Managed MOH partnership through National and Provincial MOUs which extend ZPCT II support to district medical offices/facilities and provincial hospitals, through 61 Recipient Agreements Successfully managed sub recipients (Kara Counseling and Training Trust. CHAZ, Care, Cardno, Social Impact and MSH) who contributed towards all ZPCT II objectives and achievements. Annual partnership review meetings with all ZPCT II sub partners. Two meetings were held with senior directors of MOH in 2011. International Partners- data from FHI/ZPCT II September 27, 2012 Achievement Report from the COP – unless otherwise noted--The data base on training does not include all partners’ training MSH  UnderObjective2 -- MSH responsible for the laboratory and pharmacy objectives and collaborate on the PCR/viral load lab at Laboratories with capacity to perform clinical laboratory tests – no target Individuals trained in the provision of laboratory-related activities - 725 Tests performed at USG-supported laboratories during the 92 ADCH. reporting period: (a) HIV testing, (b) TB diagnostics, (c) syphilis testing, and (d) HIV/AIDS disease monitoring -- 3,850,122  Strategic placement of FACS Count/FACS Calibur equipment in high volume facilities.  Skill enhancement training through regular equipment use and maintenance training and commodity management trainings for both pharmacy and laboratory staff.  Facilitation of specimen referral activities including provision of motorbikes, phlebotomy training and provision of blood collection tubes.  Commodity Management training to ensure uninterrupted HIV testing services  Facilitate increased access to CD4 testing and provision of timely CD4 result.  Enhanced stock storage management systems for the provision of more efficacious ARV drugs for HAART  Regular Commodity Management trainings to ensure uninterrupted commodity supply  Commodity Management training to ensure uninterrupted HIV testing services  Training in collection, packaging and transportation of DBS  Web2SMS delivery of results  Regular PCR Techniques training for lab staff  Commodity Management training to ensure uninterrupted commodity supply  Setting up of PEP corners in collaboration with other units  Facilitate availability of ARV drugs for use in PEP  Monitoring medication use counseling and adherence especially in non-ART facilities  Increased access to CD4 testing and provision of timely CD4 result.  Facilitate availability of provision of more efficacious ARV drugs for HAART  Enhanced stock storage management systems for the provision of more efficacious ARV drugs for HAART  Commodity Management training to ensure uninterrupted commodity supply  Promotion of effective medication use counseling and adherence  Quantification and forecasting for male circumcision kits, surgical instruments and other consumables  Established redistribution sub-systems to enhance provision of uninterrupted supply CARE  Under Objective 2 - Leads activities to mobilize communities to access HIV/AIDS services, Support and capacity building:  Community/lay persons trained in counseling and testing (Youth CT and CT supervision) according to national or international standards (excluding TB) -- 336 93  Enhances existing referral networks and develop new referral networks.  Provides support, and funds transport allowances to ASWs and lay counselors.  Care builds capacity in CBOs and FBOs to coordinate volunteers and deliver community-level services.  Community/lay persons trained in the provision of PMTCT services according to national or international standards (Former Traditional birth attendants) – 114  Number of district referral networks formed that are functional 37  Number of community volunteers being paid transport re￾imbursement 1,246  Contract signed between CARE and ZANACO Bank to pay volunteers through ATMs. Implementation has stated on the Copperbelt and will be rolled out to the other provinces in the coming quarter and a target of 60% of the volunteers is expected to be paid through this system.  Community/lay persons trained in the provision of ART adherence counseling services according to national or international standards – 135  The number of NGOs/FBOs that CARE built capacity: 15 organizations had 3 staff trained each in grant management, total (45 staff). The same number of organizations was approved to receive sub-grants. Three organizations have since dropped out. 5 are completing the final milestone, mile stone #3, and 7 are starting milestone one. Access: CT referred: LOP 25,000, actual referred 32,014 = 28% CT reaching: LOP 25,000, actual reaching 18,753 = 5% PMTCT referred: LOP 3,000, actual referred 14,428 = 80% PMTCT reaching: LOP 3,000, actual reaching 7,591 = 253% MC referred: LOP5,000, actual referred 9,846 = 197% MC reaching: LOP 5,000, actual reaching 9,116 = 182% Cardno  Under Objective 3 -- CARDNO/EMG trains and provides technical assistance to support PHOs and DHMTs to build their capacity for technical and program management. Conducted baseline survey in all 5 ZPCTII provinces and used results to develop capacity building plan and tools, i.e. mentorships guidelines and questionnaire, and management indicators reporting template. Developed training manuals for each of the four capacity building pillars: Finance, HR, Planning and Governance. Local organizations (PMOs and DMOs) provided with technical assistance for HIV-related institutional capacity building: 44 DMOs and PMOs. PMO and DMO staff trained: Governance Trainings – 91 Finance (Mentors’) Trainings – 20 HR (Mentors’) Trainings – 21 Planning Trainings – 83 Total: 215. Cardno U.S undertook technical assistance visits to Zambia which resulted in the development of a mentorship planning tool which is currently being used by the team to measure the extent to which the performance gaps of PMOs and DMOs are closed over time. Additionally, technical assistance visits resulted in the development of a quality assurance plan for the capacity building program as part of Cardno’s policy on quality 94 control. Cardno used the quality assurance plan to train capacity building trainers on quality assurance and training delivery among other things. SI  Social Impact provides technical assistance to develop a gender strategy for partner implementation. Trainings on gender reported by Social Impact total 190 - at the PHO and DMO level 58 – and 77 HW at the facility level Conducted a gender assessment which formed the basis for developing the gender strategy. Developed a Gender strategy to guide partner implementation The 28 indicators in the USAID approved Gender Strategy were revised and reduced to 14 to better align with PEPFAR five crosscutting strategies. Developed four manuals and a guide for documentation of success stories –infused GBV monitoring into assessment tools and referral systems. Engendered ZPCT II annual work plans. Engendered the Capacity Building Code of Ethics for Ministry of Health. technical assistance visits provided and reported by Social Impact Local Partners CHAZ  CHAZ will provide strategic services through church-run facilities to provide strategic HIV/AIDS services. CHAZ is currently supporting 10 church-run facilities in three of the ZPCT II supported provinces. Nine of the 10 facilities are providing ART and one facility is providing specimen referral services technical assistance visits provided to all supported facilities and reported by CHAZ staff The data gathered from CHAZ supported facilities contributes to the overall ZPCT II LOP targets The expansion of HIV and AIDS services through mission facilities has improved access of quality HIV and AIDS services to rural communities KCTT  KCTT/KARA will train CT supervisors under ZPCT II through contracts with FHI. Kara report documented a target of supervisor training of 131 with 130 trained. Kara also reported a target of providing TOT to 40 trainees – no number trained reported. Kara also reported couples counseling with a target of 64 – reported 100 trained. Source: ZPCT II Program Office 95 ANNEX S USAID MANAGEMENT OF ZPCT II DELIVERABLES Task 11. USAID Project Management The contractor shall: 1. Conduct an early evaluation of the environmental concerns and lay out an acceptable mitigation plan. The plan shall be submitted to the COTR and the Mission Environmental Officer shall provide approval of the submitted plan. ZPCT II submitted the evaluation of environmental concerns: the revised mitigation plan was approved by the mission. Since all partners had problems, USAID was proactive and provided training. 2. Provide sustainability plan to ensure activities continue after project ends. The COR will check and provide the sustainability plan to the evaluation team . The contractor is expected to propose additional deliverables that link the expected results and activities described in section C. F.6 REPORTS In addition to the requirements set forth for submission of reports in Sections 1 and J, and in accordance with AIDAR clause 752.242-70, Periodic Progress Reports, the Contractor shall submit reports, deliverables or outputs as further described below to the COTR (referenced in Sections F.2 and O). All reports and other deliverables shall be in the English language, unless otherwise specified by the COTR. (a) PEPFAR Country Operational Plans (COPs) Reporting PEPFAR has just started with two year COPs. a. Activity narratives and targets PEPFAR provided guidance and ZPCT II submitted activity narratives and target for semi annual and annual submissions. b. Semi-Annual and Annual Progress Reports (SAPRs and APRs) ZPCT II submitted semi annual and annual Progress Reports according to the appropriate timelines. These were reviewed by USAID in joint meetings with ZPCT II. These Progress Reports have been approved. (b) Performance Management Plan (PMP) There was conflicting information provided in team interviews with USAID, FHI 360 and Cardno and no documentation was provided to show that the PMP was approved. In addition, the M&E Plan,although not a contract deliverable for this project was tentatively approved and is being discussed further guidance from USAID is being provided to ZPCT II . The COR st ated that the review of the M&E plan was done by the M&E specialist who then departed. The M&E specialist from the regional South Africa mission focused his support on Data Quality Analysis versus the M&E plan. Other specialist assistance to the COR is planned. a. Data collection methodology for data quality assessments ZPCT II produced a guide for data quality assessments that included the data collection methodology. 96 b. Semi-annual and annual portfolio review presentations USAID reviews the ZPCT II self assessment (that is part of the process) and USAID and ZPCT II discuss progress and jointly during the two hour presentation. The review includes progress against objectives and targets, budget data, accomplishments, issues, planned next 6 month activities, constraints and procurement. (c) Ad-hoc analyses, evaluations, studies, operational research and other reports as requested ZPCT II has conducted studies, evaluations and operations research have been done by ZPCT II, such as the Gender Assessment, analysis of the use of volunteers, the Nurse Prescriber operations research, studies of the use of SMS, and others. (d) Grants manual for the sub-granting process (within three months of award) ZPCT II provided timely submission of the grants manual: it was approved by USAID. In year one there were six grants: in year two 10. Most of the grants are in the Copper Belt province where private facilities are more plentiful. (e) Quarterly Reports: Quarterly Financial and Progress Reports shall be submitted no later than one month after the end of the quarter. The scope and format of the quarterly reports will be determined in consultation with the COTR. Quarterly Financial and Progress Reports were submitted by ZPCT II on time: they were reviewed by USAID. (f) Workplans: (1) Transition period work plan: The contractor shall be required to submit upon award a transition work plan to ensure that there is no disruption with the current program. (2) Annual Workplans shall detail the work to be accomplished during the upcoming year. The scope and format of the Annual Work plan will be agreed to between the Contractor and the COTR during the first thirty days after the award of the contract. These Annual Workplans may be revised on an occasional basis, as needed, to reflect changes on the ground and with the concurrence of the COTR. The annual work plans have been submitted by ZPCT II and approved by USAID. The first Annual Work plan shall be submitted within one month of award of the contract. The work plan should include the estimated monthly funding requirements during the upcoming year of program implementation, necessary to meet all program objectives within the contract. USAID will respond to the work plan within five calendar days. The First Annual Work plan was submitted within the first month: it was approved by USAID within 5 days. (g) Final Report: The Contractor shall prepare a final report that matches accomplishments to the specific paragraphs of the Scope of Work. The final report will be drafted to allow for incremental improvements in the process, both generally within USAID and specifically with respect to this contract. 97 ANNEX T LIFE OF PROJECT & FHI PRINCIPLES Objective Indicator Targets (Aug 09 - May 14) 1.1 Counseling and Testing (Projections from ZPCT service statistics) Service outlets providing CT according to national or international standards 370 Individuals who received HIV/AIDS CT and received their test results 728,000 Individuals who received HIV/AIDS CT and received their test results (including PMTCT) 1,300,000 Individuals trained in CT according to national or international standards 2,316 1.2 Prevention of Mother-to-Child Transmission (Projections from ZPCT service statistics) Service outlets providing the minimum package of PMTCT services 359 Pregnant women who received HIV/AIDS CT for PMTCT and received their test results 572,000 HIV-infected pregnant women who received antiretroviral prophylaxis for PMTCT in a PMTCT setting 72,000 Health workers trained in the provision of PMTCT services according to national or international standards 5,325 1.3 Treatment Services and Basic Health Care and Support (Projections from ZPCT service statistics) Service outlets providing HIV-related palliative care (excluding TB/HIV) 370 Individuals provided with HIV-related palliative care (excluding TB/HIV) (adults and children) 560,000 Pediatrics provided with HIV-related palliative care (excluding TB/HIV) 60,000 Individuals trained to provide HIV palliative care (excluding TB/HIV) 3,120 Service outlets providing ART 130 Individuals newly initiating on ART during the reporting period 115,250 Pediatrics newly initiating on ART during the reporting period 11,250 Individuals receiving ART at the end of the period 146,000 Pediatrics receiving ART at the end of the period 11,700 Health workers trained to deliver ART services according to national or international standards 3,120 TB/HIV Service outlets providing treatment for TB to HIV+ individuals (diagnosed or presumed) in a palliative care setting 370 HIV+ clients attending HIV care/treatment services that are receiving treatment for TB 17,000 Individuals trained to provide treatment for TB to HIV+ individuals (diagnosed or presumed) 3,120 Registered TB patients who received HIV/AIDS CT and their test results at a USG-supported TB service outlet 30,400 1.4 Male Circumcision (ZPCT II projections) Service outlets providing MC services 50 Individuals trained to provide MC services 260 98 Number of males circumcised as part of the minimum package of MC for HIV prevention services N/A 2.1 Laboratory Support (Projections from ZPCT service statistics) Laboratories with capacity to perform: (a) HIV tests and (b) CD4 tests and/or lymphocyte tests 111 Laboratories with capacity to perform clinical laboratory tests N/A Individuals trained in the provision of laboratory-related activities 375 Tests performed at USG-supported laboratories during the reporting period: (a) HIV testing, (b) TB diagnostics, (c) syphilis testing, and (d) HIV/AIDS disease monitoring 3,813,000 2.2 Capacity Building for Community Volunteers (Projections from ZPCT service statistics) Community/lay persons trained in counseling and testing according to national or international standards (excluding TB) 2,506 Community/lay persons trained in the provision of PMTCT services according to national or international standards 1,425 Community/lay persons trained in the provision of ART adherence counseling services according to national or international standards 600 Districts graduated from intensive assistance by meeting Ministry of Health - approved minimum quality and performance criteria in technical service￾delivery areas (CT, PMTCT, ART, clinical care, laboratory and pharmacy services) and management of commodities, data and human resources 42 Provincial Medical Offices and District Medical Offices with increased capacity to manage improved HIV/AIDS services 47 Public-private partnerships for HIV/AIDS service delivery established in all target provinces through implementation of tested technical approaches from the public sector. 30 99 ANNEX U USAID EVALUATION CRITERIA • The evaluation report should represent a thoughtful, well-researched and well organized effort to objectively evaluate what worked in the project, what did not and why. • Evaluation reports shall address all evaluation questions included in the scope of work. • The evaluation report should include the scope of work as an annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline need to be agreed upon in writing by the technical officer. • Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an Annex in the final report. • Evaluation findings will assess outcomes and impact on males and females. • Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). • Evaluation findings should be presented as analyzed facts, evidence and data and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings should be specific, concise and supported by strong quantitative or qualitative evidence. • Sources of information need to be properly identified and listed in an annex. • Recommendations need to be supported by a specific set of findings. • Recommendations should be action-oriented, practical and specific, with defined responsibility for the action. Source: (USAID Evaluation Policy, January 19, 2011) 100 ANNEX V KEY PRINCIPLES AND REQUIREMENTS In addition to the life of Project service delivery targets contained in Appendix 1, FHI360 is responsible for ensuring that key principles and requirements established under the contract are applied and met during the course of implementation. These principles and requirements are: 1. Alignment: ZPCT II must continue to be fully aligned with Zambia’s and the USG’s health priorities, policies and objectives. 2. Partnerships: FHI360 shall work in partnership with the Ministry of Health at all levels to plan for and deliver a comprehensive package of HIV/AIDS services within the existing government structure. 3. Continuum of Care: FHI360 will work to provide a seamless flow of mutually reinforcing services, from prevention to care and treatment to other support, through networks of facility and community providers linked by formal referral structures. 4. Innovations: FHI360 shall continue ZPCT’s innovative, experience-based technical approaches in HIV/AIDS service delivery and implement innovative new approaches to MC, building capacity in management of HIV/AIDS as a chronic condition, integrating prevention for HIV-positive individuals in routine clinical visits and other areas. FHI360 shall conduct operational research as needed to maximize the efficacy of new approaches. 5. Linkages: FHI360 assistance will be incorporated into Ministry of Health action plans at the national, provincial and districts levels. ZPCT will coordinate with CDC and EGPAF to replicate the ZPCT model in the four non-project provinces. 6. Procurements: FHI360 will provide essential laboratory, pharmacy, medical and related equipment and supplies to ensure an uninterrupted supply of commodities for service delivery. 7. Zambian Leadership and Human Resources Support: FHI360 will fund staff development activities, including training in technical skills, leadership and finances. 8. Gender Integration: All project activities will support the GRZ’s National Gender Policy and USG gender strategies (specific approaches are described in Section 3.2 of the Task Order. 9. Sustainability: FHI360 shall ensure that project gains are sustained by implementing ZPCT’s graduation plan for districts and facilities, as well as strengthening the Ministry of Health’s capacity to maintain and manage the system in collaboration with communities, the private sector and other partners. Work will continue to build enduring networks of service providers through coordinated district-based referral systems that span the continuum of care. FHI360 will use existing community structures as the entry point for community level activities. 10. Environmental Considerations: FHI360 will comply with all environmental regulations. 101 END NOTES 1 Zambia Demographic Health Survey (ZDHS) 2007 3 Zambia National HIV/AIDS/STI/TB Council (NAC) 2009 Mode of Transmission (MOT) Report 4 Zambia Demographic Health Survey (ZDHS) 2009 5 Following USAID guidance, the evaluation focused on the five ZPCT II-supported provinces as defined before Muchinga province was created (i.e., Central, Copperbelt, Luapula, North Western, Northern). 6 All health facilities selected offered the following services: CT, PMTCT, ART, HIV/AIDS care 7 Zambia Demographic and Health Survey 2007: Preliminary Report 8 Organizational Capacity Assessment Tool 9 http://www.lusakatimes.com/2011/11/11/government-increases-funding-health-sector-abolishes-primary-health-services-fees/; UNGASS Zambia Country Report, Monitoring the Declaration of Commitment on HIV and AIDS and the Universal Access Biennial Report, submitted to the United Nations General Assembly Special Session on HIV/AIDS, Reporting period: January 2010 – December 2011, March 31st 2012 10 The team conducted FGDs with 45 groups to elicit the opinions of respondents. However, one interview was invalid because the questions asked were different that the FGD questionnaire provided in Annex G; this was the FGD with women in Luapula, Mansa Central. This case #18 was eliminated from the analysis. Thus there were 44 net valid group interviews. The participants of the FGDs are not statistically representative of the entire population of ZPCT II and non-ZPCT II beneficiaries, the team collected data from as many groups of as the time, resource, and logistical constraints would allow.