EVALUATION USAID/Rwanda: HIV/AIDS Clinical Services Project (HCSP) End of Project Evaluation DECEMBER 2012 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Carol Shepherd, Annette Nassali, and Jean Claude Karasi through the GH Tech Bridge II Project. Cover Photo © 2004 Eileen Dietrich, Courtesy of Photoshare EVALUATION USAID/Rwanda: HIV/AIDS Clinical Services Project (HCSP) End of Project Evaluation DECEMBER 2012 Global Health Technical Assistance Bridge II Project (GH Tech) USAID Contract No. AID￾OAA-C-12-00027 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. This document (Report No. 12-02-007) is available in printed or online versions. Online documents can be located in the GH Tech website at www.ghtechproject.com. Documents are also made available through the Development Experience Clearinghouse (http://dec.usaid.gov). Additional information can be obtained from: GH Tech Bridge II Project 1725 Eye Street NW, Suite 300 Washington, DC 20006 Phone: (202) 349-3900 Fax: (202) 349-3915 www.ghtechproject.com This document was submitted by Development and Training Services, Inc., with CAMRIS International to the United States Agency for International Development under USAID Contract No. AID-OAA-C-12-00027. USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION i ACKNOWLEDGMENTS The GH Tech Bridge II Project Team for the End of Project Evaluation of the Rwanda HIV Clinical Services Project (HSCP) thanks Jennifer Slotnick, Health Service Delivery Team Leader, USAID/Rwanda, and her team— Canisious Musoni, Esperance Mukamana, and Jennifer Mbabazi—for their support throughout the entire evaluation process. We are also grateful to Kirstin Krudwig, Program Manager, GH Tech Bridge II, for her vision in the design of the team structure, which not only enriched the end result but also laid the foundation for adoption of our recommendations. The individual expertise and multinational composition of the evaluation team made it easier to understand the context and perspectives of the two main stakeholders, USAID and the Rwanda Ministry of Health. It also helped the evaluation team to formulate recommendations that are realistic, which will increase the likelihood they will be adopted. The USAID/Rwanda Health Team is to be commended for their superb management of the HCSP. Most notable is the superb results the implementing partners have achieved. The project team extends its sincere appreciation to Doris Youngs, Chief of Party (COP) for the Family Health Project, for providing office space and administrative support for its time in￾country; and to Sara Lewis, COP for the IntraHealth HCSP, for supplying needed equipment. The quality of this evaluation was deeply enhanced by the candid discussions and the support provided by the three HCSP COPs, Cornelia Van Zyl, EGPAF; Kimberly Ross, FHI 360; and Sara Lewis, IntraHealth. Their perspectives were invaluable. They helped to validate the findings, interpret analyses in terms of the local context, and draft practical and specific recommendations. The COPs and their teams are to be applauded for their remarkable achievements. This evaluation could not have been carried out without the cooperation of the many individuals who took time out from busy schedules to answer questions, make datasets available, give site tours, and provide information and documents to the project team. Muyange Thierry was the behind-the-scenes Logistics Coordinator who effortlessly organized numerous priorities and schedules, procured needed supplies and services, and provided translation services as needed. We appreciate the long hours he put in to ensure that the team received what it needed. To all of you, Murakoze! Merci! Thank You! ii USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION iii CONTENTS ACRONYMS ........................................................................................................................................ v EXECUTIVE SUMMARY .................................................................................................................ix Background ................................................................................................................................ix Methodology .............................................................................................................................ix Limitations .................................................................................................................................. x Innovations ................................................................................................................................. x Findings ....................................................................................................................................... xi Conclusions .............................................................................................................................. xii Lessons Learned ..................................................................................................................... xiii Best Practices .......................................................................................................................... xiii Recommendations ................................................................................................................. xiv I. INTRODUCTION .................................................................................................................... 1 II. BACKGROUND ....................................................................................................................... 3 III. METHODOLOGY .................................................................................................................... 7 Review of Documents .............................................................................................................. 7 Interviews .................................................................................................................................... 7 Site Visits ..................................................................................................................................... 7 Tools ............................................................................................................................................ 8 Data Analysis .............................................................................................................................. 8 Triangulation .............................................................................................................................. 9 Challenges and Limitations ...................................................................................................... 9 IV. RESULTS, FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS ................. 11 Results .........................................................................................................................................11 Research Questions, Findings, Conclusions, and Recommendations ..........................15 Research Questions, Findings, Conclusions, and Recommendations ..........................21 Expansion of HIV/AIDS Clinical Services in Institutions ..................................................36 V. BEST PRACTICES AND LESSONS LEARNED ................................................................ 43 Lessons Learned ......................................................................................................................43 Best Practices ...........................................................................................................................44 iv USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION VI. SUMMARY OF RECOMMENDATIONS ........................................................................... 47 Quality and Timeliness of Routine Data Reporting .........................................................47 Financial and General Management .....................................................................................47 Service Delivery .......................................................................................................................48 Epidemiologic Outcomes ......................................................................................................49 ANNEXES ANNEX A: SCOPE OF WORK .................................................................................................... 51 ANNEX B: ORGANIZATIONS AND PERSONS CONTACTED ....................................... 67 ANNEX C: REFERENCES .............................................................................................................. 71 ANNEX D: TOOLS ......................................................................................................................... 73 ANNEX E: INDICATOR TARGETS AND ACHIEVEMENTS—MEMS ............................... 77 FIGURES Figure 1: Districts Served by the HIV Clinical Services Project. ........................................... 4 Figure 2: Infants of HIV+ Mothers Tested for HIV within 12 Months of Birth ............... 15 Figure 3: Infants Born to HIV+ Mothers Started on CTX Prophylaxis within Two Months of Birth ............................................................................................................. 16 Figure 4: Number of HIV+ Persons Receiving Cotrimoxazole ........................................... 16 Figure 5: Rwandans Given Post-exposure Prophylaxis ......................................................... 18 Figure 6: HIV+ Adults and Children Receiving at Least One Clinical Service ................. 28 Figure 7: Pregnant Women with Known HIV Status ............................................................. 37 Figure 8: Adults and Children with Advanced HIV Infection Currently Receiving ART37 Figure 9: PLWHA Reached with a Minimum Prevention Package ...................................... 38 Figure 10: Individuals who Received Their VCT Results ........................................................ 39 Figure 11: Adults and Children with Advanced HIV Infection Newly Enrolled in ART .. 40 Figure 12: Adults and Children known to be Alive and Being Treated 12 Months after ART Initiation ...................................................................................................... 40 TABLES Table 1: Site Visit Locations ......................................................................................................... 8 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION v ACRONYMS AIDS Acquired immune deficiency syndrome ANC Antenatal care ART Antiretroviral therapy ARV Antiretroviral CAMERWA Centrale d’Achat des Medicaments Essentiels de Rwanda (the Rwandan essential medicines procurement organization) CDLS District AIDS Commission CHAMP Community HIV/AIDS Mobilization Program CHW Community health worker CNLS Commission Nationale de Lutte Contre le SIDA (National Commission for the Fight Against AIDS) COP Country Operating Plan CPA Complementary package of activities CPT Contraceptive prevalence rate CT Counseling and testing CVCT Couples voluntary counseling and testing DH District hospital DHMT District Health Management Team DHN District Health Network DHS Demographic and Health Survey EDPRS Economic Development and Poverty Reduction Strategy EGPAF Elizabeth Glaser Pediatric AIDS Foundation FHI Family Health International FHP Family Health Project FP Family planning GBV Gender-based violence GH Tech Global Health Technical Assistance Bridge II Project GOR Government of Rwanda HC Health center HFPCP Health, Family Promotion, and Child Protection Unit HIV Human immunodeficiency virus HIV– HIV negative HIV+ HIV positive vi USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION HSSP Health Sector Strategic Plan ICT Information and communications technology IHDPC Institute of HIV/AIDS, Disease Prevention & Control IMCI Integrated management of childhood illnesses IRB Institutional Review Board IST In-service training IT Information technology ITN Insecticide-treated nets M&E Monitoring and evaluation MCH Maternal and child health MEMS Monitoring and Evaluation Management Systems MIYCN Maternal Infant and Young Child Nutrition MNCH Maternal, neonatal, and child health MOH Ministry of Health MTCT Mother-to-child transmission NACC National AIDS Control Commission NGO Non-governmental organization NRL National Reference Laboratory OGAC Office of the Global AIDS Coordinator in the Department of State OI Opportunistic infection OVC Orphans and vulnerable children PAQ Partenariat pour l’Amelioration de la Qualité (Partnership for Quality Improvement) PBF Performance-based financing PEPFAR President’s Emergency Plan for AIDS Relief PITC Provider-initiated testing and counseling PLWHA People living with HIV/AIDS PMTCT Prevention of mother-to-child transmission PNILT National Tuberculosis Control Program PNLP National Malaria Control Program PST Pre-service training PWP Prevention with PLWHA QMC Quality Management Committee RBC Rwanda Biomedical Center RFA Request for Applications RH Reproductive health USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION vii RN Registered nurse RPM+ Rational Pharmaceutical Management Plus RRP+ Rwandan Network of People Living with HIV SGBV Sexual and gender-based violence SM Safe motherhood STI Sexually transmitted infection STTA Short-term technical assistance TA Technical assistance TB Tuberculosis TRAC Treatment and AIDS Research Center TRAC Plus The Center for Training and Treatment for HIV/AIDS and Other Epidemics TRACNET Treatment and Research AIDS Centre Network TWG Technical working group UK United Kingdom UNC University of North Carolina UNFPA United Nations Population Fund UNICEF United Nations Children’s Fund USAID United States Agency for International Development USAID/W USAID/Washington USG United States Government VCT Voluntary counseling and testing WB World Bank WFP World Food Program WHO World Health Organization viii USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION ix EXECUTIVE SUMMARY The HIV Clinical Services Project (HCSP) was a five-year cooperative agreement, implemented by IntraHealth, the Elizabeth Glaser Pediatric AIDS Foundation, and Family Health International 360, to address the emerging issues and disease burden of HIV/AIDS. The project was tasked with building capacity in national and district institutions to support and manage health systems and with expanding HIV/AIDS clinical activities and clinical capacity for patient care in selected districts. Specifically, the goal was to decrease new cases of HIV/AIDS; increase the number of people living with HIV and AIDS (PLWHA) receiving care and support, and increase the number of HIV+ patients receiving treatment. Maternal and child health (MCH) and family planning (FP) funds were obligated in year 3 to ensure these services were integrated with HIV/AIDS services. By all measures, the HCSP more than met its objectives. BACKGROUND Health sector reforms initiated by the Government of Rwanda (GOR) have made remarkable progress. The total fertility rate (TFR) dropped from 6.1 in 2005 to 4.6 in 2010. Modern contraceptive prevalence rate (CPR) is 45% and the number of women giving birth in the presence of a skilled attendant has increased from 39% in 2005 to 69% in 2010 (2010 DHS), with benefits for maternal health. The fact that 90% of children aged 12–23 months are now fully vaccinated helped lower infant mortality from 86 in 2005 to 50 in 2010 and under-5 mortality fell from 152 to 76. Stunting in children dropped from 51% to 44%; wasting from 5% to 3%; and underweight from 18% to 11%. In terms of preventing malaria, by 2010 73% of pregnant women and 71% of children under 5 were sleeping under a mosquito net (DHS 2010). METHODOLOGY This report presents the findings of the HCSP end-of-project evaluation, which measured the efficiency, effectiveness, and relevance of resource allocation for programming and was a means for learning and capturing best practices. The lessons learned and the recommendations will influence decision-making for the next phases of programming in Rwanda and in similar projects worldwide. The evaluation will also inform district- and facility-level technical, programmatic and administrative support priorities. The evaluation team addressed the following development hypothesis: If USAID support via HCSP built capacity in district and national Rwandan institutions to support and manage Rwanda’s health systems and expanded quality health service delivery, then the results would be improvements in (1) the quality and timeliness of routine data reporting; (2) financial management; (3) service delivery and technical capacity; and (4) epidemiological outcomes among target populations. The evaluation team used both qualitative and quantitative methods to collect and analyze information relevant to the objectives, the four outcomes of the development hypothesis, and the research questions outlined in the Scope of Work (Annex A). Because a rigorous counter￾factual (control) study was not conducted before the evaluation, the team carried out a performance evaluation that focused on descriptive and normative questions. x USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION LIMITATIONS Time constraints prevented the evaluation team from accessing the Institutional Review Board (IRB) process; there was no counterfactual at initiation; implementing partners (IPs) were not asked to capture financial data to permit in-depth financial analysis; and numerous, sometimes conflicting, datasets caused confusion, all of which limited the methodologies available to the team. INNOVATIONS Each IP brought its unique corporate strength to bear on crafting innovative approaches to service delivery in Rwanda. The approaches utilized were formulated or adapted in response to clinical problems, operations research findings, and international experience. The ability of the IPs to customize approaches to the unique situation that is Rwanda rendered their innovations major contributors to the success of the project. Among the innovations are the following: The FP/HIV national integration model The HCSP helped structure the integration model and trained 165 staff on how it worked. As of September 30, 2009, among the 6,119 HIV+ women enrolled in care and treatment at HCSP sites, 3,329 (45%) were using a family planning (FP) method. Male involvement The project used integrated service models to increase partner rates of participation in HIV and FP counseling. From October 2007 to September 2008, 15,274 male partners were tested (78%); 541 were HIV+. For the 10 years between July 2002 and June 2012 male involvement averaged 84%. Palliative care With technical assistance (TA) from Mildmay International and in collaboration with the MOH, 547 health service providers were trained to provide HIV-related palliative care, and 207 other individuals (community health workers, PAQ [Partenariat pour l’Amelioration de la Qualité, Partnership for Quality Improvement] team members, and PLWHA volunteers) were trained in home-based care. The project drafted and piloted eight palliative care indicators, which documented strong community-health center linkages and stressed the need for enhanced pain management. As a direct result, oral morphine was added to Rwanda’s Essential Medicines List and the Palliative Care Model became policy. Pediatric psychosocial program The HCSP launched pediatric psychosocial programs at 21 sites serving over 728 HIV+ children. These included support for announcing children’s HIV results, counseling for children and families, and recreational and educational activities. PMTCT The HCSP’s new tracking of prevention of mother-to-child transmission (PMTCT) mother-infant pairs makes it easier to monitor service delivery, which reduces missed opportunities for treatment, ensures early initiation of prophylaxis and treatment, and improves morbidity rates for HIV-exposed infants Research-based booklet for PWP counselors The Rwanda Biomedical Center Institute of HIV/AIDS, Disease Prevention & Control (RBC/IHDPC) and Rwanda Health Communication Center (RHCC), the Rwandan Network of People Living with HIV/AIDS (RRP+), Kacyiru Health Center, and the HCSP worked together to complete a booklet in Kinyarwanda, which guides counseling on prevention with PLWHAs (PWP) in clinics and communities.  TRACnet, 2011–12. USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION xi SGBV The HCSP worked with the Ministry of Health (MOH) to put in place a one-stop model to address sexual and gender-based violence (SGBV) that brought medical care, police, lawyers, and counseling and testing to the clinic for clients, which increases attendance and improves conviction rates. In 2011, 908 SGBV cases received care and support through U.S. Government￾supported MOH facilities, and in 2012, the number was 1,107. The project trained a total of 2,542 people about gender-based violence. FINDINGS The data collected and analyzed by the evaluation team clearly supports the USAID Rwanda development hypothesis. The findings and conclusions are organized in terms of the four target areas in the hypothesis. Quality and Timeliness of Routine Data Reporting  The project has provided significant training and mentoring to data managers in addition to that provided by the RBC, Monitoring and Evaluation Management Systems (MEMS) and the health management information system (HMIS). These groups have also carried out data quality assessments and audits. While improvement has been steady, there is still a need for continuing reconciliation between the various datasets.  The data managers have limited capacity. Financial and General Management  The sub-agreement approach has built financial reporting capacity. Initially there were problems with accuracy and timeliness but recent reports have been correct and on time. The financial officers are also now able to better manage funds and procurements. In general, compliance is very good. Strong leadership was associated with the reporting improvements.  The mentoring process has created dependence; some financial officers still feel they need more training to be independent. Several who were interviewed had never completed monthly reports without help from their IP mentor and fear that they cannot function alone.  In two of the six health centers (HC) visited, management and leadership were strong. The IPs used performance-based financing (PBF) scores to monitor management of the sites they supported. Sites where clinical and managerial issues were identified had low PBF scores. But even where management is strong, patient confidentiality was often compromised.  Seconded personnel showed no evidence of transferring knowledge. However, supervisors are pleased with their performance and consider that they have done an excellent job.  There was no clear exit strategy. While implementation was decentralized, the “transition” plan was defined at the national level. There was no common understanding of the definition of “transition” between USAID, GOR levels, and IPs, which caused confusion and anxiety. Service Delivery  Most activities required major commitments to capacity building in the form of training, structural improvements, and equipment procurement. Infrastructure improvement is still needed. At the center, capacity building covered formation of policy and development of xii USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION materials. In the districts, a large amount of training was carried out, but because of high turnover of data managers and nurses, repeat trainings were necessary.  TB and HIV services are 100% integrated in health centers (HCs); providers caring for HIV patients are slightly less compliant with the TB testing protocol although more compliant in testing primary TB patents for HIV. Some outdated lab facilities carry a risk of transmission.  FP/MCH/HIV integration is provided at all supported district hospitals (DH) and HCs. Services include maternal and child health (MCH), vaccinations, HIV care, family planning (FP), and malaria care. HIV services are provided in the maternity wards, many of which have been renovated. These wards provide kangaroo care and some have incubators. Community-based FP is available.  PMTCT is accessible at all HCSP-supported HCs. The proportions of women enrolled in PMTCT whose male partners accompanied them and were tested held steady at 80%.  Nutrition for HIV+ infants has been integrated into infant feeding services The HCSP contributed to guidelines and dissemination of educational materials for adult nutrition through antenatal care (ANC), care and treatment, kitchen gardens, and cooking demonstrations.  Palliative care is being introduced. The HCSP helped to adapt service guidelines, supported drafting of the policy, and trained providers.  SGBV services are now available at HCSP sites, although in most utilization is still low. HC directors stated that there was not much SGBV in their communities. Epidemiologic Outcomes  The HCSP clearly helped to increase access to services and the capacity of health providers and managers to provide high-quality services. HCSP support for community-based health insurance (mutuelles) helps increase access. For example, between 2010 and 2012, the number of people with advanced HIV infection who were receiving ART rose from 26,159 to 34,000. CONCLUSIONS Quality and Timeliness of Routine Data Reporting  There has been significant improvement in the quality and timeliness of routine data reporting. However, there are still problems with data accuracy that requires reconciliation of databases.  Data managers have been trained and have worked with mentors, but most do not have the skills needed to carry out activities like data cleaning and monitoring and evaluation (M&E).  The very heavy workloads of data managers, especially at larger facilities and hospitals, is affecting data quality and reporting. When data managers are assigned activities not included in their scope of work, data quality and timeliness of reporting goes down.  The IPs found the subgrant approach to be an effective way to build capacity in health facility management. Financial officers must complete all documentation and cash reports correctly before they can request funds for the next month. USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION xiii  Financial management capacity building has been effective for the basic role but financial officers feel that they need more training if they are to function independently.  The mentoring process used by the IPs has created significant dependence.  General management is weak in many facilities. There is often little initiative to solve problems or make decisions. Staff is often not held accountable. Strong management makes a difference in HC and DH performance, as can be seen in the PBF scores. Service Delivery  Integration of FP, MCH, nutrition, and SGBV into HIV services is directly attributable to HCSP. The one-stop shop reduces stigma and discrimination for PLWHA, and the ease of access seems to have had a role in the uptake of FP, though there may also be other influential variables.  Supportive supervision is well-coordinated and integrated. However, mentoring has created dependence. Without the IPs to provide transportation, in some districts supervision visits have stopped. The current inefficient system does not work well.  Rwanda has been able to push down HIV/AIDS prevalence well below its neighbors through a concerted and well-managed effort to contain the epidemic. By June 2012, 467 (94.5%) of health facilities were offering PMTCT services, 55 more than in the previous year. Most also had an HIV testing system and were providing HIV care and support. **  Because of HCSP, the minimum HIV/AIDS package, as spelled out in the HIV/AIDS strategic plan, is now present at all HCSP-supported health facilities, and the DHs, working with the USG partners, have been empowered to draw up integrated plans, which include non-HIV service delivery and focus on training, supervision, and mentorship for hospital and HC staff.  Rather than being competitive, the IPs acted as teammates.  A key factor in the success of HCSP is that the GOR has created an environment that is very supportive of the project. LESSONS LEARNED Successful HCSP implementation has resulted in critical lessons learned and produced several best practices. The most important are summarized below:  Exit Strategy and Sustainability At the beginning of each project, the MOH and USAID together should prepare a well-thought-out sustainability strategy that incorporates an exit strategy. Because such a strategy has major implications for project approaches and activity planning, it is imperative that all stakeholders are in agreement. BEST PRACTICES  Partnership and Government Ownership Joint planning nurtures both productive relationships between project and district staff and harmonization of activities and priorities. Engaging the MOH at all levels reinforced government structures and facilitated shared success. Joint planning also increased MOH capacity and ownership. ** HIV Annual Report 2011; 2012. xiv USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION  Health Facility Management The subgrant approach is an effective way to build capacity in health facility management. Over the life of the project subgrantees showed improvements in their capacity to manage funds and conduct procurements and in overall compliance.  Holistic Continuum of Care, Including Palliative Care The holistic approach provides a continuum of care from diagnosis to palliative care that includes physical, emotional, social, spiritual, food support, and income-generating activities, and mutuelles, all of which contribute to the ability of patients to make choices about their own health. Linkages between teams of providers and PLWHA groups were critical to formulating a multidisciplinary and holistic palliative care model for HIV and AIDS care/treatment in Rwanda.  Task Shifting In Rwanda a pilot program was launched for nurses to provide first-line antiretroviral (ARV) treatment; treatment for simple opportunistic and for sexually￾transmitted infections (STIs); clinical and biological monitoring; and managing the side effects of ARVs. A retrospective evaluation in 2008 of the feasibility and effectiveness of the model found that nurses could effectively and safely prescribe ART when given adequate training and mentoring. After 85 nurses were trained to prescribe first-line ART, 14 additional HCs were able to introduce ART. RECOMMENDATIONS Recommendations are summarized by the four results areas in the development hypothesis. Quality and Timeliness of Routine Data Reporting  Draft a retention strategy, job descriptions, and job aids for data managers.  Build the capacity of data managers in information technology (IT), M&E, data analysis, data cleaning, data quality checks, and data for decision making.  Draft national guidelines for the Health Data Management System.  Support OpenMRS for current operations and continue to improve TRACnet data quality. Financial and General Management  Consolidate all costs by activity, including use of donor funds, in the monthly cash report.  Put in place a financial management system to monitor and report on subcontracts at the community, HC, and DH levels  Build the capacity of district administrative staff.  Manage the expectations of communities, providers, facilities, district management, the central MOH, and RBC by establishing strong lines of communication and relationships. Service Delivery  Support adoption of quality assurance/quality improvement (QA/QI) roll-out, SGBV one-stop centers, task shifting, and integration of FP/RH/MCH/SGBV/TB/malaria and HIV services. USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION xv  Ensure monthly monitoring of the quality of prenatal care, postnatal care follow-up, vaccinations, hospitalization, nutrition, and provider-initiated testing and counseling (PITC) indicators.  Revise the supervision strategy to improve efficiency and effectiveness.  Continue to collect data for the mother-infant tracking system. Epidemiologic Outcomes  Keep upgrading programs to ensure that they are state-of-the-art.  Maintain infrastructure and equipment. xvi USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 1 I. INTRODUCTION1 USAID/Rwanda requested this end-of-project performance evaluation of the five-year HIV Clinical Services Project (HCSP), which was carried out October 6–November 17, 2012, by a GH Tech Bridge II Project consulting team. The evaluation team addressed the following development hypothesis: If USAID support via HCSP built capacity in district and national level Rwandan institutions to support and manage Rwanda’s health systems and expanded quality health service delivery, then the results would be improvements in (1) the quality and timeliness of routine data reporting; (2) financial management; (3) service delivery and technical capacity; and (4) epidemiological outcomes among target populations. The evaluation, which was both formative and summative, had two purposes: (1) measure the efficiency, effectiveness, and relevance of resource allocation for programming to the stakeholders, and (2) be a resource for learning and capturing best practices. This report presents the evaluation findings and complements a recent capacity assessment to provide baseline information for the new Family Health Project (FHP). The lessons learned and recommendations will influence decision-making in the next phases of country programming and in similar projects worldwide. The evaluation also informs district and facility technical, programmatic and administrative support priorities. 1 The Introduction summarizes information from the Scope of Work which can be viewed in Annex A. 2 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 3 II. BACKGROUND The objective of the HCSP was to build capacity in national and district Rwandan institutions to support and manage health systems, and to expand HIV/AIDS clinical activities and clinical service capacity for patient care in selected districts. This project built upon the PEPFAR district￾level clinical service activities supported by USAID up to March 2007. The HCSP was implemented by cooperative agreements with three partners, Family Health International 360 (FHI360); the Elizabeth Glaser Pediatric AIDS Foundation (EGPAF); and IntraHealth. The three implementing partners (IPs) provided services to 164 health facilities, most within 14 of Rwanda’s 30 districts. This five-year project began in June 2007 and was scheduled to end December 31, 2012, after a seven-month cost extension for all three IPs. Increased capacity of all levels of Rwandan institutions to deliver expanded HIV/AIDS clinical activities Numbers of new cases of HIV/AIDS decreased Numbers of PLWHA receiving, care and support increased Numbers of HIV/AIDS cases receiving treatment increased The Results Framework above depicts the goal and expected results of HCSP. Maternal and child health (MCH) and family planning (FP) funds were obligated in year three to ensure integration of those areas with HIV and AIDS services, and as such are not included in the original Results Framework. The IPs collaborated with district health systems and national partners to increase district capacity to provide core clinical services in accordance with established GOR policies, norms, and standards; policies and guidance of the USAID Office of the Global AIDS Coordinator (OGAC); and international norms, standards, and best practices. Core clinical services included prevention of mother-to-child transmission (PMTCT); counseling and testing (CT); antiretroviral therapy (ART) services; treatment of TB/HIV; and care and support. The IPs also supported non-HIV services in MCH, family planning/reproductive health (FP/RH); and sexual and gender￾based violence (SGBV), as part of integrated programming. Provision of core clinical services was accompanied by provision of a District Package in each of the 14 districts where HCSP operated. The package included support to the district health committees, the Mayor’s Office, district pharmacy personnel, and the clinical supervisors and trainers based at the District Hospital, all of which make up the district health system. In addition to the services offered to the districts, HCSP supported hospitals to provide core HIV clinical services for their clients. The hospitals developed integrated plans, which included non-HIV service delivery especially focused on training, supervision, and mentorship for hospital and health center staff. In turn, health center staff provided not only primary health care (e.g., 4 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION MCH, FP, SGBV, health education) but also many HIV clinical services, such as patient clinical assessments, laboratory services, CT, and PMTCT services as well as ART. The three IPs supported health care services in the following districts: EGPAF: Bugesera, Gatsibo, Ngoma, Kayonza, and Rwamagana Districts, and six health facilities in Kigali City (Kicukiro and Nyarugenge) FHI 360: Nyamagabe, Nyaruguru, Ruhango, Muhanga, and Kamonyi and five health centers in Kicukiro, Nyarugenge, and Nyanza districts. IntraHealth: Gasabo, Gicumbi, Nyagatare, and Rulindo Figure 1: Districts Served by the HIV Clinical Services Project. Note: Arrows refer to the implementing partner supplying services: red = Intra-Health; blue = EGPAF; yellow: FHI 360. The IPs had the same ultimate objectives:  Build capacity in national and district institutions to support and manage Rwanda’s health systems.  Expand HIV/AIDS clinical activities and clinical service capacity for patient care in assigned district health networks (DHNs). During the project, annual workplans drawn up in consultation with the Ministry of Health (MOH) and USAID, expanded services to create a more integrated approach to service delivery, hence the inclusion of MCH and FP/RH. Each IP brought its own corporate strength and USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 5 adopted different interventions to reach the goals: IntraHealth concentrated on SGBV, palliative care, and FP; EGPAF on pediatric HIV/AIDS and PMTCT; and FHI 360 on FP/RH and research activities. Since Rwanda emerged from the devastation of the genocide in the mid-1990s, the Government (GOR) has systematically moved forward with an ambitious plan, the Economic Development and Poverty Reduction Strategy (EDPRS I 2008-12), which provides the first framework for implementing Vision 2020. The first Health Sector Strategic Plan (HSSP I 2005–09) had been designed to maximize prevention, build capacity, and improve the quality of care through community-based health insurance (mutuelles); performance-based financing (PBF); and quality assurance (QA) as cornerstones. The HSSP II (2009–12), the follow-on, was drafted to support the EDPRSI. Rwanda’s third HSSP offers strategic guidance to the health sector for the six years from July 2012 and June 2018. HSSP III has been inspired and guided by VISION 2020, which is designed to make Rwanda a middle-income country by 2020. HSSP III also draws on the 2004 Rwandan Health Policy and the priorities set by the Economic Development and Poverty Reduction Strategy (EDPRS 2008–12). The MOH identified five priorities in the draft HSSP III, which are based on Rwanda’s disease burdens and its epidemiological profile:  Achieve MDGs 1 (nutrition), 4 (child mortality reduction), 5 (MCH) and 6 (disease control) by 2015.  Improve access to health services (financial, geographic, community health).  Improve quality of health provision (QA, training, medical equipment, supervision).  Reinforce institutional strengthening (especially of district health services).  Improve quantity and quality of human resources for health (planning, quality, management). Rwanda has moved into the third stage of decentralization, devolving management and budgeting to the district level while the central MOH continues as steward. PBF first targeted district hospitals and health centers and has since been rolled out to community health workers (CHW). The reforms have brought remarkable progress. Average total fertility rate (TFR) dropped from 6.1 in 2005 to 4.6 in 2010. Modern contraceptive prevalence rate (CPR) is 45% and the number of women giving birth in the presence of a skilled attendant increased from 39% in 2005 to 69% in 2010 (2010 DHS), resulting in improved maternal health. Ninety percent (90%) of children aged 12–23 months are fully vaccinated, which between 2005 and 2010 brought infant mortality down from 86 to 50, and under-5 mortality from 152 to 76; meanwhile, children diagnosed as stunted decreased from 51% to 44%; as wasted from 5% to 3%; and as underweight from 18% to 11% (WHO’s new growth standard was applied to 2005 DHS data to compare with the 2010 RDHS). 6 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION By June 2012, 467 (94.5%) of health facilities were offering PMTCT services, an increase of 55 from the previous year. By September 2012, about 336 health facilities (62%) in Rwanda had an HIV testing system and 296 (55%) were providing HIV care and support.2 Malaria prevention has also improved: 73% of pregnant women aged 15–49 and 71% of children under 5 slept under a mosquito net the night before the survey (DHS 2010). Unfortunately, undernutrition and maternal, neonatal, infant, and under 5-mortality are still high. 2 HIV Annual Report 2011; 2012. USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 7 III. METHODOLOGY The evaluation team used both qualitative and quantitative methods and used the questions in the final SOW as the basic question set, with follow-up questions as required, and interview guides targeted to specific populations. Data sources were identified to respond to each question. A team of three experienced consultants worked together to conduct this evaluation, drawing on the datasets of each project and data from the 2010 DHS, the health management information system, Monitoring and Evaluation Management Systems (MEMS), TRACnet, PBF, IQ Chart and other assessments. Because a baseline study was not conducted, the performance evaluation focused on descriptive and normative questions. The assessment included the following: REVIEW OF DOCUMENTS Relevant documents were assembled for review and analysis, among them HCSP project documents (e.g., cooperative agreements, work plans, quarterly and annual reports, monitoring and evaluation (M&E) strategies and assessments, etc., from each of the three IPs); USAID reports; GOR documents, such as HSSP I, II, and III, Vision 2020, HIV/AIDS National Strategic Plan); and documents from donors, including the World Bank and the United Nations (UNAIDS, UNICEF, UNDP, UNODC). See Annex C for a complete list of documents consulted. INTERVIEWS The evaluation team interviewed IP teams and chiefs of party. They also met with Family Health Project (FHP), senior officers from the central ministry, USAID project staff, and staff of other donors (see Annex B for a list of those interviewed). SITE VISITS Visits were made to a district health unit and a district pharmacy, three district hospitals, and six Health Centers in project districts. Because the evaluation team had limited time, needed to visit specific types of sites, and visit an equal number of sites supported by each IP, it was not feasible to use random selection. Instead, initial meetings were held with each IP during the team’s first week in-country in which they were asked for a list of five to six sites, including district hospitals, health centers and district health units. The sites were to showcase innovations and best practices, successful facilities, and those that still faced challenges. From the recommendations provided by the IPs, the team chose nine sites, which were proposed to USAID for further discussion, after which some changes were made to better reflect the criteria and eliminate any possible appearance of bias. The final list was approved by USAID and the MOH. The MOH notified all facilities of the pending visit. Table 1 shows the district, facility, facility type, and IP providing technical assistance. 8 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION Table 1: Site Visit Locations District Facility Facility Type IP Muhanga Kabguyi District Hospital, Muhanga DHU and Pharmacy Hospital, DHU, Pharmacy FHI 360 Nyarugenge Butamwa Health Center Health Center EGPAF Nyagatare Nyagatare District Hospital Hospital IntraHealth Rulindo Rukozo Health Center Health Center IntraHealth Gicumbe Mukondo Health Center Health Center IntraHealth Ngoma Kibungo District Hospital, Ngoma Hospital EGPAF Rwamagana Nzige Health Center Health Center EGPAF Nyaruguru Ngoma Health Center Health Center FHI 360 Ruhango Ruhango health Center Health Center FHI 360 During site visits, the team met providers, managers, and a few clients. The team also reviewed management and quality reports, meeting notes, standards, the strategy and business plan, and facility data. TOOLS Using a combination of qualitative and quantitative methods with the questions in the SOW as a questionnaire guide, the evaluators constructed tables using MEMS data to look at PEPFAR indicator trends over the last three years. Before MEMS began, indicators were not comparable. The tables were computed for each IP separately and the IPs as a group. Finally, the team drew up a brief guide to use with clients. However, because clients attend clinic very early in the morning, due to the travel time from Kigali the team was unable to speak with many. DATA ANALYSIS Data collected by the evaluation team, observations of team members, and informal discussions were analyzed and consolidated as the basis for formulating findings and recommendations. The following process was used to analyze the data: 1. Description and analysis: Describing and analyzing both qualitative and quantitative findings involves organizing raw data so as to reveal basic patterns. 2. Conclusion: What do the findings mean? What are possible explanations of the findings? Conclusions go beyond the data to add context, determine meaning, and ascertain substantive effects based on deduction or inference. 3. Recommendations: The final step adds action to analysis and conclusions. What should be done? What are the action implications of the findings? All recommendations follow from and are grounded in the data. Note: quantitative data analysis was somewhat limited due to missing data and inconsistencies between datasets, especially in the early years. To deal with the data issues, the team chose to use the MEMS database exclusively for data analysis. The purpose of the MEMS Project is to help the USG Rwanda Team to design and implement a comprehensive performance management, monitoring, and reporting program. USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 9 TRIANGULATION Triangulation refers to double or triple checking of results using different methods, data sources, and experts. To increase the accuracy and credibility of the evaluation findings, whenever possible the team used several types of triangulation: 1. Methodological triangulation: data were collected by two or three methods (e.g., desk review, interviews, secondary data analysis, observation) 2. Data source triangulation: project data were collected from a variety of sources (not only desk review, USAID and GOR documents; and interviews, but also trend analysis of PEPFAR indicators in the MEMS database 3. Investigator triangulation: because the three members of the evaluation team had different backgrounds, it was possible to evaluate data from different perspectives. The team collected a large evidence base, using a systematic approach to recording and analyzing the information. Whenever possible, the information was triangulated against secondary sources to reduce bias and cover gaps. The final debriefing gave the IPs a chance to comment on the preliminary findings, conclusions, and recommendations. Circulating the PowerPoint debriefing document for comment ensured that conclusions and recommendations were as accurate and relevant as possible. Indicator targets and achievements for each IP can be found in Annex E. CHALLENGES AND LIMITATIONS Qualitative data allow for a comprehensive and multifaceted understanding of people’s experiences with the program. However, because collecting detailed data is time-consuming, the evaluation team had to limit the number of interviews and field visits they could conduct. Qualitative methodologies allow evaluators to bring their own perceptions, but also their own biases, to a study. To minimize possible distortion of the findings, the team discussed the potential biases each brought and monitored each other’s neutrality. On the team were members from Rwanda, Uganda, and the U.S., which helped in understanding the environment (political, cultural, and socioeconomic) in which HCSP operated. The team was not able to go through the Institutional Review Board (IRB) process due to time restrictions. The project did not include a rigorous counterfactual at initiation, and the HCSP IPs had not been asked to capture financial data down to a sufficient level of detail to permit in￾depth financial analysis (e.g., cost per supervisory visit) within the time allowed for the evaluation. These factors limited the methodologies available and prevented the team from conducting the requested cost- effectiveness analysis. The team did conduct secondary data and trend analysis where possible. EGPAF and IntraHealth used IQ Chart to track patient data; FHI 360 did not use IQ Chart or other patient data programs, instead waiting for the expected roll out of the OpenMRS system, which did not happen until HCSP year 5. The team therefore did not evaluate patient data. Patient information was removed to preserve privacy before the team received the data for initial review. 10 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION The team used several datasets to try to analyze facility data but found significant numbers of outliers as well as missing data. The data problems were more prevalent in the earlier years of the project. The MOH and its partners have since worked extensively to improve data quality by hiring facility data managers and building their capacity. However, these issues affected the team’s ability to carry out sophisticated data analysis, which is why trend analyses of PEPFAR indicators from the MEMS dataset were conducted. USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 11 IV. RESULTS, FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS RESULTS The development hypothesis was that the project would result in improvements in (1) the quality and timeliness of routine data reporting; (2) financial management; (3) service delivery and technical capacity; and (4) epidemiological outcomes among target populations. The evaluation team found that in all four areas the HCSP more than met project goals and was overall highly successful. During the life of the project, services expanded to integrate MCH and FP/RH into HIV service delivery and benefited from the fact that each IP brought unique corporate strengths to develop innovative approaches to service delivery. FP/HIV Integration Model Became National All three IPs worked with the MOH technical working group (TWG) on integration, provided needed but different activities EGPAF synthesized the model that received MOH approval; IntraHealth conducted needs assessments to determine training requirements for the new model and conducted much of the training; and FHI 360 seconded staff to the MOH. In early 2008 the HCSP led development of an FP/HIV integration model that was approved by the MOH. EGPAF hosted a meeting with all USG partners and the MOH to discuss challenges to integration. The revised model addresses these challenges; it maximizes women’s access to contraceptives by providing FP refresher trainings so that all health providers can make appropriate FP referrals and nurses are able to refill routine prescriptions for oral and injectable contraceptives at all service delivery points and at all times. The project also helped improve the organization of facility FP/HIV integration services to ensure that new patients, long-term methods, and complicated cases are handled by nurses who received FP/HIV integration training. FHI 360 supported a seconded staff to the MOH to oversee the national HIV/FP integration model. FP/MCH/HIV needs were assessed to ascertain current FP practices and the degree of integration with HIV services. It was determined that standardized FP/HIV training was needed throughout all health facilities; these results guided the development and rollout of FP/MCH/HIV training and tool development. Advocacy to support the drafting of standardized policy guidelines, tools, protocols, and coordinated supervision was targeted to MOH MCH/immunization/nutrition and HIV/AIDS integration in health systems task forces. This work led to the creation of integrated registers and other tools and manuals. In collaboration with the MOH and other clinical partners, HCSP also helped to draw up the 2009 national FP/HIV integration implementation and training plans. The HCSP trained 165 site staff on the new integration model. As of September 30, 2009, among the 6,119 HIV+ adult female clients enrolled in care and treatment at HCSP-supported sites, 3,329 (54%) were using an FP method. 12 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION Infant and Young Child Feeding Folded into the MIYCN The HCSP provided technical support to the Prevention Unit at the Rwanda Biomedical Center Institute of HIV/AIDS, Disease Prevention & Control (RBC/IHDPC) to draft a Questions and Answers guide on the new PMTCT guidelines which will serve as a reference for PMTCT providers. The manual answers common questions and identifies challenges facility providers face. EGPAF also led the development of Maternal Infant and Young Child Nutrition (MIYCN) materials that explain in simple language nutrition counseling and support in the context of HIV prevention. This publication was developed with support from the MOH PMTCT TWG and translated into French. Male Involvement Strategies from HCSP programs to involve male partners in PMTCT services became valuable models for replication in FP/RH programs. The project used integrated service models to increase partner rates of participation in FP counseling. When they come to the health center for HIV counseling and testing, a couple is asked if they are using an FP method. If not, the couple is triaged to FP services, where they are counseled together about FP, RH, and MCH services. From July 2002 to June 2012, 84% of male partners in PMTCT clinics were counseled and tested for HIV. Many sites routinely have over 90% of male partners attending services Male involvement in PMTCT services at many sites has become routine, with the majority of women’s husbands or partners coming with them without the need for an invitation letter. For the rest, the women may not have a regular partner, the partner may work outside the community, or he may not want to come. Future projects could better document the relationship status of women to better understand this remarkable success and improve the chance of it being adopted in other countries. Neonatal and Child Death Audit Program At the MOH, EGPAF seconds one staff member to the MCH department to do national infant death audits. The staff member has continued to support roll-out of the national infant death audit program to all district hospitals in Rwanda and has analyzed data for 173 infant deaths. The majority of neonatal deaths were due to prematurity and asphyxia during birth. Palliative Care With technical leadership from Mildmay International and in collaboration with the MOH, the project trained 547 health service providers in HIV-related palliative care and 207 others (CHWs, members of PAQ (Partenariat pour l’Amelioration de la Qualité, Partnership for Quality Improvement) teams, and PLWHA volunteers) in home-based care. Eight palliative care indicators were drafted and piloted in five sites. They documented strong community-health center linkages but underscored the need for enhanced pain management in Rwanda, especially increased availability of anesthetics. As a direct result of an HCSP study tour for three MOH representatives to learn best practices in pain management at the Mildmay Center in Uganda, oral morphine was added to Rwanda’s Essential Medicines List. The model Pediatric Palliative Care Center at Kibagabaga Hospital was inaugurated in August 2009. With Mildmay International, the HCSP supported palliative care services at project sites and provided technical leadership in drafting Rwanda’s first palliative care policy, guidelines, and USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 13 strategic plan. Within a year, the hospital added adult palliative care. Hospital staff created a multidisciplinary team that meets weekly to ensure a holistic approach to caring for adults with chronic illnesses. Kibagabaga Hospital also supported neighboring health centers, shared best practices, and provided mentorship to enable them to initiate their own palliative care units, roll out support groups, decrease clients lost to follow-up, and strengthen community linkages. Pediatric Mentoring Model The HCSP started a PMTCT/pediatric mentorship program in Ngoma district in order to improve the PMTCT cascade and ensure follow-up of pediatric HIV patients. The project trained a team at the Kibungo DH to carry out mentorship activities at sites in the catchment area. The team drew up indicators to be followed up on monthly that cover prenatal care, postnatal follow-up, vaccination, hospitalization, nutrition, and provider-initiated testing and counseling (PITC). Pediatric Psychosocial Program The HCSP launched pediatric psychosocial programs with a ceremony at Masaka and by September 2009 had expanded the program to 21 sites serving over 728 HIV+ children in 40 support groups, organized by age. Pediatric psychosocial support includes support for announcing children’s HIV results, counseling for children and families, and recreational and educational activities, including a picnic with TRAC Plus and Kibungo DH for 22 HIV+ children. The first Rwanda Ariel’s Camp for HIV+ children was held in July in collaboration with TRAC Plus and the CNLS (Commission Nationale de Lutte Contre le SIDA, National Commission for the Fight Against AIDS), with 28 children and their psychosocial counselors participating. The children came together for three days of recreation, education, and support in an atmosphere free from HIV-associated stigma. PMTCT The HCSP’s PMTCT mother-infant tracking database is a major systems improvement, the benefits of which should trickle down into clinical care. Tracking mother-infant pairs makes it easier to monitor service delivery, which should reduce missed opportunities for treatment, ensure early initiation of prophylaxis or treatment, and improve morbidity among HIV-exposed infants. The system also facilitates tracking of patients as they enter and leave the continuum of care, especially HIV+ mothers who come to seek care for their infants. Those who use the database can at any time access a current picture of PMTCT services being provided at each facility and will be able to follow up as needed. Prenuptial Counseling In its first year the HCSP also supported prenuptial counseling for couples. The counseling covered counseling on gender and parental roles, discussions of FP/RH, and HIV counseling and testing. Booklet for Counselors In the last fiscal year, RBC/IHDPC, the Rwanda Health Communication Center (RHCC), the Rwandan Network of People Living with HIV/AIDS (RRP+), Kacyiru Health Center, and the HCSP worked together to publish a booklet in Kinyarwanda to guide positive prevention counseling in clinic and community settings. The HCSP funded production of 3,000 copies and another 8,000 were produced with funds from the MOH/RBC/RHCC. In the fourth quarter, MOH and RBC distributed the booklets to IPs and associations of HIV+ people. In collaboration 14 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION with the MOH and the RHCC, the HCSP also held an orientation meeting for selected health facilities and PLWHA associations. Roll-out of OpenMRS The MOH is committed to building a strong national Electronic Medical Records (EMR) system and has chosen the Open Medical Record System (OpenMRS) to accomplish the goals of its e￾Health Strategic Plan for 2009–12. The HCSP worked closely with the MOH to train 128 people, including 68 data managers and HIV service providers from 31 facilities, in use of the OpenMRS system. To facilitate data entry, the HCSP also supported 24 MOH data entry clerks, who temporarily helped health facilities to reduce backlogs. The HCSP continued to scale up introduction of OpenMRS and until the end of the project provided TA to both health facilities and the MOH on use of the new EMR system. SGBV The HCSP worked with the MOH to put in place a one-stop model to address SGBV. The model brought medical care, police, lawyers, and counseling and testing services to the client at the clinic, which increased attendance and improved conviction rates. As co-chair of the MOH GBV Committee, HCSP facilitated the conception and content development of the national provider manual for providing services to SGBV survivors. Other SGBV Initiatives included comprehensive provider training sessions, regular collection of service data, and dissemination of SGBV community information, education and communication materials. Services began once the MOH signed off on the new SGBV manual in January 2010. The one-stop SGBV service center at Nyagatare District Hospital used radio spots to educate people about the importance of seeking health services after violence. Health center personnel also sensitize their clients about SGBV services during all consultations. The IPs provided technical and financial assistance to sensitize school and religious leaders to SGBV in each of their districts. Once they completed the sensitization session, participants committed to supporting sector anti-SGBV committees and other anti-SGBV programming in their communities. The program increased collaboration between health facility providers, police, and legal institutions by including them at the planning stage. Stakeholders from these institutions took the sensitization trainings and health care providers learned how the legal system prosecutes SGBV cases and how they can better assist the legal system. For example, the SGBV training manual includes information about forensic examinations. In 2011, 908 SGBV cases received care and support through USG-supported MOH facilities. In 2012, the number of cases increased to 1,107—246 percent of the 2012 target of 450 cases. The project trained a total of 2,542 people on SGBV. USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 15 RESEARCH QUESTIONS, FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS To what extent has the project been effective in increasing the technical and administrative capacity of Rwandan health systems, including improving quality and quantity of health services delivery? The project has been very effective. In 2010, 154 HCSP facilities3 had 186,772 deliveries. In 2011, 156 facilities had 131,910. In 2012, 156 facilities had 111,279 deliveries.4 The ability of the IPs to meet their indicator targets is a function of the process used to set the target, the quality of service provision (follow-up of those not coming to clinics), and other factors. Since the targets are somewhat arbitrary in terms of the actual number of patients in a given category, percent of target met should not be mistaken for percent of coverage. With some exceptions the IPs met the majority of their targets (see Figures 2–4). For 2010–12 infants born to HIV+ mothers who were tested for HIV within 12 months of birth range from 94% to 74 % of the target (Figure 2). Figure 2: Infants of HIV+ Mothers Tested for HIV within 12 Months of Birth The number and percentage of infants born to HIV+ mothers who started on CTX are consistent at 63%, 63%, and 62% of the target over the three years (see Figure 3) although the raw numbers fell. Since the number of HIV+ persons of all ages receiving CTX and the percentage of the target met are quite high (Figure 4), there seems to have been a problem in setting targets for newborns or of mothers bringing newborns back in for treatment. 3 Number of PMTCT sites using MEMS reports. 4 Number of deliveries, using MEMS reports, MCH program areas. 16 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION Figure 3: Infants Born to HIV+ Mothers Started on CTX Prophylaxis within Two Months of Birth Figure 4: Number of HIV+ Persons Receiving Cotrimoxazole While these figures show missed targets, there are many more examples where targets were surpassed. See Annex E for PEPFAR indicators, targets, and achievements for all IPs for the last three program years. National Data The project has been highly effective in increasing technical and administrative capacity, improving both the quality and the quantity of service delivery. In 2007 there were 14,358 new infections compared to only 10,004 in 2012, and today 98.3% of pregnant women receiving ANC are tested.5 5 Sabin Nsanzimana, MD, Head of HIV, AIDS, STIs & OBBI Division/RBC. USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 17 Numbers of mothers delivering in a health facility have increased dramatically, while the percent testing positive has gone down steadily. The trends follow what is happening in the rest of Rwanda where, according to TRACnet data, prevalence has decreased. The national data cover all sites, whether supported by the HCSP, other donors, or the Global Fund to Fight AIDS, TB, and Malaria. The increase in in-facility deliveries is related to community-based health insurance, which has worked to improve access. The trend also may have been influenced by improved quality, which is often a good proxy for patient satisfaction since regardless of insurance status women are more likely to deliver at home when they are treated poorly or feel they are receiving poor quality care. The drop in percentage of target for infants receiving an HIV test was a problem for one IP. The expected number of HIV+ pregnant women delivering in the HCs it supported was lower than expected, with some women choosing to deliver in other HCs. As a result, there were fewer infants to test. As this trend was not seen with the other two IPs, there may have been a quality problem with infants lost to follow-up. Community based follow-up in the FHP project should decrease this number. The number of HIV+ mothers started on CTX did not meet the target. The targets were set based on the expected number of HIV+ women giving birth. Because expectations were consistently high, so were targets. Part of the discrepancy is also related to infants lost to follow-up or not brought back within two months Again, community-based follow-up should help with the lost to follow-up issue. Were the objectives met during the five years of the project? If yes, to what extent did HCSP IP interventions in human and institutional capacity-building help in their attainment? Which interventions specifically contributed to attainment? If objectives were not attained, why not (e.g., no interventions deployed, interventions failed)? What needs to happen to achieve objectives? The evaluation team used the yearly PEPFAR targets and achievements to determine whether the project met its objectives. See Annex E for specific findings for each IP targets and achievement in the last three years. (The MEMS database, which the team used for consistency, has only been in existence for three years. Prior to that, another system was used to collect targets and achievements with somewhat different indicators.) Training and mentoring built the capacity of service providers to provide both first- and second￾line ART to adults and children in health centers and hospitals. Task shifting allowed nurses to provide ART and post-exposure prophylaxis (PEP) in health centers without doctors. The number of people receiving PEP almost tripled over the last three years of the project (Figure 5). 18 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION Figure 5: Rwandans given Post-exposure Prophylaxis In the last three years of the project, 34,2676 children and adults were receiving first- and second-line ART therapy The HCSP met the project objectives. Capacity was clearly built, access to HIV services increased, modern-method CPR increased to 45%, and HIV prevalence appeared to have decreased. According to the RBC, there is virtually 100% access to ARVs. In 2007 about 82.2% of patients in need of ARVs received them. By 2012 this had increased to 90%. What were the strengths and weaknesses of this project (especially in terms of those aspects detailed in the supporting questions)? Strengths The strengths, too many to list here, are described below under the supporting questions. The achievements were the result of a dynamic partnership between the MOH and the IPs for the HCSP. Many of the results presented are national data to which the HCSPs contributed. A few of the most prominent project strengths were:  Successful integration of services, with all TB patients being tested for HIV and most HIV patients being tested for TB.  Almost all pregnant women in project sites receiving PMTCT services and, if positive, both mother and baby receiving ARVs.  One-stop SGBV services introduced at HCs and hospitals.  Modern method CPR increased to 45 percent after service integration.  100% access to ARVs. 6 Data from MEMS database using achievement by program area/partners ART services indicators. USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 19  Pediatric and adult palliative care available.  Improved facility financial and data reporting.  80 percent of male partners accompanying partners wives to ANC, where they are offered testing (they are not forced to accept).  Some physician tasks shifted to nurses. Weaknesses While the IPs did collaborate, they tended to implement activities in certain program areas, such as PMTCT, in different ways, as they traditionally approached them. Only when the GOR adopted an innovative approach of one IP did the others also adopt it. The IPs did collaborate on major initiatives, taking on activities in which they were traditionally more experienced. For example, all three worked on different aspects of FP/HIV integration. In the area of data capture, two IPs chose to use IQ charting and the third waited for introduction of the new HMIS system. Capacity building centered on classroom training, followed by “mentoring.” In this situation, mentoring was defined as spending 80–90% of the program specialists’ time working alongside those being mentored, which tended to generate dependence. For example, some finance officers would not complete the monthly cash reports unless the IP specialist was available either by phone or in person to check it. HCSP staff accompanied supervisors on most visits, again generating dependence. When project activities ended September 30, 2012, supervisory and physician visits tended to end as well, with the excuse that there was no petrol, or no vehicle. Data quality is still problematic. Data managers do not know basic descriptive statistics that would help them identify outliers and improve data quality although they are improving. Conclusion The improvements in data management and quality, financial reporting, and access to the full package of high-quality HIV, FP, and RH services were very impressive. While services were not provided 100% of the time, the capacity of staff in project-supported clinics to provide services compared to the beginning of the project is a significant achievement. Although the project on the whole was very successful, there are some areas where interventions could be made more efficient and effective without running the risk of creating dependence. Recommendations  Determine the feasibility of identifying alternative supervision strategies that use technology.  Re-examine how mentoring is carried out. Mentoring can be highly effective at building capacity without requiring significant numbers of staff. Again there are technology applications that can be used.  Provide additional capacity-building activities for data managers to improve data quality and increase data use. To the extent that the project has been successful in building capacity, is the GOR at all levels now in a position for a smooth transition? 20 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION Confounding issues in capacity improvement are these:  Not all facilities received the same amount of support. Some that have had over 10 years of support are definitely ready to transition; less so others who began working with the project in the last one to two years when resources were scarce.  Staff rotation and turnover resulted in a continuous need for orientation and training as new staff came into the facility. HCSP never found an efficient way to do this without repeating trainings. However, one HC started PMTCT services without first been trained; its staff took it upon themselves to go to the DH and surrounding HCs to learn how to provide the services.  Facilities with strong managers function at higher levels than those with weak managers. For example, at one DH, the staff all seemed to go to lunch at the same time, leaving babies in incubators with no caregiver to monitor them. At the same hospital, a woman came in with a small infant that had been bitten in the head by a dog and could find no one to help her until the hospital director saw her while giving the evaluation team a tour.  At one site in a very remote area, the HC functioned well above the expected level. The director had tested business and strategic plans that he used daily. For instance, he knew where to find reporting templates, quality of care supervision reports, and follow-up reports to ensure that an issue was corrected and documented. Conclusions  The project has been very successful in capacity-building. However, because all institutions have not improved their capacity at the same pace, they are not functioning on a level playing field. However, whatever improvements have been achieved will definitely help with transition, although some will need continuing TA. The recent capacity assessment highlights areas needing further support.  The real success of the project is that it has promoted ownership, self-motivation, independence and the desire to provide quality services, thanks to strong and fair managers who show trust in their staff. These qualities will make transition smoother. What opportunities and constraints face FHP and other future USAID work? Opportunities FHP is inheriting a set of facilities that are mostly high-functioning, which will transition to the MOH in the near future. However, some will definitely need continued TA. The recent completed capacity assessment will help identify the technical areas and facilities that are most in need. The task is to keep the facilities functioning at their current level or better. This is both an opportunity, in that few projects start with such a strong base, and a constraint, in that having transitioned to the MOH these facilities and will no longer have the support experienced with the HCSP, although TA will definitely be available. The GOR generally and the Minister of Health in particular have created a very supportive environment for change and improvement. USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 21 Constraints The government has decentralized. The MOH and the DHU are fine-tuning their roles as work goes on. As a result, there are difficulties with communication both across and within levels. RESEARCH QUESTIONS, FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS Supporting Evaluation Questions Was there collaboration between IPs? If so, what was it and what outcomes were affected? Within the HCSP project, IPs were able to work together on many common issues. The collaboration on policy development, guidelines, and program implementation materials was supported through the MOH and its working groups. The ministry and its institutions, for example the RBC, provided both leadership and coordination for the expansion of HIV services. Experts from the IPs participated in the thematic technical groups that were used as discussion forums coordinated by the MOH, and the COPs of these organizations had informal exchanges to harmonize their project implementation plans. The M&E officers from the three IPs also met to discuss the indicators proposed by USAID. However, the IPs tended not to implement activities in the same way unless he MOH adopted a specific innovation. Conclusions  The differences in implementation extended to transport. Some IPs did day-to- day or long￾term car rentals for supervision or used their own vehicles, both of which created dependence. However, the rentals were very expensive.  Patient data was captured by only two of the three IPs.  The HCSP was implemented in a harmonized format, facilitating expansion of clinical services and delivery in 14 districts of Rwanda. This also facilitated coordination of the program and the direct and indirect engagement of various stakeholders.  However, organization-specific approaches resulted in some inefficiencies and may make it harder for FHP and the MOH to maintain and strengthen services at the point of delivery. Recommendation Providing services in a different manner did showcase alternative implementation strategies. Were program descriptions followed? Why or why not? In carrying out the HCSP project, the IPs did a good job of following the USAID program description. The project was adapted in terms of strategy, but in response to recommendations from the MOH and USAID, but the basic package was implemented according to plan. During year three, USAID obligated additional non-PEPFAR funds to integrate MCH and FP services with HIV services, which added another dimension to the results framework to that present at the beginning of the project. Did these projects fulfill USAID/Rwanda and IP performance monitoring plans? If not, what were the areas of underperformance? 22 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION The HCSP performance plan agreed upon with USAID in the signed contract and the adaptations made in relation to new funding and strategy were followed correctly. For certain indicators, FHI360 was able to do more than foreseen, which was approved by USAID. For example, in the HIV services expansion, the target was for FHI to extend services to 45 health facilities by project end using the new implementation strategies, e.g., task shifting. However, FHI was able to extend HIV services to 68 facilities. Annex E reports on performance on every PEPFAR indicator. EGPAF was able to extend services to 51 facilities from an original target of 35. Some targets were not met, but this was a result of over-estimation of service utilization. For example, the number of HIV patients screened for TB depends on the number of HIV patients receiving care and treatment. To what extent was routine clinical data (e.g., HMIS, IQ chart) used for management and decision-making and project improvement? The HCSP project supported data management at the facility level, in particular for reporting HIV activities. The support for monitoring health unit and district hospital activities made it possible to mentor the data managers and use the data to improve services. HCSP support complemented other support provided by the MOH, MEMS, and the Global Fund to improve district data management systems and data quality. Another important HCSP activity was support of district coordination meetings. These monthly forums allowed the district to organize meetings with district health centers and district hospitals. During the meetings the health situation in every supported facility was assessed on the basis of the information generated by the health centers and hospitals using such tools as HMIS, TRACnet, IQ Chart, MEMS, and PBF data. Issues were identified and a plan developed to correct problems. Follow￾up occurred at the next meeting. Although it is clearly improving, data quality has continued to be a problem, which makes it harder to use the data. Currently managers are unable to adequately clean the data by running descriptive statistics. The same data must be entered into multiple systems, which is very inefficient and worsens the workload. For example, the same facility data is entered into both MEMS and TRACnet. FHI did not capture patient data. Both EGPAF and IntraHealth used IQ Chart. The facilities used the data to monitor indicators, such as receipt of a CD4 count every six months, ART regimen, etc. If they identified patients who did not receive appropriate care per the protocols, they looked for trends to see this was a one-time or a chronic issue. Each facility reported their numbers and issues at the monthly district management meeting. Most facilities visited by the evaluation team posted the month’s routine graphs on the wall of the data manager’s office. While everyone stated that they used the data for decision making, very few could produce the quality reports where the data were used as the basis for discussion. They did use the data for the monthly coordination meetings, and minutes of those meetings were available. Conclusions The consistent improvement in data quality is shown by the annual data quality audits performed by MEMS. USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 23 TRACnet data quality still needs improvement. The workload for data managers is very heavy, which seems to be undermining accuracy. Workload issues are not solved because part-time data managers have other duties. Because they must enter the same data into multiple systems, the workload is heavier, which leads to errors. According to the MOH, consolidated HMIS data are not yet available to those outside the MOH, although data managers can look at their own data. The IPs did not have access to the HMIS data. Each facility presents its data for discussion at the monthly district coordination meeting. If issues are identified, a follow-up plan is developed, implemented and discussed at the monthly meeting. Thus, the data are clearly being used, although with additional support usage could well expand to cover broader facility issues. It is unclear if the district meetings will continue without project support and transport. Recommendations Policy  A strategy to retain data managers is necessary.  National guidelines should be drafted for the Health Data Management System. Capacity  The data manager’s role should be clarified. When the MOH is ready to hire additional data managers, as is necessary, these should be full-time rather than part-time, and placed at the facilities where they are most needed, most likely district hospitals, because of the workload.  Ongoing TA is necessary on data analysis and data for decision making.  An exit /graduation strategy would have enabled FHP to determine where to focus its TA to achieve the best transition results and to know when and where TA is no longer needed.  Because many of the issues are data-related, FHP should have a strong data management team to provide the necessary TA. Quality  Skill-building is needed at facilities to enable data quality checks and data cleaning.  Skill-building is also needed to improve the quality of data entry.  Data managers need basic IT and M&E skills. What has been the major contributor to successful, or to unsuccessful implementation? According to the COP interviews, the major contributor to success was the working environment. The communication between the IPS and their beneficiaries at each level was very good. Also, the MOH has created a very strong policy environment, which makes implementation easier. At the central level, strategies and other protocols and guidelines were discussed in the technical working groups coordinated by the MOH and its partners, including USAID. This allowed the IPs to adjust their strategies as needed. 24 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION At the district level, the communication was relayed through the monthly coordination meetings and also through formative and evaluative supervisions. This allowed the IPs to work with the beneficiaries to adjust strategies in real time as decisions were made at the central level. The leadership and good governance at all levels facilitated project success. What has been the major contributor to success or lack of success? The HCSP project made it possible for health facilities to conduct training on HIV care and management in many districts and health facilities. This complemented MOH training activities and allowed for rapid scaling-up of HIV services in over 200 facilities. Districts took the opportunity to rehabilitate services not exclusively linked to HIV services. For example, in many facilities laboratory, maternity, and neonatology areas were renovated. Laboratory and other equipment procured through the HCSP project are used not just for HIV services but for all health activities—all patients attending the HC or DH facility benefit, which represents a significant benefit for the entire Rwandan health system. Strategies such as task shifting have empowered HC staff, which has improved decision-making, access to services, and quality of care. What are the unintended consequences or spillover effects, positive or negative, from this project? The IPs provided services to far more clients and health facilities than initially planned. For example, FHI360 had committed to eventually implementing HIV services in 45 health facilities but was able to reach 68 facilities. The other IPS had similar experiences. There was significant overachievement on many PEPFAR indicators. While some of this was expected, the IPs were very proud of what they had managed to accomplish by leveraging other funds and activities. Equipping neonatal care units with incubators provides access to critical emergency services for premature infants. However, failure to adequately monitor the infants can eliminate potential benefits. In addition, there is a tendency to use incubators when kangaroo mother care might be a better option. Conclusion The mentoring model was very intense, with 80 to 100 percent one-on-one coverage of the mentor to the mentee. This is very expensive and not in line with traditional mentoring. It caused some individuals to become dependent on the IP supporting them. The effects of the dependence will be seen in the coming months in terms of whether the services prove to be sustainable. Recommendation  Identify when it is appropriate to use incubators and when to use kangaroo mother care.  Provide incubators for extremely premature or sick infants while stressing that kangaroo mother care is effective for most low-birth-weight infants. USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 25 Are there any best practices and lessons learned, for instance in terms of project approaches in the context of strong country ownership, integration with other services, and use of multiple partners for implementation?7 Chapter 5 contains detailed descriptions of lessons learned and best practices, but one example of a best practice is the use of evidence to review national policies or norms. A study of the pilot task-shifting activity showed that nurses can safely provide additional services, which allowed more rapid scale-up of ART. It also convinced doctors that the practice was safe. The project introduced numerous innovations and identified both lessons learned and best practices. To what extent has service delivery quality improved at the facility and district levels? Over the period when the IPs were implementing HCSP in conjunction with the MOH, service quality improved at the district hospitals and the health centers supported by the project. Other development partners such as the Global Fund were responsible for other sites. The increased quality and access was in part due to the support provided for such activities as these:  Financial support to the national PBF to pay incentives. Monitoring of PBF indicators has shown significant improvements in quality in many facilities.  Support to the Coordinated Procurement and Distribution System at the national level.  Financial and technical support for monthly district coordination meetings where facility statistics and issues were discussed.  Support to the MOH to implement data management systems, such as TRACnet and open MRS, and the support for district supervision of health centers.  Regular supervision and quality checks by technical teams of health facilities and services. The IPs also perform audits to ensure that data collected and submitted are correct and to identify quality issues.  Input-based sub-agreements, although the IPs did pay for PBF incentives through sub￾agreements.  A data collection tool developed by one IP to assess the quality of pediatric ART services and implementation of pediatric care and treatment protocols. Data collected will be analyzed and shared with districts next quarter.  10 quality indicators introduced that can be used by both the health facility and FHI360 to evaluate quality. Baseline data were collected. Sites use these indicators to identify areas for improvement, provide TA and conduct follow-up. PBF data are used for quality improvement. The IPs actively participated in the quantitative and qualitative evaluation of PBF indicators. For example, at one HC, the IP gave focused support to and worked closely with the new director and providers to address problems like frequent staff turnover and a poor burn rate. The PBF score increased from 79% to 86%. 7 See also commentary on other questions. 26 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION Whether a score is high or low, PBF evaluation data provide ample basis for open discussion with facilities, and clearly identify areas needing improvement. The results can be seen in the increased numbers of patients seeking services and the higher percentage of patients receiving the full PMTCT continuum of care. The project also re-invigorated the quality committees, which also worked to increase the quality of care. Capability of National and District Level Institutions What capacity-building results were achieved? What areas still require support? From October 2009 to September 201 2, more than 2,682 health care workers completed in￾service training supported by FHI, 6,580 were trained with IntraHealth support, and 1,975 with EGPAF support.8 (The numbers of trainees are not comparable because each IP group had different target areas.) Supported facilities now have the capacity to provide the basic package of HIV and SGBV services, kangaroo mother baby care, and palliative care. Laboratory services were central to the delivery of quality clinical care services. The IPs met the targets for upgrading laboratory services at supported facilities. Laboratories at some facilities had previously been upgraded with funds from the Global Health Fund. All supported district hospitals are now equipped with laboratories to facilitate ART services and to support district HCs. The upgrade of laboratory services (e.g., more space, new generators available, running water.) has resulted in the upgrade of other clinical services. For some HC laboratory facilities, the funds allocated were not sufficient to cover all the upgrading needed; lacking was hematology and biochemistry equipment. Health centers improvised by using the services of neighboring health centers, but this affects the quality of service because of the delays in getting patient lab results. Every health facility currently has a data management unit. The units are equipped with enough computers (funded by either the U.S.G. or the Global Fund). HCSP gave data managers significant amounts of training, monitoring, and supervision. In addition to didactic training facility and district data managers received one-on-one mentoring while they input data, made graphs, and posted and discussed the data. Annual data quality audits conducted by MEMS show steady improvement. However, the quality and use of data in health center and district level can still be improved. At the central level IPs were actively involved in the GOR health TWGs. They often provided technical support in new areas, worked with central staff to draft policies and guidelines, and vetted program practices. The IPs also worked to train trainers, so as to create district pools of trainers. There was no indication that senior staff seconded to the central ministry provided capacity-building although they did excellent work in their seconded roles. Conclusions  M&E is central to any evidence-based activity, and data management is central to M&E. 8 Human Resources for Health MEMS. USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 27  Support especially to M&E has given new life to the use of evidence in decision making. While support is still needed in this area, the IPs have made a good start.  Laboratory facilities have been instrumental in improved service delivery in HCSP￾supported facilities. Recommendations  The GOR should put in place a data management policy for health care facilities.  The M&E framework is in place but needs support for such key areas as data management. For example, data managers could use training in basic IT, M&E, and data analysis.  If possible, laboratories that were not adequately equipped during HCSP should be updated under FHP. Has the number of staff trained to deliver HIV/AIDS and other integrated services resulted in increased output, especially up to GOR recommended levels? The number of recipients of services surpassed targets in every service area except SGBV. Facilities visited by the team all had low levels of utilization of SGBV services, the highest being about 20 clients a month; most saw only one or two. The targets were drawn in accordance with GOR recommended levels. One can, therefore, state that the number of staff trained and other integrated services resulted in increased output. The number of pregnant women giving birth in 2010 in 154 project-supported sites9 was 186,772; in 2011 there were 131,900 deliveries in 156 sites and in 2012 111,279 births in 156 sites. 10 Only one IP collected data on the number of HIV+ women who delivered in the reporting period in all three years: in 2010 there were 479 HIV+ mothers; in 2011, 537; and in 2012, 667. In 2012, all three IPs collected this information, and together there were 2,932 deliveries of HIV+ women. In 2012, the number of exposed infants that tested positive at 6 months was 38 and at 18 months only 12.11 9 Number of PMTCT sites according to MEMS reports. 10 Number of deliveries, according to MEMS reports, MCH program areas. 11 Number of HIV+ deliveries and infants who tested positive at 6 and 18 months based on MEMs data. 28 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION Figure 6: HIV+ Adults and Children Receiving at least One Clinical Service Figure 6 shows the number of HIV+ adults and children who received at least one clinical service—further evidence that staff training resulted in increased outputs. From 2005 to 2010 the national maternal mortality rate fell from 750 to 476. The rate of use of modern contraceptives went up to 45% and the TFR decreased to 4.6 children per woman.12 While the HCSP cannot take credit for all these changes, it certainly contributed. Conclusions Expansion of services, changes in policies and guidelines, and replacement of staff required continuous training. The numbers of trained staff have increased, but whether the increase is sustainable is not a given; steps should be taken to ensure that this valuable resource is retained. The team believes that additional community outreach is required to increase utilization of SGBV services. Also, since many male providers stated that there was very little SGBV in their districts, it appears that they need to have their awareness raised. Recommendations Staff training deserves continued support to encourage retention, responsiveness to changes in policy and guidelines, and expansion of services. When possible, training should be hands-on at the facility level. To what extent has the capacity of staff to deliver quality services (as a result of training, coaching, mentoring) changed as a result of USAID investments? All clinical services at the facilities participating under HCSP were launched only after trained personnel were in place to deliver them, and health care providers stated that they would not be able to continue providing services without the support of the GOR or another partner. Supported health centers can now provide the minimum package of HIV services; critical services like ART that were not provided before HCSP are now accessible. Before the HCSP 12 2010 Rwanda Demographic and Health Survey. USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 29 began, there were far fewer sites providing ART and those were mainly in more urban areas. Most patients required referrals. Currently, the 416 sites providing ART services cover over 90% of those needing treatment. In 2007 there were 165 sites that covered 82.2 percent of those patients but the eligibility criteria were different. When the national immunologic eligibility criteria for the initiation of ART changed in 2008 from a CD4 count of less than 200 cells/mm3 to a count of less than 350 cells/mm3, demand for treatment increased. Even with an additional 30 ART sites, in 2008 coverage dropped to 65.3%, though it has since more than recovered.13 Monitoring of patients on ART has improved; hospital laboratories within the district are able to carry out CD4 testing. Interviews with patients revealed that they are happy that they now have access to care and treatment facilities close to their domiciles. They are also happy with the quality of care that they were receiving. However, some HCs reported that they had not received adequate coaching or mentoring. There may be many reasons for this. Some facilities that had been supported by previous projects had had more exposure to capacity building; some facilities started further down the quality scale as measured by PBF scores; and some sites that were not brought into the project until 2010 and 2011 had fewer resources. Even though all sites received significant mentoring and coaching, it was not enough for facilities that joined late or started further down the quality scale. Since the facilities did not start at the same point, they did not end at the same point, but it is reasonable to assume that the quality improvements in services seen at all facilities was in large part due to training, mentoring and supervision as well as renovations and procurement of needed equipment. The support provided to the districts and HCs has made it possible to scale up HIV services. Integration of FP, MCH, nutrition and SGBV into HIV services has resulted in their increased utilization. For example, modern CPR has increased to 45% over the course of the project, and almost 100% of TB patients are tested for HIV and vice versa. The HCSP working with the MOH was a major contributor to these nationwide successes. Has the process of capacity building been gender balanced? What was the proportion of male to female staff trained? Training was provided to both existing staff and those who were recruited to fill vacant posts at the HCSP sites. Health facilities in Rwanda have never used gender as a basis for recruitment. The MEMS database identifies 16,447 people as trained from October 2009 to September 2012 with PEPFAR funding. There is no gender breakout but in general there were far more women trained since most nurses are female. The team did attempt to match training numbers from quarterly reports with other data, but there were significant discrepancies and there was no way to reconcile the numbers. In some settings an individual who had received multiple trainings was counted only once; in others, each training was counted separately. It also is likely that there may have been double counting. 13 Sabin Nsanzimana, Head of HIV, AIDS, STIs & OBBI Division/RBC. 30 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION Therefore, the evaluators chose to use the MEMS data due to PEPFAR’s rigid rules on training counts. Conclusion Balancing the gender of the recipients in the capacity-building aspect of the program is not relevant because the facilities are not gender-balanced to start with and the central MOH chooses candidates; the HCSP could only encourage MOH to try to choose an equal mix. Recommendation Balancing genders in capacity-building has to be approached with an appreciation for the context of operation. What were successful techniques for increasing capacity for improved project implementation? The IPs looked internationally to adapt best practices for improving HCSP implementation in terms of training; supervision models, such as mentorship; task shifting; quality improvement; and PMTCT, among them:  The one-stop model for SGBV.  Task-shifting in provision of HIV clinical services.  HIV services related to maternity.  Partner testing in the PMTCT program. (See Chapter 4 for a detailed discussion.) Conclusion Although a number of the techniques introduced were successful, without support many of these interventions will not be sustained because most facilities would not have the funds to retain staff previously supported by the project. Resources are also needed for successful rollout. Without additional support supervision to ensure quality will also be problematic. All of the directors of HCs visited by the evaluation team reported that they had not had a supervision visit or a visiting MD since the project ended on September 30. Hospital directors stated that they had cut back or stopped doing supervision visits due to lack of transport, distance, and road quality after the rains. The one District Health Management Team (DHMT) visited also stated that they were not doing supervision visits to hospitals unless the hospitals were very close due to lack of transport and fuel. Granted, these observations are only a few but it seems likely that similar trends would be seen in other facilities. USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 31 Recommendations  Determine the feasibility of using alternative supervision strategies.  Adopt a more traditional mentoring strategy where mentees are visited much less often or some mentoring is done electronically using telemedicine techniques or simply mobile phones. What significant changes in capacity occurred at the national level? For example, how many central staff were trained and what tools and policies adopted? How were successes achieved? Training at the central level was passive. However, IPs seconded experts to strategic positions at the MOH to complement its capacity to manage clinical services. These experts were active participants in TWGs and other coordination forums along with the IPs and through these interactions were able to influence policies and guidelines for delivery of clinical services. They performed well. However, there is little indication that they transferred capacity and knowledge beyond the TWGs. For example, the experts are not permanent placements and there was no knowledge transfer within the MOH management structure. A number of policies and guidelines stemmed from the TWGs, led by the MOH and the IPs. An example is the Rwandan policy on palliative care, for which guidelines and protocols were adapted from programs in neighboring Uganda. Under the leadership of IntraHealth, working with the MOH, these guidelines were customized to the context of Rwanda, and a policy review was undertaken. Rwanda now has a policy on palliative care and training on the protocols has been carried out, although resources have not yet been allocated for program rollout. The IPs have supported the GOR on creating materials for a number of service areas, with each IP taking the lead in its area of comparative advantage. Conclusions Participation in building central capacity was a role that each IP appreciated at the start of the program. The IPs were involved in policy review and sponsorship (e.g., IntraHealth with palliative care); promotion of quality care through review of guidelines and protocol; and development of accompanying materials. Recommendations  Formulate a clear strategy for knowledge transfer and exit before seconding experts to the central level. Any new positions created should be defined in GOR staffing procedures, with a deadline timeframe for filling the position with permanent staff or transitioning the salary to GOR support. Were any assumptions that were made when the project began about Rwandan health system capacity-building methodologies confirmed or disproven over the course of the project? Many trained personnel were transferred to other facilities or left their posts either to secure jobs elsewhere (being more marketable due to the training) or to be closer to the city (Kigali) where they could pursue further training. The IPs did not expect this much staff turnover. Conflicts were encountered in the areas of data management when the MOH initially defined nursing experience as a requirement for data management positions, and not all positions were 32 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION full time. The nurses were often caught between data management and nursing duties. This is starting to change as new data managers are hired full-time and without nursing responsibilities. In part due to transport issues, districts rarely carried out supervision unless the IPs facilitated the activity. This created a sense of dependence. In the facilities visited by the evaluation team, staff were mentored by the IPs on an almost one on one basis bordering on providing the service themselves. The IPs had assumed that the facility staff would be completely independent by the end of the project. They also assumed that though they would start with full-time mentoring, they would gradually reduce the level and frequency of mentoring. Yet in years 4 and 5, many mentors were still spending 70 to 80% of their time with mentees. The independence assumption turned out not to be true, in large part, because of the mentoring model that was used. Although QA teams are present at virtually all HCSP facilities, most still need to be built up. Since the IPs participated in all meetings in their role as mentors, it is unclear how the teams will function without that support. Conclusions  Though the methodology of training is tried and trusted, it only works if the personnel trained actually take up the roles they are trained for and if conflicts in roles are mediated.  Part-time data managers have not worked out well. The workload is very heavy and is increasing along with demand for services.  Unintentionally the mentoring model used by the IPs created dependence. It was also very inefficient and expensive. A more traditional approach to mentoring where the mentor spends time with the mentee, formulates a plan, and then leaves to let the mentee implement the plan with occasional telephone contact would be more efficient. Recommendations  Do training as part of a personnel retention strategy.  Hire full-time data managers who have a background in IT or M&E skills rather than nursing experience.  Consider adding more data management positions when resources become available.  Explore the feasibility of using clients or NGOs to monitor service delivery. NGO monitoring has worked well internationally in a variety of contexts. It also promotes ownership and sustainability.  Review and perhaps revise supervision methodologies and mentoring models to provide a more practical and efficient system that will work within the context of the Rwanda health system. Sustainability and Transition What have been the achievements of HCSP in assisting the GOR to assume all technical and administrative oversight of HCSP-supported facilities and districts? What remains to be done and what are the challenges to achieving this? USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 33 During the HCPSP, USAID and its partners worked to complement the GOR in all technical and administrative service activities in the project districts. The following are a sample of the achievements and results:  Conducted an assessment of the needs for HIV services at the district level.  Facilitated joint planning between districts, health facilities, and partners.  Developed annual sub-agreements.  Trained and mentored administrative and technical staff to manage and monitor the sub￾agreements.  Supervised the administrative staff to ensure monitoring of the sub-agreements.  Built capacity of health personnel through training based on the tools, protocols, and needs defined by the MOH.  Supported and monitored technical training results through regular mentorship in health facilities.  Supported supervision of district health networks.  Participated in supervision to ensure quality services.  Supported improvement of district data management.  Supported districts in organizing monthly health sector coordination meetings.  Participated with other USG partners to define, update, and disseminate the standards for HIV management in conjunction with the MOH.  Participated with other USAID partners in the national PBF program and supported the community health insurance system (mutuelles de santé). Conclusions USAID IPs provided very effective support to the GOR. They shared experiences and helped build best practices through participation in TWGs, policy/strategy reviews, and revision of HIV protocols and standards. The IPs contributed greatly to strengthening the capacity of district and health facility administration. Recommendations:  Ensure the continuity of the work done by USAID and the IPs.  Put in place a strategy for retaining trained personnel.  Explore technology solutions as part of the supervision review.  Develop and implement a strategy, standards, and operating procedures for data management.  Provide TA to update policies and procedures as well as other standards related to HIV and primary health care. 34 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION  Participate in HIV M&E activities organized by the GOR. What key outcomes are considered to be sustainable? What made them sustainable? The IPs never committed to a formal process or discussed health center transition or project sustainability as this was not explicitly requested by USAID. Some initiatives, such as clinical mentorships and evaluative supervision for HIV services at district hospitals, were designed to improve quality and sustainability, but these initiatives mostly stopped at project end. For the time being, patient services continue to be of high quality. Data managers are continuing to input, analyze, and use data and are slowly improving data quality. Among achievements in the national HIV program:  Rwanda has achieved universal access to ART.  Patient retention on ARVs is at 90%.  HIV-related mortality has been lowered to 5%.  Estimated new HIV infections have been cut in half.  PLWA are supported through associations and IGA. Conclusions When resources are limited, the sustainability of achievements is relative. However, the expansion of HIV services is an important achievement because it enables access to HIV care to all Rwandans who need it. The MOH has already taken ownership and will assure continuity of HIV care, in conjunction with USAID and other development partners. Recommendations  Provide technical support to the districts for transition.  Build district capacity for management and leadership. What strategies or approaches has the GOR adopted from the IPs? Is there GOR uptake of IP interventions? USAID partners participated in the definition and adoption of several GOR health sector strategies. Experts funded by the IPs participated in several TWGs. EGPAF adopted innovations and best practices from its sister programs in other parts of Africa and initiated them in pediatric AIDS interventions in Rwanda. In conjunction with the TWG, this IP developed training materials, operating procedures, job aids, etc. to support these interventions. The MOH has adopted the Pediatric AIDS Intervention Model as national policy. The MOH approved the contents of the national SGBV kits in May 2010, which are now available at CAMERWA (Centrale d’Achat des Medicaments Essentials de Rwanda). The HCSP also printed the information, education, and communication materials developed as part of the PEPFAR Special Initiative on Sexual and Gender-based Violence, which were revised by the GBV TWG with support from HCSP. The GOR has voiced appreciation for the HCSP strategies and approaches. In one instance the mayor of a district spoke to the evaluators of his appreciation for the work of the IP in USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 35 supporting the one-stop SGBV approach in his district. He also suggested some ways things could be changed and was looking forward to showcasing the program at the World Health Day activities At a busy health center, a nurse who had just transferred from a non-project health center told the evaluation team about implementing the one-Stop SGBV initiative in her former place of work. They wanted to implement the program but had not been trained, so the nurses themselves arranged for rotations through a health center where they were providing SGBV services to learn how to do it and got permission to set it up, which they have done. As part of the GOR palliative care strategy, the MOH also approved the use of oral morphine for pain management. The EGPAF team and the ministry came up with a model for more integrated MCH/HIV services. The model ensures that waiting times are not too long, clients get all services in one visit, and they do not have to return to a health facility several times a month. IntraHealth provided the training. The model was piloted in five health facilities and is being evaluated using a mixed methods approach. Meanwhile, many health centers are integrating MCH and HIV services. HCSP provided technical support to the MOH to move away from infant feeding for HIV+ mothers only to including infant feeding generally into the national nutrition package. HCSP also supported the community-based nutrition program, which was a fore runner for integrated services. The ministry is very proud of these achievements and has introduced materials for the community-based nutrition program country-wide. Conclusions The GOR is receptive to the USAID IP strategies and approaches. Numerous program strategies have been discussed and vetted in different forums at both district and central levels. There was full disclosure about HCSP progress as it occurred and GOR has made known its appreciation. Although some interventions have not yet been rolled out to nonparticipating facilities, that may be due to resource limitations. Recommendations:  Support and strengthen task shifting, which could benefit from further monitoring and an assessment to address potential health facility issues.  Collaborate with the GOR to ensure that the M&E indicators for the new strategies are captured in TRACnet or HMIS. What was achieved with graduation or sustainability plans? All activities for each specific project had a specific project cycle with a well-defined schedule. Originally, each IPs was supposed to develop a plan to sustainably transition activities to the FHP, but this was not done because there was too much uncertainty surrounding transition to FHP and the GOR. Conclusion There was no concerted sustainability plan for HCSP activities. 36 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION Recommendations  Draft a sustainability and activity plan for transfer from the USG to the GOR.  Disseminate the plan to the districts and the health facilities. Were there any activities/outputs that were not accomplished that could affect sustainability? The HCSP was a tremendous success with regard to establishment and access to HIV services for the communities. The majority of planned activities were implemented with regular monitoring. All three IPs conducted some operational research. However, there was no suitable and sustainable supervisory structure. Due to lack of transport and fuel, supervisory visits had stopped in the sites visited by the evaluators when the project ended. Conclusions The majority of activities were successfully implemented. For a number of indicators, the results obtained for scaling up treatment go beyond what was expected. Recommendations  Ensure the continuity of HCSP achievements,  Develop an innovative and sustainable supervisory mechanism. EXPANSION OF HIV/AIDS CLINICAL SERVICES IN INSTITUTIONS How many more facilities can deliver HIV/AIDS and integrated clinical related services now compared to the beginning of this project? At the beginning of the project very few health facilities (exact number not known) had a full service package for HIV. Patients requiring ARVs were mostly referred to another site. When HCSP ended, 156 facilities were offering the full HIV services package from prevention to antiretroviral treatment. These services were treating 33,820 people and had provided PMTCT prevention to 95,662 women in the previous year (see Figure 7). Over the last three years of the project, 287,813 pregnant women knew their HIV status as shown in Figure 7. Of these, 8,150 (2.8%) were HIV+ (the range went from 2.7% in 2010, to 1.6% in 2011, and to 4.3% in 2012). USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 37 Figure 7: Pregnant Women with Known HIV Status In addition, FP/MCH/TB and SGBV services have been integrated into the HIV service package. SGBV services were implemented in the last year of the project, during which the IPs trained 2,194 service providers, and 156 HCSP sites gave care and support to 3,446 new SGBV cases Palliative care will be initiated shortly. In 2010, 154 HCSP facilities were providing ART services to 26,159 adults and children with advanced HIV. In 2011, 156 facilities provided ART services to 29,436 such patients, and in 2012, 156 facilities cared for 34,000 such patients (Figure 8). The IPs met their targets or came close in all three years. Figure 8: Adults and Children with Advanced HIV Infection Currently Receiving ART 38 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION Conclusions  A large number of health facilities started providing HIV prevention and treatment, resulting in 100 percent access to ART, due to efforts of GOR, HCSP, and other IPs and development partners.  IPs were able to support the enhanced services and go beyond the set targets.  Of the pregnant women delivering in project-supported sites, the percentage who were positive went down and then back up. The reason is unclear. Recommendation  Ensure that project-supported services are maintained and improved. How are the services that can be delivered in facilities now different from those offered at the beginning of this project? The biggest change is the introduction of a HIV services package in 156 health facilities; this package did not exist in many facilities before the HCSP project began. HCSP was also directly responsible for integrating MCH, FP, and SGBV services into HIV services. Figure 9 shows a steady increase in PLWHA reached with a minimum package of prevention over the last three years of the project, during which all targets were met. Figure 10 shows an increase and then a leveling off in the number of people receiving CT who receive their results. The project has done an excellent job in reaching out to people to get tested, including an initiative where pregnant women bring their partners in during ANC/PMTCT visits, where they are offered CT. About 80% of the fathers accompany their partners. Figure 9: PLWHA Reached with a Minimum Prevention Package USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 39 Figure 10: Individuals who Received Their VCT Results Conclusion The facilities have vastly increased their capacity to deliver a wide range of integrated services, which was not possible before the project. Initiatives such as task shifting have contributed to this achievement. Recommendation  Ensure the continuity of HIV services. To what extent has there been an increase in the number of people receiving services from the point of inception to now? To what extent has there been an increase in the percent of the population in the catchment area receiving services? Nation-wide the number of health facilities delivering HIV services increased from 165 in 2007 to 416 in 2012. USAID partners supported HIV services in 156 health facilities. Rwanda is one of seven countries where ARV coverage is virtually universal. Approximately 90% of people in need of ARV treatment receive it. Figure 8 makes clear that there has been a number of people receiving services as a result of the project. The HCSP project contributed to these achievements, as did the GOR and other partners, such as the Global Fund. Figures 11 and 12 illustrate some unexplained trends. Figure 11 shows the number of adults and children with advanced HIV infection newly enrolled in ART, which is decreasing over the last three years and is below target. The decline could be the result of decreasing HIV prevalence, which the RBC believes. Alternatively, there could be a quality issue with identifying and enrolling patients with advanced HIV in ART. The known survival trend one year after initiation of services shown in Figure 12 is of potential concern. The drop in known survival in 2011 could result from a drop in service quality in terms of patients lost to follow-up, since as the numbers increase in 2012. 40 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION Figure 11: Adults and Children with Advanced HIV Infection Newly Enrolled in ART Figure 12: Adults and Children known to be Alive and Being Treated 12 Months after ART Initiation These trends require continued monitoring. Conclusions While the evaluation team would expect that there would have been an increase in coverage in catchment areas, it did not have access to data that would have allowed them to answer this question. There has clearly been an increase in the number of people receiving HIV and FP/RH services, but there are some questionable trends that require further investigation. Recommendations  Ensure quality services and continuing M&E for the 33,820 people on ARVs in HCSP￾supported sites.  Continue to monitor potentially worrisome trends, such as patients newly enrolled in ART and one-year survival rates. USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 41 What innovative or cutting edge strategies, interventions, or techniques were used to increase quantity and improve quality services? The Rwandan politico-administrative decentralization, good governance, and appropriation of initiatives gave the HCSP project an environment favoring its success. IP coordination and experience-sharing within the MOH thematic TWGs helped to accelerate adoption of efficient initiatives and abandonment of inefficient ones. Integration of services led to an increase in modern CPR to 45%. Task shifting led to increased access to HIV services, including PEP and ART. Conclusions  Decentralization of the Rwandan health sector facilitated HCSP implementation.  Several initiatives in districts deserve special attention, such as the coordination meetings. Recommendations  Support decentralization of the health sector.  Support continued implementation of the new district management team structure to improve management of health initiatives within districts. What has been done to promote gender equality in acceptance and delivery of services? What has been done to decrease stigma and discrimination against people living with HIV/AIDS? The HCSP project did not identify a specific strategy to promote gender equality or decrease stigma and discrimination against PLWHA. However, the combination of initiatives led to gender considerations major achievements were (1) HIV screening of couples and the presence of male partners for PMTCT; and (2) promotion of one-stop SGBV centers in all health facilities supported by HCSP. Integration of services also helped reduce stigma. e.g., not separating services for HIV+ infants and HIV– infants; integrating nutrition for HIV+ children with infant feeding; one-stop service centers for the HIV+, e.g., phlebotomy services at the HIV clinic (patient is not sent to a separate location for blood draws); and support for PWP services through the NGO network. Because the IPs did not choose training participants, they could only encourage gender neutrality, although that is difficult when an industry is predominantly populated by one sex or the other. Conclusions  HCSP contributed to gender equity indirectly by, e.g., screening couples as part of HIV prevention, and implementing SGBV services at health facilities.  All three IPS also routinely included gender considerations in their programming regardless of whether they had a specific strategy. Recommendations  Support HIV gender-neutral prevention activities.  Provide TA for SGBV services at health facilities and raise community SGBV awareness. 42 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 43 V. BEST PRACTICES AND LESSONS LEARNED LESSONS LEARNED Monitoring and Evaluation The HCSP Mother Infant Tracking (MIT) database was a critical patient-level system. This helped the HCSP with required reporting and analysis as well as prospective monitoring of client interventions and outcomes. HAART training data permitted the project to identify unique individuals trained and avoid double counting. The HCSP developed feedback loops of service data and analysis to health facilities and providers, which helped build capacity, control data quality, and promote data-driven decision￾making. Routine monitoring improved staff skills, and consequently service quality, given that providers were involved in drafting monitoring materials and knew how to use them to make sure that activities were on track. Full-time data managers should be placed in district offices. Data managers who have other duties (e.g., nurses) are too busy and do not have the mandate to collect and enter data properly. Skills in IT and M&E would help data managers to function independently. Exit Strategy and Sustainability A well-thought-out sustainability strategy incorporating an exit strategy should be in place at the beginning of the project, drafted in conjunction with the MOH and USAID. Because such a strategy has major implications for project approaches and activity planning, it is imperative that all stakeholders be in agreement. While the strategy will likely need to be updated from time to time, everyone should be aware of its content and moving in that general direction throughout each project. Sustainability As the primary unit in the decentralized health system, the district is key to ensuring the sustainability of a quality health system. Structures like the DHMTs, are responsible for efficient health service delivery, coordination, and training and should continue to be supported especially in the areas of management, decision making and supervision. One intervention designed to encourage sustainability and strengthen the DHU was the creation of district pools of trainers, which allowed the district to decide where, when, and how training should take place. Previously training was organized centrally and districts had little control over its design and scheduling. Districts now have capacity to respond to changing needs, such as staff turnover. For example, when the project began, the IPs organized quarterly meetings with DHMTs, but by the end of the project the DHMTs would organize the meetings and invite the HCSP to attend. This intervention empowered them to take charge and make decisions. 44 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION BEST PRACTICES Partnership and Government Ownership Joint planning nurtures productive relationships between project and district staff with harmonized activities and priorities. Engaging the MOH at all levels reinforced governmental structures and facilitated shared success. Joint planning also increased MOH capacity and ultimately ownership of processes and activities. This is evidenced by the improvement in drawing up annual workplans and budgets. During the initial years of HCSP, planning sessions lasted from 3–5 days. However, in years 4-5, district staff prepared these documents independently and then required only one day with HCSP staff for review. Health Facility Management The subgrant approach is an effective way to build capacity in health facility management. Subgrantees showed improvements over the life of the project in their capacity to manage funds, conduct procurements, and generally comply with agreement terms. For example in year one, many sites avoided doing procurement because they did not know how to follow the correct policies and procedures. Some sites did not have an accountant on staff. Where necessary the HCSP hired and trained accountants and provided extensive training on MOH processes for procurement. The project also helped sites put in place a more robust checks and balances system as a means of risk management. By the end of the project most sites were comfortable and compliant with the procurement process. Holistic Continuum of Care Adopting a holistic approach has been shown to be beneficial to all patients regardless of country, diagnosis, and setting. The holistic approach provides a continuum of care from diagnosis to palliative care. Physical, emotional, social, and spiritual care; food support; income￾generating activities; and mutuelles all contribute to a patient’s ability to make the best choices among health care options. Multidisciplinary teams help to reinforce coordination of holistic care and formalize the links between different specializations. Continuum of Care Basic care services for HIV+ persons included clinical staging, CD4 monitoring, management of HIV-related illnesses and opportunistic infections, counseling on positive living and prevention for the HIV+, family testing, nutrition counseling, and pain management. FP and MCH services were integrated into the HIV program in 2009. Working with the MOH, the HCSP, IPs, and other donors succeeded in opening 416 ART sites that in the last three years of the project were reaching 34,267 children and adults with first and second line therapy. Palliative Care Linkages between multidisciplinary teams of providers and PLWHA groups were critical to developing a holistic palliative care model for HIV and AIDS care/treatment in Rwanda. PLWHA cooperatives can and should be leveraged as entry points to facility and community linkages. For example, CHWs, PLWHA cooperative members, PLWHA volunteers, and local associations are important in recovery of clients lost to follow up. At one point the HCSP discovered that sites near the Ugandan border experienced substantial cross-border movement of population, complicating client follow-up and home visits. CHWs, PLWHA cooperative members, PLWHA volunteers, and local associations were essential to recovering clients. USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 45 Task Shifting In September 2005 a pilot program of nurses prescribing ART was launched in three rural primary health centers in Rwanda. The feasibility and effectiveness of the task-shifting model was evaluated in 2008. During the study, nurses took on many responsibilities previously reserved for medical doctors (who were visiting health centers increasingly less often). Transferred tasks included first-line ARV treatment; treatment for simple opportunistic infections and STIs; clinical and biological monitoring; and managing the side effects of ARVs. When nurses identified therapeutic failures and other complex cases, they referred them to a doctor or a hospital. Patient outcomes in the pilot program compared favorably with other ART cohorts in sub￾Saharan Africa and with those from a recent evaluation of the national ART program in Rwanda. The findings suggested that given adequate training, mentoring, and support nurses could effectively and safely prescribe ART. Task-shifting also has the potential to substantially reduce the demand on physicians for HIV services, freeing up capacity to treat more patients, focus on more complex cases, or provide more non-HIV services. Based on this evidence, the HCSP supported task shifting and retraining in areas where there were shortages of qualified nurses, physicians, and other health professionals to help maintain access to high-quality health services. In 2011, 85 nurses were trained to prescribe first-line antiretroviral therapy, which allowed 14 additional health facilities to introduce ART in 2011. Integrated Maternal Infant and Young Child Nutrition (MIYCN) Package Over the past two years, HCSP IPs worked closely with the Nutrition Desk of the MOH MCH Department and members of the Nutrition TWG to harmonize MIYCN tools into a single package of counseling and behavior change communication materials. The package includes counseling cards, posters, brochures to take home, training curricula and training aids, and facilitator and participant guides to using the materials. The project also worked with the RBC to draft a Questions and Answers Guide on the new national PMTCT/infant and young child feeding (IYCF) guidelines. The materials make use of high-quality brightly-colored illustrations to ensure that the messages are clear to all clients, regardless of their language skills or literacy level. The package contains integrated messages on such topics as MIYCN, HIV, FP, MCH, kangaroo mother care, growth monitoring and promotion, kitchen gardening, and small animal husbandry. EGPAF supported both technically and financially the entire production and training process. The IP sent two staff members to be trained in Nairobi as master trainers in the UNICEF IYCF package and supported the training roll out, including district training of trainers, health facilities and community health care workers in selected districts. With strong leadership as demonstrated by the nutrition desk of MOH, it is possible to develop an integrated package, supported by all partners. The nutrition partners are pleased that there is one integrated package that has all the key messages around MIYCN. When there are other nutrition interventions planned by MOH or any of the partners, the MIYCN materials are used as the guiding document and relevant messages are copied. This ensures consistent messaging across interventions. The entire package is available in one folder with links to the different illustrations, and it is easy to update the materials when new messages or evidence-based practices evolve. The model appears to be quite successful but to date there has not been a formal evaluation. Quality Improvement (QI) In close collaboration with the MOH QI department, the HCSP IPs implemented a QI program at 10 sites in order to improve the quality of facility prevention and treatment services, with the 46 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION ultimate goal of better health outcomes. HCSP began with a formative workshop for managers, care providers from different services, support staff, and community representatives. The training focused on client care, QI in healthcare, performance measurement, data analysis, root causes analysis, the PDSA model for QI, preparing QI work plans, implementing QI projects, and monitoring mechanisms. The program also introduced or revitalized Quality Management Committees (QMC) to identify and manage improvement projects. Community representatives were members of the QMCs. Nine quality of care (QoC) indicators were measured at baseline and again six months after improvement projects. Overall, notable improvements in QoC indicators were observed; statistically significant improvements were noted for early DBS testing for exposed infants, enrollment rate of new HIV-positive patients, reduction in loss-to-follow up in pre-ART care, and retention of children on ART (p < 0.05). MCH/HIV Integration Model The HCSP, in collaboration with the MOH, piloted a one-stop model in five sites in the Eastern province that integrates HIV and MCH services and where possible provides same-day, co￾located services. The model is designed to improve patient flow and schedules—making sure the provider has time to offer all the services in one appointment and avoiding patients moving from room to room or coming back on different days to receive fragmented services. The HCSP, in collaboration with the MOH and district hospitals, trained staff at the selected sites on the requirements for service integration, including reorganizing patient flow and service delivery, and where necessary cross-trained staff on clinical skills. The model started in 2010 and is currently being evaluated. By helping the facilities to implement the model, health workers learned its advantages, such as better organization of their work and quality of care. EGPAF will publish the evaluation results in 2013. Psychosocial Care EGPAF, as an innovator in pediatric care, supported the national strategy on psychosocial care for children living with HIV in collaboration with TRAC Plus (today’s RBC/IHDPC). In 2008 EGPAF launched the psychosocial care program for children aged 6–15; in 2009 it was expanded to 27 sites. Psychosocial care included training health providers in announcing and disclosing HIV status to children infected and their families; providing children with age-appropriate support; therapeutic outings for children living with HIV organized by ART sites; and four-day overnight camps (Ariel Camp) for HIV-infected children and their counselors. Ariel camps give the children an opportunity to share their life experiences in a recreational environment free from stigma, and learn about positive living with HIV/AIDS and the importance of taking their drugs, good nutrition, and reproductive health. Disclosing HIV status and offering age-appropriate psychosocial support including outings with other HIV+ children helps children to adhere to their medication and cope with living with HIV. The first Ariel Club/Camp was established in Uganda in May 2006; since then, EGPAF has been able to replicate the model in Lesotho, Rwanda, South Africa, and Tanzania. However, no formal analysis has been completed to determine if these camps are a cost-effective intervention in Rwanda. USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 47 VI. SUMMARY OF RECOMMENDATIONS If USAID support through HCSP built capacity in district and national Rwandan institutions to support and manage Rwanda’s health systems and expanded quality health service delivery, then the results would be improvements in (1) the quality and timeliness of routine data reporting; (2) financial management; (3) service delivery and technical capacity; and (4) the epidemiological outcomes of the among target population. The data analyzed by the evaluation team clearly supports the USAID/Rwanda development hypothesis. While there has been great progress, much remains to be done if the service delivery and management systems are to continue to grow and improve. To this end, the team has summarized the recommendations according to the targeted results areas. QUALITY AND TIMELINESS OF ROUTINE DATA REPORTING Policy  Develop a retention strategy for data managers, preferably with their participation.  Develop national guidelines for the health data management system. Capacity  In the future the MOH should consider increasing the number of data managers considered standard.  Provide additional capacity building for managers on data analysis, data cleaning, data quality checks, and decision making.  Clarify the role of the data manager and draft detailed job descriptions and job aids.  Support OpenMRS for ongoing operations. Quality  Continue to improve TRACnet data quality.  Build the capacity of data managers in IT and M&E.  To prevent repetition and duplication of effort, streamline data entry to the extent possible. FINANCIAL AND GENERAL MANAGEMENT Financial Monitoring  Report donor funds (subagreements and subcontracts) to the DHU and the MOH and include them in monthly cash reports.  Consolidate all costs by activity in monthly cash reports.  Encourage yearly audits by district auditors per the national standard. 48 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION Capacity Building  Put in place a strong financial management system to monitor the subcontracts at the community, health center, and district hospital levels.  Build district administrative capacity for personnel training; standardization of reporting tools between IP and GOR; etc.  Clearly articulate objectives, knowledge transfer, and an exit strategy to transition to GOR funding for seconded personnel. Communication  Encourage regular communication in writing between MOH levels.  Manage the expectations of communities, providers, facilities, district management, the central MOH, and RBC by establishing strong lines of communication and relationships. SERVICE DELIVERY Quality  Update older TB laboratories and structures as soon as is financially feasible.  Support further QA/QI roll-out.  Ensure that data on the nine quality indicators tested in the pilot continue to be collected and monitored.  Ensure that prenatal care, postnatal care follow-up, vaccinations, hospitalization, nutrition, and PITC indicators are monitored monthly.  Revise the supervision strategy to improve efficiency and effectiveness. – All individuals responsible for supervision travel together in one car when possible. – Consider alternative mechanisms such as:  Technology applications, such as “Go to Meeting.”  Telemedicine (cameras) for expert consultations.  Involve the community in supervision (PAQ). Sustainability  Continue to collect data for the mother-infant tracking system.  Develop a sound exit strategy for FHP, in close collaboration with the GOR and USAID.  Continue to monitor the task shifting program, which is still relatively new.  Reinforce integration of FP/RH/MCH/SGBV/TB/malaria and HIV services reinforcement: – Standardize integration of like services across facilities and districts. – Make national guidelines available at all facilities. USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 49 Access  Support establishment of more SGBV one-stop centers.  Increase utilization of SGBV services by raising awareness, advocacy, and sensitization at the community level. EPIDEMIOLOGIC OUTCOMES  Build capacity and provide TA and support for service integration.  Continue to upgrade programs to ensure they are state-of-the-art.  Maintain infrastructure and equipment. 50 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 51 ANNEX A: SCOPE OF WORK GLOBAL HEALTH TECHNICAL ASSISTANCE BRIDGE II PROJECT GH Tech Contract No. AID-OAA-C-12-00027 SCOPE OF WORK (9-20-12) I. TITLE: USAID/Rwanda: HIV/AIDS Clinical Services Project (HCSP) Evaluation II. CONTRACT: Global Health Technical Assistance Bridge II Project (GH Tech) III. PERFORMANCE PERIOD Evaluation preparations should begin in mid-September 2012, depending on the availability of the selected consultants. Work is to be carried out over a period of approximately 11 weeks with 6 weeks in country, beginning on or about (o/a) October 1, 2012, and final report and close out concluding by December 24, 2012. IV. FUNDING SOURCE Mission-funded V. PURPOSE OF ASSIGNMENT This United States Agency for International Development (USAID)/Rwanda Statement of Work (SOW) sets forth guidelines for an end of project evaluation of the HIV/AIDS Clinical Services Project (HCSP) implemented by three implementing partners (IPs): IntraHealth International, Inc., Elizabeth Glaser Pediatric AIDS Foundation (EGPAF) and Family Health International (FHI 360). During the life of the project, other services, like maternal child health, family planning, and nutrition, were integrated into programming. The follow-on project, Family Health Project (FHP), began in February 2012 but is still in the start-up phase and is currently undergoing capacity assessments of the health facilities. The end of program evaluation may overlap with these capacity assessments. Data collection will be completed soon and preliminary results should be available by the end of August. At the very least, these capacity assessments could help inform the end of program evaluation. Data collected during this evaluation, in combination with that from the capacity assessments, will largely serve as the baseline for FHP. The new implementing partner for FHP, Chemonics, has begun the work of transitioning all clinical service delivery to the Government of Rwanda (GOR). This process will be completed over the next 3–5 years. 52 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION VI. BACKGROUND In 2007, when HCSP began, Rwanda was among the least developed countries in the world, ranking 159 of 177 in the United Nations Development Programme’s 2004 Human Development Index. Some 60% of the population of almost 8.5 million lived in poverty and over 90% were involved in small-scale agriculture. The infant mortality rate was 86 deaths per 1,000 live births (DHS 2005), the adjusted maternal mortality rate was 1,400 per 100,000 births (2005), and gross national income (GNI) per capita was $220. The 2005 DHS preliminary results reported a national HIV seroprevalence rate of 3%, with a high prevalence of 7.3% in urban areas and 2.2% in rural areas. The population of the City of Kigali had the highest prevalence rate, at 6.7%. Women were slightly more vulnerable than men to HIV infection, with 3.6% among women, and 3.0% among men. According to the National AIDS Control Commission (CNLS) website, the primary drivers of the epidemic in Rwanda were those below. CNLS reported in 2007 that: “HIV in Rwanda is spread primarily through heterosexual contact (75%) and mother-to-child transmission (20%). Young women and middle-aged men have the highest rate of infection. High￾risk populations in Rwanda include sex workers and their clients, orphans, prisoners, commercial drivers, and transportation workers. Tuberculosis is also a major public health problem; 60% of HIV patients are diagnosed with TB. Meanwhile, multi-drug-resistant TB remains a growing concern.” Prior to the initiation of HCSP, GOR demonstrated a strong response to the HIV and AIDS epidemic through collaborative national project planning and monitoring. An Emergency Plan Steering Committee was formed that included representatives of CNLS, the Office of Minister of State for HIV/AIDS, Health Care Direction (Direction des Soins de Santé), Treatment and Research AIDS Center (TRAC), the Ministry of Economic Development and Finance, the Ministry of Gender and Promotion of the Family, the Ministry of Education, and the Centrale d’Achat des Medicaments Essentiels de Rwanda (CAMERWA). The GOR and the United States Government (USG) President’s Emergency Plan for HIV/AIDS Relief (PEPFAR) Steering Committee formed five working groups to focus on results: prevention, treatment and clinical care, non-clinical care and support, USG/GOR co-management, and M&E, including epidemiological surveillance. USG/GOR collaboration led to formulation and application of national norms and standards for prevention of mother-to-child transmission (PMTCT), counseling and testing (CT), orphans and vulnerable children, antiretroviral treatment (ART), sexually transmitted infections management, and other key public health projects related to the epidemic. The USG also provided assistance in strengthening the overall quality of and demand for integrated health services within which HIV/AIDS is addressed. It was identified that for sustainability, there had to be increased attention to building capacity at all levels. This meant providing incentives to strengthen and maintain technical quality at the facility level, building stronger linkages with community care and referral systems, and generating more strategic health care financing at all levels of the system. The 2001 Rwandan Service Providers Assessment found great variation in the application of the country’s minimum and complementary (non-HIV/AIDS) packages according to established norms. Only 57% of health facilities offered all the defined range of basic outpatient, maternal, child, and reproductive health services. Almost half (49%) of the health facilities offered some USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 53 health services through community outreach, with health centers more likely to do so than hospitals. Even in 2001 (pre-Global Fund, PEPFAR) an impressive 37% of facilities offered some form of HIV/AIDS counseling or testing through community outreach. The tactical approach utilized a strategic balance of 1) rapid scale-up of prevention, care, and treatment interventions and 2) ongoing capacity building of critical institutions and systems to assure steady progress toward these ambitious targets. The HCSP was envisioned to address the emerging issue and disease burden of HIV/AIDS in Rwanda. The project focuses primarily on fourteen (14) districts and in select City of Kigali health facilities. The target populations include members of the community who utilize health facilities, those who serve the target populations, and the health delivery system. Throughout the lifespan of this project, many improvements have been made in health in Rwanda. Based on the 2010 DHS results, the maternal mortality rate has been reduced from 750 in 2005 to 476/100,000. The fertility rate has decreased to 4.6 children per woman and the use of modern contraception is up to 45%. The HIV rate, however, remains virtually unchanged at 3%. Numbers of PLWHA receiving, care and support increased Numbers of new cases of HIV/AIDS decreased Numbers of HIV/AIDS cases receiving treatment increased Increased capacity of all levels of Rwandan institutions to deliver expanded HIV/AIDS clinical activities VII. PROJECT INTENT The objective of HCSP was to build capacity in national and district-level Rwandan institutions to support and manage health systems, and to expand HIV/AIDS clinical activities and clinical service capacity for patient care in selected districts. The original objective was fully commensurate with PEPFAR, the USAID Integrated Strategic Plan 2004-2009, the GOR’s National Health Sector Strategic Plan (HSSP-I), and the GOR’s National HIV/AIDS Strategy. The Results Framework depicts the overarching goal and results of the HCSP. It is important to note that MCH and FP funds were increased in later years of the project to ensure integration of those areas with HIV and AIDS services. VIII. PROJECT APPROACH AND IMPLEMENTATION This project builds upon the PEPFAR district-level clinical service activities supported by USAID prior to March 2007. USAID’s HCSP cooperative agreements with three IPs provided services to 164 health facilities, primarily within 14 of Rwanda’s 30 districts. Implementation of this five￾year project began in June 2007 and is scheduled to end December 31, 2012. The IPs have collaborated with district health systems and key national partners to increase the capacity of districts to provide “Core Clinical Services” in accordance with established GOR 54 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION policies, norms, and standards; OGAC policies and guidance; and international norms, standards, and best practices. Core clinical services include PMTCT; CT; ARV services; TB/HIV, as well as care and support. Currently, the IPs provide non-HIV services in MCH, family planning/reproductive health (FP/RH); sexual and gender-based violence (SGBV) as part of integrated programming. Provision of core clinical services was accompanied by provision of a “District Package” in each of the 14 districts where HCSP operates. The “District Package” was comprised of support to the key clinical support structures within the district health system, including the district health committees; the Mayor’s Office; the District Pharmacy personnel; and, importantly, the clinical supervisors and trainers based at the District Hospital. In addition to the services offered at the district level at the beginning of the project, the hospitals and health centers were tasked to provide other services. The hospitals provided the core clinical services HIV package for their clients. They developed integrated plans, which included non-HIV service delivery, especially focused on training, supervision, and mentorship for hospital and health center staff. The health centers were supervised by the district hospitals. They provided primary health care (MCH, FP, health education) as well as many HIV clinical services. These included patient clinical assessments, laboratory services, counseling and testing, PMTCT services as well as ART provision. The three IPs provide health care services in the following districts: 1. IntraHealth: Gasabo, Gicumbi, Nyagatare, and Rulindo 2. EGPAF: Bugesera, Gatsibo, Ngoma, Kayonza, and Rwamagana Districts and six health facilities in Kigali City (Kicukiro and Nyarugenge) 3. FHI 360: Nyamagabe, Nyaruguru, Ruhango, Muhanga, and Kamonyi and five health centers in Kicukiro, Nyarugenge, and Nyanza districts. The following map depicts the districts served by the IPs through the use of color-coded arrows. The red arrows are IntraHealth-serviced districts, the blue arrows are EGPAF-serviced districts and the yellow arrows are FHI 360-serviced districts. USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 55 All three IPs had the following overarching project objectives, as documented in their cooperative agreements:  Build capacity in national and district-level Rwandan institutions to support and manage Rwanda’s health systems.  Expand HIV/AIDS clinical activities and clinical service capacity for patient care in selected District Health networks (DHNs). During the life of the project, annual workplans shifted, in consultation with USAID, primarily to expand services in a more integrated approach to service delivery, hence, the inclusion of MCH and FP/RH. While this is not seen in the above project objectives, it should be considered during the design and implementation of the evaluation. Each IP brings its own corporate strength to the task and focuses on different key interventions to reach the overarching goals. IntraHealth concentrates on SGBV, palliative care, and family planning. EGPAF focuses on pediatric HIV/AIDS and PMTCT. FHI has strengths in FP/RH and research activities. IX. FAMILY HEALTH PROJECT The evaluation will assist Chemonics to target program priorities and transition strategies for the newly begun follow-on, FHP. In addition to assuming the work of HCSP, FHP will also assume the in-country work of the Maternal Child Health Integrated Project (MCHIP). FHP support to the public health system will cover facility operational expenses, trainings, health sensitization activities, supportive supervision, mentorship, performance-based financing (PBF) management, and technical assistance. The goal of these activities is to support integrated service provision at the facility level in HIV/AIDS, MCH, FP/RH, and malaria. The service delivery priorities of FHP at the community level include integrated community case management, community-based distribution of contraceptives, and the supportive supervision to strengthen 56 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION their provision. Specifically, in MCH and FP, FHP will seek to strengthen the following interventions: emergency obstetric and neonatal care, Kangaroo Mother Care, Helping Babies Breath, antenatal care, post-abortion care, family planning counseling, and fistula care. While HCSP covered the payments of salaries and PBF incentives, that work will not continue under FHP, as it is being transitioned to direct financing by the GOR. X. SCOPE OF WORK Evaluation Objectives This is an end of project evaluation of the five-year HCSP and will also provide some baseline information for FHP. It will be both formative and summative in nature and its purpose is twofold. First, this evaluation will measure the efficiency, effectiveness and relevance of resource allocation for programming to the stakeholders. Second, this evaluation will serve as a source for learning and capturing best practices. Those lessons learned and recommendations can influence wide-reaching decision-making in the next phases of country programming and in similar projects worldwide. Third, it will inform district- and facility-level technical, programmatic, and administrative support priorities. The audience for the findings from this evaluation will include USAID/Rwanda, USAID/W, HCSP IPs, Chemonics, GOR and other interested parties. Chemonics and USAID will use information gleaned from this evaluation in the implementation of FHP. The GOR can use the evaluation report, as most HCSP supporting services will be assumed by GOR over the next two years. To the extent possible, this evaluation should include those HCSP-supported facilities where the MCHIP (Jhpiego) and Fistula Care (EngenderHealth) projects also operate in order to include those activities in the baseline data for FHP. Evaluation Questions The Evaluation Team is expected to address the following development hypothesis: If USAID support via HCSP built capacity in district and national level Rwandan institutions to support and manage Rwanda’s health systems and expanded quality health service delivery, then the results would be improvements in (1) The quality and timeliness of routine data reporting; (2) financial management; (3) service delivery and technical capacity ; and (4) epidemiological outcomes among target populations. In order for the Evaluation Team to answer the development hypothesis, the following key and supporting questions should be addressed: Key Evaluation Questions To what extent has the project been effective in increasing the technical and administrative capacity of Rwandan health systems, including improving quality and quantity of health services delivery? Were the project objectives (see page 7) met during the 5-year implementation? If yes, to what extent did HCSP IP interventions in human and institutional capacity building assist in the attainment of project objectives? Which HCSP IP interventions specifically contributed to these  Indicators could be PBF quality scores and quantitative results; proportion of HIV+ women receiving ARV for PMTCT, clients lost to follow-up among enrolled HIV+ patients.  HCSP PMP with indicator targets and results will be provided to the Evaluation Team. USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 57 attainments? If objectives were not attained, what were the challenges (e.g., no interventions were deployed, interventions failed)? What needs to happen to achieve the objectives? What were the strengths and weaknesses of this project (especially in terms of those aspects detailed in the supporting questions)? To the extent that the project has had success in capacity building, is the GOR at all levels now in a position that will make the transition smoother? What opportunities and/or constraints face FHP and other future work to be done by USAID? Supporting Evaluation Questions General Was there collaboration between IPs? If so, what was the collaboration and what outcomes were affected? Were the program descriptions followed? Why or why not? Did these projects fulfill USAID/Rwanda and IP performance monitoring plans? If not, what were the areas of underperformance? To what extent was routine clinical data (i.e. HMIS, IQ chart) used for management decision￾making/project improvement? What has been the major contributor for successful and/or non-successful implementation? What have been the benefits versus the costs of implementing selected aspects of this project, specifically training, renovations, and equipment purchases? What are any unintended consequences or spillover effects, positive or negative, from this project? Are there any best practices and lessons learned? For example, those around project approaches in the context of strong country ownership, a large amount of HIV funding (i.e. integration with other services), and the use of multiple partners for implementation. (Can also be pulled from other questions.) To what extent has service delivery quality improved at the facility and district levels? Were USAID and/or GOR gender policy principles incorporated in project design and implementation? What were the challenges and lessons learned? What was the cost-effectiveness of HCSP compared with other sub-Saharan Africa health systems strengthening projects? Capacity of the National and District Level Institutions What capacity-building results were achieved? What areas still require support? Has the number of staff trained to deliver HIV/AIDS and other integrated services resulted in increased output levels, especially up to GOR recommended levels??  Question is optional as it may increase the time needed to complete the evaluation. However the proposal can suggest how to answer this within a reasonable timeframe. 58 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION To what extent has the capacity (ability to deliver quality services as a result of training, coaching, mentoring) of staff changed as a result of USAID/USG investments? Has the process of capacity building been gender balanced? What was the proportion of male to female staff trained? What were successful techniques used to increase capacity for improved project implementation? What significant changes in capacity occurred at the central (national) level? For example, the number of central level staff trained and the development of tools and policies. How were they achieved? Were any assumptions that were made at the initiation of the project about Rwandan health system capacity-building methodologies confirmed or disproven over the course of the project? Sustainability and transition What have been the achievements of HCSP in assisting GOR to assume all technical and administrative oversight of HCSP-supported facilities and districts? What remains to be done and what are the challenges to achieving this? What are the key outcomes that are considered to be sustainable? What made them sustainable? What strategies or approaches has the GOR adopted from the IPs? Is there GOR uptake of IP interventions? What was achieved with graduation or sustainability plans? Were there any activities/outputs that were not accomplished that could affect sustainability? Expansion of HIV/AIDS Clinical Services in Institutions How many more facilities can deliver HIV/AIDS and integrated clinical-related services now, compared to the beginning of this project? What has been the change in services that can be delivered in the facilities now, different than those offered at the beginning of this project? To what extent has there been an increase in the number of people receiving services from the point of inception to now? To what extent has there been an increase in the percent of the population in the catchment area receiving services? What innovative or cutting-edge strategies, interventions, or techniques were used to increase the quantity and improve the quality of services? What has been done to promote gender equality in acceptance and delivery of services? What has been done to decrease discrimination and stigma among people living with HIV/AIDS? What is being done to address stigma and discrimination against MARPS and youth and to increase their access to facility-based services? What remains to be done? USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 59 Methodology The choice of design and methodology for this evaluation will be to maximize the highest quality and most credible evidence that will answer the hypothesis and accompanying questions, as well as to serve as a needs assessment for FHP. Therefore, the study team should use sound social science methods and tools. The final design and methodology will be developed by the team and approved by USAID/Rwanda. It is assumed that there will be a combination of tools, both quantitative (e.g., cost/benefit analysis, sampled surveys) and qualitative (e.g., key informant interviews, focus groups, and semi-structured interviews) research as well as desk reviews and field visits. No rigorously defined counterfactual (control) was defined prior to implementation; therefore, this will be a performance evaluation and will focus on the descriptive and normative questions: how was it implemented; what are the results; and how is it perceived and valued? There are general baseline data from the DHS and other assessments, but a specific baseline survey was not done prior to HCSP implementation. Each IP can provide nonscientific baseline information. A self-assessment done by the IPs will also be provided prior to the Evaluation Team’s initiation of in-country work. To gain a broader perspective, the team will draw upon international literature and standards. The Evaluation Team will also review existing documentation related to USAID support for Rwanda. The team will provide recommendations on remaining gaps and needs for consideration in future programming, especially for FHP. The team will consider the context in which USAID support is provided, with a special focus on contributions to national achievements, host country ownership, and transitioning to direct host government financing. Prior to the finalization of the design and development of tools, USAID/Rwanda will conduct a collaborative meeting with the Evaluation Team, HCSP IPs, and FHP to review the development hypothesis; evaluation questions; lists for interviews; and potential design, methodology, and timeframes. There should be another consultative meeting between the Evaluation Team and USAID/Rwanda and partners to discuss and finalize the process. The Evaluation Team will be responsible for incorporating suggestions, defining team responsibilities; developing a work plan, design matrix, and other research tools; and finalizing timelines. All work of the Evaluation Team must be reviewed and approved by USAID/Rwanda prior to commencement of work. Data analysis is an integral component of the entire evaluation process. It is envisioned that the Evaluation Team, after conferring with USAID/Rwanda, will develop and present a work plan and a design matrix. This plan should detail how both quantitative and qualitative data will be used to do the evaluation. Data will be disaggregated by gender in all applicable areas and into the three zones where IPs worked, both whenever possible. The Evaluation Team, when designing the evaluation and the methodology, should present any limitations of these methods. Such limitations could be related to timing, financial, language barriers or access. The Team should suggest any mitigation strategies to reduce any perceived limitations. 60 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION Implementer self-assessments might include:  IntraHealth  EGPAF  FHI 360 Key informant interviews might include:  USAID/Rwanda Health Service Delivery (HSD) Team Leader  USAID/Rwanda HSD team activity managers  Chiefs of Party and other appropriate staff from IntraHealth, EGPAF and FHI 360  Chief of Party, FHP  Head of HIV, MOH/Rwanda Biomedical Center  Head of MCH, MOH  Head of Clinical Services, MOH  Director, Performance-Based Financing, MOH/CAAC Field visits might include selected:  Administrative districts  District hospitals  Health centers  Faith-based organizations, e. g., Dioceses Team Composition, Skills, and Level of Effort (LOE) It is envisioned that the team will consist of three health and evaluation consultants, including a team leader, technical expert, and one local (or with significant Rwanda experience). The Evaluation Team members should be senior evaluation professionals who are well-respected in their areas of expertise and in the development community. The Team Leader should have additional skills and experience in project and team management. The other members should have solid public health (HIV/AIDS preferred) and/or evaluation research skills. Based on the agreed-upon evaluation design, additional local staff may be needed. GH Tech will be responsible for recruiting a logistics coordinator who would be responsible for making sure that the schedule was met. He or she will work with getting transportation needs met and will help with scheduling appointments and making hotel and other travel arrangements. GH Tech will also hire a local translator/driver while the team is conducting interviews of beneficiaries in the field. There will be a USAID/Rwanda representative assigned to work with the Evaluation Team who will be responsible for providing historical context and desk review documents and assisting with certain logistics and other agreed-upon tasks. Either USAID/Rwanda and/or its IPs will provide support staff to assist in evaluation activities. USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 61 The following is a list of some requirements for all Team Members:  Excellent general research skills, including experience doing performance evaluations using both quantitative and qualitative methods (5–7 years).  Knowledge of development project implementation management.  Knowledge of HIV/AIDS and other clinical services (5–7 years).  Experience with the Rwandan country context, required for 1 team member.  Experience working in Sub-Saharan Africa, desirable.  Fluency in English, fluency in French and/or Kinyarwanda desirable.  Excellent report writing abilities.  Excellent project management skills.  Ability to work well in a team.  Master’s or higher degree in research, epidemiology, public health, medicine, or other related discipline. The following is a list of more specific requirements for the Evaluation Team Leader:  Ability to liaise well with senior management in USAID, IPs, and GOR.  Proven ability to lead a team of highly qualified individuals (5+ years).  Excellent organizational and time management skills.  Working knowledge of finances and budget management.  Ability to manage and produce high quality deliverables. In any selection of internal support team members, gender balance should be a consideration where applicable and where the quality of work would not be compromised. Timeline USAID/Rwanda anticipates that the entire review would be completed within an eleven-week period, approximately seven weeks in country and four weeks home-based. In-country activities would include, but are not limited to, desk review, evaluation (including key informant interviews, focus groups, site visits, and review of site data), briefing, debriefing, and presentations. Home-based activities would include, but are not limited to, preparation, report writing, and dissemination. A six-day work week is approved while in Rwanda. XI. LOGISTICS USAID/Rwanda will offer assistance by assigning a Health Team staff member to work part-time, as needed, with the Team. The staff member may assist with identifying key documents, assisting with scheduling high-level meetings/appointments, intervening between all stakeholders, and assisting with logistics issues. 62 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION GH Tech, however, will be responsible for, but not limited to, the following:  Making all travel, transportation and lodging arrangements both to and from the country and internally  Hiring vehicles and drivers, as needed  Arranging for work space, computers, Internet access, phone access, printing. and photocopying  Arranging for space for focus groups and interviews  Making all payments for goods and services  Storing all evaluation data in a secure place  Returning all evaluation data to USAID/Rwanda at completion USAID/Rwanda will be available to the team for consultations regarding resources and technical issues during the assessment process. Preparatory Materials The following documents will be made available to the Evaluation Team prior to arrival for desk review:  HCSP work plans  HCSP Cooperative Agreements  HCSP Performance Monitoring Plans (including targets and results)  HCSP Quarterly and Annual Reports  HSSP-II  HSSP-III  HIV/AIDS National Strategic Plan Other documents that will be made available to the Team upon arrival:  HCSP Self-Assessments Also recommended for review, prior to arrival and during the evaluation, are public project documents found on the website of USAID, the IPs, and the GOR. In particular, review of these documents would be helpful: http://rwanda.usaid.gov/health%20Doc.html and http://moh.gov.rw. Additional background on USAID activities in Rwanda that may be of interest is available in USAID's Development Experience System at www.dec.org. Information concerning Rwanda HIV/AIDS statistics can be found at: www.cnls.gov.rw. Meetings and Briefings Upon arrival, the first full day will be devoted to meetings with USAID/Rwanda staff. The Statement of Work will be reviewed, discussed, and amended as appropriate. This will also be a time to finalize the scheduling of meetings with key informants. USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 63 There will be a consultative meeting with USAID/Rwanda, the HCSP IPs, and FHP prior to finalization of evaluation design and methodology. A debrief will be done with USAID/Rwanda to discuss preliminary findings prior to the departure of the Evaluation Team. Either a 3–5-page document or PowerPoint presentation highlighting methodology, key findings, and recommendations can be submitted at that time. XII. DELIVERABLES AND PRODUCTS GH Tech will provide all required reports, presentations, and other products as outlined in the SOW to the USAID/Rwanda AOR. All deliverables, excluding any required interviews in Kinyarwanda, will be submitted in English. All deliverables will be shared electronically. Deliverables will include:  IP self-evaluation forms  Final design and methodology  Final evaluation work plan  Copy of consultative meeting agenda and notes  Copies of all evaluation tools  Original data collected  PowerPoint presentation or debriefing document of findings, as applicable  Draft evaluation report  Final evaluation report  Final executive summary report (3–5 pages) A work plan and evaluation design for the evaluation shall be completed by the lead evaluator within two weeks of the start of this assignment and presented to the activity manager. The evaluation design will include a detailed evaluation design matrix (including the key questions, the methods, and data sources used to address each question), draft questionnaires and other data-collection instruments, and known limitations to the evaluation design. The final design requires activity manager approval. The work plan will include the anticipated schedule and logistical arrangements and delineate the roles and responsibilities of members of the Evaluation Team. The PowerPoint presentation should not exceed 30 slides. In lieu of a PowerPoint presentation, the Evaluation Team may decide with USAID/Rwanda to submit a 3–5 page debrief document during a debrief meeting. It should briefly describe the methodology, key findings, and recommendations. The presentation should be tailored to the specified audience as decided by USAID/Rwanda. The Evaluation Team will submit a draft report in English of its findings and recommendations to the activity manager within 10 business days from the time of return to their base offices. The final draft will be provided to the USAID/Rwanda activity manager in electronic form within three business days following receipt of comments from USAID. The report shall include an executive 64 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION summary and a list of abbreviations/acronyms and not exceed 50 pages (excluding appendices). The executive summary should be 3–5 pages in length and summarize the purpose, background of the project being evaluated, main evaluation questions, methods, findings, conclusions, and recommendations and lessons learned. The final report should follow USAID branding procedures and USAID Evaluation Policy (www.usaid.gov/evaluation/USAIDEvaluationPolicy.pdf). GH Tech Bridge will provide the edited and formatted final document after USAID provides final approval of the content. This process usually takes 30 days once GH Tech receives signoff on the final draft; therefore, GH Tech must receive sign-off by Nov. 23 in order to provide a final report. If USAID/Rwanda is not able to sign off on the report by this date, the final deliverable would be a final draft and the report could be finalized under a separate mechanism. If the report is finalized by GH Tech, the final report will be released as a public document on the USAID Development Experience Clearinghouse (DEC) (dec.usaid.gov) and the GH Tech project web site (www.ghtechproject.com). XIII. RELATIONSHIPS AND RESPONSIBILITIES GH Tech will coordinate and manage the Evaluation Team and will undertake the following specific responsibilities throughout the assignment:  Recruit and hire the Evaluation Team.  Make logistical arrangements for the consultants, including travel and transportation, country travel clearance, lodging, and communications. USAID/RWANDA will provide overall technical leadership and direction for the evaluation team throughout the assignment and will provide assistance with the following tasks: Before In-country Work  SOW. Respond to queries about the SOW and/or the assignment at large.  Consultant Conflict of Interest (COI). To avoid conflicts of interest or the appearance of a COI, review previous employers listed on the CVs for proposed consultants and provide additional information regarding potential COI with the project contractors evaluated/assessed and information regarding their affiliates.  Documents. Identify and prioritize background materials for the consultants and provide them to GH Tech, preferably in electronic form, at least one week prior to the inception of the assignment.  Local Consultants. Assist with identification of potential local consultants, including contact information.  Site Visit Preparations. Provide a list of site visit locations, key contacts, and suggested length of visit for use in planning in-country travel and accurate estimation of country travel line item costs.  Lodgings and Travel. Provide guidance on recommended secure hotels and methods of in￾country travel (i.e., car rental companies and other means of transportation) and if necessary, identify a person to assist with logistics (e.g., visa letters of invitation etc.). USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 65 During In-country Work  Mission Point of Contact. Throughout the in-country work, ensure constant availability of the Point of Contact person and provide technical leadership and direction for the team’s work.  Meeting Space. Provide guidance on the team’s selection of a meeting space for interviews and/or focus group discussions (USAID space if available, or other known office/hotel meeting space).  Meeting Arrangements. Assist the team in arranging and coordinating meetings with stakeholders.  Facilitate Contact with Implementing Partners. Introduce the Evaluation Team to implementing partners and other stakeholders, and where applicable and appropriate prepare and send out an introduction letter for team’s arrival and/or anticipated meetings. After In-country Work Timely Reviews. Provide timely review of draft/final reports and approval of deliverables. XIV. MISSION CONTACT PERSON Jennifer Slotnick Health Service Delivery Team Leader USAID/Rwanda +250-252-596-548 jslotnick@usaid.gov XV. COST ESTIMATE GH Tech will provide a cost estimate for this activity. 66 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 67 ANNEX B: ORGANIZATIONS AND PERSONS CONTACTED RWANDA U.S. Agency for International Development Slotnick, Jennifer USAID/Rwanda Mukamana, Esperance USAID/Rwanda Vasquez, Carol USAID/Rwanda Musoni, Census USAID/Rwanda HIV Care and Support Project—IntraHealth Lukaya Kassa, Jean-Luc Technical Advisor, Clinical Services Sempaswa, Emile Senior TA, MCH/FP Kayiranywe, Rose M. District Coordinator Ndikubwimana, Chantal District Coordinator Rutayisire, Jean-Bosco District Coordinator Mukasahaha, Diane Palliative Care Officer Mukakanona Viviane FP/MCH Coordinator Kamwesiga Julius Technical Advisor Clinical Services Uwamanow Kayiko, Chantal Integrated Services Technical Advisor Mahoro Mukabirasa, Maire-Grace District Coordinator (IH) Musabyimana, Edith M&E Team Leader Milligan, Crystal Program Manager Lewis, Sara Acting Chief of Party Chanda, Jonas Chief of Party (outgoing) Yambabariye, Cedric Health Systems TA HIV Care and Support Project—Family Health International 360 Seruntaga, Reverien AD, Finance and Operations Ross, Kimberly Chief of Party, HCSP Shumbusho, Fabienne Deputy Director Hutchison, Deborah Senior Program Officer 68 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION HIV Care and Support Project—Elizabeth Glaser Pediatric AIDS Foundation Ndatimana, Dieudonee M&E Coordinator Mukandanga, Odette PMTCT and Prevention, Technical Advisor and District Support Team Leader Gasore, Emile Technical Advisor and District Support Team Leader in Charge of Community Activity Dusabunurenyi, Jean Marie Grants Manager Van Zyl, Cornelia Country Director Dielemans, Paul Senior Technical Advisor MCH/FP/RH Nukaminega, Martha Senior Technical Officer, EGPAF Global Technical Director, Rwanda Program Habinshuti, Leon Augustin Acting Operations Associate Director Gahumba, Diane Senior Team Leader for Quality, FHP (Formerly QA for EGPAF) Family Health Project—Chemonics Youngs, Doris Chief of Party Gahumba, Diane Senior Team Leader for Quality Kamugundu, David Deputy Chief of Party for Technical Activities GTZ Girrbach, Elisabeth Health Coordinator GDC MCHIP Mukarugwiro, Dr. Beata Pascal, Dr. Musoni Rugwizangoga, Dr. Eugène MEMS Ekochu, Elizabeth Chief of Party Gasana, Emmanuel Deputy Chief of Party Tayebwa, Edwin M&E Officer Katengulia, Francoise M&E Officer Rwanda Ministry of Health Kankindi, Ida Director, Decentralization and Integration Unit Mugeni, Catherine Director, Community Health Ngabo, Fidel Director, Maternal and Child Health Sabimana, Sabin Director for HIV, RBC USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 69 United Nations AIDS (UNAIDS) Karago, Susan, M&E Advisor, UNAIDS (Rwanda) FIELD VISITS Kibungo District Hospital in Ngoma District (EGPAF) Namanya, Dr. William Hospital Director Hategekimana, Jean-Baptiste In charge of HIV services Collette Hospital Administrator Nzige Health Center in Rwamagana District (EGPAF) Ntwali, Jean-Damascène Deputy Director Butamwa Health Center in Nyarugenge District (EGPAF) Mugeni, Christine Deputy Head Mukarugema, Pauline In charge of health services Nyagatare District Hospital in Nyagatare District (IntraHealth) Sangara, Dr. Fred Acting Hospital Director Ndayambaje, Joseph Data Manager Rukozo Health Center in Rulindo District (IntraHealth) Bajyinama, Bernard Director Mukondo Health Center in Gicumbi District (IntraHealth) Ngegezehoguhora, Fidèle Deputy Director Kabgayi District Hospital in Muhanga District (FHI 360) Zulu, Dr. Chantal HIV Services Osée, Dr. Sebatunzi Director Sibomana, Vincent District Hospital Administrator Gahungu, Zacharie HR Manager Hakizimana, Peter Chief Accountant Kandama, Jacqueline IV Clinic Nurse Muhanga District Health Unit and Pharmacy in Muhanga District (FHI 360) Ndicunguye, Janvier Muhanga District Health Director Kamana, Sosthène District Pharmacy Director Ngoma Health Center in Nyaruguru District (FHI 360) Musabyimana, Favien Deputy Director Ruhango Health Center in Ruhango District (FHI 360) Bajyinama, Bernard Director 70 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 71 ANNEX C: REFERENCES BACKGROUND DOCUMENTS GOR/MOH, Oct. 2004 Health Sector Strategic Plan (HSSP I, 2005–2009) GOR/MOH, March, 2012 Health Sector Strategic Plan (HSSP II, 2012–2018) GOR/MOH, Dec. 2011 Health Situation Analysis 2011 GOR, Feb. 2005 Health Sector Policy GOR, July 2006 Rwanda Aid Policy (French) GOR/MOFEP, Sept. 2007 Economic Development and Poverty Reduction Strategy (EDPRS 2008–2012, English). External Evaluation Team, August 2011 Mid-Term Review of the HSSP I (2009–012), Final Evaluation Report MOH, Dec. 2011 Health Sector Performance Self-Assessment, 2011 MOH, Dec. 2008 Health Sector Performance Report, 2009 MOH, April 2009 Annual Report 2008, Final GOR/MOH, July 2009 Health Sector Strategic Plan (HSSP II, July 2005–June 2009) WB, Sept. 2009 Rwanda, Country Status Report (CSR) on Health and Poverty MINALOC, Dec 2009 The Rwanda Citizen Report and Community Score Cards 2009 (final) GOR/MOH, July 2009 Health Sector Strategic Plan (HSSP II, July 2009–June 2012) MOH, Oct. 2009 Joint Health Sector Performance Report (mini budget Jan-June) 2009 MOH, Oct. 2010 Annual Report July 2009–June 2010 final Terwindt, Frank, July 2010 Roadmap for Further Development of the Rwanda Health SWAp MOH, Oct. 2010 SWAp Procedures Manual (28 pages) MOH, Sept. 2010 SWAp Procedures Manual (PowerPoint presentation) GOR, Oct. 2010 Economic Development and Poverty Reduction Strategy (EDPRS) MOH, 2010 Implementation Report (June 2009–July 2010), WITH: Health Sector Performance Report July 2009–June 2010 MOH, NISR, ICF–Macro, April 2007 Interim Demographic and Health Survey (I-DHS 2007–2008) 72 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION BACKGROUND DOCUMENTS MOH, NISR, ICF-Macro, 2010 Demographic and Health Survey, (DHS 2010-2011), Preliminary Results (PowerPoint presentation) DHS, 2010 Report, October, July 2011 MOH website, undated Rwanda Indicators MOH, Oct 2011 Annual Statistical Booklet 2010 SERVICE DELIVERY AND PROGRAMS MOH, July 2003 National Reproductive Health Policy MOH, July 2005 National Medical Laboratory Policy (draft) CNLS, Dec, 2009 HIV National Strategic Plan 2009–2012 RBC, 2012 MTR, HIV National Strategic Plan 2009–2012 MOH/PNILT, Aug. 2005 Policy Statement on TB/HIV Collaborative Activities MOH, 2006 Family Planning Policy and Strategy 2006–2010 (in Kinyarwanda) GOR/MOH, Dec 2006 National Behaviour Change Communication Policy for Health Sector MOH/NCBT, May 2006 National Policy for Blood Transfusion USG DOCUMENTS AND REPORTS USAID, 2007 HCPS Project Description USAID Implementer Partners: Mapping for Services USAID/IP IP Progress Reports; Quarterly and Annual USAID/IP IP publications USAID/IP IP internal research USAID/IP IP work plans USAID/IP IP PMPs USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 73 ANNEX D: TOOLS Research Tools Research Questions Findings If USAID support via HCSP built capacity in district and national level Rwandan institutions to support and manage Rwanda’s health systems and expanded quality health service delivery, then the results would be improvements in (1) the quality and timeliness of routine data reporting; (2) financial management; (3) service delivery and technical capacity ; and (4) epidemiological outcomes among target populations. To what extent has the project been effective in increasing the technical and administrative capacity of Rwandan health systems, including improving quality and quantity of health services delivery? Were the project objectives met during the 5-year implementation? If yes, to what extent did HCSP IP interventions in human and institutional capacity building assist in the attainment of project objectives? Which HCSP IP interventions specifically contributed to these attainments? If objectives were not attained, what were the challenges (e.g., no interventions were deployed, interventions failed)? What needs to happen to achieve the objectives? What were the strengths and weaknesses of this project (especially in terms of those aspects detailed in the supporting questions)? To the extent that the project has had success in capacity building, is the GOR at all levels now in a position that will make the transition smoother? What opportunities and constraints face FHP and other future work to be done by USAID? Supporting Evaluation Questions General Was there collaboration between IPs? If so, what was the collaboration and what outcomes were affected? Were the program descriptions followed? Why or why not? Did these projects fulfill USAID/Rwanda and IP performance monitoring plans? If not, what were the areas of underperformance? To what extent was routine clinical data (i.e. HMIS, IQ chart) used for management decision-making/project improvement? What has been the major contributor for successful and/or non-successful implementation?  Indicators could be performance-based financing quality scores and quantitative results; proportion of HIV+ women receiving ARV for PMTCT, and lost to follow-up among enrolled HIV+ patients. 74 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION Research Tools Research Questions Findings What have been the benefits, specifically training, renovations, and equipment purchases? What are any unintended consequences or spillover effects, positive or negative, from this project? Are there any best practices and lessons learned, such as those around project approaches in the context of strong country ownership, a large amount of HIV funding (i.e. integration with other services), and the use of multiple partners for implementation? To what extent has service delivery quality improved at the facility and district levels? Capability of the national and district level institutions What capacity building results were achieved? What areas still require support? Has the number of staff trained to deliver HIV/AIDS and other integrated services resulted in increased outputs, especially up to GOR recommended levels? To what extent has the capacity of staff (ability to deliver quality services as a result of training, coaching, mentoring) changed as a result of USAID/USG investments? Has the process of capacity building been gender-balanced? What was the proportion of male to female staff trained? What techniques to increase capacity for improved project implementation were successful? What significant changes in capacity occurred at the central (national) level, such as the number of central staff trained and the development of tools and policies? How were they achieved? Were any assumptions that were made at the initiation of the project about Rwandan health system capacity-building methodologies confirmed or disproven over the course of the project? Sustainability and transition What have been the achievements of HCSP in assisting GOR to assume all technical and administrative oversight of HCSP￾supported facilities and districts? What remains to be done and what are the challenges to doing it? What are the key outcomes that are considered to be sustainable? What made them sustainable? What strategies or approaches has the GOR adopted from the IPs? Is there GOR uptake of IP interventions? What was achieved with graduation or sustainability plans? Were there any activities/outputs that were not accomplished that could affect sustainability? USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 75 Research Tools Research Questions Findings Expansion of HIV/AIDS clinical services in institutions How many more facilities can deliver HIV/AIDS and integrated clinical related services now, compared to the beginning of this project? How are the services that can be delivered in the facilities now different from those offered at the beginning of this project? To what extent has there been an increase in the number of people receiving services from the point of inception to now? To what extent has there been an increase in the percent of the population in the catchment area receiving services? What innovative or cutting-edge strategies, interventions, or techniques were used to increase quantity and improve quality services? What has been done to promote gender equality in acceptance and delivery of services? What has been done to decrease discrimination and stigma among people living with HIV/AIDS? 76 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 77 ANNEX E: INDICATOR TARGETS AND ACHIEVEMENTS—MEMS PEPFAR Indicators: Summary for Last Three Years of HCSP Implemented by EGPAF Indicators TGT10 (target) APR10 (result) Percentage of Achievement Y3 TGT11 (targets) APR11 (results) Percentage of Achievement Y4 TGT12 (target) APR12 (result) Percentage of Achievement Y5 PMTCT # Service outlets 35 47 134.3% 38 43 113.2% 41 51 124.4% P1.1.D. # Pregnant women with known HIV status (women who were tested for HIV and received their results) 32,851 31,699 96.5% 35,381 36,263 102.5% 40,455 3,479 8.6% Those who were counseled in PMTCT setting 0 30,878 0 35,676 0 34,156 # Those who were tested (including ANC + Maternity) 0 30,738 0 35,676 0 34,156 # Those who were tested and found HIV+ (even if results not taken) 0 898 0 667 0 498 # HIV+ women who delivered during the reporting period 0 1,345 623 1,425 228.7% # Exposed infants who tested HIV+ at 6 weeks 0 14 # Exposed infants who tested HIV+ at 18 months 0 3 P1.2.D. # HIV+ pregnant women who received ARVs to reduce risk of MTCT 0 1,652 1,338 1,051 78.6% 1,413 1,126 79.7% # women by ART dosage received 0 144 0 1,051 0 1,126 # Trained 70 247 352.9% 132 187 141.7% 48 92 191.7% 78 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION PEPFAR Indicators: Summary for Last Three Years of HCSP Implemented by EGPAF Indicators TGT10 (target) APR10 (result) Percentage of Achievement Y3 TGT11 (targets) APR11 (results) Percentage of Achievement Y4 TGT12 (target) APR12 (result) Percentage of Achievement Y5 Male Circumcision # Service outlets 0 0 0 0 0 0 P5.1.D. # males circumcised as part of the minimum package of HIV prevention services 0 0 0 0 0 0 # Clients experiencing one or more adverse events 0 0 0 0 # Trained 0 0 0 0 0 0 Post-Exposure Prophylaxis # Service outlets 27 30 111.1% 30 30 100.0% 30 50 166.7% P6.1.D # Persons provided with PEP 27 101 374.1% 90 169 187.8% 180 395 219.4% # Trained 0 0 0 0 0 48 PWP HIV-Community Based # Service outlets 40 0 0.0% 0 0 0 0 P7.1.D. # PLWHA reached with a minimum package of Prevention with PLWHA (PwP) interventions 7,178 0 0.0% 17,908 0 0.0% 0 0 # Trained 0 0 43 0 0.0% 0 0 PWP HIV-Facility # Service outlets 0 41 0 42 0 49 P7.1.D. # PLWHA reached with a minimum package of prevention with PLWHA (PwP) interventions 0 13,521 0 13,353 15,532 17,905 115.3% USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 79 PEPFAR Indicators: Summary for Last Three Years of HCSP Implemented by EGPAF Indicators TGT10 (target) APR10 (result) Percentage of Achievement Y3 TGT11 (targets) APR11 (results) Percentage of Achievement Y4 TGT12 (target) APR12 (result) Percentage of Achievement Y5 # Trained 0 0 0 0 0 48 Sexual and other Risk Prevention # Service outlets 0 0 0 0 0 0 P8.1.D. # Target population reached with individual or small group preventive interventions 0 0 0 0 0 0 P8.2.D. # Target population reached with individual or small group preventive interventions primarily focused on AB 0 0 0 0 0 0 P8.3.D. # MARP reached with individual and/or small group level interventions based on evidence and/or meet the minimum standards required 0 0 0 0 0 0 P8.5.D. # Individuals from target audience who participated in community-wide event 0 0 0 0 0 0 # Trained 0 0 0 0 0 0 Testing and Counseling # Service Outlets 36 38 105.6% 38 41 107.9% 42 48 114.3% # Individuals who received TC services for HIV and received their test results under VCT program 114,753 152,829 133.2% 175,002 171,399 97.9% 21,088 197,204 935.1% 80 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION PEPFAR Indicators: Summary for Last Three Years of HCSP Implemented by EGPAF Indicators TGT10 (target) APR10 (result) Percentage of Achievement Y3 TGT11 (targets) APR11 (results) Percentage of Achievement Y4 TGT12 (target) APR12 (result) Percentage of Achievement Y5 # Individuals who received TC services for HIV and received their test results through PIT 0 36,599 0 70,407 41,003 103,148 251.6% # Individuals who received TC services for HIV and received their test results under TB Program 0 0 0 0 0 0 # Individuals who received TC services for HIV and received their test results under EID Program 0 1,114 0 1,074 8,785 898 10.2% # Males who received TC services for HIV and received their test results under MC Program 0 0 0 0 0 0 # Male partners who received TC services for HIV and received their test results under PMTCT Program 0 24,004 0 28,312 32,217 26,408 82.0% # Couples Tested 0 34,125 0 37,189 0 33,787 # Individuals found HIV+ 0 5,876 0 3,864 0 3,164 # Trained 66 151 228.8% 90 60 66.7% 65 38 58.5% Clinical Care # Service outlets 40 44 110.0% 75 44 58.7% 46 50 108.7% C2.1.D. # HIV+ adults and children receiving at least one clinical service 20,727 1,683 8.1% 17,503 16,645 95.1% 19,413 17,905 92.2% C2.2.D. # HIV+ persons receiving cotrimoxazole prophylaxis 18,654 13,825 74.1% 15,752 16,645 105.7% 17,472 17,757 101.6% # HIV+ clinically malnourished 0 0 0 0 0 0 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 81 PEPFAR Indicators: Summary for Last Three Years of HCSP Implemented by EGPAF Indicators TGT10 (target) APR10 (result) Percentage of Achievement Y3 TGT11 (targets) APR11 (results) Percentage of Achievement Y4 TGT12 (target) APR12 (result) Percentage of Achievement Y5 C2.3.D. Clinical Malnutrition # HIV+ who received therapeutic or supplementary food 0 0 0 0 0 0 C2.3.D. Clinical Malnutrition # HIV+ who received therapeutic or supplementary food 0 0 0 0 0 0 C2.3.D. Clinical Malnutrition # HIV+ who received therapeutic or supplementary food 0 0 0 0 C2.3.D. Clinical Malnutrition # HIV+ who received therapeutic or supplementary food 0 0 0 0 # HIV+ pregnant women who received therapeutic or supplementary food 0 0 0 0 C2.4.D TB/HIV: # HIV+ patients screened for TB in HIV care or treatment settings 18,654 13,521 72.5% 15,752 13,353 84.8% 14,561 15,843 108.8% C2.5.D TB/HIV: # HIV+ patients in HIV care or treatment (pre-ART or ART) who started TB treatment 207 147 71.0% 175 171 97.7% 193 101 52.3% C4.1.D. # Infants born to HIV+ women who were tested within 12 months of birth 1,918 1,048 54.6% 1,137 1,023 90.0% 1,344 898 66.8% C4.2.D. # Infants born to HIV+ pregnant women who are started on CTX prophylaxis within two months of birth 1,918 947 49.4% 1,137 951 83.6% 1,344 958 71.3% 82 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION PEPFAR Indicators: Summary for Last Three Years of HCSP Implemented by EGPAF Indicators TGT10 (target) APR10 (result) Percentage of Achievement Y3 TGT11 (targets) APR11 (results) Percentage of Achievement Y4 TGT12 (target) APR12 (result) Percentage of Achievement Y5 # Trained 70 157 224.3% 120 208 173.3% 76 218 286.8% Support Care # Service outlets 0 0 0 0 0 0 C5.1.D. # Eligible clients who received food and/or other nutrition services 0 0 0 0 0 0 C5.2.D. # Eligible children provided with shelter and given care 0 0 0 0 0 0 C5.3.D. # Eligible children provided with health care referral 0 0 0 0 0 0 C5.4.D. # Eligible children provided with education or vocational training 0 0 0 0 0 0 C5.5.D. # Eligible adults and children provided with Protection and Legal Aid services 0 0 0 0 0 0 C5.6.D. # Eligible adults and children provided with psychological, social, or spiritual support 0 0 0 0 0 0 C5.7.D. # Eligible adults and children provided with Economic Strengthening services 0 0 0 0 0 0 # Trained 0 0 0 0 0 0 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 83 PEPFAR Indicators: Summary for Last Three Years of HCSP Implemented by EGPAF Indicators TGT10 (target) APR10 (result) Percentage of Achievement Y3 TGT11 (targets) APR11 (results) Percentage of Achievement Y4 TGT12 (target) APR12 (result) Percentage of Achievement Y5 OVC Program # OVC served 0 0 0 0 # Known HIV + 0 0 0 0 ART Services # Service outlets 27 36.0 133.3% 30 32 106.7% 30 51.0 170.0% T1.1.D. # Adults and children with advanced HIV infection newly enrolled on ART 1,844 2,052 111.3% 1,525 1,705 111.8% 1,914 1,941 101.4% # Pregnant women newly initiated on ART 0 194 0 154 0 382 # Newly initiated on ART in similar previous reporting period (includes transfers-in with known initiation dates) 0 1,704 0 1,569 0 1,916 T1.2.D. # Adults and children with advanced HIV infection receiving antiretroviral therapy (current) 7,777 7,516 96.6% 8,837 8,752 99.0% 10,739 11,059 103.0% T1.3.D. # Adults and children known to be alive and on treatment 12 months after initiation of ART 1,539 1,425 92.6% 166 1,451 874.1% 153 1,826 1193.5% T1.4.D. # Adults and children with advanced HIV-infection who EVER started on ART 9,357 9,569 102.3% 10,882 11,848 108.9% 14,593 14,993 102.7% # Trained 54 141 261.1% 340 61 17.9% 150 48 32.0% 84 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION PEPFAR Indicators: Summary for Last Three Years of HCSP Implemented by EGPAF Indicators TGT10 (target) APR10 (result) Percentage of Achievement Y3 TGT11 (targets) APR11 (results) Percentage of Achievement Y4 TGT12 (target) APR12 (result) Percentage of Achievement Y5 Laboratory Services # Service outlets 0 45 0 48 48 54 112.5% H1.1.D. # Testing facilities (laboratories) with capacity to perform clinical laboratory tests 22 45 204.5% 48 45 93.8% 48 48 100.0% H1.2.D. # Testing facilities (laboratories) accredited according to national or international standards 0 0 0 0 0 0 # Trained 0 21 0 5 0 103 Human Resources for Health # Service outlets 0 45 0 0 48 0 0.0% H2.1.D. # New health care workers who graduated from a pre-service training institution 0 0 0 0 0 0 H2.2.D. # Community health and para-social workers who successfully completed a pre-service training program 0 0 0 0 0 0 H2.3.D. # Health care workers who successfully completed an in-service training program 180 676 375.6% 569 802 140.9% 439 497 113.2% Sexual and Gender Based Violence (SGBV) # Service outlets 0 30 0 50 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 85 PEPFAR Indicators: Summary for Last Three Years of HCSP Implemented by EGPAF Indicators TGT10 (target) APR10 (result) Percentage of Achievement Y3 TGT11 (targets) APR11 (results) Percentage of Achievement Y4 TGT12 (target) APR12 (result) Percentage of Achievement Y5 P12.1.D. # People reached by an individual, small-group, or community intervention or service that explicitly addresses norms about masculinity related to HIV/AIDS. 0 0 0 0 P12.2.D. # People reached by an individual, small group or community intervention or service that explicitly addresses gender-based violence and coercion related to HIV/AIDS. 0 0 300 147 49.0% P12.4.D. # People reached by an individual, small group, or community intervention or service that explicitly aims to increase access to income and productive resources of women and girls impacted by HIV/AIDS 0 0 # new SGBV cases that received care and support through USG￾supported programs 0 85 90 229 254.4% 86 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION PEPFAR Indicators Summary for Last Three Years of HCSP Implemented by FHI Indicators TGT10 (target) APR10 (results) Percentage of Achievement Y3 TGT11 (target) APR11 (results) Percentage of Achievement Y4 TGT12 (target) APR12 (results) Percentage of Achievement Y5 PMTCT # Service outlets 60 62 103.3% 61 64 104.9% 60 64.0 106.7% P1.1.D.# Pregnant women with known HIV status (includes women who were tested for HIV and received their results) 3,867 35,424 35,345 39,497 41,474 37,398 90.2% # Counseled in PMTCT setting 0 35,606 349 38,928 40,875 37,398 91.5% # Tested (including ANC + Maternity) 0 35,159 349 38,928 40,875 395 1.0% # Tested and found HIV+ (even if results not taken) 0 989 976 614 644 1,185 184.0% # HIV+ women who delivered during the reporting period 0 888 1,135 84 74.0% # Infants exposed who tested HIV+ at 6 weeks 0 15 # Infants exposed who tested HIV+ at 18 months 0 4 P1.2.D. # HIV+ pregnant women who received antiretrovirals to reduce risk of MTCT 0 1,129 113 1,138 1,195 1,032 86.4% # Women by ART dosage received 0 1,129 509 1,138 1,195 1,032 86.4% # Trained 313 265 84.7% 473 1,199 253.5% 100 518 518.0% Male Circumcision # Service outlets 0 0 0 0 0 0 P5.1.D.# Males circumcised as part of the minimum package of HIV prevention services 0 0 0 0 0 0 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 87 PEPFAR Indicators Summary for Last Three Years of HCSP Implemented by FHI Indicators TGT10 (target) APR10 (results) Percentage of Achievement Y3 TGT11 (target) APR11 (results) Percentage of Achievement Y4 TGT12 (target) APR12 (results) Percentage of Achievement Y5 # Clients experiencing one or more adverse events 0 0 0 # Trained 0 0 0 0 0 0 Post-Exposure Prophylaxis # Service outlets 0 47 45 72 160.0% 68 73 107.4% P6.1.D # persons provided with PEP 0 106 55 282 512.7% 340 472 138.8% # Trained 107 245 229.0% 72 186 258.3% 100 555 555.0% PWP HIV-Community Based # Service outlets 0 0 0 0 0 0 P7.1.D.# PLWHA reached with a minimum package of prevention with PLWHA (PwP) interventions 15,165 0 21,032 0 0 0 # Trained 153.0 0.0 148.0 0.0 0.0 0.0 PWP HIV-Facility # Service Outlets 0 70 0 71 0 73 P7.1.D.# PLWHA reached with a minimum package of prevention with PLWHA (PwP) interventions 0 19,471 23,562 21,724 92.2% 2,417 21,537 891.1% # Trained 0 323 138 128 92.8% 100 555 555.0% Sexual and other Risk Prevention # Service outlets 0 2 2 2 3 18 600.0% P8.1.D.# Target population reached with individual or small group preventive interventions 0 0 0 128 160 0 0.0% 88 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION PEPFAR Indicators Summary for Last Three Years of HCSP Implemented by FHI Indicators TGT10 (target) APR10 (results) Percentage of Achievement Y3 TGT11 (target) APR11 (results) Percentage of Achievement Y4 TGT12 (target) APR12 (results) Percentage of Achievement Y5 P8.2.D.# Target population reached with individual or small group preventive interventions primarily focused on AB 0 0 0 0 0 0 #DIV/0! P8.3.D.# MARP reached with individual or small group interventions based on evidence or meet the minimum standards required 363 147 40.5% 180 128 71.1% 160 85 53.1% P8.5.D.# Individuals from target audience who participated in community-wide event 0 0 0 0 0 0 # Trained 17 120 705.9% 0 175 0 33 Testing and Counseling # Service outlets 0 66 62 66 106.5% 61 73 119.7% # Individuals who received TC services for HIV and received their test results under VCT program 11,091 145,356 1310.6% 130,417 175,224 134.4% 183,985 175,188 95.2% # Individuals who received TC services for HIV and received their test results through PIT 0 21,672 0 56,898 59,742 79,911 133.8% # Individuals who received TC services for HIV and received their test results under TB program 0 751 0 591 621 485 78.1% # Individuals who received TC services for HIV and received their test results under EID program 0 0 0 1,013 1,195 970 81.2% # Males who received TC services for HIV and received their test results under MC program 0 0 0 0 0 0 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 89 PEPFAR Indicators Summary for Last Three Years of HCSP Implemented by FHI Indicators TGT10 (target) APR10 (results) Percentage of Achievement Y3 TGT11 (target) APR11 (results) Percentage of Achievement Y4 TGT12 (target) APR12 (results) Percentage of Achievement Y5 # Male partners who received TC services for HIV and received their test results under PMTCT program 0 30,352 0 34,017 35,718 31,897 89.3% # Couples tested 0 45,036 0 46,374 48,694 42,913 88.1% # Individuals found HIV+ 0 5,414 0 4,312 4,521 4,538 100.4% # Trained 586 228 38.9% 132 126 95.5% 183 555 303.3% Clinical Care # Service outlets 0 70 69 72 104.3% 68 73 107.4% C2.1.D.# HIV+ adults and children receiving a minimum of one clinical service 22,955 22,064 96.1% 26,181 2,208 8.4% 24,664 21,767 88.3% C2.2.D.# HIV+ persons receiving cotrimoxazole prophylaxis 21,667 19,471 89.9% 23,562 21,724 92.2% 2,417 21,537 891.1% # HIV+ persons clinically malnourished 0 0 0 0 0 0 C2.3.D. Clinical Malnutrition: # HIV+ persons who received therapeutic or supplementary food 0 0 0 0 0 0 C2.3.D. Clinical Malnutrition # HIV+ persons who received therapeutic or supplementary food 0 0 0 0 0 0 C2.3.D. Clinical Malnutrition # HIV+ persons who received therapeutic or supplementary food 0 0 0 0 C2.3.D. Clinical Malnutrition # HIV+ persons who received therapeutic or supplementary food 0 0 0 0 0 0 90 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION PEPFAR Indicators Summary for Last Three Years of HCSP Implemented by FHI Indicators TGT10 (target) APR10 (results) Percentage of Achievement Y3 TGT11 (target) APR11 (results) Percentage of Achievement Y4 TGT12 (target) APR12 (results) Percentage of Achievement Y5 # HIV+ pregnant women who received therapeutic or supplementary food 19,174 19,583 102.1% 23,562 19,691 83.6% 0 0 C2.4.D TB/HIV: # HIV+ patients who were screened for TB in HIV care or treatment settings 532 343 64.5% 22,196 19,497 87.8% C2.5.D TB/HIV: # HIV+ patients in HIV care or treatment (pre-ART or ART) who started TB treatment 262 503 192.0% 1,107 1,013 91.5% 360 293 81.4% C4.1.D.# Infants born to HIV+ women who received an HIV test within 12 months of birth 1,006 1,788 177.7% 1,195 970 81.2% C4.2.D.# Infants born to HIV+ women started on CTX prophylaxis within two months of birth 1,198 1,218 101.7% 1,107 1,003 90.6% 1,195 951 79.6% # Trained 222 323 145.5% 138 130 94.2% 100 555 555.0% Support Care # Service outlets 0 0 0 0 0 0 C5.1.D.# Eligible clients who received food or other nutrition services 0 0 0 0 0 0 C5.2.D.# Eligible children provided with shelter and given care 0 0 0 0 0 0 C5.3.D.# Eligible children provided with health care referral 0 0 0 0 0 0 C5.4.D.# Eligible children provided with education or vocational training 0 0 0 0 0 0 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 91 PEPFAR Indicators Summary for Last Three Years of HCSP Implemented by FHI Indicators TGT10 (target) APR10 (results) Percentage of Achievement Y3 TGT11 (target) APR11 (results) Percentage of Achievement Y4 TGT12 (target) APR12 (results) Percentage of Achievement Y5 C5.5.D.# Eligible adults and children provided with Protection and Legal Aid services 0 0 0 0 0 0 C5.6.D.# Eligible adults and children provided with psychological, social, or spiritual support 0 0 0 0 0 0 C5.7.D.# Eligible adults and children provided with economic strengthening services 0 0 0 0 0 0 # Trained 0 0 0 0 0 0 OVC Program # OVC served 0 0 0 0 # Known HIV + 0 0 0 0 ART Services # Service outlets 41 47 114.6% 45 52 115.6% 45 73 162.2% T1.1.D.# Adults and children with advanced HIV infection newly enrolled on ART 2,754 2,316 84.1% 2,577 2,644 102.6% 2,777 2,399 86.4% # Pregnant women newly initiated on ART 0 157 180 278 154.4% 292 404 138.4% # Newly initiated on ART in similar previous reporting period (includes transfers-in with known initiation dates) 0 2,671 2,505 2,617 104.5% 2,644 2,922 110.5% T1.2.D.# Adults and children with advanced HIV infection receiving antiretroviral therapy (current) 13,775 12,847 93.3% 15,287 14,199 92.9% 16,976 15,735 92.7% 92 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION PEPFAR Indicators Summary for Last Three Years of HCSP Implemented by FHI Indicators TGT10 (target) APR10 (results) Percentage of Achievement Y3 TGT11 (target) APR11 (results) Percentage of Achievement Y4 TGT12 (target) APR12 (results) Percentage of Achievement Y5 T1.3.D.# Adults and children known to be alive and on treatment 12 months after initiation of ART 2,716 3,019 111.2% 2,565 2,333 91.0% 2,498 2,498 100.0% T1.4.D.# Adults and children with advanced HIV-infection who EVER started on ART 15,642 15,387 98.4% 17,561 18,768 106.9% 21,545 21,167 98.2% # Trained 48 253 48 186 387.5% 100 555 555.0% Laboratory Services # Service outlets 0 33 42 67 159.5% 68 68 100.0% H1.1.D.# Testing facilities (laboratories) with capacity to perform clinical laboratory tests 0 33 35 67 191.4% 68 68 100.0% H1.2.D.# Testing facilities (laboratories) accredited according to national or international standards 0 0 0 0 0 0 # Trained 0 14 35 0 0 0 Human Resources for Health # Service outlets 0 80 69. 84 121.7% 68 83 122.1% H2.1.D.# New health care workers who graduated from a pre-service training institution 0 0 0 0 0 0 H2.2.D.# Community health and para-social workers who successfully completed a pre-service training program 0 0 0 0 0 0 H2.3.D.# Health care workers who successfully completed an in-service training program 0 640 473 2,127 449.7% 400 555 138.8% USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 93 PEPFAR Indicators Summary for Last Three Years of HCSP Implemented by FHI Indicators TGT10 (target) APR10 (results) Percentage of Achievement Y3 TGT11 (target) APR11 (results) Percentage of Achievement Y4 TGT12 (target) APR12 (results) Percentage of Achievement Y5 Sexual and Gender Based Violence (SGBV) # Service outlets 0 73 0 68 P12.1.D # People reached by an individual, small-group, or community intervention or service that explicitly addresses norms about masculinity related to HIV/AIDS. 0 0 0 0 P12.2.D # People reached by an individual, small group or community intervention or service that explicitly addresses gender-based violence and coercion related to HIV/AIDS. 0 60,298 72,358 137,196 189.6% P12.4.D # People reached by an individual, small group, or community intervention or service that explicitly aims to increase access to income and productive resources of women and girls impacted by HIV/AIDS 0 0 # New SGBV cases who received care and support through USG￾supported programs 0 679 816 2,109 258.5% # Individuals trained in SGBV service provision and clinical management 0 208 0 0 94 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION PEPFAR Indicators Summary for Last Three Years of HCSP Implemented by IntraHealth International Indicators TGT10 (targets) APR10 (results) Percentage of Achievement Y3 TGT11 (targets) APR11 (results) Percentage of Achievement Y4 TGT12 (targets) APR12 (results) Percentage of Achievement Y5 PMTCT # Service outlets 39 45 115.4% 39 41 105.1% 39 41 105.1% P1.1.D.# Pregnant women with known HIV status (includes women who were tested for HIV and received their results) 22,443 2,157 9.6% 24,037 27,573 114.7% 28,028 2,411 8.6% # Counseled in PMTCT setting 0 21,632 24,037 27,086 112.7% 28,028 24,108 86.0% # Tested (including ANC + Maternity) 0 2,157 24,037 26,629 110.8% 28,028 23,718 84.6% # Tested and found HIV+ (even if results not taken) 0 515 716 392 54.7% 736 257 34.9% # HIV+ women who delivered during the reporting period 0 479 0 537 998 667 66.8% # Infants exposed who tested HIV+ at 6 weeks 0 478 0 9 # Infants exposed who tested HIV+ at 18 months 404 451 111.6% 0 5 P1.2.D.# HIV+ pregnant women who received ARVs to reduce risk of MTCT 716 731 102.1% 751 619 82.4% # Women by ART dosage received 0 733 751 619 82.4% # Trained 404 433 107.2% 78 138 176.9% Male Circumcision # Service outlets 0 0 0 0 0 0 P5.1.D.# Males circumcised as part of the minimum package of HIV prevention services 0 0 0 0 0 0 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 95 PEPFAR Indicators Summary for Last Three Years of HCSP Implemented by IntraHealth International Indicators TGT10 (targets) APR10 (results) Percentage of Achievement Y3 TGT11 (targets) APR11 (results) Percentage of Achievement Y4 TGT12 (targets) APR12 (results) Percentage of Achievement Y5 # Clients experiencing one or more adverse events 0 0 0 0 # Trained 0 0 0 0 0 0 Post-Exposure Prophylaxis # Service outlets 45 41 91.1% 45 44 97.8% 45 45 100.0% P6.1.D # Persons provided with PEP 82 274 334.1% 82 250 304.9% 350 373 106.6% # Trained 404 372 92.1% 404 427 105.7% 90 131 145.6% PWP HIV-Community Based # Service outlets 39 40 102.6% 39 0 0.0% 35 0 0.0% P7.1.D.# PLWHA reached with a minimum package of prevention with PLWHA (PwP) interventions 92 12,521 13609.8% 12,415 0 0.0% 9,312 0 0.0% # Trained 280 368 131.4% 404 0 0.0% 304 0 0.0% PWP HIV-Facility # Service outlets 0 44 0 45 P7.1.D.# PLWHA reached with a minimum package of Prevention with PLWHA (PwP) interventions 0 14,296 15,425 11,625 75.4% # Trained 0 650 90 228 253.3% Sexual and Other Risk Prevention # Service outlets 0 0 0 0 0 0 P8.1.D.# Target population reached with individual or small￾group preventive interventions 0 0 0 0 0 0 96 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION PEPFAR Indicators Summary for Last Three Years of HCSP Implemented by IntraHealth International Indicators TGT10 (targets) APR10 (results) Percentage of Achievement Y3 TGT11 (targets) APR11 (results) Percentage of Achievement Y4 TGT12 (targets) APR12 (results) Percentage of Achievement Y5 P8.2.D.# Target population reached with individual or small group preventive interventions primarily focused on AB 0 0 0 0 0 0 P8.3.D.# MARP reached with individual or small group interventions based on evidence or meet the minimum standards required 0 0 0 0 0 0 P8.5.D.# Individuals from target audience who participated in community-wide event 0 0 0 0 0 0 # Trained 0 0 0 0 0 0 Testing and Counseling # Service outlets 39 42 107.7% 39 45 115.4% 39 42 107.7% # Individuals who received TC services for HIV and received their test results under VCT program 114 120,502 105703.5% 12,885 106,196 824.2% 1,349 77,895 5774.3% # Individuals who received TC services for HIV and received their test results through PIT 0 0 0 29,785 31,027 35,786 115.3% # Individuals who received TC services for HIV and received their test results under TB program 0 663 0 679 1,355 529 39.0% # Individuals who received TC services for HIV and received their test results under EID program 0 0 0 498 818 584 71.4% USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 97 PEPFAR Indicators Summary for Last Three Years of HCSP Implemented by IntraHealth International Indicators TGT10 (targets) APR10 (results) Percentage of Achievement Y3 TGT11 (targets) APR11 (results) Percentage of Achievement Y4 TGT12 (targets) APR12 (results) Percentage of Achievement Y5 # Males who received TC services for HIV and received their test results under MC program 0 0 0 0 0 0 # Male partners who received TC services for HIV and received their test results under PMTCT program 0 17,219 0 21,886 22,023 19,099 86.7% # Couples tested 0 26,052 0 29,406 28,028 25,245 90.1% # Individuals found HIV+ 0 3,375 3,864 3,089 79.9% 397 2,036 512.8% # Trained 404 379 93.8% 404 111 27.5% 78 26 33.3% Clinical Care # Service outlets 39 41 105.1% 39 44 112.8% 45 45 100.0% C2.1.D.# HIV+ adults and children receiving at least one clinical service 115 12,521 10887.8% 15,518 12,754 82.2% 13,099 11,625 88.7% C2.2.D.# HIV+ persons receiving cotrimoxazole prophylaxis 115 10,733 9333.0% 15,518 10,317 66.5% 11,527 11,486 99.6% # HIV+ persons clinically malnourished 0 1,163 0 1,148 86 985 1145.3% C2.3.D. # Clinically malnourished HIV+ persons who received therapeutic or supplementary food 112 923 824.1% 103 0 0.0% 1,058 921 87.1% C2.3.D. # Clinically malnourished HIV+ persons who received therapeutic or supplementary food 112 923 824.1% 103 0 0.0% 1,058 921 87.1% 98 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION PEPFAR Indicators Summary for Last Three Years of HCSP Implemented by IntraHealth International Indicators TGT10 (targets) APR10 (results) Percentage of Achievement Y3 TGT11 (targets) APR11 (results) Percentage of Achievement Y4 TGT12 (targets) APR12 (results) Percentage of Achievement Y5 # HIV+ pregnant women who received therapeutic or supplementary food 0 0 0 413 C2.4.D TB/HIV: # HIV+ patients who were screened for TB in HIV care or treatment settings 115 1,227 1067.0% 15,518 11,027 71.1% 11,527 11,486 99.6% C2.5.D TB/HIV: # HIV+ patients in HIV care or treatment (pre￾ART or ART) who started TB treatment 115 200 173.9% 155 208 134.2% 185 124 67.0% C4.1.D.# Infants born to HIV+ women tested for HIV within 12 months of birth 608 481 79.1% 680 498 73.2% 676 524 77.5% C4.2.D.# Infants born to HIV+ pregnant women started on CTX prophylaxis within two months of birth 608 449 73.8% 680 533 78.4% 676 536 79.3% # Trained 404 372 92.1% 404 102 25.2% 90 228 253.3% Support Care # Service outlets 0 0 0 0 0 0 C5.1.D.# Eligible clients who received food or other nutrition services 0 0 0 0 0 0 C5.2.D.# Eligible children provided with shelter and given care 0 0 0 0 0 0 C5.3.D.# Eligible children provided with health care referral 0 0 0 0 0 0 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 99 PEPFAR Indicators Summary for Last Three Years of HCSP Implemented by IntraHealth International Indicators TGT10 (targets) APR10 (results) Percentage of Achievement Y3 TGT11 (targets) APR11 (results) Percentage of Achievement Y4 TGT12 (targets) APR12 (results) Percentage of Achievement Y5 C5.4.D.# Eligible children provided with education or vocational training 0 0 0 0 0 0 C5.5.D.# Eligible adults and children provided with Protection and Legal Aid services 0 0 0 0 0 0 C5.6.D.# Eligible adults and children provided with psychological, social, or spiritual support 0 0 0 0 0 0 C5.7.D.# Eligible adults and children provided with economic strengthening services 0 0 0 0 0 0 # Trained 0 0 0 0 0 0 OVC Program # OVC served 0 0 0 0 # Known HIV + 0 0 0 0 ART Services # Service outlets 25 25 100.0% 25 25 100.0% 25 34 136.0% T1.1.D.# Adults and children with advanced HIV infection newly enrolled on ART 1,584 1,369 86.4% 213 1,078 506.1% 146 1,065 729.5% # Pregnant women newly initiated on ART 0 227 0 294 663 212 32.0% # Newly initiated on ART in similar previous reporting period (includes transfers-in with known initiation dates) 0 1,168 0 146 1,596 1,088 68.2% 100 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION PEPFAR Indicators Summary for Last Three Years of HCSP Implemented by IntraHealth International Indicators TGT10 (targets) APR10 (results) Percentage of Achievement Y3 TGT11 (targets) APR11 (results) Percentage of Achievement Y4 TGT12 (targets) APR12 (results) Percentage of Achievement Y5 T1.2.D.# Adults and children with advanced HIV infection receiving ART currently 6,105 5,796 94.9% 7,569 6,485 85.7% 7,984 7,206 90.3% T1.3.D.# Adults and children known to be alive and on treatment 12 months after initiation of ART 993 1,073 108.1% 1,426 1,073 75.2% 1,435 925 64.5% T1.4.D.# Adults and children with advanced HIV infection who ever started on ART 8,327 8,869 106.5% 10,626 9,917 93.3% 11,057 11,051 99.9% # Trained 280 261 93.2% 404 63 15.6% 50 86 172.0% Laboratory Services # Service outlets 45 42 93.3% 45 45 100.0% 45 45 100.0% H1.1.D.# Testing facilities (laboratories) with capacity to perform clinical laboratory tests 45 42 93.3% 45 45 100.0% 45 45 100.0% H1.2.D.# Testing facilities (laboratories) that are accredited according to national or international standards 0 0 0 0 0 0 # Trained 45 20 44.4% 45 84 186.7% 45 85 188.9% Human Resources for Health # Service outlets 45 0 0.0% 45 45 100.0% 45 44. 97.8% H2.1.D.# New health care workers who graduated from a pre-service training institution 0 0 0 0 0 0 H2.2.D.# Community health and para-social workers who successfully completed a pre￾service training program 0 0 0 0 0 0 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION 101 PEPFAR Indicators Summary for Last Three Years of HCSP Implemented by IntraHealth International Indicators TGT10 (targets) APR10 (results) Percentage of Achievement Y3 TGT11 (targets) APR11 (results) Percentage of Achievement Y4 TGT12 (targets) APR12 (results) Percentage of Achievement Y5 H2.3.D.# Health care workers who successfully completed an in￾service training program 280 822 293.6% 280 1,017 363.2% 90 4,741 5267.8% Sexual and Gender Based Violence (SGBV) # Service outlets 0 45 0 45 P12.1.D. # People reached by an individual, small-group, or community intervention or service that explicitly addresses norms about masculinity related to HIV/AIDS. 0 0 0 0 P12.2.D. # People reached by an individual, small-group, or community intervention or service that explicitly addresses gender-based violence and coercion related to HIV/AIDS. 0 0 15 0 0.0% P12.4.D. # People reached by an individual, small-group, or community intervention or service that explicitly aims to increase access to income and productive resources for women and girls impacted by HIV/AIDS 0 0 # New SGBV cases who received care and support through USG￾supported programs 0 908 450 1,107 246.0% # Individuals trained in SGBV service provision and clinical management 0 441 3,669 2,101 57.3% 102 USAID/RWANDA: HIV/AIDS CLINICAL SERVICES PROJECT (HCSP) END OF PROJECT EVALUATION For more information, please visit http://www.ghtechproject.com/resources GH Tech Bridge II Project 1725 Eye Street NW, Suite 300 Washington, DC 20006 Phone: (202) 349-3900 Fax: (202) 349-3915 www.ghtechproject.com