EVALUATION PERFORMANCE EVALUATION OF THE CLINICAL HIV/AIDS SYSTEM STRENGTHENING PROJECT IN NIASSA PROVINCE December 30, 2015 This report was produced by AGEMA Consultoria Lda, for the United States Agency for International Development/Mozambique. Authors: Peter S. Wandiembe, PhD, Rosemary Barber-Madden, PhD, Esther Kazilimani-Pale, MPH, and Verona Parkinson, PhD PERFORMANCE EVALUATION OF CLINICAL HIV/AIDS SYSTEM STRENGTHENING PROJECT IN NIASSA PROVINCE December 30, 2015 Agema Consultoria Lda. submits this report to USAID/Mozambique as a deliverable under Contract No. 656-O-00-15-00049-00 for the Performance Evaluation of the Clinical HIV/AIDS System Strengthening Project in Niassa, FHI 360 with sub-award to Abt Associates Inc. DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Performance Evaluation of the CHASS Niassa Project i CONTENTS ACRONYMS AND ABBREVIATIONS .............................................................................................. iii EXECUTIVE SUMMARY ..................................................................................................................... v I. EVALUATION PURPOSE AND EVALUATION QUESTIONS .............................................. 1 I.1 EVALUATION PURPOSE .............................................................................................................................................. 1 I.2 EVALUATION QUESTIONS ........................................................................................................................................ 1 II. PROJECT BACKGROUND .......................................................................................................... 3 II.1 GENERAL CONTEXT .................................................................................................................................................. 3 II.2 OVERVIEW OF CHASS NIASSA IMPLEMENTATION ........................................................................................ 4 III. EVALUATION METHODS AND LIMITATIONS .................................................................... 5 III.1 EVALUATION METHODS ......................................................................................................................................... 5 III.2 SITE SELECTION AND DATA COLLECTION .................................................................................................... 5 III.3 DATA ANALYSIS .......................................................................................................................................................... 6 III.4 STUDY LIMITATIONS ................................................................................................................................................. 6 IV. FINDINGS, CONCLUSIONS AND RECOMMENDATIONS ................................................. 7 IV.1 FINDINGS ....................................................................................................................................................................... 7 IV.2 OVERALL CONCLUSIONS ..................................................................................................................................... 18 IV.3 FUTURE DIRECTIONS ............................................................................................................................................. 18 IV.4 SUMMARY OF RECOMMENDATIONS ............................................................................................................... 20 REFERENCES ..................................................................................................................................... 23 ANNEX I. EVALUATION SCOPE OF WORK ............................................................................... 25 ANNEX II. DATA COLLECTION INSTRUMENTS: KII, FGD ..................................................... 36 ANNEX III. LIST OF INTERVIEWEES ............................................................................................ 48 ANNEX IV. SOURCES OF INFORMATION .................................................................................. 49 ANNEX V. SUMMARY OF QUANTITATIVE DATA .................................................................... 50 ANNEX VI. EVALUATION TEAM MEMBERS ............................................................................... 51 ii Performance Evaluation of the CHASS Niassa Project LIST OF TABLES, FIGURES AND CHARTS Table 1: Key indicators for HIV care, ART and PMTCT programs .............................................................................. vii Table 2: Categories of key informant interview respondents ......................................................................................... 6 Table 3: Recommendations for future programs .............................................................................................................. 20 Figure 1: Patients already on ART at the beginning of the year and those newly enrolled ...................................... 7 Figure 2: Indicators in the PMTCT Cascade ........................................................................................................................ 8 Figure 3: Retention and community linkage indicators .................................................................................................... 13 Chart 1: Weaknesses in Activities in Clinical Services and Recommendations .......................................................... 9 Chart 2: Weaknesses in Activities for HSS and Recommendations ............................................................................. 10 Chart 3: Weaknesses in Activities for HSS and Recommendations (2) ...................................................................... 11 Chart 4: Challenges to Community linkages ...................................................................................................................... 13 Chart 5: Constraints to activities aimed at improving retention and recommendations ....................................... 14 Chart 6: Challenges associated with improvement of health facility management ................................................... 15 Chart 7: Challenges to HF data management capacities ................................................................................................. 16 Chart 8: Challenges and weaknesses in scaling-up GBV services ................................................................................. 17 Performance Evaluation of the CHASS Niassa Project iii ACRONYMS AND ABBREVIATIONS Acronym or Abbreviation Definition (English) Definition (Portuguese) AIDS Acquired Immune Deficiency Syndrome Síndrome de Imunodeficiência Adquirida ANC Antenatal care Cuidado pré natal APEs Agentes Polivantes Elementer ART Antiretroviral treatment Tratamento Anti-retroviral ARV Antiretroviral Anti-retroviral CBO Community-based organization Organização Baseada na Comunidade CCM Community case managers CD4 Cluster of Differentiation 4 Cluster de Diferenciação 4 CHASS Clinical HIV/AIDS System Strengthening Project DPS Provincial Health Directorate Direcção Provincial de Saúde EPTS Electronic patient tracking system Sistema Electrónico de Seguimento de Pacientes FILA ARV drug pick-up form Folha de Informação de Levantamentos de ARV FOGELA Strengthening Lab Management for Accreditation Fortalecimento da Gestão de Laboratório para Acreditação FY Fiscal year GAACs Community Support and Adherence Groups Grupo de Apoio e Adesão da Comunidade GBV Gender-based violence Violência Baseada no Género HCT HIV counseling and testing Aconselhamento e Testagem em Saúde HF Health facility Unidade de saúde HIV Human Immunodeficiency Virus Vírus da Imunodeficiência Humana HR Human resources Recursos Humanos HSS Health system strengthening M2M Mother-to-Mother Groups Mãe para Mãe MCH Maternal and child health Saúde Materno-Infantil MOH Ministry of Health Ministério da Saúde PCC Community Care Project Projecto de Cuidados Comunitários PCR Polymerase Chain Reaction Reacção em Cadeia da Polimerase PEPFAR U.S. President’s Emergency Plan for AIDS Relief PIMA CD4-analyzing machine Máquina para Analisar as Amostras de CD4 PLHIV People living with HIV Pessoas que Vivem com o HIV PMTCT Prevention of mother-to-child transmission Prevenção da Transmissão Vertical QA Quality assurance Controlo da qualidade QI Quality improvement Avaliação da Qualidade de Dados RH Reproductive health iv Performance Evaluation of the CHASS Niassa Project Acronym or Abbreviation Definition (English) Definition (Portuguese) SA Sub-agreement SDSMAS District Health Directorate Serviços Distritais de Saúde, Mulher e Acção Social SIFO Continuing Training Information System Sistema de Informação de Formação Contínua TB Tuberculosis Tuberculose USAID United States Agency for International Development Performance Evaluation of the CHASS Niassa Project v EXECUTIVE SUMMARY EVALUATION PURPOSE The Clinical HIV/AIDS System Strengthening Project in Niassa province (CHASS Niassa) was a five-year project funded by the United States Agency for International Development (USAID). It was implemented by FHI 360 with an overall goal of strengthening the Niassa provincial health system by maximizing access, quality and sustainability in the delivery of comprehensive HIV/AIDS and related primary health services. The purpose of this performance evaluation was to determine how the project activities were performing relative to their objectives. The evaluation was to provide an objective view of progress toward the expected results. The main objectives were to (a) assess CHASS Niassa achievements, emphasizing project-level results; (b) identify implementation successes, as well as any internal and external constraints that hindered the implementation of planned activities; and (c) propose recommendations for future directions of the CHASS project. KEY EVALUATION QUESTIONS The evaluation was guided by 10 questions, 1 with the following four key questions: 1) What are the strengths and weaknesses of the activities as seen by the stakeholders, and how can these weaknesses be improved? 2) What constraints have the activities faced in improving retention of patients in pre-ART and ART programs? 3) To what extent have knowledge (community and health worker) and utilization of gender-based violence (GBV) services increased over the life of the project? 4) What are the benefits and challenges of the activities’ model of working with government (at provincial and district levels) through sub-agreements? PROJECT BACKGROUND The CHASS Program was designed to address the HIV situation in Mozambique and focused on three components: (a) Improving service quality in HIV prevention, care and treatment services; (b) Enhancing program linkages and integration to provide a continuum of accessible services; and (c) Creating stronger and more sustainable systems and institutions. In Niassa, CHASS was implemented by FHI360 from August 2010 to July 2015. EVALUATION DESIGN AND METHODS The performance evaluation of CHASS Niassa was conducted September 22-30, 2015, with the aim of achieving the aforementioned objectives. The evaluation was based on a non-experimental design using a mixed-methods approach. In addition to reviewing the existing quantitative project data and documents, the evaluation team conducted field key informant interviews with eight project staff, staff from five health facilities, 24 staff from health and social welfare departments at the province and the three districts, three staff from the Ministry of Health (MOH), five staff from USAID/Mozambique, and five staff from community-based organizations (CBOs) supporting CHASS Niassa. The evaluation used multi￾channel data collection tools, using mobile technology when possible and paper-based methods when handheld computers were not appropriate. All of the interviews were recorded on tablets. Some of the 1The 10 questions are listed in Section 1 of this report: Evaluation Purpose and Evaluation Questions. vi Performance Evaluation of the CHASS Niassa Project members handwrote their field notes directly onto the tablets using a stylus beta enhanced Open Data Kit (ODK) program. The qualitative data were analyzed using a qualitative data analysis matrix. Information from this analysis was triangulated with the available quantitative data from CHASS project documents. EVALUATION LIMITATIONS The limitations of the evaluation are as follows: Firstly, in many cases, the evaluation team was unable to ascertain consistent factors responsible for the project’s success and weaknesses as the project evolved over time. Therefore, some of the findings reported are from a historical perspective. Secondly, the findings reported here are responses provided by the interviewees to the evaluation team. The likelihood of interviewee bias and recall bias cannot be ruled out. FINDINGS, CONCLUSIONS AND RECOMMENDATIONS The findings for each evaluation question are summarized below: 1) Project achievements and challenges:  Activities in HSS: The project focused directly on delivering HIV/AIDS and related clinical services at the health facility (HF) level. The project made efforts to train the provincial directorate of health (DPS) and district directorate of health (SDSMAS) staff in program management, supply and logistics, and financial management. In addition, laboratory services were improved through renovations, provision of logistics for transportation of medications, supplies, and CD4 and PCR samples and results. The project contributed significantly to the human resource (HR) system and capacities through training and providing logistics for implementation of HR systems such as the Continuing Training Information System (SIFO) at DPS and SDSMAS, providing on-the-job training, training nurses to prescribe antiretrovirals (ARVs), and funding pre-service training of about 60 health workers. Six health workers received Master’s degree-level training in management. In addition, procurement staff were recruited for all SDSMAS. It was clear that the project lacked a well-planned and appropriate exit strategy. Most of the project’s operational costs were managed by the project team, and hence limited experiential learning by DPS and SDSMAS. The quality assurance/improvement (QA/QI) approach that helped DPS to identify service-delivery weaknesses and develop, test and implement improvement strategies was not regularly done, and some key informants from the DPS were not well versed with the concepts of the approach. Overall, the project staff focused more on helping the DPS and SDSMAS staff to implement activities and did little to build their capacities.  Activities in clinical services delivery: The HIV/AIDS clinical services were improved and decentralized or expanded to new HFs through staff trainings and mentoring, minor facility renovations, and demand creation and active referrals from the communities through activists, in coordination with CBOs. The project expanded the number of antiretroviral treatment (ART) sites and scaled up Option B+ in prevention of mother-to-child transmission (PMTCT). A summary of the indicators, showing trends in clinical services, coverage and quality, is presented in Table 1 below. Most of the project’s set targets were achieved by fiscal year (FY) 2014. The improvements made by the project are threatened by scarce human resources and high rates of staff transfers. Stakeholders at provincial, district and HF levels reported an inadequate number of staff to manage improved activities, meet increased demands and maintain high￾quality services. The logistics and supply chain was not working well, and many health facilities reported periodic stock-outs of drugs and HIV test kits. Children reached and quality of services were below the project targets. Performance Evaluation of the CHASS Niassa Project vii Table 1: Key indicators for HIV care, ART and PMTCT programs Indicator FY 2011 FY 2012 FY 2013 FY 2014 % of 2014 Target achieved† Number currently enrolled on ART 3,316 (2,699) 7,383 (7,622) 10,768 (7,189) 13,334 (11,012) 121% Number newly enrolled on ART 861 (1,615) 2,812 (5,103) 3,967 (2,518) 5,261 (2,919) 180% Percent of currently enrolled on ART patients that are men 27% 36% 30% 26% Percent of currently enrolled on ART patients that are children 7% 6% 11% 9% Retention rate in 12-month ART cohort (adult) 71% 75% 63% 70% 82% Percent of pregnant women with known HIV status (newly tested and known positive at ANC entry) * 92% (90%) 83% (90%) 90% (90%) 100% Percent of HIV-positive pregnant women in ANC who have initiated Cotrimoxizole* 52% 33% 59% Percent of HIV-positive pregnant women who received ARVs to reduce risk of MTCT* 72% 56% 82% Percent of infants born to HIV-positive women who received an HIV test within 12 months of birth* 26% 24% 53% Percent of HIV test results for infants born to HIV￾positive women who received an HIV test within 12 months of birth that are positive* 15% 7% 4% †Indicators without targets have this column left blank. *No data were available for most of the PMTCT-related indicators in FY 2011 and are left blank in the table. 2) Most-improved HSS component: The project supported the strengthening of systems and capacities of the various components of the health system, with the HR component identified by key informants from the DPS and SDSMAS as the most significantly improved. The DPS and SDSMAS staff were trained on HR management and use of electronic HR information systems, and health workers were trained through mentoring and tutoring, on-the-job training and pre-service training. 3) Ready-to-transition project activity or component: The project worked jointly with SDSMAS and DPS in executing most of its activities. Whereas most of the routine project activities were led by the project team, they were part of MOH activities that were supposed to be implemented by provincial and district technical teams. The key project-specific activity that should be transitioned to the DPS and SDSMAS team leadership is the QI/QA methodology that was effective in improving clinical practices and processes. 4) Strengthening community linkages: The project had exemplary community linkage strategies for clinical and GBV services, including developing the graduation path for HFs. Through three CBOs, the project provided logistics to activists or community case managers (CCMs) for case￾finding of pre-ART care and ART defaulters, community sensitization, mobilization and community￾based HIV testing. The facility-based case managers hired by CBOs were instrumental in generating lists of defaulters, receiving referrals from the communities and communicating with health workers. Through provision of training and logistics to the adherence support groups (GAACs), people living with HIV (PLHIV) and mother-to-mother (M2M) activities, the project improved knowledge of services and managed to maintain retention in ART programs by at least 70 percent. 5) Constraints to improving retention rates: Retention rates persisted at below 85 percent of target across the province. One of the key constraints was the insufficient number of staff providing viii Performance Evaluation of the CHASS Niassa Project adequate counseling at some of the HFs and communities. Retention is a cycle that begins with counseling, and if counselors do not take time to counsel patients and answer their questions so that they understand that HIV/AIDS needs regular clinical reviews or adherence to ARVs, retention becomes a problem. Other constraints that limited the achievements of retention-improvement strategies (see point 6 below) were the long distances traveled by CCMs, stigma in the community, wrong addresses given by patients, community movements during the farming season, and time constraints, especially for working men. 6) Most effective method for improving retention: The key strategies aimed at improving retention included (1) active case-finding, where CCMs use a defaulters’ list generated by the HF case managers to search for defaulters in communities; (2) GAAC members picking up drugs for each other and also providing psychosocial support; (3) community mobilization by CCMs, encouraging retention and adherence to ART; (4) M2M following HIV-positive pregnant women; and (5) psychosocial support from PLHIV groups. Although it was reported to be costly and did not cover all communities, active case-finding was the most effective strategy. The activities of the M2M groups were moderately effective, followed by the activities of the PLHIV and GAACs. Community mobilization by activists and lay counselors also played a supportive role. 7) Improvement in HF management: Use of one-stop PMTCT and TB models, training of HF staff in process organization, and reinstallation of the fluxogram (flowchart) card system led to improved patient flow and reduced waiting times. The HFs’ restricted physical spaces limited the improvement of patient flow. 8) Improvement in HF data management capacity: The error rates of filling in the HIV/AIDS registry decreased substantially, and correct filing and summarizing of the registry records for data entry improved moderately. However, these improvements are still below the desired levels, especially the pediatric care registries. 9) Knowledge of GBV (among health workers and community) and service utilization: CHASS provided training for focal points at provincial and district levels and trained activists who could refer cases to HFs. Community sensitization was done by activists and supported by other organizations, such as MULEIDE. The CCM did an active search for the GBV victims in the community. Overall, there was an increase in cases reported at the HF from the community, police and the judiciary. However, the GBV services are not well integrated with other clinical services, except maternal and child health (MCH). In addition, sociocultural reasons were cited to have limited victims’ reporting to HFs or police. 10) Benefits and challenges of working with government through sub-agreements (SAs): Project activities were implemented through SAs, which enabled the development of systems and capacities of the SDSMAS and DPS through “forced” learning, and to some extent fostered mutual understanding of the activities and wider ownership. However, the SAs were based on the project activities, with no allowance for engaging in any other activities outside that scope. Further, there were always delays in reimbursement throughout the five years of the project. The capacity of the DPS and SDSMAS to manage direct funding needs further strengthening. CONCLUSION In general, the project achieved most of its targets, but more efforts are needed to improve these further and sustain them. In particular, more efforts and resources are required in health system strengthening (HSS). There was limited direct support to HSS at the DPS and SDSMAS, and limited operations research was done to understand reasons for low retention rates, poor patient flow– especially in rural HFs, and low recruitment of children on treatment. Performance Evaluation of the CHASS Niassa Project ix RECOMMENDATIONS Health System Strengthening 1) Involve the DPS and SDSMAS in the project design, and give them an active role at all stages of the project. The transition plan should be agreed on with the SDSMAS, DPS and MOH and should be explicitly incorporated in the project documents through the first year. Face-to-face interactions should be a preferred method of communication with these entities. The MOH should actively play its role as agreed with the project to ensure its deliverables as well as success and sustainability. 2) Introduce a component of operations research in project activities and also in local capacity building. 3) Develop tools and methodologies for regular assessment of the project’s support for HSS. The graduation path assessment that uses a 23-item tool to assess systems and capacities of various components against standards should be used to measure progress in HSS.2 The action plans following the assessment will benefit from the use of the QI approach. 4) Put a training mechanism in place. This could include a peer-to-peer model, in which trained health workers can do tutoring at the nearby HFs; work with training institutions to establish courses in HSS for all health workers; and training a group of national and district-level trainers. 5) Expand mHealth in project areas to more sites. Most aspects of data management and patient follow-up can be performed with mobile technology. In the future, the project should expand on its mHealth activities. This will include the use of tablets, PDAs or mobile phones by CCMs or activists to input client information; text messaging reminders for patient clinic visits, including mothers returning babies for HIV testing and adherence to ARVs; and working with a mobile phone company to provide online tutorials for health workers on the tablets. 6) The supply chain and logistics system is still weak. A private company should be hired to manage it until the DPS and SDSMAS have developed the necessary capacity. 7) Advocate for and support the MOH in rolling out the electronic patient tracking system (EPTS) to moderate- to large-volume HFs. In the meantime, encourage and train health workers to correctly fill in ARV drug pick-up forms (FILAs) and other forms on retention and file them appropriately, and initiate conversations with DPS about the roll-out plan for EPTS once the MOH approves it, including but not limited to data validation processes and tools, plan and terms of reference for data managers, use of data from EPTS for reporting, and the QI program. Clinical Services (enrollment and retention) 8) The GRM/MOH should design a better staff development and retention package or plan to promote staff retention in post for at least three years to allow for consolidation of experience. It should also increase staffing levels to cope with increased client load, especially for HIV/AIDS services. 9) The GRM/MOH, with support from partners, should invest in improving HFs’ physical structure and space. 10) Implement an augmented MOH strategy of a lay counselors’ workforce by recruiting and training lay counselors or retraining case managers and Agentes Polivantes Elementer (APEs) as lay counselors. 2The graduation path assessment tool was developed by Abt Associates Inc., one of the implementing partners of CHASS￾Niassa. x Performance Evaluation of the CHASS Niassa Project Some counselors should be stationed at the HF and one in each community to provide services such as adequate counseling for ART initiation, community sensitization against HIV stigma, psychosocial support to HIV-positive pregnant women to disclose HIV status, psychosocial support to GAACs and individual PLHIV, and adherence counseling. 11) Reinvigorate linkages between HF and community-based services, and promote regular information exchange for cross-referrals and health information messaging to communities:  Conduct operations research to map and improve the functioning of community linkages.  Explore adding new activities to CBOs, including creating links with faith-based organizations, traditional healers, traditional birth attendants, APEs and others.  Expand the mobile ART outreach into a mobile clinic with mHealth to provide HIV counseling and testing (HCT), HIV care, ARVs, antenatal care (ANC) and PMTCT, CD4 testing, TB screening services and health education.  Maintain the family approach model and community drama to reach couples and children. 12) Expand the HIV prevention, care and support services for adolescents at HFs, with emphasis on adolescent girls. Develop a school-based integrated health program that includes adolescent health and HCT services. 13) Improve HIV testing and ART initiation among infants through training all HF health workers in: PCR sample-taking to provide HIV testing in all pediatric entry points, including the children of patients on ART; linkage of records in labor/delivery units and at-risk child consultations; and adequate counseling of pregnant women. In addition, distribute SMS printers to rural health facilities. Community Linkages 14) Expand the current activities of the project (see Recommendation 8). 15) Partner with employer-based health programs to provide training and technical support to institute worksite HCT, referral, and dispensing ARVs, particularly for male workers. GBV Services 16) Expand the current GBV community linkages and collaborations to other communities. 17) Develop the capacity of all health workers so that GBV screening is done at all HF points of care. Sub-agreements 18) Establish a budget item for non-HIV-related services: This can be used to fund emergencies or disease outbreaks that are not necessarily within the scope of CHASS. 19) Improve on the reimbursement process: Review internal financial controls to make them realistic for GRM collaborators Performance Evaluation of the CHASS Niassa Project 1 I. EVALUATION PURPOSE AND EVALUATION QUESTIONS I.1 EVALUATION PURPOSE The five-year Clinical HIV/AIDS System Strengthening Project in Niassa province (CHASS Niassa) was funded by the United States Agency for International Development (USAID). It was implemented by FHI 360 from August 2010 to July 2015. The project’s goal was to strengthen the Niassa provincial health system by maximizing access, quality and sustainability in the delivery of comprehensive HIV/AIDS and related primary health services. This performance evaluation of CHASS Niassa was commissioned by USAID, and its purpose was to determine how the project’s activities were performing relative to their objectives. The evaluation was to provide an objective view of progress toward the expected results. The outcomes of the evaluation will inform the transition to the future activity that will support USAID system strengthening and clinical service delivery activities. The evaluation’s main audience is USAID. In addition, results will be shared with FHI 360, the Ministry of Health (MOH), U.S. Government agencies, and other stakeholders. The results will also be made available on the Development Experience Clearinghouse. The objectives of the evaluation are to:  Assess CHASS Niassa achievements, emphasizing objectives and activity- and project-level results  Identify implementation successes, as well as any internal and external constraints that hindered the implementation of planned activities  Propose recommendations for future directions of CHASS Niassa and for future activities in system strengthening and service delivery to support improved performance in addressing the HIV epidemic, in line with the GRM HIV/AIDS acceleration plan An additional goal was to use mobile technology, when possible and appropriate, to increase the efficiency, transparency and accuracy of performance data, and to take advantage of multiple data sources (pictures, videos, GPS data). I.2 EVALUATION QUESTIONS The three main evaluation question areas include a total of 10 evaluation questions. All areas were answered fully and completely, underscoring both positive and negative outcomes. Where sufficient data were available, gender analysis was done with quantitative data. In order to accomplish the above￾identified evaluation objectives, the evaluation sought to answer the following 10 questions: Question Area 1: Project Achievements and Challenges 1) What are the strengths and weaknesses of the activities, as seen by the implementing partner staff, Provincial Directorate of Health (DPS), District Directorate of Health and Social Welfare (SDSMAS) and chief medical officer, HFs, USAID, and the USAID-funded Community Care Project (PCC), and how can weaknesses be improved according to the stakeholders listed here? 2) Where has the most progress been seen in strengthening systems (e.g., planning, financial management, supply and logistics, information systems)? 3) Which activities or project components will be most feasible to transition from the project to the GRM? 2 Performance Evaluation of the CHASS Niassa Project Question Area 2: Linkages 4) To what extent has the project been able to create and strengthen linkages between health facilities and communities to allow for increased service uptake, specifically in the areas of: a. Community-based counseling and testing to treatment (for both men and women); b. Retention of pre-ART and ART patients (through the use of adherence groups, active case-finding and other community groups); and c. Knowledge, demand and access of services by men 5) What constraints have the activities faced in improving retention of patients in pre-ART and ART? 6) What has been the most effective method found by the project to improve retention? Question Area 3: Health System Strengthening 7) To what extent has HF management (improved patient flow, etc.) improved over the life of the project? 8) To what extent is data management capacity built at the HFs with regard to HIV/AIDS registry data? 9) To what extent has knowledge (community and health worker) and utilization of GBV services increased over the life of the project? 10) What are the benefits and challenges of the activities’ model of working with government (at provincial and district levels) through sub-agreements? Specific questions that guided the performance evaluation are stated in the evaluation matrix included in Appendix II. Performance Evaluation of the CHASS Niassa Project 3 II. PROJECT BACKGROUND II.1 GENERAL CONTEXT In July 2009, USAID/Mozambique issued a request for applications for a results-oriented five-year project to improve HIV clinical services in Manica, Niassa, Sofala and Tete provinces within a strengthened, comprehensive primary health care system. This project was designed to address the HIV situation in Mozambique and focused on three components: 1) Improving service quality in six important areas: HIV counseling and testing (HCT), laboratory services, prevention of mother-to-child transmission (PMTCT), adult care and treatment, pediatric care and treatment, and the prevention, diagnosis and treatment of HIV/tuberculosis (TB) co-infection 2) Enhancing program linkages and integration to provide a continuum of accessible services, including MCH and reproductive health (RH) services, within facilities and between facility and community-based services 3) Creating stronger and more sustainable Mozambican systems and institutions At the time of project design in 2010, the average HIV prevalence in Mozambique was estimated at 16 percent nationwide (ANC survey in pregnant women aged 15-49 years). Nearly 1.6 million people were living with HIV, and nearly half of all HIV-infected (48.4 percent) were identified at the time as having active TB. Within the Central Region, Sofala and Manica provinces have the highest prevalence, 23 percent and 16 percent respectively. Tete was identified as having a very mature epidemic and existing infrastructure that was unable to accommodate the numbers of patients requiring care and treatment. Niassa was considered a particularly underserved province, with vastly inadequate infrastructure. To address these issues, two separate agreements were awarded for the “Clinical HIV/AIDS System Strengthening (CHASS) Project (Sofala, Manica, Tete, Niassa)” to two implementing partners: FHI 360 for CHASS Niassa, and Abt Associates Inc. for CHASS-SMT (Sofala, Manica and Tete). CHASS Niassa was funded at $36,538,233 for the period of August 1, 2010 to July 31, 2015. The implementation of CHASS Niassa started in late 2010 and was to support USAID/Mozambique’s Country Assistance Strategy’s (CDCS DO4) priority goal number three, “Improved health of Mozambicans,” and the following focal areas in USAID’s Health Results Framework: (a) Improved access to and delivery of quality integrated services; (b) Increased adoption of healthy behaviors and informed use of services; and (c) Strengthened health systems. After these two activities were awarded, the MOH developed the HIV Acceleration Plan in 2011; signed the Political Declaration on HIV/AIDS: Intensifying the Efforts to Eliminate HIV/AIDS; and created the “Mozambique HIV and AIDS Response–Strategic Acceleration Plan 2013-2015” to respond to the commitment to an end of AIDS. The latter three-year strategy reflects the united vision of all stakeholders to achieve a Generation Free of AIDS in Mozambique, focusing on three major goals, and one added objective on gender-based violence:  Increase the percentage of eligible HIV-infected adults and children receiving ART to 80 percent by 2015.  Reduce the rate of transmission of HIV from mother to child to less than 5 percent by 2015.  Reduce the number of new infections by 50 percent by 2015.  In addition, both CHASS activities (Niassa and SMT) included programming to address GBV within the HIV platform, with a total life-of-project funding of approximately $1.5 million. GBV fosters the spread of HIV by limiting a person’s ability to negotiate safe sexual practices, disclose 4 Performance Evaluation of the CHASS Niassa Project HIV status, access services (due to fear of reprisal), adhere to treatment, and access care. Activities were implemented in all four provinces to: (a) Expand and improve coordination and effectiveness of GBV prevention efforts; (b) Improve policy implementation in response to GBV; and (c) Improve the availability and quality of GBV services. The project’s results framework is included in Appendix VII. II.2 OVERVIEW OF CHASS NIASSA IMPLEMENTATION CHASS Niassa’s goal was to strengthen the Niassa provincial health system by maximizing access, quality and sustainability in the delivery of comprehensive HIV/AIDS and related primary health services. The activity’s objectives were to: 1. Increase access, quality and utilization of HCT, laboratory services, PMTCT, adult care and treatment, pediatric care and treatment, and HIV/TB co-infection services. 2. Provide a continuum of accessible HIV and related primary health care services, including MCH and RH services, and to improve linkages and referrals within and between facilities and communities. 3. Support stronger and more sustainable Mozambican systems and institutions through emphasis on strengthening government and community capacity to deliver and manage services. The project has supported implementation and scaling up of HIV services by the DPS/MOH through training, mentorship, structural refurbishments, strengthening of monitoring and evaluation systems, and provision of equipment and medical supplies. Through sub-agreements, the project worked jointly with the DPS and the SDSMAS on activities that led to improved health system and expansion of high-quality HIV/AIDS services. The bulk of the funds identified in the SAs were expended directly by CHASS team on behalf of the DPS and SDSMAS. The project introduced a QI collaborative methodology that was reported by the DPS and SDSMAS to be very effective in systematically identifying weaknesses in clinical and auxiliary service practices, processes and environment at the HFs, and in developing elaborate action plans. In addition, through the project’s peer education program, which included community-based peer case management and mobilization, community knowledge and utilization of clinical services and GBV services increased over time. The project introduced a graduation path for large facilities. Following a QI/QA process, a HF with sustained standard clinical and auxiliary service practices and processes and that delivers high-quality care according to MOH standards was supposed to be graduated from the project’s support to run independently. The project worked with three CBOs (Associação Renascer a Vida, Conselho Cristão de Moçambique and Concelho Islamico de Moçambique) through SAs to strengthen linkages between the HFs and the communities, but also in coordination with other projects such as PCC and TB Care. Performance Evaluation of the CHASS Niassa Project 5 III. EVALUATION METHODS AND LIMITATIONS III.1 EVALUATION METHODS This performance evaluation was based on a non-experimental design and was executed by a team independent and external to the USAID/Mozambique CHASS Project. The evaluation used a mixed￾methods approach, utilizing mostly qualitative data collection and evaluation methods. The evaluation involved extensive desk review and analysis of existing quantitative project data and documentation, and primary collection and analysis of qualitative data. Quantitative data were extracted from the quarterly progress reports and project datasets held by the USAID/Mozambique team. The evaluation was conducted September 22-30, 2015. Primary data collection methods included key informant interviews and observational analysis at HFs. The key informant interviews served to validate and, where possible, verify project approaches or activities, interventions, achievements, extent of gains and changes over time. They also identified gaps and weaknesses in project activities or performance. The key informants included staff from the MOH, DPS, SDSMAS, HFs, the CHASS implementing partner and CBOs that participated in the project. Some members of the USAID/Mozambique team were interviewed. The structured key informant interview guides are included in Appendix III. They were pre￾tested in Beira city before fieldwork began. Interviews were not held with clients or patients, as this required ethical approval that would have substantially shifted the timeframe for the evaluation. The evaluation used multi-channel data collection: mobile technology when possible and paper-based methods when handheld computers/tablets were not appropriate. The team noted that the use of mobile technology is feasible, and some of the members handwrote their field notes using the ODK program, which was enhanced with a stylus beta program. However, the team notes that for the mobile technology to be most useful for collection of qualitative data, the key informant and focus group guides should be semi-structured. Further, time is required for extensive practice of handwriting on the tablets. This was not possible in this evaluation due to time constraints. III.2 SITE SELECTION AND DATA COLLECTION Three districts were selected at random to represent dominantly rural, semi-urban and urban districts. These included Lichinga, Chimbunila and Lago districts. Within each district, the team selected HFs purposively to represent low and high client volumes, extent of CHASS activities at the site, and rural vs. urban locations. Five HFs were visited and various categories of health workers interviewed (Table 2). The DPS/SDSMAS senior staff included the directors and chief medical officers while the HF management team included the health facility in-charges. A total of eight CHASS Niassa staff, eight PCC staff, and five staff from three CBOs that worked closely with CHASS Niassa were interviewed. In addition, two staff from the MOH and five from USAID were interviewed. 6 Performance Evaluation of the CHASS Niassa Project Table 2: Categories of key informant interview respondents Group Number of male key informants Number of female key informants Total key informants DPS/SDSMAS senior staff 5 1 6 Health facility management team 2 1 3 HIV focal points/technicians (at DPS, SDSMAS and HF) 2 1 3 SMI focal points/technicians (at DPS, SDSMAS and HF) 0 4 4 HSS/Monitoring and evaluation focal points (at DPS and SDSMAS) 2 0 2 Logistics/finance (at DPS and SDSMAS) 1 1 2 GBV focal points (at DPS, SDSMAS and HF) 0 3 3 Other (lab, pharmacy) 0 1 1 Total 12 12 24 Percent target 100 III.3 DATA ANALYSIS The data analyzed included those in the performance monitoring system and program reports. Trends in results and progress made on planned results were analyzed. The qualitative data were analyzed using a qualitative data analysis matrix. Information from this analysis was integrated or triangulated with the available quantitative data from CHASS documents and reports. The analysis was guided by the 10 evaluation questions listed in the scope of work (Appendix I) and in Section 1 of this report. For the evaluation questions where gender-related data or information are relevant, gender-related differences are presented. The end summary of the analysis was focused on the priority issues for CHASS to address and the main lessons learned, based on the answers provided in examining the 10 evaluation questions. III.4 STUDY LIMITATIONS The limitations of this evaluation are as follows: Firstly, the performance evaluation was delayed; it was initially planned to be in the fourth year of the project but was conducted a year later. Secondly, the findings reported here are based on what the respondents reported to the evaluation team. The likelihood of interviewee bias and recall bias cannot be ruled out. Thirdly, the project was closing at the time of the evaluation, and to some extent, this constrained the sample size for the key informant interviews. Lastly, the evaluation was conducted when the project staff was transitioning, which delayed some of the activities of the evaluation team. Performance Evaluation of the CHASS Niassa Project 7 IV. FINDINGS, CONCLUSIONS AND RECOMMENDATIONS IV.1 FINDINGS Question 1: What are the strengths and weaknesses of the activities, as seen by the interviewed stakeholders, and how can the weaknesses be improved? Findings on the strengths and weaknesses of the project activities are grouped according to project results I-III. The findings are as reported by the key informants and project reports. The achievements reported here should not be attributed solely to the CHASS strategies, considering the interventions of other actors in the province. Strengths and Weaknesses of Activities to Improve Clinical Service Delivery The project made considerable progress toward achievement of its set targets for the results. The HIV/AIDS clinical services were improved and decentralized or expanded to new HFs through staff training and mentoring, improvement of processes and practices through the QI/QA collaborative model, minor facility renovations, and creation of demand and active referrals from the communities to HFs through activists (peer case managers) and coordination with CBOs. The project trained nurses to prescribe ART and provided clinical, laboratory and pharmaceutical supplies, equipment, job aids, care and treatment protocols, and logistical support for the transportation of drugs, lab supplies and samples and results for CD4 and PCR. HIV care and treatment support: In addition to decentralization of HIV/AIDS care and treatment from 25 sites in FY 2011 to 46 sites in FY 2014, the project established mobile HIV care ART outreach teams to take ARVs to the outlying posts for pickup by the clients. The peer or community case managers actively reached out to their community members to test for HIV and enroll into pre-ART care and the ART program. Figure 1: Patients already on ART at the beginning of the year and those newly enrolled Numbers 2,455 4,571 6,801 8,073 861 2,812 3,967 5,261 27% 36% 30% 26% 7% 6% 11% 9% -22% -12% -2% 8% 18% 28% 38% 48% - 2,000 4,000 6,000 8,000 10,000 12,000 14,000 16,000 2011 2012 2013 2014 New on ART Already on ART 8 Performance Evaluation of the CHASS Niassa Project The number of individuals currently on ART increased from 3,318 in FY 2011 to 13,334 in FY 2014, achieving at least 120 percent of the set target in 2014. The percentage of clients currently on ART who are men declined from 36 percent in 2012 to 26 percent in 2014. This is partly due to scale-up of Option B+ among pregnant women. This is expected, as HIV-positive men are mainly reached through HCT programs, but HIV-positive women are reached through both HCT and PMTCT programs. The number of clients newly enrolled on ART increased five-fold since 2011 (Figure 1). In FY 2014, the project achieved more than 100 percent of its set target for clients newly enrolled on ART. PMTCT services and integration with MCH services: In addition to expanding PMTCT services to 65 sites by the end of FY 2014 and training nurses to prescribe ART, the project developed innovative strategies to get more pregnant women tested and put on Option B+. Through active mobilization by the CCMs, the number of women seen in ANC increased from 34,961 in 2012 to 53,693 in 2014 (Figure 2), above the project’s initial targets. Of these women, at least 82 percent were tested for HIV for the first time in ANC. The percentages initiated on ART in 2012 and 2013 were, however, below the set target of 80 percent. The involvement of M2M groups in following up with HIV-positive women also played a role in getting pregnant women onto ART. The one-stop shop model for PMTCT services was important in ensuring better patient flow and quality of services at the health facility. Figure 2: Indicators in the PMTCT Cascade No. of women who attended ANC Source: CHASS USAID progress report data Pediatric care and treatment: Effective strategies to improve enrolment into pediatric care and treatment included training of MCH nurses, the one-stop shop model, ensuring that HIV testing is part of the services at all points of care at the HF, reactivation of adolescent health services at some HFs, and community mobilization through the CCM and M2M. Nurses reported an increased number of youth accessing services. Further, the percentage of patients currently on ART who are children increased from 7 percent in 2011 to 9 percent in 2014. Despite this, there is still more work needed to improve enrolment into pediatric care and treatment services. Although the percentage of infants tested for HIV within 12 months increased from 26 percent in 2011 to 53 percent in 2014, this is still below target. TB/HIV co-infection: In most HFs, TB patients were referred to an ART clinic to initiate ART after completion of TB treatment. Similarly, the project made HCT a pre-requisite to access TB treatment. In large-volume facilities, a one-stop shop model for TB and HIV services was introduced, in collaboration with the TB Care project. 40,295 54,726 60,616 92% 83% 90% 72% 56% 82% 16% 18% 34% -22% -2% 18% 38% 58% 78% 98% 118% - 10,000 20,000 30,000 40,000 50,000 60,000 70,000 % pregnant women with known HIV status (newly tested + known positive at ANC entry) 2012 2013 2014 Numbers Performance Evaluation of the CHASS Niassa Project 9 HCT established at health facilities: The project has significantly improved access to HCT services, achieving more than 100 percent of its set targets. A total of 370,281 clients were tested for HIV since 2011, of whom about 50 percent were from provider-initiated counseling and testing, and 30 percent from community HCT. Community-based HCT has become an increasingly large source of HCT clients. Laboratory services: Through minor infrastructural renovations, training of staff at 18 micro and functioning labs, and provision of logistical support for the transportation of test kits, reagents, and test samples and results, the coverage of CD4, PCR and other relevant services have improved. For example, the number of samples subjected to CD4 counts using PIMAs increased from below 200 per month in 2011 to more than 600 in 2014. The project also supported registration of the Provincial Hospital Clinical Laboratory into the Management of Laboratory Accreditation Building Programme (FOGELA). The summary of other findings and recommendations are listed in Chart 1. Chart 1: Weaknesses in Activities in Clinical Services and Recommendations Weaknesses Recommendations  Adequate space for auditory and visual privacy is very limited in many HFs in Niassa, and structural modifications are critical for effective PMTCT interventions.  Storage facility for medication is lacking in some HFs.  The quality and quantity of laboratory supplies in place were not sufficient to meet the needs of the HIV/AIDS response.  In some HFs, standard laboratory equipment, especially the biochemistry and hematology analyzer, were not available or were not functioning at the time of visit.  There are frequent stock-outs, with a poor logistical system between district warehouses and the HFs.  There is an insufficient number of staff, and improved activities aimed at high quality of care have placed more demands on health workers.  Service coverage of children is limited.  The GRM, with support from partners, should invest in structural modifications of the physical space at HFs. Expansions should include storage space.  Improve on the logistics and supply chain by hiring a private company to run the system until the DPS has the capacity to run it independently. Put a system in place for reporting consumption rates for the various drugs, kits, reagents, e.g., through electronic tracking system with SMS facility to link the HF to staff at the warehouses.  Develop a kit básico (basic package of clinical and lab consumables and equipment) and store at the district warehouses and develop inter￾district route maps to ease regular supply of the kits.  The GRM/MOH and partners should urgently invest in buying necessary equipment for the different levels of HFs to ensure better quality of service.  Reinforce community-HF linkages and communication, and expand on the number and type of CBOs (e.g., actively involve faith-based organizations, traditional birth attendants, APEs, traditional healers, etc.).  Advocate with the MOH to increase the number of health workers and develop better transfer plans.  Expand the coverage of a mobile clinic model to include CD4 and PCR sample taking, HCT, PMTCT, and TB screening services.  Promote and extend school-based integrated health program, including adolescent health and HCT services.  Implement an augmented MOH strategy of a lay counselors’ workforce by recruiting and training lay counselors or retraining case managers and APEs as lay counselors to provide services.  Improve HIV testing and ART initiation among infants through training all HF health workers in: taking PCR samples to provide HIV testing in all pediatric entry points, including children of patients on ART; linkage of records in labor/delivery units and at-risk child consultations (improve pediatric care registries); adequate counseling of pregnant women about early ART initiation among infants and HIV status disclosure to partners, and counseling against HIV stigma; and DBS processing and storage. Also, distribute SMS printers to rural HFs. 10 Performance Evaluation of the CHASS Niassa Project Strengths and Weaknesses of HSS Activities The project focused directly on delivering HIV/AIDS and related clinical services at the HF level. The major activities implemented at the HFs were those identified in SAs with DPS and SDSMAS. The involvement of DPS and SDSMAS in planning and implementation of the project activities led to improved planning and management capacities of the DPS and SDSMAS staff. In addition, the project made efforts to train the DPS and SDSMAS staff in supply/logistics and financial management, and improved laboratory services through renovations and provision of logistics for transportation of medications, supplies, and CD4 and PCR samples and results. The project contributed significantly to the HR for health system and capacities through training the DPS and SDSMAS and providing logistics for implementation of HR systems such as SIFO, providing on-the-job training and funding pre-service training of about 60 health workers, providing Master’s degree-level training to six staff in management, and recruiting procurement staff for all SDSMAS. The introduction of the QA/QI approach helped DPS to identify service-delivery weaknesses and develop, test and implement improvement strategies. However, this was not regularly done, and few key informants from DPS understood the approach very well. Further, the project staff focused more on helping the DPS and SDSMAS staff to implement the activities and did little to build their capacities. Other challenges are listed in Chart 2. Chart 2: Weaknesses in Activities for HSS and Recommendations Challenges/Findings Recommendations  Limited efforts were aimed at HSS. The project team concentrated more on directly doing the work for the DPS and SDSMAS instead of mentoring them to execute the activities themselves.  There were no specific indicators to track improvements in HSS, except for labs.  There was poor communication between DPS and the project. o The planning did not work well: The project would change some activities without communicating to DPS/SDSMAS. o The project team sometimes effected changes in clinical procedures at the HF without informing the chief medical officers. In other words, there was a lack of respect for the management structure in place. o There was weak communication between the project’s field technical team and central finance team (e.g., TOs not being sure of when funds will be released).  Communication between DPS/SDSMAS and the project was delayed: The project did not return TSV reports and action plans to the DPS/SDSMAS in time.  Some line items approved in activity plans were not honored, and no justifications were given for these changes of plans. Health workers reported that technical assistance could have been better and needed to be more professional.  The large bulk of activity funds were expended directly by FHI 360 on behalf of DPS and SDSMAS. This potentially limited the improvements in financial management capacities.  Introduce a peer-to-peer mentorship program in which SDSMAS staff can move from one district to another to be mentored by experienced peers and receive training from an academic institution or project staff.  Develop tools and methodologies for regular assessment of project support to HSS. The graduation path assessment with indicators to measure progress in HSS should also be applied to HSS. The action plans that follow the assessment will benefit from the use of the QI approach.  Donors should develop a transition process with the government in which government can begin to plan from year 1 and through dialogue and planning can begin to make budgetary adjustments to assume management and financial responsibility for project advances.  The project should improve on the reimbursement process by having an adequate number of dedicated staff for reviewing reports and improving the internal financial controls.  The supply chain and logistics system is still weak. A private company should be hired to manage it until the DPS and SDSMAS have developed the desired capacity. Performance Evaluation of the CHASS Niassa Project 11 Challenges/Findings Recommendations  Few project staff available to handle SAs and HSS led to delays in reimbursements and limited activities to strengthen systems and capacities at the district level.  The project lacked a well-planned and appropriate exit strategy to promote adequate continuity. Strengths and Weaknesses of Activities to Improve Primary Health Care Program Integration and Community Linkages Details about the primary health care program integration are discussed in Evaluation Question 1, while details about the role of community linkages are given in Evaluation Questions 4 and 5. Question 2: Where has the most progress been seen in strengthening systems (e.g., planning, financial management, supply and logistics, information systems)? Considering the new skills acquired by the government staff and the increase in number of MCH and medical technicians, it is logical that the DPS and SDSMAS key informants identified HR management and capacity development as the most significantly improved component of the HSS. There were also improvements in the supply chain and logistics, including the setting up of procurement units. However, similar to improvements in financial management, improvement in information systems and monitoring and evaluation did not reach the desired levels. Stock-outs were also reported at many HFs. The project staff also reported poor data quality in some HFs supported by the project. The project provided new staff by training about 60 health workers and recruiting them through a gap￾funding mechanism, improved clinical and management skills and various competencies of the existing staff, and trained the MCH nurses to prescribe ARVs. A summary of additional findings is included in Chart 3. Chart 3: Weaknesses in Activities for HSS and Recommendations (2) Challenges/Findings Recommendations  Monitoring and evaluation systems are still weak.  The staff numbers are still insufficient, and the expanded activities have increased the workload, limiting the quality of some services.  Relatively high staff transfer/rotation rates implies that sustaining HR capacities at SDSMAS and HFs is limited.  There was no mechanism put in place for SDSMAS to train new staff (see Chart 5 in Evaluation Question 1).  The MOH could not hire all the new staff recruited under the project gap-funding system to stay at SDSMAS or facilities.  The GRM/MOH and partners should work together to: establish or strengthen the monitoring and evaluation systems, including better use of available strategic information, and health information systems; introduce use of electronic databases; and train staff.  The GRM/MOH should develop a better staff development and retention package or plan to ensure staff stay in post for at least three years to enable consolidation of experience. Increase staffing level to meet increasing client load, especially for HIV/AIDS services.  Expand mHealth in project areas: Tablets, PDAs or phones should be used by CCMs and activists to input client information. Text messaging reminders for patient clinic visits, including mothers returning babies for HIV testing and adherence to ARVs, is an effective strategy that has worked elsewhere. Working with a mobile phone company, the tablets can also be used for online tutorials for health workers.  Put in place a training mechanism (see Chart 2). 12 Performance Evaluation of the CHASS Niassa Project Question 3: Which activities or project components will be most feasible to transition from the project to the GRM? The project worked jointly with SDSMAS and DPS in executing most of its activities, which has improved the skills and capacities of the GRM staff. Most key routine activities, such as TSV to improve quality of clinical services and increasing demand through community linkages, are in the mandate of the DPS and SDSMAS senior staff. Most of these activities were led by the project team, including the management of operational costs. One new activity that is effective in improving practices and processes, the QI/QA methodology, can be transitioned to the DPS and SDSMAS. Nonetheless, it was noted that the speed at which these activities will continue will be low. Some of the DPS staff cited heavy workload and the lack of funds, while others cited staff transfer and turnover as limiting factors. Question 4: To what extent has the project been able to create and strengthen linkages between health facilities and communities to allow for increased service uptake? The project had strong community linkages through SAs with CBOs and collaboration with other groups and organizations such as MULEIDE, M2M groups, PLHIV groups and PCC. The CBOs mobilized volunteers/activists to support the community and HF linkages as CCMs and HF-based case managers. The CCMs held health talks with the general community and PLHIV, made referrals, including escorting TB cases to the HF, tracked ART defaulters, mobilized the community for ANC and HCT, conducted community-based HCT, encouraged couple counseling through drama and couple dialogues, and looked for GBV victims. The project also supported the reactivation and expansion of the youth-friendly clinical and counseling services at selected HFs. The formation of GAACs and M2M groups were actively promoted through mobilization and provision of training and logistics. The extent to which these activities improved linkages is summarized below. (a) Linkages to increase service uptake of community-based counseling and testing to treatment (for both men and women) More than 112,898 individuals (47 percent men) were tested from the community during the life of the project (more than 100 percent achievement of the set targets). The contribution of community-based HCT to the overall number of HCT clients improved from 16 percent in 2012 to 40 percent in 2014. The 1,898 people who tested HIV-positive were referred to the HFs and in many cases received by the case managers. The project staff reported over 80 percent of referrals reaching the HF. The CCMs had mobile phones for communication with HF staff and text message reminders to the patients. Findings about challenges are summarized in Chart 3. (b) Linkages to increase retention of pre-ART and ART patients Considering expansion of HCT, ART and PMTCT services to rural areas, sustaining retention rates for the 12-month cohort at 70 percent is an achievement (Figure 3). The HF case managers generate lists of defaulters, and the CCMs follow up with them in the communities. The M2M groups also followed up with the HIV-positive women for psychosocial support. The project supported the formation of GAACs in three districts. Expansion of GAACs to other districts was pending DPS approval. The project also provided logistics for these groups, and other community members discuss stigma and discrimination in the communities. The project supported formation of district-wide referral networks; some stakeholders believe it would be very useful if logistic support were available for frequent meetings of the network members to share and reconcile the patient transfer records. Performance Evaluation of the CHASS Niassa Project 13 Despite the efforts to track down the defaulters in ART, PMCT and exposed infants, over 40 percent of the defaulters were not returned to the HF, mainly due to wide population dispersion and frequent movement of communities. As reasons for persistent retention rates below the targeted 85 percent, key informants cited long distances and transportation challenges, limited coverage of the communities by activists, poor counseling at HFs due to insufficient number of staff and lack of privacy due to limited physical space at some HFs, fear to disclose HIV status to spouses, HIV stigma, and sociocultural factors, Other findings about the challenges are summarized in Chart 4. (c) Linkages to improve knowledge, demand and access of services by men Through a family-centered approach for HIV services, community drama and couple dialogues, and invitation of male partners of pregnant women, the project reached many men in the community. The project also supported CBOs to (1) form the Men2Men groups in each of the five target districts, focusing on issues of masculinity and HIV; (2) introduce invitations to prenatal care for men to accompany wives, receive HTC, and HIV/AIDS treatment if needed; and (3) establish a Gender Committee. Whereas the percentage of men on ART is still low, 47 percent of community HCT clients were men. Further, through partner invitations, more than 30 percent of the pregnant women had their partners tested for HIV. Findings about the challenges are summarized in Chart 4. Figure 3: Retention and community linkage indicators Percentage Chart 4: Challenges to Community linkages Challenges Recommendations  Reimbursements to the CBOs were always slow and often delayed the activities of CBOs and case managers.  Catchment areas of the HFs were too large for the few CCMs to cover.  Speaking with community about sexual behavior and safe sex is challenging, especially with men and women who do not know each other together.  There are stock-outs of HIV test kits in the community.  Defaulters have low return rates.  Improve on the reimbursement process (Chart 2).  Reinforce community-HF linkages further to include the faith-based organizations, traditional birth attendants, traditional healers, APEs and social assistants from the Ministry of Social Action.  Train CCMs/CBOs on requesting test kits.  Strengthen linkages between HF and community￾based services and promote regular information exchange for case-finding and referrals: o Conduct operations research to map linkages and measures to improve communication and information exchange. 63% 67% 43% 74% 71% 75% 63% 68% 0% 10% 20% 30% 40% 50% 60% 70% 80% 2011 2012 2013 2014 Retention on ART at 12 M (Adults) Retention on ART at 12 M (Children) Adult retention on ART at 12 M excluding preg. women 14 Performance Evaluation of the CHASS Niassa Project Challenges Recommendations  Case managers change constantly, mainly due to unpaid allowances and delayed reimbursements.  The lack of electronic databases in place made it hard to track retention rates accurately in large-volume HFs.  Pre-ART and ART retention rates are low. o Expand use of phone calls and text messaging to patients to provide reminders for clinic visits and drug dosing. o Pilot and expand HIV service coverage in underserved rural areas using mobile clinics with mHealth to provide: HCT, HIV care, ARVs, ANC/PMTCT, CD4, TB screening services and health education. Question 5: What constraints have the activities faced in improving retention of patients in pre￾ART and ART? The key strategies aimed at improving retention included active case-finding by the CCMs; GAAC members picking up drugs for each other and providing psychosocial support; community mobilization by CCMs, encouraging retention and adherence to ART; M2M following HIV-positive pregnant women; and psychosocial support from the PLHIV groups. The key constraints reported to have impeded these strategies from achieving desired results are summarized in Chart 5. Retention rates persisted below the 85 percent target across the provinces. A major constraint was the insufficient number of staff to provide adequate counseling in some of the HFs and communities. Retention is a cycle that begins at the HF or community level with counseling when an individual tests HIV-positive. If lay counselors or HF staff do not have time to counsel patients so that they understand that HIV needs continuous follow-up, retention problems arise. In CHASS, staff training on high-quality counseling did not translate into adequate counseling, because the staff workload was still high or even higher compared to the period before training. One HF in-charge noted that: “My greatest challenge is to get patients counseled at all points of entry of this facility. Staff have too much workload and some health workers just forget or do not think that it is important.” The influence of inadequate counseling is potentially higher on retention among HIV-positive pregnant women. Eligibility criteria have changed frequently among this group; from Option A to B and then to B+. Adequate explanations of these changes and the rationale for immediate initiation of an otherwise healthy woman on ART are necessary. Chart 5: Constraints to activities aimed at improving retention and recommendations Constraints/Weaknesses Recommendations  There is no electronic database for tracking retention, and records on defaulters are poorly organized.  Not all health workers conduct HCT, despite having test kits at all points of care.  The catchment area of each HF is far wider than the coverage of the CBOs.  Activists and volunteers must travel long distances to reach defaulters.  Break up of some GAACs: Due to stigma, not all GAACs are functioning well; some fear inadvertent disclosure of status to the community by other members.  Wrong addresses given by patients; migration or movement of the communities, e.g. during planting seasons, limited desired results of CBO activities.  CBOs reported stigma and fear of disclosure in many communities.  Advocate for and support the MOH in rolling out the EPTS to moderate- to large-volume HFs. In the meantime: encourage and train health workers to correctly fill in FILAs and other forms on retention and file them appropriately; initiate conversations with DPS about the roll-out plan for EPTS once the MOH approves it, including but not limited to data validation processes and tools, plan and terms of reference for data managers, and use of data from EPTS for reporting and the QI program.  Reinforce community linkages (see Chart 3).  Advocate for the MOH to develop a rational plan for HR transfers to ensure that staff stay in post for at least three years, and also to increase staffing levels.  Pilot and expand HIV service coverage in underserved rural areas, using mobile clinics to Performance Evaluation of the CHASS Niassa Project 15 Constraints/Weaknesses Recommendations  Reach to infants is complex, due to lack of disclosure of mothers to spouses, as well as distance; few receive PCR test and results within 12 months.  Limited physical space at some HFs is a barrier to adequate counseling and consequently to retention.  Babies who test HIV-positive are transferred to ART clinics where mothers sometimes do not go. dispense ARVs, provide HCT, HIV care, ANC/PMTCT, CD4 and TB screening services.  Partner with employer-based health programs to provide worksite HCT, referral, and ARVs and to ensure data collection and reporting to SDSMAS, using PDAs.  Promote referral of HIV-positive children to under-5 clinics, and train MCH nurses to provide treatment there. Question 6: What has been the most effective method found by the project to improve retention? The common strategies for improving retention included active case-finding by the case managers; community sensitization by activists and CBO staff, the GAACs and the PLHIV groups; and the follow￾up of HIV-positive pregnant women by the M2M groups. In many health facilities, these were working jointly to reduce cases lost to follow-up. Nonetheless, active case-finding stood out as an effective strategy but was noted to be costly and required more human resources. Question 7: To what extent has HF management (patient flow, etc.) improved over the life of the project? The HF management of patients and clients improved significantly over the life of the project. In medium to large facilities visited by the evaluation team, fluxograms were visible, showing patients/clients where to go for services. The one-stop shop models for PMTCT, MCH and TB services have improved patient flow, as patients receive all services in one place. In addition, the formation of GAACs, in which the members pick up medications for each other, has reduced the volume of patients at the clinics. Case managers were also helping to register or direct the patients at the facility. The project staff noted that in some of the large-volume facilities they helped to reorganize, the patient waiting times were reduced by more than half. Findings about the challenges are summarized in Chart 6. Chart 6: Challenges associated with improvement of health facility management Challenges Recommendations  Many HFs have limited physical space to sort the patients. With a projected increase in demand for services, this will create more patient flow challenges.  The improved patient flow is challenged by excessive staff workload, especially in facilities run by two or three health workers and with staff rotations or transfers.  There is limited understanding of factors associated with high patient turnout at the facility at certain times and days but not others.  A re-engineering strategy was implemented late in the project’s life, and therefore the long-run sustainability and efficiency are not known.  The GRM, with support from partners, should urgently invest in improving the physical space of HF clinics, labs, pharmacy and reception areas.  Understanding reasons for poor patient flow: The project should support the HFs to collect data on factors that explain high patient turnout at the facility at certain times and days but not others.  Put a training mechanism in place for sustainability of improvements: Institute peer-to-peer training protocol for 5 S at the HFs. This will help with training of new staff members and contribute to sustainability. 16 Performance Evaluation of the CHASS Niassa Project Question 8: To what extent is data management capacity built at the HFs with regard to HIV/AIDS registry data? Overall, the data management capacity at the HFs moderately improved over the life of the project, but not to the desired levels. In particular, the error rates of filling in the HIV/AIDS registry decreased, and correct filing of the registers and summarization of the registry records for data entry also improved. The project provided filing cabinets in most of the HFs. Some voices of praise for the contribution of the project to improved data management are noted below: “So at the end of data collection, one week later we gather all the health facilities for a meeting, we point out the errors found and give orientation on how they can improve both filing records and retention it all takes a day for the assistance.” (Nurse at Chimbunila Hospital) “There was a lack of the lockers, filing cabinets, which was a challenge to work without.” (HIV focal person Lichinga Hospital) Findings about the challenges are summarized in Chart 7. Chart 7: Challenges to HF data management capacities Challenges Recommendations  The insufficient numbers of HF staff and the high rate of turnover will always be a limiting factor. In HFs where the trained staff moved out, there were still some challenges with correct filling in of some forms and the registry.  There are too many forms/registries to manage without ignoring some clinical activities.  In some HFs the case managers acted as records personnel to fill in some of the registers. However, these personnel will no longer be available in most of the HFs after project closure.  The electronic databases, especially the EPTS, were still in pilot phase. As mentioned before, the DPS approval for EPTS roll-out was said to be slow. To some extent, the project was forced to create a parallel system.  Advocate for the roll-out of electronic databases, such as EPTS, by the MOH: The project should advocate with the MOH and support the ministry to roll out the electronic databases.  Put a training mechanism in place for sustainability of improvements: Institute a peer-to-peer training protocol for data management and quality improvement at the HFs. This will help with training of new staff members and contribute to sustainability.  Reduce the number of forms or registries: The ministry should consider reducing on the number of forms/registers.  Develop data management and use materials: Jointly with the MOH, develop materials to explain data management and use to health workers at the HF level. This will help them understand the purpose of ensuring high-quality data.  The GRM, with support from partners, should hire records personnel at medium-sized health facilities, and records and data management personnel at hospitals. Question 9: To what extent has knowledge (community and health worker) and utilization of GBV services increased over the life of the project? CHASS-Niassa engaged in activities that follow the MOH protocol to prevent GBV. The project focal point worked in partnership with MULEIDE (an NGO working on domestic violence), a DPS psychologist, the focal point for gender, and physicians to provide training at the provincial level and in five target districts. Training at the district level was directed to emergency care staff, MCH nurses, and psychiatry technicians. At the community level, these partnership members trained 164 activists, community leaders, traditional healers and traditional birth attendants. The project also assisted in organizing integrated services to provide victims of GBV with immediate health services, such as HCT, HIV exposure prophylaxis, emergency contraception, counseling and access to the judiciary courts. Performance Evaluation of the CHASS Niassa Project 17 In addition, the project engaged community leaders and men through Men-to-Men groups to increase their understanding of gender issues and to encourage community leaders to report violators and mobilize victims to seek care within 72 hours. The project also supported the Gender Committee (Comite de Genero), composed of community leaders, health focal points, police and judiciary. The emphasis of this committee is to encourage community leaders to follow up on cases at the community level. The main challenges were cultural perspectives that domestic violence is acceptable and the cultural gender roles. A summary of the other findings is included in Chart 8. Chart 8: Challenges and weaknesses in scaling-up GBV services Challenges/weaknesses Recommendations  Challenges included the cultural perspective that domestic violence is not a problem.  Violence victims are reluctant to identify the violators or report to police.  At the community level, violators pay, and the victims do not go to the HF for treatment for fear of being asked for the identity of the perpetrator.  Trained staff often transferred to other HFs.  Health worker knowledge increased, but attitudes toward GBV are still poor.  Reinforce messages to health workers to be more sensitive and concentrate more on GBV victims.  Encourage research to better understand cultural issues that impact GBV.  Train health workers to integrate GBV assessment into HIV care at all points of service.  Increase community engagement; work with Gender Committees and Social Assistants from the Ministry of Women and Social Action. Question 10: What are the benefits and challenges of the activities’ model of working with government (at provincial and district levels) through sub-agreements? Benefits: The project signed SAs with the provinces, districts and CBOs. Management of the SAs evolved slowly over time, but even so, they enabled the development of management and financial systems and capacities at the districts and provinces. The SDSMAS and DPS reported that the SAs provided them an opportunity to participate in the management of SA activities and report writing. Challenges: The activities defined in SAs are always inflexible and in the short-run did not allow for correction of planning errors. Similarly, the change in leadership at the districts and DPS posed challenges for the smooth running of SA activities. It was reported that whenever the directors or chief medical officers changed, they wanted to change some of the activities in the SAs. The SDSMAS and DPS reported substantial bureaucracy and delays in reimbursements or fund releases. The project reported having few project staff to review the reports. Project staff also reported that strict internal controls within FHI 360 affected timely disbursement of funds both for project staff activities and to provide reimbursements. This delayed some project activities, forcing minor changes in work plans. Further, there were no budget lines for non-HIV-related services to support emergencies. Recommendations: 1. Establish a budget item for non-HIV-related services: The project should consider establishing a fund for emergencies or disease outbreaks that are not necessarily within the scope of CHASS. 2. The DPS and SDSMAS should be involved in the initiation design of the projects and also play an active role at all stages of the project. The transition plan should be agreed upon by the DPS and “Health workers don’t see gender as their job, not important. Some health workers do not want to ask women when they come for PMTCT about violence issues. Seems like they wait for the patient to tell them instead of asking.” (GBV focal point) 18 Performance Evaluation of the CHASS Niassa Project MOH and be explicitly incorporated in the project documents, with clear strategies for sustainability. 3. Improve the disbursement and reimbursement process: The project should have an adequate number of dedicated staff for reviewing and quickly turning around the activity reports from CBOs, DPS and SDSMAS. Further, the internal financial controls need continuous review, with the aim of making them flexible enough to enable smooth running of project activities. IV.2 OVERALL CONCLUSIONS In general, the project achieved most of its targets, but more efforts are needed to improve these further and sustain them. In particular, more efforts and resources are required in HSS. There was limited direct support to HSS at the DPS and SDSMAS, and limited operations research was done to understand reasons for low retention rates, poor patient flow–especially in rural HFs–and the low recruitment of children on treatment. The project supported the decentralization of ART services from hospitals to the health centers. In total, 46 ART sites were supported. The project also supported PMTCT and HCT services in 65 sites and the scale-up of Option B+, in which at least four in every five HIV-positive pregnant women receive ARVs for PMTCT. The insufficient number of staff at the HF level is still a challenge. The long distances traveled by patients and community movements remain big challenges to retention efforts. The next project should utilize mobile ART clinics, mobile phones and strong communities. IV.3 FUTURE DIRECTIONS This list of recommendations for future directions was derived from lessons learned from the project and also the findings from the evaluation. A. Health systems strengthening 1. Involve the DPS and SDSMAS in the project’s design; they should play an active role at all stages of the project. The transition plan should be agreed upon with the SDSMAS, DPS and MOH and be explicitly incorporated in the project documents through the first year of the project. Face￾to-face interactions should be a preferred method of communication with the SDSMAS, DPS and MOH. The MOH should actively play its role as agreed with the project to ensure its deliverables, success and sustainability. 2. Introduce a component of operations research into project activities and in local capacity building. 3. Develop tools and methodologies for regular assessment of the project’s HSS support. The graduation path assessment that uses a 23-item tool to assess systems and capacities of various components against standards should be used to measure progress in HSS.3 The action plans that follow the assessment will benefit from the use of the QI approach. 4. Put a training mechanism in place. This could include a peer-to-peer model in which trained health workers provide tutoring at the nearby health facilities; work with training institutions to establish courses in HSS for all health workers; and training a group of national and district-level trainers. 5. Expand mHealth in project areas to more sites: Most of the aspects of data management and patient follow-up can be performed with mobile technology. A future project should expand on what CHASS did in this area. This would include the use of tablets, PDAs or mobile phones by 3 This tool was developed by Abt Associates, one of the implementing partners of CHASS Niassa project. Performance Evaluation of the CHASS Niassa Project 19 CCMs or activists to input client information; text messaging reminders for patient clinic visits, including mothers returning babies for HIV testing and adherence to ARVs; and working with a mobile phone company to provide online tutorials for health workers on the tablets. 6. The supply chain and logistics system is still weak. A private company should be hired to manage it until the DPS and SDSMAS have developed the necessary capacity. 7. Advocate and support the MOH to roll out the EPTS to moderate- to large-volume HFs. In the meantime: encourage and train health workers to correctly fill in FILAs and other forms on retention and file them appropriately; initiate conversations with DPS about the roll-out plan for EPTS once the MOH approves it, including but not limited to data validation processes and tools, plan and terms of reference for data managers, use of data from EPTS for reporting and the QI program. B. Clinical Services (enrolment and retention) 8. Advocate with the MOH to develop a plan to rationalize HR development to: a. Ensure a better staff retention package that also ensures that the current staff remain in their posts for more than two years. b. Increase the number of staff contracted by the GRM over time, considering projected increases in demand for services in general, and HIV services in particular. 9. Implement an augmented MOH strategy of a lay counselors’ workforce by recruiting and training lay counselors or retraining case managers and APEs as lay counselors to provide services. Some counselors should be stationed at the HFs and one in each community, providing services such as: a. adequate counseling for pre-ART care and ART initiation b. community sensitization against HIV stigma c. psychosocial support to HIV-positive pregnant women to disclose HIV status to their partners and also to return the baby on schedule for HIV testing and ART initiation d. targeting male partners of pregnant women for HIV testing e. generating a list of defaulters at HFs and motivating them (directly and through phone text messaging) to visit HFs or mobile clinics f. tracking children with HIV-positive status to deliver PCR results g. psychosocial support to GAACs and individual PLHIV h. adherence counseling 10. Communication between lay counselors and other health workers could be through mobile phones. The data collection could also be done through mobile phones or tablets. 11. Reinvigorate linkages between HF and community-based services, and promote regular information exchange for inter-referrals and health information messaging to communities: a. Conduct operations research to map linkages and measures to improve communication and information exchange. 12. Explore adding new activities to CBOs, including creating links with faith-based organizations, traditional healers, traditional birth attendants and others. 13. Expand the mobile ART outreach into mobile clinics with mHealth to provide: HCT, HIV care, ARVs, ANC/PMTCT, CD4, TB screening services and health education. 14. Maintain the family approach model and community drama to reach couples and children. 20 Performance Evaluation of the CHASS Niassa Project 15. Expand the HIV prevention, care and support services for adolescents at health facilities, with emphasis on adolescent girls. Develop a school-based integrated health program that includes adolescent health and HCT services. 16. Improve HIV testing and ART initiation among infants through training all HF health workers in: taking PCR samples to provide HIV testing in all pediatric entry points, including the children of patients on ART; linkage of records in labor/delivery units and at-risk child consultations (improve pediatric care registries); and adequate counseling of pregnant women about early ART initiation among infants and HIV status disclosure to partners, and counseling against HIV stigma; and DBS processing and storage and distribution of SMS printers to rural health facilities. C. Community Linkages 17. Expand the current activities of the project (see Recommendation 8). 18. Partner with employer-based health programs to provide training and technical support to institute worksite HCT testing, referral, and dispensing ARVs, particularly for male workers. D. GBV Services 19. Expand the current the GBV community linkages and collaborations to other communities. 20. Develop the capacity of all health workers so that GBV screening is done at all HF points of care. E. Sub-agreements 21. Establish a budget item for non-HIV-related services: This can fund emergencies or disease outbreaks that are not necessarily within the scope of CHASS. 22. Improve on the reimbursement process: Review internal financial controls to make them flexible enough for smooth running of project activities. IV.4 SUMMARY OF RECOMMENDATIONS The recommendations are summarized in the following table, indicating what USAID partners and the MOH are responsible for and where the responsibility is shared. Table 3: Recommendations for future programs Health Systems Strengthening Partners MOH 1. Involve the DPS and SDSMAS in the design of the project. They should play an active role at all stages of the project through face-to-face interaction. The transition plan should be agreed upon with the DPS and MOH, and its implementation should begin in the first year of the project. The MOH should actively play its role as agreed with the project to ensure its deliverables, success and sustainability. √ √ 2. Introduce a component of operations research into project activities and also into local capacity building. √ 3. Develop tools and methodologies for regular assessment of the project’s support for HSS. The graduation path assessment, with indicators to measure progress in HSS, should also be applied. The action plans following from the assessment will benefit from the use of the QI approach. √ 4. Put a training mechanism in place. This could include a peer-to-peer model in which senior/trained health workers provide tutoring at the nearby √ √ Performance Evaluation of the CHASS Niassa Project 21 health facilities; working with training institutions to establish courses in HSS for all health workers; and training a group of national and district￾level trainers. 5. Expand mHealth in project areas. √ √ 6. Advocate for and support the MOH to roll out the EPTS to moderate- to large-volume HFs. In the meantime: encourage and train health workers to correctly fill in FILAs and other forms on retention and file them appropriately; initiate conversations with DPS about the roll-out plan for EPTS once MOH approves it, including but not limited to data validation processes and tools, plan and terms of reference for data managers, and use of data from EPTS for reporting and the QI program. √ √ Clinical Services (enrolment and retention) 7. The GRM/MOH should design a better staff development and retention package or plan to promote staff retention in post for at least three years to allow for consolidation of experience. The GRM/MOH should also increase staffing levels to cope with increased client load, especially for HIV/AIDS services. √ 8. The GRM/MOH, with support from partners, should invest in improving the physical structure and spaces of the HFs. √ 9. Reinvigorate linkages between HF and community-based services, and promote regular information exchange for case-finding, inter-referrals and health messaging to communities. √ √ 10. Pilot and expand HIV service coverage in underserved rural areas using mobile clinics with mHealth. √ √ 11. Implement an augmented MOH strategy of a lay counselors’ workforce by recruiting and training lay counselors or retraining case managers and APEs as lay counselors to provide services. √ √ 12. Improve HIV testing and ART initiation among infants through training all HF health workers in providing infant HIV testing and counseling at all points of entry at the facility, including children of patients on ART/pre￾ART care. √ √ 13. Expand the integration of HIV prevention, care and support in adolescent health services to more health facilities. √ √ 14. Reinforce QI and QA: Future projects should strengthen these components for clinics, pharmacies and laboratories. √ √ Community Linkages 15. Reinforce community-HF linkages and communication and expand the number and type of CBOs, such as faith-based organizations, traditional birth attendants, APEs and traditional healers. √ √ 16. Research cultural barriers to develop culturally sensitive and acceptable strategies to promote behavior change. √ 17. Partner with private-sector employer-based health programs to provide HIV services, particularly for male workers. √ GBV Services 18. Expand the current the GBV community linkages and collaborations to other communities. Develop the capacity of all health workers so that GBV screening is done at all HF points of care. √ 22 Performance Evaluation of the CHASS Niassa Project Sub-agreements 19. Establish a budget item for non-HIV-related services: This can fund emergencies or disease outbreaks that are not necessarily within the scope of CHASS. √ √ 20. Improve on the reimbursement process: Review internal financial controls to make them flexible enough for the smooth running of project activities. √ Performance Evaluation of the CHASS Niassa Project 23 V. REFERENCES Clinical HIV/AIDS System Strengthening Request for Applications 2009 CHASS NIASSA annual work plans 2011 CHASS NIASSA annual work plans 2012 CHASS NIASSA annual work plans 2013 CHASS NIASSA annual work plans 2014 CHASS NIASSA annual work plans 2015 CHASS NIASSA Quarter 1 Report 2011 CHASS NIASSA Quarter 2 Report 2011 CHASS NIASSA Quarter 3 Report 2011 CHASS NIASSA Quarter 4 Report 2011 CHASS NIASSA Quarter 1 Report 2012 CHASS NIASSA Quarter 2 Report 2012 CHASS NIASSA Quarter 3 Report 2012 CHASS NIASSA Quarter 4 Report 2012 CHASS NIASSA Quarter 1 Report 2013 CHASS NIASSA Quarter 2 Report 2013 CHASS NIASSA Quarter 3 Report 2013 CHASS NIASSA Quarter 4 Report 2013 CHASS NIASSA Quarter 1 Report 2014 CHASS NIASSA Quarter 2 Report 2014 CHASS NIASSA Quarter 3 Report 2014 CHASS NIASSA Quarter 4 Report 2014 OIG findings and report of CHASS Niassa audit Public Financial Management Risk Assessment Framework reports for the four provinces CHASS USAID progress report data: Data from President’s Emergency Plan for AIDS Relief (PEPFAR) annual and semiannual reports, including both community and clinical data (2011, 2012, 2013, 2014) Partner- and USAID-conducted data quality assessments FHI Technical Proposal 2010 24 Performance Evaluation of the CHASS Niassa Project ANNEXES Performance Evaluation of the CHASS Niassa Project 25 ANNEX I. EVALUATION SCOPE OF WORK Performance Evaluation of USAID’s Clinical HIV/AIDS System Strengthening project in Niassa Province (CHASS Niassa) implemented by FHI 360 (with sub-award to Abt Associates Inc.) Anticipated Period of Performance: August, 2015–December 31, 2015 BACKGROUND USAID/Mozambique’s Clinical HIV/AIDS System Strengthening Project (CHASS) is a results-oriented five-year project to improve HIV clinical services in Manica, Niassa, Sofala and Tete4 provinces within a strengthened, comprehensive primary health care system. This project was designed to address the HIV situation in Mozambique, and focused on three components: 1) Improving service quality in six important areas: HCT, laboratory services, PMTCT, adult care and treatment, pediatric care and treatment, and the prevention, diagnosis and treatment of HIV-TB co-infection 2) Enhancing program linkages and integration to provide a continuum of accessible services, including MCH and RH services, within facilities and between facility and community-based services 3) Creating stronger and more sustainable Mozambican systems and institutions At the time of project design, the average HIV prevalence was estimated at 16 percent nationwide (ANC survey in pregnant women aged 15-49 years). Nearly 1.6 million people were living with HIV in Mozambique, and at the time of writing the request for applications, nearly half of all HIV-infected (48.4 percent) were identified as having active tuberculosis. The burden of HIV is considerably higher in Mozambique’s Central Region, with a HIV prevalence of 18 percent5 compared to national prevalence of 16 percent6. Within the Central Region, Sofala and Manica provinces have the highest prevalence rates, 23 percent and 16 percent respectively. Tete was identified as having a very mature epidemic and existing infrastructure that was unable to accommodate the numbers of patients requiring care and treatment. Niassa is considered a particularly underserved province, with vastly inadequate infrastructure, but with a much lower HIV prevalence of approximately 3 percent. To address these issues, two separate agreements were awarded to two implementing partners: FHI 360 (CHASS Niassa) and Abt Associates Inc. (CHASS SMT, for Sofala, Manica, and Tete). Table 1 provides details of each award. Funding details for CHASS Niassa: See Attachment 1–Table 1. Both activities support USAID/Mozambique’s Country Assistance Strategy’s (CAS 2009-2014) priority goal number three, “Improved health of Mozambicans,” and, more specifically, contribute to the following focal areas in USAID’s Health Results Framework: 4 The CHASS project in Sofala, Tete and Manica, implemented by Abt Associates, will be evaluated through a separate purchase order. 5 Statistics included here are from the original RFA; however, the burden of disease continues to be high in the Central Region. 6 Relatório sobre a Revisao dos Dados de Vigilância Epidemiológica do HIV, Ronda 2007. Grupo Técnico Multisectorial de Apoio A Lutta Contra o HIV/SIDA em Moçambique. Fevereiro, 2008. 26 Performance Evaluation of the CHASS Niassa Project  Improved access to and delivery of quality integrated services  Increased adoption of healthy behaviors and informed use of services  Strengthened health systems While the two activities are aligned in terms of the results they are expected to achieve, each activity has its own approach for achieving those results. CHASS Niassa’s goal is to strengthen the provincial health system and enhance the DPS’s capacity to manage its own health systems and finances, increase human resources for health, improve quality and use of strategic information, strengthen local organizations, and align with national priorities and plans in health and HIV. The activity’s objectives are to: 1. Increase access, quality and use of HIV care and treatment services to rural communities by intervention in seven areas: CT, laboratory services, PMTCT, adult care and treatment, pediatric care and treatment, palliative care, and prevention, diagnosis and treatment of HIV-TB co￾infection; 2. Provide a continuum of accessible HIV and related primary health care services, including MCH and RH services (including support at clinics that do not provide ART or PMTCT), and to improve linkages and referrals within and between facilities and communities; and 3. Support stronger and more sustainable Mozambican systems and institutions through emphasis on strengthening government and community capacity to deliver and manage services. A key shift in the period of time since the two CHASS activities were awarded was the development of an HIV Acceleration Plan by the MOH. In 2011, Mozambique signed the Political Declaration on HIV/AIDS: Intensifying our Efforts to Eliminate HIV/AIDS. The “Mozambique HIV and AIDS Response– Strategic Acceleration Plan 2013-2015” was created to respond to this commitment to an end of AIDS. This three-year strategy reflects the united vision of all stakeholders to achieve a Generation Free of AIDS in Mozambique, focusing on three major goals: 1. Increase the percentage of eligible HIV-infected adults and children receiving antiretroviral therapy to 80 percent by 2015. 2. Reduce the rate of transmission of HIV from mother to child to less than 5 percent by 2015. 3. Reduce the number of new infections by 50 percent by 2015. Both CHASS SMT and Niassa also include programming for GBV prevention, with a total life-of-project funding of approximately $1.5 million. GBV is a pervasive social behavior that leads to disability and death, and undermines the quality and productivity of community life as a whole. GBV prevention activities are implemented in all four provinces to: (i) strengthen the quality of care provided to victims of GBV presenting at health facilities, (ii) promote a continuum of care through supporting functional links between facilities and community-level support services for GBV survivors in conjunction with FHI 360’s PCC project, and (iii) conduct advocacy and sensitivity training among DPS and district health management teams and influential community members to support implementation of the national GBV strategy. While GBV activities make up only a small part of the budget and activities under CHASS Niassa and CHASS-SMT, it is an area that requires increased understanding and more information about what strategies have worked to improve access to GBV services. 1. EVALUATION PURPOSE AND OBJECTIVES This evaluation comes during the fourth year of project implementation. The purpose of this process/performance evaluation is to determine how the activity is performing relative to its objectives, as well as to provide an objective view of progress toward the expected results. The outcomes of this evaluation will provide information to be included in future activity design, as USAID begins to consider Performance Evaluation of the CHASS Niassa Project 27 future system-strengthening and clinical-service-delivery activities. Further, results will be used to identify gaps in project performance and to help USAID and the implementers determine what changes may be necessary to solidify progress during the remaining project period. The main audience for this evaluation is USAID; however, results will also be shared with CHASS Niassa, the Mozambique Ministry of Health, other U.S. Government agencies, other implementers and other stakeholders. The objectives of the CHASS Evaluation are to: 1. Assess CHASS Niassa achievements, emphasizing objectives and activity- and project-level results; 2. Identify implementation successes, as well as any internal and external constraints that hindered the implementation of planned activities; and 3. Propose recommendations for future directions of CHASS Niassa and for future activities in system strengthening and service delivery to support improved performance in addressing the HIV epidemic, in line with the GRM HIV/AIDS acceleration plan and new PEPFAR guidance. An additional goal, in terms of evaluation process, is to use mobile technology, when possible and appropriate, to increase the efficiency, transparency and accuracy of performance data, and to take advantage of multiple data sources (e.g., pictures, videos, GPS data). Electronically collected data will ultimately feed into a platform that is being developed simultaneously with this evaluation. Moving forward, USAID intends to incorporate mobile technology into more and more evaluations, utilizing the new platform to organize, analyze and report data. The CHASS evaluation will be one of the first evaluations to feed into the new platform and will help contribute to the platform’s development. 2. EVALUATION QUESTIONS The USAID team has developed a set of 10 evaluation questions, each of which belongs to one of three main evaluation question areas: (1) project achievements and challenges, (2) linkages and (3) HSS. A complete list of the evaluation questions is shown below. The awardee will ensure that each question is answered fully and completely, underscoring both positive and negative outcomes. All evaluation questions must be answered with a gender focus, going beyond sex disaggregated information, where applicable, and analyzing outcomes in relation to gender and sex. The awardee will apply, administer and evaluate each question separately. The findings for each question will be addressed in a separate section of the evaluation report. A complete list of the evaluation questions, organized by question area, is as follows: Question Area 1: Project Achievements and Challenges 1) What are the strengths and weaknesses of the activities, as seen by the implementing partner staff, DPS, District Directorate of Health and Social Welfare (DDS and Chief Medical Officer), health facilities, USAID, and the USAID-funded PCC Project, and how can weaknesses be improved according to the stakeholders listed here? 2) Where has the most progress been seen in strengthening systems (e.g., planning, financial management, supply and logistics, information systems)? 3) Which activities or project components will be most feasible to transition from the project to the GRM? Question Area 2: Linkages 4) To what extent has the project been able to create and strengthen linkages between health facilities and communities to allow for increased service uptake, specifically in the areas of: a. Community-based counseling and testing to treatment (for both men and women); 28 Performance Evaluation of the CHASS Niassa Project b. Retention of pre-ART and ART patients (through the use of adherence groups, active case-finding, and other community groups); and c. Knowledge, demand and access of services by men 5) What constraints have the activities faced in improving retention of patients in pre-ART and ART? 6) What has been the most effective method found by the project to improve retention? Question Area 3: Health System Strengthening 7) To what extent has health facility management (improved patient flow, etc.) improved over the life of the project? 8) To what extent is data management capacity built at the health facilities with regard to HIV/AIDS registry data? 9) To what extent has knowledge (community and health worker) and utilization of GBV services increased over the life of the project? 10) What are the benefits and challenges of the activities’ model of working with government (at provincial and district levels) through sub-agreements? 3. EVALUATION DESIGN AND METHODOLOGY This performance evaluation will be based on a non-experimental design. The awardee will conduct an evaluation that, to the extent possible, uses a mixed-methods approach. However, the awardee will mostly utilize qualitative data collection and evaluation methods in the evaluation. This awardee will not focus on evaluating quantitative outputs, but will be required to use USAID’s monitoring data for triangulation purposes. Data Collection Methods: Because this will be a performance evaluation that utilizes mostly qualitative data, the awardee will focus on using key informant interviews, stakeholder consultations, focus group discussions, observational analysis and beneficiary interviews as the main data collection methods. The awardee will consider a variety of methodologies to ensure that evaluation questions are adequately addressed. Mobile Data Collection: In this evaluation, the awardee will use multi-channel data collection; using mobile technology when possible and paper-based methods when mobiles are not appropriate. The USAID/Mozambique mission has limited experience using mobile technology to conduct evaluations, either on its own, or through evaluation contractors. Generally, evaluation tasks, such as those mentioned above, have been paper-based. When using this paper-based system, results have to be collated, which takes time; also, transcription is sometimes inaccurate, and costs can be substantial. New technologies can make data collection (both quantitative and qualitative) easier, faster, more accurate and cheaper. It is becoming more commonplace that researchers in the field use simple mobile phones, smart phones and tablets to carry out their surveying tasks. USAID/Mozambique is interested in leveraging these new technologies and methods in this evaluation. In using mobile technology for the CHASS Evaluation, USAID has four key goals: 1) Efficiency: Collecting and sharing information as quickly and reliably as possible 2) Transparency and sharing: Using open data to share field-level data with key stakeholders 3) Accuracy: Collecting information in real time to best capture realities on the ground 4) Cost-saving: Applying mobile devices to decrease expensive transcription and enumerator costs, when possible and appropriate Performance Evaluation of the CHASS Niassa Project 29 The awardee will conduct a desk review prior to arrival in country to ensure that evaluators are familiar with existing data and project progress according to written documentation. Data collection will take place in Niassa, as well as in Maputo. Data collection in Maputo will consist mostly of desk review and related data collection with USAID and implementing partners’ staff located in the city. Each evaluation group–comprised of members of the awardee’s evaluation team–will cover two provinces. (Details on the evaluation team and evaluation groups can be found below in the “Evaluation Team Composition” section.) The awardee will create and finalize data collection methodology and instruments that must be approved by USAID prior to any key informant interviews or site visits. The selected awardee will be required to use a mobile data collection tool for the evaluation. Mobile devices with GPS functionality will need to be provided by the selected firm. Additionally, for purposes of quality assurance, USAID will be provided with all raw data collected during the evaluation. Existing USAID and external reports, data and documentation to be used: The USAID/Mozambique IHO will provide the awardee with all necessary background documents (request for applications, project’s annual work plans and reports, MOH health sector strategy, etc.) and data. The awardee will conduct a desk review of the documents and data, some of which are written in Portuguese, and may request further documents from USAID and/or the CHASS projects as needed. Below is a list of documents that are currently available for use and review:  Clinical HIV/AIDS System Strengthening Request for Applications  CHASS Niassa annual work plans and quarterly and annual reports  Site visit reports  Public Financial Management Risk Assessment Framework reports for the four provinces  MOH health sector strategy  MOH HIV Acceleration Plan  Data from PEPFAR annual and semiannual reports, including both community and clinical data  Analysis conducted based on PEPFAR annual and semiannual reports  Partner- and USAID-conducted data quality assessments Proposed Stakeholders: For the purposes of this evaluation, health sector stakeholders in Mozambique include, but are not limited to, the MOH, DPS, SDSMAS, U.S. Government agencies (USAID, CDC, DoD, State Department), implementing partners, CBOs, health care workers and clients who receive services at the facilities that the projects support. The awardee will hold meetings with the following groups:  MOH: 2-4 staff from DNAM (Departamento Nacional de Assistencia Medica)  DPS (Niassa): 2-3 staff each  DDS: 2-4 district DDS leads in each of the four provinces  Health facility staff: 2-3 staff in 3-5 health facilities per province  Implementing partner staff in Maputo (CHASS Niassa and PCC): 4-6 staff each  Implementing partner staff in the provinces (CHASS Niassa and PCC): 4-6 staff per project per province  CBO: Eight CBOs supported by the project with 1-2 members per CBO Table 2 below highlights illustrative data collection methodologies that the awardee may use for this evaluation, organized by evaluation questions (See Attachment 2, Table 2). 30 Performance Evaluation of the CHASS Niassa Project Data Analysis and Disaggregation: The exact data analysis methods used may differ for and within each evaluation question and may depend on the data available. The awardee will specify the exact methods to be used for each evaluation question in the inception report, which will be discussed with and approved by USAID. In general, the awardee’s analysis of quantitative and qualitative data will consist of four components: (1) data reduction (i.e., open/initial coding, focused coding, axial coding); (2) displaying data; (3) drawing conclusions; and (4) verification through data triangulation. Where applicable and feasible, the awardee will disaggregate all data by sex and evaluate the disaggregated data for any gender-related differences. All qualitative and quantitative raw data, both soft- and hard-copy, will be owned by USAID. At the end of the evaluation, the awardee will release all materials to USAID, including summaries, interview and focus group data, databases, and notes, and any other data collection tool(s) used for this evaluation. 4. DELIVERABLES Aside from applicable survey tools, the awardee will submit and present all deliverables in English. However, certain presentations and discussions may be conducted in Portuguese. The awardee’s evaluation team will be responsible for providing the following deliverables to USAID for approval: a) Inception Report: The awardee will submit an inception report, which must include the following:  Detailed evaluation design and methodology  Identifying data to be collected, including precise explanation of data collection methods that will be used for each evaluation question  Sampling plan  Proposed list of key informants, focus group participants and workshop attendees  Data collection tools developed and pre-tested  Detailed data analysis plan  Detailed evaluation schedule and logistics, including debriefing with USAID  Roles and responsibilities of each evaluation team member. The submitted inception report must be approved by USAID prior to beginning key informant interviews, focus discussion groups, site visits, etc. b) Midpoint Briefing and Out-briefing/PowerPoint Presentation with USAID: The awardee will provide a midpoint briefing to USAID during the in-country evaluation period, preferably midway into the data collection process. c) Draft Evaluation Report: The awardee will submit a draft evaluation report, incorporating feedback and comments received from the debriefing. The awardee will also submit any raw data (qualitative and quantitative) collected. After receiving the report, USAID will have 10 calendar days to provide the team with one set of written comments. This may include an open source website that shares the multimedia data collected (pictures, videos, GPS data) and mapping that allows USAID to track performance, progress and challenges visually. d) Final Report (for USAID and for the public) and Final Presentation: The awardee will submit a final report after receiving final comments from USAID/Mozambique. This report should not exceed 30 pages in length (not including references, appendices, etc). The final format (see below) must meet the quality standards outlined in the Evaluation Policy (http://www.usaid.gov/sites/default/files/documents/1868/USAIDEvaluationPolicy.pdf, attached at Performance Evaluation of the CHASS Niassa Project 31 Annex A is the standard and required USAID Report Format Template), and must include the following:  Table of contents  Acronyms  List of tables  List of graphs  Executive summary: not to exceed five pages. Brief summary of project purpose and background, key evaluation questions, methods, findings and recommendations  Introduction and background: purpose, audience and synopsis of task, brief overview of the project, USAID program strategy and activities implemented in response to the problem, brief description of implementing partners  Evaluation methodology, limitations and gaps  Findings, conclusions and recommendations: Each evaluation question should be answered in its own section of the report. This section should include recommendations to USAID for future project design.  Issues: Provide a list of key technical and/or administrative issues, if any.  Future directions, to inform the design of any new intervention  References, including bibliographical documentation, meetings, interviews and focus group discussions  Annexes should include the evaluation scope of work and any amendments, evaluation tools, schedules, interview lists, tables, information sources, statements of differences, and any other information and data that was not required in the report. The awardee will submit the final report electronically, in English, and the report will, at a minimum, contain background, rationale, methodology, the evaluation’s key objectives, evaluation questions, major findings, and recommendations/conclusion. The awardee will develop a final presentation based on the final report and will deliver that presentation to the USAID mission. The awardee will also develop a public version of the final report, to exclude any potentially procurement sensitive information, which will also be submitted electronically and in English. This public version of the final report is meant for dissemination among implementing partners and stakeholders. The public version of the report will be released as a public document on the USAID Development Experience Clearinghouse (DEC) http://dec.usaid.gov. Deliverables Due date 1. Inception report August 17, 2015 2. Midpoint briefing to USAID September 7, 2015 3. Data collection completed September 28, 2015 4. Draft evaluation report submitted to USAID October 12, 2015 5. Final presentation to USAID, with PowerPoint presentation October 22, 2015 6. Final report (for USAID dissemination), including clean and identified datasets November 13, 2015 7. Public final presentation November 24, 2015 EVALUATION SCHEDULE OF ACTIVITIES The evaluation is anticipated to begin in January or February. 32 Performance Evaluation of the CHASS Niassa Project Team Planning Meeting (TPM): The awardee will participate in a one-day TPM that will be held at the beginning of the assignment with USAID. The meeting is essential for the following reasons:  Agreeing upon approach for working with USAID staff and partners throughout the assignment  Reviewing and finalizing the evaluation schedule to share with and be approved by the mission (a draft of this schedule should be prepared prior to the TPM)  Enabling USAID staff to discuss with the team the overview and purpose of the evaluation  Finalizing a field visit and meeting schedule for the time in country  Developing data collection methods, instruments, tools and guidelines, software and hardware  Developing a timeline that allows for technology piloting and corrections. See sample roll-out below (Attachment 3, Table 3). USAID/Mozambique Meetings: During the course of the field work, meetings with USAID/Mozambique will include:  Initial organizational/introductory meeting(s) at which the awardee will present an outline and explanation of the design of the evaluation  Additional consultation meetings with USAID staff, as needed  Final evaluation presentation–summary of the data, draft recommendations and draft report prior to departure from country Logistics: USAID/Mozambique will assist the awardee in scheduling initial meetings and interviews. The awardee will be responsible for scheduling follow-up meetings and other activities that are deemed necessary once in country. The awardee’s costs should include transportation and international travel to/from Mozambique and the awardee will be responsible to arrange for lodging, travel concurrence, local travel expenses, etc. Given the distance between Maputo and relevant provinces, transportation shall be by air; travel within Niassa may be by car. EVALUATION TEAM COMPOSITION The awardee will propose an evaluation team that is comprised of at least nine people: the team leader, one senior evaluation expert, two technical experts; a logistics/administrative specialist; and at least four enumerators. At least one technical expert, the logistics/administration specialist, and the enumerators must be Mozambican. USAID/Mozambique and MOH staff may also be part of the evaluation team, under the supervision of the team leader, excluding any drafting or any evaluation deliverables. The awardee’s evaluation team will have a combination of technical skills per the aspects and linkages related to HIV/AIDS care and treatment outlined in the objective section of this scope of work. During field work, the entire evaluation team will be split into two separate evaluation groups to ensure that all four provinces are covered in the planned time period. One team will be led by the team leader, joined by one technical expert and any enumerators; the other team will be led by the senior evaluation expert, joined also by one technical expert and any enumerators. Each team will need to function independently and have a full range of skills needed (included technical background, HIV/PEPFAR experience and past experience in conducting evaluations). An outline of how the evaluation team will be divided into evaluation groups will be included in the awardee’s proposal. In order to preserve integrity and transparency, none of the consultants/individuals proposed as members of the awardee’s evaluation team will have worked for the areas or partners to be evaluated, or may not have been part of the design of these projects or activities. Below are descriptions of the required characteristics for each member of the evaluation team: Performance Evaluation of the CHASS Niassa Project 33 Evaluation Team Leader: The team leader must have at least eight years of experience in implementation of health activities, including experience leading a minimum of two health sector project/activity performance evaluations. S/he should be familiar with the Mozambican health care system and ongoing health care transformation in the country and have three or more years of experience in HIV/AIDS care and treatment programs, preferably PEPFAR-funded. S/He will have demonstrated experience as an evaluation team leader as well as leading qualitative and quantitative interviews/evaluations. S/He must also have developed communication skills (both verbal and written), the ability to conduct interviews and facilitate discussions in both English and Portuguese, and extensive report writing experience. S/He should have solid client interaction skills, leadership, flexibility, and management skills, and experience interacting with host government officials, civil society partners, and other stakeholders. Additionally, s/he should also have the skills detailed below for the technical experts. The team leader will be responsible for designing, implementing and managing the evaluation and providing team leadership. S/He will be the principal interlocutor between the evaluation team and USAID/Mozambique. S/He will, in collaboration with USAID and other team members, draft assessment tools, finalize the evaluation design, coordinate activities, arrange meetings, consolidate individual input from team members, and coordinate the process of assembling the final findings and recommendations. S/He will also lead the preparation and presentation of the key evaluation findings and recommendations for future directions of the project to the USAID/Mozambique IHO team and key partners. Senior Evaluation Expert: The senior evaluation expert will be responsible for working closely with the team leader in preparing evaluation deliverables and performing data collection and analysis, and other tasks assigned by the team leader. S/he will also lead one of the two evaluation groups (see above), providing general evaluation management, quality control, and oversight during the field work. S/he must have led at least one health￾sector evaluation that included significant qualitative work, must have at least five years of evaluation experience, and must be able to demonstrate experience using qualitative evaluation methodologies, and triangulating with quantitative data. S/He must be able to conduct all interviews and background research in English and Portuguese, and must have excellent data interpretation and presentation skills. Additionally, s/he should also have the skills detailed below for the technical experts. Technical Experts: The technical experts, in collaboration with the team leader and the senior evaluation specialist, will provide significant input to the assessment tools and evaluation design. They will participate in meetings, provide input to the final presentations and evaluation report, and may write some aspects of the report, as determined by the team leader. The two technical experts must both have more than five years of experience working on clinical and/or health system strengthening activities. They should have substantial knowledge about health programming, including past experience in provision of HIV clinical services or health system strengthening. They should have solid client interaction skills, flexibility, and communication skills, as well as experience interacting with host government officials, civil society partners and other stakeholders. Both technical experts must have technical evaluation skills, including experience with SPSS or similar data analysis software, experience analyzing HIV/AIDS data and evaluation/study results, experience drafting and presenting evaluation results, and experience conducting qualitative and quantitative evaluations. They also should have experience coding qualitative survey responses as well as experience working in sub-Saharan Africa. Both technical experts must be able to communicate in Portuguese, meaning they must either be fluent in Portuguese or Spanish. In addition, one of the two technical experts must be Mozambican. 34 Performance Evaluation of the CHASS Niassa Project Local Logistics/Administrative Specialist: This person must have at least three years of experience coordinating events and travel, both international and within Mozambique. Based in Mozambique, s/he will manage all in-country travel, logistics, and other duties as assigned by the team leader. S/He may also be responsible for administrative and communications tasks such as procurement of good and services, including consumables, arranging travel, making photocopies and arranging venues for workshops/large group meetings. S/He will assist with communication with relevant evaluation participants, from the implementing partners to government officials, where appropriate. S/He will perform other duties as assigned by the evaluation team leader, and preferably will also be able to communicate in English. Local Enumerators: The evaluation team will include at least four Mozambican enumerators, two tasked to each of the two evaluation groups. These team members may interview key informants and assist with facilitating focus group discussions. They should have basic familiarity with health topics, as well as experience with interviews and note-taking. Ideally, enumerators will be based in Niassa. It is preferable for them to be able to speak at least one major dialect from Niassa to facilitate interview communication, specifically with local CBOs. They will join the evaluation team on site visits as determined by evaluation team leader. Attachment 1 Table 1: Funding Details of CHASS Niassa Attachment 2 Table 2: Evaluation Data Collection Methodology Matrix Question Proposed Data Collection Method (illustrative) 1. What are the project’s strengths and weaknesses, as seen by project staff, DPS, DDS and chief medical officer, HFs, USAID, and PCC, and how can weaknesses be improved, according to these stakeholders? Key informant interviews: It is important to capture perspectives from USAID, the project (including prime partner and SA holders), DPS, DDS, district chief medical officer 2. Where has the most progress been seen in strengthening systems (e.g., planning, financial management, supply and logistics, information systems)? Key informant interviews, review of project data (qualitative and quantitative) in systems areas 3. Which activities or project components will be most feasible to transition from the project to the government? Key informant interviews, extrapolation from project data and other reports (provincial public financial management risk assessment framework reports) 4. To what extent has the project been able to create and strengthen linkages between facilities and communities to allow for increased service uptake, specifically in the areas of: a. Community-based counseling and testing to treatment (for both men and women) Comparison of retention data/men’s access to services/referrals utilized in sites where there is a strong CBO presence and those where there is no CBO (or less activity) Focus groups with communities in sites where there is a strong CBO presence and those where there is no CBO (or less activity), surveys Activity Award Dates Activity Funding Implementing Partner CHASS Niassa August 1, 2010-July 31, 2015 $35,983,233 FHI 360 Performance Evaluation of the CHASS Niassa Project 35 Question Proposed Data Collection Method (illustrative) b. Retention of pre-ART and ART patients (through the use of adherence groups, active case-finding, and other community groups) c. Knowledge, demand and access of services by men 5. What constraints have the activities faced in improving retention of patients in pre-ART and ART? 6. What has been the most effective method found by the project to improve retention? 7. To what extent has HF management (improved patient flow, etc.) improved over the life of the project? Interviews with HF and project staff, observations within facilities, review of project reports Observation of patient flow, data registers, etc. 8. To what extent is data management capacity built at the HFs with regard to HIV/AIDS registry data? Analysis of data quality assessments previously conducted, results of routine data review meetings, key informant interviews, review of project reports 9. To what extent has knowledge (community and health worker) and utilization of GBV services increased over the life of the project? Focus groups with communities, key informant interviews, surveys, interviews with health workers, review of community and clinical data 10. What are the benefits and challenges of the projects' model of working with government (at provincial and district levels) through SAs? Key informant interviews: It is important to capture perspectives from USAID, the project (including prime partner and SA holders), DPS, and health workers) Attachment 3 Table 3: Sample Roll-out Pilot instruments on paper Test on mobile as appropriate Set up devices and test connectivity Check dummy data from pilot (and revise surveys accordingly) Roll out 36 Performance Evaluation of the CHASS Niassa Project ANNEX II. DATA COLLECTION INSTRUMENTS: KII, FGD Key Informant Interview Guide for DPS and DDS Staff Date: ___________________________ Province/District _________________________ Name of respondent: ___________________________________________ __________ Gender: M/F Designation: _____________________________________________________________ Number Question General 101. What do you consider as the main achievements of the CHASS project in this province/district? Was there a particular period when achievements were most accelerated (please explain)? 102. Are there differences in achievements across the districts? If yes, what could explain these differences? HIV/AIDS Care and Treatment 201. What do you consider as the main achievements of the CHASS project in this province/district in the delivery of HIV/AIDS care and treatment services? Ask the respondent to discuss pediatric care (if not mentioned). 202. What project activities or strategies worked well? Why? 203. What project activities or strategies did not work well? Why? 204. In your opinion, did the CHASS project adequately target the men and women? State some examples. 205. In your opinion, what do you think were the benefits of working with the following groups in improving coverage and quality of HIV care and treatment services? State the extent of their relevancy. a) Equipa Polivalentes b) Equipa de Apoio c) CBOs d) GAACs e) Adherence Committees f) Case managers 206. Please comment on the sustainability of the activities performed by the CHASS project? 207. In your opinion, what were the key challenges that the project faced in implementing some of its activities or strategies? 208. In your opinion, what project activities or approaches could have been done better by the CHASS project? 209. How did the project promote the integration of primary health care or maternal and child health with HIV/AIDS services? What approach worked, and what did not? 210. How did the project create and strengthen linkages between facilities and communities to promote increased service uptake of HIV/AIDS care and treatment services? 211. To what extent was the project successful in creating knowledge, demand and access to HIV care and treatment services among men? 212. How did the project create and strengthen intra-health facility linkages to ensure high quality of services? (How did the project engage providers across sectors at the health facility to integrate information on health records?) 213. What are your constraints in improving retention of patients in pre-ART and ART? How did the project help improve these constraints? 214. What are your constraints in improving pediatric recruitment and retention in care? How did the project help improve these constraints? 215. What was the most effective method found by the project to improve retention? 216. What are your recommendations for improving linkages and referral systems in the future? 217. Which of the approaches were most effective in improving use of healthcare services by men and women? 218. Which activities aimed at improving HIV/AIDS care and treatment services are most feasible to transition to DPS/SDMAS with little or no technical support from CHASS? 219. Are there activities that have been transitioned to the DPS/SDMAS staff? If yes, what are the successes achieved or failures faced? 220. What are some of the best practices and lessons learned from this project in improving recruitment and retention in health care services? Performance Evaluation of the CHASS Niassa Project 37 Number Question 221. In your opinion, what challenges will health facilities face in adopting some of the best practices or approaches in improving recruitment and retention of patients (men, women and children) in care and treatment, without technical support from CHASS? 222. What are your recommendations for sustainability of the project’s activities and achievements? PMTCT 301. In your opinion, what do you consider as the main achievements of the CHASS project in this province/district in delivery of the prevention of mother-to-child transmission services? 302. What project activities or strategies worked well? Why? 303. What project activities or strategies did not work well? Why? 304. In your opinion, what do you think were the benefits of working with the following groups in improving coverage and quality of PMTCT services? State the extent of their relevancy. a) Equipa Polivalentes b) Equipa de Apoio c) CBOs d) GAACs e) Adherence Committees f) Case managers g) MpM or M2M groups 305. Do you think the activities performed by these groups can continue without CHASS Niassa financial support? Why? Why not? 306. In your opinion, what are the key challenges that the project faced in implementing its activities or strategies? 307. In your opinion, what project activities or approaches could have been done better? 308. How did the project promote the integration of primary health care or maternal and child health with HIV/AIDS services? What approach worked, and what did not? 309. How did the project create and strengthen linkages between facilities and communities to promote increased service uptake of PMTCT services? 310. How did the project create and strengthen intra-health facility linkages to ensure high quality of services? (How did the project engage providers across sectors at the health facility to integrate information on health records? Mention patient tracking system if not mentioned.) 311. What are your constraints in improving pediatric recruitment and retention in care? How did the project help to improve these constraints? 312. What was the most effective method found by the project to improve retention in PMTCT cascade (ANC, testing, HIV testing for mother and baby and mother and baby receiving ART)? 313. Which of the approaches were most effective in improving healthcare service utilization by both men and women? 314. What are some of the best practices and lessons from this project in improving recruitment and retention in PMTCT cascade (ANC, testing, HIV testing for mother and baby and mother and baby receiving ART)? 315. What challenges will health facilities face in adopting some of the best practices or approaches in improving recruitment and retention of mothers in PMTCT cascade, without technical support from CHASS? 316. Which activities aimed at improving PMTCT services are most feasible to transition to DPS/SDMAS with little or no technical support from CHASS? 317. Are there activities that have been transitioned to the DPS/SDMAS staff? If yes, what are the successes achieved or failures faced? 318. What are your recommendations for sustainability of the project’s activities and achievements? HCT 401. What do you consider as the main achievements of the CHASS project in this province/district in delivery of the HCT services (at the facility and in the community)? 402. What project activities or strategies worked well? Why? 403. What project activities or strategies did not work well? Why? 404. In your opinion, did the CHASS project adequately target the men and women? State some examples. 405. In your opinion, what do you think were the benefits of working with the following groups or tools in achieving project results? 38 Performance Evaluation of the CHASS Niassa Project Number Question a) Equipa Polivalentes b) CBOs 406. Do you think the activities performed by these groups can continue without CHASS Niassa financial support? Why? Why not? 407. In your opinion, what are the key challenges that the project faced in implementing its activities or strategies? 408. In your opinion, what project activities or approaches could have been done better? 409. How did the project promote the integration of primary health care or maternal and child health with HIV/AIDS services? What approach worked, and what did not? 410. How did the project create and strengthen linkages between facilities and communities to promote increased service uptake of HCT services? 411. To what extent was the project successful in creating knowledge, demand and access to HCT services among men? 412. How did the project create and strengthen intra-health facility linkages to ensure high quality of services? (How did the project engage providers across sectors at the health facility to integrate information on health records?) 413. In your opinion, were there constraints in linking HIV-positive men and women tested in C-HCT to the health facility? How were they overcome? 414. Which of the community-level approaches were effective in improving healthcare service utilization by both men and women? 415. What challenges will health facilities face in adopting some of the best practices or approaches in improving recruitment of HIV positive men, women and children, without technical support from CHASS? 416. Which activities aimed at improving HCT coverage are most feasible to transition to DPS/SDMAS with little or no technical support from CHASS? 417. Are there activities that have been transitioned to the DPS/SDMAS? If yes, what are the successes achieved or failures faced? 418. What are your recommendations for sustainability of the project’s activities and achievements? HSS 501. (FOR SDMAS ONLY) What activities were targeted at improving the following: a) Planning system and capacity b) Financial management c) Supply and logistics at health facility (discuss lab results, stock-outs, storage infrastructure, etc.) d) Information systems (discuss Sistema de Informação de Medicamentos e Artigos Médicos-Pharmaceuticals and Medical Commodities Information System, health facility records) e) Monitoring and evaluation system f) Human resource development and management 502. Which of the activities worked well? 503. What was the value added by the various health system strengthening approaches? (Please talk about clinical tools if not mentioned.) 504. Which of the activities did not work well? Why? 505. Where has the most progress been seen in strengthening systems (e.g., planning, financial management, supply and logistics, information systems, monitoring and evaluation)? 506. Did the graduation path system produce improvements in service management at district and/or health facilities, if so, what improvements were noted in clinical services, management of information systems, monitoring and evaluation, etc.? 507. What are the associated challenges, and how can this graduation path system be improved? 508. To what extent did patient flow improve over the life of the project? 509. To what extent did patient waiting times improve over the life of the project? 510. How have leadership roles and decision-making structure improved at health facilities over the project life? Please explain. 511. Describe the current strategies and processes defined to ensure continuity of functions in the event of staff turnover or other unanticipated disruptions. Performance Evaluation of the CHASS Niassa Project 39 Number Question 512. To what extent did CHASS build the health facility data management capacity with regard to HIV/AIDS registry data? 513. How has the correct completion of different registries/forms improved at health facilities? 514. How has filing of records or registries at facilities improved over time? 515. Are health facilities analyzing data and using it in their planning? How has CHASS project contributed to their capacity to do this? 516. To what extent has the review and use of records across different sections in health facilities been improved? 517. How has the sharing of patients records within the facility improved? 518. Has transportation and management of CD4 and PCR results improved over the project life? If so, how? 519. How have the record keeping or lab information systems improved over the project life? 520. How has the stock-out of clinic consumables improved over the project life? 521. How have the record keeping or pharmacy information systems improved over the project life? 522. [IF NOT MENTIONED, ASK] How have the CHASS sub-agreements with training centers helped to improve human resource capacities and the number of health workers at district health facilities? (Discuss gap funding if not mentioned.) 523. What are the benefits of working with sub-agreements? 524. What are the challenges of working with sub-agreements? 525. How well did the CHASS program management structure support or facilitate HIV/AIDS response programming of this province/district? 526. Please share with us some lessons learned/best practices that can be scaled up in the near future? 527. What recommendations do you have for strengthening the organizational/institutional structures of DPS and DDS to perform their functions more effectively? 528. What measures should the district put in place in order to sustain what has been put in place by CHASS project? GBV (Gender-based Violence) 601. What are your perceptions about gender issues in your province, district? 602. What do you consider as the main achievements of CHASS project in this province/district relating to GBV issues? 603. What are the key challenges that the project faced in implementing some of its GBV activities or strategies? 604. What are your recommendations for sustainability of what the project has achieved in GBV? 605. To what extent has health workers’ knowledge and utilization of GBV services increased over the life of the project? 606. To what extent has community’s knowledge and utilization of GBV services increased over the life of the project? 607. To what extent has the project been able to create and strengthen linkages between facilities and communities to allow for increased service uptake of GBV services? (If not mentioned, ask about cabinet de atendimento de vitimas de violencia) 608. What constraints did the project activities face in linking GBV victims at the health facility to other services (If not mentioned, ask about cabinet de atendimento de vitimas de violencia)? How was it overcome? 609. What challenges will health facilities face in adopting some of the best practices or approaches in improving GBV services, without technical support from CHASS? 610. What are your recommendations for sustainability of the project’s activities and achievements? 40 Performance Evaluation of the CHASS Niassa Project Key Informant Interview Guide for Health Facility Clinical Staff Date: ___________________ Province/District _________________________ HF: ________________ Name of respondent: ___________________________________________ _______________________ Designation: __________________________________________________________________________ Number Question 101 What do you consider as the main achievements of the CHASS project in this health facility? Treatment and care 201 What do you consider as the main achievements of the CHASS project in this health facility in the delivery of HIV/AIDS care and treatment services? Ask the respondent to discuss pediatric care if not mentioned. 201b In your opinion, what do you consider as the main achievements of the CHASS project in this health facility in delivery of the prevention of mother-to-child transmission services? 202 What project activities or strategies worked well? Why? 203 What project activities or strategies did not work well? Why? 204 In your opinion, did the CHASS project adequately target men and women? State some examples. 205 In your opinion, what do you think were the benefits of working with the following groups in improving coverage and quality of HIV care and treatment services? State the extent of their relevancy. 205a Equipa Polivalentes 205b Equipa de Apoio 205c OBCs 205d GAACs 205e Comissões de adesão 205f Gestores de processo 205g Grupos de MpM 206 Do you think the activities performed by these groups can continue without financial support from CHASS? 207 What were the key challenges that the health facility faced in implementing some of its activities or strategies? 208 In your opinion, what project activities or approaches could have been done better? 209 How did the project promote the integration of primary health care or maternal and child health with HIV/AIDS services in this facility? What approach worked, and what did not? 210 How did the project create and strengthen linkages between your health facilities and communities to promote increased service uptake of HIV/AIDS care and treatment services? 211 To what extent was the project successful in creating knowledge, demand and access to HIV care and treatment among men? 212 How did the project help you to create and strengthen intra-health facility linkages to ensure high quality of services? How did the project engage providers across sectors at the health facility to integrate information on health records? 213 What constraints does the health facility face in improving retention of patients in pre-ART and ART? 214 What constraints did the health facility face in improving pediatric recruitment and retention in care? 215 What was the most effective method found by the health facility to improve retention in pre-ART care, ART or PMTCT? 216 What are your recommendations for improving linkages and referral systems in the future? 217 Which activities aimed at improving HIV/AIDS care and treatment services are most feasible to transition to this health facility with little or no technical support from CHASS? 217b Which activities aimed at improving PMTCT services are most feasible to transition to this health facility with little or no technical support from CHASS? 218 What are some of the best practices and lessons learned from this project in improving recruitment and retention in health care services? 219 What challenges will you, at this health facility, face in adopting some of the best practices or approaches in improving recruitment and retention of patients men, women and children in care and treatment, without technical support from CHASS? Performance Evaluation of the CHASS Niassa Project 41 Number Question PMTCT/MCH Nurses 301 In your opinion, what do you consider as the main achievements of the CHASS project in this health facility in delivery of the prevention of mother-to-child transmission services? 302 What activities or strategies worked well at this health facility in improving recruitment and retention in PMTCT services or pediatric care services? Why? 303 What activities or strategies did not work well at this health facility? Why? 304 In your opinion, what do you think were the benefits of working with the following groups in improving coverage and quality of PMTCT services or pediatric care services? State the extent of their relevancy. 304a Equipa Polivalentes 304b Equipa de Apoio 304c OBCs 304d GAACs 304e Comissões de adesão 304f Gestores de processo 304g Grupos de MpM 305 Do you think the activities performed by these groups can continue without CHASS Niassa financial support? Why? Why not? 306 What are the key challenges that the health facility faced in implementing some of its activities or strategies? Discuss recruitment and retention in PMTCT services or pediatric care services. 307 In your opinion, what activities or approaches could have been done better at this health facility? 308 How did the project promote the integration of primary health care or maternal and child health with HIV/AIDS services at this health facility? What approach worked, and what did not? 309 How did the project create and strengthen linkages between facilities and communities to promote increased service uptake of PMTCT services? 309b How did the project create and strengthen linkages between facilities and communities to promote increased service uptake of pediatric care services, including HIV testing? 310 How did the project help you to create and strengthen intra-health facility linkages to ensure high quality of services? How did the project engage providers across sectors at the health facility to integrate information on health records? 311 What were the constraints faced by the health facility in improving pediatric recruitment and retention in care? 312 What was the most effective method found by the health facility to improve retention in PMTCT cascade ANC, testing, HIV testing for mother and baby and mother and baby receiving ART? 313 What are some of the best practices and lessons from this health facility in improving recruitment and retention in PMTCT cascade? 314 What challenges will you face at this health facility in adopting some of the best practices or approaches in improving recruitment and retention of mothers in PMTCT cascade, without technical support from CHASS? 315 Which activities aimed at improving PMTCT services are most feasible to transition to this health facility with little or no technical support from CHASS? 315b Which activities aimed at improving pediatric care services are most feasible to transition to this health facility with little or no technical support from CHASS? 316 What are your recommendations for sustainability of what the project has achieved? HCT 401 In your opinion, what are the main achievements of the CHASS project in this health facility in delivery of the HCT services at the facility and in the community? 402 What activities or strategies worked well at this health facility? Why? 403 What activities or strategies did not work well at this health facility? Why? 412 To what extent was the project successful in creating knowledge, demand and access to HCT services among men? 414 What constraints did the project’s activities face in linking HIV-positive men and women tested in C-HCT to the health facility? How were they overcome? 417 Which activities aimed at improving HCT coverage are most feasible to transition to the health facility with little or no technical support from CHASS? 42 Performance Evaluation of the CHASS Niassa Project Number Question 418 What are your recommendations for sustainability of what the project has achieved? Health Systems Strengthening 501 Where has the most progress been seen in strengthening systems, e.g., planning, financial management, supply and logistics, information systems, monitoring and evaluation? 502a To what extent has the patient flow at your health facility improved over the life of the project? 502b To what extent have the patient waiting times at your health facility improved over the life of the project? 503 Have the leadership roles and the decision-making structure improved at this facility over the project’s life? Please explain. 504 Describe the current strategies and processes defined to ensure continuity of functions in the event of staff turnover or other unanticipated disruptions. 505 To what extent did CHASS build the health facility data management capacity with regard to HIV/AIDS registry data? 504 How has the correct completion of different registries/forms improved at the health facility? 505 How has the filing of the records or registries at this facility improved over time? 504 Do you analyze data and use it in your planning? How has CHASS project contributed to your capacity to do this? 505 How has CHASS project contributed to your capacity to do this? 504 Are health facilities analyzing the data and use it in your planning? How has CHASS project contributed to your capacity to do this? 505 To what extent has the review and use of records across different sections in your health facility been improved? 504 Did the graduation path system produce improvements in service management in this facility, if so, what improvements were noted in clinical services, management of information systems, monitoring and evaluation, etc.? 505 How has the supply chain improved over the life of the project? 504 How is the sharing of the patients’ records within the facility? 505 How has the transportation and management of CD4 and PCR results improved over the project’s life? 504 How have the record keeping or lab information systems improved over the project’s life? 505 How has the stock-out of clinic consumables improved over the project’s life? 504 How have the record keeping or pharmacy information systems improved over the project’s life? Gender-based Violence 601 What training have you received in gender issues? 602 What do you consider as the main achievements of CHASS project in this health facility relating to GBV issues? 602a What forms of GBV do you handle at this facility? 603 What are the key challenges that the health facility faced in implementing GBV services? 604 To what extent has health workers’ knowledge and utilization of GBV services increased over the life of the project? 605 To what extent has the project been able to create and strengthen linkages between facilities and communities to allow for increased service uptake of GBV services? 606 What constraints did the health facility face in linking GBV victims in the community to the other services? How were they overcome? 607 What challenges will health facilities face in adopting some of the best practices or approaches in improving GBV services, without technical support from CHASS? 608 What are your recommendations for sustainability of what the project has achieved? Performance Evaluation of the CHASS Niassa Project 43 Key Informant Interview Guide for Implementing Partner Staff Date: ___________________________ Province/District _________________________ Name of respondent: ___________________________________________ Gender: M/F Designation: _______________________________________ Number Question General 101. What do you consider as the main achievements of the CHASS project in this province/district? Was there a particular period when achievements were most accelerated (please explain)? 102. Are there differences in achievements across the districts? If yes, what could explain these differences? HIV/AIDS Care and Treatment 223. What do you consider as the main achievements of the CHASS project in this province/district in the delivery of HIV/AIDS care and treatment services? Ask the respondent to discuss pediatric care (if not mentioned). 224. What project activities or strategies worked well? Why? 225. What project activities or strategies did not work well? Why? 226. In your opinion, did the CHASS project adequately target men and women? State some examples. 227. In your opinion, what do you think were the benefits of working with the following groups in improving coverage and quality of HIV care and treatment services? State the extent of their relevancy. a) Equipa Polivalentes b) Equipa de Apoio c) CBOs d) GAACs e) Adherence Committees f) Case managers 228. Please comment on the sustainability of the activities performed by these groups by the district and health facilities? 229. What were the key challenges that the project faced in implementing some of its activities or strategies? 230. In your opinion, what project activities or approaches could have been done better? State how. 231. How did the project promote the integration of primary health care or maternal and child health with HIV/AIDS services? What approach worked, and what did not? 232. How did the project create and strengthen linkages between facilities and communities to promote increased service uptake of HIV/AIDS care and treatment services? 233. To what extent was the project successful in creating knowledge, demand and access to HIV care and treatment services by men? 234. How did the project create and strengthen intra-health facility linkages to ensure high quality of services? (How did the project engage providers across sectors at the health facility to integrate information on health records?) 235. What constraints did the project activities face in improving retention of patients in pre-ART and ART? 236. What constraints did project activities face in improving pediatric recruitment and retention in care? 237. What was the most effective method found by the project to improve retention? 238. What are your recommendations for improving linkages and referral systems in the future? 239. Which of the community-level approaches were most effective in improving use of healthcare services by men and women? 240. Which activities aimed at improving HIV/AIDS care and treatment services are most feasible to transition to the government system with little or no technical support from CHASS? 241. Are there activities that have been transitioned to the government staff? If yes, what are the successes achieved or failures faced? 242. What are some of the best practices and lessons learned from this project in improving recruitment and retention in health care services? 44 Performance Evaluation of the CHASS Niassa Project 243. What challenges will health facilities face in adopting some of the best practices or approaches in improving recruitment and retention of patients (men, women and children) in care and treatment, without technical support from CHASS? 244. What are your recommendations for sustainability of what the project has achieved? PMTCT 319. What do you consider as the main achievements of the CHASS project in this province/district in delivery of the prevention of mother-to-child transmission services? 320. What project activities or strategies worked well? Why? 321. What project activities or strategies did not work well? Why? 322. In your opinion, what do you think were the benefits of working with the following groups in improving coverage and quality of PMTCT services? State the extent of their relevancy. a) Equipa Polivalentes b) Equipa de Apoio c) CBOs d) GAACs e) Adherence Committees f) Case managers g) MpM or M2M groups 323. Please comment on the sustainability of the activities performed by these groups by the district and health facilities. 324. What are the key challenges that the project faced in implementing some of its activities or strategies? 325. In your opinion, what project activities or approaches could have been done better? 326. State how. 327. How did the project promote the integration of primary health care or maternal and child health with HIV/AIDS services? What approach worked, and what did not? 328. How did the project create and strengthen linkages between facilities and communities to promote increased service uptake of PMTCT services? 329. How did the project create and strengthen intra-health facility linkages to ensure high quality of services? (How did the project engage providers across sectors at the health facility to integrate information on health records?) 330. What constraints did project activities face in improving pediatric recruitment and retention in care? 331. What was the most effective method found by the project to improve retention in PMTCT cascade? 332. Which of the community-level approaches were most effective in improving healthcare service utilization by both men and women? 333. What are some of the best practices and lessons from this project in improving recruitment and retention in PMTCT cascade? 334. What challenges will health facilities face in adopting some of the best practices or approaches in improving recruitment and retention of mothers in PMTCT cascade, without technical support from CHASS? 335. Which activities aimed at improving PMTCT services are most feasible to transition to the government system with little or no technical support from CHASS? 336. Are there activities that have been transitioned to the government staff? If yes, what are the successes achieved or failures faced? 337. What are your recommendations for sustainability of what the project has achieved? HCT 419. What do you consider as the main achievements of the CHASS project in this province/district in delivery of the HCT services (at the facility and in the community)? 420. What project activities or strategies worked well? Why? 421. What project activities or strategies did not work well? Why? 422. In your opinion, did the CHASS project adequately target men and women? State some examples. 423. In your opinion, what do you think were the benefits of working with the following groups or tools in achieving project results? a) Equipa Polivalentes Performance Evaluation of the CHASS Niassa Project 45 b) CBOs 424. Please comment on the sustainability of the activities performed by these groups by the district and health facilities? 425. What are the key challenges that the project faced in implementing some of its activities or strategies? 426. In your opinion, what project activities or approaches could have been done better? 427. State how. 428. How did the project promote the integration of primary health care or maternal and child health with HIV/AIDS services? What approach worked, and what did not? 429. How did the project create and strengthen linkages between facilities and communities to promote increased service uptake of HCT services? 430. To what extent was the project successful in creating knowledge, demand and access to HCT services by men? 431. How did the project create and strengthen intra-health facility linkages to ensure high quality of services? (How did the project engage providers across sectors at the health facility to integrate information on health records?) 432. What constraints did the project activities face in linking HIV-positive men and women tested in C-HCT to the health facility? How were they overcome? 433. Which of the community-level approaches were effective in improving healthcare service utilization by both men and women? 434. What challenges will health facilities face in adopting some of the best practices or approaches in improving recruitment of HIV-positive men, women and children, without technical support from CHASS? 435. Which activities aimed at improving HCT coverage are most feasible to transition to the government system with little or no technical support from CHASS? 436. Are there activities that have been transitioned to the government staff? If yes, what are the successes achieved or failures faced? 437. What are your recommendations for sustainability of what the project has achieved? HSS 529. What activities were targeted at improving the following: a) Planning system and capacity b) Financial management c) Supply and logistics at health facility (discuss lab results, stock-outs, storage infrastructure, etc.) d) Information systems (discuss Sistema de Informação de Medicamentos e Artigos Médicos-Pharmaceuticals and Medical Commodities Information System, health facility records) e) Monitoring and evaluation system f) Human resource development and management 530. Which of the activities worked well? 531. What was the value added by the various health system strengthening approaches? 532. Which of the activities did not work well? Why? 533. Where has the most progress been seen in strengthening systems (e.g., planning, financial management, supply and logistics, information systems, monitoring and evaluation)? 534. To what extent did patient flow improve over the life of the project? 535. To what extent did patient waiting times improve over the life of the project? 536. How have leadership roles and decision-making structure improved at health facilities over the project life? Please explain. 537. Describe the current strategies and processes defined to ensure continuity of functions in the event of staff turnover or other unanticipated disruptions. 538. To what extent did CHASS build the health facility data management capacity with regard to HIV/AIDS registry data? 539. How has the correct completion of different registries/forms improved at health facilities? 540. How has filing of records or registries at facilities improved over time? 541. Are health facilities analyzing data and using it in their planning? How has the CHASS project contributed to their capacity to do this? 542. To what extent has the review and use of records across different sections in health facilities been improved? 543. Did the graduation path system produce improvements in service management at district and/or health facilities? If so, what improvements were noted in clinical services, management of information systems, monitoring and evaluation, etc.? 46 Performance Evaluation of the CHASS Niassa Project 544. How has the supply chain improved over the life of the project? 545. How has the sharing of patients’ records within the facility improved? 546. Has transportation and management of CD4 and PCR results improved over the project’s life? If so how? 547. How have the record keeping or lab information systems improved over the project’s life? 548. How has the stock-out of clinic consumables improved over the project’s life? 549. How has the record keeping or pharmacy information systems improved over the project’s life? 550. [If not mentioned, ask] How have the CHASS sub-agreements with training centers helped to improve human resource capacities and the number of health workers at district health facilities? (Discuss gap funding if not mentioned) 551. What are the benefits of working with sub-agreements? 552. What are the challenges of working with sub-agreements? 553. What measures should the district put in place in order to sustain what has been put in place by the CHASS project? Gender-based Violence 554. What training have you received in gender issues? 555. What do you consider as the main achievements of the CHASS project in health facilities relating to GBV issues? 556. What are the key challenges that the health facilities faced in implementing GBV services? 557. To what extent has health workers’ knowledge and utilization of GBV services increased over the life of the project? 558. To what extent has the project been able to create and strengthen linkages between facilities and communities to allow for increased service uptake of GBV services? 559. What constraints did the project activities face in linking GBV victims in the community to the health facility? How were they overcome? 560. What challenges will health facilities face in adopting some of the best practices or approaches in improving GBV services, without technical support from CHASS? 561. What are your recommendations for sustainability of what the project has achieved? Key Informant Interview Guide for Participating CBOs’ Staff Date: ___________________________ Province/District _________________________ Name of respondent: ___________________________________________ Gender: M F Designation: _______________________________________ 1. What do you consider as the main achievements of the CHASS project in this district? What has not been achieved? 2. Specifically, comment on the role of your organization in: (a) Improving the demand and knowledge about health services in the health facilities (b) Improving retention in HIV/AIDS care services (c) GBV related services 3. How sustainable are your groups’ activities without the technical support from CHASS? 4. What can you tell us about the case managers, GAACs and the Adherence Committees? How can groups work better? 5. Are there some key challenges that members of GAACs or adherence groups face in accessing or staying on treatment? 6. To what extent has the project been able to create and strengthen linkages between facilities and communities to allow for increased service uptake, specifically in the areas of: a. Community-based counseling and testing to treatment (for both men and women)____________________ b. Retention rates in pre-ART and ART patients (through the use of adherence groups, active case-finding, and other community groups); and in PMTCT cascade_________________________________________ c. Knowledge, demand and access of services by men and women (PEP) __________________________ d. Inter-facility referral and linkages ______________________________________________________ e. Linkage and referral of HIV-positive men and women to HIV/AIDS care ________________________ 7. What are the challenges associated with C-HCT? How can C-HCT be improved or sustained? _ 8. What constraints have your activities faced in improving retention of patients in pre-ART and ART? Performance Evaluation of the CHASS Niassa Project 47 9. What constraints have your activities faced in improving pediatric recruitment and retention in care? 10. What has been the most effective method found to improve retention? 11. Provide your overall rating on the improvement of linkages and referrals since CHASS began. What are your recommendations for the future? 12. What are some of the gender issues in your community? Are you aware of GBV in this community? 13. To what extent has knowledge (community and health worker) and utilization of GBV services increased over the life of the project in your community members and among members of your team? 14. What kind of information, education and communication materials do you have on GBV? Who is in the target group? How do you identify them? 15. What are the challenges of implementing GBV services? How can they be overcome? 16. How does the community assist those who have suffered from GBV? In your opinion, what can be done to reduce GBV in your community? 17. What are your thoughts about the benefits and challenges of the project’s model of working with you through sub￾agreements with CBOs? 18. Please share with us some lessons learned/best practices of the project in working through sub-agreements with CBOs that should be maintained or scaled-up in the near future? 19. How is your CBO prepared to sustain what has been put in place by CHASS? Key Informant Interview Guide for MOH and USAID Staff Date: ___________________________ Province/District _________________________ Name of respondent: ___________________________________________ Gender: M F Designation: _______________________________________ Number Question 245. What do you consider as the main achievements of the CHASS project in the province of Niassa? 246. Are there differences in achievements across the provinces? If yes, what could explain these differences? 247. In your opinion, what were the key challenges that the project faced in implementing some of its activities or strategies? 248. In your opinion, what project activities or approaches could have been done better by the CHASS project teams? State how. 249. Where has the most progress been seen in strengthening systems (e.g., planning, financial management, supply and logistics, information systems, monitoring and evaluation)? 250. What are the benefits of working with sub-agreements? 251. What are the challenges of working with sub-agreements? 252. How well did the CHASS program management structures support or facilitate HIV/AIDS response programming in the different provinces? 253. What are some of the best practices and lessons learned in this project that can be scaled-up in the near future? 254. What recommendations do you have for strengthening the organizational/institutional structures of DPS and DDS to perform their functions more effectively? GBV (Gender-based Violence) [Gender Focal Points] 611. What do you consider as the main achievements of CHASS project relating to GBV issues in the different provinces? 612. What are the key challenges that the project faced in implementing some of its GBV activities or strategies? 613. What are your recommendations for sustainability of what the project has achieved in GBV? 48 Performance Evaluation of the CHASS Niassa Project ANNEX III. LIST OF INTERVIEWEES Maputo Province Nuno Miguel Militar Antonio Oficial Prog, FHI/Sofala Timotio Mario Filite Oficial FHI/Sofala MOH Dr. Alene Couto, HIV Chief USAID Performance Evaluation of the CHASS Niassa Project 49 ANNEX IV. SOURCES OF INFORMATION List of Documents Reviewed:  Clinical HIV/AIDS System Strengthening Request for Applications, 2009  CHASS Niassa annual work plans and quarterly and annual reports (2011, 2012, 2013, 2014)  Site visit reports  Public Financial Management Risk Assessment Framework reports for the four provinces  MOH health sector strategy  MOH HIV Acceleration Plan  Data from PEPFAR annual and semiannual reports—this includes both community and clinical data (2011, 2012, 2013, 2014)  Analysis conducted based on PEPFAR annual and semiannual reports (2011, 2012, 2013, 2014)  Partner- and USAID-conducted data quality assessments 50 Performance Evaluation of the CHASS Niassa Project ANNEX V. SUMMARY OF QUANTITATIVE DATA Indicator 2011 2012 2013 2014 % of 2014 Target achieved† Number currently enrolled on ART 3,316 (2,699) 7,383 (7,622) 10,768 (7,189) 13,334 (11,012) 121% Number newly enrolled on ART 861 (1,615) 2,812 (5,103) 3,967 (2,518) 5,261 (2,919) 180% Current on ART (men) 908 2,627 3,213 3,507 Percentage of current on ART who are men 27% 36% 30% 26% Percentage of current on ART who are children 7% 6% 11% 9% Percentage of newly enrolled (all) 26% 38% 37% 39% Percentage of newly enrolled children 9% 12% 9% 7% Number of women registered at ANC* 48,874 61,782 60,616 145% Percent of women receiving HIV test and results in a PMTCT ANC setting (first test) * 42,404 50,937 53,693 113% Percent of pregnant women with known HIV status (newly tested + known positive at ANC entry) * 92% 83% 90% Number of HIV-positive pregnant women who received ARVs to reduce risk of MTCT* 1,965 2,003 2,259 151% Percent of HIV-positive pregnant women who received ARVs to reduce risk of MTCT* 72% 56% 82% Number of HIV-positive pregnant women in ANC who have initiated Cotrimoxizole 1,426 1,192 1,641 Percent of HIV-positive pregnant women in ANC who have initiated Cotrimoxazole* 52% 33% 59% Percent of infants born to HIV-positive women who received an HIV test within 12 months of birth* 26% 24% 53% Percent of HIV test results for infants born to HIV-positive women who received an HIV test within 12 months of birth that are positive* 15% 7% 4% Number of partners of women who are HIV tested in a PMTCT setting* 7,651 10,925 20,551 133% Percent of partners of women who are HIV tested in a PMTCT setting* 16% 18% 34% Number of HIV-positive patients receiving a minimum of one clinical service 11,874 21,123 18,508 25,346 139% Percent of HIV-positive patients receiving a minimum of one clinical service (male) 28% 37% 35% 31% Percent of total HIV-positive patients receiving cotrimoxazole prophylaxis in last visit 37% 41% 55% 59% Percent of HIV-positive patients screened for TB at last visit in HIV care 0% 51% 68% 60% † Indicators without targets have this column left blank. *No data were available for most of the PMTCT-related indicators in FY 2011 and are left blank in the table. Performance Evaluation of the CHASS Niassa Project 51 ANNEX VI. EVALUATION TEAM MEMBERS The evaluation team was external to the CHASS project and consists of members of AGEMA Consultoria Lda. Dexis Consulting Group’s Global Health Program Cycle Improvement (GH Pro) Project is providing technical assistance to the evaluation team. Peter Symon Wandiembe, PhD (AGEMA Consultoria Lda): Team Leader Rosemary Barber-Madden, PhD (GH Pro): Senior Technical Specialist Esther Kazilimani-Pale, MPH, (AGEMA Consultoria Lda): Senior Evaluation Officer/Public Health and Gender Specialist Verona Parkinson, PhD (AGEMA Consultoria Lda): Evaluation Team Member and Project Director Jeffrey Kilama, MSc (AGEMA Consultoria Lda): Data Manager and Analyst Danubio Cumbane (AGEMA Consultoria Lda): Evaluation Team Member and Data Collector Three data collectors U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 2052