EVALUATION PERFORMANCE EVALUATION OF THE CLINICAL HIV/AIDS SYSTEM STRENGTHENING PROJECT IN SOFALA-MANICA-TETE PROVINCES December 30, 2015 This report was produced by AGEMA Consultoria Lda, for the United States Agency for International Development/Mozambique. Authors: Peter S. Wandiembe, PhD, Rosemary Barber-Madden, PhD, Esther Kazilimani-Pale, MPH, and Verona Parkinson, PhD Performance Evaluation of the CHASS-SMT Project iii PERFORMANCE EVALUATION OF THE CLINICAL HIV/AIDS SYSTEM STRENGTHENING PROJECT IN SOFALA￾MANICA-TETE PROVINCES December 30, 2015 Agema Consultoria Lda. submits this report to USAID/Mozambique as a deliverable under Contract No. 656-O-00-15-00048-00 for the Performance Evaluation of the Clinical HIV/AIDS System Strengthening Projects–Sofala, Manica, and Tete, implemented by Abt Associates Inc, with a sub-award to FHI 360. DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Performance Evaluation of the CHASS-SMT Project i CONTENTS ACRONYMS AND ABBREVIATIONS .......................................................................... iii EXECUTIVE SUMMARY .................................................................................................. v 1 EVALUATION PURPOSE AND QUESTIONS ...................................................... 1 1.1 EVALUATION PURPOSE .................................................................................................................... 1 1.2 EVALUATION QUESTIONS ............................................................................................................... 1 2 PROJECT BACKGROUND ....................................................................................... 3 2.1 GENERAL CONTEXT ........................................................................................................................ 3 2.2 OVERVIEW OF CHASS-SMT IMPLEMENTATION............................................................................... 3 3 EVALUATION METHODS AND LIMITATIONS .................................................. 5 3.1 EVALUATION METHODS .................................................................................................................. 5 3.2 SITE SELECTION AND DATA COLLECTION..................................................................................... 5 3.3 DATA ANALYSIS ............................................................................................................................... 6 3.4 STUDY LIMITATIONS ........................................................................................................................ 7 4 FINDINGS, CONCLUSIONS AND RECOMMENDATIONS ............................... 8 4.1 FINDINGS.......................................................................................................................................... 8 4.2 OVERALL CONCLUSIONS...............................................................................................................24 4.3 FUTURE DIRECTIONS .....................................................................................................................25 4.4 SUMMARY OF RECOMMENDATIONS .............................................................................................27 5 REFERENCES ............................................................................................................29 ANNEXES ........................................................................................................................31 ANNEX I: SCOPE OF WORK .......................................................................................33 ANNEX II: DATA COLLECTION INSTRUMENTS: KII, FGD ..................................44 ANNEX III: LIST OF INTERVIEWEES .........................................................................57 ANNEX IV: SOURCES OF INFORMATION ...............................................................60 ANNEX V: SUMMARY OF QUANTITATIVE DATA .................................................61 ANNEX VI: EVALUATION TEAM MEMBERS ............................................................65 ii Performance Evaluation of the CHASS-SMT Project LIST OF TABLES, FIGURES AND CHARTS Table 1: Selected districts and health facilities ............................................................................................................ 6 Table 2: Categories of key informant interview respondents ................................................................................ 6 Table 3: Summary of selected indicators to illustrate achievements in clinical services ................................ 11 Table 4: Recommendations for future programs ..................................................................................................... 27 Figure 1: Patients already on ART and newly enrolled ............................................................................................. 9 Figure 2: Key PMTCT indicators .................................................................................................................................. 10 Figure 3: Performance of systems and capacities at the districts within the provinces at different rounds of assessments (Source: Project WPY 2015) ............................................................................................................ 14 Figure 4: Retention and community linkage indicators ........................................................................................... 17 Chart 1: Clinical services weaknesses and recommendations .............................................................................. 10 Chart 2: HSS weaknesses and recommendations .................................................................................................... 15 Chart 3: HSS weaknesses and recommendations (2) ............................................................................................. 16 Chart 4: Challenges to community linkages .............................................................................................................. 18 Chart 5: Constraints to activities aimed at improving retention and recommendations .............................. 19 Chart 6: Challenges associated with improvement of health facility management .......................................... 21 Chart 7: Challenges to data management capacities at health facilities .............................................................. 22 Chart 8: Challenges and weaknesses in scaling-up GBV services ........................................................................ 23 Performance Evaluation of the CHASS-SMT Project iii ACRONYMS AND ABBREVIATIONS Acronym/ Abbreviation Definition (English) Definition (Portuguese) AIDS Acquired Immune Deficiency Syndrome Síndrome de Imunodeficiência Adquirida ANC Antenatal care Cuidado pré natal APEs Agentes Polivantes Elementer ART Antiretroviral treatment Tratamento Anti-retroviral ARV Antiretroviral Anti-retroviral CBO Community-based organization Organização Baseada na Comunidade CD4 Cluster of Differentiation 4 Cluster de Diferenciação 4 CHASS-SMT Clinical HIV/AIDS System Strengthening Project in Sofala, Manica and Tete DPAF Provincial Administration and Finance Department Departamento Provincial de Administracao e Financas DPS Provincial Health Directorate Direcção Provincial de Saúde EP Multidisciplinary team Equipa Polivalente EPTS Electronic patient tracking system Sistema Electrónico de Seguimento de Pacientes FILA ARV drug pick-up form Folha de Informação de Levantamentos de ARV GAACs Community Support and Adherence Groups Grupo de Apoio e Adesão da Comunidade GBV Gender-based violence Violência Baseada no Género HAI Health Alliance International HCT HIV counseling and testing Aconselhamento e Testagem em Saúde HF Health facility Unidade de saúde HIV Human Immunodeficiency Virus Vírus da Imunodeficiência Humana HR Human resources Recursos Humanos M2M Mother-to-Mother Groups Mãe para Mãe MCH Maternal and child health Saúde Materno-Infantil MOH Ministry of Health Ministério da Saúde MSF Médecins Sans Frontières (Doctors without Borders) Médicos sem Fronteiras PCC Community Care Project Projecto de Cuidados Comunitários PCR Polymerase Chain Reaction Reacção em Cadeia da Polimerase PEP Post-exposure prophylaxis Profilaxia Pós-Exposição PEPFAR U.S. President’s Emergency Plan for AIDS Relief PIMA CD4-analyzing machine Máquina para Analisar as Amostras de CD4 PLHIV People living with HIV/AIDS Pessoas que Vivem com o HIV/SIDA iv Performance Evaluation of the CHASS-SMT Project Acronym/ Abbreviation Definition (English) Definition (Portuguese) PMTCT Prevention of mother-to-child transmission Prevenção da Transmissão Vertical QA Quality assurance Controlo da qualidade QI Quality improvement Avaliação da Qualidade de Dados SA Sub-agreement SDSMAS District Health Directorate Serviços Distritais de Saúde, Mulher e Acção Social TB Tuberculosis Tuberculose Performance Evaluation of the CHASS-SMT Project v EXECUTIVE SUMMARY EVALUATION PURPOSE The Clinical HIV/AIDS System Strengthening Program in Sofala, Manica and Tete provinces (CHASS￾SMT) was a five-year project funded by the United States Agency for International Development. It was implemented by Abt Associates Inc. with an overall goal of improving public sector HIV clinical services in the three provinces within a strengthened comprehensive primary health care system. The purpose of this performance evaluation of the CHASS-SMT was to determine how the project’s activities were performed relative to their objectives. The evaluation was to provide an objective view of progress towards the expected results. The main objectives were to (a) assess CHASS-SMT achievements, emphasizing project-level results; (b) identify implementation successes, as well as any internal and external constraints that hindered the implementation of planned activities; and (c) propose recommendations for future directions of the CHASS project. KEY EVALUATION QUESTIONS The evaluation was guided by 10 questions, 1 with the following four key questions: 1) What are the strengths and weaknesses of the activities as seen by the stakeholders, and how can these weaknesses be improved? 2) What constraints have the activities faced in improving retention of patients in antiretroviral treatment (ART) and pre-ART programs? 3) To what extent has knowledge (community and health worker) and utilization of gender-based violence (GBV) services increased over the life of the project? 4) What are the benefits and challenges of the activities’ model of working with government (at provincial and district levels) through sub-agreements? PROJECT BACKGROUND The CHASS Program was designed to address the HIV situation in Mozambique and focused on three components: (a) Improving service quality in HIV prevention, care and treatment; (b) Enhancing program linkages and integration to provide a continuum of accessible services; and (c) Creating stronger and more sustainable Mozambican systems and institutions. In Sofala, Manica and Tete, CHASS was implemented by Abt Associates Inc. from November 2010 to July 2015. EVALUATION DESIGN AND METHODS The evaluation data were collected September 4–28, 2015. The evaluation was based on a non￾experimental design using a mixed-methods approach. The evaluation team reviewed the existing quantitative project data and documentation and also conducted field key informant interviews with staff from the project (18), 15 health facilities (HFs), health and social welfare departments at the three provinces and eight districts (89), the Ministry of Health (MOH) (3), USAID/Mozambique (5) and community-based organizations (CBOs) supporting CHASS-SMT (9). The evaluation used a multi￾channel data collection approach, using mobile technology when possible and paper-based methods when handheld computers/tablets were not appropriate. All of the interviews were recorded on tablets. Some of the evaluation team members handwrote their field notes directly onto the tablets using a stylus beta enhanced Open Data Kit (ODK) program. The qualitative data were analyzed using a 1The ten questions are listed in Section 1 of this report: Evaluation Purpose and Evaluation Questions. vi Performance Evaluation of the CHASS-SMT Project qualitative data analysis matrix. Information from this analysis was triangulated with the available quantitative data from CHASS project documents. EVALUATION LIMITATIONS The limitations of the evaluation are as follows: Firstly, in many cases, the evaluation team was unable to ascertain consistent factors responsible for the project’s success and weaknesses as the project evolved over time. Therefore, some of the findings reported are from a summative perspective. Secondly, the findings reported here are responses provided by the interviewees to the evaluation team. The likelihood of interviewee bias and recall bias cannot be ruled out. FINDINGS, CONCLUSIONS AND RECOMMENDATIONS The findings for each evaluation question are summarized below: 1) Project achievements and challenges:  HSS activities: The introduction of the graduation path to assess for weaknesses and develop action plans for Provincial Health Directorate (DPS) systems and capacities was successful. This led to improvement in the management capacities from below 50 percent to 70 percent, according to MOH management standards. Manica province was the most improved. At the HF level, the project revised a tool used for supportive supervision and converted it to an electronic clinical monitoring and tutoring tool aimed at quality improvement. This helped in systematic identification of training needs, on-the-spot targeted tutoring, and development of detailed action plans. The project helped reduce the human resource shortage through a pre-service training of more than 400 health workers, who were deployed in the districts. Technical skills of existing staff were improved through on-the-job and in-service training, and maternal and child health (MCH) nurses were trained to prescribe antiretrovirals (ARVs). The project lacked a well-planned, appropriate exit strategy. Tools and methodologies such as the graduation path and the clinical monitoring tool that were key in HSS have not been institutionalized in any DPS/District Health Directorate (SDSMAS), and hence their experiential learning was limited. Further, most of the project’s operational costs were managed by the project team. The sustainability of HSS is subject to staff turnover and transfers; high transfer rates reduce the capacity at the SDSMAS and HFs. The project also did not put a mechanism in place for SDSMAS/health staff to train new staff. Limited operations research was completed, and quality improvement approaches were not sufficiently promoted.  Clinical service delivery activities: HIV/AIDS clinical services were improved and expanded to new HFs through staff training and mentoring, minor facility renovations, improvement of auxiliary services, and creation of demand and active referrals from the communities through activists, in coordination with CBOs. The project contributed to structuring and organization of HIV services, data review, expanded number of ART sites, and scale-up of Option B+, leading to more than a threefold increase in HIV counseling and testing (HCT), ART and Option B+ coverage. The table below summarizes key indicators. These impressive improvements are threatened by the insufficient human resources and high rates of staff transfers. Provincial, district and HF-level stakeholders reported an insufficient number of staff to manage improved activities, meet increased demands and maintain high-quality services. For example, MCH nurses in some HFs reported attending to more than 60 patients per day. One of the notable consequences of this is the little time taken to counsel clients. Inadequate counseling was noted as a barrier to improvements in retention rates in care and treatment, and to recruitment of infants into care. Clients who do not understand the importance of staying in care or Performance Evaluation of the CHASS-SMT Project vii early infant diagnosis are less likely to return for scheduled clinic visits. In addition to insufficient staff, the infrastructural design and small physical spaces at some HFs limit the privacy of clinical consultations and pose a barrier to adequate counseling. Table 1: Key indicators for HIV care, ART and prevention of mother-to-child transmission (PMTCT) programs Indicator FY 2011 FY 2012 FY 2013 FY 2014 Percent of 2014 Target Achieved Number of patients currently enrolled on ART 57,894 (48,106) 61,016 (70,012) 102,198 (77,039) 155,588 (107,879) 144% Number of patients newly enrolled on ART 13,955 (19,558) 23,922 (22,908) 32,481 (29,966) 47,675 (44,305) 107% Percent of patients currently enrolled on ART that are men 33% 31% 30% 29% No target Percent of patients currently enrolled on ART that are children 8% (N/A) 8% (N/A) 8% (11%) 8% (13%) Retention rate in 12-month ART cohort (Adult) 68% 65% 74% 72% 85% Percent of pregnant women with known HIV status (newly tested and known positive at antenatal care (ANC) entry)* 90% (90%) 92% (90%) 91% (90%) 101% Percent of HIV-positive pregnant women in ANC who have initiated Cotrimoxazole* 21% 33% 64% No target Percent of HIV-positive pregnant women who received ARVs to reduce the risk of MTCT 88% 86% 86% No target Percent of infants born to HIV-positive women who received HIV test within 12 months of birth* 45% 46% 52% No target Percent of HIV test results for infants born to HIV-positive women who received an HIV test within 12 months of birth that are positive* 11% 9% 7% No target *No data were available for most of the PMTCT-related indicators in FY 2011 and are left blank in the table. **Targets were not set for these indicators. 2) Most improved HSS component: The project strengthened systems and capacities of various components of the health system. The human resources (HR) component was identified by both DPS and SDSMAS key informants as the most significantly improved. The DPS and SDSMAS staff were trained on HR management and use of electronic HR information systems, and health workers were trained through mentoring and tutoring, on-the-job training and pre-service training. 3) Ready-to-transition project activity or component: Most routine project activities were part of MOH activities that are supposed to be executed by provincial and district technical teams. Whereas the project team led the implementation of these activities, they involved DPS and SDSMAS. Key project-specific activities that should be transitioned to the DPS and SDSMAS team leadership are the graduation path and clinical monitoring methodology and assessments. 4) Strengthening community linkages: The project moderately improved linkages between the HFs and communities in all 36 districts. This led to increased uptake of HCT, PMTCT care, ART and TB care services and slight improvements in retention of patients in pre-ART and ART services. By working with eight CBOs (three each in Sofala and Tete, and two in Manica), the project created new linkages between the HFs and the communities. Through CBOs, the project provided logistics to community activists for case-finding of pre-ART care and ART defaulters, community sensitization, mobilization and community-based HIV testing. The facility-based case managers hired by CBOs were instrumental in generating lists of defaulters, receiving referrals from the viii Performance Evaluation of the CHASS-SMT Project communities and communicating with health workers. Through provision of training and logistics to the adherence support groups (GAACs), people living with HIV (PLHIV) and Mother-to-Mother (M2M) activities, the project improved knowledge of services and improved retention in ART programs. Overall, across all three provinces, adult and pediatric retention rates increased from 65 percent and 64 percent in FY 2011 to 74 percent and 72 percent in FY 2014, respectively, though these are still below the 85 percent target. However, the community reach to men was still low. Less than 10 percent of male partners of pregnant women were tested in the PMTCT setting. Stakeholders cited various barriers to retention, including long distances to the facilities, poor quality of counseling due to insufficient number of HF staff and lack of privacy due to limited physical space, and time constraints, especially for working men. 5) Constraints to improving retention rates: Retention rates persisted below the 85 percent target across the provinces. A major retention constraint was the insufficient number of staff to provide counseling in some HFs and communities. Retention is a cycle that begins with counseling, and if counselors do not take time to counsel and answer patients’ questions so that they understand the need for regular clinical reviews or adherence to ARVs, retention becomes a problem. Other constraints (see point 6 below) included the long distances activists had to travel to reach defaulters; an insufficient number of activists to cover wide catchment areas; wrong addresses given by patients; patients moving out of the community during farming season; small physical space at HFs, limiting privacy of counseling sessions; and time constraints, especially for men. 6) Most effective method for improving retention: Key retention strategies included (1) active case-finding, where activists use a defaulters’ list generated by the HF case managers to search for defaulters in communities; (2) GAAC members picking up drugs for each other and providing psychosocial support; (3) community mobilization by activists, encouraging retention and adherence to ART; and (4) psychosocial support from lay counselors, PLHIV and M2M groups. In many HFs, these activities were executed in a complementary manner. Although it was reported to be costly and did not cover all communities, active case-finding was said to have been the most effective strategy. The GAAC activities were moderately effective, followed by the activities of the PLHIV and M2M groups. Community mobilization by activists and counselors also played a supportive role. 7) Improvement in HF management: The use of one-stop PMTCT and TB models, training of HF staff in process organization, reinstallation of the fluxogram (flowchart) card system and reengineering of patient flow led to impressive improvements in patient flow and waiting times. 8) Improvement in HF data management capacity: Error rates of filling in the HIV/AIDS registry decreased substantially, and correct filing of the registers and summarization of registry records for data entry improved moderately. HF data from Sofala, Manica and Tete passed the data quality audit conducted by the External Data Quality Assessments evaluation team in late 2014. 9) Knowledge of GBV (among health workers and community) and service utilization: CHASS trained provincial and district focal points and trained activists to refer cases to HFs. Activists also conducted community sensitization. Cases reported at the HF from the community, police and judiciary increased. However, community linkages for GBV services are still weak, with no active follow-up on victims, and GBV services are not well integrated with other services, except in MCH. Questions remain regarding accessibility of these services to men and boys. 10)Benefits and challenges of working with government through SAs: SAs enabled the development of systems and capacities of the SDSMAS and DPS, “forcing” them to learn. To some extent, SAs fostered mutual understanding of the activities and wider ownership. However, they were based on project activities and had no allowance for executing any activities outside that Performance Evaluation of the CHASS-SMT Project ix scope. Further, there were delays in reimbursement. Overall, the capacity of the DPS and SDSMAS to manage direct funding needs further strengthening. CONCLUSION The project achieved some of its key objectives, including a significant contribution to HSS in the three provinces. However, limited operations research was conducted to understand reasons for low retention, poor patient flow (especially in rural HFs) and low recruitment of children on treatment. The effectiveness of new strategies and quality improvement (QI) initiatives introduced by the project–e.g., reengineering of patient flow, ARV stock management through mobile phones, and improvement of PCR sample quality and turnaround of results–was also not assessed. Overall, Sofala province performed best on most targets. RECOMMENDATIONS Health Systems Strengthening 1) Involve the DPS and SDSMAS in the project’s design; they should play an active role in all stages. A transition plan should be agreed on with the SDMAS, DPS and MOH and be explicitly incorporated in the project documents starting the first year. Face-to-face interactions with the DPS and MOH should be a preferred method of communication. The MOH should actively play its agreed role to ensure its deliverables, success and sustainability. 2) Introduce a component of operations research in project activities and in local capacity building. 3) Provide logistics (including tools, manuals, computer tablets, and financial logistics for DPS travel to districts and DPS/SDSMAS travel to HFs) for institutionalization of the graduation path and clinical tutoring tool by DPS. 4) Establish a training mechanism: This could include a peer-to-peer model in which trained health workers tutor at nearby HFs, work with training institutions to establish courses in HSS for all health workers, and training a group of national and district-level trainers. 5) Expand mHealth (mobile health) in project areas. Most aspects of data management and patient follow-up can be performed with mobile technology. Tablets or PDAs could be used by community case managers or activists to input client information and for text message reminders for patient clinic visits, including mothers returning babies for HIV testing and adherence to ARVs. Working with a mobile phone company, the tablets can also be used for online tutorials for health workers. This has worked well in agricultural health systems in other African countries. 6) Advocate for and support the MOH in rolling out the electronic patient tracking system (EPTS) to moderate- to large-volume HFs. Meanwhile, encourage and train health workers to correctly fill in ARV drug pick-up forms (FILAs) and other retention forms and file them appropriately, and initiate conversations with DPS about the roll-out plan for EPTS once the MOH approves it, including but not limited to data validation processes and tools, plan and terms of reference for data managers, and use of EPTS data for reporting and the QI program. Clinical Services (enrollment and retention) 7) The GRM/MOH should develop a better staff development and retention package or plan to guarantee staff retention in post for at least three years to allow for consolidation of experience. Also, increase staffing levels to meet increasing client load, especially for HIV/AIDS services. 8) The GRM/MOH, with partners’ support, should invest in improving HF physical structure and space. x Performance Evaluation of the CHASS-SMT Project 9) Implement an augmented MOH strategy of a lay counselors’ workforce by recruiting and training lay counselors or retraining case managers and Agentes Polivantes Elementer (APEs) as lay counselors. Some counselors should be stationed at HFs and one in each community to provide services such as adequate counseling for ART initiation, community sensitization against HIV stigma, psychosocial support to HIV-positive pregnant women to disclose HIV status, psychosocial support to GAACs and individual PLHIV, and adherence counseling. 10) Reinvigorate linkages between HF and community-based services, and promote regular information exchange for inter-referrals and health information messaging to communities:  Conduct operations research to map and improve the functioning of community linkages.  Explore opportunities to partner with private telephone companies to introduce mHealth for direct communication between HFs and the community–CBOs, faith-based organizations, activists, traditional healers, religious leaders, etc.–using mobile phones or phone credit, patient monitoring devices, PDAs and tablets, to assure continuous information exchange, data collection and reporting, and health messaging.  Pilot and expand HIV service coverage in underserved rural areas, using mobile clinics with mHealth to provide HCT, HIV care, ARVs, ANC/PMTCT, CD4 testing, etc. 11) Integrate HIV prevention, care and support in adolescent health services at HFs, with emphasis on adolescent girls. Develop a school-based integrated health program with adolescent health and HCT services. 12) Improve HIV testing and ART initiation among infants through training all HF health workers in: PCR sample-taking to provide HIV testing in all pediatric entry points, including the children of patients on ART; linkage of records in labor/delivery units and at-risk child consultations; and adequate counseling of pregnant women. In addition, distribute SMS printers to rural HFs. Community Linkages 13) Reinforce community-HF linkages, and introduce mHealth in mobile clinics and HFs. 14) Partner with employer-based health programs, to provide training and technical support to institute worksite HCT testing, referral, and dispensing ARVs, particularly for male workers. GBV Services 15) Strengthen the GBV community linkages and work with a NGO/CBO that specializes in gender issues to support community activities. Sub-agreements 16) Establish a budget item for non-HIV-related services: This can fund emergencies or disease outbreaks that are not necessarily within the scope of CHASS. 17) Improve on the reimbursement process: Review internal financial controls to make them realistic for GRM collaborations. Performance Evaluation of the CHASS-SMT Project 1 1 EVALUATION PURPOSE AND QUESTIONS 1.1 EVALUATION PURPOSE The five-year Clinical HIV/AIDS System Strengthening Project in Sofala, Manica and Tete provinces (CHASS-SMT) was funded by the United States Agency for International Development (USAID). It was implemented by Abt Associates Inc. from November 2010 to July 2015. The project’s goal was to improve HIV/AIDS and related primary health services in the three provinces by strengthening the primary health care system. This performance evaluation of CHASS-SMT was commissioned by USAID, and its purpose was to determine how the project activities were performing relative to their objectives. The evaluation was to provide an objective view of progress toward the expected results. Its outcomes will inform the transition to the future activity that will support USAID system strengthening and clinical service delivery activities. The main audience for the evaluation is USAID. In addition, results will be shared with Abt Associates, the MOH, U.S. Government agencies, and other stakeholders. The results will also be made available on the Development Experience Clearinghouse. The objectives of the CHASS Evaluation are to:  Assess CHASS-SMT achievements, emphasizing objectives and activity- and project-level results  Identify implementation successes, as well as any internal and external constraints that hindered the implementation of planned activities  Propose recommendations for future directions of CHASS-SMT and for future activities in system strengthening and service delivery to support improved performance in addressing the HIV epidemic, in line with the GRM HIV/AIDS acceleration plan. An additional goal was to use mobile technology, when possible and appropriate, to increase the efficiency, transparency and accuracy of performance data, and to take advantage of multiple data sources (pictures, videos, GPS data). 1.2 EVALUATION QUESTIONS The three main evaluation question areas include 10 evaluation questions. All question areas were answered fully and completely, underscoring both positive and negative outcomes. Where sufficient quantitative data were available, gender analysis was conducted. In order to accomplish the above￾identified evaluation objectives, the evaluation sought to answer the following questions: Question Area 1: Project Achievements and Challenges 1) What are the strengths and weaknesses of the activities, as seen by the implementing partner staff, Provincial Directorate of Health (DPS), District Directorate of Health and Social Welfare (SDSMAS and chief medical officer), HFs, USAID, and the USAID-funded Community Care Project (PCC), and how can weaknesses be improved, according to these stakeholders? 2) Where has the most progress been seen in strengthening systems (e.g., planning, financial management, supply and logistics, information systems)? 3) Which activities or project components will be most feasible to transition from the project to the GRM? 2 Performance Evaluation of the CHASS-SMT Project Question Area 2: Linkages 4) To what extent has the project been able to create and strengthen linkages between HFs and communities to allow for increased service uptake, specifically in the areas of: (a) Community-based counseling and testing to treatment (for both men and women) (b) Retention of pre-ART and ART patients (through the use of adherence groups, active case-finding, and other community groups) (c) Knowledge, demand, and access of services by men 5) What constraints have the activities faced in improving retention of patients in pre-ART and ART? 6) What has been the most effective method found by the project to improve retention? Question Area 3: Health System Strengthening 7) To what extent has HF management (improved patient flow, etc.) improved over the life of the project? 8) To what extent is data management capacity built at the HFs with regard to HIV/AIDS registry data? 9) To what extent has knowledge (community and health worker) and utilization of GBV services increased over the life of the project? 10) What are the benefits and challenges of the activities’ model of working with government (at provincial and district levels) through sub-agreements? Specific questions that guided the performance evaluation are stated in the evaluation matrix included in Annex I. Performance Evaluation of the CHASS-SMT Project 3 2 PROJECT BACKGROUND 2.1 GENERAL CONTEXT In July 2009, USAID/Mozambique launched a request for applications for a results-oriented five-year project to improve HIV clinical services in Manica, Niassa, Sofala and Tete provinces within a strengthened, comprehensive primary health care system. The CHASS project was designed to address the HIV situation in Mozambique and focused on three components: a) Improving service quality in six important areas: HCT, laboratory services, prevention of mother-to-child transmission (PMTCT), adult care and treatment, pediatric care and treatment, and the prevention, diagnosis and treatment of HIV-TB co-infection b) Enhancing program linkages and integration to provide a continuum of accessible services, including MCH and reproductive health (RH) services, within facilities and between facility and community-based services c) Creating stronger and more sustainable Mozambican systems and institutions At the time of project design in 2010, the average HIV prevalence in Mozambique was estimated at 16 percent nationwide (ANC survey in pregnant women aged 15-49 years). Nearly 1.6 million people were living with HIV, and nearly half of all HIV-infected (48.4 percent) were identified at the time as having active tuberculosis. Within the Central Region, Sofala and Manica provinces have the highest prevalence, 23 percent and 16 percent respectively. Tete was identified as having a very mature epidemic, and existing infrastructure that was unable to accommodate the numbers of patients requiring care and treatment. Niassa was considered a particularly underserved province, with vastly inadequate infrastructure. To address these issues, two separate agreements were awarded to two implementing partners: FHI 360 for CHASS Niassa, and Abt Associates Inc. for CHASS-SMT (Sofala, Manica and Tete). Activity funding for CHASS-SMT was $111,545,465 for the period of November 1, 2010 to October 31, 2015 and was to support USAID/Mozambique’s Country Assistance Strategy’s (CDCS DO4) priority goal number three, “Improved health of Mozambicans.” It was to contribute specifically to the following focal areas in USAID’s Health Results Framework: (a) Improved access to and delivery of quality integrated services; (b) Increased adoption of healthy behaviors and informed use of services; and (c) Strengthened health systems. Implementation of CHASS-SMT started in early 2011. After the two CHASS activities were awarded, the MOH developed the HIV Acceleration Plan in 2011. In addition, both activities included programming to address gender-based violence (GBV) within the HIV platform, with a total life-of-project funding of approximately $1.5 million. GBV fosters the spread of HIV by limiting a person’s ability to negotiate safe sexual practices, disclose HIV status, access services (due to fear of reprisal), adhere to treatment and access care. Activities were implemented in all four provinces to:  Expand and improve coordination and effectiveness of GBV prevention efforts  Improve policy implementation in response to GBV  Improve the availability and quality of GBV services 2.2 OVERVIEW OF CHASS-SMT IMPLEMENTATION CHASS-SMT aimed at achieving the following four objectives: 4 Performance Evaluation of the CHASS-SMT Project 1. Strengthen Mozambican health systems and institutional capacity to provide high-quality services and ultimately receive and manage direct support from the U.S. Government 2. Improve integration of HIV and related primary health care services and linkages between the community and the health system 3. Increase demand, use and provision of high-quality HIV services 4. Increase coverage of care and treatment as well as PMTCT services CHASS-SMT followed the Health Alliance International (HAI) project that was mainly focused on delivery of HIV/AIDS and related services at the HF level. Building of local leaders’ capacity and HSS were done at both the provincial and district levels. Integral to this was the gradual transition of resources and responsibilities to the GRM staff at these levels. The project operated within the framework of GRM decentralization plan for health services delivery as well as MOH protocols and guidelines, and it aimed to align its work-plans with the Planos Económicos e Social (economic and social plan) and Plano Económico e Social Operacional Distrital (District Social Economic Operational Plan). After 18 months of implementation, the project restructured its operational model to focus, support and integrate technical support to improve capacity of the districts (SDSMAS) to manage and supervise health programs and HIV/AIDS service delivery. The revised model organized project staff into multidisciplinary teams, or Equipas Polivalentes (EPs), to collaborate with DPS to integrate technical assistance (TA) to SDSMAS and health facilities. TA by EPs involved short training, mentorship, supportive supervision and clinical tutoring of the DPS, SDSMAS and HF staff. They also developed tools, standard operating procedures and guidelines. Most project activities were achieved through SAs between the project and the three provinces, 36 districts and eight CBOs. In addition to TA and financial support, the project procured office materials, equipment (for laboratory and information systems) and cars or motorbikes, and it did some basic infrastructural rehabilitation. A basic package of clinical and lab consumables and equipment (kit básico) was distributed. CHASS-SMT trained new health workers through SAs with training institutions in the three provinces. Since the government takes more than a year to clear new staff into their system, the project provided 12 months of gap funding for these new staff. The project also used s graduation path strategy with which EPs assisted SDSMAS to systematically identify specific service delivery and management needs and gaps using a 23-item Standards Compliance Tool. This was followed by technical support from EPs to address identified needs. These efforts contributed towards the districts’ ‘graduation’ from direct TA and subsequent ability to receive and manage support through funding from SAs only. Further, the project developed a clinical monitoring and tutoring tool, used jointly with DPS and SDSMAS staff to monitor HF service delivery. Some of the USAID-funded projects CHASS-SMT worked with included PCC and ROADS for community mobilization activities; TB Care for TB/HIV co-infection activities; and SCMS/SPS/Deliver for supply chain logistics in the three provinces. Performance Evaluation of the CHASS-SMT Project 5 3 EVALUATION METHODS AND LIMITATIONS 3.1 EVALUATION METHODS This performance evaluation was based on a non-experimental design and was executed by a team independent and external to the CHASS project. The evaluation used a mixed-methods approach, utilizing mostly qualitative data collection and evaluation methods. The evaluation involved extensive desk review and analysis of existing quantitative project data and documentation, and primary collection and analysis of qualitative data. Quantitative data were extracted from the quarterly progress reports (QPR) and project datasets held by the USAID team. The evaluation was conducted September 4-28, 2015. Primary data collection methods included key informant interviews and observational analysis at HFs. The key informant interviews served to: (a) validate and, where possible, verify project approaches or activities, interventions and achievements, extent of gains and changes over time; and (b) identify gaps and weaknesses in project activities or performance. Key informants included staff from the MOH, DPS, SDSMAS, HFs, CHASS implementing partner and CBOs that participated in the project. Some members of the USAID/Mozambique team were also interviewed. The structured interview guides are included in Appendix III. The survey instruments were pre-tested in Beira city before fieldwork began. Interviews with patients were not within the remit of the scope of work. The evaluation used multi-channel data collection, using mobile technology when possible and paper￾based methods when handheld computers/tablets were not appropriate. The team noted that the use of mobile technology is feasible, and some of the members handwrote their field notes using the ODK program, enhanced with a stylus beta program. However, for the mobile technology to be most useful for collection of qualitative data, it was noted that the key informant and focus group guides should be semi-structured. Also, for the locally recruited research assistants, more time was required for extensive practice of handwriting on the tablets. This was not possible in this evaluation due to time constraints. 3.2 SITE SELECTION AND DATA COLLECTION Districts were stratified based on their performance on SAs, relative to the average performance for their province in 2014. Within each stratum, a random sample was taken to represent dominantly rural, semi-urban and urban districts. Data collection took place in the following locations: Beira, Nhamatanda and Dondo in Sofala province; Chimoio, Gondola and Manica in Manica province; and Tete and Changara in Tete province. Within each district, the team selected HFs purposively to represent low and high client volumes, extent of CHASS activities at the site and rural or urban location. Fifteen HFs were visited, and various categories of health workers were interviewed (Table 1). Categories of respondents are summarized in Table 2. The DPS/SDSMAS senior staff included the directors and medical chiefs, and the HF management team included the HF in-charges. Respondents are listed in Appendix III. 6 Performance Evaluation of the CHASS-SMT Project Table 1: Selected districts and health facilities Province District Location District’s performance on SA activities relative to the province’s performance in 2014 Name of HFs selected Sofala Beira Urban Below Munhava and Ponta Gea Nhamatanda Rural Above Nhamatanda Sede and Tica Dondo Semi-urban Below Dondo Sede & Mafambisse Manica Chimoio Urban Below 1 de Maio Manica Semi-urban Above Manica DH and Messica Gondola Rural Below Gondola Sede and Amatongas Tete Nucleo Provincial de Combat ao Sida Changara Rural Average Changara Sede, Chipembere Tete Urban Above Bairro Francisco Manianga (No. 3) and Bairro Muthemba (No. 4) A total of 18 CHASS-SMT staff, eight PCC staff and nine staff from CBOs that worked closely with CHASS-SMT were interviewed. In addition, two staff from the MOH and five from USAID were interviewed. Table 2: Categories of key informant interview respondents Group Sofala Manica Tete Total DPS/SDSMAS senior staff 4 4 2 10 HF management team 6 5 4 15 HIV focal points/technicians (at DPS, SDSMAS and HF level) 7 7 4 18 SMI focal points/technicians (at DPS, SDSMAS and HF level) 7 8 4 19 HSS/monitoring and evaluation focal points (at DPS & SDSMAS) 3 3 3 9 Logistics/finance (at DPS and SDSMAS) 2 0 2 4 GBV focal points (at DPS, SDSMAS and HF level) 3 3 2 8 Other (lab, pharmacy) 1 2 3 6 Total 33 32 24 89 Percent of target 100% 97% 86% 95% 3.3 DATA ANALYSIS The analysis included data in the performance monitoring system and program reports, and included trend analysis of results and progress made on planned results. The qualitative data were analyzed using a qualitative data analysis matrix. Information from this analysis was integrated or triangulated with the available quantitative data from CHASS documents and reports. The analysis was guided by the 10 evaluation questions listed in the scope of work (Appendix I) and in Section 1 of this report. For questions where gender-related data or information are relevant, gender￾related differences are presented. The end summary of the analysis was focused on the priority issues Performance Evaluation of the CHASS-SMT Project 7 for CHASS to address and main lessons learned, based on the answers provided in examining the evaluation questions. 3.4 STUDY LIMITATIONS Limitations of the evaluation include the following: First, the evaluation was initially planned to take place in the project’s fourth year, but instead was conducted at the end of the project. Secondly, the evaluation team was not able to ascertain consistent factors responsible for the project’s success and weaknesses, as the project evolved substantially over time. Therefore, some of the factors reported are from a summative perspective. The findings reported here are according to what the interviewees told the evaluation team. The likelihood of interviewee bias cannot be ruled out. Lastly, the time allocated for the evaluation was limited compared to the complexity of the project, and this, in turn, constrained the detailed evaluation of each of the components within each province. To some extent, this also constrained the sample size for the key informant interviews. Further, because the evaluation was conducted at the project’s end, when staff were transitioning, some of the interview scheduling was delayed. 8 Performance Evaluation of the CHASS-SMT Project 4 FINDINGS, CONCLUSIONS AND RECOMMENDATIONS 4.1 FINDINGS The CHASS-SMT project suffered a slow start in the first year, partly because the team was trying to reorient the DPS to the concepts of HSS and working jointly through SAs. In the third year, the project had to change its approach from HSS at DPS only to HSS at both DPS and the districts. Despite this, the project achieved most of its objectives, including improved HSS (project objective 1) and coverage of HIV/AIDS services. The evaluation findings on project achievements and challenges are organized in the subsequent sections according to the evaluation questions. Question 1: What are the strengths and weaknesses of the activities, and how can weaknesses be improved, according to the interviewed stakeholders? This section summarizes strengths and weaknesses, organized according to project objectives 1-4. The team notes that the achievements listed should not solely be attributed to CHASS-SMT strategies, considering the interventions of other actors in the provinces. Strengths and weaknesses of activities to improve clinical service delivery HIV care and support: In line with MOH guidelines and protocols, CHASS-SMT provided training and technical support to DPS’ and SDSMAS’ efforts toward the decentralization and integration of ART services at the primary health care level, with an emphasis on introducing ART at HFs in peripheral areas. As a result, 77 new ART sites were opened. In addition, the project provided SDSMAS and HFs with technical support to improve logistics and transport of CD4, PCR samples and some medications, and it provided a Kit Basico with basic equipment for SDSMAS and HFs in selected districts. Health workers were also trained and provided with logistics for quality improvement/assurance (QI/QA) activities at HFs and labs. Health facility clinical staff emphasized that technical support by EPs contributed to structuring and organization of HIV services, data review and increased demand for HIV care and treatment. The strength of the technical support was that it integrated many aspects, and not clinical issues only. These included tutoring in data quality, improved patient flow, record keeping and sharing, stock management and staff motivation, among others. Through training and technical support, enrollment of patients on ART increased threefold between 2010 and 2014 (Figure 1). Similarly, the number of individuals currently enrolled on ART increased nearly threefold from 57,894 in 2011 to 155,588. These were more than 100 percent of the project’s targets. Children represented 1 percent of ART enrollees in 2011, increasing to 8 percent by 2014. More women than men enrolled for ART in all three provinces, which is consistent with stakeholder reports. Further, the percentage of clients currently on ART who are men declined from 33 percent in 2011 to 29 percent in 2014 (Figure 1). This is partly due to scale-up of Option B+ among pregnant “We saw improved performance in four HFs and ART increase to 11 sites; their indicators improved over time.” (Director, DDS, Messica, Manica) “With structuring and organization of Clinical HIV,TB, PMTCT, mapping HFs using indicators, team data review and database graphs, we detected mistakes and could improve.” (Monitoring and Evaluation Technician, Nhamatanda) Performance Evaluation of the CHASS-SMT Project 9 women. This is expected, as HIV-positive men are mainly reached through HCT programs, but HIV￾positive women are reached through both HCT and PMTCT programs. Figure 1: Patients already on ART and newly enrolled Numbers Integration of HIV services in primary care: CHASS-SMT provided assistance to SDSMAS and HFs to integrate HIV services for pregnant women and also for TB/HIV co-infected patients, adopting the one-stop service delivery model. MCH nurses were trained to test and counsel for HIV, take blood samples and prepare dry blood spots (DBS) for PCR, prescribe ARVs to HIV-positive patients, provide Option B+, promote Mother-to-Mother activities onsite, and promote male partner involvement in testing in ANC/PMTCT services. Chief medical officers and MCH nurses reported how service delivery in ANC improved with the one-stop model. The community activities led to an improved number of women attending ANC, from 223,168 in 2012 to 295,267 in 2014, of which 87 percent were tested for HIV in 2012 and 88 percent in 2014. At least 86 percent of the HIV-positive pregnant women were initiated on ART (Figure 2). However, the percentage of children born to HIV-positive mothers who receive a PCR test within 12 months is still low (Table 3). HCT established at HFs: To increase facility-based HCT services, the project provided logistics for counseling, testing and referral at all points of entry at the HFs. Between 2011 and 2014, 966,155 individuals were tested at HFs, and the 142,081 who tested positive were referred into pre-ART care. “One-stop approach worked very well; there is no waiting time and treatment of patients improved.” (Nurse, Tete) “With one-stop approach, our services are integrated at each HF; we nurses test women and babies.” (Medical Chefe, Nhamatanda) “We were successful in bringing all pregnant women who are HIV-positive to ART, and babies born are negative. And since 2013, we provide ART at maternity ward for positive pregnant women.” (MCH nurses in Gondola and Changara Districts) 43,939 37,094 69,717 107,913 13,955 23,922 32,481 47,675 33% 31% 30% 29% 8% 8% 8% 8% -25% -15% -5% 5% 15% 25% 35% 45% - 20,000 40,000 60,000 80,000 100,000 120,000 140,000 160,000 180,000 2011 2012 2013 2014 New on ART Already on ART % Current ART clients who are men % Current ART clients who are children 10 Performance Evaluation of the CHASS-SMT Project Sixty percent of HCT was through the provider-initiated approach. Health workers emphasized that community mobilization contributed significantly to the increases. Figure 2: Key PMTCT indicators A summary of other findings is presented in Chart 1. Chart 1: Clinical services weaknesses and recommendations Findings Recommendations 1) Low retention rates (poor quality of counseling due to insufficient number of staff, long distances, HIV status disclosure issues, stigma) 2) Provincial, district and HF stakeholders report insufficient number of staff to manage improved activities, meet increased demands and maintain high-quality services. For example, MCH nurses in some HFs reported attending to more than 60 patients per day. Evaluators observed this in two HFs. 3) Integration of primary health care services and HIV is still a) The GRM/MOH should develop a better staff development and retention package or plan to ensure staff stay in post at least three years to enable consolidation of experience. Increase staffing levels to meet increasing client load, especially for HIV/AIDS services. b) Implement an augmented MOH strategy of a lay counselors’ workforce by recruiting and training lay counselors or retraining case managers as lay counselors to provide various services. c) Reinvigorate linkages and communication between HF and community￾based services:  Conduct operations research to map linkages and measures to improve communication.  Explore opportunities to partner with private telephone companies to introduce mHealth.  Use mobile phone SMS/calls to follow-up on mothers with exposed infants. “We set up an area for HCT and it works; we test everyone who comes in, even I do testing, and it works!” (Clinic Director, Ponta Gea) “We test even in the dental sector.” (Director, Gondola) 223168 234607 295267 90% 92% 91% 88% 86% 86% 21% 33% 64% -20% 0% 20% 40% 60% 80% 100% 0 50000 100000 150000 200000 250000 300000 350000 2012 2013 2014 No. women registered at ANC % pregnant women with known HIV status (newly tested + known positive at ANC entry) % HIV+ pregnant women Initiated on ART % HIV+ pregnant women in ANC who have initiated CTX Performance Evaluation of the CHASS-SMT Project 11 Findings Recommendations weak, except for one-stop PMTCT and TB models. 4) Few men and children are being reached. 5) Overall, sustainability of achievements is questionable.  Lay counselors should visit homes of new enrollees in pre￾ART, ART and PMTCT cascade. d) Pilot and expand HIV service coverage in underserved rural areas using mobile clinics with mHealth component to deliver HIV services. e) Partner with private-sector based health programs to provide worksite HIV services. f) Improve HIV testing and ART initiation among infants through training all HF health workers in: PCR sample-taking to provide HIV testing in all pediatric entry points, including the children of patients on ART; linkage of records in labor/delivery units and at-risk child consultations; and adequate counseling of pregnant women. Distribute SMS printers to rural HFs. Table 3: Summary of selected indicators to illustrate achievements in clinical services Indicator/Province/FY 2011 2012 2013 2014 % of 2014 Target achieved† Number currently enrolled on ART Sofala 26,387 (20,935) 26,246 (32,323) 49,176 (32,555) 70,565 (45,303) 156% Manica 16,440 (14,132) 20,637 (20,053) 30,277 (25,434) 49,346 (34,361) 144% Tete 15,067 (13,039) 14,133 (17,636) 22,745 (19,050) 35,677 (27,386) 130% Total 57,894 (48,106) 61,016 (70,012) 102,198 (77,039) 155,588 (107,879) 144% Men currently enrolled on ART Sofala 8,369 7,898 14,280 19,401 123% Manica 5,473 6,433 9,149 13,575 116% Tete 5,370 4,821 7,593 11,506 119% Total 19,212 19,152 31,022 44,482 120% Number newly enrolled on ART Sofala 7,248 (9,833) 11,387 (10,920) 16,596 (12,405) 21,529 (16,077) 131% Manica 3,391 (6,100) 8,262 (6,729) 9,483 (10,770) 16,226 (18,638) 87% Tete 3,316 (3,626) 4,273 (5,260) 6,402 (6,791) 9,920 (9,386) 106% Total 13,955 (19,558) 23,922 (22,908) 32,481 (29,966) 47,675 (44,305) 108% Percent of pregnant women with known HIV status (newly tested and known positive at ANC entry) * Sofala 99% 98% 91% Manica 88% 95% 93% Tete 84% 86% 89% Total 90% 92% 91% 12 Performance Evaluation of the CHASS-SMT Project Indicator/Province/FY 2011 2012 2013 2014 % of 2014 Target achieved† Number of HIV-positive pregnant women who received ARVs to reduce risk of MTCT* Sofala 11,213 10,774 9,447 95% Manica 6,101 5,732 6,487 78% Tete 4,634 6,110 5,055 112% Total2 21,948 22,616 20,989 92% Percent of HIV-positive pregnant women who received ARVs to reduce risk of MTCT* Sofala 100% 89% 85% Manica 73% 78% 87% Tete 92% 90% 86% Total 88% 86% 86% Number of HIV-positive pregnant women in ANC who have initiated Cotrimoxizole* Sofala 2,528 3,936 7,724 155% Manica 1,714 2,859 4,750 114% Tete 1,032 1,888 3,043 135% Total 5,274 8,683 15,517 136% Percent of HIV-positive pregnant women in ANC who have initiated Cotrimoxazole* Sofala 22% 32% 70% Manica 21% 39% 63% Tete 20% 28% 52% Total 21% 33% 64% Percent of infants born to HIV-positive women who received an HIV test within 12 months of birth* Sofala 69% 54% 58% Manica 30% 45% 47% Tete 35% 39% 51% Total 45% 46% 52% Percent of HIV test results for infants born to HIV-positive women who received an HIV test within 12 months of birth that are positive* Sofala 11% 10% 7% Manica 8% 6% 6% Tete 11% 9% 8% Total 11% 9% 7% 2 Data quality and data availability has changed over time, due to the nature and use of registers at HFs. Due to data quality improvement efforts, duplicate reporting of women receiving ARVs has likely decreased over time, hence the net reduction in numbers of women initiated on ART. Performance Evaluation of the CHASS-SMT Project 13 Indicator/Province/FY 2011 2012 2013 2014 % of 2014 Target achieved† Number of partners of women who are HIV tested in a PMTCT setting* Sofala 834 2,534 6,048 20% Manica 562 1,456 2,629 8% Tete 348 1,297 3,576 10% Total 1,744 5,287 12,253 12% Percent of partners of women who are HIV tested in a PMTCT setting* Sofala 1% 3% 6% Manica 1% 2% 3% Tete 1% 2% 3% Total 1% 2% 4% *No data were available for most of the PMTCT-related indicators in FY 2011 and are left blank in the table. Source: CHASS USAID progress report data Strengths and weaknesses of HSS activities Through training of staff and provision of logistics, equipment and materials, the project helped the DPS and SDSMAS to develop planning, health program management and financial management skills. All supported districts reported improved capacities to manage SAs. All DAF personnel who had been working on SAs reported confidence in managing them. One DAF noted: At the SDSMAS level, the project introduced a graduation path methodology that enabled SDSMAS to systematically identify weaknesses and gaps and to develop action plans for further improvement of their health systems and capacities. At the HF level, the project revised a tool for supportive supervision into an electronic clinical monitoring (for quality) and tutoring tool that helped in systematic identification of training needs, on-the-spot targeted tutoring, and development of elaborate action plans. The QI approach was used in developing strategies for action. These two tools and associated methodologies were effective in improving HSS, health workers’ skills and their work environment. Data in Figure 3 show the extent of HSS in the three DPS for the delivery of HIV/AIDS response. The figures show an increase in scores from one star (inner part of the web) to five stars (outer). Summaries of other findings are in presented in Chart 2. “I have learned a lot because of operating the sub-accords; I can operate all activities and associated budgets fully, including reporting, without any further support.” (DAF Manica SDSMAS) 14 Performance Evaluation of the CHASS-SMT Project Figure 3: Performance of systems and capacities at the districts within the provinces at different rounds of assessments (Source: Project WPY 2015) At initial assessment, the systems and capacities for planning and managing health programs were better in Sofala and Tete than in Manica. Nonetheless, by the third assessment, all the provinces had a score of at least four stars (on 1-5 scale). Manica also had the weakest laboratory and supply chain systems and capacities and the weakest community linkages. By the fourth assessment, all these components had attained a score of at least four stars. This could partly explain faster rate of scale-up in Manica between 2013 and 2014. The fluctuations in scores for planning and financial and health program management were reportedly due to transfer or resignation of staff at DPS and SDSMAS. The application of the graduation path at the SDSMAS helped to improve the management and planning systems. Overall, the capacity of the districts and provinces to conduct program monitoring as a means of identifying priority problems and linking service utilization patterns with resource planning improved from 50 percent to over 70 percent. The project contributed considerably to human resources at DPS, SDSMAS and the HFs. The project provided funding for pre-service training of more than 350 health workers (30 pharmacy and 47 health technicians, and about 273 nurses) and deployed them into the health system. The project also recruited new staff, including 40 monitoring and evaluation staff in Manica. The EP, through integrated HF visits, improved the skills, attitudes and competencies of health workers, the quality of HIV/AIDS services, and the data. MCH nurses were trained to prescribe ARVs, leading to the expansion of Option B+. In addition, through training and financial support, the project supported rollout of GRM human resources for health management systems, such as the personnel information system. Performance Evaluation of the CHASS-SMT Project 15 The supply chain of drugs and lab commodities and the transport logistics for CD4 and PCR samples and results were among the most improved HSS components. The project improved linkages of laboratory services to HFs through organizing transportation logistics for mobile CD4 analyzing machines (PIMAs) and SMS printers. Additional findings are included in Chart 2. Chart 2: HSS weaknesses and recommendations Challenges/Findings Recommendations 1. The sub-agreements took too long to be operationalized due to poor communication and not involving DPS at initial project design. 2. To date, the graduation path and the clinical monitoring tool that were key in HSS were not institutionalized in any DPS/SDSMAS. Not all required logistics were in place at the provinces. 3. The project did not build operations research capacity, which limited understanding of what management measures were most successful in improving overall performance. a. The DPS and SDSMAS should be involved in the design of the project and play an active role in all stages through face-to-face interactions. The transition plan should be agreed with the DPS and MOH and be explicitly incorporated in the project documents. The MOH should actively play its role as agreed with the project to ensure its deliverables, success and sustainability. b. Introduce an operations research component in project activities and local capacity building. c. Provide logistics (including the printed tools, manuals and provision of computer tablets, and financial logistics for DPS’ travel to districts and DPS/SDSMAS travel to HFs) for institutionalization of the graduation path and clinical tutoring tool by DPS. 4. Provincial and district authorities did not contract some of the new staff contracted by CHASS after the gap funding ended. 5. HSS sustainability depends on staff turnover/ transfer. SDSMAS did not create a mechanism to train new staff, and insufficient numbers of staff and a high rate of transfers were noted. 6. The project lacked a well-planned and appropriate exit strategy to promote adequate continuity. d. Put in place a training mechanism at SDSMAS. This could include a peer-to-peer model in which senior or trained health workers can conduct monitoring and tutoring at nearby HFs; working with training institutions to establish courses in HSS for all health workers; and training a group of national trainers. e. Involve DPS/SDSMAS/MOH in the design of an exit strategy early to ensure continuity of services. Strengths and weaknesses of activities to improve PHC program integration and community linkages Details of the primary health care program integration are discussed under evaluation question 1, while details about the role of community linkages are given under questions 4 and 5. Question 2: Where has the most progress been seen in strengthening systems (e.g., planning, financial management, supply and logistics, information systems)? The DPS and SDSMAS key informants identified HR management as the most significantly improved component of HSS. Overall, the M&E systems and capacities were among the least developed; these include the limited rollout of the electronic databases. The project staff noted that the use of strategic information for planning and management was still weak in Sofala and Tete provinces. Similarly, the financial management capacity of the districts and provinces improved over time, though these improvements were far below the desired level. This is reportedly due to insufficient manpower in the districts and provinces; 16 Performance Evaluation of the CHASS-SMT Project additionally, available staff had low management capacity. The increased workload due to project activities was nicely summarized by the PDAF of Tete: Additional findings are summarized in Chart 3. Chart 3: HSS weaknesses and recommendations (2) Challenges/Findings Recommendations 1. The M&E system is still weak. 2. Staff numbers are still insufficient, and the expanded activities have increased workloads, limiting the quality of some services. 3. The relatively high rate of staff transfers or rotation implies that the ability to sustain HR capacities at SDSMAS and HFs is limited. 4. No mechanism was put in place for SDSMAS to train new staff. 5. The MOH could not hire all the new staff recruited under the project gap￾funding system to stay at SDSMAS or facilities. 6. Most of the project’s operational costs were managed by the project team, hence limiting experiential learning by DPS and SDSMAS. a. Expand mHealth in project areas: Tablets, PDAs or phones should be used by CCMs or activists to input client information. Text messaging reminders for patient clinic visits, including mothers returning babies for HIV testing and adherence to ARVs, are an effective strategy that has worked elsewhere. Working with a mobile phone company, the tablets can also be used for online tutorials for health workers. b. Establish a training mechanism (see description in chart 2). c. The GRM/MOH should develop a better staff development and retention package or plan, e.g. to ensure staff stay in post for at least three years to enable consolidation of experience; and increase staffing levels to meet increasing client loads, especially for HIV/AIDS services. Question 3: Which activities or project components will be most feasible to transition from the project to the GRM? The project worked jointly with SDSMAS and DPS in executing most of its activities. Nonetheless, some activities predominantly relied on the project staff, including the management of operational costs for supply chain and transport logistics, the graduation path tools and assessments, and the clinical monitoring and tutoring tool and assessments. The graduation path and clinical monitoring tools, methodology and assessments should be transitioned to the DPS and SDSMAS. However, SDSMAS staff mentioned that progress would be very slow without project support. Some SDSMAS/DPS staff cited heavy workload, while others cited staff turnover as a limiting factor. Question 4: To what extent has the project been able to create and strengthen linkages between HFs and communities to allow for increased service uptake? The project worked with eight CBOs (three each in Sofala and Tete, and two in Manica) through SAs to strengthen linkages between HFs and communities in all 36 districts. For each selected HF, CBOs hired a CCM to be based at the facility and also mobilized activists within its catchment area. The case managers were community focal persons at the facility. The activists and CBOs worked with mobile brigade teams, community HIV counselors and community leaders to hold meetings and health talks (palestras), trace patients lost to follow-up and do community-based HCT. “CHASS did best they could, the problem is the level of staffing at district level is low and even the few available are either overworked or have weak management capacity” Performance Evaluation of the CHASS-SMT Project 17 The project also provided logistics for the activities of the local adherence support groups (GAACs), M2M groups and PLHIV groups. GAACs are groups of HIV-positive individuals who live in the same community. Each member travels to a HF every six months on a rotational basis to have a medical consultation and pick up drugs for the members. In the following sections, the contribution of these community engagements is summarized for the various aspects of health services. Linkages for increased service uptake of community-based counseling and testing to treatment (for both men and women) Between FY 2011 and FY 2014, CBO and community counselors provided HCT in the community to a total 546,299 persons, 44 percent of whom were men. Community HCT brought in 32 percent of the total number of HCT clients in 2011 and 33 percent in 2014. Through community HCT, 26,727 HIV￾positive individuals were identified and referred to a HF. Of those referred, 40–60 percent reached the facility (Figure 4). Other findings are summarized in Chart 4. Figure 4: Retention and community linkage indicators Linkages to allow for increased retention of pre-ART and ART patients (through the use of adherence groups, active case-finding, and other community groups) The community-based individual and groups involved in improving retention were case managers, community activists, volunteers, GAACs, and PLHIV and M2M groups. CBOs trained activists to handle outreach to patients, consisting of active client outreach (‘busca activo’), especially aimed at convincing ART defaulters to come back to the HF to continue care. The case managers would retrieve a list of defaulters and hand it over to the activists. In some HFs, the activists, themselves generated these lists and followed up in the communities. Each GAAC group member picks up medication once every three or six months in a rotational manner. This helped members to save on money for transportation–a barrier to retention. The members come to the HF once every six months for review by a physician. The M2M group members support other 0% 10% 20% 30% 40% 50% 60% 70% 80% 2011 2012 2013 2014 % referred from community & reached HF % ART defaulters returned by Activists Retention on ART at 12 M (Adults) Retention on ART at 12 M (Children) Percent “Having case managers at HFs review the data and follow up on abandonments directly or through community activists really helps.” (DPS HIV Technician, Tete Province) 18 Performance Evaluation of the CHASS-SMT Project HIV-positive women to adhere to ART and encourage them to disclose their status to their husbands. This indirectly contributes to improved retention. The project managed to increase the retention rate on ART for the 12-month adult cohort by about 4 percent, from 68 percent in 2011 to 72 percent in 2014. The improvement in retention was slightly higher in Sofala in 2014 (about 78 percent) (Table 3). Over 50 percent of ART patients in Sofala were from Beira, where transport challenges due to long distances were relatively few compared to rural districts. More than 50,000 ART defaulters were tracked over the course of the project’s life, and at least 50 percent were returned to the HF (Figure 4). Informants cited the following reasons for persistent retention rates below the targeted 85 percent: long distances and transportation challenges; limited coverage of the communities by activists; poor counseling at the HF due to insufficient number of staff and lack of privacy; fear to disclose HIV status to spouses; HIV stigma; and sociocultural factors. Other findings about the challenges are summarized in Chart 4. Linkages to improve knowledge, demand and access of services by men There were no specific activities targeting men, except for the community health talks about men’s involvement in ANC services and the invitation of men to accompany their wives for prenatal care in PMTCT services. Nonetheless, less than 3 percent of male partners in Tete and Manica and 6 percent in Sofala were tested for HIV at the HFs. Only one CBO mentioned using a family approach model introduced by PCC to reach men and children in some communities. Other findings are summarized in Chart 3 and in Evaluation question 5 and in Chart 4. Chart 4: Challenges to community linkages Challenges Recommendations 1) Low pre-ART and ART retention rates 2) Low reach to men 3) CBO sub-agreements pre￾defined activities without addressing CBOs’ priorities and potential to do more. 4) Stock-outs of HIV test kits 5) CBO activities at the district level are not well coordinated; some CBOs do not report to SDSMAS but to the project. a) Recruit and train lay counselors (see description in Chart 1). b) Reinforce community-HF linkages and communication, and expand on the number and types of CBOs (e.g., actively involve the faith￾based organizations, traditional birth attendants, traditional healers, etc.). Include the existing GRM structures of Agentes Polivantes Elementares (APEs) and social assistants from the Ministry of Social Action. c) Train CBOs on requesting test kits in a timely manner through mobile phones. d) To reach men, the community activities should include drama and meetings encouraging couples’ counseling. Partner with private sector health programs to provide HIV services. Encourage staff to ensure invitations of male partners in ANC. e) Research into the role of masculinity in access and utilization of services would be valuable. f) Pilot and expand HIV service coverage in underserved rural areas using mobile clinics with mHealth component to deliver HIV services. “Stigma and discrimination is challenging; many women do not want spouses, family to know their HIV status. The project did not deal effectively with this challenge.” (HW, Changara, Tete) Performance Evaluation of the CHASS-SMT Project 19 Question 5: What constraints have the activities faced in improving retention of patients in pre￾ART and ART? Key strategies aimed at improving retention included: active case-finding by the activists with support from case managers; GAAC members picking drugs for each other and providing psychosocial support; community mobilization activists encouraging retention and adherence to ART; and psychosocial support from the PLHIV and M2M groups. Stakeholders reported various constraints that limited the achievements of these strategies. Retention rates persisted below the 85 percent target across the provinces. A major constraint was the insufficient number of staff to provide adequate counseling in some of the HFs and communities. Retention is a cycle that begins at the HF or community level with counseling when an individual tests HIV-positive. If lay counselors or HF staff do not have time to counsel patients so that they understand that HIV needs continuous follow-up, retention problems arise. In CHASS SMT, staff training on high￾quality counseling did not translate into adequate counseling, because the staff workload was still high, or even higher, compared to the period before the training. The influence of inadequate counseling on retention is potentially higher among HIV-positive pregnant women. Eligibility criteria have changed frequently among this group: from Option A to B, and then to B+. Adequate explanations of these changes and the rationale for immediate initiation of an otherwise healthy woman on ART are necessary. Other constraints are summarized in Chart 5. Chart 5: Constraints to activities aimed at improving retention and recommendations Constraints/Weaknesses Recommendations 1) Lack of an electronic database for tracking retention and poorly organized records on defaulters 2) The catchment area of each HF is far wider than the coverage of the CBOs, which often have few activists. 3) Long distances to be travelled by activists and volunteers to reach defaulters 4) Wrong addresses given by patients, and patients moving out of the community during the farming season 5) Breakup of some GAACs. Due to stigma, not all GAACs are functioning well; some fear inadvertent disclosure of status to community by other members. 6) Fear of disclosing HIV status to partners and family, especially in polygamous families a) Advocate for and support the MOH to roll out the EPTS to moderate- to large-volume HFs. In the meantime: (1) encourage and train health workers to correctly fill in FILAs and other forms on retention and file them appropriately; (2) initiate conversations with DPS about the roll-out plan for EPTS, including but not limited to data validation processes, tools/plan and terms of reference for data managers, and use of data from EPTS for reporting and QI program. b) Improve and expand measures to use data for tracking retention, especially extending electronic systems set up by the project. c) Reinforce community linkages (see chart 3). d) Advocate with the MOH to develop a rational plan for HR transfers. e) Expand mHealth to community linkages, in particular use of mobile phones for texting reminders for clinic visits to patients, and also to ease activists’ linkage to HF staff. “My greatest challenge is to get patients counseled at all points of entry of this facility. Staff have too much workload and some health workers just forget or do not think that it is important.” (HF in-charge) 20 Performance Evaluation of the CHASS-SMT Project Constraints/Weaknesses Recommendations 7) Limited physical space at some HFs is a barrier to adequate counseling and consequently to retention. 8) Stigma and discrimination still exists. 9) Babies who test HIV-positive are transferred to ART clinics to which mothers sometimes do not report. f) Pilot and expand HIV service coverage in underserved rural areas using mobile clinics to dispense ARVs and provide HCT and ANC/PMTCT. g) Partner with employer-based health programs to provide technical support to institute worksite HCT, dispense ARVs and assure data collection and reporting to SDSMAS. Question 6: What has been the most effective method found to improve retention? As noted above under question 5, retention strategies included active case-finding, GAAC activities, community mobilization, and psychosocial support from lay counselors, PLHIV and M2M groups. In many HFs, these activities were executed in a complementary manner. Active case-finding was the most effective strategy. The activities of the GAACs were moderately effective, followed by the activities of the PLHIV and M2M groups. Community mobilization by activists and lay counselors also played a supportive role. GAACs are cheaper and sustainable, but challenges related to stigma need to be addressed for this model to be effective. More than 50,000 ART defaulters were tracked by the case managers during the course of the project, and at least 50 percent were returned to the HF (Figure 4). This also implies that at least 40 percent of current ART clients were at one time defaulters. Without active case-finding, many of these would have been lost to follow-up. By each GAAC group member picking up medication once every three or six months in a rotational manner, the burden of transportation for an individual member is reduced. In addition to this, the GAAC members provide psychosocial support to each other. These activities reduced the chances of defaulting. The role of GAACs was evident in Tete province, where almost 90 percent of GAACs supported by the project still existed (GAACs formerly formed by MSF). According to the data in Project Quarter 2 Progress Report of Year 4 (Pages 65-66), the percentage of identified (by project team) ART defaulters in Tete was about 11 percent, as compared to 24 percent in Sofala and 45 percent in Manica. Whereas these data are not necessarily of good quality due to lack of proper records on retention, their trend showed the relevance of the GAACs to retention. Further, Tete also had the highest number of PLHIV groups, whose activities partly explained the low percentage of defaulters in that province. Question 7: To what extent has HF management (improved patient flow, etc.) improved over the life of the project? Management of HF patients and clients improved significantly over the life of the project. In medium- to large-sized facilities visited by the evaluation team, patients or clients were many, but well sorted according to service being sought, organized and orderly, and certain priority cases were attended to “Case finding and GAACs are good strategies in rural health facilities; they worked in a complementary way; case finding is much effective.” M&E to CHASS MANICA Performance Evaluation of the CHASS-SMT Project 21 first (i.e., improved fluxogram). The nurses reported the EP to have helped them to learn to organize the processes. In Gondola and Ponta Gea hospitals in Sofala, where the project piloted patient flow reengineering, the health workers reported that patient flow had improved significantly in the last year, since EP started training them on the 5S strategy. The project staff noted that in some of the large-volume facilities they helped to reorganize, the patient waiting times were reduced by more than half. The one-stop shop models for PMTCT and MCH services and TB services have improved on patient flow, as they provide all services in one place. In addition, the formation of GAACs, in which the members pick up medications on behalf of the others, has reduced on volume of patients at the clinics. Case managers were also helping to register or direct patients at the facility. Findings about the challenges are summarized in Chart 6. Chart 6: Challenges associated with improvement of health facility management Challenges Recommendations 1) Many HFs have limited physical space to sort patients. With projected increase in demand for services, this will create more patient flow challenges. 2) The improved patient flow is challenged by staff rotations/transfers and staff work overload, especially in facilities run by two or three health workers. 3) There is limited understanding of factors associated with high patient turnout at facilities at certain times and particular days. 4) The reengineering strategy was implemented late in the project, and therefore the long-run sustainability and efficiency are not known. a) The GRM, with support from partners, should invest in improving the physical space of HF clinics, labs, pharmacy and reception areas. b) Understanding reasons for poor patient flow: The project should support the HFs to collect data on factors that explain high patient turnout at certain times and particular days but not others. c) Put a training mechanism in place for sustainability of improvements: Institute a peer-to-peer training protocol for 5S at the HFs. This will help with training of new staff members and contribute to sustainability. Question 8: To what extent is data management capacity built at the HFs with regard to HIV/AIDS registry data? Error rates of filling in the HIV/AIDS registry decreased substantially, and correct filing of the registers and summarization of the registry records for data entry improved moderately. The quality of the data from HFs in Sofala, Manica and Tete provinces passed the data quality audit conducted by the External Data Quality Assessments evaluation team in late 2014. The audit team was from the MOH, UCSF, ITECH and JSI/MCHIP. On a score scale of 0 (poor) to 3 (excellent), HFs in Sofala scored 2.6, while facilities in Manica and Tete scored 2.1. However, the improvements in pediatric care registries were below desirable levels. The project also provided filing cabinets in most of the health facilities. Frequent district-level strategic information meetings led to improved HF data quality, especially through reorientations for clinicians to improve missed or incorrectly filled fields or indicators in the registers. The completion and sharing of data or registries also improved and is likely to continue without the project’s support. In some HFs, the registers are stored in a central location where all staff can access them for checks and consultations. Some informants praised the project’s contribution to improved data management, as noted below. “The team (CHASS team, SDSMAS, DPS team) helped in improving the triage process and record reviews and hence easy to identify patients who are eligible for ARVs.” (Director, Gondola hospital) 22 Performance Evaluation of the CHASS-SMT Project Findings about the challenges are summarized in Chart 7. Chart 7: Challenges to data management capacities at health facilities Challenges Recommendations 1) The insufficient numbers of HF staff and the high rate of turnover will always be a limiting factor. In HFs where the trained staff moved out, there were still some challenges with correct filling in of some forms and the registry. 2) There are too many forms/registries to manage without ignoring some clinical activities. 3) In some HFs, the case managers acted as records personnel to fill in some of the registers. However, these personnel will no longer be available in most HFs after project closure. 4) The electronic databases, especially the EPTS were still in the pilot phase. DPS approval for EPTS rollout was said to be slow. To some extent, the project was forced to create a parallel system. a) Advocate for and support MOH in rolling out the EPTS to moderate- to large-volume HFs (Chart 5 #1). b) Put a training mechanism in place for sustainability of improvements: Institute a peer-to-peer training protocol for data management and quality improvement at HFs. This will help with training of new staff members and contribute to sustainability. c) The MOH should consider reducing the number of forms/registers. d) Develop data management and use materials: Jointly with the MOH, develop materials to explain data management and use to HF health workers. This will help them understand the purpose of ensuring high-quality data. e) Hire records personnel at medium-sized HFs and records and data management personnel at hospitals. Question 9: To what extent has knowledge (community and health worker) and utilization of GBV services increased over the life of the project? Health worker knowledge: CHASS trained focal points at the provincial and district levels and trained activists who could refer cases to HFs. DPS now has a team to ensure prophylaxis for STIs, HIV and any lesions, provide psychosocial support to victims, and refer cases to police to initiate the legal process. The project provided logistics for SDSMAS to hold community meetings and health talks. In addition, the project supported training for all district-level MCH nurses. Training was also conducted with the police, members of legal courts and community leaders. Most HF staff were aware of GBV, but only a few were actually assessing to identify GBV victims. Assessment was mainly done by MCH nurses. Post-exposure prophylaxis kits and job aids were available at facilities visited by the evaluation team. There is increased coordination between the police, community and HFs as noted by informants. Community knowledge: Information to the community about GBV was provided through CBO activities. The project provided training and logistics for CBOs and activists to promote GBV information in the communities. “I personally benefited from training in dispensing ARVs to mothers and they also taught me how to apportion my time to correctly document the processes without straining.” (Nurse at HF in Gondola) “In the beginning, I personally benefitted a lot; I learned how to maintain the registry books in order. If today, the HFs and I manage this well, it is because CHASS worked hard with us on this aspect by correcting errors, regular review meetings, with M&E and HF health workers.” (SMI Nhamatanda Hospital) Performance Evaluation of the CHASS-SMT Project 23 Other findings are summarized in Chart 8. Chart 8: Challenges and weaknesses in scaling-up GBV services Challenges/weaknesses Recommendations 1) Weak community linkages: Activists concentrate on retention activities and little on sensitization. 2) Reluctance by violence victims to identify the violators or report to police 3) At the community level, violators pay, and the victims do not go to the HF for treatment for fear of being asked for the identity of the perpetrator. 4) Trained staff are often transferred to other HFs. 5) Some districts do not consider the GBV program a priority. 6) Infrequent supportive supervision visits to outlying HFs by trained teams a) Increase community engagements and sensitization; work with Gender Committees and Social Assistants from the Ministry of Women and Social Action. b) Train all clinical staff on GBV issues so they can identify victims. c) Plan for and conduct regular reviews of services for GBV cases to assure quality. d) Involve a specialized CBO or NGO to support GBV activities. All logistics should be channeled through such a CBO/NGO. e) Train health workers to integrate assessment for GBV at all HF points of care. f) The GRM/MOH should develop a better staff development and retention package or plan (see Chart 3 #3). Question 10: What are the benefits and challenges of the activities’ model of working with government (at the provincial and district levels) through sub-agreements? Benefits: The project signed SAs with the provinces, districts, CBOs and training institutions. Their management evolved slowly over time, but even so, SAs enabled the development of systems, and capacities of the districts and provinces were developed. The management of the SAs also helped to strengthen the health program, management and ownership of the project’s results. For many SDSMAS, the percentage of SA activities executed annually increased from below 40 percent to above 60 percent. “I have learned a lot because of operating the sub-accords; I can operate all activities and associated budgets fully, including reporting, without any further support.” (DAF Manica SDSMAS) “During the first visit of victim to health services, when she reports the case, we take the history provide the appropriate treatment and then we accompany her to the police, where she reports the case. The police does the crime process and they call a public lawyer from the Instituto de Patrocinio de Advocacia (IPAG). The lawyer prepares case to present to the tribunal for a determination.” “Before the police did not refer victims to us for they did not understand that victims need to be treated; now the coordination has improved and we are working together, now they are referring the victims to health facility within the recommended 72 hours.” (SDSMAS focal point) “At first we didn’t have a focal person for GBV; now (we) have GBV focal points and trained staff, people can easily be attended to in hospitals” 24 Performance Evaluation of the CHASS-SMT Project The DPS and SDSMAS informants also reported confidence in disbursement or reimbursement processes; as a result, they were able to implement planned activities without worrying about a lack of funds or refunds. Challenges: The activities defined in SAs are always inflexible in the short-run and did not allow for correction of planning errors. Given that the SDSMAS staff are involved in many other activities, sometimes modifications in work plans are necessary. Further, SAs were based on project activities and had no allowance for executing any activities outside their scope. The SDSMAS and DPS also complained of the substantial delays in reimbursements or disbursements. This was mainly due to the associated financial management restrictions. Where the project staff were not available to review the district reports, the delay in reimbursements was said to be much longer. As noted in project reports, the number of districts submitting their reports on time was low, mostly due to a lack of adequate training of SDSMAS administrative staff, and insufficient support from project EPs and Equipas de Apoio to districts to retain and process the necessary documents and justify the activities and products to support the respective invoices. The verification process undertaken by the project staff has also been very slow, with some reports being returned for further review and clarification. Recommendations: 1. Limit financial sub-agreements to a few districts: This will enable the project staff to monitor and support the districts more efficiently and document best practices to share with a set of new districts. 2. Establish a budget item for non-HIV-related services: The project should consider establishing a fund for emergencies or disease outbreaks that are not necessarily within the scope of CHASS. 3. Involve the DPS and SDSMAS in the initial design of the projects: They should play an active role in all project stages. The transition plan should be agreed on with the DPS and MOH and be explicitly incorporated in project documents, with clear strategies for sustainability. 4. Improve the reimbursement process: The project should have an adequate number of dedicated staff for reviewing and quickly turning around activity reports from CBOs, DPS and SDSMAS. 4.2 OVERALL CONCLUSIONS The project achieved some of its key objectives including a significant contribution to the HSS in the three provinces. All components of HSS improved significantly, although all did not reach the desired levels. The capacity of the SDSMAS to manage health programs improved almost two-fold, which was seen by all as a major success. The ability to manage sub-agreement activities also improved. Similarly, the supply chain of drugs and lab commodities, and the transport logistics for CD4 and PCR samples improved. Nonetheless, there is still work needed to develop the M&E systems and financial management to desired levels. Most DPS and SDSMAS want a project similar to CHASS to continue. The project over-achieved on most of its targets for clinical service delivery except for the recruitment of infants in ART and retention rates in ART programs. Overall, the project almost tripled the number of patients on ART in the last four years, which is a substantial achievement given the nature of the health system. The ART and Option B+ services were expanded to rural health facilities. The insufficient number of staff at the health facility level and the high rate of scale-up of services may have compromised the quality of counseling. The long distances traveled by patients’ remain a challenge to retention efforts. Performance Evaluation of the CHASS-SMT Project 25 4.3 FUTURE DIRECTIONS These recommendations for future directions were derived from lessons learned from the project and the findings from the evaluation. Recommendations Health Systems Strengthening 1) Involve the DPS and SDSMAS in the design of the project; they should play an active role in all stages of the project cycle. The transition plan should be agreed on with the SDMAS, DPS and MOH and be explicitly incorporated in the project documents starting in the first year of the project. Face￾to-face interactions should be a preferred method of communication with the DPS and MOH. The MOH must play its role as agreed on with the project to ensure its deliverables, success and sustainability. 2) Introduce a component of operations research in project activities and local capacity building. 3) Provide logistics (including the printed tools, manuals and provision of computer tablets, and financial logistics for DPS’ travel to districts and DPS/SDSMAS travel to HFs) for institutionalization of the graduation path and clinical tutoring tool by DPS. 4) Put a training mechanism in place: This could include a peer-to-peer model in which trained health workers tutor at nearby HFs; work with training institutions to establish courses in HSS for all health workers; and training a group of national and district-level trainers. 5) Expand mHealth in project areas: Most aspects of data management and patient follow-up can be conducted using mobile technology. Tablets or PDAs could be used by CCMs or activists to input client information and for text message reminders for patient clinic visits, including mothers returning babies for HIV testing and adherence to ARVs. Working with a mobile phone company, the tablets can also be used for online tutorials for health workers. This has worked well in agricultural health systems in other countries in Africa. 6) Advocate for and support the MOH in rolling out the EPTS to moderate- to large-volume HFs. In the meantime: Encourage and train health workers to correctly fill in FILAs and other forms on retention and file them appropriately, and initiate conversations with DPS about the roll-out plan for EPTS once the MOH approves it, including but not limited to data validation processes and tools, plan and terms of reference for data managers, and the use of data from EPTS for reporting and the QI program. Clinical Services (enrollment and retention) 7) The GRM/MOH should develop a better staff development and retention package or plan, e.g. to ensure that staff stay in post for at least three years to enable consolidation of experience; it should also increase staffing levels to meet increasing client loads, especially for HIV/AIDS services. 8) The GRM/MOH, with support from partners, should invest in improving HFs’ physical structure and space. 9) Implement an augmented MOH strategy of a lay counselors’ workforce by recruiting and training lay counselors or retraining case managers and APES as lay counselors to provide services. Some counselors should be stationed at the HF and one in each community, providing services such as: adequate counseling for pre-ART care and ART initiation; community sensitization against HIV stigma; psychosocial support to HIV-positive pregnant women to disclose HIV status to their partners and to return babies on schedule for HIV testing and ART initiation; targeting male partners of pregnant women for HIV testing; generating a list of defaulters at HFs and encouraging 26 Performance Evaluation of the CHASS-SMT Project them (directly and through phone text messaging) to visit a HF or mobile clinic; tracking HIV￾positive children to deliver PCR results; psychosocial support to GAACs and individual PLHIV; and adherence counseling. Communication between lay counselors and other health workers could be through mobile phones. Data collection could also be done through mobile phones or tablets. 10) Reinvigorate linkages between HF and community-based services, and promote regular information exchange for inter-referrals and health information messaging to communities:  Conduct operations research to map linkages and measures to improve communication and information exchange.  Explore opportunities to partner with private telephone companies to introduce mHealth to enable direct communication between HFs and the community–CBOs, faith-based organizations, activists, traditional healers and others–using mobile phones or phone credit, patient monitoring devices, PDAs, and tablets, to ensure continuous information exchange, data collection and reporting, and health messaging.  Pilot and expand HIV service coverage in underserved rural areas using mobile clinics with mHealth to provide: HCT, HIV care, ARVs, ANC/PMTCT CD4, TB screening services and health education. 11) Integrate HIV prevention, care and support in adolescent health services at HFs, with emphasis on adolescent girls. Develop a school-based integrated health program that includes adolescent health and HCT services. 12) Improve HIV testing and ART initiation among infants through training all HF health workers in: PCR sample-taking to provide HIV testing in all pediatric entry points, including the children of patients on ART; linkage of records in labor/delivery units and at-risk child consultations (improve pediatric care registries); and adequate counseling of pregnant women about early ART initiation for infants, HIV status disclosure to partners, and HIV stigma; and DBS processing and storage and distribution of SMS printers to rural HFs. Community Linkages 13) Reinforce community-HF linkages and introduce mHealth in mobile clinics and HFs (see Recommendation 8), and, in addition, promote use of text messaging to patients to remind them of clinic visits, adherence and drug pickup. 14) Partner with employer-based health programs to provide training and technical support to institute worksite HCT testing, referral and dispensing ARVs, particularly for male workers. GBV Services 15) Strengthen the GBV community linkages and work with an NGO/CBO that specializes in gender issues to support community activities. Sub-agreements 16) Establish a budget line item for non-HIV-related services to fund emergencies or disease outbreaks that are not necessarily within the scope of CHASS. 17) Improve the reimbursement process: The project should have an adequate number of dedicated staff for reviewing and quickly turning around activity reports from CBOs, DPS and SDSMAS, as well as reimbursements. Review internal financial controls to make them realistic for GRM collaboration. Performance Evaluation of the CHASS-SMT Project 27 4.4 SUMMARY OF RECOMMENDATIONS The recommendations are summarized in the following table, indicating what USAID partners and the MOH are responsible for and where the responsibility is shared. Table 4: Recommendations for future programs Health Systems Strengthening Partners MOH 1) Involve the DPS and SDSMAS in the design of the project. They should play an active role at all stages of the project through face-to-face interactions. The transition plan should be agreed on with the DPS and MOH, and its implementation should begin in year 1 of the project. The MOH should actively play its role as agreed to ensure its deliverables, as well as success and sustainability. √ √ 2) Introduce a component of operations research in project activities and in local capacity building. √ 3) Provide logistics for institutionalization of the graduation path and clinical tutoring tool by DPS. √ 4) Put a training mechanism in place. This could include (a) a peer-to-peer model in which senior/trained health workers can tutor at nearby HFs, (b) working with training institutions to establish courses in HSS for all health workers; and (c) training a group of national and district-level trainers. √ √ 5) Expand m-Health in project areas. √ √ Clinical Services (enrollment and retention) 6) The GRM/MOH should introduce a better staff development and retention package or plan to promote staff retention in post for at least three years to allow consolidation of experience. Increase staffing levels to cope with increased client load, especially for HIV/AIDS services. √ √ 7) Reinvigorate linkages between HF and community-based services, and promote regular information exchange for case-finding, inter-referrals and health messaging to communities. √ √ 8) Implement an augmented MOH strategy of a lay counselors’ workforce by recruiting and training lay counselors or retraining case managers and APEs as lay counselors to provide various services. √ √ 9) Improve HIV testing and ART initiation among infants through training all HF health workers in providing infant HIV testing at all points of entry at the facility, including children of patients in ART/pre-ART care. √ √ 10) Pilot and expand HIV service coverage in underserved rural areas using mobile clinics with mHealth. √ √ 11) Integrate HIV prevention, care and support in adolescent health services at HFs, with emphasis on adolescent girls. √ 12) Reinforce QI and QA: Future projects should strengthen these components for clinics, pharmacies and laboratories. √ Community Linkages 13) Reinforce community-HF linkages and communication and expand the number and type of CBOs, such as faith-based organizations, traditional birth attendants, APEs and traditional healers. √ √ 14) Research cultural barriers to develop culturally sensitive and acceptable strategies to promote behavior change. √ √ 15) Partner with employer-based health programs to provide HIV services, particularly for male workers. √ 28 Performance Evaluation of the CHASS-SMT Project GBV Services 16) Strengthen the GBV community linkages and work with an NGO/CBO that specializes in gender issues to support community activities. √ Sub-agreements 17) Establish a budget item for non-HIV-related services: This can fund emergencies or disease outbreaks that are not necessarily within the scope of CHASS. √ Performance Evaluation of the CHASS-SMT Project 29 5 REFERENCES Clinical HIV/AIDS System Strengthening RFA 2009 CHASS SMT annual work plans 2011 CHASS SMT annual work plans 2012 CHASS SMT annual work plans 2013 CHASS SMT annual work plans 2014 CHASS SMT annual work plans 2015 CHASS SMT Quarter 1 Report 2011 CHASS SMT Quarter 2 Report 2011 CHASS SMT Quarter 3 Report 2011 CHASS SMT Quarter 4 Report 2011 CHASS SMT Quarter 1 Report 2012 CHASS SMT Quarter 2 Report 2012 CHASS SMT Quarter 3 Report 2012 CHASS SMT Quarter 4 Report 2012 CHASS SMT Quarter 1 Report 2013 CHASS SMT Quarter 2 Report 2013 CHASS SMT Quarter 3 Report 2013 CHASS SMT Quarter 4 Report 2013 CHASS SMT Quarter 1 Report 2014 CHASS SMT Quarter 2 Report 2014 CHASS SMT Quarter 3 Report 2014 CHASS SMT Quarter 4 Report 2014 OIG findings and report of CHASS SMT audit Public Financial Management Risk Assessment Framework reports for the four provinces CHASS USAID progress report data: Data from President’s Emergency Plan for AIDS Relief (PEPFAR) annual and semiannual reports, including both community and clinical data (2011, 2012, 2013, 2014) Partner- and USAID-conducted data quality assessments CHASS-SMT Project Strategy, August 9, 2012 MISAU (2013). Mozambique National Treatment Acceleration Plan 2013-2015. Maputo, Mozambique: Ministerio da Saude Mocambique. Performance Evaluation of the CHASS-SMT Project 31 ANNEXES Performance Evaluation of the CHASS-SMT Project 33 ANNEX I: SCOPE OF WORK Performance Evaluation of USAID’s Clinical HIV/AIDS System Strengthening project in Sofala, Manica and Tete (CHASS-SMT), implemented by Abt Associates Inc. (with sub-award to FHI 360) Anticipated Period of Performance: August 2015–December 31, 2015 BACKGROUND USAID/Mozambique’s Clinical HIV/AIDS System Strengthening Project (CHASS) is a results-oriented five-year project to improve HIV clinical services in Manica, Niassa, Sofala and Tete3 provinces within a strengthened, comprehensive primary health care system. This project was designed to address the HIV situation in Mozambique, and focused on three components: 1) Improving service quality in six important areas: HCT, laboratory services, PMTCT, adult care and treatment, pediatric care and treatment, and the prevention, diagnosis and treatment of HIV-TB co-infection 2) Enhancing program linkages and integration to provide a continuum of accessible services, including MCH and RH services, within facilities and between facility and community-based services 3) Creating stronger and more sustainable Mozambican systems and institutions At the time of project design, the average HIV prevalence was estimated at 16 percent nationwide (ANC survey in pregnant women aged 15-49 years). Nearly 1.6 million people were living with HIV in Mozambique, and at the time of writing the request for applications, nearly half of all HIV-infected (48.4 percent) were identified as having active tuberculosis. The burden of HIV is considerably higher in Mozambique’s Central Region, with a HIV prevalence of 18 percent,4 compared to the national prevalence of 16 percent.5 Within the Central Region, Sofala and Manica provinces have the highest prevalence rates, 23 percent and 16 percent respectively. Tete was identified as having a very mature epidemic and existing infrastructure that was unable to accommodate the numbers of patients requiring care and treatment. Niassa is considered a particularly underserved province, with vastly inadequate infrastructure, but with a much lower HIV prevalence of approximately 3 percent. To address these issues, two separate agreements were awarded to two implementing partners: FHI 360 (CHASS Niassa) and Abt Associates Inc. (CHASS-SMT, for Sofala, Manica and Tete). Table 1 provides details of each award. Funding details for CHASS-SMT: See Attachment 1–Table 1. Both activities support USAID/Mozambique’s Country Assistance Strategy’s (CAS 2009-2014) priority goal number three, “Improved health of Mozambicans,” and more specifically contribute to the following focal areas in USAID’s Health Results Framework: 3 The CHASS project in Niassa, implemented by FHI 360, will be evaluated through a separate purchase order. 4 Statistics included here are from the original request for applications; however, the burden of disease continues to be high in the Central Region. 5 Relatório sobre a Revisao dos Dados de Vigilância Epidemiológica do HIV, Ronda 2007. Grupo Técnico Multisectorial de Apoio A Lutta Contra o HIV/SIDA em Moçambique. Fevereiro, 2008. 34 Performance Evaluation of the CHASS-SMT Project  Improved access to and delivery of quality integrated services  Increased adoption of healthy behaviors and informed use of services  Strengthened health systems While the two activities are aligned in terms of the results they are expected to achieve, each activity has its own approach for achieving those results. CHASS-SMT aims to achieve the following objectives related to its overall goal of improving public sector HIV clinical services in Sofala, Manica and Tete, within a strengthened comprehensive primary health care system: 1) Strengthen Mozambican health systems and institutional capacity to provide high-quality services and ultimately receive and manage direct support from the U.S. Government; 2) Improve integration of HIV and related primary health care services and linkages between the community and the health system; 3) Increase demand, use and provision of high-quality HIV services; and 4) Increase coverage of HIV care and treatment and PMTCT services. A key shift in the period since the two CHASS activities were awarded was the development of an HIV Acceleration Plan by the MOH. In 2011, Mozambique signed the Political Declaration on HIV/AIDS: Intensifying our Efforts to Eliminate HIV/AIDS. The “Mozambique HIV and AIDS Response–Strategic Acceleration Plan 2013-2015” was created to respond to this commitment to an end of AIDS. This three-year strategy reflects the united vision of all stakeholders to achieve a Generation Free of AIDS in Mozambique, focusing on three major goals: 1. Increase the percentage of eligible HIV-infected adults and children receiving antiretroviral therapy to 80 percent by 2015; 2. Reduce the rate of transmission of HIV from mother to child to less than 5 percent by 2015; and 3. Reduce the number of new infections by 50 percent by 2015. Both CHASS SMT and Niassa also include programming for the prevention of GBV, with a total life-of￾project funding of approximately $1.5 million. GBV is a pervasive social behavior that leads to disability and death and undermines the quality and productivity of community life as a whole. GBV prevention activities are implemented in all four provinces to: (i) strengthen the quality of care provided to victims of GBV presenting at HFs; (ii) promote a continuum of care through supporting functional links between facilities and community-level support services for GBV survivors, in conjunction with FHI 360’s PCC project, and (iii) conduct advocacy and sensitivity training among DPS and district health management teams and influential community members to support implementation of the national GBV strategy. While GBV activities make up only a small part of the budget and activities under CHASS Niassa and CHASS-SMT, it is an area that requires increased understanding and more information about what strategies have worked to improve access to GBV services. 1. EVALUATION PURPOSE AND OBJECTIVES This evaluation comes during the fourth year of project implementation. The purpose of this process/performance evaluation is to determine how the activity is performing relative to its objectives, as well as to provide an objective view of progress toward the expected results. The outcomes of this evaluation will provide information to be included in future activity design, as USAID begins to consider future system-strengthening and clinical service delivery activities. Further, results will be used to identify gaps in project performance and to help USAID and the implementers determine what changes may be necessary to solidify progress during the remaining project period. The main audience for this Performance Evaluation of the CHASS-SMT Project 35 evaluation is USAID; however, results will also be shared with CHASS SMT, the MOH, other U.S. Government agencies, other implementers and other stakeholders. The objectives of the CHASS Evaluation are to: 1. Assess CHASS SMT achievements, emphasizing objectives and activity- and project-level results; 2. Identify implementation successes, as well as any internal and external constraints that hindered the implementation of planned activities; and 3. Propose recommendations for future directions of CHASS SMT and for future activities in system strengthening and service delivery to support improved performance in addressing the HIV epidemic, in line with the GRM HIV/AIDS acceleration plan and new PEPFAR guidance. An additional goal, in terms of evaluation process, is to use mobile technology, when possible and appropriate, to increase the efficiency, transparency and accuracy of performance data, and to take advantage of multiple data sources (e.g. pictures, videos, GPS data). Electronically collected data will ultimately feed into a platform that is being developed simultaneously with this evaluation. Moving forward, USAID intends to incorporate mobile technology into more and more evaluations, utilizing the new platform to organize, analyze and report data. The CHASS evaluation will be one of the first evaluations to feed into the new platform and will help contribute to the platform’s development. 2. EVALUATION QUESTIONS The USAID team has developed a set of 10 evaluation questions, each of which belongs to one of three main evaluation question areas: (1) project achievements and challenges, (2) linkages, and (3) HSS. A complete list of the evaluation questions is shown below. The awardee will ensure that each question is answered fully and completely, underscoring both positive and negative outcomes. All evaluation questions must be answered with a gender focus, going beyond sex-disaggregated information, where applicable, and analyzing outcomes in relation to gender and sex. The awardee will apply, administer and evaluate each question separately. The findings for each question will be addressed in a separate section of the evaluation report. A complete list of the evaluation questions, organized by question area, is as follows: Question Area 1: Project Achievements and Challenges 1) What are the activities’ strengths and weaknesses, as seen by implementing partner staff, DPS, District Directorate of Health and Social Welfare (DDS and Medical Chefe), HFs, USAID, and the USAID-funded PCC project, and how can weaknesses be improved, according to these stakeholders? 2) Where has the most progress been seen in strengthening systems (e.g., planning, financial management, supply and logistics, information systems)? 3) Which activities or project components will be most feasible to transition from the project to the GRM? Question Area 2: Linkages 4) To what extent has the project been able to create and strengthen linkages between HFs and communities to allow for increased service uptake, specifically in the areas of: a. Community-based counseling and testing to treatment (for both men and women); b. Retention of pre-ART and ART patients (through the use of adherence groups, active case-finding and other community groups); and c. Knowledge, demand and access of services by men 36 Performance Evaluation of the CHASS-SMT Project 5) What constraints have the activities faced in improving retention of patients in pre-ART and ART? 6) What has been the most effective method found by the project to improve retention? Question Area 3: Health System Strengthening 7) To what extent has HF management (improved patient flow, etc.) improved over the life of the project? 8) To what extent is data management capacity built at the HFs with regard to HIV/AIDS registry data? 9) To what extent has knowledge (community and health worker) and utilization of GBV services increased over the life of the project? 10) What are the benefits and challenges of the activities’ model of working with government (at provincial and district levels) through SAs? 3. EVALUATION DESIGN AND METHODOLOGY This performance evaluation will be based on a non-experimental design. The awardee will conduct an evaluation that, to the extent possible, uses a mixed-methods approach. However, the awardee will mostly utilize qualitative data collection and evaluation methods in the evaluation. This awardee will not focus on evaluating quantitative outputs, but will be required to use USAID’s monitoring data for triangulation purposes. Data Collection Methods: Because this will be a performance evaluation that utilizes mostly qualitative data, the awardee will focus on using key informant interviews, stakeholder consultations, focus group discussions, observational analysis and beneficiary interviews as the main data collection methods. The awardee will consider a variety of methodologies to ensure that the evaluation questions are adequately addressed. Mobile Data Collection: In this evaluation, the awardee will use multi-channel data collection, using mobile technology when possible and paper-based methods when mobiles are not appropriate. The USAID/Mozambique mission has limited experience using mobile technology to conduct evaluations, either on its own, or through evaluation contractors. Generally, evaluation tasks, such as those mentioned above, have been paper-based. When using this paper-based system, results have to be collated, which takes time; also, transcription is sometimes inaccurate, and costs can be substantial. New technologies can make data collection (both quantitative and qualitative) easier, faster, more accurate and cheaper. It is becoming more commonplace that researchers in the field use simple mobile phones, smart phones and tablets to carry out their surveying tasks. USAID/Mozambique is interested in leveraging these new technologies and methods in this evaluation. In using mobile technology for the CHASS Evaluation, USAID has four key goals: 1) Efficiency: collecting and sharing information as quickly and reliably as possible; 2) Transparency and sharing: using open data to share field-level data with key stakeholders; 3) Accuracy: collecting information in real time to best capture realities on the ground; 4) Cost-saving: applying mobile devices to decrease expensive transcription and enumerator costs, when possible and appropriate. The awardee will conduct a desk review prior to arrival in country to ensure that evaluators are familiar with existing data and project progress according to written documentation. Data collection will take place in Sofala, Manica, Tete, as well as in Maputo. Data collection in Maputo will consist mostly of desk review and related data collection with USAID and implementing partners’ Performance Evaluation of the CHASS-SMT Project 37 staff located in the city. Each evaluation group–comprised of members of the awardee’s evaluation team–will cover two provinces. (Details on the evaluation team and evaluation groups can be found below in the “Evaluation Team Composition” section.) The awardee will create and finalize data collection methodology and instruments that must be approved by USAID prior to any key informant interviews or site visits. The selected awardee will be required to use a mobile data collection tool for the evaluation. Mobile devices with GPS functionality will need to be provided by the selected firm. Additionally, for purposes of quality assurance, USAID will be provided with all raw data collected during the evaluation. Existing USAID and external reports, data and documentation to be used: The USAID/Mozambique IHO will provide the awardee with all necessary background documents (request for applications, project’s annual work plans and reports, MOH health sector strategy, etc.) and data. The awardee will conduct a desk review of the documents and data, some of which are written in Portuguese, and may request further documents from USAID and/or the CHASS projects as needed. Below is a list of documents that are currently available for use and review:  Clinical HIV/AIDS System Strengthening Request for Applications  CHASS SMT annual work plans and quarterly and annual reports  Site visit reports  OIG findings and report of CHASS SMT audit  Public Financial Management Risk Assessment Framework reports for the four provinces  MOH health sector strategy  MOH HIV Acceleration Plan  Data from PEPFAR annual and semiannual reports (both community and clinical data)  Analysis conducted based on PEPFAR annual and semiannual reports  Partner and USAID conducted data quality assessments Proposed Stakeholders: For the purposes of this evaluation, health sector stakeholders in Mozambique include, but are not limited to, the MOH, DPS, SDSMAS, U.S. Government agencies (USAID, CDC, DoD, State Department), implementing partners, CBOs, health care workers and clients who receive services at the facilities that the projects support. The awardee will hold meetings with the following groups:  MOH: 2-4 staff from DNAM (Departamento Nacional de Assistencia Medica)  DPS (Sofala, Manica, Tete): 2-3 staff each  DDS: 2-4 district DDS leads in each of the four provinces  Health facility staff: 2-3 staff in 3-5 health facilities per province  Implementing partner staff in Maputo (CHASS SMT and PCC): 4-6 staff each  Implementing partner staff in the provinces (CHASS SMT and PCC): 4-6 staff per project per province  CBO: Eight CBOs supported by the project with 1-2 members per CBO Table 2 below highlights illustrative data collection methodologies that the awardee may use for this evaluation, organized by evaluation questions (See Attachment 2, Table 2). Data Analysis and Disaggregation: The exact data analysis methods used may differ for and within each evaluation question and may depend on the data available. The awardee will specify the exact methods 38 Performance Evaluation of the CHASS-SMT Project to be used for each evaluation question in the inception report, which will be discussed with and approved by USAID. In general, the awardee’s analysis of quantitative and qualitative data will consist of four components: (1) data reduction (i.e., open/initial coding, focused coding, axial coding); (2) displaying data; (3) drawing conclusions; and (4) verification through data triangulation. Where applicable and feasible, the awardee will disaggregate all data by sex and evaluate the disaggregated data for any gender-related differences. All qualitative and quantitative raw data, both soft- and hard-copy, will be owned by USAID. At the end of the evaluation, the awardee will release all materials to USAID, including summaries, interview and focus group data, databases, and notes, and any other data collection tool(s) used for this evaluation. 4. DELIVERABLES Aside from applicable survey tools, the awardee will submit and present all deliverables in English. However, certain presentations and discussions may be conducted in Portuguese. The awardee’s evaluation team will be responsible for providing the following deliverables to USAID for approval: a) Inception Report: The awardee will submit an inception report, which must include the following:  Detailed evaluation design and methodology  Identifying data to be collected, including precise explanation of data collection methods that will be used for each evaluation question  Sampling plan  Proposed list of key informants, focus group participants and workshop attendees  Data collection tools developed and pre-tested  Detailed data analysis plan  Detailed evaluation schedule and logistics, including debriefing with USAID  Roles and responsibilities of each evaluation team member The submitted inception report must be approved by USAID prior to beginning key informant interviews, focus discussion groups, site visits, etc. b) Midpoint Briefing and Out-briefing/PowerPoint Presentation with USAID: The awardee will provide a midpoint briefing to USAID during the in-country evaluation period, preferably midway into the data collection process. c) Draft Evaluation Report: The awardee will submit a draft evaluation report, incorporating feedback and comments received from the debriefing. The awardee will also submit any raw data (qualitative and quantitative) collected. After receiving the report, USAID will have 10 calendar days to provide the team with one set of written comments. This may include an open source website that shares the multimedia data collected (pictures, videos, GPS data) and mapping that allows USAID to track performance, progress and challenges visually. d) Final Report (for USAID and for the public) and Final Presentation: The awardee will submit a final report after receiving final comments from USAID/Mozambique. This report should not exceed 30 pages in length (not including references, appendices, etc). The final format (see below) must meet the quality standards outlined in the Evaluation Policy (http://www.usaid.gov/sites/default/files/documents/1868/USAIDEvaluationPolicy.pdf, attached at Annex A is the standard and required USAID Report Format Template), and must include the following: Performance Evaluation of the CHASS-SMT Project 39  Table of contents  Acronyms  List of tables  List of traphs  Executive summary: not to exceed five pages. Brief summary of project purpose and background, key evaluation questions, methods, findings and recommendations  Introduction and background: purpose, audience, and synopsis of task, brief overview of the project, USAID program strategy and activities implemented in response to the problem, brief description of implementing partners  Evaluation methodology, limitations and gaps  Findings, conclusions and recommendations. Each evaluation question should be answered in its own section of the report. This section should include recommendations to USAID for future project design.  Issues: Provide a list of key technical and/or administrative issues, if any.  Future directions, to inform the design of any new intervention  References, including bibliographical documentation, meetings, interviews and focus group discussions  Annexes should include the evaluation scope of work and any amendments, evaluation tools, schedules, interview lists, tables, information sources, statements of differences, and any other information and data that was not required in the report. The awardee will submit the final report electronically, in English, and the report will, at a minimum, contain background, rationale, methodology, the evaluation’s key objectives, evaluation questions, major findings and recommendations/conclusion. The awardee will develop a final presentation based on the final report and will deliver that presentation to the USAID mission. The awardee will also develop a public version of the final report, to exclude any potentially procurement-sensitive information, which will also be submitted electronically and in English. This public version of the final report is meant for dissemination among implementing partners and stakeholders. The public version of the report will be released as a public document on the USAID Development Experience Clearinghouse http://dec.usaid.gov. Deliverables Due date 1. Inception report August 17, 2015 2. Midpoint briefing to USAID September 7, 2015 3. Data collection completed September 28, 2015 4. Draft evaluation report submitted to USAID October 12, 2015 5. Final presentation to USAID, with PowerPoint presentation October 22, 2015 6. Final report (for USAID dissemination), including clean and identified datasets November 13, 2015 7. Public final presentation November 24, 2015 EVALUATION SCHEDULE OF ACTIVITIES The evaluation is anticipated to begin in January or February. Team Planning Meeting (TPM): The awardee will participate in a one-day TPM that will be held at the beginning of the assignment with USAID. The meeting is essential for the following reasons: 40 Performance Evaluation of the CHASS-SMT Project  Agreeing upon approach for working with USAID staff and partners throughout the assignment  Reviewing and finalizing the evaluation schedule to share with and be approved by the mission (a draft of this schedule should be prepared prior to the TPM)  Enabling USAID staff to discuss with the team the overview and purpose of the evaluation  Finalizing a field visit and meeting schedule for the time in country  Developing data collection methods, instruments, tools and guidelines, software and hardware  Developing a timeline that allows for technology piloting and corrections. See sample rollout below (Attachment 3, Table 3). USAID/Mozambique Meetings: During the course of the field work, meetings with USAID/Mozambique will include:  Initial organizational/introductory meeting(s) at which the awardee will present an outline and explanation of the design of the evaluation  Additional consultation meetings with USAID staff, as needed  Final evaluation presentation: summary of the data, draft recommendations and draft report, prior to departure from country Logistics: USAID/Mozambique will assist the awardee in scheduling initial meetings and interviews. The awardee will be responsible for scheduling follow-up meetings and other activities that are deemed necessary once in country. The awardee’s costs should include transportation and international travel to/from Mozambique, and the awardee will be responsible to arrange for lodging, travel concurrence, local travel expenses, etc. Given the distance between Maputo and relevant provinces, transportation shall be by air; travel within Sofala, Manica, Tete, may be by car. Given recent instability in Sofala, the awardee may adjust travel plans, as needed, due to the security situation in the provinces. If travel is limited based on security concerns, USAID will work with the awardee to understand any potential impact of these limitations on the scope of the evaluation. EVALUATION TEAM COMPOSITION The awardee will propose an evaluation team comprised of at least nine people: the team leader, one senior evaluation expert, two technical experts, a logistics/administrative specialist and at least four enumerators. At least one technical expert, the logistics/administration specialist, and the enumerators must be Mozambican. USAID/Mozambique and MOH staff may also be part of the evaluation team, under the supervision of the team leader, excluding any drafting or any evaluation deliverables. The awardee’s evaluation team will have a combination of technical skills per the aspects and linkages related to HIV/AIDS care and treatment outlined in the objective section of this SOW. During field work, the entire evaluation team will be split into two separate groups to ensure that all four provinces are covered in the planned time period. One team will be led by the team leader, joined by one technical expert and any enumerators; the other team will be led by the senior evaluation expert, joined also by one technical expert and any enumerators. Each team will need to function independently and have a full range of skills needed (included technical background, HIV/PEPFAR experience and past experience in conducting evaluations). An outline of how the evaluation team will be divided into evaluation groups will be included in the awardee’s proposal. In order to preserve integrity and transparency, none of the consultants/individuals proposed as members of the awardee’s evaluation team will have worked for the areas or partners to be evaluated, or may not have been part of the design of these projects or activities. Below are descriptions of the required characteristics for each member of the evaluation team: Performance Evaluation of the CHASS-SMT Project 41 Evaluation Team Leader: The team leader must have at least eight years of experience in implementation of health activities, including experience leading a minimum of two health sector project/activity performance evaluations. S/he should be familiar with the Mozambican health care system and ongoing health care transformation in the country and have three or more years of experience in HIV/AIDS care and treatment programs, preferably PEPFAR-funded. S/He will have demonstrated experience as an evaluation team leader as well as leading qualitative and quantitative interviews/evaluations. S/He must also have developed communication skills (both verbal and written), the ability to conduct interviews and facilitate discussions in both English and Portuguese, and extensive report writing experience. S/He should have solid client interaction skills, leadership, flexibility and management skills, and experience interacting with host government officials, civil society partners and other stakeholders. Additionally, s/he should also have the skills detailed below for the technical experts. The team leader will be responsible for designing, implementing and managing the evaluation and providing team leadership. S/He will be the principal interlocutor between the evaluation team and USAID/Mozambique. S/He will, in collaboration with USAID and other team members, draft assessment tools, finalize the evaluation design, coordinate activities, arrange meetings, consolidate individual input from team members and coordinate the process of assembling the final findings and recommendations. S/He will also lead the preparation and presentation of the key evaluation findings and recommendations for future directions of the project to the USAID/Mozambique IHO team and key partners. Senior Evaluation Expert: The senior evaluation expert will be responsible for working closely with the team leader in preparing evaluation deliverables and performing data collection and analysis, and other tasks assigned by the team leader. S/he will also lead one of the two evaluation groups (see above), providing general evaluation management, quality control and oversight during the field work. S/he must have led at least one health￾sector evaluation that included significant qualitative work, must have at least five years of evaluation experience, and must be able to demonstrate experience using qualitative evaluation methodologies and triangulating with quantitative data. S/He must be able to conduct all interviews and background research in English and Portuguese, and must have excellent data interpretation and presentation skills. Additionally, s/he should also have the skills detailed below for the technical experts. Technical Experts: The technical experts, in collaboration with the team leader and the senior evaluation specialist, will provide significant input to the assessment tools and evaluation design. They will participate in meetings, provide input to the final presentations and evaluation report, and may write some aspects of the report, as determined by the team leader. The two technical experts must both have over five years of experience working on clinical and/or HSS activities. They should have substantial knowledge about health programming, including past experience in provision of HIV clinical services or HSS. They should have solid client interaction skills, flexibility and communication skills, as well as experience interacting with host government officials, civil society partners and other stakeholders. Both technical experts must have technical evaluation skills, including experience with: SPSS or similar data analysis software; experience analyzing HIV/AIDS data and evaluation/study results; experience drafting and presenting evaluation results; and experience conducting qualitative and quantitative evaluations. They also should have experience coding qualitative survey responses as well as experience working in sub-Saharan Africa. Both technical experts must be able to communicate in Portuguese, meaning they must either be fluent in Portuguese or Spanish. In addition, one of the two technical experts must be Mozambican. 42 Performance Evaluation of the CHASS-SMT Project Local Logistics/Administrative Specialist: This person must have at least three years of experience coordinating events and travel, both international and within Mozambique. Based in Mozambique, s/he will manage all in-country travel, logistics, and other duties as assigned by the team leader. S/He may also be responsible for administrative and communications tasks such as procurement of good and services, including consumables, arranging travel, making photocopies, and arranging venues for workshops/large group meetings. S/He will assist with communication with relevant evaluation participants, from the implementing partners to government officials, where appropriate. S/He will perform other duties as assigned by the evaluation team leader, and preferably also will be able to communicate in English. Local Enumerators: The evaluation team will include at least four Mozambican enumerators, two tasked to each of the two evaluation groups. These team members may interview key informants and assist with facilitating focus group discussions. They should have basic familiarity with health topics, as well as experience with interviews and note-taking. Ideally, enumerators will be based in Sofala, Manica, or Tete. It is preferable that they able to speak at least one major dialect from Sofala, Manica, or Tete to facilitate interview communication, specifically with local CBOs. They will join the evaluation team on site visits as determined by evaluation team leader. Attachment 1 Table 1: Funding Details of CHASS SMT Annex 2 Table 2: Evaluation Data Collection Methodology Matrix Question Proposed Data Collection Method (illustrative) 1) What are the project’s strengths and weaknesses, as seen by project staff, DPS, DDS and chief medical officer, HFs, USAID, and PCC, and how can weaknesses be improved, according to these stakeholders? Key informant interviews: It is important to capture perspectives from USAID, the project (including prime partner and SA holders), DPS, DDS, District Medical Chefe 2) Where has the most progress been seen in strengthening systems (e.g., planning, financial management, supply and logistics, information systems)? Key informant interviews, review of project data (qualitative and quantitative) in systems areas 3) Which activities or project components will be most feasible to transition from the project to the government? Key informant interviews, extrapolation from project data and other reports (provincial public financial management risk assessment framework reports) 4) To what extent has the project been able to create and strengthen linkages between facilities and communities to allow for increased service uptake, specifically in the areas of: a. Community-based counseling and testing to treatment (for both men and women) Comparison of retention data/men’s access to services/referrals utilized in sites where there is a strong CBO presence and those where there is no CBO (or less activity) Focus groups with communities in sites where there is a strong CBO presence and those where there is no CBO (or less activity), surveys Activity Award Dates Activity Funding Implementing Partner CHASS SMT Nov 1, 2010-October 31, 2015 $111,435,465 Abt Associates Performance Evaluation of the CHASS-SMT Project 43 Question Proposed Data Collection Method (illustrative) b. Retention of pre-ART and ART patients (through the use of adherence groups, active case-finding, and other community groups) c. Knowledge, demand and access of services by men 5) What constraints have the activities faced in improving retention of patients in pre-ART and ART? 6) What has been the most effective method found by the project to improve retention? 7) To what extent has HF management (improved patient flow, etc.) improved over the life of the project? Interviews with HF and project staff, observations within facilities, review of project reports Observation of patient flow, data registers, etc. 8) To what extent is data management capacity built at the HFs with regard to HIV/AIDS registry data? Analysis of DQAs previously conducted, results of routine data review meetings, key informant interviews, review of project reports 9) To what extent has knowledge (community and health worker) and utilization of GBV services increased over the life of the project? Focus groups with communities, key informant interviews, surveys, interviews with health workers, review of community and clinical data 10) What are the benefits and challenges of the project’s model of working with government (at provincial and district levels) through SAs? Key informant interviews: It is important to capture perspectives from USAID, the project (including prime partner and SA holders), DPS and health workers) Attachment 3 Table 3: Sample rollout: Pilot instruments on paper Test on mobile as appropriate Set up devices and test connectivity Check dummy data from pilot (and revise surveys accordingly) Roll out 44 Performance Evaluation of the CHASS-SMT Project ANNEX II: DATA COLLECTION INSTRUMENTS: KII, FGD Key Informant Interview Guide for DPS and DDS Staff Date: ___________________________ Province/District _________________________ Name of respondent: ___________________________________________ __________ Gender: M/F Designation: _____________________________________________________________ Number Question General 101. What do you consider as the main achievements of the CHASS project in this province/district? Was there a particular period when achievements were most accelerated (please explain)? 102. Are there differences in achievements across the districts? If yes, what could explain these differences? HIV/AIDS Care and Treatment 201. What do you consider as the main achievements of the CHASS project in this province/district in the delivery of HIV/AIDS care and treatment services? Ask the respondent to discuss pediatric care (if not mentioned). 202. What project activities or strategies worked well? Why? 203. What project activities or strategies did not work well? Why? 204. In your opinion, did the CHASS project adequately target the men and women? State some examples. 205. In your opinion, what do you think were the benefits of working with the following groups in improving coverage and quality of HIV care and treatment services? State the extent of their relevancy. a) Equipa Polivalentes b) Equipa de Apoio c) CBOs d) GAACs e) Adherence Committees f) Case managers 206. Please comment on the sustainability of the activities performed by the CHASS project. 207. In your opinion, what were the key challenges that the project faced in implementing some of its activities or strategies? 208. In your opinion, what project activities or approaches could have been done better by the CHASS project? 209. How did the project promote the integration of primary health care or maternal and child health with HIV/AIDS services? What approach worked, and what did not? 210. How did the project create and strengthen linkages between facilities and communities to promote increased service uptake of HIV/AIDS care and treatment services? 211. To what extent was the project successful in creating knowledge, demand and access to HIV care and treatment services among men? 212. How did the project create and strengthen intra-health facility linkages to ensure high quality of services? (How did the project engage providers across sectors at the health facility to integrate information on health records?) 213. What are your constraints in improving retention of patients in pre-ART and ART? How did the project help improve these constraints? 214. What are your constraints in improving pediatric recruitment and retention in care? How did the project help improve these constraints? 215. What was the most effective method found by the project to improve retention? 216. What are your recommendations for improving linkages and referral systems in the future? 217. Which of the approaches were most effective in improving use of health care services by men and women? Performance Evaluation of the CHASS-SMT Project 45 Number Question 218. Which activities aimed at improving HIV/AIDS care and treatment services are most feasible to transition to DPS/SDMAS with little or no technical support from CHASS? 219. Are there activities that have been transitioned to the DPS/SDMAS staff? If yes, what are the successes achieved or failures faced? 220. What are some of the best practices and lessons learned from this project in improving recruitment and retention in health care services? 221. In your opinion, what challenges will health facilities face in adopting some of the best practices or approaches in improving recruitment and retention of patients (men, women and children) in care and treatment, without technical support from CHASS? 222. What are your recommendations for sustainability of the project activities and achievements? PMTCT 301. In your opinion, what do you consider as the main achievements of the CHASS project in this province/district in delivery of the prevention of mother-to-child transmission services? 302. What project activities or strategies worked well? Why? 303. What project activities or strategies did not work well? Why? 304. In your opinion, what do you think were the benefits of working with the following groups in improving coverage and quality of PMTCT services? State the extent of their relevancy. a) Equipa Polivalentes b) Equipa de Apoio c) CBOs d) GAACs e) Adherence Committees f) Case managers g) MpM or M2M groups 305. Do you think the activities performed by these groups can continue without CHASS SMT financial support? Why? Why not? 306. In your opinion, what are the key challenges that the project faced in implementing its activities or strategies? 307. In your opinion, what project activities or approaches could have been done better? 308. How did the project promote the integration of primary health care or maternal and child health with HIV/AIDS services? What approach worked, and what did not? 309. How did the project create and strengthen linkages between facilities and communities to promote increased service uptake of PMTCT services? 310. How did the project create and strengthen intra-health facility linkages to ensure high quality of services? (How did the project engage providers across sectors at the health facility to integrate information on health records? Mention patient tracking system if not mentioned.) 311. What are your constraints in improving pediatric recruitment and retention in care? How did the project help to improve these constraints? 312. What was the most effective method found by the project to improve retention in the PMTCT cascade (ANC, testing, HIV testing for mother and baby, and mother and baby receiving ART)? 313. Which of the approaches were most effective in improving health care service utilization by both men and women? 314. What are some of the best practices and lessons from this project in improving recruitment and retention in the PMTCT cascade (ANC, testing, HIV testing for mother and baby, and mother and baby receiving ART)? 315. What challenges will health facilities face in adopting some of the best practices or approaches in improving recruitment and retention of mothers in the PMTCT cascade, without technical support from CHASS? 316. Which activities aimed at improving PMTCT services are most feasible to transition to DPS/SDMAS with little or no technical support from CHASS? 317. Are there activities that have been transitioned to the DPS/SDMAS staff? If yes, what are the successes achieved or failures faced? 318. What are your recommendations for sustainability of the project activities and achievements? 46 Performance Evaluation of the CHASS-SMT Project Number Question HCT 401. What do you consider as the main achievements of the CHASS project in this province/district in delivery of HCT services (at the facility and in the community)? 402. What project activities or strategies worked well? Why? 403. What project activities or strategies did not work well? Why? 404. In your opinion, did the CHASS project adequately target men and women? State some examples. 405. In your opinion, what do you think were the benefits of working with the following groups or tools in achieving project results? a) Equipa Polivalentes b) CBOs 406. Do you think the activities performed by these groups can continue without CHASS SMT financial support? Why? Why not? 407. In your opinion, what are the key challenges that the project faced in implementing its activities or strategies? 408. In your opinion, what project activities or approaches could have been done better? 409. How did the project promote the integration of primary health care or maternal and child health with HIV/AIDS services? What approach worked, and what did not? 410. How did the project create and strengthen linkages between facilities and communities to promote increased service uptake of HCT services? 411. To what extent was the project successful in creating knowledge, demand and access to HCT services among men? 412. How did the project create and strengthen intra-health facility linkages to ensure high quality of services? (How did the project engage providers across sectors at the health facility to integrate information on health records?) 413. In your opinion, were there constraints in linking HIV-positive men and women tested in C-HCT to the health facility? How were they overcome? 414. Which of the community-level approaches were effective in improving health care service utilization by both men and women? 415. What challenges will health facilities face in adopting some of the best practices or approaches in improving recruitment of HIV-positive men, women and children, without technical support from CHASS? 416. Which activities aimed at improving HCT coverage are most feasible to transition to DPS/SDMAS with little or no technical support from CHASS? 417. Are there activities that have been transitioned to the DPS/SDMAS? If yes, what are the successes achieved or failures faced? 418. What are your recommendations for sustainability of the project activities and achievements? HSS 501. (FOR SDMAS ONLY) What activities were targeted at improving the following: a) Planning system and capacity b) Financial management c) Supply and logistics at health facility (discuss lab results, stock-outs, storage infrastructure, etc.) d) Information systems (discuss Sistema de Informação de Medicamentos e Artigos Médicos–Pharmaceuticals and Medical Commodities Information System, health facility records) e) Monitoring and evaluation system f) Human resource development and management 502. Which of the activities worked well? 503. What was the value added by the various health system strengthening approaches? (Please talk about clinical tools if not mentioned) 504. Which of the activities did not work well? Why? 505. Where has the most progress been seen in strengthening systems (e.g., planning, financial management, supply and logistics, information systems, monitoring and evaluation)? Performance Evaluation of the CHASS-SMT Project 47 Number Question 506. Did the graduation path system produce improvements in service management at district and/or health facilities? If so, what improvements were noted in clinical services, management of information systems, monitoring and evaluation, etc.? 507. What are the associated challenges, and how can this graduation path system be improved? 508. To what extent did patient flow improve over the life of the project? 509. To what extent did patient waiting times improve over the life of the project? 510. How have leadership roles and decision-making structure improved at health facilities over the project’s life? Please explain. 511. Describe the current strategies and processes defined to ensure continuity of functions in the event of staff turnover or other unanticipated disruptions. 512. To what extent did CHASS build the health facility data management capacity with regard to HIV/AIDS registry data? 513. How has the correct completion of different registries/forms improved at health facilities? 514. How has filing of records or registries at facilities improved over time? 515. Are health facilities analyzing data and using it in their planning? How has the CHASS project contributed to their capacity to do this? 516. To what extent has the review and use of records across different sections in health facilities been improved? 517. How has the sharing of patients’ records within the facility improved? 518. Have transportation and management of CD4 and PCR results improved over the project life? If so, how? 519. How have the record keeping or lab information systems improved over the project’s life? 520. How has the stock-out of clinic consumables improved over the project’s life? 521. How have the record keeping or pharmacy information systems improved over the project’s life? 522. [IF NOT MENTIONED, ASK] How have the CHASS sub-agreements with training centers helped to improve human resource capacities and the number of health workers at district health facilities? (Discuss gap funding if not mentioned) 523. What are the benefits of working with sub-agreements? 524. What are the challenges of working with sub-agreements? 525. How well did the CHASS program management structure support or facilitate HIV/AIDS response programming of this province/district? 526. Please share with us some lessons learned/best practices that can be scaled up in the near future? 527. What recommendations do you have for strengthening the organizational/institutional structures of DPS and DDS to perform their functions more effectively? 528. What measures should the district put in place in order to sustain what has been put in place by the CHASS project? GBV (Gender-based Violence) 601. What are your perceptions about gender issues in your province, district? 602. What do you consider as the main achievements of the CHASS project in this province/district relating to GBV issues? 603. What are the key challenges that the project faced in implementing some of its GBV activities or strategies? 604. What are your recommendations for sustainability of what the project has achieved in GBV? 605. To what extent has health workers’ knowledge and utilization of gender-based violence services increased over the life of the project? 606. To what extent has communities’ knowledge and utilization of gender-based violence services increased over the life of the project? 607. To what extent has the project been able to create and strengthen linkages between facilities and communities to allow for increased service uptake of GBV services? (If not mentioned, ask about cabinet de atendimento de vitimas de violencia) 608. What constraints did the project activities face in linking GBV victims at the health facility to other services (If not mentioned, ask about cabinet de atendimento de vitimas de violencia)? How were they overcome? 48 Performance Evaluation of the CHASS-SMT Project Number Question 609. What challenges will health facilities face in adopting some of the best practices or approaches in improving GBV services, without technical support from CHASS? 610. What are your recommendations for sustainability of what the project activities and achievements? Key Informant Interview Guide for Health Facility Clinical Staff Date: ___________________ Province/District _________________________ HF: ________________ Name of respondent: ___________________________________________ _______________________ Designation: __________________________________________________________________________ Number Question 101 What do you consider as the main achievements of the CHASS project in this health facility? Treatment and care 201 What do you consider as the main achievements of the CHASS project in this health facility in the delivery of HIV/AIDS care and treatment services? Ask the respondent to discuss pediatric care if not mentioned. 201b In your opinion, what do you consider as the main achievements of the CHASS project in this health facility in delivery of the prevention of mother-to-child transmission services? 202 What project activities or strategies worked well? Why? 203 What project activities or strategies did not work well? Why? 204 In your opinion, did the CHASS project adequately target men and women? State some examples. 205 In your opinion, what do you think were the benefits of working with the following groups in improving coverage and quality of HIV care and treatment services? State the extent of their relevancy. 205a Equipa Polivalentes 205b Equipa de Apoio 205c OBCs 205d GAACs 205e Comissões de adesão 205f Gestores de processo 205g Grupos de MpM 206 Do you think the activities performed by these groups can continue without financial support from CHASS? 207 What were the key challenges that the health facility faced in implementing some of its activities or strategies? 208 In your opinion, what project activities or approaches could have been done better? 209 How did the project promote the integration of primary health care or maternal and child health with HIV/AIDS services in this facility? What approach worked, and what did not? 210 How did the project create and strengthen linkages between your health facilities and communities to promote increased service uptake of HIV/AIDS care and treatment services 211 To what extent was the project successful in creating knowledge, demand and access to HIV care and treatment among men? 212 How did the project help you to create and strengthen intra-health facility linkages to ensure high quality of services? How did the project engage providers across sectors at the health facility to integrate information on health records? 213 What constraints does the health facility face in improving retention of patients in pre-ART and ART? 214 What constraints did the health facility face in improving pediatric recruitment and retention in care? 215 What was the most effective method found by the health facility to improve retention in pre-ART care, ART or PMTCT? 216 What are your recommendations for improving linkages and referral systems in the future? Performance Evaluation of the CHASS-SMT Project 49 Number Question 217 Which activities aimed at improving HIV/AIDS care and treatment services are most feasible to transition to this health facility with little or no technical support from CHASS? 217b Which activities aimed at improving PMTCT services are most feasible to transition to this health facility with little or no technical support from CHASS? 218 What are some of the best practices and lessons learned from this project in improving recruitment and retention in health care services? 219 What challenges will you, at this health facility, face in adopting some of the best practices or approaches in improving recruitment and retention of patients (men, women and children) in care and treatment, without technical support from CHASS? PMTCT/MCH Nurses 301 In your opinion, what do you consider as the main achievements of the CHASS project in this health facility in delivery of prevention of mother-to-child transmission services? 302 What activities or strategies worked well at this health facility in improving recruitment and retention in PMTCT services or pediatric care services? Why? 303 What activities or strategies did not work well at this health facility? Why? 304 In your opinion, what do you think were the benefits of working with the following groups in improving coverage and quality of PMTCT services or pediatric care services? State the extent of their relevancy. 304a Equipa Polivalentes 304b Equipa de Apoio 304c OBCs 304d GAACs 304e Comissões de adesão 304f Gestores de processo 304g Grupos de MpM 305 Do you think the activities performed by these groups can continue without CHASS SMT financial support? Why? Why not? 306 What are the key challenges that the health facility faced in implementing some of its activities or strategies? Discuss recruitment and retention in PMTCT services or pediatric care services. 307 In your opinion, what activities or approaches could have been done better at this health facility? 308 How did the project promote the integration of primary health care or maternal and child health with HIV/AIDS services at this health facility? What approach worked, and what did not? 309 How did the project create and strengthen linkages between facilities and communities to promote increased service uptake of PMTCT services? 309b How did the project create and strengthen linkages between facilities and communities to promote increased service uptake of pediatric care services, including HIV testing? 310 How did the project help you to create and strengthen intra-health facility linkages to ensure high quality of services? How did the project engage providers across sectors at the health facility to integrate information on health records? 311 What were the constraints faced by the health facility in improving pediatric recruitment and retention in care? 312 What was the most effective method found by the health facility to improve retention in the PMTCT cascade (ANC, testing, HIV testing for mother and baby and mother, and baby receiving ART)? 313 What are some of the best practices and lessons from this health facility in improving recruitment and retention in the PMTCT cascade? 314 What challenges will you face at this health facility in adopting some of the best practices or approaches in improving recruitment and retention of mothers in the PMTCT cascade, without technical support from CHASS? 315 Which activities aimed at improving PMTCT services are most feasible to transition to this health facility with little or no technical support from CHASS? 315b Which activities aimed at improving pediatric care services are most feasible to transition to this health facility with little or no technical support from CHASS? 316 What are your recommendations for sustainability of what the project has achieved? 50 Performance Evaluation of the CHASS-SMT Project Number Question HCT HCT 401 In your opinion, what are the main achievements of the CHASS project in this health facility in delivery of the HCT services at the facility and in the community? 402 What activities or strategies worked well at this health facility? Why? 403 What activities or strategies did not work well at this health facility? Why? 412 To what extent was the project successful in creating knowledge, demand and access to HCT services among men? 414 What constraints did the project activities face in linking HIV-positive men and women tested in C-HCT to the health facility? How were they overcome? 417 Which activities aimed at improving HCT coverage are most feasible to transition to the health facility with little or no technical support from CHASS? 418 What are your recommendations for sustainability of what the project has achieved? HSS Health Systems Strengthening 501 Where has the most progress been seen in strengthening systems, e.g., planning, financial management, supply and logistics, information systems, monitoring and evaluation? 502a To what extent has the patient flow at your health facility improved over the life of the project? 502b To what extent have patient waiting times at your health facility improved over the life of the project? 503 Have the leadership roles and the decision-making structure improved at this facility over the project’s life? Please explain. 504 Describe the current strategies and processes defined to ensure continuity of functions in the event of staff turnover or other unanticipated disruptions. 505 To what extent did CHASS build the health facility data management capacity with regard to HIV/AIDS registry data? 504 How has the correct completion of different registries/forms improved at the health facility? 505 How has the filing of the records or registries at this facility improved over time? 504 Do you analyze data and use it in your planning? How has CHASS project contributed to your capacity to do this? 505 How has CHASS project contributed to your capacity to do this? 504 Are health facilities analyzing the data and using it in planning? How has CHASS project contributed to the capacity to do this? 505 To what extent have the review and use of records across different sections in your health facility been improved? 504 Did the graduation path system produce improvements in service management in this facility? If so, what improvements were noted in clinical services, management of information systems, monitoring and evaluation, etc.? 505 How has the supply chain improved over the life of the project? 504 How is the sharing of the patients’ records within the facility? 505 How have the transportation and management of CD4 and PCR results improved over the project’s life? 504 How have the record keeping or lab information systems improved over the project’s life? 505 How has the stock-out of clinic consumables improved over the project’s life? 504 How have the record keeping or pharmacy information systems improved over the project’s life? Gender-based Violence 601 What training have you received in gender issues? 602 What do you consider as the main achievements of CHASS project in this health facility relating to GBV issues? 602a What are forms of the GBV do you handle at this facility? 603 What are the key challenges that the health facility faced in implementing GBV services? 604 To what extent has health workers’ knowledge and utilization of gender-based violence services increased over the life of the project? Performance Evaluation of the CHASS-SMT Project 51 Number Question 605 To what extent has the project been able to create and strengthen linkages between facilities and communities to allow for increased service uptake of GBV services? 606 What constraints did the health facility face in linking GBV victims in the community to the other services? How were they overcome? 607 What challenges will health facilities face in adopting some of the best practices or approaches in improving GBV services, without technical support from CHASS? 608 What are your recommendations for sustainability of what the project has achieved? Key Informant Interview Guide for IP staff Date: ___________________________ Province/District _________________________ Name of respondent: ___________________________________________ Gender: M/F Designation: _______________________________________ Number Question General 101. What do you consider as the main achievements of the CHASS project in this province/district? Was there a particular period when achievements were most accelerated (please explain)? 102. Are there differences in achievements across the districts? If yes, what could explain these differences? HIV/AIDS Care and Treatment 223. What do you consider as the main achievements of the CHASS project in this province/district in the delivery of HIV/AIDS care and treatment services? Ask the respondent to discuss pediatric care (if not mentioned). 224. What project activities or strategies worked well? Why? 225. What project activities or strategies did not work well? Why? 226. In your opinion, did the CHASS project adequately target men and women? State some examples. 227. In your opinion, what do you think were the benefits of working with the following groups in improving coverage and quality of HIV care and treatment services? State the extent of their relevancy. a) Equipa Polivalentes b) Equipa de Apoio c) CBOs d) GAACs e) Adherence Committees f) Case managers 228. Please comment on the sustainability of the activities performed by these groups by the district and health facilities. 229. What were the key challenges that the project faced in implementing some of its activities or strategies? 230. In your opinion, what project activities or approaches could have been done better? State how. 231. How did the project promote the integration of primary health care or maternal and child health with HIV/AIDS services? What approach worked, and what did not? 232. How did the project create and strengthen linkages between facilities and communities to promote increased service uptake of HIV/AIDS care and treatment services? 233. To what extent was the project successful in creating knowledge, demand and access to HIV care and treatment services by men? 234. How did the project create and strengthen intra-health facility linkages to ensure high quality of services? (How did the project engage providers across sectors at the health facility to integrate information on health records?) 52 Performance Evaluation of the CHASS-SMT Project Number Question 235. What constraints did the project activities face in improving retention of patients in pre-ART and ART? 236. What constraints did project activities face in improving pediatric recruitment and retention in care? 237. What was the most effective method found by the project to improve retention? 238. What are your recommendations for improving linkages and referral systems in the future? 239. Which of the community-level approaches were most effective in improving use of health care services by men and women? 240. Which activities aimed at improving HIV/AIDS care and treatment services are most feasible to transition to the government system with little or no technical support from CHASS? 241. Are there activities that have been transitioned to the government staff? If yes, what are the successes achieved or failures faced? 242. What are some of the best practices and lessons learned from this project in improving recruitment and retention in health care services? 243. What challenges will health facilities face in adopting some of the best practices or approaches in improving recruitment and retention of patients (men, women and children) in care and treatment, without technical support from CHASS? 244. What are your recommendations for sustainability of what the project has achieved? PMTCT 319. What do you consider as the main achievements of the CHASS project in this province/district in delivery of the prevention of mother-to-child transmission services? 320. What project activities or strategies worked well? Why? 321. What project activities or strategies did not work well? Why? 322. In your opinion, what do you think were the benefits of working with the following groups in improving coverage and quality of PMTCT services? State the extent of their relevancy. a) Equipa Polivalentes b) Equipa de Apoio c) CBOs d) GAACs e) Adherence Committees f) Case managers g) MpM or M2M groups 323. Please comment on the sustainability of the activities performed by these groups by the district and health facilities? 324. What are the key challenges that the project faced in implementing some of its activities or strategies? 325. In your opinion, what project activities or approaches could have been done better? 326. State how? 327. How did the project promote the integration of primary health care or maternal and child health with HIV/AIDS services? What approach worked, and what did not? 328. How did the project create and strengthen linkages between facilities and communities to promote increased service uptake of PMTCT services? 329. How did the project been create and strengthen intra-health facility linkages to ensure high quality of services? (How did the project engage providers across sectors at the health facility to integrate information on health records?) 330. What constraints did project activities face in improving pediatric recruitment and retention in care? 331. What was the most effective method found by the project to improve retention in the PMTCT cascade? 332. Which of the community-level approaches were most effective in improving healthcare service utilization by both men and women? 333. What are some of the best practices and lessons from this project in improving recruitment and retention in the PMTCT cascade? 334. What challenges will health facilities face in adopting some of the best practices or approaches in improving recruitment and retention of mothers in the PMTCT cascade, without technical support from CHASS? Performance Evaluation of the CHASS-SMT Project 53 Number Question 335. Which activities aimed at improving PMTCT services are most feasible to transition to the government system with little or no technical support from CHASS? 336. Are there activities that have been transitioned to the government staff? If yes, what are the successes achieved or failures faced? 337. What are your recommendations for sustainability of what the project has achieved? HCT 419. What do you consider as the main achievements of the CHASS project in this province/district in delivery of the HCT services (at the facility and in the community)? 420. What project activities or strategies worked well? Why? 421. What project activities or strategies did not work well? Why? 422. In your opinion, did the CHASS project adequately target men and women? State some examples. 423. In your opinion, what do you think were the benefits of working with the following groups or tools in achieving project results? a) Equipa Polivalentes b) CBOs 424. Please comment on the sustainability of the activities performed by these groups by the district and health facilities. 425. What are the key challenges that the project faced in implementing some of its activities or strategies? 426. In your opinion, what project activities or approaches could have been done better? 427. State how. 428. How did the project promote the integration of primary health care or maternal and child health with HIV/AIDS services? What approach worked, and what did not? 429. How did the project create and strengthen linkages between facilities and communities to promote increased service uptake of HCT services? 430. To what extent was the project successful in creating knowledge, demand and access to HCT services by men? 431. How did the project create and strengthen intra-health facility linkages to ensure high quality of services? (How did the project engage providers across sectors at the health facility to integrate information on health records?) 432. What constraints did the project activities face in linking HIV-positive men and women tested in C-HCT to the health facility? How were they overcome? 433. Which of the community-level approaches were effective in improving health care service utilization by both men and women? 434. What challenges will health facilities face in adopting some of the best practices or approaches in improving recruitment of HIV-positive men, women and children, without technical support from CHASS? 435. Which activities aimed at improving HCT coverage are most feasible to transition to the government system with little or no technical support from CHASS? 436. Are there activities that have been transitioned to the government staff? If yes, what are the successes achieved or failures faced? 437. What are your recommendations for sustainability of what the project has achieved? HSS 529. What activities were targeted at improving the following: a) Planning system and capacity b) Financial management c) Supply and logistics at health facility (discuss lab results, stock-outs, storage infrastructure, etc.) d) Information systems (discuss Sistema de Informação de Medicamentos e Artigos Médicos–Pharmaceuticals and Medical Commodities Information System, health facility records) e) Monitoring and evaluation system f) Human resource development and management 530. Which of the activities worked well? 531. What was the value added by the various health system strengthening approaches? 54 Performance Evaluation of the CHASS-SMT Project Number Question 532. Which of the activities did not work well? Why? 533. Where has the most progress been seen in strengthening systems (e.g., planning, financial management, supply and logistics, information systems, monitoring and evaluation)? 534. To what extent did patient flow improve over the life of the project? 535. To what extent did patient waiting times improve over the life of the project? 536. How have leadership roles and decision-making structure improved at health facilities over the project’s life? Please explain. 537. Describe the current strategies and processes defined to ensure continuity of functions in the event of staff turnover or other unanticipated disruptions. 538. To what extent did CHASS build the health facility data management capacity with regard to HIV/AIDS registry data? 539. How has the correct completion of different registries/forms improved at health facilities? 540. How has filing of records or registries at facilities improved over time? 541. Are health facilities analyzing data and using it in their planning? How has CHASS project contributed to their capacity to do this? 542. To what extent have the review and use of records across different sections in health facilities been improved? 543. Did the graduation path system produce improvements in service management at district and/or health facilities? If so, what improvements were noted in clinical services, management of information systems, monitoring and evaluation, etc.? 544. How has the supply chain improved over the life of the project? 545. How has the sharing of patients’ records within the facility improved? 546. Have transportation and management of CD4 and PCR results improved over the project’s life? If so, how? 547. How have the record keeping or lab information systems improved over the project’s life? 548. How has the stock-out of clinic consumables improved over the project’s life? 549. How have the record keeping or pharmacy information systems improved over the project’s life? 550. [IF NOT MENTIONED, ASK] How have the CHASS sub-agreements with training centers helped to improve human resource capacities and the number of health workers at district health facilities? (Discuss gap funding if not mentioned) 551. What are the benefits of working with sub-agreements? 552. What are the challenges of working with sub-agreements? 553. What measures should the district put in place in order to sustain what has been put in place by the CHASS project? 554. Gender-based Violence 555. What training have you received in gender issues? 556. What do you consider as the main achievements of the CHASS project in health facilities relating to GBV issues? 557. What are the key challenges that the health facilities faced in implementing GBV services? 558. To what extent have health workers’ knowledge and utilization of gender-based violence services increased over the life of the project? 559. To what extent has the project been able to create and strengthen linkages between facilities and communities to allow for increased service uptake of GBV services? 560. What constraints did the project activities face in linking GBV victims in the community to the health facility? How was it overcome? 561. What challenges will health facilities face in adopting some of the best practices or approaches in improving GBV services, without technical support from CHASS? 562. What are your recommendations for sustainability of what the project has achieved? Performance Evaluation of the CHASS-SMT Project 55 Key Informant Interview Guide and Data Capture Form A. Participating CBO Staff Date: ___________________________ Province/District _________________________ Name of respondent: ___________________________________________ Gender: M F Designation: _______________________________________ 1. What do you consider as the main achievements of the CHASS project in this district? What has not been achieved? _________________________________________________________________________________________ 2. Specifically, comment on the role of your organization in: (a) Improving the demand and knowledge about health services in the health facilities (b) Improving retention in HIV/AIDS care services (c) GBV-related services 3. How sustainable are your groups’ activities without the technical support from CHASS? 4. What can you tell us about the case managers, GAACs and the Adherence Committees? How can groups work better? 5. Are there some key challenges that members of GAACs or adherence groups face in accessing or staying on treatment? 6. To what extent has the project been able to create and strengthen linkages between facilities and communities to allow for increased service uptake, specifically in the areas of: a. Community-based counseling and testing to treatment (for both men and women)________________ b. Retention rates in pre-ART and ART patients (through the use of adherence groups, active case-finding, and other community groups), and in PMTCT cascade_________________________________________ c. Knowledge, demand and access of services by men and women (PEP) __________________________ d. Inter-facility referral and linkages _____________________________________________________ e. Linkage and referral of HIV-positive men and women to HIV/AIDS care ________________________ 7. What are the challenges associated with C-HCT? How can C-HCT be improved or sustained? _______________________________________________________________________________________ 8. What constraints have your activities faced in improving retention of patients in pre-ART and ART? _______________________________________________________________________________________ 9. What constraints have your activities faced in improving pediatric recruitment and retention in care? _______________________________________________________________________________________ 10. What has been the most effective method found to improve retention? _______________________________________________________________________________________ 11. Provide your overall rating on the improvement of linkages and referrals since CHASS began. What are your recommendations for the future? ______________________________________________________________ 12. What are some of the gender issues in your community? Are you aware of GBV in this community? 13. To what extent have knowledge (community and health worker) and utilization of gender-based violence services increased over the life of the project in your community members and among members of your team? ________________________________________________________________________________________ 14. What kind of information, education and communication materials do you have on GBV? Who is in the target group? How do you identify them? _______________________________________________________________________________________ 15. What are the challenges of implementing GBV services? _____________________________________________ How can they be overcome? __________________________________________________________________ 16. How does the community assist those who have suffered GBV?____________________________________ In your opinion, what can be done to reduce GBV in your community?_________________________________ 17. What are your thoughts about the benefits and challenges of the project’s model of working with you through sub￾agreements with CBOs? ______________________________________________________________________ 18. Please share with us some lessons learned/best practices of the project in working through sub-agreements with CBOs that should be maintained or scaled-up in the near future? _____________________________________________ 19. How is your CBO prepared to sustain what has been put in place by CHASS? ______________________________ 56 Performance Evaluation of the CHASS-SMT Project Key Informant Interview Guide and Data Capture Form B. MOH Staff/USAID Staff Date: ___________________________ Province/District _________________________ Name of respondent: ___________________________________________ Gender: M F Designation: _______________________________________ Number Question 201. What do you consider as the main achievements of the CHASS project in the provinces of Sofala, Manica and Tete? 202. Are there differences in achievements across the provinces? If yes, what could explain these differences? 203. In your opinion, what were the key challenges that the project faced in implementing some of its activities or strategies? 204. In your opinion, what project activities or approaches could have been done better by the CHASS project teams? State how. 205. Where has the most progress been seen in strengthening systems (e.g., planning, financial management, supply and logistics, information systems, monitoring and evaluation)? 206. What are the benefits of working with sub-agreements? 207. What are the challenges of working with sub-agreements? 208. How well did the CHASS program management structures support or facilitate HIV/AIDS response programming in the different provinces? 209. What are some of the best practices and lessons learned in this project that can be scaled-up in the near future? 210. What recommendations do you have for strengthening the organizational/institutional structures of DPS and DDS to perform their functions more effectively? GBV (Gender-based Violence) [Gender Focal Points] 601. What do you consider as the main achievements of the CHASS project relating to GBV issues in the different provinces? 602. What are the key challenges that the project faced in implementing some of its GBV activities or strategies? 603. What are your recommendations for sustainability of what the project has achieved in GBV? Performance Evaluation of the CHASS-SMT Project 57 ANNEX III: LIST OF INTERVIEWEES Maputo Province Diego Frederico Paulo, HR, CHASS Angelica Solomao, Clinical Director, CHASS Alzira Pereira dos Santos de Louvado, Clinical, CHASS Tete Laide Boaze, Tech Provincial Coordinator, CHASS/Sofala Nuno Miguel Militar Antonio, Oficial Prog, FHI/Sofala Timotio Mario Filite Oficial, FHI/Sofala Manuel Joao Cala, Oficial Cuidados e Tratamento, CHASS Joao Luanga, Conselheiro, Formacao CHASS Sofala Province Dra. Ivone Chenena, Beira City, Dir Clinica Juliao Munkepe Armando, Tec. Admin, Gestao Processo (M&E) Dra. Tania Mulhanga, Medico Chef Sr. Rafael, Tec M&E Dr. Corado Lionel, Med Chef, Nhamatanda Luis Mabasso, M&E Nordeno Cp,e. Tec Medicina, Gondola Sede Belmira Gilda Antonio Moiama, Emf SMI Gondola Sede IP Coordinator IP Technical Coordinator DPS, HIV Focal Person MCH Nurses, one for prenatal care and another for PCR DPS Director DPS Gender CS Munhava Figueira Ponto Gea HF director Ponto Gea HF MCH nurse MCH nurses at Nhamatanda health facility Dr. Sonia Director Centro de Saude Ponta Beira CHASS Provincial Coordinator, CHASS SMT Coordinator Leando Adriano, DPPC (M&E) Provincial M&E Abdieulremse Abduls, abduljamila@yahoo.com, DPAF (DPS Staff), Provincial Planning/DAF Paulino Tatoluis, ptatoluis@gmail.com, DPPC (Logistics), Provincial Planning/Logistics Clinical Director, Muhanva, CHASS, worked for 2 months, Director HF Fernando Chenene, fcheueue@fhi360.org, Tel: 837948820, Provincial coordinator PCC 58 Performance Evaluation of the CHASS-SMT Project Clinical Director Mafambise Hospital, mioneliachiau@hotmail.com (CHASS 8 months), Director HF Technical General, 10-09-2015 Manuel Guerra, Coordinator Kugarissica CBO, Email: guerrasitole@gmail.com Manica Province Albano de Camangueira, Diretor Distrital Nilda da Conceicao Felipe, Enf SMI Anizia Felisberto Capossa, Enf SMI/Messica DPS Director, Manica Medical Chefe, Manica Clinical Director, HF I de Maio Chimoio Vila de Manica, MCH Nurse Vila de Manica, Gender Focal Point Vila de Manica, HIV Focal Person Claila Julicio Nasshanganhe, Responsauel de Estatistica (CHASS-SMT since 2011), ME Elisa Mpfumo, Elisampfumo2015@gmail.com, SDMAS Gender Focal Person, GBV Ulga Rejge, Director Gondola H, Director HF Olivia Francisco Bene, Tel: 845711213 (Refused to be recorded; been at HF since 2012), SMI Josefe Falstino do Silva Nola, Technica de Admininstracao Hospital, 825808180, Josefanola280@yahoo.com Manica District DAF Jeremias Wandi, Technico Estatistica Sanitaria, Responsavel de NED, 845566094 (been at district since 2012) Garvalho Filipe Fido, Farmaciatico, 845205464 (been at hospital 10 months), Pharmacy Pita Vaseo Daniels, 82320487, Tec Medicine General (New 3 months), Director HF Narvia Beira, 824914896, lundsbeira@gmail.com, (Sub-Accords Officer), CHASS SMT Manica Gima Madige, 826684299, gimomad74@gmail.com, (Sub-Accords Officer), CHASS SMT Manica Emilio Nhamirre -824237171, emilionhamirre@gmail.com, (Logistics and Supply Officer), CHASS SMT Manica Antonio Avaiano Manjate -823075397, Antonio.manjate@yahoo.com, (Team Leader HSS), CHASS SMT Manica Amos Lucas Amosse, 823917777/843917777, amos23234@gmail.com, (CHASS Provincial M&E), CHASS SMT Manica Tete Province Gilda Joaquim Nabasso, Resp Distrital HIV Osvaldo Roldao dos Santos, GBV/Changara Ana Bela Pires Joao, Enf SMI Chipenbere Henrique Vilanculo, Rsp Genero, Prov Tete Milagros Sidoe, Coord SMI Prov Tete Dias Bartolomeu Maibeque, M&E Prov Tete Performance Evaluation of the CHASS-SMT Project 59 Nogueice Ismail Faguir Rassue Joaquim Alfinete, Dir Clinico Centro Saude, Tete City Arlete Amelia Clemente Picardo Pontes, Enf SMI IP staff, Provincial Technical Coordinator Changara, H/F Director Centro Saude 4 Clinical Director Director District 17/9/2015 Kati Marina Manual, 844813063/823435387, and Baragoa Vaseo Semba (842216626), Pharmacy HF Fungai Patroque Chiringa, Tel: 861012047, Technical Director, Director HF Amandio Augusto Sulo Gentos HIV/SIDA Tete, amandio.sulo@gmail.com, Tel: 828684745, Provincial HIV/AIDS Clinical TO Manuel Joan, Responsavil do Envovimento Communitonio, Tel: 820633743/848771115, Provincial HIV/AIDS Community TO MOH Dr Alene Couto HIV Chief 60 Performance Evaluation of the CHASS-SMT Project ANNEX IV: SOURCES OF INFORMATION List of Documents Reviewed  Clinical HIV/AIDS System Strengthening Request for Applications, 2009  CHASS SMT annual work plans and quarterly and annual reports (2011, 2012, 2013, 2014)  Site visit reports  OIG findings and report of CHASS SMT audit  Public Financial Management Risk Assessment Framework reports for the four provinces  MOH health sector strategy  MOH HIV Acceleration Plan  Data from PEPFAR annual and semiannual reports—this includes both community and clinical data (2011, 2012, 2013, 2014)  Analysis conducted based on PEPFAR annual and semiannual reports (2011, 2012, 2013, 2014)  Partner- and USAID-conducted data quality assessments Performance Evaluation of the CHASS-SMT Project 61 ANNEX V: SUMMARY OF QUANTITATIVE DATA Table A1. Indicators of Project Progress Indicator/Province FY 2011 FY 2012 FY 2013 FY 2014 % of 2014 Target Achieved† Number currently enrolled on ART Sofala 26,387 (20,935) 26,246 (32,323) 49,176 (32,555) 70,565 (45,303) 156% Manica 16,440 (14,132) 20,637 (20,053) 30,277 (25,434) 49,346 (34,361) 144% Tete 15,067 (13,039) 14,133 (17,636) 22,745 (19,050) 35,677 (27,386) 130% Total 57,894 (48,106) 61,016 (70,012) 102,198 (77,039) 155,588 (107,879) 144% Number newly enrolled on ART Sofala 7,248 11,387 16,596 21,529 134% Manica 3,391 8,262 9,483 16,226 87% Tete 3,316 4,273 6,402 9,920 106% Total 13,955 23,922 32,481 47,675 108% Currently on ART (men) Sofala 8,369 7,898 14,280 19,401 123% Manica 5,473 6,433 9,149 13,575 116% Tete 5,370 4,821 7,593 11,506 119% Total 19,212 19,152 31,022 44,482 120% Percent currently on ART who are men Sofala 32% 30% 29% 27% Manica 33% 31% 30% 28% Tete 36% 34% 33% 32% Total 33% 31% 30% 29% Percentage of currently on ART who are children Sofala 9% 10% 9% 9% Manica 8% 8% 7% 8% Tete 8% 7% 8% 7% Total 8% 8% 8% 8% Percentage of newly enrolled (All) Sofala 27% 43% 34% 31% Manica 21% 40% 31% 33% Tete 22% 30% 28% 28% Total 24% 39% 32% 31% Percentage of newly enrolled children 62 Performance Evaluation of the CHASS-SMT Project Indicator/Province FY 2011 FY 2012 FY 2013 FY 2014 % of 2014 Target Achieved† Sofala 11% 11% 10% 9% Manica 8% 8% 7% 9% Tete 8% 10% 8% 8% Total 10% 10% 9% 9% Number of women registered at ANC* Sofala 68,684 73,748 96,540 Manica 80,133 79,007 95,257 Tete 74,351 81,852 103,470 Total 223,168 234,607 295,267 Percent of women receiving HIV test and results in a PMTCT ANC setting (First Test) * Sofala 95% 92% 87% Manica 86% 92% 91% Tete 82% 82% 86% Total 87% 89% 88% Percent of pregnant women with known HIV status (newly tested and known positive at ANC entry) * Sofala 99% 98% 91% Manica 88% 95% 93% Tete 84% 86% 89% Total 90% 92% 91% Number of HIV-positive pregnant women who received ARVs to reduce risk of MTCT* Sofala 11,213 10,774 9,447 95% Manica 6,101 5,732 6,487 78% Tete 4,634 6,110 5,055 112% Total 21,948 22,616 20,989 92% Percent of HIV-positive pregnant women who received ARVs to reduce risk of MTCT* Sofala 100% 89% 85% Manica 73% 78% 87% Tete 92% 90% 86% Total 88% 86% 86% Number of HIV-positive pregnant women in ANC who have initiated Cotrimoxizole* Sofala 2,528 3,936 7,724 155% Manica 1,714 2,859 4,750 114% Tete 1,032 1,888 3,043 135% Total 5,274 8,683 15,517 136% Percent of HIV-positive pregnant women in ANC who have initiated Cotrimoxazole* Sofala 22% 32% 70% Manica 21% 39% 63% Tete 20% 28% 52% Performance Evaluation of the CHASS-SMT Project 63 Indicator/Province FY 2011 FY 2012 FY 2013 FY 2014 % of 2014 Target Achieved† Total 21% 33% 64% Percent of infants born to HIV-positive women who received an HIV test within 12 months of birth* Sofala 69% 54% 58% Manica 30% 45% 47% Tete 35% 39% 51% Total 45% 46% 52% Percent of HIV test results for infants born to HIV-positive women who received an HIV test within 12 months of birth that are positive* Sofala 11% 10% 7% Manica 8% 6% 6% Tete 11% 9% 8% Total 11% 9% 7% Number of partners of women who are HIV tested in a PMTCT setting* Sofala 834 2,534 6,048 20% Manica 562 1,456 2,629 8% Tete 348 1,297 3,576 10% Total 1,744 5,287 12,253 12% Percent of partners of women who are HIV tested in a PMTCT setting* Sofala 1% 3% 6% Manica 1% 2% 3% Tete 1% 2% 3% Total 1% 2% 4% Number of HIV-positive patients receiving a minimum of one clinical service Sofala 37,812 134,727 42,795 84,056 176% Manica 15,177 83,107 28,164 45,378 144% Tete 111,595 138,143 31,057 41,064 138% Total 164,584 355,977 102,016 170,498 156% Percent of HIV-positive patients receiving a minimum of one clinical service (Male) Sofala 34% 35% 33% 33% Manica 34% 35% 34% 32% Tete 37% 36% 39% 35% Total 35% 36% 35% 34% Percent of total HIV-positive patients receiving cotrimoxazole prophylaxis in last visit Sofala 21% 16% 44% 49% Manica 52% 58% 53% 67% Tete 31% 68% 38% 67% Total 31% 46% 45% 58% Percent of HIV-positive patients screened for TB at last visit in HIV care Sofala 38% 56% 64 Performance Evaluation of the CHASS-SMT Project Indicator/Province FY 2011 FY 2012 FY 2013 FY 2014 % of 2014 Target Achieved† Manica 50% 64% Tete 36% 54% Total 41% 58% † For indicators without targets in 2014, the column is left blank *No data were available for most of the PMTCT related indicators in FY 2011 and are left blank in the table Performance Evaluation of the CHASS-SMT Project 65 ANNEX VI: EVALUATION TEAM MEMBERS The evaluation team was external to the CHASS project and consists of members of AGEMA Consultoria Lda. Dexis Consulting Group’s Global Health Program Cycle Improvement (GH Pro) project provided technical assistance to the evaluation team. Peter Symon Wandiembe, PhD (AGEMA Consultoria Lda), Team Leader Rosemary Barber-Madden, PhD (GH Pro), Senior Technical Specialist Esther Kazilimani-Pale, MPH, (AGEMA Consultoria Lda), Senior Evaluation Officer/Public Health and Gender Specialist Verona Parkinson, PhD (AGEMA Consultoria Lda), Evaluation Team Member and Project Director Jeffrey Kilama, MSc (AGEMA Consultoria Lda), Data Manager and Analyst Danubio Cumbane (AGEMA Consultoria Lda), Evaluation Team Member and Data Collector Three data collectors U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523