October 2014 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by MIDEGO Inc.: Ruth Hope, Team Leader with Saul Onyango and Celso Mondlane. Eliminating Pediatric AIDS in Swaziland Project Evaluation Report ii iii ELIMINATING PEDIATRIC AIDS IN SWAZILAND PROJECT EVALUATION REPORT: EVALUATION OF THE LARGEST DONOR PROJECT SUPPORTING THE SWAZILAND EFFORT TO ELIMINATE PEDIATRIC AIDS AND KEEP THEIR MOTHERS ALIVE August 2014 Project Cooperative Agreement No. 674-A-00-l l-00009-00; Evaluation Mechanism Number: USAID Contract No. AID-674-C-14-00007 Additional information can be obtained from: MIDEGO, Inc. 4710 Olley Lane Fairfax VA 22032 Ph: 703-978-3331 Fx: 703-991-8293 www.midego.com DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Responsibility for the contents lies entirely with MIDEGO and the authors. iv ACKNOWLEDGEMENTS The evaluation team would like to thank the Swaziland Ministry of Health for all assistance provided with conducting the evaluation: Mrs. Rejoice Nkambule, Deputy Director Health Services, and Dr. Velephi Okello, National ART Coordinator, Swaziland National AIDS Programme for briefing the evaluators; Mumcy Twala, Lindiwe Shongwe, and Thabsile Gama who assisted the field visits with introductions to the Regional Health Management Teams and health facility staff; and all the Ministry of Health participants in evaluation key informant interviews and site visits. We are most grateful for your assistance, taking time out from your busy schedules to assist the evaluation. The evaluators are also grateful for the many health service users who met with the evaluation team in small groups and occasionally one on one, to give their perspectives on prevention of mother-to-child transmission of HIV in Swaziland. The Swaziland PEPFAR Team provided invaluable briefing of the evaluation team and shared additional documents that provided context and greater understanding for the evaluation. The evaluation team would also like to thank the EPAS partners and their staff, and the other PEPFAR implementing partners who responded to the evaluation questions thoughtfully and fully. Ruth Hope, Team Leader Saul Onyango, PMTCT Expert Celso Mondlane, Evaluation Expert/Data Analyst Gwen Hadfield, Local Logistics & Admin Consultant v CONTENTS ACKNOWLEDGEMENTS ............................................................................................................................. iv ABBREVIATIONS & ACRONYMS ............................................................................................................... vii EXECUTIVE SUMMARY .............................................................................................................................. ix EVALUATION PURPOSE, FRAMEWORK & EVALUATION QUESTION ................................................... 1 Evaluation Purpose ....................................................................................................................................... 1 The Evaluation Framework ........................................................................................................................... 1 Evaluation Questions .................................................................................................................................... 2 BACKGROUND ............................................................................................................................................ 3 EVALUATION METHODS & LIMITATIONS ................................................................................................. 7 Triangulation ................................................................................................................................................. 8 Sampling ....................................................................................................................................................... 8 Limitations of the Evaluation and its Findings ............................................................................................... 9 FINDINGS, CONCLUSIONS & RECOMMENDATIONS ............................................................................ 11 Discussion of the Findings .......................................................................................................................... 11 Conclusions ................................................................................................................................................. 45 Lessons Learnt ............................................................................................................................................ 49 Recommendations ...................................................................................................................................... 49 ANNEXES ...................................................................................................................................................... I ANNEX I: EVALUATION SCOPE OF WORK ............................................................................................... II ANNEX II: EPAS EVALUATION FRAMEWORK ....................................................................................... XIII ANNEX III: EVALUATION METHODOLOGY .......................................................................................... XXII ANNEX IV: DATA COLLECTION INSTRUMENTS................................................................................ XXVII ANNEX V: SOURCES OF INFORMATION ......................................................................................... XXXV 1. Persons The Evaluation Team Met With ......................................................................................... XXXV 2. Documents And Other Resources Accessed..................................................................................XXXVIII ANNEX VI: DISCLOSURE OF ANY CONFLICTS OF INTEREST .................................................................. XLVI vii ABBREVIATIONS & ACRONYMS AIDS acquired immunodeficiency syndrome AMICAALL Alliance of Mayors’ Initiative on Community Action on AIDS at Local Level ANC(s) antenatal clinic(s) ART antiretroviral therapy ARV antiretroviral (drug) AZT zidovudine CTX cotrimoxazole DBS dried blood spot DQA data quality audit EGPAF Elizabeth Glaser Pediatric AIDS Foundation EID early infant diagnosis EPAS Eliminating Pediatric AIDS in Swaziland GOKS Government of Kingdom of Swaziland HCW health care worker HIV human immunodeficiency virus HTC HIV testing and counseling IEC information education and communication M&E monitoring and evaluation m2m mothers2mothers MM(s) mentor mother(s) MNCH maternal, newborn & child health MOH Ministry of Health MTCT mother-to-child transmission [of HIV] NVP nevirapine PCR polymerase chain reaction PEPFAR The United States President’s Emergency Plan for AIDS Relief PHU Public Health Unit PMTCT prevention of mother-to-child transmission [of HIV] PNC post natal care viii RHMT(s) Regional Health Management Team(s) SAM Service Availability Mapping SDHS 2006 Swaziland Demographic and Health Survey of 2006 SHIMS Swaziland HIV Incidence Measurement Survey SNAP Swaziland National AIDS Programme SOW scope of work SRH sexual and reproductive health SRHU Sexual and Reproductive Health Unit TA technical assistance UN United Nations USAID U.S. Agency for International Development ix EXECUTIVE SUMMARY EVALUATION PURPOSE AND EVALUATION QUESTIONS The goal was to assess the Eliminating Pediatric AIDS in Swaziland (EPAS) project effectiveness in supporting and strengthening the technical capacity for a high quality, integrated and comprehensive prevention of mother-to-child transmission of HIV (PMTCT) program in Swaziland. The purpose was to:  assess the quality of implementation  document lessons learnt  explore challenges and accomplishments, and  provide strategic guidance for the remaining years of the project and any follow-on activities of the USAID/EPAS project The primary audiences for the evaluation report are the Swaziland Ministry of Health (MOH), and the EPAS implementers: The Elizabeth Glaser Pediatric AIDS Foundation (EGPAF) and its partners, along with PEPFAR/Swaziland, and USAID/Southern Africa. The USAID/Washington Global Health Office is a further important audience. The final evaluation questions to be addressed were: 1. To what extent have the project's four main objectives been achieved; what have been the successes, failures and gaps in EPGAF's approach? a) Expanded PMTCT services b) Comprehensive and quality PMTCT services integrated into antenatal care (ANC), labor & delivery and post-natal care (PNC), including antenatal & postnatal visits, HIV counseling and testing and antiretroviral uptake, adherence, facility deliveries, HIV transmission rates at 6-8 weeks, etc. c) National PMTCT system strengthened d) Protocols, guidelines, and job aids developed with MOH 2. What were the successes, challenges and gaps in EPAS’s community approach? 3. How sustainable are the gains made under EPAS? How many facilities have been graduated? How many have improved, and to what extent? What factors contributed to sustainable improvement? 4. How has EPAS strengthened maternal, newborn and child health (MNCH) services more broadly? For example, in the areas of family planning, prevention and management of obstetrical complications, newborn health, infant feeding and reducing post-natal transmission of HIV? What are the remaining gaps? 5. What are the remaining challenges to improving PMTCT outcomes in Swaziland? 6. How effective has been EPAS’s approach in providing facility level support? PROJECT BACKGROUND EPAS is a five-year agreement value $16,495,000 being implemented between October 2010 and September 2015. The overall goal of the EPAS project is to eliminate pediatric AIDS in Swaziland, x by achieving the following objectives: 1) To achieve universal access to PMTCT through increasing geographical coverage of services and addressing social and gender norms that create barriers to service uptake and retention in care 2) To provide and sustain quality, comprehensive and integrated PMTCT services through supporting clinical mentoring and supportive supervision of health care workers at health facilities across the country 3) To strengthen the national health systems in line with the Ministry of Health plans for PMTCT through technical assistance (TA) and capacity building to ensure sustainability 4) To support the MOH develop and review PMTCT policies, protocols and guidelines on a regular basis EVALUATION QUESTIONS, DESIGN, METHODS AND LIMITATIONS The evaluation used a four-level evaluation approach: national, regional, facility, and the community and employed five methods of data collection, triangulation and verification of the information: 1. document review; 2. email survey of EGPAF and sub-partners’ staff; 3. analysis of secondary data including from the HMIS; 4. key informant interviews and guided group discussions; and 5. checklists for focused observations. An Evaluation Framework that detailed the evidence to be collected, the sources of the data, and methods of data analysis was completed. The evaluation is a programmatic performance evaluation that includes analysis of primary qualitative data collected by the evaluation team. It provides re-aggregation and analysis of the MOH health management information system (HMIS) quantitative data to identify results achieved from 2010-2013, the first three years of the project implementation. The sample provided valuable qualitative understanding on what is going well and less well. The analysis supports the recommendations for the last year of EPAS implementation and for any follow on activity. FINDINGS AND CONCLUSIONS 1. To what extent have the project's four main objectives been achieved; what have been the successes, failures and gaps in EPGAF's approach? a) Expanded PMTCT services. The push to expand the number of health facilities in Swaziland providing PMTCT services preceded the EPAS project. There were 8% more facilities providing PMTCT service in 2013 than there were in 2010. However, EPAS support to health facilities occurred over the period when uptake of PMTCT services increased—by as much as 33.5% from 2010 to 2013 for pregnant women HIV tested and receiving their result. xi Health facilities also performed better in terms of the proportion of eligible clients receiving PMTCT services. In 2013 in EPAS supported facilities, 98% of pregnant women were tested for HIV and received their result. The proportion of HIV positive pregnant women initiated on cotrimoxazole in EPAS supported facilities, increased from 31% in 2010 to 97% in 2013; in 2013, 97% of exposed infants in EPAS supported facilities had DBS taken at 6-8 weeks of age for early infant diagnosis. The rates for the 18 health facilities providing PMTCT services not yet supported by EPAS, were lower but the numbers were small. Over all facilities providing PMTCT, ninety-eight percent of pregnant women in Swaziland attending ANC were HIV tested and received their result in ANC. Ninety-five percent of known exposed infants had DBS taken for early infant diagnosis. It is estimated that 98% of pregnant women in Swaziland access ANC at least once. Thus it is possible to estimate the rate for all pregnant women in Swaziland accessing HIV testing and receiving their result in ANC—96%; and the overall rate for exposed infants having DBS at 6-8 weeks of age—93%. Conclusion: If “universal access” is defined as 90%, EPAS has supported the MOH to attain universal access for pregnant women in Swaziland to be HIV tested and receive their result in ANC. EPAS has also supported the MOH to attain universal access for exposed infants having DBS taken for PCR at 6-8 weeks of age. While not yet achieving universal access to the other PMTCT services, EPAS has achieved its objective of expanded PMTCT services. b) Comprehensive and quality PMTCT services integrated into ANC, labor & delivery and post￾natal clinics, including ANC & PNC visits, HTC and ARV uptake, adherence, facility deliveries, HIV transmission rates at 6-8 weeks, etc. EPAS has supported the integration of a comprehensive package of PMTCT services into the MNCH platform at health facilities throughout Swaziland. Mentor mothers, are a vital part of the delivery of PMTCT services in ANC & PNC supporting HTC and ARV uptake as well as adherence, and PMTCT behaviors such as exclusive breastfeeding for the first 6 months and safer sex with condoms during pregnancy and lactation. EPAS reviews the performance of health facilities by grading the performance against PMTCT indicators and tailors its level of TA to health facilities by the grade. High volume sites, Maternity Units, newly supported sites and sites that are performing poorly are visited monthly and receive other TA such as onsite training. Sites with performance that falls to medium performance are visited twice a quarter; after two years of monthly EPAS mentoring, consistently high performing sites “graduate” to quarterly mentoring visits. Some high performing sites’ performance has rapidly deteriorated—for example when key staff in small facilities go on leave or when there is staff turnover at larger facilities with incoming staff not having had PMTCT training. Currently, two health centers and 22 clinics meet the EPAS requirements for facility graduation. EPAS has not defined PMTCT service quality more widely than service performance. The xii evaluation assessed service quality from the pillars of structure, process, and outcome, as well as from the perspectives of the health service manager, the medical professional, and the client, and found good quality services in the PHUs, health centers and clinics visited, although the premises were frequently cramped and congested. The main weakness was in Maternity Units where there was general lack of privacy and respect for the dignity of the birthing women. In two units, birthing women were seen to be laboring completely naked, in open wards with nurses and students walking in and out. Women were observed to be walked—naked—from the labor ward to the delivery room. One of four Maternity Units visited had run out of infant nevirapine in the previous 6 months, attributed to poor record keeping at peak times and failure to reorder on time. As there isn’t an indicator that includes mother-baby pairs, the HMIS doesn’t track mother-baby pairs. EPAS has supported the MOH to introduce ART services at PHUs for HIV positive pregnant women, HIV positive mother-baby pairs until the child reaches 2 years old and is negative on antibody testing, or until the HIV infected child reaches 5 years old. The care of the mother and her older child living with HIV is then transferred to the Hospital ART clinic. The care of mothers on ART, or ARV prophylaxis during lactation, and their infant/child is tracked by the PHU ART clinic where Expert Clients follow up positive women who miss an appointment. The provision of ART in PHUs facilitates a “family centered approach” and reduces the burden of health care visits on the mother. Exposed infants of mothers who are not on ART or ARV prophylaxis are followed after the 6-8 weeks post natal check in the child welfare clinic. Child health cards now include a line for the mother’s HIV status and this stimulates the health worker to request DBS and HIV antibody tests when the infant returns to the child welfare clinic. If the mother’s status is initially negative, the child welfare card stimulates the health worker to initiate HTC for the mother every 8 weeks, when the infant is seen in the child welfare clinic. Mentor mothers follow up with mothers of exposed infants who do not return for DBS at 6-8 weeks. However there is no other active tracing of exposed infants who are not brought for follow up. EPAS supported facilities achieved an HIV transmission rate at 6-8 weeks of 2.85% in 2013. The overall HIV transmission rate at 6-8 weeks for all health facilities in Swaziland providing PMTCT services was 2.89% in 2013. Conclusions: EPAS has in general achieved its objective of good quality comprehensive, PMTCT services integrated into the ANC, PNC and child welfare services of the MNCH platform. In maternity units, the lack of definition of quality in PMTCT care, and lack of service delivery standards enabled EPAS to miss the unacceptable standard of care in labor and delivery wards. This remains to be addressed in the 5th year of implementation. A proactive approach to following mother-baby pairs in child welfare clinics—particularly for exposed infants of mothers who are not on ART/ARVs and infants of initially negative mothers who may have sero-converted during breastfeeding is a challenge with the fall off in uptake of PMTCT services after the 6-8 week postnatal check. The HIV transmission rate at 6-8 weeks is an excellent achievement. xiii c) National PMTCT system strengthened The national PMTCT system has been strengthened at national policy-level as well as at regional and health facility levels. EGPAF is reported to be a “trusted partner” providing TA when the MOH is short of technical capacity at policy level. EPAS has funded posts in the MOH, some but not all of which have been absorbed: initially a training officer and supervision officer in the Sexual and Reproductive Health Unit; later an MNCH advisor was added and an ART physician at King Sobhuza II Clinic, who supported ART at other facilities, too. EPAS has invested heavily in training: initial offsite, onsite refresher training, and mentoring. This has undoubtedly strengthened the MOH capacity to consistently provide quality PMTCT services. Health worker performance delivering the package of PMTCT services integrated into MNCH has in general increased from 2010 to 2013, supported by EPAS mentoring. Although EPAS has also provided considerable assistance to the HMIS/M&E system at national, regional and health facility levels the evaluation identified a weakness in the quality of the monthly reporting by health facilities. A major issue is the burden of reporting the large number of indicators that have data collected routinely. The evaluation looked at the proportional agreement between the registers for 2013 and the monthly reports for the same period for three indicators. There was variance with some facilities under￾reporting and others over-reporting for all three indicators. The number of pregnant women HIV tested and receiving their result this month showed the greatest variance as is shown in the figure where less than 100% indicates under reporting and more than 100% indicates over reporting. Conclusions: EPAS training, mentoring and other TA has supported health facilities to increase their performance providing PMTCT services integrated into MNCH from 2010 to 2013. EPAS has worked with the MOH to strengthen the MNCH platform in PHUs, health centers and clinics as far as it is able within its manageable interest and within the limitation on use of PEPFAR funding imposed by the U.S. Congress. To fully achieve this objective, EPAS must work with the RHMTs to address the quality of reporting by health facilities. The burden of reporting might be reduced by prioritizing which data is essential to be included in the monthly reports and which items could be collected through periodic health facility surveys and which indicators might be better calculated by periodic population-based survey. Improvements in the registers—particularly in the user Figure: Proportion of agreement between register book and reports - Number of pregnant women tested for HIV at the first ANC visit, 2013 92% 108% 95% 80% 74% 108% 77% 105% 57% 96% 100% 103% 0% 20% 40% 60% 80% 100% 120% GOOD SHEPHARD HOSPITAL MANKAYANE HOSPITAL EMKHUZWENI HC SITHOBELA HC LOMAHASHA CLINC NTFONJENI CLINIC LUBULI CLINIC LUSHIKISHINI CLINIC MAWELAWELA MLIBA NAZ CLINIC SILELE RED CROSS CLINIC KING SOBUZA II CLINIC xiv friendliness of the family planning register—might also reduce the burden of reporting. Reduced burden of reporting when combined with supportive supervision and mentoring that acknowledges and rewards (praises) service providers whose performance has improved is likely to improve the accuracy of the monthly reporting. d) Protocols, guidelines, and job aids developed with MOH EGPAF inter alia contributed to the development of the Revised National PMTCT Guidelines, the National Strategic Framework for accelerated action for the elimination of new HIV infections among Children by 2015 and keeping their mothers alive, and assisted the MOH with the development of the new HMIS tools (registers) that mirrored the National Strategic Framework and funded the printing of the tools. The evaluation noted many job aids posted on health facility walls that were provided by EPAS. Conclusion: EPAS has contributed TA greatly valued by the MOH to the development and printing of protocols, guidelines, and job aids—developed with the MOH at policy level. EPAS has achieved this objective. 2. What were the successes, challenges and gaps in EPAS’s community approach? EPAS community approach is organized through its community linkages team that seeks to empower communities with PMTCT knowledge through community dialogues, school debates and children and adolescents support group meetings. Male dialogues target male dominated industries. Community events organized by EPAS partners are successful in mobilizing 200-300 persons per event raising awareness through drama and facilitating peer group discussions to reinforce lessons learnt from the dramas about topics such as women’s status, their lack of agency in health decision making related to MNCH and PMTCT, gender-based violence and stigma. Voluntary HIV counseling and testing is provided at the event by PSI. Rural Health Motivators reported that these events stimulate more men to engage in discussion with the Motivators when they go door to door—for a period, but then the men’s engagement tails off back to earlier low levels. The EPAS approach is not evidence-based yet it has not been set up as a pilot intervention to demonstrate its effectiveness. No routine data is collected to monitor the effectiveness—at outcome level—in changing PMTCT behaviors and use of services. Conclusion: There is great scope for addressing male involvement, post-delivery PMTCT behaviors and service uptake, and AIDS stigma within the family using the EPAS approach to community mobilization. However, the community approach is not evidence-based and has not been established as a pilot project with monitoring of the outcomes to demonstrate effectiveness. 3. How sustainable are the gains made under EPAS? How many facilities have been graduated? How many have improved, and to what extent? What factors contributed to sustainable improvement? EPAS has worked extensively with the RHMTs and the approach to mentoring is well established with the MOH committed to the approach at national, regional and health facility levels, and the MOH reported that it will continue with mentoring. However, the MOH has not yet finalized its xv requirements for mentors. The MOH reported that its thinking is moving towards integrating PMTCT and MNCH mentor functions so that all RHMT mentors support integrated delivery of PMTCT/MNCH services. With regular mentoring and supportive supervision by the RHMTs, and on-site training for new health care workers, the gains in delivery and uptake of PMTCT services should continue after EPAS. Apart from the weakness the evaluation identified in health facility reporting, the HMIS system is working well and, as a government system, will continue beyond EPAS. The RHMTs are well versed through working with EPAS in the need for monthly data quality review meetings to review the monthly reports from health facilities and identify gaps [missed opportunities] for providing PMTCT services. The mentor mother program is not institutionalized and neither are the community programs implemented by EPAS; AMICAALL and Lutsango community programs cannot continue after EPAS without funding. Conclusions: Implementing through the existing health care system in support of the National Strategic Framework has institutionalized the gains. The enhanced health worker PMTCT knowledge and skills will continue while they are employed providing PMTCT services. The weaknesses in the monthly reporting by health facilities indicate a need for a new approach to mentoring and potentially to the collection of performance data to improve reporting accuracy. The mentor mother program, AMICAALL and Lutsango community activities require donor funding to continue. Retention of the EPAS PMTCT Coordinators who are seconded to the RHMTS is likely to be a challenge. These positions may not be appropriate within the emerging MOH policies and plans for integrated mentorship. Without systematic and iterative raising of socio-cultural and gender barriers to use of PMTCT behaviors and services, and changes in gender norms and health seeking behaviors stimulated by EPAS partners are unlikely to be sustained beyond EPAS. 4. How has EPAS strengthened MNCH services more broadly? For example, in the areas of family planning, prevention and management of obstetrical complications, newborn health, infant feeding and reducing post-natal transmission of HIV? What are the remaining gaps? EPAS training and mentorship has addressed knowledge, skills and approaches to client care that have wider resonance through the delivery of all MNCH services. Health workers report that they have received EPAS training and mentorship in family planning, dual protection, exclusive breastfeeding, and complementary feeding, as well as in compassionate care for clients—topics that benefit both postnatal PMTCT, and wider MNCH clients. Job aids printed [and some laminated] seen by the evaluation supported wider MNCH services. Improvements in client flows at health facilities that EPAS initiated, benefit all MNCH clients. EPAS has provided training to Maternity Unit staff in reducing PMTCT in maternity care, and use of the partogram during labor—knowledge and skills that benefit all birthing mothers and infants. EPAS has xvi donated small equipment to Maternity Units and PHUs that benefit wider MNCH clients. The evaluation identified the remaining need for strengthening of the platform—beyond use of the partogram and teaching midwives about PMTCT—in Maternity Units. The evaluation observed widespread lack of respect for birthing women’s privacy and dignity: birthing women were naked without covers in all the Hospital Maternity Units and in two Units women were walked naked from the labor ward to the delivery room. EPAS mentoring in Maternity Units had not identified the lack of respectful care for birthing women. Conclusions: EPAS has in general strengthened the MNCH services as far as possible within its manageable interest and within the limits on use of PEPFAR funding imposed by the U.S. Congress. There remains a need for EPAS to focus on addressing quality of care in Maternity Unit labor and delivery wards in the last year of implementation. 5. What are the remaining challenges to improving PMTCT outcomes in Swaziland? The challenges remaining to improving PMTCT outcomes include: (1) Sustaining the standards of integrated PMTCT/MNCH clinical service delivery as personnel change at health facilities, through onsite training, and regular, supportive supervision and quality mentoring by RHMTs (2) Lack of point of service CD4 count laboratory equipment (3) The need to address health care delivery around birthing to ensure respectful clinical care in labor and delivery wards (4) Effective interventions to reduce socio-cultural and gender barriers to uptake of ART by HIV positive pregnant women. Male involvement is increasing but still lags behind the level needed to achieve universal uptake of PMTCT services beyond the 6-8 week postnatal visit; stigma within the family and self-stigma remain significant issues. (5) Effective strategies to increase retention of mother-infant pairs in follow-up, increasing use of PMTCT services after the 6-8 week post-delivery checkup and increasing retesting of exposed infants. Conclusions: Lack of CD4 laboratory equipment will reduce as a barrier with the roll out of Option B+. Effective strategies for retaining mother-infant pairs in PMTCT follow up are needed. Socio-cultural and gender barriers to improving PMTCT outcomes remain challenges to be effectively addressed. 6. How effective has been EPAS’s approach in providing facility level support? EPAS approach to facility level support has been to “work within” the health system and avoid setting up parallel support systems. Co-locating EPAS Program Coordinators with RHMTs has leant itself to close working relations and traveling together on supervision/mentoring visits to health facilities. The combination of initial offsite training with follow up onsite refresher training, along with regular mentoring has demonstrated effectiveness in supporting integration of PMTCT/MNCH service delivery. The evaluation found that the quality of antenatal and postnatal services in the PHUs, health centers and clinics it visited was good. However there were weaknesses in the quality of care provided by maternity units. The EPAS approach to strengthening PMTCT service delivery with its PMTCT mentors working separately from RHMT xvii MNCH mentors, has been effective but is becoming dated now that service delivery is integrated. Continued vertical support to PMTCT rather than integrated support to comprehensive MNCH services might discourage “joined up thinking” in service delivery. Conclusions: EPAS approach to providing facility level support through training, in service training and mentoring was effective in integrating PMTCT services in to the MNCH platform at PHUs, health centers and clinics. EPAS emphasis on performance against specific PMTCT indicators to the exclusion of a wider focus on delivery of quality PMTCT services let EPAS mentors miss the unacceptable quality of basic midwifery care to birthing mothers in maternity units. RECOMMENDATIONS 1. For the remaining term of the EPAS project, EPAS should take a fresh approach to clinical mentoring, and to its approach to reviewing performance reports and use of HMIS data with health facility staff. 2. There remains a huge need for the MOH to improve the quality of care of laboring and birthing women in maternity units. Midwifery care must respect women’s rights to be treated with dignity and have privacy. 3. EPAS partners should document the outcomes of their community activities in the last year of implementation to demonstrate that their community approaches are making a difference to male involvement and use of PMTCT behaviors and services. 4. EPAS senior management should review and formalize the EPAS exit plan with the MOH, identifying (1) the gains that need sustaining beyond the end of project; (2) what the MOH will be able to provide to sustain the gains; and (3) how EPAS implementation in the final year will move towards sustaining its gains. 5. Health development implementing partners should coordinate their support and visits to health facilities to minimize the burden on health facility staff, to maximize the potential for complementarity and to reduce duplication of effort. EPAS staff should continue to its approach of working from within, to support the RHMT leadership and to develop and implement a strategy for retaining mother-infant pairs in follow up until after cessation of breastfeeding, and increasing retesting of exposed infants. 6. Although m2m is no longer a subpartner of EGPAF and thus not within the purview of this evaluation, the evaluation identified the value of MMs work as a contribution to the quality of PMTCT care at the facilities visited. There is need for expansion of the m2m program with continuation after the EPAS project. 7. USAID should focus any follow on activity to EPAS on addressing demand side barriers to xviii uptake of PMTCT services and adoption of new behaviors, and to increase retention of mother￾baby pairs in PMTCT services in the child welfare clinics until after the infants cease breast feeding and increase retesting of HIV exposed infants. There should be emphasis on increased uptake of ART. 1 EVALUATION PURPOSE, FRAMEWORK & EVALUATION QUESTIONS EVALUATION PURPOSE The external performance evaluation of the Eliminating Pediatric AIDS in Swaziland (EPAS) project was requested in line with the policy that encourages The United States President’s Emergency Plan for AIDS Relief (PEPFAR) countries to invest in building evidence to assess the effectiveness of their HIV/AIDS programs. The goal was to assess EPAS effectiveness in supporting and strengthening the technical capacity for a high quality, integrated and comprehensive prevention of mother-to-child transmission of HIV (PMTCT) program in Swaziland. The purpose was to:  assess the quality of implementation  document lessons learnt  explore challenges and accomplishments, and  provide strategic guidance for the remaining years of the project and any follow-on activities of the United States Agency for International Development (USAID)/EPAS project. This evaluation provided independent evidence on the effectiveness of the project and whether it was meeting its intended objectives; as well as detailed information on the elements of the project that worked well, and the ones that worked less well. The evaluation also provided an overall assessment of the implementation model to facilitate decision-making, for instance on whether the model can be scaled-up or which elements of the model can be re-aligned or strengthened for potential follow-on activity. The evaluation was conducted from May to August of Year 4 of implementation to allow for adoption of evaluation recommendations in Year 5 to maximize the effectiveness and to ensure the gains achieved are sustained after the end of the project. The primary audiences for the evaluation report are the Swaziland Ministry of Health (MOH) and the EPAS implementers: The Elizabeth Glaser Pediatric AIDS Foundation (EGPAF), its partners, as well as PEPFAR/Swaziland, USAID/Southern Africa. The USAID/Washington Global Health Office is a further important audience. THE EVALUATION FRAMEWORK The scope of work (SOW) for the evaluation1 included an outline evaluation framework—a matrix of the 6 final evaluation questions with columns for i. evidence to be sought, ii. data sources and collection methods, and iii. data analysis methods. Annex II to this report includes the evaluation framework that was completed by the evaluation team during the inception 1 The full SOW for the EPAS evaluation is included in Annex I 2 phase of the evaluation. EVALUATION QUESTIONS The final evaluation questions to be addressed were: 1. To what extent have the project's four main objectives been achieved; what have been the successes, failures and gaps in EPGAF's approach? a) Expanded PMTCT services b) Comprehensive and quality PMTCT services integrated into antenatal care (ANC), labor & delivery and post-natal care (PNC), including ANC & PNC visits, human immunodeficiency virus (HIV) testing and counseling and ante-retroviral uptake, adherence, facility deliveries, HIV transmission rates at 6-8 weeks, etc. c) National PMTCT system strengthened d) Protocols, guidelines, and job aids developed with MOH 2. What were the successes, challenges and gaps in EPAS’s community approach? 3. How sustainable are the gains made under EPAS? How many facilities have been graduated? How many have improved, and to what extent? What factors contributed to sustainable improvement? 4. How has EPAS strengthened maternal, newborn and child health (MNCH) services more broadly? For example, in the areas of family planning, prevention and management of obstetrical complications, newborn health, infant feeding and reducing post-natal transmission of HIV? What are the remaining gaps? 5. What are the remaining challenges to improving PMTCT outcomes in Swaziland? 6. How effective has been EPAS’s approach in providing facility level support? 3 BACKGROUND A small landlocked country, slightly smaller than New Jersey, the Kingdom of Swaziland has an estimated 2014 population of 1,106,000 to 1,258,121. This is a moderate increase from the 2007 census, which found 1,018,000 residents in the country2 . Swaziland has a very high disease burden as a result of HIV infection. The Swaziland Demographic and Health Survey of 2006 (SDHS 2006) reported that the prevalence of HIV in Swaziland among the general population age 15-49 was 26%, the highest in the world. UNAIDS estimates the prevalence in this age group increased to 27.43 in 2013, still the highest globally. The Swaziland HIV Incidence Measurement Survey (SHIMS) identified a national HIV prevalence of 31% among adults 18-49 years in 20114 . Life expectancy decreased significantly from 58.8 years in 1997, to 43 years in 20075 but this has risen to 49 for the period 2010-20156 . There is a gender disparity with women disproportionately affected and infected at younger ages than men. Women of reproductive age, 15-49 years, have a prevalence of 31%, compared to 20% among the males of similar age range. Among young people aged 15 to 24 the percentage living with HIV in 2013 was 12.4 for females and 7.1 for males7 . The SHIMS findings show that peak in HIV prevalence has shifted to older ages for both sexes to 30 - 34 year olds for women and 35 – 39 year olds for men when compared with the SDHS 20006. Prevalence among women attending antenatal clinics increased more than 10 fold from 3.9 per cent in 1992 to 41.1 per cent in 20108 . In 2013, an estimated 10,000 pregnant women were living with HIV9 with serious implications for mother-to-child transmission of HIV (MTCT) and maternal mortality. Infant HIV infection adversely influences the infant and under-five mortality rates which in 2012 were 56 per 1,000 live births (infant mortality) and 80 per 1,000 live births 2 http://worldpopulationreview.com/countries/swaziland-population/ (accessed Aug 5, 2014). However, the UNICEF estimate for the 2012 population was 1,231,000 http://www.unicef.org/infobycountry/swaziland_statistics.html (accessed July 1, 2014) 3 UNAIDS (2014) The Gap Report. Joint United Nations Programme on HIV/AIDS, Geneva, Switzerland 4 CDC & ICAP-Columbia University (2012) Swaziland HIV Incidence Measurement Survey (SHIMS) First Analysis Report. MOH, GOKS, Mbabane, Swaziland 5 WHO Swaziland Health Status: Analytical Summary http://www.aho.afro.who.int/profiles_information/index.php/Swaziland:Analytical_summary_- _Health_Status_and_Trends accessed June 24, 2014 6 UNFPA (2013) State of the World’s Population 2013 - Motherhood in Childhood. United Nations Population Fund, New York 7 UNAIDS (2014) The Gap Report. Joint United Nations Programme on HIV/AIDS, Geneva, Switzerland 8 WHO Swaziland Health Status: Analytical Summary http://www.aho.afro.who.int/profiles_information/index.php/Swaziland:Analytical_summary_- _Health_Status_and_Trends accessed June 24, 2014 9 UNAIDS (2014) The Gap Report. Joint United Nations Programme on HIV/AIDS, Geneva, Switzerland 4 (under-five mortality), with nearly half of infant deaths related to HIV10. There is evidence that infection with HIV in pregnant women negatively affects the maternal mortality ratio. The maternal mortality ratio rose from an estimated 370 per 100,000 live births in 1995 to 589 per 100,000 live births in 200711, but in 2010 was estimated to be 320 per 100,000 live births12. The Government of Kingdom of Swaziland (GKOS) utilized the opportunity presented by the high rates of utilization of antenatal care to roll out interventions for PMTCT. By the end of 2008, about 67% of HIV infected pregnant women received antiretrovirals (ARVs) for PMTCT prophylaxis13. Revised PMTCT Guidelines were issued in 2010 in support of the national scale￾up of more effective interventions aimed at preventing MTCT, underscoring the strong commitment towards elimination of MTCT in Swaziland by 2015. Supported by funding from PEPFAR through USAID and the other development partners, the strategy for the Swazi program to eliminate MTCT was strengthening the capacity of the existing national health care delivery system. A comprehensive family-centered approach was employed in line with World Health Organization guidance14 to address all four prongs of PMTCT: 1. Primary prevention of HIV infection 2. Prevention of unintended pregnancies among HIV infected women 3. Prevention of HIV transmission from infected mothers to their children, and 4. HIV treatment, care and support for infected women and their families However, there were significant challenges that hindered the MOH in the optimal expansion and provision of sustainable, high quality PMTCT services. The public sector was constrained by inadequate number of health facilities, weak management capacity, insufficient human resources for health, weak procurement and supplies chain management system, plus limited capacity for monitoring and evaluation. In addition, the community mobilization and education system was also weak, and a large proportion of clients were not consistently followed-up demonstrated by a high lost to follow-up rate. In 2010 USAID/Swaziland awarded a Five Year Cooperative Agreement for the EPAS project to EGPAF. The original agreement value was $11,968,250 with cost sharing of $1,115,509 (9.3%). Through a modification in February 2013, the agreement was increased to value $16,495,000. EPAS is a follow-on activity to the global award, Call to Action, which supported the MOH from 2003 to increase access to effective PMTCT services through a direct service delivery model. 10 WHO Swaziland Health Status: Analytical Summary http://www.aho.afro.who.int/profiles_information/index.php/Swaziland:Analytical_summary_- _Health_Status_and_Trends accessed June 24, 2014 11 WHO Swaziland Health Status: Analytical Summary http://www.aho.afro.who.int/profiles_information/index.php/Swaziland:Analytical_summary_- _Health_Status_and_Trends accessed June 24, 2014 12 UNFPA (2013) State of the World’s Population 2013 - Motherhood in Childhood. United Nations Population Fund, New York 13 MOH (2010) Guidelines for the Prevention of Mother-to-Child Transmission of HIV 3rd edition. GOKS, Mbabane, Swaziland 14 Report (2010) Towards the Elimination of Mother-to-Child-Transmission of HIV. WHO, Geneva 5 EPAS is the largest PMTCT project in the country supporting the MOH to achieve the GKOS’ goal of eliminating pediatric AIDS and keeping mothers alive. EPAS project uses PMTCT entry points to strengthen the wider MNCH platform as well as to address sexual and reproductive health issues including prevention of sexual spread of HIV infection. The initial project design had EGPAF as the primary implementer, in partnership with two non￾governmental organizations: mothers2mothers (m2m) and Alliance of Mayors’ Initiative on Community Action on AIDS at Local Level (AMICAALL). m2m was engaged to provide one-on￾one peer education and psychosocial support based on the m2m “mentor mothers” model and community-based support to retaining mothers and infants in follow up, while AMICAALL was engaged to focus on urban-based community interventions and strengthen linkages between urban communities and health facilities. The project funding and structure were revised in 2012 when m2m graduated to being a direct grantee of USAID/Swaziland while AMICAALL remained as sub-grantee but was joined by two new partner nongovernmental organizations: Swaziland Infant Nutrition Action Network (SINAN) and Lutsango Lwakangwane. The overall goal of the EPAS project is to eliminate pediatric AIDS in Swaziland, by achieving the following objectives: 1) To achieve universal access to PMTCT through increasing geographical coverage of services and addressing social and gender norms that create barriers to service uptake and retention in care 2) To provide and sustain quality, comprehensive and integrated PMTCT services through supporting clinical mentoring and supportive supervision of health care workers at health facilities across the country 3) To strengthen the national health systems in line with the Ministry of Health plans for PMTCT through technical assistance (TA) and capacity building to ensure sustainability 4) To support the MOH develop and review PMTCT policies, protocols and guidelines on a regular basis The EPAS project results framework provided by EGPAF is given in Figure 1, below. As the higher level goal is not specifically stated15, the evaluation inferred from EPAS documentation that the Strategic Objective was “Achievement of a Swaziland pediatric HIV infection rate of less than 5% of exposed infants by 2015”. 15 “Moving towards elimination of pediatric HIV” is not worded as a result. 6 Figure 1: The EPAS Results Framework 7 EVALUATION METHODS & LIMITATIONS The MIDEGO evaluation team included as broad and representative involvement of as many stakeholders as possible including USAID/Southern Africa and the Swaziland PEPFAR team. The team conducted key informant interviews with senior MOH policy level personnel, and managers and service oversight staff at national and regional level. They interviewed United Nations (UN) stakeholders; PEPFAR partners; EPAS management, technical and monitoring and evaluation (M&E) staff. The evaluators interviewed health facility and community service delivery personnel; and held discussions with service users and their partners, and others in the community. To address the evaluation questions, MIDEGO used a four-level evaluation approach: national, regional, facility, and the community. The evaluation employed five methods of data collection, triangulation and verification of the information to evaluate the EPAS and answer the evaluation questions: 1. Desk review of the documents made available by USAID Southern Africa and EPAS16; other relevant documentation including local studies, guidelines, and best practice documents; and other Swaziland and Sub-Saharan Africa- specific HIV and PMTCT literature available through the internet or in the evaluators’ personal collections 2. An email survey of EGPAF and sub-partners’ staff to give insights into where more detailed questions should be asked in key informant interviews. [The Survey was limited to EPAS staff because, in Swaziland, the RHMTs and health facility staff do not currently have access to email.] 3. Re-aggregation and analysis of secondary data from the health management information system (HMIS), and EGPAF and partners’ plans, budgets and reports 4. Key informant interviews and guided group discussions with a wide range of stakeholders—including PEPFAR partner staff, national level MOH staff, RHMTs, health care providers and service users—who consent to participate, to obtain a wide range of stakeholders’ perceptions. The evaluators were also briefed in depth by USAID Southern Africa and members the PEPFAR Swaziland team, and by the EPAS Senior Management Team. The evaluators took detailed handwritten notes during the briefings, interviews and discussions which they typed up and analyzed the content. 5. Checklists for focused observations for assessing service delivery at selected facilities and in the community including counseling and testing, antenatal clinics, labor and delivery wards, postnatal clinic, and MNCH services, coordination and oversight The evaluation tools were developed during the inception phase in the evaluators’ home offices, 16 A full list of the documents used in the desk review is included in ANNEX V: SOURCES OF INFORMATION page XXX9,,, 8 and modified in the light of briefings by USAID and the MOH. The check list was adapted as needed during field data collection. The evaluation tools are included in Annex IV. The evaluation used both primary data—that is qualitative and subjective— and secondary data from the routine HMIS, EGPAF reporting to USAID and PEPFAR, and documents and reports from a review of the relevant literature available to the evaluators. The evaluation team was sensitive to time constraints and the ownership of the data by the MOH; and worked with the MOH to set up KIIs with the persons and organizations identified by the MOH, and site visits. In all the regions except Manzini, a Sexual and Reproductive Health (SRH) Mentor represented the MOH and assisted the team with directions to facilities and introductions to the senior nurse on duty at each health facility during the field visits. All the Manzini regional staff were in a workshop and unable to participate. The evaluation collected HMIS data sets from the MOH at national level and from EGPAF country office. The evaluation data analyst re-aggregated and analyzed the HMIS data sets; evaluated EGPAF project management and implementation monitoring data from the EGPAF reports to USAID/PEPFAR, and assessed EPAS activities to strengthen the PMTCT/MNCH information collection and use by the MOH. TRIANGULATION To increase the credibility and validity of the findings, the evaluation team triangulated their findings, comparing data from different sources (data triangulation), using different methods (methodological triangulation) and between the evaluators (evaluator triangulation). SAMPLING The MOH advised which national level MOH personnel were to be interviewed and also advised which UN and PEPFAR partner organizations should be interviewed. The evaluation used a stratified, purposive sampling to select sites to be visited during field data collection. The sample included the regional health management team, a hospital Maternity Unit and Public Health Unit (PHU), and three lower facilities in all four regions (Hhohho, Shiselweni, Manzini, and Lubombo). As EPAS supports one private sector hospital and one mission hospital in different regions, the evaluation sample was stratified to include both the private sector and the mission hospital, and two public hospitals one each from the remaining regions. EPAS grades health centers and clinics as high performing—ones that are graduated and receive supervisory/mentoring visits quarterly; medium performing—receive supervisory/ mentoring visits monthly; and low performing—these receive supervisory/mentoring visits monthly and additional support to improve performance. The sample of health centers and clinics ensured representation of private sector, mission and public facilities, and high, medium, and low performing facilities. In regions where there was more than one facility in the category, the sample was taken blind of any further details such as facilities’ geographic situation. The evaluation included sites 30 or more kilometers along graded roads and did not substitute hard to reach facilities with ones closer to the blacktop roads. This purposive sampling captured qualitative information on sites where EPAS has been particularly successful or unsuccessful, to inform lessons learned. Table 1, page 9, summarizes the sample. At each health facility, the team interviewed the most senior nurse available and a person with HMIS responsibilities. In facilities with a Maternity Unit, the team also interviewed the senior person on duty in the labor and delivery ward. Group discussions were entirely opportunistic— where there were service users available and willing to come into a group discussion. One group discussion was with rural health motivators who had come into a health facility for a monthly 9 meeting. As the local logistics consultant became more skilled and confident, several individual health facility users—including two men outside one health center—were approached and agreed to talk to the evaluator with the logistics consultant interpreting. Table 1: Summary of the Sample for the Field Evaluation Region Type of Facility Ownership Performance Sample Hhohho Hospital Maternity Unit & PHU Government PHU: High 1/2 (2 Government) Lubombo Hospital Maternity Unit & PHU Mission PHU: High 1/1 (1 Mission) Manzini Hospital Maternity Unit & PHU Private PHU: Medium 1/2 (1 Government) (1 Private) Shiselweni Hospital Maternity Unit & PHU Government PHU: High 1/1 (1 Government) Hhohho Health Center NGO High 1/2 (1 Government) (1 NGO) Lubombo Health Center Mission Medium 1/1 (1 Mission) Manzini [no health centers in Manzini] 0/0 Shiselweni Health Center Government High 1/1 (1 Government) Hhohho Clinic Government Low 2/32 (23 Government) (9 Mission) Hhohho Clinic Government Low Lubombo Clinic Mission Low 2/34 (23 Government) (7 Mission) (1 Private) (3 NGO) Lubombo Clinic Government Low Manzini Clinic Government Low 3/35 (17 Government) (6 Mission) (1 Private) (1 NGO) Manzini Clinic Mission Low Manzini Clinic Private High Shiselweni Clinic Mission Medium 2/25 (17 Government) (6 Mission) (1 Private) (1 NGO) Shiselweni Clinic NGO Low Added in the field [Directed there by Regional Administrator] Manzini Clinic/PHU Government High LIMITATIONS OF THE EVALUATION AND ITS FINDINGS The evaluation was a programmatic performance evaluation that includes analysis of primary qualitative data collected by the evaluation team. It provided re-aggregation and analysis of the MOH HMIS quantitative data, to identify results achieved from 2010-2013, the first three years of the project implementation. As the primary data collected was a cross-sectional assessment of the implementation of EPAS, the evaluators were not able to ascertain long-term achievements from the primary data. However, 10 the evaluation re-aggregated and analyzed MOH HMIS data to assess the progress so far and improvements in performance to date. The quality of the HMIS data was assessed by comparing the data for three indicators in the 2013 registers at health facilities with the HMIS reports for 2013, to determine reporting reliability. The evaluation data analyst was not able to conduct formal data quality audits (DQAs), fill in gaps in information, or correct errors. He reviewed the USAID DQA reports for September 2011 and July 2012. With a small evaluation team and limited time for field data collection, the evaluation was conducted with only a small sample of project sites, activities and informants. Although the team made every effort to make the sample representative by stratifying, it is not large enough to be analyzed statistically, and may not be generalizable. For practical reasons, the sample was purposefully stratified across 4 levels, by region, by type of facility, by facility performance including graduation status and to ensure the evaluators saw the best of EPAS and its most challenged. Inevitably, the sample in each stratification is small. Triangulation and verification reduced biases and errors, but the generalizability may be limited. Specifically, evaluating the high performing and low performing sites was needed for lessons learned. Generalizability may have been traded off against obtaining evidence to support specific, strategic recommendations for maximizing the results in the last year of implementation, and for possible follow on activities. The sample provided valuable qualitative understanding on what is going well and less well. Analysis of the primary data collected supports the evaluation recommendations for the last year of EPAS implementation, and any follow on activity. The key informant interviews depended on the informants’ availability at the time of the field visit. In Manzini region it was not possible to meet with the Regional Health Management Team (RHMT) as they were in a residential workshop. The interviews were conducted in English and the evaluators took handwritten notes during the interviews; the evaluators relied on local persons to translate guided group discussions where the participants used SiSwati. The local logistics consultant was an impartial translator and quickly learned how to conduct guided group discussions and one on one discussions with some service users, translating the responses into English, while the evaluators wrote handwritten notes. The evaluators discussed their fieldwork findings daily, typed up transcriptions, and analyzed the contents for common and contrasting qualitative findings. The size of USAID/PEPFAR investment and the scale of EPAS support at regional health team and facility level enabled the evaluation to assess the contribution made by EPAS project to the supply side, at regional and health facility levels. The inception phase of the evaluation coincided with the absence on leave of key USAID/Southern Africa and EPGAF staff. This delayed delivery of the package of desk review documents to the evaluation team. The evaluation tools—drafted in the evaluators’ home offices from experience of PEPFAR PMTCT programs elsewhere—were amended and finalized in the light of briefings by the Swaziland PEPFAR team and the EGPAF senior management team. The evaluators found it necessary to further amend the checklist for site visits while collecting data from Maternity Units after the first site visit to the Maternity Unit at Mbabane Hospital. 11 FINDINGS, CONCLUSIONS & RECOMMENDATIONS DISCUSSION OF THE FINDINGS Introduction: The evaluation findings gathered through all methods are presented by evaluation question. The evaluation distributed 26 email surveys to EGPAF and its subpartner staff and received 21 responses, and conducted key informant interviews with 61 persons including national level MOH and PEPFAR partner staff, RHMTs and health facility staff. The evaluators visited 17 health facilities where they conducted key informant interviews with the most senior nurse available and a person responsible for the HMIS, as well as observing the clinical services with the facility checklist. The evaluators talked with service users where possible at the health facilities, and community members individually or in small groups nearby health facilities and during a community event held by EPAS partners. The evaluation team was briefed in detail by USAID/Southern Africa and the Swaziland PEPFAR Team, and EGPAF Senior Management Team. Terminology: The evaluators very quickly found that the terms “Eliminating Pediatric AIDS in Swaziland project” and “EPAS” were not understood outside the EPAS partners and other PEPFAR implementing partners. The MOH from national to facility level understood the project as “the EGPAF project” or simply “EGPAF”. A further difficulty for MOH respondents was that many EGPAF staff worked on EGPAF’s previous direct service delivery project. This made it difficult for MOH respondents to identifying what the EGPAF staff implemented prior to EPAS and what they implemented and achieved through EPAS. To reduce confusion, the evaluators reworded questions using EGPAF instead of EPAS. When the evaluators asked about EPAS partner activities, they referred to the partner organization by name. The evaluators clarified that the period of interest was from 2010 and specifically asked about which year events had happened that were reported by facility staff to check that they were referring to the period from October 2010. Organization of the Swaziland PMTCT Program at national level: Key informant interviews and GOKS documents and websites identified that the Sexual and Reproductive Health Unit (SRHU) of the MOH manages the PMTCT program and promotes integration into maternal and child health services. A Technical Working Group (TWG) brings the PMTCT and the antiretroviral therapy (ART) programs together. The ART Coordinator sits in the Swaziland National AIDS Programme (SNAP) and is also responsible for pediatric HlV/AIDS. There is a designated PMTCT Coordinator in the SRHU and a Technical Advisor who facilitates linkages between the technical staff in the SRHU and SNAP. The latter position is supported through the EPAS project. However, at the time of this evaluation the PMTCT Coordinator had gone on study leave and a new officer had been assigned but was not very conversant with the project activities. The shared responsibility of the SRHU and SNAP for the PMTCT program leads to complications in determining oversight at regional and national level. 12 QUESTION 1. THE EXTENT TO WHICH THE PROJECT'S FOUR MAIN OBJECTIVES HAVE BEEN ACHIEVED; THE SUCCESSES, FAILURES AND GAPS IN EPGAF'S APPROACH A) EXPANDED PMTCT SERVICES: Baseline Availability of PMTCT Services Prior to the EPAS Project During the 2 years prior to the Service Availability Mapping (SAM) 2006-2007, more than 50% of health facility staff had been trained in HIV counseling and testing as well as in PMTCT. At that time there were 154 health facilities in Swaziland. [Please see Table 2, below.] The SAM found that more than half the facilities in all four regions had staff trained in the “key service” PMTCT.17 The MOH website states “Progress has been made in improving geographical coverage of PMTCT of HIV from 3 sites in 2003, to 16 sites in 2004, 54 sites in 2005, 88 sites in 2006 to 137 sites out of 162 health facilities in 2009.”18 This is shown in Figure 2 on page 13 below. Availability of PMTCT Services During EPAS Project Implementation In 2010, during EPAS first year, it was supporting PMTCT services in 59 health facilities (of 150 health facilities then providing PMTCT services in the country19). EPAS surpassed its target of supporting 118 facilities in 2012, and by 2014 was supporting 144 facilities. EGPAF states that it is currently supporting all the public health facilities providing PMTCT services but the evaluation was unable to independently confirm this because there have been changes in the definition of facility ownership in the SAM. The evaluation team reviewed the EGPAF data for public health facilities that demonstrate EPAS is providing TA to all the public facilities providing PMTCT services. The SAM 2013 mapped 287 facilities in the country in 2013 and of these 162 (64.3%) provided PMTCT.20 Figure 3 on page 13, below, shows the number of health facilities in Swaziland, and the number providing PMTCT services taken from the 2013 SAM21, with the numbers of EPAS supported facilities in 2010 and 2014 taken from EPAS documents. EPAS is currently providing 17 WHO (2008) Service Availability Mapping (SAM) 2006-2007. MOH&SW and MOE, Mbabane, Swaziland 18 http://www.gov.sz/index.php?option=com_content&view=article&id=473&Itemid=286 [accessed 07/02/2014] 19 World Bank (2014) Service Availability Mapping (SAM) 2013. Final April 2014, MOH, Mbabane, Swaziland 20 World Bank (2014) Service Availability Mapping (SAM) 2013. Final April 2014, MOH, Mbabane, Swaziland 21 The number of sites providing PMTCT for period 2006-07 in the SAM 2013 is less than the number given by the MOH for 2006 [see figure 2]. This is probably because the 2006-7 SAM omitted some facilities. Table 2: in 2006-2007 13 PMTCT TA to half of the 287 health facilities in Swaziland. Re-analysis of the HMIS data for 2010 and 2013 indicates that uptake of PMTCT services increased during the period. Examples of increased uptake of services include: HIV testing on first ANC visit increased by 33.5% to 20,757 in 2013; the uptake of intrapartum dose of ARV by positive pregnant women and of exposed infants receiving ARVs also increased but not as much [12% and 15% respectively]. The HMIS data for dried blood spot (DBS) polymerase chain reaction (PCR) testing at 6-8 weeks are not directly comparable between 2010 and 2013 as the 2010 data includes all DBS PCR between 6 weeks and 11 months. Nonetheless there was a nearly 20% increase in 2013— indicating that an even larger increase in uptake of DBS PCR at 6-8 weeks occurred during the implementation of EPAS. [Please see Figure 4, page 14 below.] 3 16 54 88 137 0 20 40 60 80 100 120 140 160 '2003 2004 2005 2006 2009' Number of facilities providing PMTCT Year Figure 2: Expansion of coverage of PMTCT services prior to EPAS activity Figure 3: Numbers of health facilities, of health facilities providing PMTCT, and of health facilities supported by EPAS by year Health facilities Providing PMTCT EPAS TA 2006‐07 2010   2013           2014 14 The HMIS data indicate significant increases from 2010 to 2013 in the numbers of HIV positive pregnant women commenced on prophylactic cotrimoxazole (CTX) and in the proportion of HIV positive pregnant women commenced on CTX. The denominator is Total number of HIV+ pregnant women (known positive, plus newly identified HIV+). The proportion of clients receiving a service indicates the performance of the facilities providing PMTCT services in not missing opportunities to provide services. As shown in Figure 5, on page 15 below, the proportions of HIV positive pregnant women initiated on AZT and on ART also increased from 2010 to 2013, but the HMIS denominators changed. In 2010 the denominator for HIV positive women initiated on AZT and HIV positive women initiated on ART was the total number of HIV positive women excluding those who were already on ART. In 2013, the denominator for women commenced on ART was women eligible for ART with CD4 <350 /WHO Stage III-IV. Between 2010 and 2013 the number of women already on ART [and thus excluded from the denominators] jumped from 1107 in 2010 to 3090 in 2013. Following the Revised PMTCT Guidelines, pregnant women who are newly identified as being HIV positive and others who are known to be positive but not on ART are started on AZT and the eligible positive women fast tracked onto ART. Further, forty-six percent more exposed infants were identified in 2013—a total of 9025—and ninety-two percent of the exposed infants received NVP. In summary, the push to expand the number of health facilities in Swaziland providing PMTCT services preceded the EPAS project. The HMIS data indicate that number of health facilities providing PMTCT services in 2013 was 8% more than in 2010. However, EPAS has successfully scaled up its TA to facilities providing PMTCT services from an initial 59 facilities in 2010 to 144 of the 162 facilities in Swaziland providing PMTCT in 2014. Importantly, EPAS support has occurred over the period when the uptake of PMTCT services has increased—by as much as 33.5% for pregnant women HIV tested and receiving their results—and health facilities performed better in Figure 4: Uptake of PMTCT services in 2010 and 2013 15,552 5,368 7,276 7,134 20,757 6,025 8,303 8,462 Number of pregnant women tested for the first time and received results (tested at D & E exclude re‐test) Number of HIV positive pregnant women provided with Intra‐ Partum Dose (IPD) this month HIV exposed infants initiated on AZT/NVP Number of exposed infants DBS taken for DNA/PCR test at 6‐8 weeks visit 2010 2013 6wks-11 months 15 terms of the proportions of eligible clients receiving services. Only 31% of HIV positive pregnant women were initiated on CTX in 2010 but this increased to 97% in 2013 with EPAS support. B) COMPREHENSIVE AND QUALITY PMTCT SERVICES INTEGRATED INTO ANC, LABOR & DELIVERY AND POST-NATAL CLINICS, INCLUDING ANC & PNC VISITS, HTC AND ARV UPTAKE, ADHERENCE, FACILITY DELIVERIES, HIV TRANSMISSION RATES AT 6-8 WEEKS, ETC. At the time of the evaluation, a comprehensive package of PMTCT services had been integrated into the MNCH platform at health facilities throughout Swaziland. The evaluators observed that HIV testing and counseling (HTC) of pregnant women on their first ANC attendance was routine— women could opt out but HTC was routinely provided at all the facilities visited. At health facilities with Mentor Mothers (MMs), pregnant women attending ANC were counseled by a MM on each attendance. The interviewed nurses in ANC and MMs reported that MMs counsel HIV negative pregnant women to attend ANC regularly, advise them on HIV prevention [PMTCT Prong 1] and encouraged them to retest after 8 weeks and in the last trimester of pregnancy. Interviewed nurses and MMs, as well as m2m personnel, reported that MMs also promote exclusive breastfeeding for the first 6 months. Condoms were seen to be freely available for ANC clients to help themselves at facilities visited. Pregnant women living with HIV are commenced on ARV prophylaxis from as early as 14 weeks gestation or their first ANC visit if that was after 14 weeks. Pregnant women living with HIV also receive intrapartum ARVs to take home at their first ANC visit if that is at 14 weeks or later. Health facility staff, MMs and m2m reported that MMs encouraged pregnant women living with HIV to attend ANC regularly, and adhere to ARV prophylaxis and CTX prophylaxis, and to deliver in a health facility (Maternity Unit). The pregnant woman’s antenatal Figure 5: Proportions of PMTCT clients receiving PMTCT services in 2010 and 2013 98% 31% 67% 13% 98% 97% 95% 83% 74% 92% Proportion of ANC women HIV tested & received their result Proportion of positive pregnant women initiated on CTX Proportion of positive pregnant women initiated on AZT Proportion of positive pregnant women initiated on ART Proportion of exposed infants given NVP 2010 2013 (15552/15874) (20757/21075) (2620/8452) (5690/6855) (4877/7279) (9705/10005) (927/7279) (1512/2043) (5862/6170) (8303/9025) 16 record card has her HIV status and requirements for retesting [if needed] included so that the testing is routinely offered. As there isn’t an indicator that includes mother-baby pairs, the HMIS doesn’t track mother-baby pairs. EPAS has supported the MOH to introduce ART services at PHUs for HIV positive pregnant women, HIV positive mother-baby pairs until the child reaches 2 years old and is negative on antibody testing, or until the HIV infected child reaches 5 years old; the care of the mother and her older child living with HIV is then transferred to the hospital ART clinic. The care of mothers on ART, or ARV prophylaxis during lactation, and their infant/child is tracked by the PHU ART clinic where Expert Clients follow up positive women who miss an appointment. The provision of ART in PHUs facilitates a “family centered approach” and reduces the burden of health care visits on the mother. Other positive mothers who are not on ART are followed in ANC and, where MMs have EPAS cellphones and airtime, MMs will contact mothers who miss an ANC appointment. Exposed infants of mothers who are not on ART/ARV prophylaxis are followed in child welfare clinics after the 6-8 week post natal check. Child health cards now include a line for the mother’s HIV status and this stimulates the health worker to request DBS and HIV antibody tests when the infant returns to the child welfare clinic. If the mother’s status is initially negative, the child welfare card stimulates the health worker to initiate HTC for the mother every 8 weeks, when the infant is seen in the child welfare clinic. Mentor mothers follow up with mothers of exposed infants who do not return for DBS at 6-8 weeks. However there is no other active tracing of exposed infants who are not brought for follow up a child welfare clinic care is mainly reactive to a mother or infant/young child coming to the health facility rather than proactive keeping track of mothers and infants. Higher volume sites visited by the evaluators had a MM who counseled all women returning for post natal care and mothers returning with exposed infants. HIV negative women were said to be routinely offered retesting at the postnatal clinic attendance. The evaluators were assured by nurses in child welfare clinics that nurses routinely offer HTC to women who attend child welfare clinics at the health facilities, now that they are reminded by the mother’s status line on the child health card. Some staff interviewed in Maternity Units assured that evaluators that pregnant women who come in labor to the Maternity Unit and have forgotten their intrapartum ARVs are no longer sent home to get them but receive the ARVs in the labor ward. Midwives reported that “almost all” pregnant women known to be HIV positive have taken their intrapartum ARVs at home before attending the labor ward. One of the Maternity Units visited had a MM although the MM was not on duty at the time of the evaluation visit. One of that MM’s responsibilities was tracking labor ward stocks of ARVs and alerting the senior midwife when stocks needed reordering. Midwives reported that they do HIV rapid testing in the labor ward for pregnant women who have not had a negative test in the last trimester. One Maternity Unit visited had had stockouts of nevirapine (NVP) because they had not reordered in time. This was attributed to midwives on the night shift using NVP but not recording it in the client’s medical records when the night shift staff are very busy. Another Maternity Unit did not give infants NVP; instead they waited for the “PMTCT nurse” to come from the PHU and give infant NVP. This was a problem when there was not a “PMTCT nurse” on duty on Sundays. QUALITY OF PMTCT SERVICES Although one of the EPAS project objectives was “to provide and sustain quality, comprehensive and integrated PMTCT services through supporting clinical mentoring and supportive supervision 17 of health care workers at health facilities across the country” it has not formally defined quality for PMTCT services. EPAS staff report they conduct regular internal quarterly review meetings and regional quarterly review meetings that focus on health facility performance (rather than quality of service delivery per se.) EPAS reports that it supports monthly Multi-Disciplinary Team/Quality Improvement meetings at 12 facilities where PMTCT data are reviewed, weak areas are identified and strategies to address weak areas were discussed. These focus on analysis of the HMIS data. In the EGPAF M&E Plan for EPAS, it was stated that they intended to conduct annual client satisfaction surveys in selected sites22, but these surveys have not been reported on; they were not in annual workplans and the evaluation did not find evidence that the surveys happened. EPAS and health facility staff reported that EPAS PMTCT training for health facility staff included topics that contribute to service delivery quality including clinical skills strengthening and those topics concerned with empathy and being kind to clients—aspects of the training that several nurses highlighted as benefits of EGPAF training over other trainings they have participated in. Training on implementing PMTCT guidelines and training that improves clinical knowledge and skills—such as ART initiation—as well as provision of job aids and small equipment, all contribute to improved service delivery quality. The evaluation assessed quality of PMTCT services from the pillars of structure, process and outcomes as described by Avedis Donabedian, an academic expert in health care quality measurement23, 24, 25. Structural factors related to PMTCT service quality Staffing and reporting: In Swaziland, the PMTCT service was observed to be a nurse-led service delivered principally as primary medical care through Public Health Units (PHUs), health centers and clinics under the oversight and leadership of the RHMTs. The exception was reported to be that Maternity Units in hospitals and some health centers provide intrapartum and immediate postpartum support to PMTCT but hospitals report directly to the Ministry of Health at central level. Physical facilities, equipment and job aids: Very many of the health facilities visited—particularly high volume facilities—were housed in cramped and overcrowded buildings: this was frequently noted by EPAS in its workplans “Space remains a significant challenge for provision of comprehensive PMTCT services in most health facilities.” EPAS project supported modest renovations of some health facilities and has provided small items of equipment—including fetal stethoscopes in supported Maternity Units—to aid delivery of PMTCT services. EPAS has also supported the MOH by printing and distributing copies of the 2010 Revised PMTCT Guidelines and other job aids. 22 EGPAF (2011) EPAS M&E Plan Final Dec 2011 23 Avedis Donabedian (1966) Evaluating the Quality of Medical Care. Reprinted in: The Milbank Quarterly, Vol. 83, No. 4, 2005 (pp. 691–729) 24 Donabedian, A., (1980), The definition of Quality and Approaches to its Assessment, vol. 1: Explorations in Quality Assessment and Monitoring. Ann Arbor, Michigan, Health Administration Press 25 Donabedian, A. et al, (1988), The Quality of Care: How can it be Assessed? JAMA, Vol.260, No.12, p1743-48 18 Although the evaluators did not see the guidelines in all the facilities, facility staff had seen the guidelines and referred to the contents. Many facilities visited by the evaluation had photocopied (and some laminated: see box) A-4 sized job aids including standard operating procedures stuck on the walls in clinics for the staff to refer to. Health facility staff reported that only EGPAF has provided them with job aids. Available job aids varied between the different facilities indicating that the supply was tied to specific needs in each facility. The health workforce: EPAS reports that it has been deeply involved in training—advising on the nursing pre-service training curriculum, and in retraining of health workers. EPAS trainings reported in the semiannual and annual reports and confirmed by health workers, have included:  aspects of PMTCT and orientation to the 2010 Revised National PMTCT Guidelines  basic PMTCT  PMTCT in maternity settings  care & treatment of people living with HIV including integrated management of adult illness, and nurse ART initiation  pediatric and adolescent HIV counseling and psychosocial support  HIV testing & counseling; couple HTC  M&E  MNCH M&E tools  training of trainers EPAS has also conducted ad hoc, on-site trainings/updates on selected PMTCT, M&E, and care & treatment topics such as more efficacious ARV regimens, EID and HMIS quality improvement. RHMT MNCH mentors reported that they held records of which staff have received training—one MNCH mentor was able to assert that a Maternity Unit had PMTCT trained staff but that the named trained staff were not on duty at the time of the evaluation visit, when the evaluation was told by the Maternity Unity staff that no one had had training. To implement the MOH’s adoption of the “Option B+” policy, there is currently a push to train nurses as ART initiators. Regional PMTCT coordinators and registered nurses reported to the evaluators that nurse initiators have phone back up from a doctor in the ART clinic at the referral facility. Some PHUs reported that the doctor from the ART clinic attends the PHU regularly one day a week to provide clinical support and address clinical issues. At one PHU the evaluators met the doctor from the ART clinic doing her regular weekly support visit. The nurse initiators are also able to use Whatsapp26—an application on their cellphones—to send questions and photographs to their back up doctor. The adoption of nurse ART initiators appears to be working appropriately, decentralizing PMTCT services to health facilities closer to service users’ homes. 26 See http://www.whatsapp.com/ for details 19 Health facility staff reported that they greatly appreciate the training they receive from EPAS: there is some difference of opinion about the preference for onsite training versus off-site. Health facility staff assert that they have learned a lot more about PMTCT and have improved PMTCT service delivery skills as a result of the training. A few specifically noted that they had learned the importance of “being kind” to all clients or of “caring about their clients” from their EPAS trainings and were striving to always be considerate in their clinical work. Availability of birthing facilities: Many of the clinics visited reported that although they do not have a Maternity Unit, they are sometimes called on to deliver infants “in an emergency” when a woman arrives in advanced labor and delivers before she can be transferred to a facility with a Maternity Unit. Staff at these clinics reported that they would prefer to be set up to conduct normal deliveries—so that they would have the right equipment for safe deliveries—rather than conduct unplanned for deliveries without the right equipment. They said that pregnant women find it easier to get to their clinics because the clinic is nearer the women’s homes than the facility with the Maternity Unit. Process factors related to PMTCT service quality Organization of service delivery: PMTCT services were observed to be well integrated into antenatal care in all the health facilities visited during the evaluation. Some facilities—generally small clinics with only a few nursing staff—were observed to provide a fully integrated service, although one such facility visited had a phlebotomist who said she was seconded from the EPAS project, who was also providing HTC27. She reported that she collected the blood for DBS/ EID and venous samples for blood tests ordered by the nursing staff. Her register was reviewed by the evaluators as she complained of too large a work load. Larger facilities were observed to provide a “one stop shop” service where all components of PMTCT were available each clinic day, but from different service providers. Even within the one stop shop, the evaluators observed variations. Whereas some were observed to dispense their own ARVs, both for prophylaxis and to treat mothers and HIV infected infants, others were observed to refer positive mothers and infants to the main pharmacy in the facility to collect their refill prescriptions. At one PHU, the evaluators noted that in the very cramped ANC the nurses were dispensing ARVs. However, in the new wing—that housed the PHU ART clinic, mentor mothers, and HTC— ART clients were sent to the main pharmacy to collect their medications. Nursing staff, MMs and women clients on ART in health facilities where clients were sent to the 27 The person concerned reported to the evaluators that she was seconded by EGPAF in March 2013. She may have been seconded as an HTC counselor although she identified herself professionally as a phlebotomist. She said her post would go when EGPAF leaves: but her role in HTC would still be needed. ART nurse who dispenses ARVs in a PHU 20 main pharmacy, reported to the evaluators that women on ART do not like going to the main pharmacy to collect medications. This was said to be in part because that incurred another wait in line, but also because those in line behind them would know that they were on ART. Pharmacies did not observe practices that would provide privacy and confidentiality for each client, and others in line crowd round the person receiving services. This evaluation finding mirrors the finding in the EGPAF 2012 study on barriers to uptake of ART by eligible pregnant women.28 Challenges obtaining CD4 counts: The most common challenge reported to the evaluators to providing PMTCT services is obtaining CD4 counts. Smaller health facilities do not have equipment to conduct CD4 counts; in some higher volume facilities CD4 machines are no longer working. Many health facilities depend on samples being taken to referral facilities or to Mbabane. Referral facilities limited use of their CD4 machines to specific sessions for each clinic. Clients had to return to the clinic on specific days for blood draws if they attended for clinical follow up on a day when blood could not be taken for CD4 or if they attended on a day after the collection of samples for the referral laboratory. This situation was exacerbated when the transportation for the blood samples was irregular—sometimes coming before the appointed hour before women had had their blood taken. The 2012 EGPAF study on barriers to uptake of ART by eligible pregnant women also identified CD4 testing as an issue. “Health care workers (HCWs) also expressed frustration at the length of time it takes from sample collection to receiving results. HCWs believed that a lot of the challenges with receiving the CD4 test results were structural challenges such as fuel shortages, problems with transport pickup and drop, and stock-outs of reagents. Problems with CD4 result delivery were the most common facility-level challenge discussed by HCWs.”29 Availability of DBS: The 2013 SAM reported that the availability of DBS for early infant diagnosis (EID) testing has decreased since 2010. “In 2010 DBS for EID was available in 88% facilities and in the current SAM this proportion had declined remarkably to 66%. This is particularly worrying given the high HIV related infant mortality.”30 The SAM sheds no light on why or at which facilities there has been a decrease in point of care DBS; a total of 139 facilities were reported as collecting DBS at the point of care in the 2012 SAM.31 DBS PCR was collected at all the facilities visited by the evaluators and was reported to be working well. Nursing staff reported that infants that are brought back to the facility at 6-8 weeks routinely have DBS for EID and the results are available when the infant returns 4 weeks later. Some facilities reported that when a DBS is positive, the laboratory staff call the facility, so that the facility can in turn call the mother to bring the infant back sooner than the next appointment. No health facility staff member was able to tell the evaluators what proportion of exposed infants have DBS at 6-8 weeks and how that has changed over the last 4 years. Some simply reported that “they all do” others said “most do”. 28 EGPAF (2012) Swaziland Barriers to Antiretroviral Therapy Initiation for Eligible HIV-Positive Pregnant Women in Antenatal Care. EGPAF, Washington, DC 29 EGPAF (2012) Swaziland Barriers to Antiretroviral Therapy Initiation for Eligible HIV-Positive Pregnant Women in Antenatal Care. EGPAF, Washington, DC 30 World Bank (2014) Service Availability Mapping Report 2013 (Final). MOH, Mbabane, Swaziland. See Page vi 31 World Bank (2014) Service Availability Mapping Report 2013 (Final). MOH, Mbabane, Swaziland. See Page 186 21 Delivery of post natal PMTCT services: Postnatally, the facilities visited were observed to be providing integrated services. Health facility nursing staff delivering postnatal and child welfare services reported that they had been trained in PMTCT. The high volume facilities visited had MMs in the child welfare clinics who both m2m and the MMs interviewed reported they routinely encourage exclusive breastfeeding until the infant is 6 months old as well as use of condoms and family planning methods. The child health cards were observed to have a line item where the mother’s HIV status is written, and retesting every two months is reported by the nursing staff to be a routine part of the package of services provided. Nurses reported family planning to be “provider initiated” in many clinics, with 2 month injectables and oral progesterone only pills reported as being the most popular methods with clients. Some clients reported that they were not taking “family planning” but then said later in the discussion that they were on the “2 month injection”. Condoms were observed to be freely available: MMs give out condoms and mothers can pick up supplies from open boxes in the clinics and the clients’ toilets. Nurses reported that taking blood for DBS/EID is routine at 6-8 weeks. Weakness in postnatal PMTCT: The major weakness in postnatal PMTCT services at the facilities visited was not on the supply side. The major weakness was the drop off in mothers and infants returning for follow up after the 6-8 week postnatal visits. This is demonstrated in Figure 6 below. A high proportion of young children seen at health facilities aged 12 to 18 months (81%) and aged 18 to 24 months (88%) are tested for HIV infection indicating that child welfare clinic staff are providing PMTCT/EID services. However the actual numbers of young children seen at age 12-18 months (981) and age 18 to 24 months (661) are very small, reflecting the drop off in uptake of services after the 6-8 week post natal check. Some clinics offer outreach services including immunizations but nursing staff reported that these outreaches do not provide PMTCT. Figure 6: Proportion of Infants & young children tested at different ages, and proportion testing positive in 2013 97% 81% 11% 88% 12% Proportion of exposed infants DBS taken for DNA PCR at 6‐8WKS Proportion of exposed infants tested HIV+ at 6‐ 8wks Proportion of children tested at 12‐ 18 months Proportion of children tested HIV+ at 12‐18 months Proportion of exposed infants who return for the first time at 18‐24 months and are tested Proportion of exposed infants who return for the first time at 18‐24 months that are positive 8,462/8,757 241/8,462 795/981 84/795 582/661 67/582 2.82% 22 Weakness in delivery of PMTCT: the main weakness in the supply side—delivery of PMTCT services—was found to be in the Maternity Units. The MOH staff interviewed reported that Maternity Units in hospitals are staffed separately from the PHUs, health centers and clinics providing ANC and post-delivery care. Hospitals and their Maternity Units report to the MOH centrally not to the RHMT. EPAS reported that it is focusing on 11 high volume Maternity Units to provide TA. EPAS reports, confirmed by health facility staff, it has provided small equipment such as fetal stethoscopes. Although staff at one Maternity Unit reported EPAS had provided a lot more equipment, all the equipment seen was labeled “UNFPA”. In addition to training in PMTCT, EPAS and nursing staff report it has also conducted training and mentoring in use of the partogram in labor. The evaluation visited four Maternity Units in referral hospitals and one in a health center. At the time of the visits, deliveries were being conducted in all five units. The evaluators observed a general lack of privacy and respect for the dignity of the women laboring and delivering. In two of the units, the evaluators observed women laboring completely naked, in open wards with nurses and students walking in and out. Women in both of these units were observed to be walked— naked—from the labor ward into the delivery room. Only one of the maternity units visited had screened delivery beds. That unit had fitted wooden room dividers in the delivery room and had curtains at the windows and to the open end of the delivery cubicles.32 A delivery during the evaluation visit was conducted with the curtains closed [the midwife was unaware that there was an evaluator in the unit.] All four Maternity Units in Hospitals were using the partogram. At one of the Maternity Units where women were laboring naked and being walked from labor ward through a corridor to delivery room naked, a Muslim woman in hijab and full length outer clothes was laboring in the garden33, accompanied by two of her sisters. When the midwives were asked about how she would be treated in the delivery room, the evaluator was told that the midwives would have to do what the woman wanted and they would comply with that “for the sake of the infant”. Although some of the Maternity Units have MMs attached, no MMs were on duty at the time of the evaluation visits to the Maternity Units. One Maternity Unit admitted to running out of infant NVP within the last 6 months—initially attributed to the MM not reordering. When this responsibility of the MM was questioned by the evaluator, the nursing staff decided that it was a nursing responsibility to reorder infant NVP but that at night it is used and not recorded in the client records because the night staff are over worked. In one Maternity Unit—one with far higher staff to delivery ratios than all the other units—the Maternity Unit staff reported that they do not give infant NVP or ARVs to the mother. The Maternity Unit staff reported that mothers have their own intrapartum ARVs dispensed to them to take home from ANC and the “PMTCT Nurse” comes from the PHU to give the infant NVP—except on Sundays when there isn’t a “PMTCT nurse” in the PHU. The nurses on duty in this Maternity Unit reported that they had not been trained by EPAS although at least one who was not on duty had received training in April 2014—reported to the evaluators by the RHMT MNCH mentor accompanying the evaluators. The nurses on duty had 32 Nursing staff in that unit and one other unit said that the old curtains around delivery beds didn’t work as laboring women sometimes reached out “while thrashing around” and pulled them down. 33 Although ambulatory labor is considered good practice in the west, it was not generally an option for women in the maternity units visited. 23 many complaints to the evaluators about workloads [they had 6-7 midwives on each shift and conducted 250 deliveries a month. This compared with two other maternity units the evaluation had visited that had half the staffing levels and fewer at night, and conducted 300 deliveries a month]. The evaluators had been warned by the RHMT MNCH mentor before going to the hospital to “not expect a warm welcome” as the attitude of the Midwifery Unit staff was well known to the RHMT. EPAS support facilitating service provision: EPAS initially posted a vehicle in each region 2 days a week, but this was later increased to 5 days a week, providing much valued transport to the RHMTs. EPAS vehicles are used to take both the EPAS staff and RHMT MNCH mentors and supervisors to health facilities for supervision/mentorship visits and to assist the RHMTs with redistribution of drugs and commodities between facilities. By moving short shelf life stock from low volume facilities and providing high volume facilities with stock, EPAS support both avoided stockouts and wastage of drugs and commodities through date expiry. Most of the facilities visited had not experienced stockouts in the preceding 6 months although infant NVP was not always available at facilities in both 25ml and 240ml volumes. Staff at one Maternity Unit said that maintaining stocks of infant NVP was a challenge but attributed this to staff not reordering stock as supplies were used, rather than a supply-side issue. Additionally, EPAS provides cellphones and funding for airtime for staff to access technical support and also for following up persons who miss their scheduled follow up appointment. mother2mother support to PMTCT service delivery: Most of the high volume facilities visited by the evaluation team—including all of the PHUs—have MMs attached to the health care teams. This cadre of peer support worker was introduced by m2m, an African regional non-governmental organization, initially a subpartner within EPAS that has graduated and become a USAID direct grantee since 2012. Although MMs are not a grade within the civil service commission, health facility staff and service users reported appreciation of the MMs to the evaluation.34 Registered nurses in the facilities reported that they are able to shift lower level administrative tasks including follow up of clients who do not attend for scheduled appointments to MMs, relieving some of the work burden on the nurses and enabling the nurses to focus more of their time on patient care. This confirmed what m2m had reported to the evaluation. Nursing staff and MMs reported to the evaluation that MMs see all clients attending ANC and child welfare clinics, and some are assigned to maternity units and see clients there, too. Women who test positive for HIV infection are referred back to the MMs for posttest peer support and encouragement to accept PMTCT services. Nursing staff and MMs reported that MMs see positive clients at each clinic visit to encourage adherence to PMTCT ARVs and promote exclusive breast feeding for the infants first 6 months of life. Mentor mothers see HIV negative clients to counsel them prior to provider initiated retesting in the third trimester of pregnancy, and throughout lactation. Some health facility respondents reported that they also have one community-based 34 This finding stands in contrast to the findings of the EGPAF 2012 study of barriers to uptake of ART by eligible pregnant women that documented complaints from health care workers and clients about volunteers including MMs: EGPAF (2012) Swaziland Barriers to Antiretroviral Therapy Initiation for Eligible HIV-Positive Pregnant Women in Antenatal Care. EGPAF, Washington, DC 24 MM who encourages mothers to return with their infants for follow up, supports exclusive breast￾feeding and helps with contracting mothers who miss appointments35. Clients, MMs and many health facility staff reported that they greatly value the quality of counseling and support that MMs provide as they are speaking from their own experience and are caring and empathetic. Some regional PMTCT coordinators did not to recognize that speaking from experience uniquely strengthens the support MMs provide to clients. They only reported the MMs’ value in task shifting, and stated that if facilities were fully staffed they would not need MMs. Outcome measures of PMTCT service quality The only outcome data collected by the routine HMIS is for positive DBS PCR and positive antibody testing in older infants. The rate of positive DBS PCR at 6-8 weeks was 2.85% in 2013 [please see Figure 7 on page 25] which is an excellent result. It is not possible to compare with 2010 HMIS data in 2010 the indicator was for DBS from 6 weeks to 11 months not 6-8 weeks. Further, the 2011 Impact Assessment reported the rate of positive DBS PCR was 12% in 2010 down from 24% in 2007, but states only that the DBS PCR diagnosis was “as early as 6 weeks” without giving an upper age cut off for the result.36 A 2011 study of the effectiveness of the National PMTCT Programme in Swaziland took the infant HIV status at 6-8 weeks postpartum as the indicator. It took a sample of 52 health facilities across Swaziland and a sample of 3,592 mother baby pairs attending during the data collection period of about four months. Very sick infants were excluded from the study as were infants brought to the welfare clinic by a carer other than the mother. Challenges included that more than 600 of the 3,592 samples did not come back from the laboratory testing. The study found the national weighted MTCT rate measured at 6-8 weeks of infant age was 2.2 percent (95% CI: 1.5 - 3.1).37 The research study results are not directly comparable with this evaluation that used all the DBS samples from all the facilities receiving TA from EPAS in 2013 [which included all the public health 35 The evaluators later learned that there are not yet any community-based MMs. It is possible that the health facility staff were confusing other volunteers in the community. However, one nurse was asked if it was a Mentor Mother or an Expert Client in the community and she again said Mentor Mother. 36 MOH (2011) Impact Assessment for ART, TB Treatment and PMTCT Using Triangulation Approach Final Report December 2011. GOKS, Mbabane, Swaziland 37 MOH (2012) Evaluation of the Effectiveness of the National Prevention of Mother To Child Transmission of HIV (PMTCT) Programme at 6-8 weeks Postpartum in Swaziland. GOKS, Mbabane, Swaziland Mentor mothers & support for exclusive breastfeeding In 2006 at the end of the Linkages activity in Swaziland, many health facility staff were conflicted about promoting exclusive breast￾feeding—thinking it was better for HIV infected mothers to replacement feed their infants.** This evaluation found no evidence of such conflict— promotion of exclusive breastfeeding by all mothers was routine. However, many HIV positive mothers have deep fears about breastfeeding. Mentor mothers reported that they know that mothers have fears because they themselves had deep fears and were scared when they were breast-feeding. However, by exclusively breast-feeding and having infants that are not infected and are healthy has encouraged them. Mentor mothers reported that their own healthy children provide powerful evidence for dissipating the fears of the mothers they are supporting to exclusively breastfeed their infants. ** Linkages (2006) Linkages/Swaziland Final Report 2003-06. AED & USAID, Washington DC 25 facilities providing PMTCT services]. Comparing the data for DBS PCR at 6-8 weeks for health facilities supported by EPAS and those not supported by EPAS in 2013, the number of exposed infants seen at 6-8 weeks in EPAS sites was considerably more than in non EPAS sites, by a factor of x 10. Eighty-three percent of exposed infants had DBS taken for PCR/EID at non-supported sites whereas ninety-seven percent had DBS taken at EPAS supported sites. At non-supported sites the rate of expose infants testing positive on PCR at 6-8 weeks was 3.31% [for a relatively small sample and very small number of positive infants] whereas the rate for exposed infants testing positive on PCR at 6-8 weeks at EPAS supported sites was 2.85%. The routine HMIS collects data on infants antibody tested at 12-18 months and 18-24 months, however the data collected doesn’t differentiate between retesting at 12-18 and first testing. It is likely that infants brought back for the first time at 12 month or older are brought back because the infant is ill. This might bias the proportion of HIV positive results at older ages. The relatively small number of infants tested at 12-18 months (981) and at 18-24 months (661) is indicative of the drop off in utilization of PMTCT services after the 6-8 week post natal checkup. [Please see Figure 6 on page 21]. As so few exposed young children in the older age groups are tested for HIV, the routine HMIS data cannot be used to determine national MTCT rates after 6-8 weeks. Quality from managerial, medical professional and service user perspectives In addition to considering structural , process and outcome aspects of the quality of PMTCT 82.57% 60.00% 3.31% 96.63% 69.65% 2.85% Proportion of exposed infants DBS taken for DNA/PCR test at 6‐8 weeks visit Proportion of exposed infants Tested with DNA/PCR at 6‐8 weeks visit and Received Results Proportion of exposed infants Tested DNA/PCR + at 6‐8 weeks visit HF not Supported by EPAS EPAS Supported HF (725/878) (435/725) (8462/8757) (5894/8462) (24/725) (241/8462) Figure 7: Dried Blood Spot for PCR in Facilities Not Supported by EPAS and Facilities Supported by EPAS in 2013 26 services, the evaluation considered the quality from manager, medical professional, and service user perspectives—also determined by Avedis Donabedian to be important aspects of health care service delivery quality. All three perspectives are considered equally important in determining the quality of health care. Managerial perspectives on PMTCT quality The PMTCT services seen by the evaluators were consistently, comprehensively integrated into the MNCH platform. Health facility performance: EGPAF reviews the performance of health facilities by grading the reported services provided against the indicators for different components of PMTCT. The overall performance for a facility is derived by assigning a 5 scale point to all marks scored for the indicators for the sites as follows in Table 3: Table 3: EPAS Health Facility Performance Scale The average point scale between percent HTC and percent ARV uptake scores were initially used as combination proxy for sites’ performance. For example, if Site A scored 75 percent for HTC, and scored 49 percent for ARV Uptake, the equivalent point scales were 3 and 1 respectively. An average scale point was then calculated to come up with the site performance. For Site A, the average of 3 and 1 is 2 scale points, which falls within the performance scale of 2.00 - 2.99. Therefore Site A is a low performance site.38 In Quarter 1 Financial Year 2013, EPAS introduced two more indicators into the calculation of site performance. The new indicators are percent pregnant women with CD4 test done; and percent infants on I-NVP. A gap in the EPAS site performance measurement is that there is no measurement of performance in Maternity Units and only recently was an indicator for post-natal PMTCT adopted. EPAS senior management team reported that EPAS uses the grading of health facilities to determine the frequency of supervision and mentoring visits as it transitions sites from EPAS support back to the RHMTs. High volume sites, Maternity Units, newly supported sites and sites that are performing poorly are visited monthly but other sites are visited less frequently if their performance is acceptable. Facilities in their second year of EPAS support and those that fall from Very High Performance/High Performance to Medium Performance are visited twice a quarter. All other sites with Very High Performance/High Performance are visited quarterly. Although EPAS “graduates” very high performance/high performance to quarterly mentoring visits after two years of EPAS monthly mentoring, in practice this only applies to smaller health centers and clinics as the high volume facilities are mentored monthly regardless of their performance 38 Kudiabor, Kwashi (2011) EPAS Year 1 Site Level Baseline Performance Report. EGPAF, Mbabane, Swaziland No. Mark interval Point Scale Performance Scale Interpretation 1 <50% 1 1 – 1.99 Very Low Performance 2 50% ‐ <70% 2 2 – 2.99 Low Performance 3 70% ‐ <80% 3 3 – 3.99 Medium Performance 4 80% ‐ <90% 4 4 – 4.99 High performance 5 >  90% 5 5 High performance 27 level. The EPAS Site Level Performance, as of Quarter 2, Financial Year 2014, Table 4 between pages 27-26, demonstrates changes in health facility performance over time. Performance can be seen at times to have dropped rapidly at some facilities including “graduate clinics”. EPAS and some health facility staff attributed falls in performance to the PMTCT-trained nurse going on leave at small facilities and to turnover in staff at larger facilities, with incoming staff that have not had PMTCT training. EPAS staff report they respond to fall in health facility performance by increasing mentoring visits and TA to monthly. Currently, 2 health centers and 22 clinics meet the EPAS requirements for graduation. Health facility staff commonly reported that the EPAS mentors “come and point out our errors and encourage us to do better”. Health facility staff were uncomfortable criticizing the mentors but some reported that RHMT MNCH mentors assist with service delivery if the facility is very busy when the MNCH mentor visits the facility, but EPAS mentors do not help with service delivery. A few health facility staff reported to the evaluators that they would prefer/had thought that EPAS mentors would come and see patients with the health facility staff and by this help the facility staff to improve the clinical care of patients. From the health facility staff perspective, the EPAS mentors focus most of their effort on oversight of the HMIS and addressing missed opportunities in service provision that they identify from the monthly reports, rather than on clinical mentoring. Yet none of the facility staff interviewed was able to give the evaluators information about how their performance had improved the PMTCT cascades for their health facility. The interviews with the health facility staff did not elicit that EPAS staff use the PMTCT cascades for performance discussions at health facilities. However, EPAS Senior Management Team are unanimous in their assertion, confirmed by USAID Southern Africa, that EPAS does indeed use the PMTCT cascades for service delivery performance discussions at health facilities, especially at PHUs where the multi￾disciplinary team/quality improvement monthly meetings are conducted. Clearly there is a disconnect between the process that EPAS is using and the experience of the health facility staff. RHMT MNCH mentors and one Regional PMTCT Coordinator reported that initially EPAS mentoring was seen as duplicating RHMT supervision and happened without the RHMT supervisor and MNCH mentor. However that was rapidly changed and RHMT MNCH mentors reported to the evaluation that they now mainly travel with the EPAS mentors and make joint visits. Reanalysis of the HMIS data for 2010 and 2013 shows health facilities performed better in terms of the proportions of eligible clients receiving services in 2013 with fewer missed opportunities for PMTCT service delivery. (For examples of improved performance, please see Figure 5 on page 15.) Efficiency of service delivery: EPAS project inputs such as modest renovations to some health facilities, along with working with the facility staff to improve client flows, provision of guidelines and job aids, as well as small equipment, are likely to have contributed to more efficient delivery of PMTCT services. Similarly, EPAS contributions at national level to the quantification of PMTCT drugs has supported efficiency in supplies without stockouts. Additionally, providing transport to take RHMT supervisors and MNCH mentors to health facilities, and for redistribution of drugs and commodities between health facilities will, at the time, have increased efficiency although this will not be sustained beyond the EPAS project. Coordination of development partner support to PMTCT service delivery: EPAS and other PEPFAR Implementing Partners reported that PMTCT Implementing Partners found coordinating their EPAS Site Level Performance, as of Quarter 2, Financial Year 2014 NO NAME OF SITE REGION Baseline Performance Q2FY11 Performance Q3FY11 Performance Q4 FY11 Performance Q1 FY12 Performance Q2 FY12 Performance Q3 FY12 Performance Q4 FY12 Performance Q1 FY13 Performance Q2 FY13 Performance Q3 FY13 Performance Q4 FY13 Performanc e Q1 FY14 Performanc e Q2 FY14 Performanc e 1 Bhalekane Nazarene Clinic HHOHHO HP MP VHP VHP VHP VHP HP VHP VHP VHP VHP VHP VHP VHP 2 Bhekinkosi Nazarene Clinic MANZINI MP HP HP VHP VHP VHP VHP HP VHP VHP VHP VHP VHP VHP 3 Cana Alliance Clinic MANZINI HP FY12 New site FY12 New site FY12 New site HP HP VHP VHP VHP HP VHP VHP VHP VHP 4 Dvokolwako Health Centre HHOHHO VLP HP HP VHP VHP HP VHP VHP HP VHP VHP VHP VHP 5 Emkhuzweni Health Center HHOHHO MP VHP HP HP VHP HP VHP VHP HP VHP VHP VHP VHP 6 Gebeni Clinic MANZINI LP VHP VHP HP HP VHP HP HP HP VHP VHP VHP VHP 7 Gege Clinic SHISELWENI HP VHP VHP VHP VHP 8 Hlane Clinic LUBOMBO MP VHP HP VHP VHP VHP VHP HP VHP HP HP VHP VHP VHP 9 Hluti Clinic SHISELWENI HP FY12 New site FY12 New site FY12 New site HP HP HP HP VHP HP HP VHP VHP VHP 10 Horo Clinic HHOHHO HP HP VHP VHP VHP 11 Ka‐Mfishane Clinic SHISELWENI LP MP MP MP VHP HP HP HP HP VHP HP VHP VHP 12 Luyengo Clinic MANZINI MP MP MP VHP HP VHP VHP HP VHP VHP VHP VHP HP VHP 13 Manzini Town Council MANZINI VHP FY12 New site FY12 New site FY12 New site VHP HP HP HP VHP HP VHP HP HP VHP 14 Mbabane Phu HHOHHO HP HP HP VHP VHP HP VHP VHP VHP VHP VHP MP HP VHP 15 Mbikwakhe Clinic MANZINI HP VHP VHP HP VHP HP VHP HP VHP HP VHP MP HP VHP 16 Mhlosheni Clinic SHISELWENI HP FY12 New site FY12 New site FY12 New site VHP VHP VHP HP VHP HP VHP MP HP VHP 17 Mliba Nazarene Clinic MANZINI LP VHP HP MP VHP HP MP HP VHP VHP VHP MP VHP 18 Moti Clinic SHISELWENI VLP HP VHP MP VHP VHP HP VHP HP VHP VHP MP VHP 19 Motshane Clinic HHOHHO LP MP HP VHP HP MP HP VHP VHP HP VHP MP VHP 20 Ncabaneni Clinic MANZINI HP HP VHP VHP VHP VHP VHP HP VHP VHP VHP HP MP VHP 21 Ndzingeni Nazarene Clinic HHOHHO HP VHP HP MP VHP 22 New Haven Clinic SHISELWENI MP VHP VHP VHP HP VHP HP HP VHP MP VHP HP MP VHP 23 Ngculwini Nazarene Clinic MANZINI HP MP VHP HP VHP HP MP HP VHP HP HP HP MP VHP 24 Nhlangano Health Center SHISELWENI VHP VHP VHP HP VHP HP HP HP VHP VHP VHP MP MP VHP 25 Nhletjeni Clinic SHISELWENI HP HP HP VHP VHP VHP HP HP HP HP VHP MP MP VHP 26 Nkalashane Clinic LUBOMBO LP MP VHP HP HP HP HP VHP VHP HP HP MP MP VHP 27 Ntshanini Clinic SHISELWENI LP HP VHP VHP VHP VHP VHP VHP VHP VHP VHP LP MP VHP 28 Our Lady of Sorrows Clinic SHISELWENI New Site MP VHP 29 Pigg'S Peak Phu HHOHHO FY12 New site FY12 New site FY12 New site FY12 New site HP HP VHP VHP VHP VHP VHP LP VHP 30 Regina Mundi Clinic HHOHHO VHP FY12 New site FY12 New site FY12 New site VHP VHP HP HP VHP MP VHP VHP LP VHP 31 Sigangeni Clinic HHOHHO HP HP VHP VHP VHP HP HP VHP VHP VHP VHP HP LP VHP 32 Sigcineni Clinic MANZINI VHP VHP VHP VHP HP HP HP MP HP HP HP HP LP VHP 33 Sigombeni Red Cross Clinic HHOHHO MP HP HP VHP MP HP HP HP VHP MP HP HP LP VHP 34 SOS Clinic (Ekutfokomeni clinic) LUBOMBO HP FY12 New site FY12 New site FY12 New site VHP HP VHP VHP VHP VHP HP LP LP VHP 35 Tikhuba Clinic LUBOMBO VHP HP VHP NA VHP 36 Ubombo Sugar Clinic LUBOMBO HP VHP VHP HP VHP VHP HP HP VHP LP VHP LP NA VHP 37 Zombodze Clinic SHISELWENI New Site NA VHP 38 Bhahwini Clinic MANZINI VHP VHP VHP VHP VHP VHP VHP HP VHP VHP VHP VHP VHP HP 39 Bulembu Clinic (Havelock) MANZINI LP MP VHP VHP VHP VHP HP VHP VHP VHP VHP VHP HP 40 Bulunga Nazarene Clinic MANZINI VHP VHP VHP VHP VHP VHP VHP VHP VHP VHP VHP VHP VHP HP 41 Dwaleni Clinic SHISELWENI MP MP HP HP VHP VHP VHP HP VHP HP VHP VHP VHP HP 42 Dwalile Clinic MANZINI VHP FY12 New site FY12 New site FY12 New site VHP VHP HP VHP VHP HP VHP VHP VHP HP 43 Ezulwini Satellite Clinic HHOHHO MP HP VHP HP VHP MP MP HP VHP HP VHP VHP VHP HP 44 Family  Life Association Clinic (Manzini) MANZINI MP VHP VHP VHP VHP VHP VHP VHP HP HP VHP VHP VHP HP 45 Gilgal Clinic LUBOMBO VHP FY12 New site FY12 New site FY12 New site VHP HP HP HP VHP MP VHP VHP VHP HP 46 Good Shepherd Public Health Unit SHISELWENI MP VHP VHP VHP HP HP VHP VHP MP VHP VHP VHP HP 47 Gucuka Clinic LUBOMBO LP VHP VHP VHP VHP HP HP VHP VHP VHP HP VHP VHP HP 48 Hhukwini Clinic HHOHHO VHP VHP VHP VHP VHP VHP HP VHP VHP HP HP VHP VHP HP 49 Hlatikhulu PHU SHISELWENI LP MP HP VHP VHP VHP MP VHP HP HP VHP VHP HP 50 JCI Clinic SHISELWENI MP VHP VHP HP HP HP MP VHP HP VHP HP VHP HP 51 KaZondwako Clinic MANZINI VHP VHP VHP VHP VHP HP HP VHP VHP HP HP HP VHP HP 52 KS II PHU MANZINI VHP FY12 New site FY12 New site FY12 New site VHP HP HP HP HP HP HP HP VHP HP 53 Lavumisa Clinic SHISELWENI LP HP HP VHP HP MP HP VHP HP MP HP VHP HP 54 Lobamba Clinic HHOHHO MP LP VHP VHP HP VHP MP VHP HP HP MP VHP HP 55 Magubheleni Clinic SHISELWENI VHP VHP VHP HP HP 56 Maguga Clinic HHOHHO MP MP VHP VHP VHP HP HP VHP VHP HP VHP VHP HP HP 57 Mahlandle Clinic SHISELWENI VL HP HP VHP HP HP VHP HP HP HP VHP VHP HP HP 58 Malindza Refugee Camp Clinic LUBOMBO MP MP HP VHP VHP VHP VHP HP VHP HP HP VHP HP HP 59 Matsanjeni Health Center SHISELWENI HP HP HP HP VHP VHP VHP VHP VHP HP HP HP HP HP 60 Mkhulamini Clinic MANZINI VHP MP VHP MP VHP HP HP MP VHP VHP VHP VHP MP HP 61 Mpolonjeni Clinic LUBOMBO MP MP MP HP VHP HP VHP HP MP VHP HP VHP MP HP 62 Mshingishingini Nazarene Clinic HHOHHO No data FY12 New site FY12 New site FY12 New site VHP HP HP HP HP HP HP VHP MP HP 63 Musi Clinic MANZINI MP HP VHP HP VHP VHP HP HP MP HP VHP MP HP 64 Nkaba Clinic HHOHHO HP MP MP VHP VHP HP VHP MP VHP VHP HP MP MP HP 65 Nkwene Clinic SHISELWENI LP MP HP VHP VHP HP HP HP HP MP HP MP MP HP 66 Nyonyane Clinic HHOHHO HP HP VHP VHP VHP VHP VHP VHP VHP VHP VHP LP MP HP 67 OSSU MANZINI MP MP HP HP VHP HP HP VHP VHP VHP VHP LP MP HP 68 Pigg's Peak Nazarene Clinic HHOHHO VHP VHP VHP VHP VHP VHP VHP HP VHP VHP VHP VHP LP HP 69 Shewula Nazarene Clinic LUBOMBO HP HP VHP LP HP 70 Sidvokodvo Railway Clinic (High Care Clinic) LUBOMBO VHP LP HP 71 Sigcaweni Nazarene Clinic LUBOMBO LP MP MP HP HP VHP VHP MP HP HP VHP HP LP HP 72 Sinceni Clinic LUBOMBO MP MP HP VHP VHP VHP VHP HP VHP HP VHP MP LP HP 73 Siphocosini Clinic HHOHHO HP VHP MP LP HP 74 Siteki Nazarene Clinic LUBOMBO No data FY12 New site FY12 New site FY12 New site VHP HP VHP HP HP VHP HP MP LP HP 75 Sitsatsaweni Nazerene Clinic LUBOMBO LP VHP VHP HP VHP HP HP VHP HP VHP LP LP HP 76 St. Florence Clinic MANZINI No Data VHP LP HP 77 St. Juliana's Clinic MANZINI HP HP LP HP 78 St. Mary's Clinic HHOHHO VHP FY12 New site FY12 New site FY12 New site VHP HP HP HP HP MP HP HP VLP HP 79 St. Theresa's Clinic MANZINI HP HP LP VLP HP 80 Tfokotani Clinic SHISELWENI MP VHP LP VHP HP HP HP VHP HP VHP VHP VHP NA HP 81 Tsambokhulu Clinic LUBOMBO MP MP VHP VHP HP HP VHP HP HP HP VHP NA HP 82 Wellness Center Clinic MANZINI MP VHP MP MP *VLP LP VHP VHP VHP VHP VHP NA HP 83 Khuphuka Clinic New Site HP 84 Bethany Clinic                                                              MANZINI MP MP HP VHP VHP HP VHP HP VHP VHP VHP VHP VHP MP 85 Bholi Clinic LUBOMBO MP VHP HP VHP VHP VHP VHP HP VHP VHP VHP VHP VHP MP 86 Bulandzeni Clinic HHOHHO LP MP VHP VHP VHP HP HP VHP VHP VHP VHP VHP MP 87 Ekuphileni Clinic HHOHHO HP HP VHP VHP VHP VHP VHP HP VHP HP VHP VHP VHP MP 88 Herefords Clinic HHOHHO HP FY12 New site FY12 New site FY12 New site HP HP HP HP MP VHP HP VHP VHP MP 89 High Care Clinic (Mpaka Railway) LUBOMBO MP MP MP HP VHP HP HP HP VHP HP HP VHP VHP MP 90 Ikwezi Joy Clinic LUBOMBO HP VHP HP VHP HP VHP VHP VHP HP MP HP VHP VHP MP 91 Jericho Clinic SHISELWENI HP VHP VHP VHP HP MP HP HP HP VHP HP VHP MP 92 Ka‐Phunga Gov Clinic SHISELWENI MP VHP HP VHP MP 93 Ka‐Phunga Nazarene Clinic SHISELWENI HP FY12 New site FY12 New site FY12 New site VHP HP VHP HP VHP HP HP HP VHP MP 94 Lamvelase Clinic MANZINI VHP FY12 New site FY12 New site FY12 New site VHP HP VHP HP VHP LP HP HP VHP MP 95 Mafutseni Nazerene Clinic MANZINI VHP VHP VHP HP MP 96 Mahlangatsha Clinic MANZINI MP MP MP HP VHP VHP VHP HP HP HP VHP VHP HP MP 97 Mahwalala Red Cross Clinic HHOHHO HP FY12 New site FY12 New site FY12 New site VHP HP VHP VHP HP HP VHP VHP HP MP 98 Malandzela Nazarene Clinic HHOHHO HP VHP VHP HP MP EPAS Site Level Performance, AS OF Q2 FY14 NO NAME OF SITE REGION Baseline Performance Q2FY11 Performance Q3FY11 Performance Q4 FY11 Performance Q1 FY12 Performance Q2 FY12 Performance Q3 FY12 Performance Q4 FY12 Performance Q1 FY13 Performance Q2 FY13 Performance Q3 FY13 Performance Q4 FY13 Performanc e Q1 FY14 Performanc e Q2 FY14 Performanc e 99 Maloyi Clinic MANZINI MP MP VHP MP VHP HP HP HP VHP HP HP VHP HP MP 100 Mangcongco Clinic MANZINI VLP LP HP HP VHP VHP HP HP HP HP VHP HP MP 101 Mankayane Phu MANZINI VLP VHP VHP VHP VHP VHP MP VHP VHP VHP HP HP MP 102 Manyeveni Nazarene Clinic LUBOMBO MP VHP VHP VHP HP HP VHP VHP VHP VHP HP HP MP 103 Mashobeni Clinic SHISELWENI VHP FY12 New site FY12 New site FY12 New site VHP HP HP MP VHP HP HP HP HP MP 104 Msunduza Salvation Army Clinic HHOHHO VLP HP VHP VHP VHP VHP VHP MP HP HP VHP MP MP 105 Ndvwabangeni Nazarene Clinic HHOHHO MP HP MP VHP HP HP HP HP VHP VHP HP MP MP 106 Ndzevane Clinic LUBOMBO VHP HP HP VHP VHP HP HP HP HP HP VHP HP MP MP 107 New Thulwane Clinic LUBOMBO MP HP HP VHP VHP HP VHP HP MP VHP HP MP MP 108 Ngowane Clinic HHOHHO VHP VHP VHP VHP HP HP HP HP HP HP HP HP MP MP 109 Raleigh Fitkin Hospital MANZINI VLP MP HP VHP VHP HP HP VHP HP VHP VHP LP MP 110 RSSC Medical Services (Simunye) LUBOMBO No data FY12 New site FY12 New site FY12 New site VHP HP HP VHP MP MP VHP VHP LP MP 111 Silele Red Cross Clinic SHISELWENI MP HP VHP VHP VHP HP HP HP MP HP HP LP MP 112 Siphofaneni Clinic LUBOMBO MP VHP MP LP MP 113 Siteki Phu LUBOMBO MP VHP VHP HP VHP VHP MP HP HP HP HP MP LP MP 114 Sithobela Health Center LUBOMBO HP MP LP MP 115 Family Life Association Clinic (Mbabane) HHOHHO MP MP HP VHP VHP VHP VHP HP HP VHP VHP VHP LP 116 Lomahasha Clinic LUBOMBO VHP VHP VLP VHP LP 117 Lubuli Clinic LUBOMBO MP HP VHP HP VHP VHP HP VHP HP HP VLP VHP LP 118 Lushikishini Clinic MANZINI HP VHP VHP VHP VHP VHP HP HP VHP VHP VHP VHP HP LP 119 Mangweni Clinic HHOHHO VLP HP HP HP VHP HP HP MP HP HP VHP HP LP 120 Mawelawela Women Correctional Services ClinicMANZINI LP HP VHP VHP HP HP VHP HP HP HP HP HP LP 121 Nkonjwa Clinic LUBOMBO LP LP VHP VHP VHP HP VHP HP HP HP MP MP LP 122 Ntfonjeni Clinic HHOHHO HP MP MP MP LP 123 SOS Children's Village Clinic (Mbabane) HHOHHO MP HP VHP VHP VHP VHP VHP HP VHP HP VHP LP LP LP 124 St. Phillip's Clinic LUBOMBO LP MP VHP VHP HP VHP HP HP MP VHP LP VLP LP 125 Vuvulane Clinic LUBOMBO HP VHP VLP NA LP 126 Ebenezer Clinic MANZINI LP 127 SOS Clinic (Nhlangano) SHISELWENI MP HP HP VHP VHP VHP VHP MP HP HP HP LP LP VLP 128 Luyengo Student Clinic MANZINI MP VHP VHP VHP HP VHP HP HP VHP VHP VHP HP NA 129 Mhlume Medical Services LUBOMBO HP FY12 New site FY12 New site FY12 New site HP VHP HP HP HP HP HP LP HP NA 130 Mpuluzi Clinic MANZINI VHP FY12 New site FY12 New site FY12 New site VHP VHP VHP VHP VHP HP HP VHP MP NA 131 National Baptist Clinic HHOHHO MP HP HP VHP VHP VHP HP HP VHP VHP VHP HP MP NA 132 Ngomane Clinic MANZINI MP MP MP VHP HP HP HP HP VHP HP HP HP MP NA 133 Tabankulu Estates Clinic LUBOMBO LP VHP VLP NA 134 Tambuti Estate Clinic LUBOMBO VLP MP VLP NA 135 Etetsembisweni Clinic MANZINI New Sites NA 136 Kwaluseni Campus Clinic MANZINI New Sites NA 137 Mbabane Campus HHOHHO New Sites NA 138 U‐TECH Clinic LUBOMBO New Sites NA *Had zero report for Q1 FY12 Scoring Methodology Score Point Performance % >90 and above 1 VHP 37% >80‐<90 2 HP 20% >70‐<80 3 MP 23% >50‐<70% 4 LP 19% <50% 5 VLP 3% Total 126 n=7 sites Calculations not possible due to either zero first ANC, Zero HIV+ women etc I 28 support and activities at regional level and health facilities challenging, although they are all represented at policy level in the TWG. EPAS reported that even when the US PEPFAR team mandated coordination, efforts only lasted a few months before Implementing Partners returned to previous autonomous behaviors. A non-PEPFAR development partner reported to the evaluation that it has its own agenda, and it does not coordinate with RHMTs or PEPFAR implementing partners. The lack of coordination risks i. lost opportunities to complement activities, and ii. duplication of effort. Health facility staff have to adapt to each partner’s ways rather than partners supporting a single approach—burdening facility staff rather than increasing partner staff effort in coordination. Medical professional perspectives on PMTCT service quality: EPAS funding of technical personnel seconded to the MOH SRHU and an ART physician seconded to King Sobhuza II Clinic, in Manzini, will have strengthened the PMTCT services from a medical/ professional perspective. The large investment in training and mentoring health facility staff in PMTCT, and providing guidelines and other job aids supporting delivery of PMTCT services integrated into the ANC and MNCH standard package of services will have increased the health facility staff knowledge and skills, and thus their service delivery competencies. The improved clinical/professional quality of PMTCT services will continue while the staff are retained in facilities providing PMTCT services beyond the end of the EPAS activity. Clearly displayed local health facility policy statements, such as the one photographed in Lubuli clinic (please see box) provide a clear framework for the service providers to be aware of their responsibilities to provide standardized, quality services. Many of the health facilities visited had clearly displayed job aids and standard procedures, developed from the National PMTCT Guidelines, which support application of knowledge and skills by service providers in their patient care. Service user perspectives on PMTCT quality: Health workers reported that EPAS training included a component on compassion in service delivery. Most clients met with by the evaluation stated that their health care was “good”. Respondents in the community and at one clinic reported that the occasional health worker is known to “have attitude”. Women in the community reported to the evaluators that they talk about their experiences and some are able to choose to attend a different facility when they know of a health care worker with a bad attitude. Some respondents in the community and some clients at a health center visited also reported that clinics do not always open at the published opening hour and service users are forced to wait long hours to receive services. When asked about the quality of maternity care, clients who met the evaluators outside Maternity Units had some dissatisfaction with the care but were happy they had a healthy baby. Complaints included being shouted at and not being fed after delivery until the next scheduled meal even when that was many hours away. One group when asked if they were covered when they were in labor and delivery said they were not. When asked what they thought about that said simply, “When we get 29 in there we are all in it together. We just don’t think about it and forget it afterwards.” A discussion [in English] with a mother after she had delivered a breech infant at a health center and her male midwife elicited that she had chosen to come to the health center because of her bad prior experience at a hospital Maternity Unit. She stated that the staff there were “unkind” and “shouted” at her. The midwife who delivered her in the health center said, “Tell her I was kind – I rubbed your back.” The mother agreed with him39. Undoubtedly, the improvement in quality of PMTCT services that clients notice and reported on most, was the introduction of mentor mothers. Mentor mothers were universally appreciated by the clients the evaluation met with, who reported that they meet with a mentor mother on each PMTCT visit in ANC and often postnatally too. This evaluation finding contrasts with EGPAF barriers to ART initiation study40 that identified health care worker and client concern about MMs: knowledge, training, communication skills (health care workers) and clients being treated inappropriately—harshly (client concern). The difference in the finding probably results from the m2m program maturing in the intervening years with the MMs possibly having better training, but certainly the health care workers have developed better understanding the role of MMS and have also developed more trusting working relations with the MMs. C) NATIONAL PMTCT SYSTEM STRENGTHENED The MOH respondents reported they view EGPAF as a “trusted partner” that will step forward with technical assistance (TA) when the government is short of technical human resources at policy level. EGPAF participates in Technical Working Groups and their sub-committees (PMTCT, Care & Treatment, Pediatric HIV, M&E) and is co-chairing the National Option B+ Task Team; the EGPAF Country Director is recognized as providing vision and leadership to elimination of pediatric AIDS through the TWG. The MOH and UN organizations reported to the evaluation that EGPAF technical leadership is “invaluable” and “essential” because the MOH has a lack of technical capacity at higher levels. Respondents remarked that EGPAF led the way on mentoring in Swaziland. In acknowledgement of his technical leadership and expertise, UNAIDS had recently issued an invitation for the EGPAF Country Director to participate in a meeting in Geneva on setting indicators and standards for universal access to post natal PMTCT services.41 EGPAF reported it provided technical assistance to:42  the development and in the finalization of the National Strategic Framework for the 39 The evaluator did not enter the delivery room but had observed the male midwife talking encouragingly to the client and rubbing her lower back in the labor ward. When the midwife realized that the baby was a breech presentation, he immediately sent for the nurse in charge of the health center [who was in an evaluation interview.] The nurse in charge of the health center came immediately to assist arriving as the baby was delivered safely by the midwife. 40 EGPAF (2012) Swaziland Barriers to Antiretroviral Therapy Initiation for Eligible HIV-Positive Pregnant Women in Antenatal Care. EGPAF, Washington, DC 41 Although EGPAF headquarters declined to give permission for him to attend because a large contingent from EGPAF headquarters was participating in the meeting. 42 The details are taken from EGPAF Annual Reports. However much of the TA at national level was described appreciatively to the evaluation by senior MOH respondents. UN respondents also gave examples of EGPAF technical leadership at national/policy level. EGPAF participants in activities that produced MOH guidelines, frameworks and other reports is detailed in the publications’ acknowledgements. 30 Elimination of New HIV Infections Among Children by 2015 and Keeping Their Mothers Alive by participating in the National Steering Committee  the SRHU in the finalization of the Revised National PMTCT Guidelines and implementation of the Guidelines at all the 4 Regions  the development of M&E tools and indicators to monitor the new PMTCT guidelines  the MOH in the finalization of the Revised National Infant and Young Child Feeding Guidelines  the development of the study protocol for the National PMTCT Impact Evaluation, the study data analysis and report writing  the technical leadership to the National Option B+ Task Team formed by the National HIV Care and Treatment Technical Working Group  the revision of the National Sexual Reproductive Health (SRH) Policy and Integrated SRH Strategic Plan  the costing of the Eliminating Mother To Child Transmission Plan for Swaziland  the finalization of Family Planning/ART Standard Operating Procedures  the compilation of the National PMTCT Report for 2012  and technical leadership in the adaptation of the PMTCT, Pediatric and Adult HIV Care & Treatment, and HIV Testing and Counseling Guidelines in line with the 2013 WHO Consolidated Guidelines on the Use of Antiretroviral Drugs for Treating and Preventing HIV Infection  to the National Emergency Response Council on HIV and AIDS (NERCHA) and UNAIDS in developing the 2012 Global AIDS Response Progress Report—EGPAF assisted with the review the PMTCT section of the report INSTITUTIONAL STRENGTHENING EGPAF reported that EPAS has adopted the World Health Organization building blocks approach to institutional strengthening and so the evaluation considered the three building blocks that are relevant to EPAS interventions—a well performing health workforce; a well-functioning health information system; and good health services delivering effective, safe, quality services when and where they are needed.43 EPAS has funded various posts in the MOH, some but not all of which have been absorbed: initially a training officer and supervision officer in the SRHU, later an MNCH Advisor was added; and an ART physician at King Sobhuza II Clinic. The latter position had responsibilities for supporting ART at other facilities as well as at his base clinic. Currently, four of EGPAF’s Program Coordinators have been redesignated PMTCT Coordinators and are reporting to the RHMT, although still employed by EGPAF. Building Block: Well Performing Health Work Force EGPAF reported EPAS has invested heavily in training: off site, onsite and mentoring, and has provided TA to inclusion of PMTCT in pre-service nurse training. The topics covered by EPAS training are detailed above on page 18. This investment has undoubtedly strengthened the MOH capacity to consistently provide quality PMTCT services. The evaluation observed that—with the exception of in Maternity Units—health care workers are often working within cramped 43 WHO (2007) Everybody business: strengthening health systems to improve health outcomes: WHO’s framework for action. World Health Organization, Geneva, Switzerland 31 overcrowded buildings yet delivering good quality, comprehensive PMTCT services integrated into MNCH services. Analysis of the HMIS demonstrated that health worker performance delivering the package of PMTCT services increased from 2010 to 2013. (For examples of improved performance, please see Figure 5 on page 15.) EPAS reports that its approach to training and mentoring of health facility staff on avoiding missed opportunities for delivery of PMTCT contributed to the service delivery performance. By introduction of mentor mothers to PMTCT services, EPAS has enabled task shifting—to mentor mothers—in line with government policy on task shifting. This is effective in relieving the burden of lower level tasks on registered nurses. Building Block: Well-functioning health information system: HMIS/M&E The HMIS in Swaziland is comprised of three levels of reporting, starting from the health facility level where the service providers complete registers each time a client is seen. At the end of the month, the service providers complete a health facility monthly summary report. The next level is regional; data is aggregated into monthly summaries and entered electronically into the national database. Quarterly, the RHMT office produces and disseminates a report to health facilities as feedback. The third level is the national level where the national M&E Unit prepares a quarterly report that is shared with the national PMTCT program, the Public Health Directorate and implementing partners. The national M&E Unit prepares the annual report. Thus, key PMTCT service indicators are reported on a monthly, quarterly and annual basis to monitor coverage, uptake and progress in the scale-up of PMTCT interventions toward universal coverage. EGPAF and MOH respondents reported that EPAS supports the HMIS/M&E system at all three levels: National Level EPAS played a key role in providing TA to the Strategic Information Department for the revision of PMTCT tools and indicators to align them with the National Strategic Framework for Accelerated Action for the Elimination of New HIV Infections Among Children by 2015 and Keeping Their Mothers Alive. EPAS prepared data collection guidelines that were included in the register books used at the health facility level. EPAS also supported the Strategic Information Department in printing and distributing the National Strategic Framework, and the new tools: register books with the guidelines for the health facilities. EPAS participates and supports national dissemination meetings that are held quarterly. Both the M&E and HMIS unit staffs at the national level reported that they did not have enough information on EPAS plans. They said that although EPAS always comes to their rescue when needed, they wanted this support to be planned and systematic rather than only in response to emergency situations. They would like a forum and regular sharing with the M&E and HMIS units the EPAS plans and information in areas related to the MOH M&E and HMIS. Regional Level At regional level, EPAS provides TA to improve data quality by working with the regional offices to conduct joint supervision visits to health facilities. Technical assistance is also given to data cleaning, analysis, and preparation of regional quarterly reports. Regional health management team staff were trained in advanced EXCEL by EPAS, and are currently using cascade analysis templates introduced by the project. This evaluation confirmed the findings in two USAID DQAs in September 2011 and July 2012 that the aggregation and use of data at regional level from the 32 health facility monthly summary reports is accurate and the HMIS system is established appropriately from health facility to national level.44, 45 However, this evaluation did far more checking the monthly reports against their data sources—the health facility registers—than the DQAs. From the 2011 DQA report we understand that USAID checked two sets of data back to source, ART data at Mbabane PHU and PMTCT data at Siteki PHU, both of which were high performing facilities in the 2011 EPAS Year 1 Site Level Baseline Performance Report.46 This evaluation looked at three indicators at all the facilities visited—including low performing sites— and checked the monthly reports for 2013 against the registers for 2013. Please see Quality of data below. Health Facility Level EPAS supports the HMIS at facility level by providing transportation for RHMT personnel and through direct TA by EPAS staff. Both EPAS and USAID reported that EPAS uses a data driven mentorship strategy. Mentors use the specific PMTCT cascades for the Facility they are visiting and encourage the facilities to use theme to identify gaps [missed opportunities to provide PMTCT services] that need strengthening. EPAS also provided training for service providers on the use of new integrated MNCH/PMTCT register books and supports monthly meetings at health facilities to analyze data and address data issues. In selected sites, EPAS staff said they conducted DQAs and disseminated results to RHMTs. Quality of data: This evaluation conducted an analysis of primary data collected at service sites in 2013 and compared it with monthly summaries produced for the same period, at the health facilities visited by the evaluators. The initial comparison involved three indicators: i. Number of pregnant women attending first ANC visit this month; ii. Number of pregnant women tested for HIV at the first ANC visit this month; and iii Number of Exposed infants who tested for HIV using DNA PCR at 6-8 weeks and received result. The evaluation would have liked to have used a fourth indicator: Family planning (FP) attendees. However the FP register book is not user friendly and that made it difficult for the evaluation team to reconstitute the figures to compare with the summary reports and so that indicator was not analyzed by the evaluation. The comparison on the three indicators was carried out by counting the entries in the ANC and child welfare clinic register books for each month in 2013 and comparing the counts with the PMTCT monthly summary reports for 2013. All the information was not collected from every facility visited as availability of the registers depended on the timing of the evaluation visit in relation to availability of staff and other factors. The three figures below show the degree of agreement for each indicator and for each health facility visited. 100% represent full agreement. Below 100% means under-reporting and above 100% over-reporting. For number of pregnant women attending first ANC visit this month the proportion of agreement is very high, with only two facilities over-reporting data (King Sobuza II Clinic, and Mawelawela Clinic). Both Sithobela HC and Mankayane hospital under-reported pregnant women attending for their first ANC visit. [Please see Figure 8, on page 33 below.] 44 USAID (2011) DQA 1_EGPAF_September_20-21_2011_Final Report. USAID, Mbabane, Swaziland 45 USAID (2012 ) DQA 2_EGPAF. SZ_July 2012. USAID, Mbabane, Swaziland 46 Kudiabor, Kwashi (2011) EPAS Year 1 Site Level Baseline Performance Report. EGPAF< Mbabane, Swaziland 33 The number of pregnant women tested for HIV at the first ANC visit this month, showed greater variance, with many facilities underreporting the number of pregnant women tested for HIV at the first ANC visit. [Please see Figure 9, below.] Figure 8: Proportion of agreement between register book and reports - Number of pregnant women attending first ANC visit, year of 2013 101% 85% 100% 84% 99% 100% 98% 114% 98% 144% 0% 20% 40% 60% 80% 100% 120% 140% 160% GOOD SHEPHARD HOSPITAL MANKAYANE HOSPITAL EMKHUZWENI HC SITHOBELA HC LOMAHASHA CLINC LUBULI CLINIC LUSHIKISHINI CLINIC MAWELAWELA MLIBA NAZ CLINIC KING SOBUZA II CLINIC Figure 9: Proportion of agreement between register book and reports - Number of pregnant women tested for HIV at the first ANC visit , 2013 92% 108% 95% 80% 74% 108% 77% 105% 57% 96% 100% 103% 0% 20% 40% 60% 80% 100% 120% GOOD SHEPHARD HOSPITAL MANKAYANE HOSPITAL EMKHUZWENI HC SITHOBELA HC LOMAHASHA CLINC NTFONJENI CLINIC LUBULI CLINIC LUSHIKISHINI CLINIC MAWELAWELA MLIBA NAZ CLINIC SILELE RED CROSS CLINIC KING SOBUZA II CLINIC 34 The number of HIV exposed infants tested for HIV using DBS PCR at the 6-8 week visit showed high variance in the facilities visited. [Please see Figure 10, above.] EPAS senior management was unaware of the reporting issues prior to the outbrief by the evaluators. The variance demonstrated by the evaluation cannot be generalized as some facilities under report on some indicators and over report on others. However, the variance shows that EPAS efforts to improve reporting by health facilities has not yet achieved desirable accuracy across these indicators by all the health facilities. The monthly mentoring activities—which the service providers commonly reported to the evaluation team focuses on service provider reporting errors and data quality—may perhaps have become a meaningless routine rather than an important activity for demonstrating to the service providers improvements in health facility performance delivering PMTCT services. Building Block Good health services delivery The two aspects of the delivery of good health services EGPAF reported that EPAS addresses are support to logistics and supplies, and strengthening of the MNCH platform—a priority of the MOH. Logistics & Supplies EGPAF collaborates with MSH and Central Medical Stores on drug supply management including providing assistance with quantification and forecasting, and to build health facility staff capacity to manage CTX and ARVs without stockouts. EGPAF provided assistance with the costing of the Eliminating Mother to Child Transmission Plan for Swaziland. Together with other partners, EGPAF Figure 10: Proportion of agreement between register book and reports - Number of HIV exposed infants tested for HIV using DBS DNA PCR at 6-8 weeks, 2013 105% 103% 147% 64% 71% 96% 77% 71% 100% 96% 124% 0% 20% 40% 60% 80% 100% 120% 140% 160% GOOD SHEPHARD HOSPITAL MANKAYANE HOSPITAL EMKHUZWENI HC SITHOBELA HC LOMAHASHA CLINC NTFONJENI CLINIC LUBULI CLINIC LUSHIKISHINI CLINIC MAWELAWELA MLIBA NAZ CLINIC KING SOBUZA II CLINIC 35 provided support to the MOH National Reference Laboratory to develop an emergency strategy to resolve a 2012 national stock out CD4 test reagents.47 Although many EPAS staff reported strengthening logistics and supplies, the interventions described were mainly of filling local gaps in logistics and supplies to keep services running rather than strengthening the logistics system. Both EPAS staff and health facility staff reported that EPAS has provided transport to the RHMTs and has supported redistribution of drugs and commodities between health facilities. The redistribution has reduced stockouts and loss of stock through date expiry. Drugs and supplies have been redistributed to high volume sites that had stocks running low from lower volume sites with adequate stocks. Short shelf life drugs and other supplies have been redistributed from low volume facilities to high volume facilities for use before they expire. While not strengthening the system per se, these interventions have increased the efficiency of the health facility services and reduced wastage. The MNCH Platform The main entry points to PMTCT are reported to be through MNCH services: women learn their HIV status through testing in ANC, in PNC and child welfare clinics. EGPAF reports that EPAS has worked hard on building the knowledge and skills of the nurses and midwives delivering ANC, PNC and child welfare clinic services. Many of the skills and increased knowledge—for example in promoting family planning, safer sex, and encouraging women to exclusively breastfeed for the first 6 months of their infants’ life—directly benefit the delivery of ANC, PNC and child welfare clinics. The small equipment provided also benefit the wider population of clients attending the health facilities who do not require PMTCT services. Changes in client flow through clinics to make them more efficient benefit all service users. EGPAF reported that EPAS has specifically worked to strengthen maternity care in high volume sites, with provision of small equipment such as fetal stethoscopes and training in use of the partogram as well as training in reducing MTCT of HIV during labor and delivery. This should improve the outcomes for the infant although it is beyond the scope of the current evaluation to demonstrate this improvement. The evaluation found that there is still room for improvement—for example in strengthening leadership and management of health facilities and particularly in maternity units to provide privacy and respect for birthing women’s dignity—within EPAS remit, in the last year of implementation. Community Linkages As the service delivery side has improved, linkages with the community to address demand side issues—barriers to uptake of services and adherence to treatment—become increasingly important. Client reluctance to accept ARV prophylaxis and ART: Many health workers and MMs at facilities visited reported that women are reluctant to accept PMTCT prophylaxis that continues after the infant is born: drugs taken during pregnancy are seen by the wider family as for the benefit of the infant, but drugs the mother takes after delivery are stigmatized as indicative that she has AIDS. These opinions were confirmed by the MNCH mentors who accompanied the evaluators and EPAS 47 As evidenced by EGPAF Semiannual and Annual Reports. The evaluators were unable to meet with Central Medical Stores and MSH. 36 partners at the community event observed by the evaluators and mirror findings in the Barriers to ART Initiation study 48. Many nurse respondents volunteered that they were not hopeful about the success of rolling out Option B+ as that requires the mother to acknowledge her status and that she will benefit from treatment. Swazi women’s lack of agency: Evaluation respondents said that women in Swaziland must defer to their partners and to an extent to their parents-in-law and do not have autonomy in decision making about themselves or their children. They may not have autonomy in health seeking behavior. Some respondents thought that some women may say that they cannot accept prophylaxis or treatment without their partner’s consent when it is the woman herself who is reluctant, however most women do indeed have to seek permission from the partner to attend for health care for themselves and their children. Women who are reluctant to accept prophylaxis or treatment may have not disclosed their status to their partner and fear abandonment and loss of financial support. As in many other cultures, respondents asserted that if the woman tests first and discloses she is HIV positive, it is assumed that it is she who brought the infection into the family. EGPAF and partners AMICAALL and Lutsango report that their community programs address gender and other barriers to PMTCT, (please see the next page). Partner involvement in PMTCT: Women need more than their partner’s permission to attend for PMTCT services and the bus fare to enable her. Male involvement in PMTCT requires male partners of negative pregnant women to accept HTC so that HIV positive partners can receive ART and reduce the chance of infecting their HIV negative pregnant partners. Meaningful involvement requires shared responsibility and decision making, and support to the HIV positive pregnant woman to enable her to accept and adhere to prophylaxis and treatment for her own health care. It requires shared responsibility for family planning and support for exclusive breastfeeding of their infant for the first 6 months of life. For HIV negative women, male involvement requires consistent and correct use of condoms during pregnancy and lactation, which implies acknowledgement that the man can infect the woman with HIV during pregnancy and lactation. It is not possible to adequately address these socio-cultural and gender issues from within the health care delivery setting alone. EPAS efforts to increase partner involvement: EGPAF has introduced invitation letters that health facility staff are able to address and send home with a pregnant woman to her partner. [Please see the box]. The cover is an attractive color photograph of an infant in “protecting hands” and the letter is written in both English and SiSwati. Supplies of these invitation letters were seen by the evaluators in several health facilities and nursing staff reported that the letters are helpful to some women to get their partners to come into the health facility for testing. Several health 48 EGPAF (2012) Swaziland Barriers to Antiretroviral Therapy Initiation for Eligible HIV-Positive Pregnant Women in Antenatal Care. EGPAF, Washington, DC 37 facilities reported that they are seeing an increase in partners coming into ANC—but still very few. The evaluators saw two male partners attending King Sobhuza II Clinic where the nurse in charge reported that their main difficulty with men attending the clinic is that there are no male toilets. Men who need the toilet use the patient toilet which does not have a urinal and is used by women clients. EPAS has provided cellphones and airtime for following up positive pregnant women who miss appointments. Mentor mothers interviewed by the evaluators said that they record pregnant women’s phone numbers and call those who have missed appointments. Mostly this system is reported to be working well except near the border with South Africa where pregnant women may use South African cellphone services—and may actually reside in South Africa—because the cost of calls to South African numbers is very high compared with the cost of calls to Swaziland numbers. EPAS partners, AMICAALL, Lutsango Lwakangwane and SINAN, have implemented approaches to linking communities to health services, and addressing the socio-cultural and gender barriers to uptake of services and adherence to treatment: AMICAALL reports that it has expanded to 11 urban areas (out of the official 12 in the country) although its funding through EPAS is limited. It uses volunteer community health workers to promote PMTCT—and male involvement including facilitating fatherhood and motherhood support groups. The volunteers go door to door and identify pregnant women who they “refer” to health facilities for ANC. There is no mechanism to track the referrals and to measure any increased uptake in services that result. AMICAALL reports that it regularly meets with health workers—in the community and at the facility—and discusses the referral network in effort to strengthen the linkages of the community to the health facility. AMICAALL reported that it offers stipends to community leaders but not to the volunteer community health workers which has resulted in low leverage capacity to demand activities and outputs (the volunteers think they should be remunerated in return for their activities providing services to the community). AMICAALL attempted to implement an income generating activity to motivate the community health workers but that was not as successful as expected, leaving motivation of volunteers a challenge. Lutsango Lwakangwane reports that it works with traditional leaders and structures mobilizing the women’s “regiment” [traditional grouping of women] and informing them about issues related to MNCH/ PMTCT. EGPAF and Lutsango respondents stated that the women’s regiment then mobilizes the community—men as well as women—to attend community events. These events are organized by Lutsango in collaboration with EGPAF and PSI. The evaluation team attended one event in Shiselweni that followed the model for events described by EGPAF & partner respondents. The local health facility set up a small clinic for treating minor ailments; PSI was there with voluntary counseling and testing facilities in a tent. EGPAF provided a marquee, refreshments, and some prizes. Kwasa Lokungali Drama Group produced a moving drama starting with the funeral of an infant and exploring the relationship of the “traditionalist” father and mother who wanted them both to seek help and HIV testing. The drama touched on relationships within the family, gender￾based violence and blaming the mother for the death of her infant. The audience divided into peer groups for discussions after the drama: there were two groups of women, two groups of men, an older adolescent peer group and a younger adolescent group; all were large groups (30-40 38 persons). The groups were facilitated and went through a series of questions for the peers groups to discuss and understand what they had seen and how it related to their lives. This went far beyond traditional “IEC” [information, education and communication] and other expert driven approaches. Many of the key informants at health facilities mentioned that periodically EPAS had held events in their communities and that these events included “education” or dramas and discussions. One group of rural health motivators met with in Hhohho also described events in their area. When asked how effective the events are, the rural health motivators said that the events are good—in the weeks after an event they notice that more men come to the door when the rural health motivator calls at the house. The men are more willing to talk for a while, but that this doesn’t last. If there aren’t follow up events that continue to engage the men, they soon go back to their usual behavior and don’t engage with the rural health motivators. Lutsango isn’t funded at a level to conduct events at scale throughout Swaziland and to follow each event with a full program of follow up events with different dramas, to sustain any change in relationship behaviors or health seeking behaviors. Importantly, the events are not set up as a pilot intervention with monitoring and evaluation of outcome indicators that would measure their effectiveness in changing attitudes and behaviors. SINAN is spearheading interventions focusing on infant and young child feeding, training health workers at the facility and in the community and MMs at the facility. SINAN reported that it believes the approach has increased the rates of exclusive breast feeding although the training was compressed into a single two hour session which is less than ideal. Furthermore, only 200 health facility based and 200 community based service providers were reached against an estimated 5,000 community health workers who are active in the country, due to resource constraints. Rural Health Motivators have not been actively involved in the program although this cadre of salaried health worker exists throughout Swaziland and works door to door, at household level. SINAN noted that there is only limited advocacy for maternity protection to support working mothers (the International Labour Organization convention of 2000 has not been ratified by Swaziland). Feeding infants whose mothers work outside the home—particularly in industrial settings where mothers return to work only three to four weeks post-delivery—is a major challenge for PMTCT and for infant nutrition in general. d) Protocols, Guidelines, and Job Aids Developed with MOH EGPAF technical assistance to the development of the revised National PMTCT Guidelines was appreciated by the MOH respondents. EGPAF 2011 Annual Report documents that it provided support to the MOH in the finalization of revised National Infant and Young Child Feeding Guidelines and technical assistance for the development of tools for piloting of nurse-led ART initiation. As documented above, on page 31, EGPAF provided technical assistance to the development of the National Strategic Framework for Accelerated Action for the Elimination of New HIV Infections Among Children by 2015 and Keeping Their Mothers Alive and provided funding for its printing and distributing; EGPAF also assisted the MOH with the development of the new HMIS tools [registers] that mirrored the Strategic Framework and funded the printing of the M&E tools. The registers include guidance on their use [and EPAS provides training and mentoring on the HMIS at health facility level.] EGPAF reported that it assisted the MOH in the development of 39 family planning/ART standard operating procedures. EGPAF’s 2013 Annual Report details working with the MOH in the adaptation of the PMTCT, Pediatric and Adult HIV Care & Treatment, and HIV Testing and Counselling Guidelines in line with the 2013 WHO Consolidated Guidelines on the Use of Antiretroviral Drugs for Treating and Preventing HIV Infection. EGPAF co-chaired the PMTCT guideline revision team and actively participated in the other guideline revision teams. The evaluation observed many standard operating procedures and other job aids posted on health facility walls that were provided by EPAS. Some were locally printed on A4 paper—presumably in response to a specific health facility need; others were laminated and included MOH and EGPAF logos. One health facility had a recently received PMTCT counseling flipchart produced by EGPAF and ICAP, please see box. IMPLEMENTATION GAPS Service delivery gaps The lack of definition of quality and service delivery quality standards documented above [please see pages 16-17] is a gap. The few areas of weakness that the evaluation identified—for example not dispensing ARVs in PHUs but referring clients to hospital pharmacies [please see page 19 above]; gaps in recording dispensing of ARVs in labor and delivery wards, and consequential stockouts of infant nevirapine [please see page 22 above], late opening of some clinics [please see page 28 above]—and the more serious gap in respect for the dignity and privacy of women in labor and delivery wards [please see page 22 above] indicate there may also be a gap in EPAS support to leadership and good management of health facilities. Gaps in indicators and reporting on activities There are also gaps in indicators and reporting for the community activities implemented by AMICAALL and Lutsango Lwakangwane [please see page 41 below]. m2m reporting—which the evaluation found to be hugely burdensome against a very large number of indicators in m2m registers—is not linked to the reporting from the health facilities. Gaps in institutionalization and bringing to scale demand side interventions The most important gaps in EPAS implementation are lack of (i) effective strategies to retain mother-baby pairs in follow up until after cessation of breastfeeding; and (ii) institutionalization and bringing to scale interventions to address demand side gaps and link communities with health facilities. [Please see page 41 below and Question 2. The successes, challenges and gaps in EPAS’s community approach, below.] With the expanded provision of PMTCT services [please see pages 13-14 above] and the improved performance of health facilities to deliver quality facility-based PMTCT services [Please see pages 14-15 above], demand side gaps have become increasingly important barriers to achieving universal access to PMTCT services in Swaziland. Although AMICAALL and Lutsango Lwakangwane are providing potentially effective interventions, they do not have funding to implement at scale and yet have not set up in a limited number of com￾munities as a pilot intervention to demonstrate the effectiveness of their interventions. After the end of EPAS project, AMICAALL and Lutsango Lwakangwane community mobilization will cease. 40 QUESTION 2. THE SUCCESSES, CHALLENGES AND GAPS IN EPAS’S COMMUNITY APPROACH Please also see Community Linkages on page 35 above. EPAS community approach: EGPAF reported that EPAS community approach is organized through its community linkages team who seek to empower communities around the country with knowledge about PMTCT through conducting community dialogues, school debates and children and adolescents support group meetings. Community linkages staff reported that male dialogues target male dominated industries to learn from them the reasons men do not support their partners during pregnancy, birthing, and the breastfeeding period. EPAS staff reported they work with some facilities to support children and adolescents living with HIV, those who were infected by MTCT before the widespread push to deliver PMTCT services. As detailed above (please see page 16) EPAS has also provided health facilities with cellphones and airtime for MMs to reach out to positive pregnant women who have missed appointments at the health facility; and has provided invitation letters for positive pregnant women to take home to their partners (please see page 36 above). Successes and challenges: EGPAF reported that community events organized by EPAS partners are successful in mobilizing 200-300 persons per event, for sensitizing about the importance of pregnant women and their partners knowing their HIV status and utilizing PMTCT services at the health facility. The awareness raising—through drama—addresses important topics such as status of women in Swaziland, their lack of agency related to MNCH and PMTCT, gender-based violence and stigma. The model for events described to the evaluation include facilitated peer group discussions to reinforce lessons learnt from the dramas; voluntary HIV counseling and testing is available at the events, provided by PSI. Rural Health Motivators reported to the evaluation that these events stimulate more men to engage in discussion with the Motivators when they go door to door—for a period, but then men’s engagement tails off over time, back to earlier low levels. AMICAALL Reported it has paid stipends to its community leaders who champion PMTCT but it has not had EPAS funding for payments to its community volunteers. In response to the lack of funding for community volunteer payments, AMICAALL reported that it attempted an income generating project with the volunteers. AMICAALL reported that this was unsuccessful but didn’t specify why. It reported that maintaining the enthusiasm and motivation of the AMICAALL volunteers in the absence of stipends has proven a challenge. Sustainability: The support groups at facilities for children and adolescents living with HIV are likely to continue while there are health facility staff interested in facilitating the support groups—one nurse who facilitates a group for adolescents living with HIV who met with the evaluators said that her group predated EPAS but it continues to struggle because of a lack of an appropriate space for the young people to meet in. Both AMICAALL and Lutsango reported they have existing relations to communities prior to their involvement in EPAS. AMICAALL reported its volunteers have previously volunteered in earlier AMICAALL activities and AMICAALL says they will continue to be volunteers with AMICAALL after EPAS. Similarly Lutsango reported it has an ongoing relationship with the Women’s Regiments. Both organizations reported to the evaluation that they believe that PMTCT will remain on community leaders’ agendas after the end of EPAS. 41 Gaps: The EPAS model does not have an evidence base for its effectiveness although it has similarities with Stepping Stones, the social mobilization methodology that has demonstrated effectiveness. However, Stepping Stones is a far more iterative process than EPAS is implementing in its community approach.49 Stepping Stones is implemented through 13 three hour sessions including drama and role play, with three facilitated peer group discussions in each session and critical reflection. This is very much more intensive than the EPAS one off events. Although the EPAS budget for community interventions and its subgrants to Lutsango and AMICAALL are not large enough to take the community approach adopted to scale throughout Swaziland, EPAS could have implemented its community approach within its budget, in a sample of communities, as a pilot project. With monitoring and evaluation of the outcomes of the activities in terms of increasing male involvement, reducing gender-based violence, increasing uptake of PMTCT services and increasing adoption of PMTCT services—particularly postnatally from the 6 week check-up through to cessation of breastfeeding—EPAS could have generated useful evidence on the effectiveness of its approach. Thus, the gap in the community approaches EPAS has used is the lack of M&E to provide evidence for the effectiveness of the EPAS community approach in changing PMTCT behaviors and use of services. QUESTION 3. SUSTAINABILITY OF THE GAINS MADE UNDER EPAS Implementation of EPAS has been through the existing health care delivery system in support of the MOH Elimination of New HIV Infections Among Children By 2015 And Keeping their Mothers Alive National Strategic Framework for Accelerated Action contributing to and implementing the 2010 National PMTCT Guidelines. Thus the policy level framework and tools are in place to sustain the technical gains supported by EPAS. PMTCT Mentorship: EGPAF reported that EPAS has worked extensively with the RHMTs and the approach to mentoring is well established. The MOH respondents at national, regional and facility level all reported to the evaluation their commitment to the approach. Although EGPAF has seconded Regional PMTCT Coordinators to the RHMTs, they are remunerated by EPAS. EGPAF reported that its early discussions with the MOH indicate that the PMTCT coordinators could only be absorbed at staff nurse grade. Retention of Regional PMTCT Coordinators at staff nurse grade on MOH pay scales and conditions might be challenging for individuals who have international NGO experience. Discussions the evaluation team had with the MOH demonstrated that the MOH have not yet finalized their requirements for mentors but that they are moving towards integrating PMTCT and RHMT mentor functions so that all RHMT mentors support integrated PMTCT/ MNCH services. The MOH had not fully decided the professional qualifications of RHMT mentors to meet the future need for supporting integrated PMTCT/MNCH at the time of this evaluation. Nonetheless, the mentorship approach will be sustained by the MOH after the EPAS activity ends. The HMIS. Apart from the weaknesses the evaluation found at health facility level, the HMIS is working well and, as a government system, will continue beyond EPAS. The MOH reported that it 49 Jewkes, Rachel et al (2007) Evaluation of Stepping Stones: A gender transformative HIV prevention intervention. The Medical Research Council, Pretoria, South Africa 42 has trained regional HMIS coordinators who are responsible for data entry into the national database, which is accessed online by the national staff HMIS. RHMTs reported that their staff also carry out data cleaning. The two DQAs conducted by USAID in 2011 and 2012 showed high accuracy of data entered into the HMIS at regional level and aggregated at national level. The RHMTs report that they supervise the health facilities and provides technical assistance. The data files used by EPAS, and this evaluation, are taken from the HMIS dataset. EGPAF reported that the RHMTs are well versed, through working with EPAS, in the HMIS including the need for regular monthly data quality review meetings to review the monthly reports from health facilities and identify gaps [missed opportunities for providing PMTCT services] in service delivery. The EPAS developed system for grading health facility performance could be used post EPAS to assist RHMTs prioritize where their mentoring visits should focus greatest effort. Site graduation. The most recent site performance data available to the evaluation demonstrates the trends and variability of facility performance. Please see Table 4 between pages 27 and 28. EGPAF reports that EPAS supports sites with monthly mentoring visits and other TA including onsite training for two years and continues monthly mentoring until the site consistently achieves very high/high performance. At this stage, clinics and some smaller health centers are considered “graduated” and their level of mentoring reduced to quarterly. Nonetheless Table 4 demonstrates that sites’ performance can rapidly deteriorate—EPAS and some health facility staff reported that deterioration in performance is often a result of personnel changes at the facility: at small facilities, a fall in performance can occur when the PMTCT-trained nurse goes on leave; staff turnover at larger facilities can cause a fall in performance if the new staff are not PMTCT-trained. Table 4, shows that all health facilities have initially improved performance and achieved very high performance during the period that they are receiving monthly mentoring and TA visits. However performance at 50 facilities has later fallen to medium performance and 39 fallen to low/very low performance. The trend is that performance again improves when EPAS increases the mentoring visits to monthly and provides additional TA for example on-site training. Currently in Table 4, 37% of facilities show very high performance; 20% high performance; 23% medium performance; 19% low performance and 3% very low performance. Two health centers and 22 clinics are “graduated” to quarterly mentoring by EPAS criteria. EPAS reported to the evaluation that it does not graduate high volume sites, PHUs and Maternity Units. They are all visited for mentoring and TA monthly, irrespective of their site performance. The health workforce: The enhanced PMTCT knowledge and skills of health workers throughout Swaziland are very likely to be sustained beyond the EPAS intervention, while the health workers are retained in health sector employment at facilities delivering PMTCT services. HMIS tools and health worker job aids: EGPAF reported that EPAS has provided resources to print HMIS tools—the health facility registers—and other health worker job aids including protocols and guidelines. These additional resources will cease at the end of the EPAS project although the HMIS tools—the health facility registers—will still be needed. The job aids will have utility after the end of EPAS and could be continued to be used if the MOH includes printing of tools and job aids, and laminating where needed, for resupplies in its future years’ budgets. The evaluation noted that very many of the MOH reports it reviewed had been printed by donors, the World Bank and UN organizations. 43 The EPAS community programs: The community programs implemented by EPAS, are not institutionalized and interventions implemented by AMICAALL and Lutsango Lwakangwane cannot continue without donor funding. It is possible that the interest that EPAS and partners AMICAALL and Lutsango have generated in PMTCT and the awareness of the socio-cultural and gender barriers to use of PMTCT services and behaviors could be sustained by mass media [programs on Swazi television in urban and peri-urban areas or radio in rural areas.] However, the person-to￾person “sensitization” in the EPAS events may not have been intensive enough. Stepping Stones is a recognized approach to working with communities on socio-cultural and gender issues that has been evaluated and shown to lead to lasting changes at community level when done well.50 However, Stepping Stones is a program of 13 three hour sessions including drama and role play, with three facilitated peer group discussions in each session and critical reflection. This is very much more intensive than the one off events that EPAS organizes. QUESTION 4. HOW EPAS HAS STRENGTHENED MNCH SERVICES MORE BROADLY Please also see Institutional Strengthening, The MNCH Platform on page 35 above, that specifically addresses how EPAS has strengthened MNCH services. EGPAF reported that EPAS training and mentorship has addressed knowledge, skills and approaches to client care that have wider resonance in the delivery of all MNCH services. Health workers report that they have received EPAS training and mentorship in family planning, dual protection, exclusive breastfeeding, and complementary feeding, as well as in compassionate care for clients—topics that benefit both PMTCT postnatally and wider MNCH clients. Many of the job aids printed [and some laminated] seen by the evaluation supported wider MNCH services. Improvements in client flows at health facilities that EGPAF reported it had initiated, benefit all MNCH clients. EGPAF also reported EPAS has provided training to Maternity Unit staff in reducing PMTCT in maternity care, and use of the partogram during labor—knowledge and skills that benefit all birthing mothers and infants born in the 11 high volume maternity units it supports. EGPAF reported it has donated small equipment to Maternity Units and PHUs that benefit PMTCT and wider MNCH clients. The evaluation identified the remaining need for strengthening of the platform—beyond use of the partogram and teaching midwives about PMTCT—in Maternity Units. The evaluation observed that hospital Maternity Units visited are able to provide emergency caesarian section and blood transfusion when needed, although the operating theatre in the health center Maternity Unit visited was not operational. However, the evaluation also observed widespread lack of respect for birthing women’s privacy and dignity: birthing women were naked without covers in all the Hospital Maternity Units and in two Units women were walked naked from the labor ward to the delivery room. EPAS mentoring in Maternity Units had not identified the lack of compassionate care for birthing women. QUESTION 5. THE REMAINING CHALLENGES TO IMPROVING PMTCT OUTCOMES IN 50 Jewkes, Rachel et al (2007) Evaluation of Stepping Stones: A gender transformative HIV prevention intervention. The Medical Research Council, Pretoria, South Africa 44 SWAZILAND The challenges remaining to improving PMTCT outcomes include: (1) Sustaining the standards of integrated PMTCT/MNCH clinical service delivery as personnel change at health facilities, through onsite training, and regular, supportive supervision and quality mentoring by RHMTs to continue the “well performing health workforce” (see page 30 above) and the strengthened MNCH platform (see page 35 above) after the end of EPAS. (2) Lack of point of service CD4 count analysis laboratory equipment (see page 20, above) although this challenge will diminish as Option B+ is rolled out because it doesn’t depend on the CD4 count for initiating ART. (3) The need to address health care delivery around birthing to ensure respectful clinical care in labor and delivery wards, with women’s rights to privacy and dignity respected—so that health facility delivery rates can be further increased (see pages 22 above). (4) Implementation of effective strategies for HIV positive mothers to accept ART, and to retain mother-baby pairs in follow up, until after cessation of breast-feeding. (5) Effective interventions to reduce socio-cultural and gender barriers to uptake of ART by HIV positive pregnant women and to uptake of all PMTCT services after the 6-8 week post-delivery checkup (see Community Linkages page 35 above.) Beliefs and practices around pregnancy and lactation—particularly with the erosion of traditional prohibition of sexual intercourse while breast feeding—are also a challenge to implementation of PMTCT, reported by PEPFAR partners. Gender issues, reported to the evaluation by m2m, impact on health seeking behavior, uptake of PMTCT services and continued use of PMTCT services and behavior: i. women do not have autonomy in decision-making for themselves and their infants and do not control family resources; ii. men have fewer contacts with the health care system and are less well informed regarding HIV and PMTCT; iii. MNCH clinics are not “man friendly; and iv. women fear being blamed and abandoned if they disclose their status as men/in-laws often assume the woman brought HIV infection into the family if she tests first and is found to be HIV positive. (4.1) Male involvement in PMTCT is reported to be increasing but still lags behind the level needed to achieve universal uptake of PMTCT services beyond the 6-8 week postnatal visit (see male involvement, page 36 above). One respondent stated “Male involvement is more than permission and bus fare to go to the MNCH clinic.” (4.2) AIDS stigma—particularly self-stigma by positive women and stigma within the extended family—remains a challenge to implementing PMTCT according to many nurse respondents, MMs and EPAS partners working in the community. QUESTION 6. THE EFFECTIVENESS OF EPAS’S APPROACH IN PROVIDING FACILITY LEVEL SUPPORT EGPAF staff reported that the EPAS approach to facility level support has been to “work within” the health system and to avoid setting up parallel support systems. That approach to development assistance is both efficient and best practice. Co-locating EPAS Program Coordinators—now designated PMTCT coordinators—with RHMTs has leant itself to close working relations and 45 traveling together on supervision/ mentoring visits to health facilities. The availability of EPAS transportation to the RHMTs has relieved a critical shortage on the government side and enabled RHMT support to health facilities as well as EPAS mentoring and HMIS related site visits. The combination of initial offsite training with follow up onsite refresher training, along with regular mentoring has demonstrated effectiveness in supporting integration of PMTCT/MNCH service delivery. The evaluation observed that clinical standards in PHUs, health centers and clinics are now good, despite the services often being delivered from very congested and cramped premises. The evaluation observed that health workers are also enabled in their service delivery by EPAS supplied job aids and small equipment. In terms of effectiveness of supporting improved PMTCT clinical service delivery, EPAS approach with its PMTCT mentors working separately from RHMT MNCH mentors, has been effective but is becoming dated. Now that PMTCT services are integrated at facility level with the standard package of MNCH services including PMTCT during ANC, PNC and in child welfare clinics, the MOH reported that continued vertical support to PMTCT rather than integrated support to comprehensive MNCH services might discourage “joined up thinking” in service delivery. To ensure that all pregnant women and all infants receive the best primary care, the MOH stated that service providers need to understand and practice comprehensive integrated MNCH service delivery. This might now be best supported by integrated PMTCT/MNCH mentoring focusing primarily on clinic care skills, and related clinical record keeping. That would be emphasizing the components of clinical care that have to be delivered rather than emphasizing the components of clinical care that have to be reported. In relation to the HMIS and reporting, there is a disconnect between the support the EPAS team state that they are giving and the data quality of the reporting by health facilities. The key to the disconnect might be in the oft repeated “EGPAF staff come and point out our errors”. Many of the health facilities visited are very busy, with high client to staff ratios. There is a huge burden in collecting data against very many variables, and reporting to the HMIS. Some of the registers—for example the family planning register—are frankly not user friendly. The oversight is perceived by health facility staff as chastising rather than helping them or rewarding them with knowledge that their performance has improved. Service delivery staff need to understand why they are collecting so much data and how it benefits staff who are burdened with reporting, so that they have a stake in reported data quality. CONCLUSIONS 1. To what extent have the project's four main objectives been achieved; what have been the successes, failures and gaps in EPGAF's approach? A) Expanded PMTCT services EPAS has provided TA to the MOH from 2010 over a period when there has been a modest (8%) expansion in the number of sites providing PMTCT services and much larger increases in uptake in PMTCT services provided by the facilities. The uptake of services at EPAS supported health facilities is greatest for pregnant women being HIV tested and receiving their results in ANC (98%); for exposed infants receiving nevirapine in maternity units (92%) and for exposed infants receiving DBS for PCR at 6-8 weeks (97%) 46 UNAIDS has recently defined “universal access” for exposed infants as 90% of exposed infants receiving DBS/PCR at 6-8 weeks; 90% of infants positive on DBS/PCR receiving ART; and 90% of DBS/PCR positive infants on ART achieving viral suppression.51 Thus 90% might be considered as the standard for universal access to PMTCT services in general. The number of pregnant women being seen at health facilities that EPAS is not yet supporting is very low: in 2013, 27448 pregnant women were seen for the first time in EPAS supported facilities and only 2173 seen for the first time at facilities EPAS is not yet supporting. The rate for pregnant women being HIV tested and receiving their results in ANC in the 18 facilities providing PMTCT services that EPAS does not yet support is 94% (1413/1504). Thus the rate for pregnant women being HIV tested and receiving their results in ANC for all Swazi health facilities providing PMTCT is (20757+1413)/(21075+1504) or 98%. In Swaziland it is currently estimated that 98% of women attend ANC at least once and so 96% of pregnant women [98% x 98%] are being HIV tested and receiving their results in ANC. Thus it can be estimated that EPAS has supported the MOH to attain universal access for pregnant women being tested and receiving their results in ANC. Similarly, reviewing the data presented in Figure 7 on page 25, the rate for exposed infants having DBS for PCR at 6-8 weeks in all health facilities providing PMTCT services is (725+8462)/(878+8757) or 95%. And so 93% of HIV exposed infants [98% x 95%] in Swaziland have DBS taken for PCR at 6-8 weeks of age. Thus it can be estimated that EPAS has supported the MOH to attain universal access for exposed infants having DBS taken for PCR at 6-8 weeks of age. There have also been large increases in numbers of HIV positive pregnant women commenced on CTX, AZT and ART although the proportions of such women do not approach universal access for communities served by EPAS-supported facilities as yet. Nonetheless, EPAS has supported expansion of these PMTCT services and thus EPAS has achieved its objective of expanded PMTCT services. B) Comprehensive & quality PMTCT services integrated into ANC, labor & delivery and post natal clinics, including ANC & PNC visits, HTC and ARV uptake, adherence, facility deliveries, HIV transmission rates at 6-8 weeks, etc EPAS has in general achieved its objective of good quality and comprehensive PMTCT services, integrated into the ANC, PNC and child welfare services of the MNCH platform. The exception being in Maternity Units where the lack of a definition of quality in PMTCT care and lack of service delivery quality standards—with EPAS focused more narrowly on performance against the National Framework for Eliminating MTCT indicators—enabled EPAS to miss the unacceptable standard of care in labor and delivery wards. This remains to be addressed in the 5th year of implementation. The persistent challenges in obtaining CD4 counts because of lack of point of care CD4 equipment or failure of point of care CD4 equipment underscore the need to the roll out Option B+ because it does not require a CD4 count to initiate ART. It also indicates a need for greater emphasis on regular maintenance and servicing of the CD4 equipment at referral hospitals that will continue to 51 Personal communication to the EPAS evaluation team by UNAIDS. June 16, 2014 47 need CD4 counts for monitoring ART. EPAS has achieved very high rates for DBS taken from exposed infants for PCR at 6-8 weeks; the proportion of infants tested by PCR at 6-8 weeks that were positive was 2.85% at EPAS supported facilities in 2013: a commendable result. A small number of women attend 18 facilities that provide PMTCT services that are not yet supported by EPAS. The overall rate for exposed infants testing positive on DBS for PCR at 6-8 weeks (24+241)/(725+8462) at all health facilities in Swaziland providing PMTCT services was 2.89% in 2013. The remaining barriers to uptake of PMTCT services after the infant is 6-8 weeks old are mainly demand side—sociocultural and gender issues, complicated by women in the industrial sector returning to work at only 3-4 weeks after delivery. EPAS has introduced the innovation of letters of invitation from the health care worker to male partners of women attending ANC, who can take it home to their partner, in an attempt to get more male involvement in PMTCT. C) National PMTCT system strengthened EPAS has strengthened the national PMTCT system at national—policy level—as well as at RHMT and service delivery level in health facilities. Although EPAS has also provided considerable assistance to the HMIS/M&E system at all three levels, there is a weakness in the quality of the reporting by health facilities identified by the evaluation. EPAS still needs to work with the RHMTs to address this in year 5 of implementation. It is important to analyze performance data with the staff who generate the data and disseminate the results of the analysis to all the health facility staff for them to understand the significance of the data they are reporting and thus have a stake in data quality. However, an issue is the burden of reporting the large number of indicators that have data collected routinely. The burden of reporting might be reduced by prioritizing which data is essential to be included in the monthly reports and which items could be collected through periodic health facility surveys and which indicators might be better calculated by periodic population-based survey. Improvements in the registers—particularly in the user friendliness of the FP register—might also reduce the burden of reporting on health care providers. Reduced burden of reporting—when combined with supportive supervision and mentoring that acknowledges and rewards (praises) service providers whose performance has improved—is likely to improve the accuracy of the monthly reporting by health facilities. EPAS has worked with the MOH to strengthen the MNCH platform in PHUs, health centers and clinics as far as it is able within its manageable interests and within the limitation on the use of PEPFAR funding imposed by the U.S. Congress. D) Protocols, guidelines, and job aids developed with MOH EPAS has contributed TA valued by the MOH to the development and printing of protocols, guidelines, and job aids developed, working with MOH at policy level, achieving its objectives in this area. EPAS has also produced individual job aids for health facilities—printed and often laminated, but varying by health facility. This perhaps indicates that these health facility level job aids are tailored to specific needs and requests from individual health facilities. 2. What were the successes, challenges and gaps in EPAS’s community approach? There is great scope for addressing male involvement, post-delivery PMTCT behaviors and service uptake, and AIDS stigma within the family using the EPAS approach to community mobilization. However, the community approach is not evidence based and has not been established as a pilot 48 project with monitoring of the outcomes to demonstrate effectiveness. 3. How sustainable are the gains made under EPAS? How many facilities have been graduated? How many have improved, and to what extent? What factors contributed to sustainable improvement? Implementation through the existing health care delivery system in support of the National Strategic Framework has institutionalized the gains. Health worker PMTCT knowledge and skills are very likely to continue beyond EPAS while the trained and mentored staff remain within the health sector delivering PMTCT services. The mentor mother program is not institutionalized and requires donor funding to continue. Retention of the EPAS PMTCT Coordinators who are currently seconded to the RHMTs is likely to be a challenge. These positions may not be appropriate within the emerging MOH policies and plans for integrated mentorship. Performance increased at all health facilities with EPAS monthly mentoring visits and additional technical support, for example onsite training, but at some facilities performance later rapidly fall. As most falls in performance are related to staff leave or turnover, with incoming staff not trained in PMTCT, RHMTs might be able to anticipate the need for increased mentoring and TA when they know of staff changes at health facilities, before there are falls in performance. Additional transport that EPAS provides to support the supervision and mentoring of health facilities and the redistribution of drugs at regional level will cease with EPAS. Thus the frequency of mentoring and supervision may fall and the RHMTs’ ability to redistribute drugs between facilities might be reduced unless EPAS negotiates an exit strategy with the MOH that addresses transportation. That EPAS partners have raised the PMTCT awareness of formal community leaders—through training in PMTCT issues—may ensure that HIV and PMTCT issues remain on the agenda at community meetings. However the community leaders have been paid stipends by EPAS partners and may no longer be interested in promoting PMTCT issues after their stipends ceased. Without systematic and iterative raising of socio-cultural and gender barriers to use of PMTCT behaviors and services—any changes in gender norms and health seeking behaviors stimulated by EPAS partners are unlikely to be sustained beyond EPAS. 4. How has EPAS strengthened MNCH services more broadly? For example, in the areas of family planning, prevention and management of obstetrical complications, newborn health, infant feeding and reducing post-natal transmission of HIV? What are the remaining gaps? While EPAS has in general strengthened the MNCH services as far as possible within its manageable interest and within the limits on use of PEPFAR funding imposed by the U.S. Congress, there remains a need for EPAS to focus on addressing quality of care in Maternity Unit labor and delivery wards in the last year of implementation 5. What are the remaining challenges to improving PMTCT outcomes in Swaziland? While the RHMTs must maintain clinical standards at health facilities and support the continued delivery of the integrated package of MNCH services, the RHMTs and the MOH are not well placed to address the remaining demand side challenges to increasing uptake of services and to rolling out Option B+. The important barriers to women accepting ARVs that continue after delivery; to 49 remaining in follow up after the 6-8 week post-natal visit until cessation of breastfeeding and retesting of the exposed infant; to adopting safer sex behaviors in pregnancy and during lactation; to exclusive breast feeding for the first 6 months of life; and accepting ART for infected infants are sociocultural and gender related. Although MMs are peer counseling positive mothers on how they might address these issues, and EPAS has tried to increase male involvement by use of invitation letters, these issues are not generally amenable to resolution from within PMTCT/MNCH service delivery at the health facility, the sphere of influence of the RHMTS and MOH. Male partners need to be actively involved in PMTCT decision making and behavior change to reduce the number of women who seroconvert in pregnancy and during lactation. Partners must also support mothers to exclusively breastfeed their infants for the first 6 months of life in the face of traditional feeding practices encouraged by grandmothers and others in the family who believe that infants need early introduction of pap and other “real food” to grow well. Male involvement requires improved communication between the male partner and the mother, and increased empowerment of women in decision making about their own and their infants’ health. These socio-cultural and gender barriers to improving PMTCT outcomes remain challenges to be effectively addressed. 6. How effective has been EPAS’s approach in providing facility level support? EPAS approach to providing facility level support through training, in service training and mentoring was effective in integrating PMTCT services in to the MNCH platform at PHUs, health centers and clinics. However there is evidence there may have been too great an emphasis on performance against specific PMTCT indicators to the exclusion of clinical care mentoring with a wider focus on delivery of quality PMTCT services. The narrower focus on performance indicators, after initial training in PMTCT and use of the partogram, let EPAS mentors miss the unacceptable quality of basic midwifery care to birthing mothers in Maternity Units. The focus on improving performance indicators also seems to have somewhat lost its “mentoring” as distinct from “policing” approach. This is evidenced by health workers stating that “EPAS staff come to correct our mistakes” and weaknesses in the accuracy of the health facility monthly reports. LESSONS LEARNT 1. The EPAS model of working from inside the health system, with the RHMTs to deliver a combination of training—initial off site and then on site refresher training—combined with mentorship and supplies of related job aids and small clinical equipment, has been effective in integrating PMTCT into the delivery of an integrated package of MNCH services in Swaziland. The resulting quality of PMTCT services delivered in the PHUs, health centers and clinics is generally good. Yet, the EPAS focus on mentoring around improved performance against PMTCT indicators has excluded consideration of wider PMTCT service quality issues. The tone of the support may have moved away from clinical mentoring of health workers for improvement in the clinical care they provide, to policing of health facility performance detailed in the monthly reports. The lesson learnt is that the drive for better performance against indicators can be detrimental to wider quality of care (and to the quality of health facility HMIS reports) if there isn’t also a push to address wider clinical care quality issues and to use performance data to recognize and praise those who are doing well, as an incentive for them to do better. 50 2. The EPAS project has not been implementing an exit strategy negotiated with the MOH from the project inception. There might have been changes in MOH policy over the years since 2010, but revising an exit strategy annually with the MOH is preferable to introducing an exit strategy late in project implementation. Looking to the MOH to absorb personnel that it had not planned to absorb from the beginning of the project, and that are not in line with MOH determined service staffing needs, is unlikely to be a successful strategy for sustaining the gains that EPAS has achieved over the life of the project. The lesson learnt is that projects should have a clearly stated exit strategy negotiated and agreed with the partner lead ministry from inception so that all parties are working to the mutual goal of the lead ministry sustaining project gains after planned withdrawal of project support. RECOMMENDATIONS 1. For the remaining term of the EPAS project, EPAS should take a fresh approach to clinical mentoring, and to its approach to reviewing performance reports and use of HMIS data with health facility staff. 1.1 EPAS should review and negotiate with the MOH the essential data required to be collected and reported monthly, and data that might be better collected through periodic health facility surveys, or not collected if it is not a vital component of monitoring and improving service delivery performance. 1.2 EPAS should work closely with RHMTs on support to health facilities and health facility reporting in the last year. The approach should be supportive of health delivery staff and their needs; recognize and validate staff who are doing well; and not be policing of the monthly reports and the HMIS. 1.3 EPAS should only conduct joint mentoring and supervision visits with the RHMTs; at health facilities, EPAS should coach and support the RHMT staff to take the lead on clinical care mentoring, and supportive supervision of the compilation of the health facility monthly reports to the HMIS. 2. There remains a huge need for the MOH to improve the quality of care of laboring and birthing women in Maternity Units. Midwifery care must be respectful of women’s rights to be treated with dignity and have privacy. Therefore: 2.1 In line with good practice, EPAS should work with the MOH to define service delivery quality standards and publish the standards at health care delivery points for service providers and clients to see. 2.2 EPAS should consider implementing leadership and management strengthening in the coming year, to address the serious gaps in service delivery quality in Midwifery Units, and also to enhance the leadership and management of PHUs, health centers and clinics. Other aspects of good leadership and management that EPAS should address include: clinical service delivery with a stronger client focus rather than the current health provider focus; acknowledgement and validating clinical providers who are doing well and who are at of risk becoming dispirited and less interested in good performance. 3. EPAS partners should document the outcomes of their community activities in the last year of implementation to demonstrate that their community approaches are making a difference. 51 3.1 If the community activities’ objective is to increase male involvement [for example] EPAS should work with the partners to define male involvement and determine how increased male involvement might be measured. 3.2 If the community activities objective is to increase the uptake of PMTCT services, EPAS should work with the partners to focus activities where increased uptake is needed—PMTCT after the 6-8 post natal attendance—and on the behaviors needed such as exclusive breast feeding for 6 months and use of condoms during pregnancy and lactation. 3.3 EPAS should monitor and evaluate community activities at outcome level rather than only monitoring activity/process. 4. EPAS senior management should review and formalize the EPAS exit plan with the MOH, identifying (1) the gains that need sustaining beyond the end of project, (2) what the MOH will be able to provide to sustain the gains; and (3) how EPAS implementation in the final year will move towards sustaining its gains. 4.1 EPAS should write the exit plan agreed with the MOH—including handover details—in a formal document and use the plan to guide EPAS final year of implementation. 4.2 In the final year, EPAS staff should support MOH cadres at all levels [national, regional and facility] who will continue beyond the project, to assume their full responsibilities sustaining the gains. Thus EPAS mentors should mentor RHMT MNCH mentors rather than EPAS mentors directly mentor health facility staff. 5. Health development implementing partners should coordinate their support and visits to health facilities to minimize the burden on health facility staff, to maximize the potential for complementarity and to reduce duplication of effort. 5.1 EPAS staff should continue to its approach of working from within, to support the RHMT leadership in coordination of health development implementing partners. 5.2 EPAS and the RHMTs should work together to develop a strategy for retaining mother-infant pairs in follow up until after cessation of breastfeeding, and increasing retesting of exposed infants 6. Although m2m is no longer a subpartner of EGPAF and thus not within the purview of this evaluation, the evaluation identified the value of the m2m program as a vital contribution to the quality of PMTCT care at the facilities visited. There is need for expansion of the m2m program with continuation after the EPAS project. 6.1 USAID might explore with the MOH, the potential for mentor mother positions being formalized, accredited and supervised by registered nurses in the facilities. 6.2 Support to the m2m program could then be tailored to the MOH ability to absorb the positions as USAID support is phased out. 7. USAID should focus any follow on activity to EPAS on addressing demand side barriers to uptake of PMTCT services and adoption of new behaviors, and to increase retention of mother-baby pairs in PMTCT services in the child welfare clinics until after the infants cease breast feeding. 7.1 USAID should design activities to address sociocultural and gender barriers so that they are institutionalized (not stand alone), sustained and brought to scale throughout Swaziland until cultural norms related to PMTCT behaviors have changed. 52 7.2 Areas that USAID should focus particular attention in a follow on activity include: i. women’s uptake of ART [which is increasingly important as Option B+ is rolled out]; ii. women and their infants continuing in follow up for PMTCT after the 6-8 week post natal checkup, during breastfeeding; iii. women and their partners practicing safer sex during pregnancy and lactation; iv. exclusive breastfeeding infants for 6 months; v. meaningful involvement of men in PMTCT. I ANNEXES ANNEX I: EVALUATION SCOPE OF WORK II ANNEX II: EPAS EVALUATION FRAMEWORK XIII ANNEX III: EVALUATION METHODOLOGY XXII ANNEX IV: DATA COLLECTION INSTRUMENTS XXVII ANNEX V: SOURCES OF INFORMATION XXXV 1. PERSONS THE EVALUATION TEAM MET WITH XXXV 2. DOCUMENTS AND OTHER RESOURCES ACCESSE XXX9III ANNEX VI: DISCLOSURE OF ANY CONFLICTS OF INTEREST XLVI II ANNEX I: EVALUATION SCOPE OF WORK Evaluation Scope of Work for Eliminating Pediatric Aids in Swaziland (EPAS) Project PROJECT TO BE EVALUATED Project name: Eliminating Pediatric Aids in Swaziland (EPAS) Cooperative Agreement No.: 674-A-00-11-00009-00 Project Dates: October 2010 - September 2015 Agreement Value: $16,495,000 Implementing Organizations: Elizabeth Glaser Pediatric AIDS Foundation (EGPAF) 1. Evaluation Objective Eliminating Pediatric AIDS in Swaziland (EPAS) is the largest PMTCT program in Swaziland supporting the Ministry of Health (MoH) to achieve its goal of eliminating pediatric HIV/AIDS and keeping mothers alive. PEPFAR countries are encouraged to invest in building evidence to assess the effectiveness and impact of their HIV/AIDS programs. It is the goal of this evaluation to assess EPAS’s effectiveness in supporting and strengthening the technical capacity for a quality, integrated and comprehensive PMTCT program in Swaziland. The purpose of this assignment is to conduct an external performance evaluation of the Swaziland USAID/EPAS project. The evaluation should (1) assess the quality of implementation, (2) document lessons learned, (3) explore challenges and accomplishments, and (5) provide strategic guidance for the remaining years of the project and any follow-on activities. The evaluation is expected to provide results not only on the likely effectiveness of the program and whether it met its intended objectives, but the evaluation should also provide detailed input into which elements of the program worked and which did not. These results will provide an overall assessment of the model and will lead to several decisions, including whether the model should be scaled up or which elements of the model should be re-aligned or strengthened for potential follow-on activities. The audience of the evaluation report will be EGPAF as implementing partner, Swaziland Ministry of Health, PEPFAR/Swaziland, as well as USAID/Southern Africa and the Global Health office in USAID/Washington. 2. Background of project III EPAS is a five-year project funded by USAID/Swaziland and is implemented through Elizabeth Glaser Pediatric AIDS Foundation (EGPAF). The project was awarded in October 2010 and ends in September 2015. EPAS is a follow-on project to the global award, Call to Action, which from 2003 supported the MoH to increase access to effective PMTCT services to HIV infected pregnant women in Swaziland through a direct service delivery model. EPAS was built on the successes of the global Call to Action program and sought to further expand service delivery to reach more women through increasing access to services; sustaining quality, comprehensive and integrated services; and strengthening the health system for sustainability, as espoused in PEPFAR 2. The overall goal of this award is to eliminate pediatric AIDS in Swaziland. In order to achieve this goal, the project is expected to achieve the following objectives: 1. Objective 1: To achieve universal access to PMTCT through increasing geographical coverage of services and addressing social and gender norms that create barriers to service uptake and retention in care 2. Objective 2: To provide and sustain quality, comprehensive and integrated PMTCT services through supporting clinical mentoring and supportive supervision of health care workers at health facilities across the country 3. Objective 3: To strengthen the national health systems in line with the Ministry of Health plans for PMTCT through technical assistance and capacity building to ensure sustainability 4. Objective 4: To support the MoH develop and review PMTCT policies, protocols and guidelines on a regular basis In 2012, EGPAF received additional funds to accelerate EPAS activities in Swaziland. These resources were over and above the amount that was awarded to EGPAF in the original award. The additional resources were used to further expand EPAS efforts and the extra activities are commonly referred to as EPAS+. The additional activities include, (a) strengthening voluntary family planning integration at PMTCT and ART sites, (b) increasing access to and utilization of MNCH services, (c) improving infant feeding practices, (d) improving quality of care during labor and delivery, and (e) Strengthening community participation in PMTCT and MNCH. EPAS currently supports 144 facilities throughout Swaziland’s four regions (Hhohho – 37, Shiselweni – 27, Manzini – 41, Lubombo - 39). EPAS supports the regional health teams through the Regional PMTCT coordinators that are stationed in each region. However, there are discussions to transition these positions to government. In addition, EPAS also supports the national program through seconding an MNCH advisor and a training officer to the SRHU program. In an effort to maximize efficiencies and effectiveness, EGPAF has been working with a number of sub-partners to implement the EPAS project. While some of the sub-partners have just recently joined EPAS under the acceleration plan, one partner was recently graduated to become a direct recipient of USAID funding (see table below for sub-partner details) IV It is estimated that there are between 33,000 and 35,000 deliveries in Swaziland each year. In FY13, EGPAF supported 27,051 pregnant women to attend at least one ANC visit. 20,664 of the 20,914 pregnant women eligible for HIV testing were tested and received their results, representing an HTC uptake rate of 99.2%. 95% of the almost 10,000 HIV positive pregnant women that were identified in the reporting period were started on antiretrovirals to reduce the risk of mother-to-child transmission of HIV. 97.2% of exposed babies had Dried Blood Spot (DBS) done for Early Infant Diagnosis (EID) at 6-8 weeks after delivery The final evaluation questions to be addressed are: 1. To what extent have the project's four main objectives been achieved; what have been the successes, failures and gaps in EPGAF's approach? a) Expanded PMTCT services b) Comprehensive and quality PMTCT services integrated into ANC, labor & delivery and post-natal clinics, including ANC & PNC visits, HTC and ARV uptake, adherence, facility deliveries, HIV transmission rates at 6-8 weeks, etc. c) National PMTCT system strengthened d) Protocols, guidelines, and job aids developed with MOH 2. What were the successes, challenges and gaps in EPAS’s community approach? 3. How sustainable are the gains made under EPAS? How many facilities have been graduated? How many have improved, and to what extent? What factors contributed to sustainable improvement? 4. How has EPAS strengthened MNCH services more broadly? For example, in the areas of family planning, prevention and management of obstetrical complications, newborn health, infant feeding and reducing post-natal transmission of HIV? What are the remaining gaps? 5. What are the remaining challenges to improving PMTCT outcomes in Swaziland? 6. How effective has been EPAS’s approach in providing facility level support? The Table below provides a summary of EPAS sub-partners and the technical areas and geographic focus. Sub-partner Focus Areas Coverage Alliance of Mayors' Initiative for Community Action on AIDS at the Local Level (AMICAALL) Increase PMTCT demand and linkages between peri￾urban communities and facilities 12 towns in Hhohho, Manzini, Lubombo, Shiselweni regions mothers2mothers (m2m) (graduated to become a direct USAID partner in January 2013) Provides psychosocial support and education to newly diagnosed HIV positive pregnant women and mothers Hhohho, Manzini, Lubombo, Shiselweni Lutsango Lwakangwane (EPAS+ sub partner) Support the dissemination of MNCH/HIV messages to women in rural areas V Swaziland Infant Nutrition Action Network (SINAN) (EPAS+ sub partner) A local NGO providing training support on infant feeding and nutrition to health care workers Hhohho, Manzini, Lubombo, Shiselweni 4. Methodology USAID is interested in evaluation proposals that use objective evidence to answer the evaluation questions listed above. USAID desires an evaluation that provides an objective assessment of the quality of program implementation using a non-experimental design approach, with extensive use of both quantitative and qualitative methods. These methods will include review of project documents, interviews, and extensive use of routinely collected program data. Prior to commencing fieldwork, the evaluators will review and analyze information from key documents to assess trends such as service delivery and facility coverage over time. The evaluators will have access to routinely collected program and Health Management Information System (HMIS) data. However, no patient level data is expected to be collected. Upon award, but before fieldwork is conducted, the contractor will submit a detailed evaluation design, methodology and implementation plan for review and approval by USAID. It is expected that the evaluators will discuss the relative strengths and limitations of the methodology proposed within the proposal. The methodology should take into account – and independently assess, where possible – the quality of the routine monitoring data that is collected at selected facilities and use appropriate tools to verify EPAS site performance assessments scores at select facilities. Additionally, the evaluators should discuss data disaggregation (by gender and other categories) and gender considerations in the evaluation. Before data collection, the contractor in coordination with USAID will finalize the data analysis methods as part of the methodology plan. The primary audience of this evaluation is USAID, other USG agencies, EPAS partners, and MOH. Sampling Given the largely qualitative nature of the primary data to be collected, it will be important for evaluators to propose appropriate sampling methods to minimize bias. This will ensure balanced responses and minimize bias Data Analysis The evaluators will analyze and present quantitative data using appropriate methods. Qualitative data should be analyzed according to key themes that emerge from the interviews. VI Clean and coded qualitative and quantitative datasets will need to be shared with USAID upon project completion. Evaluators are requested to complete the evaluation matrix below based on their proposed evaluation approach and data collection methods. Before data collection, the evaluators, in coordination with USAID will finalize the matrix and include it in the overall evaluation design and methodology plan. Evaluation Questions What evidence would you look for? Data Source(s) and Collection Methods Data Analysis Methods 1. To what extent have the project's four main objectives been achieved; what have been the successes, failures and gaps in EPGAF's approach in: (a) Expanding PMTCT services (b) Comprehensive and quality PMTCT services integrated into ANC, labor & delivery and post-natal clinics, including ANC & PNC visits, HTC and ARV uptake, adherence, facility deliveries, HIV transmission rates at 6-8 weeks, etc. (c) National PMTCT system strengthening (d) Supporting MoH in the development of protocols, guidelines, and job aids developed with MOH 2. What were the successes, challenges and gaps in EPAS’s community approach? 3. How sustainable are the gains made under EPAS? How many facilities have been graduated? How many have improved, and to what extent? What factors contributed to sustainable improvement? 4. How has EPAS strengthened MNCH services more broadly? For example, in the areas of family planning, prevention and management of obstetrical complications, new-born health, infant feeding and reducing post-natal transmission of HIV? What are the remaining gaps? 5. What are the remaining challenges to improving PMTCT outcomes in Swaziland? 6. How effective has been EPAS’s approach in providing facility level support? VII The following documents will be available for review o EPAS Cooperative Agreement and the associated modifications o EPAS/EPAS+ work plans o Acceleration Plan quarterly reports o USAID DQA Reports o Site Visits Reports o EPAS progress reports o EPAS baseline assessments for select facilities o Swaziland HMIS data o HIV Sentinel Surveillance reports o PMTCT outcome evaluation study o Annual MOH PMTCT reports o 2007 Demographic and Health Survey (DHS) (reports and datasets available online at Measure DHS) o 2010 Multi-Indicator Cluster Survey (MICS) o 2011 Swaziland HIV Incidence Measurement Survey (SHIMS) o Other EGPAF studies, including the “Barriers to ART uptake study”, etc. o Sub partner documents  Program Descriptions  Project Agreements  Quarterly, semi-annual and annual reports  Annual work plans and budgets • Additional data for evaluation to potentially collect o Key informant interviews with program staff, health workers and other stakeholders (including Ministry of Health, NERCHA, etc.) o Focus Group Discussions (FGD) with clients and/or communities (Qualitative) Evaluation Limitations The evaluators should discuss potential limitations to the proposed methods and their plans for mitigating these limitations. 5. Team Composition The overall evaluation team should have: a. Extensive evaluation experience with substantial maternal and child health experience in Africa. b. A comprehensive understanding of PMTCT-related programming. c. Experience working with donor-funded development programs, preferably USAID and PEPFAR. d. Experience in qualitative and quantitative data collection methods e. Experience with strengthening Ministry of Health systems f. In-depth knowledge and understanding of the Swaziland HIV/AIDS epidemic and response, as well as hands on experience working on PMTCT/MNCH programs in country. VIII Core team members should include: 1. Team Leader/Senior Evaluation Specialist should have a post graduate degree in public health or social sciences. S/he should have extensive experience in conducting mixed methods (combining quantitative and qualitative) evaluations in Sub-Saharan Africa focusing on maternal and child health. Excellent oral and written skills in English are required. The Team Leader should also have experience in leading evaluation teams and preparing high quality documents. The Team Leader will take specific responsibility for assessing and analyzing the organization‘s progress towards targets, factors for such performance, benefits/impact of the strategies, and compare with other possible options. S/he will also suggest ways of improving the present performance, if any. S/he will provide leadership for the team, finalize the evaluation design, coordinate activities, arrange periodic meetings, consolidate individual input from team members, and coordinate the process of assembling the final findings and recommendations into a high quality document. S/he will write the final report. S/he will also lead the preparation and presentation of the key evaluation findings and recommendations to the USAID/SD team and other major partners. 2. Senior Technical Advisor for PMTCT / MNCH should have a medical, nursing, public health or related degree. S/he should have several years’ experience with PMTCT / MNCH programs in Sub-Saharan Africa. S/he should be knowledgeable in program assessment and evaluation methodologies. S/he should have extensive experience, and demonstrate state-of- the-art knowledge, in conducting evaluations/assessments of PMTCT / MNCH projects. S/he should have extensive knowledge of the HIV/AIDS epidemic in Swaziland and its response, and experience working on PMTCT/MNCH or other health programs in country is desirable. 3. Quantitative data analyst: should have a post-graduate degree in public health, statistics, epidemiology, economics, or another applicable social science. S/he should have several years’ experience doing advanced data analysis, including using regression models and quasi-experimental methods. S/he should be knowledgeable in program assessment and evaluation methodologies. 4. In-Country Logistics/Administrative Specialist should have several years’ experience coordinating events and/or international travel. S/he should be knowledgeable about traveling throughout Swaziland. S/he will manage all in-country travel, logistics and other duties as assigned by the team leader. USAID recommends that this position be filled by a Swaziland national, if possible. 5. Deliverables and timetable It is estimated that the evaluation activities will take approximately 48 working days. The Offeror is expected to provide in the proposal a detailed timeline for the performance of the evaluation (from the planning work until the submission of the final report). IX 1. Team-planning meeting At least one team planning meeting will be conducted in advance of any international travel. This meeting will allow USAID to present the evaluation team with the purpose, expectations, and agenda of the assignment as well as provide any feedback on the evaluation design and methodology submitted as part of the proposal. 2. Work plan Before international travel is authorized; the team will prepare a detailed work plan that will include the methodologies to be used in the evaluations. The work plan will be submitted to the Contracting Officer’s Representative (COR) at USAID for approval one week after the project is awarded. 3. Pre-fieldwork briefing Discuss program literature review and initial findings from secondary data analysis conducted on key program documents. This briefing is expected to happen before the team travels to Swaziland and shortly after the workplan submission. 4. Initial in-country briefing with USAID The team will meet with the USAID team in Swaziland for an in-brief before commencing fieldwork. 5. Interim briefings, including status reports The team leader will provide weekly status reports to USAID on work plan implementation. 6. Key findings debriefings with USAID and stakeholders The team will present the major findings of the evaluation to USAID, the USAID partners (as appropriate and as defined by USAID) and the Ministry of Health, Swaziland after the completion of the field work and analysis and prior to completion of the draft report. The debriefing will include a discussion of achievements, activities, and recommendations. The evaluation team will consider USAID and stakeholder comments in the draft report accordingly, as appropriate. 7. Draft evaluation report The draft report of the findings and recommendations should be submitted to the USAID COR 10 working days after the field work. One hard copy and one electronic copy will be provided to USAID. 8. Final evaluation report The USAID has 14 working days to review the draft report and provide written comments to the contractor on the draft report. The contractor will have 5 working days to incorporate these comments into a final report. Contractor must provide both an electronic version and 5 hard copies of the final report to USAID. The reporting format is listed in Section 7. The report will be released as a public document on the USAID Development Experience Clearinghouse (DEC) (http://dec.usaid.gov) after the COR provides formal written approval. X 7. Reporting Format and Criteria Format The report format should be restricted to Microsoft products and 12-point type font should be used throughout the body of the report, with page margins 1 inch top/bottom and left/right. The report should not exceed 35 pages, excluding references and annexes. The evaluation report should include the following sections: 1. Executive Summary: summarizes project purpose and background, key evaluation questions, methods, findings, and recommendations. (3-5 pgs.); 2. Table of Contents (1 pg.); 3. Introduction and Background: purpose, audience, and synopsis of task, brief overview of the project, USAID program strategy and activities implemented in response to the problem, brief description of implementing partners (1-2 pages); 4. Methodology—describes evaluation methods, including constraints and gaps (1-2 pg.); 5. Findings/Conclusions/Recommendations—for each objective area; and also include data quality and reporting system that should present verification of spot checks, issues, and outcome (17–20 pgs.); 6. Issues—provide a list of key technical and/or administrative, if any (1–2 pgs.); 7. Future Directions - to inform the design of any new intervention (if appropriate) (2- 3 pgs.); 8. References (including bibliographical documentation, meetings, interviews and focus group discussions); 9. Annexes—annexes that document the evaluation tools, schedules, interview lists, tables, all sources of information, the evaluation statement of work, statements of differences— should be succinct, pertinent and readable. In addition, the evaluators should prepare a power point presentation that summarizes the final evaluation findings/conclusions/recommendations. Final Report Criteria • The evaluation report should represent a thoughtful, well-researched, and well organized effort to objectively evaluate what worked in the project, what did not, and why. • Evaluation reports shall address all evaluation questions included in this Scope of Work. • The evaluation report should include the scope of work (as written in the signed contract) as an annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline need to be agreed upon in writing by the COR. • Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists, and discussion guides will be included in an Annex in the final report. • Evaluation findings will assess outcomes and impacts on males and females. XI • Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). • Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or the compilation of people’s opinions. Findings should be specific, concise, and supported by strong quantitative or qualitative evidence. • Sources of information need to be properly identified and listed in an annex. • Recommendations should be action-oriented, practical, and specific, with defined responsibility for the action. 8. Relationships, Scheduling and Logistics The Contracting Officer will formally appoint a Contracting Officer’s Representative (COR) and an Alternate Contracting Officer’s Representative in a formal letter to the contractor. The Contracting Officer and the COR are the only official representatives of USAID for this contract and are the only ones authorized to provide technical direction to the contractor throughout the evaluation. The contractor is expected to work together with the COR to implement the scope of work. 9. Proposal Requirements Proposals must include the following: 1. Specific methodology and techniques used to gather information and answer the evaluation questions. 2. A complete timeline that illustrates key milestones in the evaluation. The timeline should show a realistic schedule that lists specific activities that will occur each week that the evaluation research is conducted. It should also specify when the final evaluation report will be submitted to USAID. It is anticipated that one contract will be awarded on/about April 21, 2014. The contract start date should be no later than one week after the award of the contract. Offerors should use these dates when developing a timeline. Timelines should include an initial in-briefing with USAID and an oral presentation of the final evaluation (delivered prior to the submission of the written evaluation). The latest possible date for receiving final USAID evaluations proposals is April 7, 2014. Proposals will be evaluated on the basis of the evaluation criteria below. 3. The number of consultants and each consultant’s background. 4. A detailed budget 10. Evaluation Criteria Technical Approach (50 points) Evaluation under this factor will focus on the soundness and innovativeness of the overall technical approach presented in the proposal. The following considerations will be XII evaluated under this factor. These factors will not be scored individually, but are included to provide potential offerors with additional information regarding this evaluation criterion: • Proposed design/methods to answer evaluation questions should demonstrates a clear and complete understanding of the PMTCT program/activities in Swaziland • The offerer should suggest innovative, practical and efficient data collection methods/techniques, balanced with rigor, in order to attain planned results and outputs during the contract period • Execution of required tasks should be clearly defined, feasible, and technically sound Personnel (40 points) The personnel factor evaluates the extent to which the qualifications, skills and experience of proposed personnel meet or exceed those required in Section 5. The evaluation will assess the overall staffing plan and approach, as well as the capabilities of specific personnel. In addition, the offeror should demonstrate the appropriateness of the persons proposed for the positions, including a review of their experience in areas relevant to the successful implementation of the proposed activity, education, other skills and performance history as shown through references and/or other sources. Experience working in Swaziland is desirable. Past Experience (10 points) Past performance information will be used for both the responsibility determination and best value decision. USAID may use performance information obtained from other than the sources identified by the offerer. The following factors will be considered during the past performance evaluation: • Quality of service, including consistency in meeting targets and goals • Timeliness of performance; including adherence to contract schedules, and other time sensitive project conditions Proposed Costs Proposed costs shall be evaluated for cost realism, completeness, reasonableness, allowability, allocability and the competitiveness of the fee proposed. This analysis is intended to determine the degree to which the costs included in the cost/price proposal are fair and reasonable. An overall evaluated price (cost plus fixed fee) will be determined and will be used as part of the tradeoff analysis in determining source selection. XIII ANNEX II: EPAS EVALUATION FRAMEWORK Purpose of the EPAS evaluation, to: Questions that address the evaluation purpose Data Source(s) and Collection Methods Data Analysis Methods (1) assess the quality of implementation (1) What is the quality of services provided from (a) managerial, (b) technical and (c) client perspectives? (1) Key in formant interviews (2) Guided group discussions w beneficiaries & community members (3) Direct observation w checklists (1) transcription of interviews and discussions; analysis of detailed notes taken by discussion observers (2) Content analysis of transcriptions, notes and checklists (2) document lessons learned (1) What have been the implementation successes, and why? (2) What has been less successful, and why? (3) Which are lessons learned are context specific and which are generalizable to other resource poor settings in Africa? (1) document review particularly EGPAF quarterly and annual reports (2) Key in formant interviews (3) Guided group discussions w beneficiaries & community members (4) Evaluation team technical knowledge & resources (1) transcription of interviews and discussions; analysis of detailed notes taken by discussion observers (2) Content analysis of transcriptions, notes and checklists (3) explore challenges and accomplishments, and (1) What have been the main implementation challenges? How were the challenges managed and what were the solutions? Did the lessons learned from managing the challenges avert similar later implementation challenges? (1) Document review particularly EGPAF quarterly and annual reports (2) Key in formant interviews (3) Guided group discussions w beneficiaries & community members (4) Evaluation team technical knowledge & resources (1) content extraction to spread sheets (2) transcription of interviews and discussions; analysis of detailed notes taken by discussion observers (3) Content analysis of spread sheets transcriptions, notes and checklists XIV Purpose of the EPAS evaluation, to: Questions that address the evaluation purpose Data Source(s) and Collection Methods Data Analysis Methods (2)What accomplishments are EPAS personnel and service providers most proud of? Why? Are the accomplishments sustainable? Is the approach transferrable to resource poor settings elsewhere in Africa (1) Document review particularly EGPAF quarterly and annual reports (2) Key in formant interviews (3) Guided group discussions w beneficiaries & community members (4) Evaluation team technical knowledge & resources (1) content extraction to spread sheets (2) transcription of interviews and discussions; analysis of detailed notes taken by discussion observers (3) Content analysis of spread sheets transcriptions, notes and checklists (4) Provide strategic guidance for the remaining years of the project and any follow‐on activities. (1) Is the program design appropriate? (2) What are the gaps, if any, in implementation? (3) Are there any increased efficiencies possible? (4) How might sustainability be increased? (1) Document review particularly those documenting the initial program design [for example the results framework] and any modifications to the design (2) Key informant interviews (3) ) Guided group discussions w beneficiaries & community members (4) Evaluation team technical (1) document content analysis (2) transcription of interviews and discussions; analysis of detailed notes taken by discussion observers (3) Content analysis of transcriptions and notes Evaluation Question 1.To what extent have the project's four main objectives been achieved; what have been the successes, failures and gaps in EPGAF's approach in: XV EPAS Objective: What evidence will be looked for? Data Sources & Collection Methods Data Analysis Methods (a) Expanding PMTCT services [To achieve universal access to PMTCT through increasing geographical coverage of services and addressing social and gender norms that create barriers to service uptake and retention in care]  Increased geographical coverage with PMTCT services  Increased # service delivery sites  Increased # clients served  Proportion of clients served to eligible pregnant women increased to >95%  Proportion of clients retained in care increased to >95%  HMIS data extraction to evaluation spread sheets  Project M&E data review  KIIs with health managers & EPAS personnel (1) Re‐aggregate HMIS data (2) Compare HMIS data with Project data (3) Calculate indicators based on the findings and compare with the reported ones (4) Comparison to the targets set in project document and Strategic Plan b) Comprehensive and quality PMTCT services integrated into ANC, labor & delivery and post‐natal clinics, including ANC & PNC visits, HTC and ARV uptake, adherence, facility deliveries, HIV transmission rates at 6‐8 weeks, etc. [To provide and sustain quality, comprehensive and integrated PMTCT services through supporting clinical mentoring and supportive supervision of health care workers at health facilities across the country]  Sustained availability of components of PMTCT in ART clinics, ANC, labor & delivery, and post‐natal clinics:  HTC  involvement of men  appropriate per WHO Guidelines infant feeding counseling & support  ART available to positive pregnant women as per WHO guidelines  ART adherence support  skilled delivery care  DBS from HIV–exposed newborns  Positive infants receiving ART & CTX  Peer support for positive mothers  Availability of FP counseling & commodities postnatally & in ART clinics  Direct observation with checklists  Review of health facility records  KIIs with health managers & health workers delivering PMTCT services; Guided discussions with service users & partners  HMIS data extraction to evaluation spread sheets  Project M&E data review (1) Create database to help analyze questionnaire/checklist data (2) Create data sheets to compile HMIS and Project M&E data (3) Cross‐check information from the different sources (4) Use of Master‐sheet to summarize qualitative data into appropriate codes and themes (5) Use of verbatim quotes from respondents to validate findings (6) Comparison to the targets set in project document and national Strategic Plan XVI EPAS Objective: What evidence will be looked for? Data Sources & Collection Methods Data Analysis Methods  % women in ANC who are HIV tested and receive results  % positive women in ANC who uptake ARVs/ART  % negative pregnant women at booking who are retested at 36‐40 weeks  % HIV‐exposed infants who have DBS taken  % HIV‐exposed infants who receive recommended course ARVs  % positive DBS  % positive infants who receive ART & CTX  HMIS data extraction to evaluation spread sheets  Project M&E data review (1)Re‐aggregate HMIS data (2)Compare HMIS data with Project data (3)Calculate indicators based on the findings and compare with the reported ones (4) Comparison to the targets set in project document and national Strategic Plan (c) National PMTCT system Strengthening [To strengthen the national health systems in line with the Ministry of Health plans for PMTCT through technical assistance and capacity building to ensure sustainability]  On the job coaching & mentoring of service delivery staff established & regularly conducted  training support for health workers on infant feeding and nutrition established & conducted  Service delivery staff aware of and practicing service delivery to operational standards defined by the MOH  Reduced stockouts of essential PMTCT drugs & reagents  KIIs with health managers at national and regional levels & EPAS personnel  KIIs with health managers at regional and facility level & EPAS personnel  Review of health facility pharmacy and laboratory records against checklists  Review of project training reports and data with extraction of data to evaluation spreadsheets (1)Transcription of interviews (2)Content analysis of transcriptions, notes and checklists (3)Analysis of Master spreadsheet for analysis of qualitative data into appropriate codes and themes (4)Use of verbatim quotes from respondents to validate findings (5) Comparison to the targets set in project document and national Strategic Plan XVII EPAS Objective: What evidence will be looked for? Data Sources & Collection Methods Data Analysis Methods (d) Supporting MoH in the development of protocols, guidelines, and job aids [To support the MoH develop and review PMTCT policies, protocols and guidelines on a regular basis]  MOH protocols, guidelines, and job aids, developed with EPAS support, available to evaluation & in use in health facilities  KIIs with health facility personnel  Observation of service delivery and verification of physical availability with checklists (1)Transcription of interviews (2)Content analysis of transcriptions, notes and checklists EPAS Evaluation Question: 2.What were the successes, challenges and gaps in EPAS’s community approach? Aspect of community approach: What evidence will be looked for? Data Sources & Collection Methods Data Analysis Methods Linkages between communities and facilities  “Ownership” of PMTCT services by community  KIIs with EPAS partner staff  KIIS with health facility personnel  Guided group discussions with PMTCT service users & communities served (1)Transcription of interviews (2)Content analysis of transcriptions, notes and checklists m2m strategy  Active m2m groups in communities  Subpartner reports  KIIs with health facility personnel  Guided group discussions with PMTCT service users & communities (1)Transcription of interviews (2)Content analysis of reports, transcriptions, notes and checklists Dissemination of MNCH/HIV messages to women in rural areas  Outreach processes for health promotion within communities  Subpartner reports  KIIs with health facility personnel  Guided group discussions with MNCH and PMTCT service users and communities (1)Transcription of interviews (2)Content analysis of reports, transcriptions, notes and checklists XVIII Aspect of community approach: What evidence will be looked for? Data Sources & Collection Methods Data Analysis Methods Involvement of men  Partners attending antenatal/ PMTCT services  Men in community aware of key PMTCT messages  % of male partners of pregnant women tested for HIV  HMIS data extraction  EPAS report analysis  KIIs with health facility personnel  Observation with checklists  guided group discussions with men in community (1)Re‐aggregate HMIS data (2)Content analysis of reports transcriptions, notes and checklists (3)Use of verbatim quotes from respondents to validate findings EPAS Evaluation Question: 3.How sustainable are the gains made under EPAS? How many facilities have been graduated? How many have improved, and to what extent? What factors contributed to sustainable improvement? Aspect of gains under EPAS: What evidence will be looked for? Data Sources & Collection Methods Data Analysis Methods Sustainability  MOH has personnel & budget to continue PMTCT services and activities initiated with EPAS support  HMIS data collected, analyzed and regularly used for PMTCT service improvement by regional and facility managers  GOKS & MOH document review  KIIs—national regional & facility levels (1) Content analysis of reports transcriptions, notes and checklists Graduation of facilities  Number of facilities graduating by year of program  % of all facilities graduated at time of evaluation  Review of EPAS reports and M&E data (1)Re‐aggregate HMIS data (2)Content analysis of reports Improvement of facilities & services  Comparison of service quality at graduated and non‐graduated facilities  Improvement in current over historic service delivery data  Facility observation with checklists  HMIS & facility registers data extraction (1) Analysis of service delivery statistics from HMIS & facility registers over length of the project (2) Analysis of checklists XIX Aspect of gains under EPAS: What evidence will be looked for? Data Sources & Collection Methods Data Analysis Methods Factors contributing to sustained improvement  MOH budgets—national & regional  MOH HR plans & HR availability  Leadership—national, regional, facility  Staff moral & absenteeism  Involvement of communities/ service users in facility  GOKS & MOH document review  KIIs—national regional & facility levels  Guided interviews with members of communities served (1) Content analysis of reports, transcriptions, notes EPAS Evaluation Question: 4.How has EPAS strengthened MNCH services more broadly? For example, in the areas of family planning, prevention and management of obstetrical complications, new‐born health, infant feeding and reducing post‐natal transmission of HIV? What are the remaining gaps? Aspect of MNCH services: What evidence will be looked for? Data Sources & Collection Methods Data Analysis Methods Family planning  FP counseling & support integrated into PMTCT service delivery  Availability of condoms and other modern FP methods  KIIs with health facility personnel  Guided group discussion with service users  Direct observation with checklists (1)Transcription of interviews (2)Content analysis of transcriptions, notes and checklists Prevention and management of obstetric complications  Delivery ward health workers knowledge & skills  Availability & use of caesarian section  KIIs with health facility personnel  Direct observation with checklists (1)Transcription of interviews (2)Content analysis of transcriptions, notes and checklists Newborn care  Clearing of nasopharynx after delivery  Prevention of hypothermia [wrapping newborn, not bathing]  Prevention of sepsis [cord care, eye care]  KIIs with health facility personnel  Direct observation with checklists (1)Transcription of interviews (2)Content analysis of transcriptions, notes and checklists XX Aspect of MNCH services: What evidence will be looked for? Data Sources & Collection Methods Data Analysis Methods Infant feeding  Support for early establishment of exclusive breast feeding [skin to skin contact with mother; putting to the breast soon after delivery; rooming in with mother]  % of infants exclusively breast fed at 6 months old  % of infants breast fed into 2nd year of life after the introduction of locally appropriate complementary feeding  KIIs with health facility personnel  Direct observation with checklists  Guided group discussions with M2M groups  Guided group discussions in the community with mothers of infants in the community (1)Transcription of interviews (2)Content analysis of transcriptions, notes and checklists Post natal transmission of HIV  Breastfeeding infant by others than biological mother  Mixed feeding  Promotion of risk reduction messages to mother & her partner  KIIs with health facility personnel  Direct observation with checklists  Guided group discussions with M2M groups  Guided group discussions in the community with mothers of infants in the community (1)Transcription of interviews (2)Content analysis of transcriptions, notes and checklists EPAS Evaluation Question: 5.What are the remaining challenges to improving PMTCT outcomes in Swaziland? What evidence will be looked for? Data Sources & Collection Methods Data Analysis Methods Availability  Geographic availability  frequency of clinics [daily/not]  HMIS  GOKS & EPAS reports  KIIs with MOH & EPAS personnel  Facility registers (1)Analysis of service delivery statistics from HMIS & facility registers (2)Transcription of interviews (3)Content analysis of XXI What evidence will be looked for? Data Sources & Collection Methods Data Analysis Methods Acceptability & affordability  Client satisfaction  Direct and opportunity costs to service users  Guided group discussions with service users and communities (1)Transcription of discussions (2)Content analysis of transcriptions, notes Quality  From management, health worker and client perspectives  KIIs with regional health managers, health facility managers and health workers  Guided group discussions with service users and communities (1)Transcription of discussions (2)Content analysis of transcriptions, notes Sustainability of EPAS gains after end of program  GOKS budget & HR commitments  Institutionalization of EPAS initiated improvements  Review of MOH reports  KIIs MOH at national and regional levels  KIIs EPAS personnel (1)Transcription of discussions (2)Content analysis of reports transcriptions, notes EPAS Evaluation Question: 6.How effective has been EPAS’s approach in providing facility level support? What evidence will be looked for? Data Sources & Collection Methods Data Analysis Methods Health worker satisfaction  KIIs with health facility health workers (1) Transcription of discussion (2) Content analysis Health Manager satisfaction  KIIs regional health managers and health facility managers (3) Transcription of discussion (4) Content analysis Service user/client satisfaction  Guided group discussions with service users & communities (1) Transcription of discussion (2) Content analysis Health worker knowledge & clinical skills  checklists to assess provider’s clinical skills and competencies in delivery of care and treatment services Content analysis of checklists Service delivery quality  As per 1 (b) above XXII ANNEX III: EVALUATION METHODOLOGY The evaluation employed the following five methods of data collection, triangulation and verification of the information to evaluate the EPAS and answer the evaluation questions: 1. Desk review of the documents made available by USAID Southern Africa; other relevant documentation including local studies, guidelines, and best practice documents; and other Swaziland and Sub-Saharan Africa- specific HIV and PMTCT literature available through the internet or in the evaluators’ personal collections 2. An online survey of EGPAF and sub-partners’ staff 3. Analysis of secondary data from the HMIS, and EGPAF and partners’ plans, budgets and reports 4. Key informant interviews and guided group discussions with a wide range of stakeholders—including health care providers and service users—who consented to participate, to obtain a wide range of stakeholders’ perceptions 5. Checklists for focused observations for assessing service delivery at selected facilities and in the community including counseling and testing, antenatal clinics, labor and delivery wards, post natal follow up, and MNCH services, coordination and oversight The evaluation used both primary data and secondary data from the routine Health Management Information System (HMIS), EGPAF reporting to USAID and PEPFAR, and documents and reports from a review of the relevant literature available to the evaluators. The evaluation team was sensitive to time constraints and the ownership of the data by the MOH; and worked with the MOH to set up KIIs and site visits. A representative of the MOH participated in the in the field visits and made the formal introductions to health facility staff. As the MOH does not permit development partners to collect monitoring data directly from the health facilities and EPAS uses HMIS data sets received from the MOH. The evaluation team re￾aggregated and analyzed the HMIS data sets from the MOH; evaluated EGPAF project management and implementation monitoring data from the reports to USAID/PEPFAR, and EPAS activities to strengthen the PMTCT/MNCH information collection and use by the MOH. TRIANGULATION The evaluators triangulated their findings, comparing a variety of data from different sources (data triangulation), using different methods (methodological triangulation) and between the evaluators (evaluator triangulation). SAMPLING The evaluation used a stratified, purposive sampling to select sites to be visited during field data collection, as well as participants for interviews and group discussions. The MOH indicated its National level personnel to be interviewed and the UN and PEPFAR partner organizations to be interviewed. The sample included the regional health management team, a hospital, and three lower facilities all four regions (Hhohho, Shiselweni, Manzini, Lubombo). EPAS supports one private sector hospital and one mission hospital in different regions: the sample was stratified to include both and two public hospitals from the other two regions. EPAS grades health centers XXIII and clinics as high performing—ones that are graduated and receive supervisory/mentoring visits quarterly; medium performing—receive supervisory/mentoring visits monthly; and low performing—these receive supervisory/mentoring visits monthly and additional support to improve performance. The sample of health centers and clinics ensured representation of private sector, mission and public facilities, and high, medium, and low performing facilities. Where there was more than one facility in the category, the sample was taken blind of any further details such as sites’ geographic situation. This purposive sampling captured mainly qualitative information on sites where EPAS has been particularly successful or unsuccessful, to inform lessons learned. SITES SELECTED AND VISITED Hhohho Region: Mbabane Hospital & Public Health Unit Emkhuzweni Health Centre Mangweni Clinic Ntfonjengi Clinic Manzini Region: Mankayane Hospital & Public Health Unit Lushikishini Clinic Mawelawela Clinic Mliba Nazarene Clinic King Sobhuza II Clinic*** Shiselweni Region: Hlatikulu Hospital Nhlangano Health Centre Silele Red Cross Clinic SOS Nhlangano Clinic Lubombo Region: Good Shepherd Hospital & Public Health Unit Sithobela Health Centre Lomahasha Clinic Lubuli Clinic *** This evaluators were directed by the Manzini Regional Administrator to visit this site; he was not present when the team were there and so the evaluators conducted a site visit while waiting. EVALUATION FRAMEWORK The evaluation framework guided the evaluation through answering each of the evaluation questions. It is included in Annex II. LIMITATIONS OF THE EVALUATION AND ITS FINDINGS As the evaluation was a cross-sectional assessment of the effectiveness of EPAS, the evaluators were not able to ascertain long-term achievements from the field data collection. They re￾aggregated and analyzed MOH HMIS data to assess the progress so far and improvements in performance to date. The quality of the data was re-assessed to determine its reliability but the team was not be able to conduct formal data quality assessments, fill in gaps in information or correct errors. The evaluation was conducted with only a sample of project sites, activities and informants, by a small team of external evaluators. Although the team has made every effort to make the sample representative by stratifying, it is not large enough to be analyzed statistically, and may not be generalizable. For practical reasons, the sample is purposefully stratified across four levels, by region, by type of facility, by facility performance including graduation status and to ensure the evaluators saw the best of EPAS and its most challenged. Inevitably, the sample in XXIV each stratification was small. Triangulation and verification will have reduced biases and errors, but the generalizability may be limited. Specifically, evaluating the high performing and low performing sites is needed for lessons learned. Generalizability may have been traded off against obtaining evidence to support specific strategic recommendations for maximizing the results in the last year of implementation, and for future funding. The key informant interviews depended on the informants’ availability at the time of the field visit. The interviews were conducted in English; the evaluators relied on local persons to translate guided group discussions where the participants used local languages. The local logistics consultant was an impartial translator but the MOH evaluation participant brought their own biases to the translation and were not used after the first few days. The Snr. Evaluation Specialist and Snr. Technical Advisor for PMTCT/MNCH are experienced in conducting evaluation group discussions in such circumstances and were alert for evidence that the interpreter was directing the responses. They will ask other participants that have some English to confirm responses. Attribution may be difficult in the presence of numerous development partners (particularly in the community/demand side) although the size of USAID/PEPFAR investment and the scale of EPAS support at facility level enabled the evaluation to assess the contribution made by EPAS project to the supply side at health facility levels. WORKPLAN The MIDEGO evaluation team approached the evaluation in three main phases—(1) inception, (2) field data collection, and (3) analysis and reporting. 1. INCEPTION PHASE MAY 20, 2014 TO MAY 30, 2014 The Inception Phase commenced with receipt of the signed contract May 20, 2014. The evaluation team began collecting and pooling relevant electronic resources. Some Desk Review documents from EGPAF arrived May 21, 2014. The Evaluation Team Leader with the Home Office Administrative Support manager participated by telephone in the First Team Planning Meeting [Deliverable 1] Thursday May 22, 2014. On May 26, 2014 MIDEGO requested USAID/ Southern Africa and EGPAF assist with supplying the missing review documents. These were made available at the beginning of June as the team were preparing for travel and during the first days in country. The evaluation team completed the Evaluation Framework informed by the desk review and their technical knowledge, drafted the MIDEGO shared the draft workplan [Deliverable 2] with USAID/SA May 27, 2014. The team went on to draft a set of evaluation tools and MIDEGO provided USAID/SA with a full packet of deliverables including the Evaluation Framework, Workplan and Tools May 29, to support the request for approval to travel, granted May 30, 2014. The Inception Phase ended with USAID/SA review of the Workplan, draft Evaluation Framework, XXV and draft Tools May 30th, and issuance of approval to travel. This exchange by email replaced the pre-fieldwork briefing, [Deliverable 3] teleconference at the request of USAID/SA during the teleconference First Team Planning Meeting, Thursday May 22, 2014. 2. IN-COUNTRY DATA COLLECTION PHASE JUNE 7, 2014 TO JULY 3, 2014 In country work phase began with the arrival of the MIDEGO evaluation team in Mbabane June 6-8, 2014. The team participated in the initial in-country briefing with the Swaziland PEPFAR Team [Deliverable 4] and a protocol visit to the MOH June 9, 2014. They participated in a briefing by EGPAF Senior Management Team and further briefing with USAID. The evaluators liaised with the MOH, regarding the scheduling of Key Informant Interviews (KIIs) at national level, the site visits and travel schedules. The evaluators finalized the sampling framework, informed by the initial briefings and the evaluation field visit schedule and finalized the evaluation tools incorporating feedback from USAID and new information from the in￾country briefings. The evaluation tools—an email survey, key informant interview question lists; group discussion guide questions; observation checklists are included in Annex IV. The list of persons met with in KIIs is included in Annex V. The team conducted KIIs at national level, KIIs at regional level; site visits, KIIs, guided group discussions, service delivery observations throughout all 4 regions, June 12-July 1, 2014. The team leader provided, weekly status reports to USAID/interim briefings, Deliverable 5 by email. The MIDEGO evaluation team leader, with the assistance of team members, presented a key findings out brief to USAID, Deliverable 6, July 2 and a debriefing for the MOH July 2, and EGPAF July 3 after completion of field work and before leaving Swaziland. 3. ANALYSIS AND REPORT PREPARATION PHASE JULY 7, 2014 TO JULY 18, 2014 In their home offices, the evaluators analyzed their findings and prepared the draft final report, delivered to USAID/Southern Africa by the beginning of their working day July 21, 2014—10 working days after the completion of the field data collection. The MIDEGO team will finalize and deliver the finalized report 1 week after receiving the feedback from the mission. 6. COMMUNICATION AND CONSULTATION PLAN (WITH STAKEHOLDERS) MIDEGO communicated with USAID/Southern Africa by email and Skype, and provided the hard copies of the final evaluation report required in the contract by international courier. The main axis of communication on contracting matters was between the USAID/CO and the MIDEGO President. The main access for communication between the evaluation team and USAID/Southern Africa was between the USAID/COR and the evaluation team leader. Communications between the USAID/COR and evaluation team leader were by email and telephone prior to arrival in county, and by email, cellphone and Short Message Service (SMS) in country. The team leader set up email groups to ensure all the relevant USAID and MIDEGO team were copied on email communications. XXVI The desk review documents from EGPAF were shared through DropBox with additional documents shared by USAID by email. When working in their home offices, the evaluators communicated by email and Skype, and shared documents amongst themselves by email attachment and DropBox. Deliverables were emailed. Hard copies of the final report will be provided as per the contract on the due dates. XXVII ANNEX IV: DATA COLLECTION INSTRUMENTS 1. EPAS PROJECT EVALUATION EMAIL SURVEY XXVIII 2. QUESTIONS FOR KEY INFORMANT INTERVIEWS XXX 3. GUIDED GROUP DISCUSSION PLANS XXXII 3.1 WITH MOTHERS-2-MOTHERS GROUP MEMBERS XXXII 3.2 WITH COMMUNITY MEMBERS/MEN’S GROUPS IN THE COMMUNITY XXXIII 3.3 WITH PMTCT SERVICE USERS AT HEALTH FACILITIES XXXIV XXVIII 1. EPAS PROJECT EVALUATION EMAIL SURVEY All information from this survey will be used anonymously We need your email address only to ensure we do not receive duplicate responses for any reason. However we will not use your email, share your email or try and identify you though your email. If you have any questions, please email the team leader, Ruth Hope: ruth@midego.com in confidence You may make your answers as detailed and as long as you wish: the boxes will enlarge as you type! email: What is your gender?: Job title: Date of hire [month and year] : Evaluation Questions Your Response 1. What is your role in PMTCT service delivery? 2. What has been your main contribution to the EPAS project? 3. How has the quality of PMTCT services changed during the EPAS project implementation? Why? 4. How has the quality of MNCH services changed during the EPAS project implementation? Why? 5. How will the changes be sustained after the end of the EPAS project? 6. What are the challenges to sustaining change after the end of the EPAS project? 7. How have clients and the communities served by the EPAS project been involved in the project? XXIX EPAS PROJECT EVALUATION EMAIL SURVEY All information from this survey will be used anonymously We need your email address only to ensure we do not receive duplicate responses for any reason. However we will not use your email, share your email or try and identify you though your email. If you have any questions, please email the team leader, Ruth Hope: ruth@midego.com in confidence You may make your answers as detailed and as long as you wish: the boxes will enlarge as you type! email: What is your gender?: Job title: Date of hire [month and year] : Evaluation Questions Your Response 8. What in your view are the three main successes of the EPAS project? 9. What are the three main lessons learned from your experience of the EPAS project? 10. What do the EPAS project evaluators and USAID need to know to ensure future support to PMTCT services—in Swaziland and elsewhere in Southern Africa—is effective and sustained? Any other comments or observations: Thank you for your time and thoughtful responses to this survey! yours sincerely, Ruth Hope, EPAS Evaluation Team Leader XXX QUESTIONS FOR KEY INFORMANT INTERVIEWS Key Informant Name: Gender: Organization & Job Title: Date & place of interview: Evaluator(s) Name(s): NB: Only ask the questions pertinent to the Informant! Tailor the questions appropriate to the informant! [for example clarify how EGPAF sub-partner staff have been involved with EPAS] 1. PROGRAM MANAGEMENT 1.1 Has the EPAS structure succeeded in supporting expansion and delivery of appropriate and high quality services? If so, how? 1.2 What are the three most important EPAS achievements? What accomplishments are you most proud of/are most commendable? 1.3 What has been less successful? Why? 1.4 What have been the three most important challenges to implementing EPAS? How were they managed? 1.5 How satisfied are you with EPAS? Probe! 1.6 In general, how satisfied are health managers in their work? Probe! 1.7 In general, how satisfied are health workers in their work? Probe! 1.8 How satisfied are service users with their local MNCH/PMTCT services? Probe! 2. SERVICE DELIVERY 2.1 Has EPAS increased the availability, quality and consistency of facility and community-based care and support to PMTCT clients and their families? If so, how? 2.2 Have components of PMTCT been consistently available in: the community? ART clinics? ANC clinics? labor and delivery wards? MNCH services/child welfare clinics? How has this been achieved? 2.3 What are the challenges to providing quality and consistency in service delivery? 2.4 What are the links between PMTCT and family planning services? [for health facility key informants:] Are condoms consistently available within PMTCT services? What modern family planning methods are consistently available to positive women after delivery? 2.5 What more could be done to increase the numbers of pregnant women accessing PMTCT? To retain positive pregnant women and their infants in treatment, care and support? 2.6 Are mentor mothers/m2m groups effective in reducing HIV infection in infants? How? 2.7 How do mentor mothers/m2m groups affect the infant feeding practices of mothers? 2.8 Has EPAS increased opportunities for HIV positive mothers and their infants to access family planning and MNCH services? How? XXXI 3. COMMUNITY INVOLVEMENT 3.1 How are clients and the communities served involved in EPAS? At the health facility? In the community? 3.2 How has EPAS influenced involvement of partners/husbands in FP/MNCH? In PMTCT? 3.3 How has EPAS mobilized communities to ensure pregnant women go to ANC? 3.4 Has EPAS contributed to reducing HIV stigma? How? 3.5 How has EPAS contributed to increasing community awareness about HIV, wider MNCH messages, and services available to mothers and their infants? 3.6 Has EPAS demonstrated significant improvement in the quality of life of HIV positive women and their infants? If so, how? 4. SYSTEM STRENGTHENING, CAPACITY BUILDING & SUSTAINABILITY 4.1 How has EPAS built the capacity/strengthened institutionally local partners? health facilities and staff? regional health teams? 4.2 How has EPAS strengthened health care service delivery? What protocols, guidelines and job aids are available as a result of EPAS support? 4.3 Does EPAS have an exit strategy that has assures continuance of the gains achieved under EPAS after project has ended? Describe the strategy: 4.4 What will be the challenges to sustaining the gains achieved under EPAS beyond the end of the project? 5. MONITORING AND EVALUATION For Key informants with knowledge of the HMIS 5.1 How has EPAS strengthened the HMIS? 5.2 In particular, how has data collected through the HMIS been used to increase service delivery performance and quality? At facility level? At regional level? At national level? 5.2 How does the HMIS measure the progress towards eliminating pediatric AIDS in Swaziland targets? 6. LESSONS LEARNED 6.1 What are the three most important lessons learned from implementing EPAS? 6.2 What are the remaining challenges to eliminating pediatric AIDS in Swaziland? How can they be managed? 6.3 In summary, what are the successes of EPAS that merit continuation or replication elsewhere in Southern Africa? XXXII GUIDED GROUP DISCUSSION PLAN WITH MOTHERS-2-MOTHERS GROUP MEMBERS Record: date of discussion Place of discussion/m2m meeting: Number of participants (M/F): Name of health facility Name(s) of evaluator(s) Name of interpreter: 1. Introductions: everyone tell their name Evaluator to give explanation for holding discussion: o Wish to learn about the women’s experience of being a member of a mothers’ support group to improve the design of future mothers’ groups elsewhere in Africa Following the introductions and translation of the evaluator’s statement, the discussion will be conducted entirely in the local language by the Team leader with translation by the local logistics consultant or MOH participant. 2. How long have you been members of the group? 3. How did you learn about the m2m group? How did you become members of the m2m group? Why did you join the group? Probe for feelings of the HIV+ mothers at the time and how they felt about the mentor mothers and the m2m group 4. How often are the m2m group meetings? Tell us what do you do in the m2m group meetings Probe for details of how the meetings are conducted, who does what, the feelings of the participants about the meetings 5. Tell us about what have you learned from the m2m group meetings Probe about the topics, new knowledge and skills 6. What other benefits are there from m2m group membership? 7. Are any male partners/husbands involved in the group? Why (not)? Probe whether male partners/husbands should join the m2m group or have their own fathers’ support group 9. Would more male partners and mothers join the m2m group if it were in the community closer to their homes? Probe Why/Why not? 10. How do you find the services provided by your [facility]? Probe about how long pregnant women have to wait to be seen? Where do they wait? How do the nursing staff behave towards the women? [bossy/respectful/angry etc] Is there enough time to talk with the nurse about how you feel? Your medication? About your problems? Would they say overall the service is OK? Good? Poor? 11. How much does it cost to attend ANC and receive PMTCT services? Probe: cost of transport, registration fees, consultation fees, any other fees? Opportunity costs such as loss of earnings? Any other costs to the pregnant woman? Over all does the [health facility] provide good value for the costs or is it not good value? 12. Round-up: Thank participants and ask if there is anything more they want to tell or ask the evaluator XXXIII GUIDED GROUP DISCUSSION PLAN WITH COMMUNITY MEMBERS/MEN’S GROUPS IN THE COMMUNITY Record date of discussion: Place of discussion: Number of participants (M/F): Name of local facility: Name(s) of evaluator(s): Name of interpreter: 1. Introductions: everyone tell their name Evaluator to give explanation for holding discussion: Wish to learn about the communities’ experience of PMTCT services, and mother newborn and child health services in general. We will use our findings to give advice on improving the design of mother newborn and child health services elsewhere here and elsewhere in Africa Following the introductions and translation of the evaluator’s statement, the discussion will be conducted entirely in the local language by the Team leader with translation by the Local logistics consultant or MOH participant. 2. What are your experiences with mother, newborn and child welfare services in general? How are the PMTCT services in this area? Probe who has used services and what others have heard about services? 3. Who provides the mother newborn and child welfare services round here? Who provides PMTCT services? 4. How good are the mother newborn and child welfare services? And the PMTCT Services? Probe: Why do you say that? So are the services OK? Good? Not so good? 5. What are the costs of the mother newborn and child welfare services to people using the services? Are there any additional costs for PMTCT services? Probe: cost of transport, registration fees, consultation fees, any other fees? Opportunity costs such as loss of earnings? Any other costs to the pregnant woman? 6. How are people in the community – like you – involved mother newborn and child health services in the community? At the health facility? How does the community ensure that a woman who is pregnant is seen in the antenatal clinic? 7. Are there any groups for HIV positive people around here? Are there any groups for HIV positive mothers round here? Probe for details of who is eligible to join any groups? any barriers to joining the groups? where the groups meet and how often? who organizes the meetings? 8. What are the benefits of joining a group/attending meetings Probe about the topics, new knowledge and skills 9. Are men involved in or members of any of the groups? Probe: why/why not? 10. Round-up: Thank participants and ask if there is anything more they want to tell or ask the evaluator XXXIV GUIDED GROUP DISCUSSION PLAN WITH PMTCT SERVICE USERS AT HEALTH FACILITIES Record: date of discussion Place of discussion: Number of participants (M/F): Name(s) of evaluator(s) Name of interpreter 1. Introductions: everyone tell their name Evaluator to give explanation for holding discussion: Wish to learn about the communities’ experience of PMTCT services, and mother newborn and child health services in general. We will use our findings to give advice on improving the design of mother newborn and child health services elsewhere here and elsewhere in Africa Following the introductions and translation of the evaluator’s statement, the discussion will be conducted entirely in the local language by the Team leader with translation by the Local logistics consultant or MOH participant. 2. What are your experiences with the PMTCT services at this health facility? How are the mother newborn and child health services in general? 3. Who provides PMTCT services, and mother newborn and child health services at this facility? 4. How good are the PMTCT services, and mother newborn and child health services at this facility? Why do you say that? Probe about waiting times; whether staff are caring; whether medicines are available; whether user has confidence in the health workers, Where male partners are welcome etc 5. What are the costs of the PMTCT services, and mother newborn and child health services to people like you using the services? Probe about fees for registration, medicines, lab tests; how long it takes to get to the clinic; transport costs; need for time off work etc. 6. How are people in the community – like you – involved in PMTCT services, and mother newborn and child health services in the community? At the health facility? 7. Are there any groups for HIV positive people around here? Are there any groups for HIV positive mothers round here? Probe for details of who are eligible to join any groups? any barriers to joining the groups? where the groups meet and how often? who organizes the meetings? What are the benefits of joining a group/attending meetings? Probe about the topics, new knowledge and skills XXXV ANNEX V: SOURCES OF INFORMATION 1. PERSONS THE EVALUATION TEAM MET WITH NAME Organization Charles Mandivenyi, COR USAID Southern African Region Munamato Mirira, Alt. COR & PMTCT Team Leader, USAID Swaziland Natalie Kruse-Levy USAID Country Director USAID Swaziland Wendy Githens Benzerga USAID Deputy Director USAID Swaziland Lucille Bonaventure PEPFAR Coordinator USAID Swaziland Peter Ehrenkranz CDC Country Director Centers for Disease Control, Swaziland Peter Preko Senior Care & Treatment Specialist Centers for Disease Control, Swaziland Sipho Kunene PEPFAR program assistant USAID Swaziland Ms Rejoice Nkambule Deputy Director Health Services Ministry of Health Dr Velephi Okello Senior Medical Officer SNAP ART Coordinator Ministry of Health Dr Simangele Mthethwa Technical Advisor Sexual and Reproductive Health Program (SRHU) Ministry of Health Mrs Phumzile Mabuza SRH/MNH Manager SNAP Ministry of Health Ms Zanele Simelane Health Monitoring Information Systems (HMIS) Ministry of Health Ms Nomsa Mulima Strategic Information Dept. (SID) Ministry of Health Ms Daniela Phiri Snr Computer Analyst Ministry of Health Ms Christabel Mahlambi PMTCT Coordinator Hhohho RHMT EGPAF/Ministry of Health, Hhohho RHMT Ms Sebentile Myeni M& E Officer Mbabane PHU XXXVI Ms Lindiwe Shongwe MNCH Mentor Ministry of Health, Hhohho RHMT Sr D. Dlamini Phlebotomist/HTC counselor Ntfonjeni Clinic Matron Raynet Sr Thandi Ndwandwe Emkhuzweni Health Clinic RN Mr Hadebe Mangweni Clinic Mr Mlondolozi Dlamini Regional Administrator RHMT Manzini Sr Elizabeth Qu. Mvila Snr Nurse/Dermatologist Reg Nurse Loncwala Nkosi, ART Data RN Dumisile Dlamini RN Precious Dlamini King Sobuza II Clinic /PHU Sr G Mavuso, Snr Reg Nurse Sr Iris Dlamini, Reg Nurse Lushikishini Clinic RN Thobile Gift Nxumalo RN P. Sibandze Mliba Nazarene Clinic Sr Phindile Mabuza Acting Matron Mankayane Hospital Sr Nomsa Mlangeni PMTCT Health Clinic Midwife Mawelawela Women’s Prison Clinic Mr Mankinathi Shongwe Lubombo Region Health Administrator RHMT, Lubombo Sr Phephile H Khumalo Maternity Unit Snr Sr Betty Mushanti Maternity Unit Sr Lindiwe Dlamini, MNCH Good Shepherd Hospital Sr Anastasia Mavundla Lomahasha Clinic Sr Thoko Hlope Lubuli Clinic Matron Ester Dlamini Sithobela Health Centre RN Thulisile Gama Lubombo Mentor RN Mr Liberty Thwala Clinic Supervisor, Hho Zone Regional Health Office Nhlangano CBD Pureen Madlopha Clinic Supervisor, Hlatikulu Zone Regional Health Office Nhlangano CBD Ms Nombekezelo Shongwe Regional M&E Regional Health Office Nhlangano CBD Sr Ester Simelane Sr Phindile Sihlonwanyane EGPAF Coodinator SOS Children’s Village Clinic (NGO) Nhlangano CBD Sr Khetsiwe Dlamini Phindile, PMTCT Reg Nurse Sandra Vilakati, Reg Nurse Jane Shongwe, Reg Nurse Mbekelezwa Shongwe, M&E Hlatikulu Government Hospital Sr Dumsile Ngwenya Silele Clinic XXXVII Appointed contact Shiselweni Area Dr Mapoana ART Clinic Nhlangano Hospital Ms Mumcy Thwala MNCH Mentor Shiselweni Dr Mohammed Ali Mahdi Chief Of Party EGPAF Swaziland Dr Caspian Chouraya Technical Director EGPAF Swaziland Kwashie Kudiabor Associate Director, SI&E EGPAF Swaziland Ms Thembi Masuku Program Manager EGPAF Swaziland Mr Musa Magongo Associate Director of Operations EGPAF Swaziland Mr Rudolf Maziya Country Director AMICALL Manzini Ms Jabu Ndzingane Program Manager Lutsango Mr Percy Chipepera SINAN Somhlolo Street Dr Sithembile Dlamini-Nqeketo TB, HIV and PMTCT Focal Person WHO Swaziland Dr Kwame Ampomah Country Director Ms Pepukai Chikukwa SI Advisor UNAIDS Swaziland Dr Florence Naluyinda-Kitabire HIV/AIDS Specialist UNICEF Swaziland Glory Mkandawire Chief of Party Johns Hopkins Health Communication Capacity Collaborative (HC3) Mr Charlie Gilman Country Manager Mr Patrick Shabangu Technical Director Mr Kelvin Sikwibele CEO Institute for Health Management Mr Luis Fernando Martinez PSI Ms Sibongile Maseko Country Director m2m Dr Ruben Sahabo Country Director ICAP Mr Bernhard Kerschberger Médecins San Frontières XXXVIII ANNEX V: SOURCES OF INFORMATION 2. DOCUMENTS AND OTHER RESOURCES ACCESSED 1. Ahanda, K & Conly, S (2013) Select Pieces from the SBCC review. USAID, Washington, DC 2. AMICAALL (2011) –Year 1 Budget and Workplan 3. AMICAALL (2012) –Year 2 Budget and Workplan 4. AMICAALL (2013) –Year 3 Budget and Workplan 5. AMICAALL (2014) –Year 4 Budget and Workplan 6. AMICAALL (2011) FY 2011 Reports Q2 and Q4 7. AMICAALL (2012) FY 2012 Reports Q1, Q2, Q3 and Q4 8. AMICAALL (2013) FY 2013 Reports Q1, Q2, Q3 and Q4 9. AMICAALL (2014) FY 2014 Report Q1 10. AMICAALL (2014) FY 2014 Monthly Reports March and April 11. CIA (2014) World Factbook https://www.cia.gov/library/publications/the-world￾factbook/geos/wz.html accessed 5/20/2014 12. CDC (2013) Site Monitoring System (SMS) Data Collection Tool. CDC, Atlanta, GA 13. CDC & ICAP-Columbia University (2012) Swaziland HIV Incidence Measurement Survey (SHIMS) First findings report. MOH, GOKS, Mbabane, Swaziland 14. Cohen, SA (2008) Hiding in Plain Sight: The Role of Contraception in Preventing HIV. Guttmacher Policy Review Vol 11, No 1 Winter 2008 15. Central Statistical Office [Swaziland], and Macro International Inc. (2008). Swaziland Demographic and Health Survey 2006-07. Central Statistical Office and Macro International Inc., Mbabane, Swaziland 16. Central Statistical Office and UNICEF (2011) Swaziland Multiple Indicator Cluster Survey 2010. Final Report. Central Statistical Office and UNICEF, Mbabane, Swaziland 17. Dlamini, M et al (201?) Addressing the Social and Behavioral Barriers to Prevention of Mother￾to-Child Transmission of HIV Through Community Dialogue: An implementer’s guide based on XXXIL the Elizabeth Glaser Pediatric AIDS Foundation’s Swaziland Experience. EGPAF, Washington DC 18. Donabedian, Avedis (1966) Evaluating the Quality of Medical Care. Reprinted in: The Milbank Quarterly, Vol. 83, No. 4, 2005 (pp. 691–729) 19. Donabedian, A., (1980), The definition of Quality and Approaches to its Assessment, vol. 1: Explorations in Quality Assessment and Monitoring. Ann Arbor, Michigan, Health Administration Press 20. Donabedian, A. et al, (1988), The Quality of Care: How can it be Assessed? JAMA, Vol. 260, No. 12, p1743-48 21. EGPAF (2010) EPAS Year1_Workplan_and_ME_Plan_Final 22. EGPAF (2011) APR11 Re-submitted 10 31 2011 23. EGPAF (2011) EPAS M&E Plan Final Dec 2011 24. EGPAF (2011) EPAS Year 2 Work Plan_Final 25. EGPAF (2011) m2m Original Sub-Agreement 02318 26. EGPAF (2011) m2m Amendment 1 02318 27. EGPAF (2011) m2m Amendment 2 02318 28. EGPAF (2011) SAPR11 04 14 11 submitted 29. EGPAF (2011) Sub-agreement U S-00-9-270-02288-0-00 AMICAALL 30. EGPAF (2011) Sub-agreement U S-00-9-270-02288-0-00 Amendment #7 AMICAALL 31. EGPAF (2012) EPAS APR12_Final Submitted 10.21.12 32. EGPAF (2012) EPAS FY12 Quarter 3 Report_Final 33. EGPAF (2012) EPAS SAPR12_Submitted 04.13.12 34. EGPAF (2012) EPAS Year 3 Work Plan Final 35. EGPAF (2012) PMTCT Report_Oct-Dec 2012 [PowerPoint slide] 36. EGPAF (2012) Swaziland Barriers to Antiretroviral Therapy Initiation for Eligible HIV-Positive XL Pregnant Women in Antenatal Care. EGPAF, Washington, DC 37. EGPAF (2013) EPAS APR13_Final submitted 10.25.13 38. EGPAF (2013) EPAS FY13 Quarter 1 & 3 Reports 39. EGPAF (2013) EPAS SAPR13_Final_Submitted 40. EGPAF (2013) EPAS Year 4 Work Plan 41. EGPAF (2013) Plan for Site Mentorship for FY13 -quarter 4 42. EGPAF (2013) Site Support Plan_FY14 Quarter 3 43. EGPAF (2013) Sub-agreement N° US-00-9-270-03301-1-00 SINAN 44. EGPAF (2013) Sub-agreement US-00-9-270-03302-1-00 Lutsango LwakaNgwane 45. EGPAF (2014) EPAS FY14 Quarter 1 Report_Final 46. EGPAF (2014) EPAS Program Overview 06.10.14 [PowerPoint presentation] 47. EGPAF (2014) EPAS SAPR 14_Final Revised 48. EGPAF (2014) EPAS Sites Performance Q2 FY14 vFinal 2 49. EGPAF (2014 ) List of PMTCT Sites_FY14_with coordinates Jun 09.2014 50. EGPAF (2014) PMTCT Cascade for EPAS FY14 Q1 51. Holmes, Charles (2011) PEPFAR Care, Treatment and PMTCT Programs: Results, Directions, Gaps & Opportunities. (Presentation to PEPFAR Scientific Advisory Board Meeting) Department of State, Washington, DC 52. ICAP (2014) Swaziland HIV Incidence Measurement Survey (SHIMS) Descriptive Data Tables April 2014. MOH, GOKS, Mbabane, Swaziland 53. Jewkes, Rachel et al. (2007) Evaluation of Stepping Stones: A gender transformative HIV Prevention Initiative. Medical Research Council, Pretoria, South Africa 54. Keiffer, Mary Pat et al (2011) Improved Detection of Incident HIV Infection and Uptake of PMTCT Services in Labor and Delivery in a High HIV Prevalence Setting.J Acquir Immune Defic Syndr. 57:e85–e91 XLI 55. Kudiabor, Kwashi (2011) EPAS Year 1 Site Level Baseline Performance Report. EGPAF, Mbabane, Swaziland 56. Leclerc-Madlala, Suzanne (2013) Debrief from Swaziland TDY, Feb. 11-15, 2013. USAID/OHA, Washington, DC 57. Lutsango Lwakangwane (2013) Budget - FINAL Revised 09.09.13 58. Lutsango Lwakangwane (2013) Work Plan-Final Draft 59. Lutsango Lwakangwane (2014) PMTCT Acceleration project reports Yr1 Q1 and Yr1 Q2 60. m2m (2010) Swaziland Budget - EGPAF – 180411 61. m2m (2010) FY11 EPAS Work Plan 62. m2m (2011) FY12 EPAS Work Plan 63. m2m (2011) EGPAF Q3 & Q4 Reports 2011 64. m2m (2012) FY13 EPAS Work Plan 65. m2m (2012) EGPAF Q1, Q2, Q3 & Q4 Reports FY 2012 66. m2m (2013) EGPAF Q1 Report FY 2013 67. MOH (2010) Guidelines for prevention of mother to child HIV transmission of HIV. 3rd Edition. GOKS, Mbabane, Swaziland 68. MOH (2010) Swaziland Annual Health Statistics Report 2010. Strategic Information Department, GOKS, Mbabane, Swaziland 69. MOH (2011) Annual Health Statistics Report 2011. Strategic Information Department, GOKS, Mbabane, Swaziland 70. MOH (2011) Elimination of new HIV infections among children by 2015 and keeping their mothers alive. National Strategic Framework for Accelerated Action. UNICEF, Mbabane, Swaziland 71. MOH (2011) Impact Assessment for ART, TB Treatment and PMTCT Using Triangulation Approach Final Report December 2011. GOKS, Mbabane, Swaziland 72. MOH (2012) 12th National HIV Serosurveillance Among Women Attending Antenatal Care Services in Swaziland. Survey Report 2010. GOKS, Mbabane, Swaziland XLII 73. MOH (2012) ART Program Annual Report 2011. Strategic Information Department, GOKS, Mbabane, Swaziland 74. MOH (2012) Evaluation of the Effectiveness of the National Prevention of Mother to Child Transmission of HIV (PMTCT) Programme at 6-8 weeks Postpartum in Swaziland. GOKS, Mbabane, Swaziland 75. MOH (2012) HTC Program Annual Report 2011. Strategic Information Department, GOKS, Mbabane, Swaziland 76. MOH (2012) PMTCT Program Annual Report 2011. Strategic Information Department, GOKS, Mbabane, Swaziland 77. MOH (2013) National ART Program Annual Report 2012. Strategic Information Department, GOKS, Mbabane, Swaziland 78. MOH (2013) National HTC Program Annual Report 2012. Strategic Information Department, GOKS, Mbabane, Swaziland 79. MOH (2013) National PMTCT Program Annual Report 2012. Strategic Information Department, GOKS, Mbabane, Swaziland 80. MOH (2013) National TB Program Annual Report 2012. Strategic Information Department, GOKS, Mbabane, Swaziland 81. MOH (2013) Service Availability Mapping (SAM) Report 2013. GOKS, Mbabane, Swaziland 82. MOH (2014) PMTCT HMIS Database 2010-2012 83. MOH (2014) PMTCT HMIS Database 2013-2014 84. MOH&SW (?2006) National Health Policy. GOKS. Mbabane, Swaziland 85. OGAC (2009) The U.S. President’s Emergency Plan for AIDS Relief Five-Year Strategy 2009- 2014. Department of State, Washington DC 86. PANOS (2012) Media Brief on Prevention of Mother-To-Child Transmission (PMTCT) of HIV. PANOS Institute Southern Africa, Lusaka, Zambia 87. PEPFAR (2011) Swaziland PMTCT Acceleration Plan FY 2012. OGAC, Department of State, Washington DC 88. PEPFAR (2012) Blueprint for Creating an AIDS-free Generation. OGAC, Department of State, XLIII Washington, DC 89. PEPFAR (2012) Workshop on ART in Pregnancy, Breastfeeding, and Beyond (presentation) 90. PEPFAR (2014) Swaziland Operational Plan Report FY 2013. Department of State, Washington DC 91. Siapka, M et al. (2014) Is there scope for cost savings and efficiency gains in HIV services? A systematic review of the evidence from low- and middle-income countries Bull World Health Organ; 92:499–511AD | doi: http://dx.doi.org/10.2471/BLT.13.127639 92. SINAN (2013) EPAS Budget - FINAL 93. SINAN EPAS+ YR1 Q1 & Q2 Reports 94. SINAN (2013) Year 1 Work PlanFinal_31072013 95. UNAIDS (2012) Promising practices in community engagement for the elimination of new HIV infections in children by 2015 and keeping their mothers alive. UNAIDS, Geneva, Switzerland 96. UNAIDS (2014) The Gap Report. Joint United Nations Programme on HIV/AIDS, Geneva, Switzerland 97. UNFPA (2011) Improving the Quality of Maternal and Neonatal Health Services in Swaziland: A Situational Analysis. GOKS, Mbabane, Swaziland 98. UNFPA (2013) State of the World’s Population 2013 - Motherhood in Childhood. United Nations Population Fund, New York 99. UNICEF (2012) Options B and B+: Key consideration for countries to implement and equity￾focused approach Draft for discussion. UNICEF, New York NY 100. UNICEF (2013) Swaziland Statistics. http://www.unicef.org/infobycountry/swaziland_statistics.html (accessed July 1, 2014) 101. United States Government (2011) Global Health Initiatvie Strategy Document, Department of State, Washington DC 102. United States Government (2012) Swaziland U.S. Global Health Initiative Country Strategy January 9, 2012. Department of State, Washington DC 103. USAID (2010) Final Cooperative Agreement No. 674-A-00-11-00009-00 EGPAF. 104. USAID (2010) Cooperative Agreement No . 674- A- 00- l l - 00009-00 EGPAF Attachment B XLIV Program Description. 105. USAID (2010) Request for Applications (RFA) No. 674-10-00040 Prevention of Mother to Child Transmission (PMTCT) Program in The Kingdom of Swaziland. USAID Southern Africa, Pretoria 106. USAID (2011) DQA 1_EGPAF_September_20-21_2011_Final Report. USAID, Mbabane, Swaziland 107. USAID (2011) Evaluation Learning from Experience. USAID Evaluation Policy. USAID, Washington, DC 108. USAID (2012 ) DQA 2_EGPAF. SZ_July 2012. USAID, Mbabane, Swaziland 109. USAID (2012) Hlathikhulu Hospital Site Visit. USAID, Mbabane, Swaziland 110. USAID (2012) PMTCT Site visit [Pigg’s Peak PHU]. USAID, Mbabane, Swaziland 111. USAID (2013) Modification of Assistance. #3 Award No . 674- A- 00- l l - 00009-00 EGPAF. 112. USAID (2013) Site visit Siteki PHU. USAID, Mbabane, Swaziland 113. USAID (2013) USAID Southern Africa Site visit to Mbabane PHU. USAID, Mbabane, Swaziland 114. USAID (2014) Site Visit Report [Mbikwakhe Clinic]. USAID, Mbabane, Swaziland 115. World Bank (2014) Service Availability Mapping Report 2013 (Final). MOH, Mbabane, Swaziland 116. World Population Review (2014) Swaziland Population 2014. http://worldpopulationreview.com/countries/swaziland-population/ (accessed Aug 5, 2014) 117. WHO (2007) Everybody’s business: strengthening health systems to improve health outcomes: WHO’s framework for action. World Health Organization, Geneva, Switzerland 118. WHO (2010) Monitoring the building blocks of health systems: a handbook of indicators and their measurement strategies. WHO Geneva, Switzerland 119. WHO (2008) Service Availability Mapping 2006-07. MOHSW & MOEd, Mbabane, Swaziland 120. WHO (2009) HIV and infant feeding Revised Principles and Recommendations Rapid Advice. WHO, Geneva, Switzerland XLV 121. WHO (2010) Service Availability Mapping Report 2010. MOH, Mbabane, Swaziland 122. WHO (2011) Swaziland Factsheets of Health Statistics 2011. WHO Regional Office for Africa 123. WHO (2012) Preventing mother-to-child transmission of HIV in Swaziland. http://www.who.int/features/2012/pmtct_swaziland/en/ (Accessed Aug 6, 2014) ANNEX VI: DISCLOSURE OF ANY CONFLICTS OF INTEREST Name Ruth HOPE Title Director RH/HIV/AIDS & Gender Equality Organization MIDEGO, Inc. Evaluation Position? X Team Leader Team member Evaluation Award Number (contract or other instrument) AID-EPAS01-001SW USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Eliminating Pediatric AIDS in Swaziland Elizabeth Glaser Pediatric AIDS Foundation Cooperative Agreement No. AID-674-C-14-0007 I have real or potential conflicts of interest to disclose. Yes X No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date: July 28, 2014 $11(;9,',6&/2685(2)$1<&21)/,&762),17(5(67 1DPH 7LWOH 2UJDQL]DWLRQ (YDOXDWLRQ3RVLWLRQ" 7HDP/HDGHU7HDPPHPEHU>GHOHWHDSSURSULDWHO\@ (YDOXDWLRQ$ZDUG1XPEHU (contract or other instrument) 86$,'3URMHFW V (YDOXDWHG (Include project name(s), implementer name(s) and award number(s), if applicable) ,KDYHUHDORUSRWHQWLDOFRQIOLFWV RILQWHUHVWWRGLVFORVH GHOHWHDSSURSULDWHO\@ ,I\HVDQVZHUHGDERYH,GLVFORVH WKHIROORZLQJIDFWV Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. 6LJQDWXUH ANNEX VI: DISCLOSURE OF ANY CONFLICTS OF INTEREST Name Celso Jeremias S. Mondlane Title Organization Evaluation Position? Team member Evaluation Award Number AID-EPAS01-01SW (contract or other instrument) USAID Project(s) Evaluated Eliminating Pediatric Aids in Swaziland (EPAS) Project (Include project name(s), implementer name(s) and award EGPAF number(s), if applicable) I have real or potential conflicts No of interest to disclose. If yes answered above, I disclose the following facts: Real or potential connicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. .. I cert1fy (1} that I have completed th1s disclosure form fully and to the best of my ab1hty and (2} that I w11l update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature f) f ( \ l \) \..1 )3· 01·o?OIY U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523