Evaluation of Postabortion Care Service Delivery in Malawi JHP-25 Prepared by Catherine Schenck-Yglesias, MHS March 2004 United States Agency for International Development technical report CREDITS Editor: Dana Lewison Production Assistance: Youngae Kim All photos were taken by the author. Copyright© 2004 by JHPIEGO Corporation. All rights reserved. ____________________________________________________________________________________ JHPIEGO, an affiliate of Johns Hopkins University, builds global and local partnerships to enhance the quality of health care services for women and families around the world. JHPIEGO is a global leader in the creation of innovative and effective approaches to developing human resources for health. ACKNOWLEDGMENTS The author would like to acknowledge Mrs. Maryjane Lacoste, JHPIEGO/Malawi Country Director, for her technical and conceptual guidance beginning in the study design phase and following throughout data collection, data analysis, and the writing of this report. Participants in the PAC Monitoring & Evaluation workshop in March 2002 in Lilongwe, Malawi, also contributed conceptually to the design of this evaluation. The assistance of the Government of Malawi, EngenderHealth, and staff of all hospitals visited in the evaluation is also greatly appreciated. Their role in communicating about the current status of PAC Program implementation, which is documented in this report, has been invaluable. The author would also like to thank the United States Agency for International Development (USAID)/Malawi for supporting this work. The evaluation team members listed below contributed to the fieldwork and initial compilation of the data for this study. EVALUATION TEAM MEMBERS Malawi Ministry of Health and Population Mr. John Gondwe Mr. Felicia Chawani Mr. B. Zuza Mr. Emily Karonga EngenderHealth Dr. F. Ndede Thanks also go to Ms. Elaine Yuan, JHPIEGO consultant, as well as Mr. Saleh Ahmed and Ms. Deborah Estep of the JHPIEGO/Baltimore office for their technical support in synthesizing the data for this report. TRADEMARKS: All brand names and product names are trademarks or registered trademarks of their respective companies. The Malawi PAC Evaluation and this report were made possible through support provided by the Service Delivery Improvement Division, Office of Population and Reproductive Health, Bureau for Global Health, U.S. Agency for International Development, under the terms of Award HRN-A-98-00041-00. The opinions expressed herein are those of JHPIEGO and do not necessarily reflect the views of the U.S. Agency for International Development. Printed March 2004 JHPIEGO Technical Report i TABLE OF CONTENTS LIST OF TABLES AND FIGURES ...............................................................................................ii ABBREVIATIONS AND ACRONYMS ........................................................................................iii EXECUTIVE SUMMARY ..............................................................................................................v BACKGROUND............................................................................................................................1 Needs Assessment Findings .....................................................................................................1 Malawi PAC Program Intervention Description..........................................................................3 METHODOLOGY .........................................................................................................................8 Risks........................................................................................................................................10 Benefits....................................................................................................................................10 Confidentiality Assurance ........................................................................................................10 FINDINGS...................................................................................................................................11 Advocacy & Policy Development.............................................................................................11 Organization of Services..........................................................................................................11 Provision of Materials & Equipment.........................................................................................11 Supervision..............................................................................................................................15 PAC Inservice and Preservice Training...................................................................................17 Community Mobilization & BCI ................................................................................................26 Increased Availability of PAC Services....................................................................................26 Increased Quality of PAC Services..........................................................................................28 Expansion of PAC Services.....................................................................................................44 DISCUSSION..............................................................................................................................45 CONCLUSION............................................................................................................................46 RECOMMENDATIONS ..............................................................................................................47 REFERENCES ...........................................................................................................................48 APPENDICES A. PAC Evaluation Study Questions and Linkages Between the JHPIEGO PAC Program Framework and the Malawi National PAC Program Objectives B. Results of T-Tests for Analysis of Knowledge Score Differences by Hospital Type, Training Type and Cadre C. Age Distribution of FP Acceptors among MVA Patients, by Method ii JHPIEGO Technical Report LIST OF TABLES AND FIGURES Table 1. Number of Respondents and Facilities Represented in Malawi PAC Evaluation.......................9 Table 2. Percentage of PAC Hospitals with PAC Supplies and Equipment on Hand.............................13 Table 3. Percentage of PAC Hospitals with PAC Training Materials......................................................18 Table 4. Percentage of Directly Observed PAC Service Providers that Satisfactorily Completed Postabortion Care Clinical Skills ...............................................................................................40 Table 5. Percentage of Directly Observed PAC Service Providers that Satisfactorily Completed Postabortion Verbal Anesthesia Skills ......................................................................................41 Table 6. Percentage of Directly Observed PAC Service Providers that Satisfactorily Completed Postabortion Family Planning Counseling Skills.......................................................................42 Table 7. Availability of Reproductive Health Referral Services and Linkages at PAC Hospitals............43 Figure 1. Rural Housing in Malawi .............................................................................................................1 Figure 2. MVA Procedure Room at a Malawi Hospital...............................................................................2 Figure 3. JHPIEGO Postabortion Care (PAC) Project Design Framework with Malawi Evaluation Areas Highlighted ..................................................................................................................................4 Figure 4. Malawi Population Distribution, Terrain, and Location of Hospitals in Phase I Interventions of Malawi PAC Program.......................................................................5 Box 1. National PAC Program Objectives..............................................................................................7 Figure 5. Percentage of PAC Trainers Who Rate Each Training Method as Useful ...............................22 Figure 6. Histogram of PAC Service Provider Knowledge Assessment Scores......................................23 Figure 7. Box Plots of PAC Service Provider Knowledge Assessment Scores, by Hospital Type, Training Type, and Cadre .......................................................................................................................24 Figure 8. Mean Section Scores on PAC Knowledge Assessment...........................................................25 Figure 9. Distribution in Length of Time Since PAC Training and Total Number of PAC Clients ............25 Figure 10. Geographic Distribution of Hospitals with PAC Services, by availability 24 hours per day, 7 days per week ....................................................................28 Figure 11. Percentage of Clinical Supervisors Reporting that These Topics were Included in Routine Counseling at their Facility........................................................................................................31 Figure 12. Client Perception of Pain Scale and Client Rating of Pain at Admission, during MVA Treatment, and Post-Treatment................................................................................................32 Figure 13. Percentage of Service Providers who Reported Managing PAC Patients’ Pain with Each Pain Management Regimen..............................................................................................................33 Figure 14. Percentage of Clinical Supervisors who Report these FP Methods are Available in the PAC Service Area..............................................................................................................................34 Figure 15. Percentage of Facilities with these FP Methods available 24 hours per day/7 days per week 35 Figure 16. PAC Caseload, Malawi PAC Program Hospitals, July–September 2002.................................36 Figure 17. Percentage of PAC Clients receiving FP counseling, Malawi PAC Program Hospitals, July– September 2002........................................................................................................................37 Figure 18. Percentage of PAC Clients leaving with an FP method, Malawi PAC Program Hospitals, July – September 2002........................................................................................................................37 Figure 19. Total number of FP acceptors among MVA patients, Malawi PAC program hospitals, July - September 2002........................................................................................................................38 Figure 20. Age distribution of MVA patients that left with a family planning method and did not accept a method, PAC program hospitals ...............................................................................................39 JHPIEGO Technical Report iii ABBREVIATIONS AND ACRONYMS AIDS Acquired Immunodeficiency Syndrome CHAM Christian Health Association of Malawi CMS Central Medical Stores D&C Dilation and Curettage DHMT District Health Management Team DHS Demographic and Health Survey EHP Essential Health Package EHPLMIS Essential Health Package Logistics Management Information System FP Family Planning HIV Human Immunodeficiency Virus IP Infection Prevention MMR Maternal Mortality Ratio MOHP Ministry of Health and Population MVA Manual Vacuum Aspiration OJT On-the-Job Training PAC Postabortion Care RH Reproductive Health RHLMIS Reproductive Health Logistics RHU Reproductive Health Unit RNM Registered Nurse-Midwife SDG Service Delivery Guidelines STI Sexually Transmitted Infection TIMS Training Information Monitoring System TRH Training in Reproductive Health (Project) USAID United States Agency for International Development iv JHPIEGO Technical Report JHPIEGO Technical Report v EXECUTIVE SUMMARY A postabortion care (PAC) needs assessment undertaken in 2000 by the Malawi Ministry of Health and Population Reproductive Health Unit, JHPIEGO, and EngenderHealth showed that systemic improvements were needed to improve comprehensive PAC services for women throughout the country. Previous study data had already shown that nearly a third of maternal deaths were due to complications of abortion, and the needs assessment confirmed deficiencies in the level of care available at the time for these patients at Malawi hospitals. In 2001, the Ministry of Health and Population (MOHP) in Malawi and JHPIEGO, through the United States Agency for International Development (USAID) Training in Reproductive Health (TRH) award, began to implement a project to introduce comprehensive PAC services on a broad scale to Malawi. The program components included: (1) advocacy and policy development, (2) PAC training, (3) provision of materials and equipment, (4) organization of services, (5) expansion of PAC services, and (6) supervision, in order to increase the availability, quality, and use of comprehensive PAC services throughout Malawi. This report presents the methodology and results of a process evaluation conducted by the MOHP RHU, JHPIEGO, and EngenderHealth in October 2002 to assess progress in achieving Malawi’s National PAC program goals at the initial 14 implementation hospitals. This evaluation was anticipated at the outset of the program and was considered part of the MOHP’s ongoing monitoring and quality assurance efforts. This evaluation had a descriptive study design with subjects from all available PAC facilities and the population of trained PAC providers. Data were collected on weekdays during the period 17–28 October 2002 by Malawian clinical trainers from the MOHP, as well as JHPIEGO and EngenderHealth evaluation staff. The subjects for this evaluation included the national PAC program coordinator, clinical supervisors at health facilities currently providing PAC services, clinical trainers and service providers standardized in 2001-2002, and their postabortion clients seen during the data collection period. In addition, institutional surveys were conducted assessing the quality of PAC facilities with regard to dedicated availability of space for manual vacuum aspiration (MVA) services, necessary equipment and supplies, as well as adequacy of the site for provision of on-the-job training. Findings showed that the National PAC Strategy had gone into effect in November 2001, and that PAC had been incorporated into the national Reproductive Health Service Delivery Guidelines (SDGs), formalizing PAC as a priority within the National Sexual and Reproductive Health Policy. However, the policy and SDGs reached a higher percentage of facility managers than line clinical staff. Stockouts of PAC equipment were a problem in many hospitals despite the availability of these supplies at the Central Medical Stores (ordering and transportation issues were cited). The external supervision system functioned such that hospitals generally received a half to full day visit from external supervisors either monthly or quarterly. These visits focused on improving clinical performance and addressing PAC supplies issues. Internal supervision did not always include direct observation of PAC providers with clients; however, onsite supervisors were successful in correcting insufficient infection prevention practices by some PAC providers. The PAC on-the-job training (OJT) program at hospitals was in progress at 13 of the 14 hospitals visited, and most sites had the proper training and reference manuals; however, nearly vi JHPIEGO Technical Report half of the hospitals lacked ZOE anatomical models for use in practicing the PAC procedure before working with clients. Once a provider completed the OJT program, the RHU would send a PAC trainer to the hospital to observe and certify that provider. At the time of the evaluation, clinical officers were the majority of PAC providers (57%), with the remaining (43%) being registered nurse-midwives. And at that time, more PAC providers had been trained on the job (56%) than in group-based courses (44%). PAC providers’ knowledge assessment scores ranged from 60–95%, with a median score of 85%. At the time of the evaluation, the median length of time post-training was 4 months, and providers had each treated a median of 8 PAC patients post-training. Caseload in some hospitals may have been low due to stigma in the surrounding community, creating fear among women in seeking postabortion care services. Only half of the hospitals visited had initiated any type of community activity related to PAC services. Often, clients waited 1–2 days after symptom onset before seeking treatment. One of the lessons learned in this evaluation is that community involvement activities should be initiated at the start of PAC program services in new hospitals. Hospitals generally had waiting times of over an hour for PAC clients to be initially assessed, and patients had to be assessed in two departments before being ushered into the PAC procedure room for MVA. Patients received different levels of treatment depending on when they arrived at the hospital, with those arriving late in the day or in the evening often being treated with MVA the following day. This patient flow pattern showed that postabortion cases were not yet consistently being treated as emergencies in all sites. Over 80% of PAC clients were counseled about the procedure that they underwent, and clinical supervisors reported that PAC counseling routinely included current diagnosis, other RH conditions, treatment options, and prognosis. Patients were generally pleased with the pain management that they received during the procedure, but some clinicians reported that patients experienced significant pain during MVA under the current pain management protocol. When observed providing the MVA procedure, over 90% of PAC providers correctly performed each step of the preparation and MVA, procedure, and over 80% performed the pain management protocol correctly. Family planning (FP) counseling was integrated into the PAC patient flow, often with counseling occurring in the MVA room immediately following the procedure. On a total client basis, the Malawi PAC program—across hospitals—provided FP counseling to 74% of MVA clients and 40% of D&C clients in the quarter preceding the evaluation. Condoms, oral contraceptives and Depo-Provera were available in the PAC service area of all but two hospitals. PAC providers who were observed completed more than 70% of the FP counseling protocol steps according to standard. PAC clients interviewed post-procedure appreciated the patient education they received in addition to the empathetic life-saving treatment. Recommendations based on the findings of this evaluation addressed: 1. Strengthening of facility-level SDG orientation and dissemination efforts 2. Assuring an ongoing adequate supply of ZOE models and trainee logbooks at all PAC on￾the-job training sites 3. Integrating referrals to additional services such as cervical cancer screening or gender violence counseling (as they become available locally) into PAC counseling and patient education sessions 4. Assuring that all PAC facilities are using a routine PAC supply ordering system, such as the Essential Health Package Logistics Management Information System (EHPLMIS), and that JHPIEGO Technical Report vii they have access to adequate transport to get supplies from the Central Medical Stores to the facility 5. Supporting the geographic expansion of PAC services with parallel expansion of the supportive system for training, supplies, monitoring, supervision, and community involvement 6. Standardizing external supervision visits, encouraging more involvement of internal clinical supervisors, and capacity building of both external and internal supervisors in supportive supervision and coaching skills The results of this evaluation informed mid-course decisions about the program before scaling PAC services up to additional hospitals in 2003. In 2003, the PAC program was scaled up to an additional 18 hospitals, and activities are underway to support further expansion in 2004. viii JHPIEGO Technical Report JHPIEGO Technical Report 1 Evaluation of Postabortion Care Service Delivery in Malawi BACKGROUND Malawi is a sub-Saharan African nation with a population of 11.6 million (U.S. Census Bureau International Database) and has the third highest maternal mortality ratio (MMR) in the world, after Afghanistan and Sierra Leone. The life expectancy in Malawi is 39 years, and more than 60% of the population lives in poverty. Eighty-five percent of the Malawi population lives in rural areas. A recently completed set of global estimates adjusted the Demographic and Health Survey (DHS) estimated MMR for Malawi of 1,120 up to 1,800 per 100,000 live births for the period 1994-2000 (AbouZahr and Wardlaw 2003). Malawi has relatively low use of modern family planning methods at 26% of currently married women (Malawi National Statistical Office and ORC Macro 2001). In Malawi, maternal mortality related to postabortion complications is estimated to be 30% (Mtimvalye 1996); however, there are no research results that show whether the complications are from induced or spontaneous abortions. The unmet need for family planning among married women is 30%. Fifteen percent of the population aged 15 to 49 years is HIV-positive (UNAIDS 2002). Needs Assessment Findings In 1999, a postabortion care (PAC) needs assessment at the four central hospitals was carried out in Malawi by a collaborative team including the Reproductive Health Unit (RHU) of the Ministry of Health and Population, the national medical and nursing councils, and JHPIEGO. Adolescents represented at least 50% of all postabortion cases in each of the hospitals visited. Staff reported that “many abortions were induced by traditional healers using herbs, sticks and leaves, among other methods.” (Lacoste 2000) Providers reported the following complications in incomplete abortion cases in Malawi: severe bleeding, shock, retained products of conception, sepsis, and internal injury. The standard of treatment for incomplete abortion at that time was sharp curettage or evacuation. Most hospitals offered only these services, because their staff had not been trained formally on the less invasive manual vacuum aspiration (MVA) procedure and MVA supplies were not readily available. Dilation and curettage (D&C) is an invasive procedure for the patient and has to be performed under anesthesia in an operating theatre under sterile conditions and by a clinical officer, thus representing a high-resource treatment for the health system. Figure 1. Rural housing in Malawi 2 JHPIEGO Technical Report The key findings from the needs assessment included: (1) no PAC clinical standards in place, (2) lack of equipment, (3) infection prevention improvements needed, (4) lack of standard PAC curriculum for preservice education or inservice training, (5) weak linkages between school and clinic, and (6) new preservice graduates had to attend family planning (FP) inservice courses and be certified by the RHU before they could provide FP services. There were particular concerns about infection prevention from the PAC assessment team, with one report noting: “theatre staff use Jik for decontamination when it is available. When it is available, instruments are decontaminated, cleaned and sterilized—when it is not available, the decontamination step is skipped. Infection prevention needs a lot of work at this hospital.” Client flow was also an issue and interacted with supply issues. For example, in one hospital assessors reported: “no more than 5 cases can be done daily as there are only enough instruments to conduct 5 procedures. Those who come in after the 5 cases are done must wait until the next day for the evacuation. All instruments are autoclaved centrally—there is no high level disinfection by boiling.” One hospital had basic supplies but was reported not to be ready to handle a patient who went into shock/hemorrhage because it lacked emergency resuscitation equipment. Human resource issues also played a role in client flow: “FP counseling is frequently provided to [PAC] patients prior to discharge by a nurse counselor on the gynecology ward. There is only one nurse who provides this service, so PAC patients must sometimes wait for this nurse to come back on duty to get FP counseling… only one registered nurse/midwife has been trained in PAC counseling a year and a half ago.” Prioritization was also important, and perhaps a result of stigma from hospital staff: “MVA is not currently done as women come in—cases are kept until a scheduled time—abortion is not considered an emergency. As a result of this, every patient gets infected and needs antibiotics, and every patient is treated under Pethadine, a narcotic analgesic.” And at another hospital, assessors found that “women who come with incomplete abortion are given low priority. There are no evacuation services provided over the weekend. Women are scheduled for evacuation on either Tuesdays, Wednesdays or Thursdays.” Needs assessment interviews revealed that Ipas had supported the introduction of MVA into Malawi in 1993 through the supply of kits and training materials at Queen Elizabeth Central Hospital in Blantyre. However, this was a one-time activity at only one site. More recently, Dr. Vollert, an obstetrician/gynecologist at Lilongwe Central Hospital, had obtained support from German donors to provide MVA supplies for all districts for 2 years ending in 2000. Unfortunately, this procurement was not matched with PAC training inputs across all districts. On the positive side, Dr. Vollert was personally able to provide 1-day training sessions to district health officers, clinical officers, and registered nurse-midwives at selected district hospitals, which was of benefit to patients at these sites. The PAC Needs Assessment made clear to the MOHP Reproductive Health Unit that any PAC intervention would need to take a systems approach to address the above findings that were encountered in Malawi in 2000. Figure 2. MVA Procedure Room at a Malawi Hospital JHPIEGO Technical Report 3 Malawi PAC Program Intervention Description Since 2001, the Ministry of Health and Population (MOHP) in Malawi and JHPIEGO, through the United States Agency for International Development (USAID) Training in Reproductive Health (TRH) award, have been implementing a project to introduce comprehensive postabortion care (PAC) services on a broad scale to Malawi. The ultimate aim of the program is to improve the quality of emergency care, family planning, and reproductive health services provided to postabortion care patients and to increase the availability and accessibility of these services throughout the country. Program activities included the development of a National PAC Strategy, National PAC Service Delivery Guidelines, adaptation of a training package, group￾based clinical training of service providers, and development of on-the-job (OJT) trainers, using a competency-based training methodology. Figure 3 illustrates the JHPIEGO postabortion care project framework. The components shown in bold have been implemented in Malawi, and were all the subject of the process evaluation in 2002 that is reported herein. The Malawi PAC Program is being implemented in two phases. Service providers and OJT trainers from 14 hospitals received competency-based training in 2001–2002. The four central hospitals received training from JHPIEGO while the 10 additional hospitals received training from EngenderHealth using the nationally adapted PAC training package (based on the JHPIEGO PAC training package). In 2003, the program began scaling up to 18 additional hospitals (8 and 10 with support from JHPIEGO and EngenderHealth, respectively), based on lessons learned from the 2002 process evaluation. Up until October 2002 when this evaluation took place, PAC clinical services had been implemented in the above-mentioned 14 hospitals, which were located throughout 13 of Malawi’s 27 districts. See Figure 4 for maps showing the location of these PAC facilities in Malawi. The purpose of the evaluation was to determine if the National PAC Program’s objectives were being met at that stage of implementation, and if they weren’t yet met, what areas required intervention and improvement. Specifically, the evaluation team wanted to assess the advocacy and policy development, organization and expansion of services, supervision system, provision of materials and equipment to facilities, on-the-job training, availability of PAC services, job performance of trained providers, and quality of care from the client perspective. Improved clinical performance has been found to increase clinic utilization, client satisfaction and clients’ use of reproductive health and family planning services (Bruce 1990; Sullivan 1995). This evaluation was anticipated at the outset of the program and was considered part of the MOHP’s ongoing monitoring and quality assurance efforts. 4 JHPIEGO Technical Report Figure 3. JHPIEGO Postabortion Care (PAC) Project Design Framework with Malawi Evaluation Areas Highlighted 5 JHPIEGO Technical Report Population Distribution, 1998 Four Central Hospitals Where JHPIEGO Provided PAC Training Ten Hospitals Where EngenderHealth Provided PAC Training Using National PAC Training Materials, Based on the JHPIEGO PAC Training Package Figure 4. Malawi Population Distribution, Terrain, and Location of Hospitals in Phase I interventions of Malawi PAC Program 6 JHPIEGO Technical Report With USAID funds, JHPIEGO has provided technical assistance to the Malawi MOHP’s Reproductive Health Unit: to incorporate PAC into the national reproductive health (RH) policy; to develop a national PAC strategy, clinical practice standards, and training materials (both group-based and on-the-job); and has directly provided group-based clinical and training skills courses to clinicians working in four central hospitals. Also funded by USAID and in collaboration with the Malawi MOHP and JHPIEGO, EngenderHealth—another USAID cooperating agency with headquarters in the U.S.—has provided group-based PAC training, using the same instructional materials, to clinicians in 10 district hospitals. All training was conducted using a competency-based approach, which involves providing clinical trainers updates on standards, guidelines, and PAC clinical skills (Sullivan 1995). The trainers use an interactive, adult-learning methodology; students practice their clinical skills on anatomic models, and only after gaining competency on models begin to work in clinics with actual clients under trainer supervision. Students are considered competent (based on a skills assessment checklist and knowledge assessment questionnaire) when they successfully complete a number of procedures with real clients; they are observed and mentored throughout by clinical trainers. All training participants also received clinical training skills strengthening and were expected to return to their facilities to provide on-the-job training to other PAC service providers. Box 1 lists the Malawi National PAC Program objectives. Each of the main Malawi program areas is linked to the JHPIEGO PAC Framework in Box 1 by listing the corresponding level in parentheses. For example, Malawi’s goal of “increasing the accessibility of PAC services in an integrated sexual and reproductive health program” links to JHPIEGO’s framework Level C, “demand and capacity to deliver services.” The expectation was that, as of October 2002, the national PAC program would have provided the foundation for the success of the program by their work in: ♦ Advocacy and policy development ♦ Organization of services ♦ Supervision systems ♦ Provision of materials and equipment ♦ PAC inservice training JHPIEGO Technical Report 7 Box 1. National PAC Program Objectives To raise awareness of the magnitude of the problem of incomplete abortion and its complications and the availability of PAC services (Level D) 1. Establish task force to manage and coordinate PAC programme strategy development, programme design and implementation 2. Develop and disseminate a comprehensive PAC policy as one component of the national Sexual and Reproductive Health (SRH) policy 3. Increase PAC knowledge/awareness of all cadres of staff working in sites where PAC services are offered 4. Ensure involvement of community in design, development and delivery of PAC services Establish sustainable system for providing PAC services (Level D) 1. Strengthen PAC component of preservice education in Colleges of Medicine, Nursing, and Health Sciences 2. Ensure necessary equipment and supplies for PAC services are incorporated into national- and district-level procurement/requisition systems (e.g., Reproductive Health Logistics Management Information System) Increase accessibility of PAC services in an integrated SRH program (Level C) 1. Increase the number of service providers 2. Improve the availability of equipment, supplies, and drugs necessary for PAC services 3. Ensure PAC services are delivered in the most timely manner possible at the service delivery point Provide quality PAC services (Levels B and C) 2. Ensure that PAC services are offered by trained and competent service providers 3. Establish and maintain a safe environment for delivery of PAC services 4. Ensure that the appropriate technology is used in delivery of PAC services 5. Ensure linkages are made to other SRH services 6. Monitor provider performance through existing supervision systems to ensure continued quality of care The present evaluation assessed the program’s progress in these areas. The evaluation team aimed to assess the extent to which facilities where the PAC program had been implemented through October 2002 would be performing to standard in several areas, to meet national PAC program objectives: ♦ Availability of trained providers ♦ Materials and equipment for PAC emergency care, infection prevention, counseling and family planning service provision ♦ On-the-job training (OJT) ♦ Supervision system ♦ Quality of care Individual PAC providers at each facility were expected to be performing to standard on PAC: ♦ Knowledge ♦ Clinical skills ♦ Counseling skills The evaluation team also assessed client: ♦ Satisfaction with PAC service ♦ Reasons for non-acceptance of family planning methods 8 JHPIEGO Technical Report The evaluation assessed each of the areas above in order to find areas that needed strengthening before the program was scaled up to another 8 hospitals in 2003. METHODOLOGY This evaluation had a descriptive study design with subjects from all available PAC facilities and the population of trained PAC providers. Data were collected on weekdays during the period 17–28 October 2002 by Malawian clinical trainers from the MOHP, as well as JHPIEGO and EngenderHealth evaluation staff. Prior to data collection, data collectors were trained on the evaluation’s goals, protocol, and instruments. Clinician evaluators were trained in the use of the clinical checklists and standardized on their observation and recording skills. The seven data collection instruments were: 1. PAC National Program Coordinator Interview Guide 2. PAC Clinical Supervisor Interview Guide 3. PAC Facility Assessment 4. PAC Service Provider Knowledge Assessment 5. PAC Service Provider Clinical Assessment 6. PAC Service Provider Interview Guide 7. PAC Client Interview Guide All instruments are available upon request from the JHPIEGO Research and Evaluation Office. The subjects for this evaluation included the national PAC program coordinator, clinical supervisors at health facilities currently providing PAC services, clinical trainers and service providers standardized in 2001-2002, and their postabortion clients seen during the data collection period. In addition, institutional surveys were conducted to assess the quality of PAC facilities with regard to dedicated availability of space for MVA services, necessary equipment and supplies, as well as adequacy of the site for provision of on-the-job training. The MOHP RHU Training Information Monitoring System (TIMS©) database was used to provide JHPIEGO with a list of the names and current facility addresses of all PAC clinical trainers and service providers. The evaluation area included multiple districts spread across Malawi. The Malawi MOHP RHU sent administrators of facilities selected for inclusion in the evaluation a letter requesting that, on a pre-assigned date, PAC clinical supervisors, service providers, and clients be available at their worksite to participate in the evaluation. They were informed about the purpose of the evaluation and the composition of the evaluation team. A followup phone call was made to all hospital administrators before the arrival of the team. Client interviews were completely voluntary. The evaluation team used consent forms onsite with each client to be interviewed. Each client’s verbal response was noted on the consent form and signed and dated by the interviewer. Obtaining verbal consent is the preferred method in Malawi because people often do not want to sign forms they believe may be shared with the government. In addition, only 48.6% of Malawian women are literate (National Statistical Office 2001), which would have posed a challenge for the clients to read and understand the written consent form. (The verbal consent forms used are available upon request from the JHPIEGO Research and Evaluation JHPIEGO Technical Report 9 Office.) In addition, service providers and administrators were assured that their individual responses would be used for aggregate data analysis only, so they were unlikely to adversely affect the employment of the participants. The evaluation data collection was held in the PAC unit of each hospital, including the MVA procedure room(s), counseling area, gynecological ward, and training areas. A room was requested at each site, to be set aside for knowledge assessments and interviews. Only trained PAC clinician evaluators observed clinical provider subjects. This observation included all of the following: patient assessment and assignment to treatment (MVA or dilation and curettage [D&C]), MVA and counseling about the procedure, family planning counseling, and referrals for other reproductive health services. Completing interviews took each clinical supervisor and clinician no more than 1 hour. Completing skills checklists for patient care services lasted the actual duration of treatment, usually less than 2 hours. Clients were asked to participate in an interview to assess their level of satisfaction with the PAC services they received. Completing the client exit interview took no more than 30 minutes; Malawian clinicians conducted all interviews in the local language, Chichewa. Consent forms were used for voluntary participation of all clients for clinical observation and exit interviews. Table 1 shows the number of respondents and hospitals for which data on each data collection instrument were collected and analyzed. All PAC program hospitals were visited (N=14), but one hospital was not prepared for the evaluators and thus very little data were obtained at that site. Not all facilities had clients and providers that could be readily observed. For example, 12 PAC service providers from three central hospitals and five district hospitals were directly observed providing an MVA procedure to at least one client, so 12 respondents and eight facilities are shown in the PAC Clinical Assessment Checklist row of Table 1 below. Table 1. Number of Respondents and Facilities Represented in Malawi PAC Evaluation Data Collection Instrument/ Respondent Type Instrument Type Number of Respondents Number of Facilities PAC National Program Coordinator Interview Guide 1 NA PAC Facility Assessment (reviewed with clinical supervisor or service provider, based on staff availability) Observation Checklist and Interview Questions 14 14 (13 for most questions) PAC Clinical Supervisor Interview Guide 13 13 PAC Service Provider Interview Guide 26 13 PAC Service Provider Knowledge Assessment Self-Administered Multiple Choice Assessment 29 13 PAC Clinical Assessment Checklist (to assess PAC service provider) Observation Checklist 12 8 PAC Client Interview Guide 13 9 Data analysis included descriptive and multivariate analysis of items on data collection instruments, which were both quantitative (analysis in SPSS) and qualitative (the JHPIEGO evaluator read through all answers and reported on themes encountered). Appendix A shows the linkages between the JHPIEGO PAC Program Framework shown in Figure 3 and Malawi National PAC Program objectives and evaluation questions. These analyses, presented in this report, inform the program whether national objectives are being met, and indicate areas of 10 JHPIEGO Technical Report strength and weakness to be used to improve the program when scaling up. Quantitative analyses shown in this report were calculated using SPSS software; missing responses were excluded for percentage calculations. Risks There were few risks involved in participation in this evaluation. The evaluation involved observation by proficient clinical trainers of routine PAC clinical interventions being performed by staff clinicians at 14 MOHP district and central hospitals throughout Malawi. In addition, trained evaluators performed interviews of clinical supervisors, clinicians, and clients. No explicit, embarrassing or sensitive questions (e.g., number of sex partners, whether the abortion was spontaneous or induced, etc.) were included in any of the questionnaires. All questionnaires and checklists were pretested for content by local experts to avoid any unintended culturally and personally sensitive questions. Every effort was made to complete the evaluation activities as quickly and efficiently as possible. No monetary compensation was given to respondents in the evaluation. The participants’ involvement in the evaluation (i.e., their checklists and interview responses) are not being shared with any supervisor or worksite. Participants will not be adversely affected for promotion or any other job-related evaluation at the work site as a result of participating in this evaluation. Benefits There were direct benefits to PAC service providers and their supervisors from participating in this evaluation. After completing the interviews, clinical supervisors received a verbal summary of strengths and weaknesses of the site’s PAC services from the clinician evaluator. After completion of the skills observation sessions, the clinical trainers reviewed the clinical service providers’ performance and provided input on strengths and opportunities for improvement that might help the clinicians to improve their skills, or feel more confident about their current practice. Other benefits of this evaluation for all participants are in the long-term, as results will be used to inform future MOHP Reproductive Health Unit system strengthening and capacity building programs in order to prepare PAC service providers for better job performance. This is aimed at higher quality postabortion care including family planning, HIV/AIDS/STI care, and referrals for other reproductive health services, and increased utilization of facilities where PAC is available. Confidentiality Assurance Each data collection form included a unique identifier code for each respondent, and on client interview forms, a provider code was also included. Respondent and provider names were not recorded on the data forms. Respondents and their facilities were identified in the electronic data set by codes. Conducting interviews in private, training interviewers and observers appropriately, and providing adequate field supervision served to protect confidentiality in this evaluation. Access to completed data collection forms was limited to evaluation staff. The forms were stored in a safe place at each evaluation site during the process of data collection and for the duration of the field visits, and were then used for data entry and analysis by EngenderHealth and JHPIEGO evaluators. Copies of forms and electronic data files are JHPIEGO Technical Report 11 archived at JHPIEGO’s country office in Lilongwe, Malawi, and corporate office in Baltimore, Maryland. FINDINGS Findings are presented below by program component. Data for many program components are presented from several vantage points, for example the national perspective as well as client and service provider perspectives, based on the data collected when interviewing each level of actor in the system. In addition, some components are represented by direct observation of skills and supplies in addition to reported evidence from interviews. This is to provide the Malawi Ministry of Health and Population as comprehensive a view of the Malawi PAC Program as possible within the boundaries of a descriptive process evaluation study. Advocacy and Policy Development The PAC National Program Coordinator reported that a national PAC strategy had been created for Malawi and that it went into effect in November 2001. The strategy was disseminated in a stakeholder meeting, and copies were also sent to all PAC facilities. Recipients of the strategy included UN agencies, Christian Health Association of Malawi (CHAM) facilities, USAID, Save the Children, Umoyo Networks, and all preservice training institutions in Malawi. The National PAC Strategy is also available to the implementing MOHP departments, CHAM and all implementing facilities from the MOHP RHU. PAC is incorporated into the national Reproductive Health Service Delivery Guidelines, thus formalizing PAC as a priority within the National Sexual and Reproductive Health Policy. Malawi’s PAC service delivery guidelines promote a linkage to family planning, HIV/AIDS services, sexually transmitted infection (STI) clinics and infertility services. While the PAC program was being implemented, at the time of the evaluation, a National PAC Task Force had not yet been created. At the facility level, 77% of clinical supervisors reported that management personnel at their facility had been briefed on the National PAC Strategy and service delivery guidelines, while 62% reported that staff had been oriented on this strategy and the guidelines. Organization of Services The National Program Coordinator reported that PAC service delivery guidelines had been disseminated along with the strategy to the audiences mentioned above. She also reported that national efforts had been made to link PAC to such reproductive health services as STI screening and treatment and infertility services, but that others, such as cervical cancer screening and gender violence counseling, were not yet available in Malawi on a wide scale or at all in some districts. Provision of Materials and Equipment 1. PAC materials and equipment from central perspective The National Program Coordinator reported that all PAC supplies and equipment, with the important exception of MVA kits, were on the national essential equipment list. To address the issue of MVA kits, the MOHP sent a letter to the Central Medical Stores (CMS) to request that they stock them. The national coordinator reported that the CMS had adequate stocks at the 12 JHPIEGO Technical Report time and that the items at the CMS adhered to the PAC service delivery guidelines. As of the writing of this report, MVA kits have now also been included as an essential item in the national Essential Health Package. The Reproductive Health Logistics Management Information System (RHLMIS) includes family planning commodities and STI drugs that can be used as part of PAC service delivery and linkages to other RH services. 2. PAC materials and equipment from facility perspective a. Staff knowledge of standard list of PAC supplies At the facility level, 85% of the service providers interviewed reported knowing the standard list of PAC supplies for the site (they were not quizzed by evaluators to verify their knowledge of the contents of the list), but only 67% reported having a paper copy of the equipment list available to them (they were not asked to show the interviewer that list). b. Stockouts of PAC equipment Thirty-nine percent of clinical supervisors interviewed said they had experienced a stockout of PAC supplies during the past 6 months (46% had not had a stockout and 15% didn’t know). All of those who experienced a stockout were from district or mission hospitals, indicating that the availability of supplies is better for central hospitals, perhaps due to their location in urban areas. The issues surrounding the stockouts were systemic. Supervisors reported: “The hospital is not buying any Jik even though the PAC trainer has asked the medical superintendent for it. He says it will be ordered but none has shown up for almost a year.” In another hospital, procuring bleach was difficult—the supervisor stated that it is ordered from the stores and it is not there due to the financial crisis. Yet another hospital reported difficulty in obtaining Jik because there was no transport available to the Central Medical Stores. Gloves were also reported as low in supply and sometimes of poor quality. In addition to the above infection prevention supplies, the supervisors who had experienced stockouts had run out of MVA syringes, cannulae, curettes, and dilators. Some reported that the equipment was old and not available in sufficient quantities. c. Accessibility, adequate storage, and resupply systems in place for supplies and equipment Unfortunately, only 31% of the clinical supervisors interviewed reported that a system was in place for the monthly resupply of PAC equipment, to prevent stockouts. Eight percent said there was no system and 61% didn’t know. When asked to further explain details on how PAC equipment is ordered at their sites, all 13 clinical supervisors had an answer: 69% reported some variation on ordering through the Central Medical Stores and their pharmacy system; 31% reported that they get their PAC supplies directly from the RHU or that they were still using the supplies given to them by the RHU at training or on supervision visits and did not yet know how to order otherwise. JHPIEGO Technical Report 13 Table 2. Percentage of PAC Hospitals with PAC Supplies and Equipment on Hand Item % of PAC Hospitals with Supply on Hand The following instruments, equipment, and supplies are needed for PAC: Bivalve speculum, medium or large 100% Uterine tenaculum or vulsellum forceps 100% Sponge or ring forceps (2) 100% Kidney dish 100% 10–12 ml syringe and 22-gauge needle, for paracervical block (or ordinary 21-gauge needle) 100% MVA instruments including MVA vacuum single or double syringes, flexible cannulae of different sizes, adapters for double valve syringes and silicone for lubricating MVA syringe O-ring 100% Light source, to see cervix and inspect tissue 92% Swabs/gauze 100% Antiseptic solution, preferably an iodophor or any locally available skin antiseptic 100% Gloves, sterile or high-level disinfected surgical gloves or new examination gloves 100% Utility gloves 77% Strainer, for tissue inspection 31% Clean container or basin, for tissue inspection 92% These items should be on hand, but are not required for all MVA procedures: Sharp curettes: small, medium, and large 92% Tapered mechanical dilators, Pratt (metal) or Denniston (plastic) 100% The treatment room should have the following furniture and equipment in working order: Examination table with stirrups 100% Strong light 100% Seat or stool for clinician (optional) 100% Plastic buckets for decontamination solution 100% Puncture-proof container for disposal of needles 100% Leakproof container for disposal of infectious waste 92% For high-level disinfection or sterilisation of instruments, these items should be available: High-level disinfection solution, e.g., 0.5% chlorine 92% Heat source and pot if HLD by boiling is chosen 92% Steamer 67% Autoclave (steam) or convection oven (dry heat) 100% Clean water for washing instruments or for HLD 100% HLD water to rinse HLD instruments 100% Sterile water to rinse sterile instruments 92% Detergent 92% Nonmetal or plastic containers 92% Utility gloves 83% Small scrub brushes 92% These items are seldom required in uterine evacuation cases but are needed for possible emergency use: IV infusion equipment and fluid (DSW or D/S) 92% Ambu bag with oxygen (tank with flow meter) 85% Oral airways suction machine 85% Essential drugs for emergency postabortion care include the following: Sedatives Diazepam 100% Anaesthetics, local Lignocaine, 1% without epinephrine 100% Lidocaine 100% 14 JHPIEGO Technical Report Item % of PAC Hospitals with Supply on Hand Analgesics Acetylsalic acid 100% Ibuprofen 77% Pethidine (or suitable substitution) 100% General anaesthetic Ketamine hydrochloride 100% Halothane 92% Trilen 77% Thiopentone sodium 92% Broad spectrum antibiotics Ampicillin 92% Chloramphenicol 100% Metronidazole 92% Sulfamethoxazole 100% Sulfamethoxazole-trimethoprim 100% Tetracycline 92% Other appropriate substitute (combining antibiotics is recommended) 85% Blood products Dried human plasma 54% Haematinics 62% Iron 100% Antiseptics Chlorhexidine, 4% (Hibitane, Hibiscrub) 85% Iodine preparations, 1-3% 92% Iodophors (Betadine) 54% Disinfectants Sodium hypochlorite, 5-10% (commercial-based solution, e.g., Jik) 85% Formaldehyde, 8% (Formalin) 85% Glutaraldehyde, 2% (Cidex) 85% Oxytocics Ergometrine injection 100% Oxytocin injection 100% Intravenous solutions Sodium lactate (Ringer’s) 100% Glucose, 5% and 50% 100% Glucose with isotonic saline 77% Potassium chloride 85% Sodium chloride 100% 5% Dextrose 100% Normal Saline 100% Water for injections 100% Other Tetanus Toxoid 100% Atropine 92% Adrenaline 100% Hydrocortisone 92% Plasma expanders (Dextran) 77% Antihistamine 100% Source: Facility Assessment Checklists. JHPIEGO Technical Report 15 d. Staff need for equipment, supplies, and infrastructure Facility assessment data helped the evaluation team to round out the data reported on interviews. Eight-five percent of hospitals visited kept an inventory of supplies. All hospitals assessed complied with the following three equipment, supplies, and infrastructure standards: ♦ Staff always observe the first-expired, first-out (FEFO) rule for stocks that expire. ♦ Drugs and other supplies are stored in a manner that ensures good preservation (for example, away from water, heat, or pests). ♦ There are handwashing facilities near procedure and exam rooms. e. PAC supplies and equipment onsite observation and availability All PAC facilities were equipped with at least one consultation section and were adequately lit. However, one hospital was not equipped with a storage cabinet for contraceptives. In the PAC treatment room, all facilities had an instrument tray/table, stethoscope, blood pressure machine, thermometer, and blood collection equipment/supplies, including vials and 2-3 ml disposable syringes. However, two hospitals lacked an emergency tray and nearly 70% of hospitals did not have eye protection (goggles). See Table 2 for a list of supplies and the percentage of hospitals that had these in stock on the day of the evaluation visit. f. Use of family planning and STI/HIV supplies Only in half of facilities visited did evaluators obtain the number of FP commodities on hand. The range in number of available commodities was very large. For example, the number of condoms available in stock in hospitals ranged from 100 to more than 175,000. There was a range from 10 to 4,000 packets of oral contraceptives (pills) in stock. Depo-Provera ranged from 10 to 3,100 doses on hand. Hospitals had from 36 to 65 Norplant implants, and from 5 to 30 IUDS available. This variation could have also resulted from some evaluators obtaining the number of supplies available in the direct PAC procedure and counseling area, with others obtaining the number of commodities on hand in the hospital’s FP method or pharmacy storage area. g. Availability of family planning patient education materials Only half of the facilities were assessed for the number of family planning patient education materials on hand. For these hospitals, one hospital had no flipcharts or posters, and the others ranged from having 1–30 of these materials on hand. Four hospitals had no FP pamphlets or instruction sheets on hand to give to patients to take home. The remaining three hospitals had 11, 20, and 100 FP pamphlets on hand. No hospital had FP clinic referral slips available to hand to clients, even though the staff stated that FP referral was done as part of comprehensive PAC service provision, indicating that referral slips may not be part of the normal patient flow in Malawi. Supervision 1. National supervisory system in place for PAC a. Components The National PAC Program Coordinator reported that there is a centrally coordinated system for regular PAC followup via supervisory visits. That system covers clinical performance, on-the-job 16 JHPIEGO Technical Report training, equipment/supplies, linkages to other reproductive health services, and other administrative issues. b. Training of supervisors The coordinator noted a need for supervisors to be trained in facilitative supervision in order to serve in this capacity for the National PAC Program. Facilitative supervision is a methodology that includes coaching and problem solving toward good performance rather than reprimanding when supervisors arrive at facilities and see shortcomings. c. Data collection and dissemination of findings Also according to the National PAC Program Coordinator, supervisors need to visit each PAC facility once every 2 months. Information is collected using a standard checklist and questionnaire at each site visit. At this time, debriefing is done to each District Health Management Team (DHMT) after each visit, but there is an intention to disseminate the findings to each PAC facility in the future. 2. Number and characteristics of external supervisory visits to each site a. Numbers of external supervisory visits, dates, and time Only one of the clinical supervisors interviewed reported that his hospital had not yet received a supervision visit. Six of the hospitals had already received three or four supervisory visits, and the remaining hospitals had all had one or two visits. Those hospitals with more visits said they occurred monthly, while those with fewer reported more of a quarterly pattern. The amount of time the followup visits took ranged quite a bit. The longest reported visits were to one central hospital, for 1 to 1½ days, while the shortest reported time was at a district hospital, where supervisors stayed 1 hour. Five hospitals reported 4-hour supervisory visits and four hospitals reported visits of 6–8 hours. b. Problems encountered External supervision visits assisted hospitals in improving clinical performance. Clinical supervisors reported that supervisors noted that the hospital clinicians were not using sterile gloves, not draping the patient, and not using the five-swab technique. These problems were all addressed using a coaching style, and clinicians were referred back to what they had been taught in class and to what was written in the PAC training manual. Infection prevention practices and supplies were addressed on visits, with problem solving on procurement at specific hospitals. At hospitals where external supervisors perceived low caseloads, they encouraged staff to start some community motivation work, for example, meeting with traditional chiefs and the community at large. At one hospital, the followup visit was focused on “getting the on-the-job training program off the ground” as it had not yet begun as planned. c. Followup visits Three-quarters of service providers interviewed reported that their facility had received external PAC supervision visits. Most reported these were quarterly visits, with only three providers reporting monthly visits. Service providers said that the visits were useful for knowledge and practice updates, and because the supervisors brought supplies such as MVA syringes and instruments, family planning books and trainee checklists. Specific feedback received assisted in improving infection prevention, the use of gloves and drapes, as well as encouraging providers to treat PAC patients as emergencies. Learning materials such as books and videotapes also arrived with the supervisors. One hospital reported still waiting for an JHPIEGO Technical Report 17 anatomical model, however. Several respondents noted that the supervisors provided them with a PAC logbook for recordkeeping. 3. Internal supervisors a. Amount of onsite supervision The reports on internal supervision at the PAC hospitals showed that it differed greatly from the nature of external supervision. Some clinical supervisors reported that their clinical staff largely worked on their own while others said they work together every day on the ward (presumably meaning that this is ample time for supervision). One supervisor reported possibly having observed one of his staff provide one MVA procedure to a client. b. Aware of PAC responsibilities Even though only 62% of clinical supervisors said that their staff had clear job descriptions that included PAC services, nearly all (96%) of PAC service providers reported that their supervisors understood their responsibilities for providing PAC services. c. Observation and provision of coaching/feedback Only 38% of clinical supervisors surveyed reported taking the time to observe their PAC clinical staff with clients in order to provide feedback. However, 70% of PAC service providers reported receiving coaching or feedback from their internal supervisors on their treatment of PAC clients. 4. PAC clinical staff job performance problems encountered Four of the clinical supervisors reported having a positive experience supervising their PAC providers: ”so far, so good.” About an equal number had some difficulties. For example, some clinical officers and interns were not following infection prevention practices and the supervisor intervened. But even after coaching, the providers did not really change their behavior unless someone was watching. In this case, they were not decontaminating the instruments after the procedure. Another supervisor reported coordination problems between the doctors and the clinical officer. One respondent said he has not seen good results from the PAC program at his district hospital, because the providers that were nominated for training are not really enthusiastic about the service and lacked commitment to the program. They were sent to training because of their availability and not their interest in this particular service, due to staff shortages (no one else was available to attend). 5. Record keeping and data review responsibilities assigned to staff Only a minority (38%) of clinical supervisors interviewed reported that PAC providers were designated to review the completeness of daily records and share the results with other hospital staff. When asked why not, only one reported that this was a time issue and that patient care and other duties superceded recordkeeping. One respondent noted this was just a “lack of insight,” perhaps meaning that there was a perceived value to data review, but that it just was not being done. PAC Inservice and Preservice Training The National Program Coordinator reported that standard inservice training packages (group￾based and OJT) had been developed and approved by the MOHP for training clinical service providers. In addition, a PAC component had been incorporated into the preservice RH learning 18 JHPIEGO Technical Report package for faculty and clinical preceptors. However, there have been barriers to implementing the training programs. The OJT program had not progressed as well as expected at some hospitals, as the trainees were not as motivated as desired, perhaps needing more incentives from the PAC program. And most of the district level hospitals still lacked training equipment, (e.g., videos, VCR, and anatomic models) for the OJT. Nonetheless, all but one of the clinical supervisors interviewed at the hospitals in this evaluation reported having a PAC OJT program in place. The faculty members had not yet been trained at the time the National Coordinator was interviewed. The percentage of hospitals with accessible and routinely maintained training materials is shown in Table 3. Three-quarters of the hospitals visited had a dedicated area for PAC OJT activities. All sites had MVA equipment and most sites had the written reference and training materials. The exception to the latter was the lack of sufficient trainee logbooks in half of the hospitals. More than 40% of the hospitals did not have ZOE (anatomical) models at the time of the evaluation, which would make it impossible for the trainees to prove competency on models before moving on to clients, as required as part of the competency-based training methodology adopted by the National PAC Program. Very few hospitals had the required equipment and materials for on-the-job trainees to view PAC training videos. Table 3. Percentage of PAC Hospitals with PAC Training Materials PAC Training Materials Percentage of Hospitals MVA Equipment 100% PAC Reference Materials 92% OJT Materials 92% Kulera FP Flipcharts 92% Dedicated Training Area 75% Completed Course Book 67% Trainer Signatures on Course Book 67% ZOE Models 58% Trainee Logbooks 50% TV/VCR 50% Videotapes 25% The evaluation team attempted to assess the total number of PAC cases on models and clients treated by the OJT trainees at each hospital. Only 11 of the 14 hospitals visited could provide these data from either their training records or from memory. Only four of the hospitals reported numbers of cases with models as well as clients; the remaining reported that the OJT trainees had treated only PAC clients. Of those hospitals where OJT trainees practiced on ZOE models, the number of cases per trainee on models ranged from 1–15. For those where OJT trainees learned with clients, the number of cases per trainee ranged from 2 to more than 30 during their training period. The National Program Coordinator reported that at the time of her interview, six on-the-job training participants had been certified as competent PAC providers by the central level. Registered nurse-midwives (RNMs) and clinical officers interviewed reported having been JHPIEGO Technical Report 19 training one to four providers each through the OJT program, with central hospital staff training two to four providers each and other hospitals from one to three providers each. Thirteen hospitals reported training/orienting support staff in aspects of postabortion care, specifically infection prevention. Clinical officers and enrolled nurses who had not been to group-based training were oriented to PAC. Patient attendants were trained on the processing of instruments (cleaning and sterilization), and hospital servants in the operating theatre and female ward were oriented on PAC and infection prevention. Additional staff trained included: ward attendants, watchmen, cleaners, hospital maids, and hospital attendants. Topics included in the training for the various cadres were: cleaning of floors and room, cleanliness, handwashing, gloves, care of instruments (syringes, sharps, cannulae), decontamination, high￾level disinfection, packing of MVA kits for sterilization, sterilization, autoclaving, and handling/caring for patients who have aborted, including intraprocedure counseling and “verbacaine.” 1. Training participants’ characterization of training components PAC service providers were interviewed about their experience with PAC group-based and on￾the-job training. Respondents reported that the “understanding was better with demonstration” by the trainer, as it “made it easy to follow the procedure” and “reinforced knowledge and memory” as “you see what you are supposed to do.” However, one provider who attended a group-based course said he never saw his trainer providing MVA to a client—only practicing on a model. Some of the providers trained on the job reported not having practiced on models as they had no ZOE anatomical model at their training site. Those who did have models stated it was “important, because it prepares for the real life situation,” that it “helps me to get ready (be confident) with the procedure” and that “many mistakes were corrected by the trainer before the real client.” Other benefits of the models were that “we could practice even on our free time” and “skill was acquired fast.” Only one provider seemed to have an aversion, stating that working with models is “hard to measure and you don’t get reaction from the models.” Regarding working with clients during the training, respondents reported that this was “very useful because it was done with the help of a supervisor” and that they “gain[ed] confidence/competence during practice, removing the fear of the unknown.” The real patients brought out lessons that the models had not, such as one provider’s “problems with counseling skills” and “real situations like pain management [were] experienced.” Also, “you get reactions from the client and know what to expect from the next one” and “this was a real life situation.” Feedback from the instructors was very important to the trainees. Respondents were pleased that their mistakes were corrected and that they were taught what to do in a real-life situation, based on the trainers’ experiences and clarifications made during the training and practice sessions. The strong points that PAC service providers reported and remembered about the training included the focus on infection prevention and family planning to prevent unwanted pregnancy. They said they would also emphasize as important the “treatment of clients [as] emergency cases and with empathy.” The OJT focus was appreciated as “you can train more of the district without waiting for group training. [This is a] good idea because you are [in] your environment and you can continue providing care to others, too.” 20 JHPIEGO Technical Report Weaknesses in the training varied. Some providers reported problems with the OJT schedule at their hospital, especially for those working night duty, as they had to come in for training during the day, which made it very difficult to concentrate. Others reported space issues, with one saying that their conference room was being used by another group so they could not do PAC training there. Equipment was also a weakness in one site, without enough MVA kits for all of the clients. One group-based course trainee reported “we had little time to do all the things, both theory and practical,” while another stated “not much practice on real clients, probably because when you are many and traveling, the clients are less.” In Malawi, there is a practice of paying per diem allowances when participants go to a group-based training course away from their normal worksite. Two respondents brought this up as an issue for the PAC OJT program: “people take [PAC] to be [the responsibility of the] individual … who had gone to group-based training—[he] was given money so he should do it alone;” and another reported “lack of commitment because [of] not attending the central training where some allowances were offered while OJT training [participants are]… demoralized,” presumably because these participants do not get the allowances because they are trained at their regular worksite. Still others focused on clinical aspects that differed in the demonstrations from what was taught in theory—for example, “failure to use sterile gloves; failure to use 5 swabs throughout” and “at the clinical site, we found the practice different, e.g., cleaning with 4 swabs and inserting the tenaculum.” Several respondents also said that one trainer was not enough, and stressed the busy hospital environment so that at some points PAC patients were kept waiting while the providers were busy with other work. Service providers who took PAC training had several suggestions for improving the training. Only two of 27 respondents stated more funding was needed for the training. Several respondents wanted the training to last longer, with one OJT participant opining that allowing 3 weeks would give the providers more clients for practice; another suggested there should be a specific number of weeks for training only. There was an emphasis on better coordination of OJT with work schedules, including night-shift work. Many suggested that the availability of PAC equipment and supplies for training was not yet adequate, with the following as examples: disposable and sterile gloves, Savlon, Cidex, gauze, ZOE models, videotapes, towels, and drapes. Others emphasized the need for more training materials, such as modules and learner’s guides. Finally, more refresher courses and regular supervision visits were requested. 2. Trainers’ characterization of training components Trainers reported that OJT was helpful because it did not interrupt the regular workflow: “since it is done while doing other jobs, there is no interruption of services, and it is cost-free.” Two hospitals reported positive spillover effect as they are also now training interns on MVA. Most hospitals actually reported that PAC training had decreased congestion on the wards by reducing duration of stay in the facility and improving quality of service. Although not representative of the larger group, clinical supervisors at two of the 14 hospitals said that PAC had not affected service delivery much because there are were either too few cases so far or not much work had been done. Like service providers, trainers emphasized the need for adequate supplies in order to have a successful training program. The training equipment cited as needed were TVs and VCRs for watching videos at the OJT site, as participants had to take the videos home under the current arrangement. Trainee logbooks were also in short supply in several hospitals. PAC supplies needed in different sites ran the gamut, and included Jik, Cidex, stirrups for the patient bed, speculums, cannulae, bivalve syringes, lubricant, light sources, and towels. Several trainers JHPIEGO Technical Report 21 expressed the need to expand PAC services and training to health centers or rural hospitals, because these were the sources of referral for many of the PAC hospitals’ clients. There was also a request for additional nursing staff to take on more of the PAC and other clinical duties. Even with the equipment needs cited, trainers were very enthusiastic about the positive points of having OJT. They cited that it was much better than sending providers out for training, so that they are in the hospital for service provision. Those who had access to them said that the videos were useful to trainees and that the model provided a good opportunity for practice so that they had few problems with their first patients. Supervisors reported that they perceived a reduction in septic cases after the PAC training. Being cost-free was also an appreciated aspect of OJT for clinical supervisors. OJT trainers asked the evaluation team to relay their requests for more TVs/VCRs, ZOE models, and trainee manuals to the National PAC Program. One trainer stressed the importance of changing specific language throughout the PAC training manuals, citing a section that said “put on gloves” rather than “put on sterile gloves,” saying this left too much to interpretation, and that providers referring to this may just use any gloves. Supervisors asked for more training of providers and trainers, including those from various departments such as the outpatient, gynecology, and obstetrics wards. Regular (bimonthly) supervision visits were encouraging to the site staff and there were requests for the MOHP RHU to maintain this external supervision. The selection of participants for PAC training was an issue, as some hospitals said they have so few registered nurses that enrolled nurses should also be considered for training to deal with the caseload. In addition, some decision-makers had said the providers trained must be registered nurse-midwives (RNMs) and there was concern expressed that all registered nurses should be allowed to be trained in PAC as there are fewer RNMs. Participants selected for training “should be people who are interested in the job,” said one supervisor. In contrast to the service provider perspective, some of the trainers thought that the OJT program was too long, perhaps because it is stretched to a longer calendar period of time than the group-based course. One trainer was concerned about the low caseload and stated that the hospital was trying to do more community outreach to inform the public about the PAC services available. 3. Performance of OJT trainers Strong points of the OJT trainers included an emphasis on teamwork and imparting valuable knowledge and skills to providers. However, there was some concern about little or no feedback going from the trainers to the immediate supervisor. Another point that needs work is the organization of the training, including having the adequate supplies and scheduling time and space for the training appropriately. Criticisms included too many competing activities to fully benefit from OJT, certification takes too long, there is little emphasis on theory, and there is a lack of incentives. Also, those who drop out of OJT should need to be retrained or updated and there is no standard mechanism for this. Supervision visits were cited as good ways to maintain standardization, including among the onsite trainers, as sometimes the providers themselves have noticed mistakes and when the external supervisors came, a higher level supervisor then corrected the OJT trainers. Figure 5 shows trainer responses on the utility of specific teaching methods. All trainers reported that the trainer’s manuals and that coaching of trainees while they practiced their skills were useful. The trainer’s manual made teaching easier because it was very logical and delineated clear steps to follow. Coaching allowed trainers to point out weaknesses and remind the trainees of specific procedures, to improve skills. While 82% of trainers found case studies 22 JHPIEGO Technical Report of use, they were split in their opinions about role plays, with some saying they seemed useful, but with one trainer and a shortage of personnel, they didn’t do them. Another said that role plays were useful to “help trainees to know what is good and bad during counseling.” Case studies were nice to have when there was no access to videos, said one trainer, as they helped to present information to the participants. The trainers who did not report models and videos as useful in OJT actually did not have access to these tools. In one hospital, the tape deck had been stolen, and there were no tapes available. In others, the VCR and TV were available but they had been supplied with no PAC training tapes. When they were available, models were appreciated because practice “gives much confidence to trainees on doing MVA on real patients.” * Those who said not useful did not have access to these items Discuss videos Discuss role plays Skill demo on models Discuss case studies Coaching w/practice Trainer's guide % of service providers 100 80 60 40 20 0 27 45 55 82 100 100 * * Trainers and providers at the hospitals visited directly asked for the assistance of the MOHP in obtaining and maintaining access to the training and service provision materials to retain a quality PAC OJT program at their sites. Figure 5. Percentage of PAC Trainers Who Rate Each Training Method as Useful JHPIEGO Technical Report 23 Total Score 60 65 70 75 80 85 90 95 Number of service providers 8 6 4 2 0 Std. Dev = 10.82 Mean = 83 N = 29.00 4. Knowledge assessments of PAC providers trained Twenty-nine providers representing 13 facilities were given written knowledge assessments consisting of 20 multiple choice questions, five questions each on four topics: (1) initial assessment, (2) infection prevention, (3) MVA provision, and (4) postabortion family planning. The providers who completed the assessment were registered nurse-midwives and clinical officers who had received either group-based or on-the-job training and worked at central and other (district and mission) hospitals. Figure 6 above shows the total score distribution. The mean knowledge assessment score was 83%, the median was 85%, and there was a bimodal distribution with seven providers each achieving 85% and 95%. The lowest score achieved was 60% and the highest was 95%, for a range of 35%. The differences in mean score between providers by hospital type, training type, and cadre are shown in Figure 7 on p. 24. A t-test performed in SPSS to compare mean scores revealed a statistically significant difference (p < 0.05) in knowledge scores achieved by hospital type, with central hospital scoring higher (mean = 87%) than other hospital providers (mean = 79%). The differences in scores between group-based and on-the-job training participants and between registered nurse midwives and clinical officers were not statistically significant. See Appendix B for more detail on the t-test results. Although providers achieved the same range of scores on all four sections, from 40% to 100% of questions correctly answered, the section with the highest mean score was initial assessment. The second highest was infection prevention; however, this was also the section . Figure 6. Histogram of PAC Service Provider Knowledge Assessment Scores 24 JHPIEGO Technical Report Figure 7. Box Plots of PAC Service Provider Knowledge Assessment Scores, by Hospital Type, Training Type, and Cadre N = 12 17 Hospital Type Central Other Total Score 100 90 80 70 60 50 Clinical Officer N = 14 15 Training Type Group-based On-the-job Total Score 100 90 80 70 60 50 N = 15 14 Cadre Clinical Officer Registered Nurse/Mid Total Score 100 90 80 70 60 that contained the single question which the most providers (50%) answered incorrectly. The third highest scored section was PAC family planning, followed by the lowest scored section, MVA provision. Figure 8 on p. 25 shows the comparison in mean section scores achieved by the 29 providers whose knowledge was assessed. 5. Impact of OJT on PAC services PAC patient caseload at each site differs greatly.1 With regard to retention of skills after training, caseload is important, as providers should be exposed to and treat a number of patients in their practice. This also speaks to the appropriateness of participant selection, as those who underwent PAC training should be actively providing PAC services. The number of cases seen per provider post-training varied considerably among hospitals, ranging from 0 for one provider who was 1 month post￾training at the time of the study, to 92 clients for one provider who had already been practicing 10 months at the time of the evaluation. Figure 9 on p. 25 shows the distribution in length of time since training and the number of clients for whom providers had directly provided PAC services. The median length of time post training was 4 months, and providers had seen a median of 8 PAC patients post￾training. Clinical supervisors reported that the provision of PAC services through the OJT program decongested the gynecological wards by lowering average length of stay, and that OJT improved infection prevention and other clinical practices. 1 Logbooks in place at each site show the total caseload at each hospital, and these are presented in the section on quality of care below with other service delivery indicators. JHPIEGO Technical Report 25 N = 29 service providers % MVA Provision % Postabortion FP % Infection Prev % Initial Assessment Mean Section Score 90 88 86 84 82 80 78 76 74 77 82 85 88 N = 27 service providers Months providing PAC services since training 0 2 4 6 8 10 12 14 Number of PAC patients 100 80 60 40 20 0 Figure 8. Mean Section Scores on PAC Knowledge Assessment Figure 9. Distribution in Length of Time Since PAC Training and Total Number of PAC Clients 26 JHPIEGO Technical Report Community Mobilization and BCI Half of the 14 hospitals visited reported some PAC-related community activities ongoing. These included working with drama and drummer groups, working with youth, and having meetings with traditional authority chiefs in the areas surrounding the hospital, in addition to working on awareness of PAC through the existing family planning commodity community-based distribution system. Those working on community activities had two key messages: (1) to explain that PAC services were not induced abortion, and (2) to encourage early treatment and recognition of the danger signs. Among the hospitals that were not carrying out any such activities, one concern was how to tell people that they are dealing with abortions in PAC without promoting induced abortion. Over half (64%) of the hospitals visited reported having community activities in at least one of the following FP/RH areas: ♦ Prevention of unwanted pregnancy ♦ Prevention of unsafe abortion ♦ Early recognition of danger signs ♦ General sexual/reproductive health Only one hospital reported having undertaken a Community COPE exercise2 . They stated this was still new and that people make changes based on new information slowly. As yet they had seen no great increase in clients. They thought that transportation may be helpful to arrange, and that there were available vehicles at the hospital. A few of the PAC clients who were interviewed in this evaluation provided the evaluation team with insight on the need for further community reproductive health education: “I am very satisfied with the services and the reception. However, I was not willing to come, if it weren’t for my mother. I was afraid of general anesthesia, which was not the case with MVA.” Another such comment showing persistent stigma in the community was: “She has seen for herself that nurses/doctors really help and they are friendly. Her friends were telling her that nobody will care for her if she goes for [postabortion care] because they might think that you have induced it.” These women’s concerns before seeking care reveal important messages for targeted information, education, and communication campaigns. Increased Availability of PAC Services 1. Number and geographic distribution of sites providing comprehensive PAC services Figure 4 on p. 5 shows the geographic distribution of the 14 hospitals that are providing comprehensive PAC services in Malawi at the time of this evaluation. There were three in the Northern Region, six in Central and five in the Southern Region. 2 Community COPE (client-oriented, provider-efficient services) is a methodology developed by EngenderHealth whereby health facility staff learn how communities feel about healthcare service delivery quality, and community members are brought into the service delivery improvement cycle. JHPIEGO Technical Report 27 2. Number and percentage of trained PAC providers at each site The majority of current PAC service providers at the hospitals visited were clinical officers (31 providers, 57%), followed by registered nurse-midwives (23 providers, 43%). Most providers had been trained on the job (30 providers, 56%) rather than in group-based courses (24 providers, 44%). Despite a total of 54 current PAC service providers being reported across the 14 hospitals, all but one hospital evaluated reported that clinicians not trained in PAC were currently providing PAC services. On a positive note, all hospitals then stated that these clinicians (most hospitals had two or three) either solely do evacuations or that they are currently in the OJT program to learn MVA. Hospitals reported that overall, an average of 37% of current PAC providers were trained in PAC, 39% of providers assigned to counsel PAC patients on FP had been certified as FP providers, and 16% of those counseling PAC patients on other RH services had been certified as HIV/STI counselors. 3. Orientation of staff and management, and adherence to PAC service delivery guidelines Seventy percent of service providers interviewed reported that they had a copy of the PAC service delivery guidelines available to them at the facility where they worked. Most kept them in the MVA procedure room or the PAC counseling room. Eighty-nine percent of providers stated that the PAC service delivery guidelines were being adhered to in the facility. 4. Availability of PAC services 24 hours a day and 7 days a week Sixty-two percent of facilities visited had PAC services available 24 hours a day and 7 days per week. This leaves a serious gap of nearly 40% of facilities either without this level of service availability or who could not report whether they had it or not. The map in Figure 10 shows the geographic location of PAC program hospitals with and without 24/7 availability of PAC services. The evaluation teams did not get the specific times and hours that the hospitals without 24/7 coverage offered PAC services; however, the differential patterns of treatment availability when patients arrive during the day versus in the evening are illustrated in the section below on emergency care and average length of stay (see p. 30). 28 JHPIEGO Technical Report Increased Quality of PAC Services 1. From central perspective, lessons learned and advice to other countries The National Program Coordinator reported that the overall PAC program implementation was proceeding well at the time of the interview in October 2002. However, there were specific barriers to implementation, including inadequate supplies and equipment at the facility level and a shortage of trained providers. The MOHP had several lessons learned to share with other countries that may be implementing PAC programs, as follows: a. A consensus among stakeholders is essential. b. It is important to have a close liaison with the Ministry of Health (i.e., the Principal Secretary, Minister, Controller of Nursing Services), so as to communicate to policymakers what PAC is. c. For the sake of progress, it is better to have one main cooperating agency as a partner in coordination of the program. d. Essential equipment should be in the Central Medical Stores and available as well as sustainable in the country. e. There is a need to have high-quality training of the service provider. OJT should be critically reviewed before expansion. f. There should be community involvement starting from the planning stage. Figure 10. Geographic Distribution of Hospitals with PAC Services, by Availability 24 Hours per Day, 7 Days per Week JHPIEGO Technical Report 29 2. Record-keeping, including HMIS, systems in place to monitor quality of PAC service delivery The National Program Coordinator reported that the service delivery sites are supposed to send in information quarterly to the MOHP, but that this was not happening at the time of the evaluation. Thus, there was a need to start discussing this with the service providers on supervision visits. The data they are to submit are based on logbooks that were disseminated to each facility to collect standard data on each PAC patient. Visits to hospitals have demonstrated that the logbooks are being completed consistently for MVA patients, but may not be used in all cases for D&C patients as a separate operating theatre logbook may be in place without all of the same data fields. The quarterly report sheets, which contain instructions on how to calculate key indicators from the logbook, are the missing link, as staff are not compiling the logbook data to complete and send in these reports to the RHU. Of the seven key indicators requested in the quarterly report template, six can be calculated from the logbook: INDICATOR #1: Total number of PAC patients INDICATOR #2: Total number of PAC patients treated for complications INDICATOR #3: Percentage of PAC patients counseled for FP INDICATOR #4: Percentage of PAC patients who accepted and received (left with) a FP method INDICATOR #5: Percentage of PAC clients referred for FP services INDICATOR #6: Percentage of PAC clients referred to or receiving other RH services The seventh indicator must be obtained from training records kept at the facility or by interviewing the staff at report time: INDICATOR #7: Number of trained PAC providers, by cadre 3. Client waiting area, waiting time and patient flow Patient waiting time is an important aspect of care to assess, not only for patient satisfaction but because cases of postabortion complications should be treated as emergencies and be seen promptly for patient safety. Three of the 13 patients interviewed were seen by a nurse or doctor within 15 minutes, but most patients experienced waiting times of 2–3 hours. The longest two waiting times reported were 12 hours and 3 days. Within the same facility, there were wide ranges as well, with three hospitals showing a range of waiting times from 1–3 hours. Even with this small sample size, these waiting times show that there is still a reason for concern with regard to treatment of PAC patients as emergencies consistently across and within facilities. Patient waiting areas varied from facility to facility, with the most well-equipped waiting area described as follows: “It has chairs for patients to sit on, a dressing area and a toilet, so patients don’t have to move a long way to find a toilet.” The least well-equipped waiting area was described as: “in maternity, they are admitted and wait in the antenatal ward on the floor (space problem).” Only two of the 14 hospitals visited reported a toilet in the patient waiting area. Many said that the patients wait on the ward, presumably on benches although this was not stated in all cases. The patient flow in all hospitals visited started with registration, initial assessment, and admission by the outpatient department. The second stop for clients in most hospitals was an additional assessment either in the gynecology or maternity ward or in the outpatient 30 JHPIEGO Technical Report department by a clinical officer. Only three hospitals reported the second step was the PAC treatment procedure itself. Most hospitals treated the patients on the third step, followed by FP counseling, recovery, and discharge. 4. Emergency care and average length of stay Clinical Supervisors were asked to assess on average the time that clients wait before seeking care. They reported that 1–2 days usually passed between onset of symptoms and patient presentation at the hospital. One provider said that induced abortion patients wait longer than spontaneous abortion patients. The average time reported between hospital admission and initiation of the MVA or D&C procedure ranged from 1 hour to 2 days, with most respondents just committing to “same day.” Two providers said, “if not an emergency, next day treatment.” This seems inconsistent with the teaching that all postabortion cases are emergencies. Another provider at a central hospital said it depends on the time of admission: “before noon, assessment and procedure within 2 hours; afternoon, reviewed but procedure done next day.” Still another combined these criteria: “emergency—immediate; day—within 8 hours; night, not emergency—18 hours.” The average time between MVA treatment and hospital discharge was 2–3 hours. After receiving the D&C procedure, patients usually stayed in the hospital 1–2 days for recovery before being discharged. It is important to note that some MVA patients actually stay longer than the 1–2 hours if they do not have transportation or are too weak to walk, if that is their only way to get home. 5. Counseling about the procedure and treatment options Sixty-nine percent of clinical supervisors interviewed reported that counseling about the procedure was offered to all PAC patients. More than 85% of clients interviewed stated that they were counseled about the procedure that they underwent. Figure 11 shows the percentage of supervisors who reported that each of various topics were covered routinely in PAC counseling at their facility. JHPIEGO Technical Report 31 Current diagnosis Other RH conditions Treatment options Prognosis Other patient concrn % of clinical supervisors 60 70 80 90 67 67 67 75 83 6. Client’s pain perception and use and sufficiency of pain management Ninety-two percent of clients reported that their healthcare provider gave them pain medication, and about a quarter had to ask for it. Only 1 of the 3 patients who asked for pain medication said that the healthcare provider’s response was not sufficient. Figure 12 shows the patients’ perception of pain upon admission, during treatment and post-treatment, along a standardized scale. Figure 11. Percentage of Clinical Supervisors Reporting That These Topics Were Included in Routine Counseling at Their Facility 32 JHPIEGO Technical Report 11 11 11 N = After treatment During treatment Came with Pain 10 8 6 4 2 0 5 5 Figure 12. Client Perception of Pain Scale and Client Rating of Pain at Admission, during MVA Treatment, and Post-Treatment 33 JHPIEGO Technical Report Consistent with nearly all clients interviewed reporting having received medication for pain during their postabortion care visit, nearly all providers reported using verbal anesthesia along with a non-narcotic local analgesic for pain management to manage pain in PAC patients. This finding is shown in Figure 13. Perhaps of concern, nearly a third of service providers reported at times using verbal anesthesia alone. N = 27 service providers Paracervical block General anesthesia Verbal anesth. only Verbal & analgesic % of service providers 100 80 60 40 20 0 7 15 30 93 Although most providers reported that their pain management strategies were sufficient, others said that it really varied by patient. One clinical officer at a district hospital reported: “patients scream with pain. From my point of view, we need stronger analgesia,” while another clinical officer at that same hospital said “most patients are comfortable.” A registered nurse/midwife at another hospital said, “some clients become uncooperative maybe because they are in great pain which means that the analgesia and verbacaine don’t work well on all clients; but some respond well.” 7. Provision of FP counseling and range of methods available onsite Two of the 14 hospitals visited had FP counseling available in a separate building in the same hospital. This is contrary to the service delivery recommendations, which are to have FP counseling and methods in the immediate PAC service area. The remaining 12 hospitals had FP in the same gynecology ward or procedure room where MVA services are provided. Eighty-five percent of clinical supervisors interviewed stated that FP counseling is provided to all clients who received PAC services. Ninety-two percent of clients interviewed said they had received FP counseling at their PAC visit. However, logbooks in the PAC section showed that in half of the hospitals visited, fewer than 50% of PAC patients received FP counseling. This does not concur with the interview data, and may either demonstrate over-reporting in interviews and/or under-recording of FP counseling in PAC logbooks. On a total client basis, the Malawi Figure 13. Percentage of Service Providers who Reported Managing PAC Patients’ Pain with Each Pain Management Regimen 34 JHPIEGO Technical Report PAC program—across hospitals—provided FP counseling to 74% of MVA clients and 40% of D&C clients in the quarter preceding the evaluation. In nearly all facilities, clinical supervisors reported the availability of condoms, oral contraceptives, and Depo-Provera injectables in the immediate PAC service area. Just over half of supervisors reported that IUDs and Norplant implants were available, and less than 10% had foam tablets in the PAC area. (See Figure 14.) N = 13 Condom Oral contraceptives Depo-Provera injec IUD Norplant Foam tablets % of clinical supervisors 0 20 40 60 80 100 54 54 92 92 92 As shown in Figure 15, more than three-quarters of facilities visited had condoms, pills and Depo- Provera injectables available to PAC clients 24 hours per day, 7 days per week. Slightly fewer had Norplant implants and IUDs available at all times. Figure 14. Percentage of Clinical Supervisors Who Report That These FP Methods Are Available in the PAC Service area JHPIEGO Technical Report 35 N = 14 facilities Condoms Pills Injectables Norplant IUDs % of facilities with method available 24 hours/7 days per week 68 70 72 74 76 78 80 82 70 70 80 80 80 8. Provision of recovery room/area onsite All but one of the 14 facilities visited had a recovery room or area. It was close to the procedure room, in the female or maternity ward in most cases. In one hospital, the recovery area was the waiting room, which did not seem optimal. In another, the female ward that was used for recovery was located in a separate building at the same hospital, a short walk away. 9. Provision of additional reproductive health services More than 60% of clinical supervisors reported that counseling for STI and HIV patients was routinely provided to PAC cases at their facility, while 38% said that infertility services were available. None had cervical cancer screening available at their hospital. 10. Caseload of PAC patient Twelve of the hospitals had caseload data available, showing from 46 to 1,279 total gynecology ward admissions in the previous quarter (July–September 2002), of which from 4 to 293 were PAC patients. The median number of gynecology cases was 219 and median PAC patient caseload was 56. The evaluation teams gathered these data from logbooks kept at each hospital. Calculations with these data showed that 2–56% of gynecological ward admissions were PAC patients, with a median value of 26% of gynecological ward admissions being PAC patients. Although these data could represent real differences, there was some concern among the data collectors that hospitals did not all record data with the same level of consistency. Figure 16 shows the hospitals and their relative quarterly PAC caseload. Figure 15. Percentage of Facilities with These FP Methods Available 24 Hours per Day, 7 Days per Week 36 JHPIEGO Technical Report 11. Number and percentage of clients accepting each FP method by age and method According to hospital PAC logbooks, the percentage of PAC clients who were counseled for FP and left with a method ranged widely between the hospitals evaluated. (See Figures 17 and 18.) The most popular FP method among MVA clients was Depo-Provera, followed by pills, condoms and bilateral tubal ligation (BTL). (See Figure 19.) The age distribution of FP acceptors and non-acceptors among MVA patients is shown in Figure 20. The highest numbers of cases are age 20–24, followed by ages 25–29, and then by ages 15–19. In every age group, there were more FP acceptors than non-acceptors. Age distribution of FP acceptors by method is shown in Appendix B. The average patient age of short-term method acceptors is younger than that of long-term method acceptors, consistent with expectations, lending credence to PAC logbook FP method provision and patient age data. Figure 16. PAC Caseload, Malawi PAC Program Hospitals, July–September 2002 JHPIEGO Technical Report 37 Figure 17. Percentage of PAC Clients Receiving FP Counseling, Malawi PAC Program Hospitals, July–September 2002 Figure 18. Percentage of PAC Clients Leaving with a FP Method, Malawi PAC Program Hospitals, July–September 2002 38 JHPIEGO Technical Report Source: PAC logbooks at each hospital Depo Provera Pill Condom BTL Norplant IUD Number of Method Acceptors 0 20 40 60 80 100 120 140 22 35 99 135 12. Number and percentage of clients receiving RH referrals In the hospitals where these data were available, a median of 7 patients had been referred for FP services and 3 patients were referred for other RH services in the quarter preceding the evaluation visit. Although this may be true, it is also a likely possibility that documentation of these referrals is very poor in the PAC logbooks, which were the data source for these indicators. 13. Skills checklists for all aspects of PAC care The evaluation teams assessed the clinical skills of six registered nurse-midwives and six clinical officers via direct observation of services provided to PAC clients during hospital visits. Four of the providers observed worked at central hospitals and the remaining 12 at district hospitals. All were observed at the site where they normally work. All providers were assessed on PAC clinical service provision, family planning counseling, and verbal anesthesia skills. a. Clinical skills As shown in Table 4 on p. 40, the majority of providers observed satisfactorily completed all of the steps necessary for PAC emergency care. (See Table 4 to review the percentage of providers who completed each step according to standard.) Figure 19. Total Number of FP Acceptors among MVA Patients, Malawi PAC Program Hospitals, July– September 2002 39 JHPIEGO Technical Report Age distribution of MVA patients who left with a Family Planning method Source: Malawi MOHP RHU 45+ 40-44 35-39 30-34 25-29 20-24 15-19 <15 Number of Patients 100 80 60 40 20 0 Age distribution of MVA patients who did not accept an FP method Source: Malawi MOHP RHU 45+ 40-44 35-39 30-34 25-29 20-24 15-19 <15 Number of Patients 100 80 60 40 20 0 Figure 20. Age Distribution of MVA Patients Who Left with a Family Planning Method and Did not Accept a Method, PAC Program Hospitals 40 JHPIEGO Technical Report Table 4. Percentage of Directly Observed PAC Service Providers Who Satisfactorily Completed PAC Clinical Skills Task % of Providers Who Completed Step Satisfactorily GETTING READY 1. Tells patient what is going to be done and encourages her to ask questions. 100% 2. Tells patient she may feel discomfort during some of the steps and that s/he will tell her in advance. 100% 3. Checks that patient has thoroughly washed her perineal area and has recently emptied her bladder. 83% 4. Determines that all required sterile or high-level disinfected instruments and cannulae are present. 100% 5. Checks MVA syringe and charges it (establishes vacuum). 100% 6. Puts on apron, washes hands thoroughly with soap and water and dries with clean, dry cloth or air dries. 100% 7. Puts new examination or sterile or high-level disinfected gloves on both hands using aseptic “no touch” technique. 100% 8. Arranges sterile or high-level disinfected instruments on sterile tray or in high-level disinfected container using aseptic “no touch” technique. 100% MVA PROCEDURE 1. Explains each step of the procedure prior to performing it. 92% 2. Performs bimanual pelvic examination to confirm uterine size, position, and degree of cervical dilation. 100% 3. Checks the vagina and cervix for tissue fragments and removes them. 83% 4. Applies antiseptic solution two times to the cervix (particularly the os) and vagina. 92% 5. Puts tenaculum or vulsellum forceps on anterior or posterior lip of cervix. 100% 6. Correctly administers paracervical block (if necessary). 100% 7. Dilates the cervix (if needed). 100% 8. While holding the cervix steady, inserts the cannula gently through the cervix into the uterine cavity. 100% 9. Attaches the prepared syringe to the cannula by holding the end of the cannula in one hand and the syringe in the other. 100% 10. Evacuates contents of the uterus by rotating the cannula and syringe and moving the cannula gently and slowly back and forth within the uterine cavity. 100% 11. Inspects tissue removed from uterus for quantity and presence of POC and to assure complete evacuation. 100% 12. When the signs of a complete procedure are present, withdraws the cannula and MVA syringe and removes forceps or tenaculum and speculum. (The speculum should not be removed until the cervix has been observed so that step 14 will not be necessary.) 100% 13. Performs bimanual examination to check size and firmness of uterus. 100% 14. Inserts speculum and checks for bleeding. (Not necessary if 13 above is done as suggested). 100% 15. If uterus is still soft or if bleeding persists, gives Ergometrine and/or repeats some of the steps 4–11 where necessary. 100% POST-MVA TASKS 1. Before removing gloves, disposes of waste materials and soaks instruments and MVA items in 0.5% chlorine solution for 10 minutes to decontaminate them. 100% 2. Immerses both gloved hands in 0.5% chlorine solution and removes gloves by turning them inside out. If disposing of gloves, places them in a leakproof container or plastic bag. If reusing surgical gloves, submerges them in 0.5% chlorine solution for 10 minutes to decontaminate them. 100% 3. Washes hands thoroughly with soap and water and dries with clean, dry cloth or air dries. 92% 4. Checks for amount of bleeding and if cramping has decreased at least once before discharge. 100% 5. Instructs patient regarding postabortion care (e.g., when patient should return to clinic). 83% 6. Discusses reproductive goals and, as appropriate, provides family planning. 83% N = 12 service providers JHPIEGO Technical Report 41 b. Pain management Similar to their performance on the above clinical aspects of MVA service provision, more than 80% of providers correctly performed each of the verbal anesthesia steps necessary for this procedure. Table 5 shows the percentage of providers who performed to standard on each step. Table 5. Percentage of Directly Observed PAC Service Providers Who Satisfactorily Completed Postabortion Verbal Anesthesia Skills Task % of Providers Who Completed Step Satisfactorily GETTING READY 1. Greets woman respectfully and with kindness. 100% 2. Assures the necessary privacy and confidentiality. 92% 3. Tells patient what you are going to do and encourage her to ask questions. 100% 4. Tells patient she may feel discomfort during some of the steps and you will tell her in advance. 92% 5. Assesses need for pain management medication. 92% PROCEDURE 1. Explains each step of the procedure prior to performing it. 92% 2. Asks the patient throughout the procedure if she is experiencing any pain. 83% 3. Waits after performing each step or task for patient to prepare for the next one. 83% 4. Moves slowly, without jerky or quick motions. 100% 5. Asks the patient to take deep breaths in an out during the procedure if there is some pain. 100% 6. Uses instruments with confidence. 100% 7. Avoids saying things such as, “This won’t hurt” when it will hurt or, “I’m almost done” when you’re not. 100% 8. Talks with the patient throughout the procedure. 100% N = 12 service providers c. Counseling skills Provision of PAC family planning counseling was not as satisfactory as provision of MVA clinical services and verbal anesthesia (see Table 6). During the initial interview, fewer than 80% of providers questioned the client about pre-pregnancy contraceptive history and discussed the patient’s needs, concerns, and fears in a thorough and sympathetic manner. During the patient screening process, fewer than 70% of providers screened patients carefully to make sure there was no medical condition that would be a problem for specific contraceptive methods (by completing the Patient Screening Checklist) and asked the patient to repeat instructions given on FP methods. Fewer than 80% answered patients’ questions. One-half of providers observed scheduled an appointment for the client to revisit the health facility, while one-third scheduled an appointment with the client for a followup visit at her home. 42 JHPIEGO Technical Report Table 6. Percentage of Directly Observed PAC Service Providers Who Satisfactorily Completed Postabortion Family Planning Counseling Skills Task % of Providers Who Completed Step Satisfactorily INITIAL INTERVIEW 1. Greets woman respectfully and with kindness. 83% 2. Assesses whether counseling is appropriate at this time (if not, arranges for her to be counseled at another time). 83% 3. Assures visual privacy. 83% 4. Assures auditory privacy. 83% 5. Obtains biographic information (name, address, etc.). 83% 6. Asks if she was using contraception before she became pregnant. If she was, finds out if she: · Used the method correctly · Discontinued use · Had any trouble using the method · Has any concerns about the method 75% 7. Provides general information about family planning. 83% 8. Explores any attitudes or religious beliefs that either favor or rule out one or more methods. 83% 9. Gives the woman information about the contraceptive choices available and the risks and benefits of each: · Shows where and how each is used · Explains how the method works and its effectiveness · Explains possible side effects and other health problems · Explains the common side effects 83% 10. Discusses patient’s needs, concerns, and fears in a thorough and sympathetic manner. 75% 11. Helps patient begin to choose an appropriate method. 83% PATIENT SCREENING 1. Screens patient carefully to make sure there is no medical condition that would be a problem for specific contraceptive methods (completes Patient Screening Checklist). 67% 2. Explains potential side effects of the various contraceptive methods and makes sure that each is fully understood. 83% 3. Performs further evaluation (physical examination), if indicated. (Non-medical counselors must refer patient for further evaluation.) 100% 4. Discusses what to do if the patient experiences any side effects or problems from contraceptive methods. 83% 5. Provides followup visit instructions. 83% 6. Assures patient she can return to the same clinic at any time to receive advice or medical attention. 83% 7. Asks the patient to repeat instructions. 67% 8. Answers patient’s questions. 75% 9. Clients are referred to other health care facility for FP services if not available at the treatment center. 100% 10. Fixes an appointment for the client to revisit the health facility. 50% 11. Fixes an appointment with the client for followup visit at her home. 33% N = 12 service providers JHPIEGO Technical Report 43 d. RH linkages The 12 clinical service providers (representing five district and three central hospitals) who were directly observed providing PAC services were questioned about the availability of services for additional reproductive health needs. They were asked whether the services shown in Table 7 were available either at their hospital or by referral from their hospital. Adolescent RH as well as reproductive tract/sexually transmitted infections and HIV/AIDS services were available onsite or by referral from all hospitals, and infertility screening and treatment were so at all but one district hospital. However, only one central hospital had onsite or referral services for cervical cancer screening and two central hospitals had gender violence service linkages. Table 7. Availability of Reproductive Health Referral Services and Linkages at PAC Hospitals Services/Referral Hospitals with This Service Available Either Onsite or by Referral 1. Cervical Cancer Screening Only one central hospital 2. RTIs, STIs, HIV/AIDS All 3. Adolescent RH Services All 4. Infertility Screening and Treatment All except one district hospital 5. Gender Violence Only two central hospitals N = 8 hospitals 14. Client satisfaction with services, and intent to return to this facility or refer others Interviews were conducted with 13 PAC clients who received treatment at each of nine hospitals. Six of the clients had received an MVA from a registered nurse-midwife, and seven had received an MVA from a clinical officer. The clients had come to their respective facilities for complications of abortion, which they described in various ways: “severe vaginal bleeding after three months amenorrhea,” “backache, abdominal pain and vaginal bleeding after two months of amenorrhea,” and “vaginal bleeding while three months pregnant.” Upon arrival at the hospital, clients all reported being received well: “I was warmly welcomed by staff, and I was examined immediately,” “they welcomed me and told me about the procedure,” “received well…. given drugs and bed” and “welcomed well and received empathy from staff in outpatient department.” All clients interviewed said they would come back themselves to this hospital for the same service and also refer others to this facility for PAC services. Seven of the clients gave additional comments to their interviewers, which provide some information on preconceptions about PAC services within the community: “she didn’t think it would take a short time for the procedure. When told she would be discharged today, she couldn’t believe it. Also, to have the pain gone after the procedure was something she didn’t expect.” Another client interviewer had recorded, “when she was just coming to the hospital, and then specifically to the MVA room, she was afraid of what will happen to her. But after the procedure, she noticed that it was a short procedure.” One patient wanted the interviewer to know she had experienced pain during the procedure, while another wanted to thank the hospital staff because she was no longer feeling any pain. Another, recognizing the lifesaving nature of the treatment she had received, said, “since I came today, I was welcomed well and if I had stayed home, I would have died.” 44 JHPIEGO Technical Report 15. Client recommendations on how to improve services PAC clients emphasized the value they placed on timely and attentive service, and patient education and counseling, through the following interview comments: The providers should continue welcoming people in time, counseling and telling them what procedures are to be done on them to allay the fears of the unknown. Nurses and doctors should continue attending to patients quickly when they come to the hospital, and be friendly by explaining whatever they are going to do to the patient. Nurses and doctors should help the patients immediately when they arrive instead of waiting so many hours or days before being treated. Nurses and doctors should see the patients frequently like what the MVA nurse did for me. Patients should be treated quickly and not left waiting a long time. Hospital staff should help patients with a kind heart. Health personnel should explain the problem found on patients and whatever they are going to do with them. 16. MVA decongesting gynecological wards’ availability, effectiveness, and impact PAC service providers and supervisors interviewed during facility assessments mentioned several effects of the introduction of MVA services on hospital gynecological wards. Overall, hospitals had noted that MVA services decongested the gynecology ward and provided a needed quality service for the patient population. A short length of stay, with most MVA patients being discharged the same day as their procedure, was listed as a benefit: “when MVAs are done in the morning, patients are discharged in the afternoon.” Having a nurse complete the procedure also means that the patients do not have to wait for the doctor. As the MVA procedure takes less time than the alternative D&C, turnover is quick, meaning that providers can move on to the next patient sooner, and beds can be used for other patients. MVA also reduces the number of patients going to the operating theatre, and waiting time is less for MVA than for the D&C procedure. One hospital representative quoted the following difference in charges for the two procedures: “MVA is less expensive to the patient. 150 Kwacha for MVA (they charge for supplies only) whereas 1,500–2,000 Kwacha for D&C.” Expansion of PAC Services In a February 2003 meeting to disseminate the results of the October 2002 PAC program evaluation, stakeholders made a strong recommendation to the MOHP to continue support to the current PAC program hospitals and to expand as quickly as possible to all other MOHP and CHAM hospitals in the country. In addition, it was suggested that PAC services be piloted at the rural hospital/urban health center level. In response to these recommendations, the MOHP worked with JHPIEGO and EngenderHealth, with USAID support, to expand in 2003 to an additional 8 and 10 sites respectively. It was determined at the evaluation meeting and in subsequent discussions with the MOHP that PAC OJT continue as well, but only at those sites that expressed interest in hosting this training program and that met the training site criteria as outlined in the national PAC training guidelines. JHPIEGO Technical Report 45 DISCUSSION The Malawi National PAC Program components include: (1) advocacy and policy development, (2) PAC training, (3) provision of materials and equipment, (4) organization of services, (5) expansion of PAC services, and (6) supervision, all aimed to increase the availability, quality, and use of comprehensive PAC services throughout Malawi. The structure of the National PAC Program and dissemination of the National PAC Strategy and service delivery guidelines were reported to be on track at the central level, but at the facility level, only about six-tenths of providers said staff had been adequately oriented on the PAC strategy and guidelines. Services are improved at the point of provider-client interactions, so it is important to keep this focus when implementing service delivery improvement programs, including PAC. Although significant strides in advocacy and policy development have been achieved raising the visibility of PAC on the national agenda, on-the-job orientation sessions and strengthened supervision may be useful to achieve greater penetration of the policy at the local level where it is needed. Group-based and on-the-job PAC inservice training produced clinical officers and registered nurse midwives who were both knowledgeable about PAC and prepared to provide high-quality care for patients. When observed directly, they largely conformed to the standard clinical protocols, with excellent performance on the MVA procedure itself and good performance on the subsequent FP counseling. Ongoing training at service delivery sites needs to be improved, however, with the adequate provision of ZOE anatomical models—as competency-based clinical training dictates that first the procedure be demonstrated successfully on models—and trainee logbooks, as each individual would optimally have a place to track her/his progress with the individualized learning package. From a central perspective, materials and equipment are being brought into the country successfully and sufficient supplies are regularly available in the Central Medical Stores. The PAC facilities had the majority of required PAC supplies on hand the day of the evaluation visit. However, the distribution of PAC equipment to the facilities is not yet optimal if nearly 40% of interview respondents had experienced a stockout of PAC supplies in the previous 6 months. Also, having nearly a third of providers depend on the MOHP RHU to provide them with PAC supplies directly rather than using a routine ordering system needs to be addressed. This will affect quality of care, for example, because standard infection prevention practices cannot be followed and FP services cannot be delivered without the needed supplies. The Essential Health Package Logistics Information System, EHPLMIS, will have the PAC supplies included, as they are now included in the Reproductive Health Logistics Management Information System (RHLMIS), which will be expanded into the EHPLMIS. However, the RHU and onsite supervisors will need to follow up to assure that the supplies are indeed in the hospitals where they are needed on a regular basis. The PAC client flow revealed in this evaluation did not demonstrate that PAC is being treated as an emergency consistently within or across hospitals. Waiting times of several hours were not uncommon. Differentials in the way clients were treated depending on what time of day they first arrive at the hospital were also problematic with regard to quality of care. And although integrating PAC into clinical service provision on hospital gynecology wards and linking PAC to other reproductive health services were envisioned at the central level, because not all hospitals have the desired services with which to link PAC, there was a limit to the linkages that could be put in place. Once cervical cancer screening and gender violence counseling are more 46 JHPIEGO Technical Report available, PAC patients and other women of reproductive age will be able to benefit from these services. Because they are by definition already connected to the formal health system in their PAC treatment visit, this will be a good opportunity to advise PAC clients on these additional services. To date, expansion of services in Malawi has meant both geographic expansion—the provision of PAC services in a greater number of district and mission hospitals over time—as well as expansion of the service to more clients through the addition of registered nurse-midwives as a cadre that can provide MVA services by amending the Reproductive Health Service Delivery Guidelines (previously, only clinical officers could perform this procedure). This evaluation shows that the MOHP RHU has been able to support the introduction of PAC services in hospitals in each region of the country through initial group-based training, OJT, and supervision visits. The experience in Malawi with registered nurse-midwives as PAC providers has been successful. The evaluation data have shown that their knowledge and clinical skills in PAC service provision are comparable to those of clinical officers trained in the same program. It would not have been possible to collect data on caseload and quality of care without the standard routine PAC logbooks in place and in active use at each hospital. The use of these data for ongoing monitoring of key indicators in between supervision visits is also important to assuring quality as services are expanded. Lessons learned from the evaluation also indicate the importance of expanding community PAC activities in parallel with service provision activities, to ensure that the population is aware of the services available and that stigma does not prevent them from accessing these lifesaving interventions. An external supervision system was envisioned and designed by the National PAC Program, and these visits to PAC facilities have occurred on a regular (at least quarterly) basis. However, the range in length of time that external supervisors are onsite, from 1 hour to 1.5 days, means that not all hospitals are getting the same level of observation and feedback. The evaluation also revealed that onsite supervision does not mean the same thing to all supervisors and providers, with only some onsite supervisors directly observing their staff with PAC clients and providing feedback. Onsite supervision did detect and correct important weaknesses in infection prevention practices, which is an important finding to be positively reinforced among clinical supervisors. CONCLUSION While the Malawi PAC program is currently providing lifesaving services to thousands of women each year, there are areas that could be strengthened both centrally and locally, with an eye to service delivery improvement. Patients already appreciate the PAC care and patient education received, but not all of the service provision is yet according to the national standards and goals for quality—especially with regard to wait times which sometimes leave women at the hospital overnight before they receive the MVA procedure. Nonetheless, great strides have been made and must be appreciated as the foundation upon which to build an even stronger National PAC Program, based on data derived from this evaluation. Through the implementation of these recommendations with existing and expansion hospitals in the PAC program, the MOHP Reproductive Health Unit and its partners will be in a better position to reduce the postabortion morbidity and mortality burdening so many Malawian women. JHPIEGO Technical Report 47 RECOMMENDATIONS The following recommendations fall within the six original intervention areas of the PAC program in Malawi: 1. Facility-level SDG orientation and dissemination efforts should be strengthened in existing and new PAC sites, to assure that these national policy and clinical protocol documents reach the service providers. These efforts should be made at the time of service introduction and on an ongoing basis, given staff turnover and transfer rates. 2. Adequate ZOE models and trainee logbooks should be distributed to all PAC OJT sites. A system for re-ordering training materials should be put in place to assure that the OJT sites are set up as successful individualized clinical learning environments. 3. As additional services such as cervical cancer screening or gender violence counseling for women of reproductive age are introduced and expanded, the linkages to PAC should be put into place by working with clinical supervisors and staff at PAC facilities to instruct them on ways to integrate these referrals into their counseling and patient education sessions. 4. All PAC facilities should be using a routine PAC supply ordering system, to get the equipment needed from the Central Medical Stores, and should order sufficient supplies for their expected caseload in advance to avoid stockouts. Since PAC equipment is now available through the Essential Health Package Logistics Management Information System (EHPLMIS), this system is in existence, but external supervisors should follow up with hospitals to ensure that the system is being used to equip providers with such essential equipment as emergency trays and to assure that stockouts are being prevented. Hospitals should be working internally and with the MOHP RHU to address supply challenges, including arranging for transport to get supplies from the Central Medical Stores to each facility. 5. As expansion of PAC services to more geographically dispersed hospitals is planned, expansion of the system for training, supplies, monitoring, and supervision should be planned with foresight to make certain that the supportive systems are in place to ensure quality of care. If expansion of PAC services to an additional cadre such as enrolled nurses is sought, a controlled pilot project and closely monitored evaluation should be designed and put in place to assure quality of care before amending the national SDGs. Community PAC interventions should be planned in parallel with service provision expansion activities. 6. Efforts should be made to standardize external supervision visits, so that each hospital gets a similar amount of time from supervisors. The MOHP RHU already has a standard supervisory checklist to guide visits, so the central level should monitor more closely that this is followed for each hospital each quarter. Onsite clinical supervisors should be well oriented to the PAC clinical protocol, including the MVA procedure, pain management, infection prevention, and FP counseling and method provision, and be encouraged to observe their clinical staff with patients to provide constructive feedback. To the extent possible, the MOHP should build the capacity of external and internal supervisors in supportive supervision and coaching skills. 48 JHPIEGO Technical Report REFERENCES AbouZahr C and T Wardlaw. 2003. Maternal Mortality in 2000: Estimates Developed by WHO, UNICEF and UNFPA. Bruce J. 1990. Fundamental elements of the quality of care: A simple framework. Studies in Family Planning. 21(2): 61–91. Kirkpatrick DL. 1994. Evaluating Training Programs: the Four Levels.: Berrett-Koehler Publishers: San Francisco, CA. Lacoste M. 2000. Malawi PAC Needs Assessment Reports. Unpublished. National Statistical Office [Malawi] and ORC Macro. 2001. Malawi Demographic and Health Survey 2000. National Statistical Office and ORC Macro: Zomba, Malawi and Calverton, MD. Sullivan R. 1995. The Competency-Based Approach to Training. Strategy Paper #1. JHPIEGO Corporation: Baltimore, MD. UNAIDS. 2002. Malawi Epidemiological Fact Sheet. JHPIEGO Technical Report A-1 APPENDIX A PAC EVALUATION STUDY QUESTIONS AND LINKAGES BETWEEN THE JHPIEGO PAC PROGRAM FRAMEWORK AND THE MALAWI NATIONAL PAC PROGRAM OBJECTIVES Broad Evaluation Study Questions How to Measure Effects? 1. Was the intervention carried out according to the plan? PAC site-based observations 2. How well was it carried out? PAC site-based observations, and interviews of trainers/service providers/administrators 3. Did it result in changes in knowledge, skills, and behaviours among the providers? What kinds of changes? PAC site-based observations, assessments and interviews of trainers/service providers/other onsite clinicians 4. Has the intervention improved services to clients? PAC client interviews, PAC site-based observations A-2 JHPIEGO Technical Report Malawi National PAC Program Objectives and Related Evaluation Questions 1. To raise awareness of the magnitude of the problem of incomplete abortion and its complications and the availability of PAC services (Level D) Specific Objective Being Evaluated JHPIEGO Project Design Linkage Type of Study Specific Evaluation Questions Establish task force to manage and coordinate PAC programme strategy development, programme design, and implementation 12 Advocacy & policy development 15 Organization of services Survey of National PAC Program Administrators Does the task force exist? Does it meet regularly? Who are the members? Are they knowledgeable? Are there clear objectives? Does it provide links to other programs? Are there meeting minutes available? Is the national programme well designed with clear objectives? Is the program being implemented? Develop and disseminate a comprehensive PAC policy as one component of the national SRH policy 12 Advocacy & policy development Survey of National PAC Program Administrators Was a policy created? Is it available? Method of dissemination? Content of policy? Who received the policy? Is it integrated into SRH policy (on paper and in practice)? Is it being implemented? Is it a realistic policy? Increase PAC knowledge/awareness of all cadres of staff working in sites where PAC services are offered 9 Expansion of PAC services 11 Inservice and preservice training PAC onsite research (OJT evaluation forms, TIMS, observation checklists) Has knowledge of staff at sites increased (knowledge retention)? Were staff trained in all 13 sites? Were there core people trained? Were staff trained on-the-job? How many were trained on-the￾job? Has management been briefed/oriented? Have all staff been oriented (top to bottom)? Staff skill and knowledge retention? Implementation of training program? Followup and refresher courses as needed? Are service delivery guidelines being adhered to? (application of knowledge and skills) Ensure involvement of community in design, development, and delivery of PAC services 8 Community Mobilization & BCI 12 Advocacy & policy development Community level survey ⇒ Evaluation Phase II Does the community know that there are PAC services available? Is the community receiving the services? Community involvement – Who (TBAs, CBDs, HSAs, youth, local leaders)? How? Community representation on national task force? How does community feel about PAC services? Community concerns identified and addressed? What channels were used to involve the community? JHPIEGO Technical Report A-3 2. Establish sustainable system for providing PAC services (Level D) Specific Objective Being Evaluated JHPIEGO Project Design Linkage Type of Study Data Collection Questions Strengthen PAC component of preservice education in Colleges of Medicine, Nursing and Health Sciences 11 Inservice and preservice training PAC onsite research (Administrator/provider surveys); preservice institution faculty/preceptor surveys and observation checklists; TIMS; National PAC Program Administrator Survey Do the curricula contain a PAC component? Have faculty been trained to implement the component? Ensure necessary equipment and supplies for PAC services are incorporated into national- and district￾level procurement/requisition systems (e.g., Reproductive Health Logistics Management Information System) 13 Provision of materials & equipment PAC onsite research (Administrator/provider surveys); National PAC Program Administrator Survey Inventory of PAC equipment established at the site level? PAC equipment/supplies included in the national essential equipment list/central medical stores? Does the list adhere to the PAC service delivery guidelines? MVA equipment included as standard supply item? Which PAC service provision items (FP, IP, emergency tx) are included in EHPLMIS? RHLMIS? 3. Increase accessibility of PAC services in an integrated SRH program (Level C) Specific Objective Being Evaluated JHPIEGO Project Design Linkage Type of Study Data Collection Questions Increase the number of service providers 6 Increased availability of PAC services PAC onsite research (Supervisory visit form), TIMS Percentage of RH providers who provide PAC services (will need to look at central hospital level and district hospital level) Percentage of RH providers trained in PAC (will need to look at central hospital level and district hospital level) Improve the availability of equipment, supplies and drugs necessary for PAC services 6 Increased availability of PAC services 7 Increased quality of PAC services PAC onsite research (Supervisory visit form, observation checklists) Have there been stockouts in the last 6 months? In the last month? Ensure PAC services are delivered in the most timely manner possible at the service delivery point 7 Increased quality of PAC services PAC onsite research (Client flow analysis) Is MVA decongesting gynecological wards? Are patients spending too much time waiting for initial treatment? A-4 JHPIEGO Technical Report 4. Provide quality PAC services (Levels B and C) Specific Objective Being Evaluated JHPIEGO Project Design Linkage Type of Study Data Collection Questions Ensure that PAC services are offered by trained and competent service providers PAC onsite research (Supervisory visit form, observation checklists), TIMS How many trained providers are onsite? Are any untrained clinicians providing PAC services? Establish and maintain a safe environment for delivery of PAC services PAC onsite research (Supervisory visit form, observation checklists) What is the complication rate following PAC procedures? What is the infection rate? Are the complications deemed preventable? Do more knowledgeable clinicians coach others to perform better when mistakes are made? Ensure that the appropriate technology is used in delivery of PAC services PAC onsite research (Supervisory visit form, observation checklists) What % of PAC patients is being serviced with D&C versus MVA? Ensure linkages are made to other SRH services PAC onsite research (Supervisory visit form, observation checklists, administrator surveys); National PAC Program Administrator Survey Are RH services available for referrals (cervical cancer screening, STI screening and tx, gender violence counseling, infertility services)? How many clients have been referred for each service in the past 6 months? In the past month? Monitor provider performance through existing supervision systems to ensure continued quality of care 3 Increased use of PAC services 7 Increased quality of PAC services 14 Supervision PAC onsite research (Administrator/provider surveys); National PAC Program Administrator Survey Can every provider identify her/his PAC supervisor? Is there a functional national supervision system? B-1 JHPIEGO Technical Report APPENDIX B RESULTS OF t-TESTS FOR ANALYSIS OF KNOWLEDGE SCORE DIFFERENCES BY HOSPITAL TYPE, TRAINING TYPE, AND CADRE t-Test to explore PAC knowledge score differential central vs. other hospitals SIGNIFICANT DIFFERENCE, with central hospitals higher Group Statistics 12 87.083 5.8225 1.6808 16 78.750 12.3153 3.0788 Hospital Type Central Other Total Score N Mean Std. Deviation Std. Error Mean Independent Samples Test 11.038 .003 2.162 26 .040 8.333 3.8538 .4116 16.2550 2.376 22.543 .026 8.333 3.5077 1.0689 15.5978 Equal variances assumed Equal variances not assumed Total Score F Sig. Levene's Test for Equality of Variances t df Sig. (2-tailed) Mean Difference Std. Error Difference Lower Upper 95% Confidence Interval of the Difference t-test for Equality of Means B-2 JHPIEGO Technical Report t-Test to explore PAC knowledge score differential group-based vs. OJT training delivery NO SIGNIFICANT DIFFERENCE Group Statistics 13 85.385 9.8872 2.7422 15 79.667 11.0948 2.8647 Training Type Group-based On-the-job Total Score N Mean Std. Deviation Std. Error Mean Independent Samples Test 1.225 .279 1.430 26 .165 5.718 3.9995 -2.5031 13.9390 1.442 25.971 .161 5.718 3.9656 -2.4339 13.8698 Equal variances assumed Equal variances not assumed Total Score F Sig. Levene's Test for Equality of Variances t df Sig. (2-tailed) Mean Difference Std. Error Difference Lower Upper 95% Confidence Interval of the Difference t-test for Equality of Means B-3 JHPIEGO Technical Report t-Test to explore PAC knowledge score differential CO vs. RNM provider cadres NO SIGNIFICANT DIFFERENCE Group Statistics 14 78.929 9.8408 2.6301 14 85.714 10.8941 2.9116 Cadre Clinical Officer Registered Nurse/Midwife Total Score N Mean Std. Deviation Std. Error Mean Independent Samples Test .038 .847 -1.729 26 .096 -6.786 3.9236 -14.8507 1.2793 -1.729 25.736 .096 -6.786 3.9236 -14.8548 1.2833 Equal variances assumed Equal variances not assumed Total Score F Sig. Levene's Test for Equality of Variances t df Sig. (2-tailed) Mean Difference Std. Error Difference Lower Upper 95% Confidence Interval of the Difference t-test for Equality of Means B-4 JHPIEGO Technical Report JHPIEGO Technical Report C-1 APPENDIX C AGE DISTRIBUTION OF FAMILY PLANNING ACCEPTORS AMONG MVA PATIENTS, BY METHOD Age distribution of Depo Provera acceptors among MVA patients Source: Malawi MOHP RHU <15 15-19 20-24 25-29 30-34 35-39 40-44 45+ Number of Patients 40 30 20 10 0 C-2 JHPIEGO Technical Report Age distribution of Pill acceptors among MVA patients Source: Malawi MOHP RHU <15 15-19 20-24 25-29 30-34 35-39 40-44 45+ Number of Patients 40 30 20 10 0 Age distribution of Condom acceptors among MVA patients Source: Malawi MOHP RHU <15 15-19 20-24 25-29 30-34 35-39 40-44 45+ Number of Patients 14 12 10 8 6 4 2 0 JHPIEGO Technical Report C-3 Age distribution of BTL acceptors among MVA patients Source: Malawi MOHP RHU <15 15-19 20-24 25-29 30-34 35-39 40-44 45+ Number of Patients 8 6 4 2 0 Age distribution of Norplant acceptors among MVA patients Source: Malawi MOHP RHU <15 15-19 20-24 25-29 30-34 35-39 40-44 45+ Number of Patients 3 2 1 0 C-4 JHPIEGO Technical Report