A Matched Case-Control Evaluation of the Knowledge and Skills of Midwives in Ghana Two Years after Graduation JHP-21 Prepared by Linda A. Fogarty, PhD Susi Wyss, MPH Kai Spratt, PhD, MPH, BSN Abigail Kyei, MPH, PHN Barbara Rawlins, MPH August 2003 United States Agency for International Development technical report CREDITS Editors: Katrin DeCamp, Jane Zadlo Sudbrink Copyright  2003 by JHPIEGO Corporation. All rights reserved. JHPIEGO is a nonprofit international health organization dedicated to improving the health of women and families. Established in 1973, JHPIEGO—affiliated with Johns Hopkins University and headquartered in Baltimore, Maryland— works in more than 30 countries through its collaborative partnerships with public and private organizations, and local communities. ACKNOWLEDGMENTS We would like to acknowledge the following organizations and individuals who helped make this evaluation possible: ♦ Ghana Ministry of Health: Dr. Henrietta Odoi-Agyarko, Dr. Ken Sagoe, and Said Al-Hussein ♦ Nurses and Midwives Council of Ghana: Ruth Gyang, Veronica Darko, Susannah Arthur, and Felicia Dagadu ♦ Clinical observers for the evaluation: Nana Kusi-Yeboah, Perfect Bleboo, Kate Agyei-Sakyi, Gladys Kankam, Mercy Ofori, Cynthia Bruce-Smith, Martha Serwah Appiagyei, Suzy Vander Bosche, Benedicta Appiah, Mary Dampson, Hectorina Yebuah, Ernestina Schandorf, Veronica Darko, and Ruth Gyang ♦ Interviewers for the evaluation: Ama N. Mensah, Wendy Boamah, Augusta Wellington, Mavis Opoku Asante, Veronica Osae, Phyllis Dako-Gyeke, Theodora Gbena, Felicity Tanoa, and Kate Badoe ♦ Consultants: Donna Sacknoff and Dr. Kwadwe Mensah ♦ United States Agency for International Development/Ghana: Ursula Nadolny, Jane Wickstrom, and Juliana Pwamang ♦ JHPIEGO/Baltimore: Dr. Chandrakant Ruparelia and Jessica Posner ♦ JHPIEGO/Ghana: Joyce Ablordeppey, Vera Ama Bimpong, and all JHPIEGO/Ghana Office staff TRADEMARKS: All brand and product names are trademarks or registered trademarks of their respective companies. NORPLANT® is the registered trademark of the Population Council for subdermal levonorgestrel implants. ZOE® is the registered trademark of Gaumard Scientific Co. for gynecologic simulators. This publication was made possible through support provided by the Service Delivery Improvement Division, Office of Population and Reproductive Health, Bureau for Global Health, U. S. Agency for International Development, under the terms of Award No. HRN-A-00-98-00041-00. The opinions expressed herein are those of JHPIEGO and do not necessarily reflect those of the U.S. Agency for International Development. Printed August 2003 JHPIEGO Technical Report i TABLE OF CONTENTS ABBREVIATIONS AND ACRONYMS .........................................................................................ii EXECUTIVE SUMMARY .............................................................................................................iii INTRODUCTION ..........................................................................................................................1 BACKGROUND............................................................................................................................1 METHODOLOGY .........................................................................................................................4 Procedure..........................................................................................................................4 Assessments and Measures .............................................................................................6 Statistical Methods ............................................................................................................8 RESULTS .....................................................................................................................................8 Demographics ...................................................................................................................8 Training and Preservice Experience .................................................................................9 Professional and Practice Information ..............................................................................9 Facility Resources and Supervision ................................................................................11 Knowledge ......................................................................................................................15 Skills................................................................................................................................15 DISCUSSION..............................................................................................................................17 CONCLUSION............................................................................................................................19 ii JHPIEGO Technical Report ABBREVIATIONS AND ACRONYMS AIDS Acquired immunodeficiency syndrome ANCOVA Analysis of covariance ANOVA Analysis of variance D&C Dilatation and curettage DHS Demographic and Health Survey EMNC Essential maternal and neonatal care EN Enrolled nurse FP Family planning HIV Human immunodeficiency virus HRDD Human Resources Development Division (of the Ministry of Health in Ghana) IP Infection prevention IR Intermediate result IUD Intrauterine device L&D Labor and delivery MVA Manual vacuum aspiration NMC Nurses and Midwives Council R&E Research and evaluation RH Reproductive health SO Strategic objective SRN State registered nurse STI Sexually transmitted infection USAID United States Agency for International Development JHPIEGO Technical Report iii EXECUTIVE SUMMARY JHPIEGO has collaborated with the Ghana Ministry of Health’s Human Resources Development Division and the Nurses and Midwives Council, with support from the United States Agency for International Development, to strengthen preservice education in family planning/reproductive health (FP/RH) and essential maternal and neonatal care (EMNC) at 12 midwifery training schools. JHPIEGO’s strategies for strengthening Ghanaian preservice education have included: ♦ Developing and implementing a standardized, competency-based curriculum ♦ Improving knowledge and skills of tutors and clinical trainers/preceptors ♦ Reinforcing service delivery sites used for clinical practice ♦ Providing schools and clinical training sites with anatomic models and supporting training materials The program has been implemented in three phases, beginning with two schools in Phase 1, three schools in Phase 2, and the remaining seven schools in Phase 3. This report summarizes an evaluation of the program at Phase 1 schools. The 2-year followup evaluation conducted in 2002 compared the knowledge and performance of midwives graduating from Phase 1 schools in 2000 (intervention group) to those of midwives graduating from the remaining schools during the same year (control group). Two years after graduation, 79 Phase 1 graduates and 79 control group graduates from non￾Phase 1 schools were matched by province, location (urban or rural), and facility type (e.g., hospital, polyclinic, health post), and invited to participate in the evaluation. Ultimately, 142 midwives participated in the study: 72 intervention group graduates and 70 control group graduates (90% overall response rate). Participants completed a 70-item knowledge questionnaire and an interview, and then rotated through four clinical skills simulation stations. Station 1 assessed proper handwashing technique, abdominal palpation, glove-wearing technique, vulval swabbing, insertion and removal of a speculum, and glove removal. Station 2 assessed placenta delivery by controlled cord traction, decontaminant preparation, and use of decontaminant to clean instruments. Station 3 assessed the quality of FP counseling. Station 4 assessed episiotomy repair and newborn resuscitation. Participants also provided data on FP and EMNC clinic services from their current workplace. Initially, the demographic characteristics and clinical experience of the midwives were assessed to verify comparability of groups. Although the midwives in the two groups were similar in most characteristics, there were some notable differences. Midwives trained at Phase 1 facilities were younger on average (42 years versus 46 years), worked fewer years as a nurse before midwifery training (15 years versus 18 years), and were more likely to be state registered nurses than those trained at other schools (61% versus 13%). After the assessment, the intervention and control groups’ average scores on the knowledge questionnaires were compared. As expected, intervention group midwives had higher knowledge iv JHPIEGO Technical Report scores overall (77%) compared to midwives in the control group (72%). In fact, intervention group midwives scored higher on every subscale and significantly higher on four of the eight subscales—antenatal care, FP services, partograph analysis, and infection prevention (IP). Finally, performance at the skills stations was compared between the intervention and control groups. The results revealed that the intervention group again performed better than the control group in several areas. Specifically, intervention group midwives performed significantly better on skills assessed in Stations 1 and 2, in particular on the handwashing, abdominal palpation, vulval swabbing, controlled cord traction, decontaminant preparation, and instrument cleaning subscales. Intervention group midwives also performed better on one of the two subscales in Station 3—they excelled at providing counseling on a FP method compared to the control group. The two groups were not different on the two subscales in Station 4. There was also a strong trend for midwives in the intervention group to perform better on total skills overall and on steps considered “critical” to performance of the skill. Potential barriers to optimal on-the-job performance were also assessed. In some cases, midwives’ ability to perform their jobs and maintain their skills was challenged by a lack of facility resources (e.g., enough beds for patients, not enough Ambu bags for infant resuscitation), and/or sporadic stockouts of supplies, particularly those needed for IP. In summary, 2 years after graduation, midwives trained at schools using the revised and updated training methods and curriculum were more knowledgeable and had better clinical skills compared to midwives trained at other schools throughout Ghana. In some content areas these differences were quite dramatic. In the area of FP, the intervention group excelled, scoring higher in FP knowledge and performing more competently in FP counseling skills compared to the control group. This evaluation also identified performance areas that were not well retained by either group (e.g., episiotomy repair and newborn resuscitation), areas that might require additional training or intensive on-the-job supervision, and support or reinforcement of the service delivery sites themselves. Further analysis of the data suggests two broad, recurring issues that affect midwifery performance and should be further examined and discussed by Ghanaian stakeholders—the need to continually strengthen the clinical portion of student training, and the need to address the environment in which midwives perform after they graduate. The study findings indicate that the competency-based technical assistance JHPIEGO provided to the midwifery training schools was effective in transferring knowledge and skills, and that the effects were maintained. These findings suggest that this training approach should be adopted for other technical areas or other cadres of providers in Ghana, and be applied to other preservice education institutions in the region. JHPIEGO Technical Report 1 A Matched Case-Control Evaluation of the Knowledge and Skills of Midwives in Ghana Two Years after Graduation INTRODUCTION JHPIEGO is collaborating with the Ghana Ministry of Health’s Human Resources Development Division (HRDD) and the Nurses and Midwives Council (NMC), with support from the United States Agency for International Development (USAID), to strengthen preservice education in family planning/reproductive health (FP/RH) and essential maternal and neonatal care (EMNC) at 12 midwifery training schools. The purpose of this program is to improve basic knowledge and skills in FP/RH and EMNC of midwives upon graduation. Improved knowledge and skills not only lead to improved quality of FP/RH and EMNC services; they also have the potential to decrease the burden on the inservice training system because new midwives are in less need of refresher (formerly “add on”) training such as FP (McDermott et al 1999). The program in Ghana was implemented in three phases, beginning with two schools in Phase 1, three schools in Phase 2, and the remaining seven schools in Phase 3. As of April 2003, the program was at the end of Phase 3. This report summarizes a 2-year followup evaluation conducted in 2002 comparing the knowledge and skills of midwives graduating from Phase 1 schools in 2000 to that of midwives graduating from the remaining schools during the same year. BACKGROUND Reproductive Health and Family Planning in Ghana With a current population of 20.2 million and a total fertility rate of 4.3, Ghana has made significant progress toward reducing population growth. The 1998 Demographic and Health Survey (DHS) showed a marked improvement in many areas related to FP and population. The total fertility rate decreased from 5.5 in 1993 to 4.6 in 1998, and modern contraceptive use increased from 10% to 13%. However, these improvements fall short of the goals set forth in the country’s 1994 population policy, and do not address the amount of unmet need for services in the country. The 1998 DHS found that 23% of Ghanaian women had an unmet need for contraceptives to space or limit births. Moreover, the high level of maternal mortality (590 per 100,000 births) indicates the need for expanded access to and improved quality of antenatal, labor and delivery (L&D), and postnatal care services (Population Reference Bureau and Ghana Statistical Service Division and Macro International, Inc. 1998). Although Ghana has shown encouraging signs of progress toward reaching the government’s population policy goals, national FP and EMNC programs need further strengthening to reach desired objectives. Program efforts must address the unmet demand for high-quality services. To this end, the Government of Ghana has supported, among other strategies, the strengthening of training in FP, RH, and EMNC for healthcare providers. 2 JHPIEGO Technical Report Midwifery Education in Ghana There are 12 midwifery training schools in Ghana,1 and two types of programs. Two schools (Kumasi and Korle Bu) have 1-year programs for state registered nurses (SRNs). The other 10 midwifery schools offer 2-year programs for enrolled nurses (ENs) and community health nurses already practicing in the health system. The 2-year program covers anatomy and physiology for antenatal, puerperium, L&D, and postnatal care (the first year of the program focuses on normal care; the second year covers abnormal care). Students follow a schedule of a 1-month class followed by a 2-month clinical rotation. By the end of the first year, when students are in normal L&D, they receive 2 weeks of FP training, enabling them to offer it in their practices. (It is recommended that every pregnant woman have an FP counseling visit and choose a method before she leaves the hospital after childbirth.) The Government of Ghana has focused on training ENs as midwives in order to put more midwives into the healthcare system, especially for service in peripheral/rural areas. Nurses attending 2-year schools are, on average, approximately 40 to 45 years old. Each school year, most schools have approximately 50 students, with one intake every September. It is known that these women have limited academic skills. They are not used to writing or reading in English, so their ability to express themselves in that language is somewhat limited. Ghana is changing to the semester system, with credits available for each semester finished. The SRN program will be six semesters long; upon successful completion, graduates will receive a diploma. (Afterwards, an additional 2 years of study is required to receive a Bachelor of Arts/Bachelor of Science degree.) If an SRN then goes on to midwifery school, there is another year of schooling. JHPIEGO’s Support to Strengthen Midwifery Education Since 1998, JHPIEGO has been working under the USAID/Ghana Results Framework2 to increase the use of RH services through quality improvement. The main objective has been to improve competencies of new midwives through strengthening their preservice education. In partnership with the NMC, and the HRDD and Reproductive and Child Health Unit in the Ministry of Health, JHPIEGO has been reinforcing midwifery education in 12 schools using a phased approach. During Phase 1, JHPIEGO worked with midwifery schools in Kumasi and Koforidua. In Phase 2, which began in 2000, activities focused on three additional schools (Korle Bu, Atibie, and Ashanti Mampong). During Phase 3, which began in late 2001 and will be completed in fiscal year 2003, JHPIEGO is working with the remaining seven midwifery schools. This phased approach allows lessons learned from initial phases to guide programming in the subsequent phases. 1 The 12 schools are: Komfo Anokye Teaching Hospital in Kumasi, Koforidua, Korle Bu, Atibie, Ashanti Mampong, Offinso–Maase, Hohoe, Berekum, Jirapa, Sekondi, Bolgatanga, and Number 37 Military Hospital. 2 USAID/Ghana’s Results Framework for the Strategic Objective (SO) of Improved Family Health (SO 3) includes Intermediate Result (IR) 3.1: Increased Use of RH Services with the following sub-IRs: (1) Improved Policies for RH Services; (2) Improved Quality of RH Services; and (3) Increased Access to RH Services. USAID/Ghana is currently revising their results framework and strategy. JHPIEGO Technical Report 3 JHPIEGO’s specific strategies for strengthening Ghanaian preservice education have included: Strategy 1: Developing and implementing a standardized, competency-based curriculum Activities under this strategy included the development of a curriculum strengthening team of approximately 20 persons at the start of the program. This group included tutors and preceptors from the schools and representatives from NMC and HRDD. This team had their clinical skills in FP and EMNC updated through a standardization course, which was followed immediately by a clinical training skills course in which they learned the “mastery learning approach”3 to training (Schaefer L et al 2000). Subsequently, the team was convened on two occasions to develop the Reproductive Health Classroom and Clinical Activity Guide for Training Midwives (also called the “RH Guide”). The RH Guide serves as a supplement to the midwifery curriculum. It includes activity outlines for specific content areas and learning guides and checklists that give a step-by￾step breakdown of clinical skills, as well as case studies and role plays. The guide is made up of four volumes (FP, antenatal care, L&D, and postnatal and neonatal care), and has one version for tutors/preceptors and another version for students. Strategy 2: Improving knowledge and skills of tutors and clinical trainers/preceptors Further to this strategy, tutors and clinical preceptors had their clinical skills in FP and EMNC updated through a standardization course and a clinical training skills course. The tutors and preceptors were provided with the RH Guide to be applied in their training (students received a student version of the guide as well). Subsequently, these tutors and preceptors received two followup visits from the national trainers who had trained them. Generally, the first followup visit took place 3 months after the training course and the second visit occurred 9 months after the training. The followup visits ensured that the tutors and preceptors were applying what they had learned in the courses, and also allowed trainers to work with them to solve specific problems. Furthermore, the visits encouraged a positive learning and working environment, and were a powerful motivator for the tutors and preceptors because the crucial role they play in the midwives’ learning process was being recognized. Strategy 3: Reinforcing service delivery sites used for clinical practice A key part of the learning experience for preservice education takes place in the clinics. In these clinics, students learn the practical skills and the clinical problem solving that they will use daily as providers. The training of preceptors had a direct effect on their competency as healthcare providers at the clinical site and on their competency as clinical trainers. In addition, the followup visits focused on the preceptors and their training sites. During these visits, specific problem-solving advice was given, as was encouragement toward ensuring that practices carried out by preceptors were consistent with what students learned in the classroom and followed in the RH Guide. 3 Mastery learning: Approach to learning that is based on the premise that all participants can master (learn) the required knowledge, attitudes, and skills, provided sufficient time is allowed and appropriate training methods are used. The goal of mastery learning is that 100 percent of the participants will “master” (learn) the knowledge, attitudes, and skills on which the training is based. 4 JHPIEGO Technical Report Strategy 4: Providing schools and clinical training sites with anatomic models and supporting training materials JHPIEGO provided each school with up-to-date reference materials, as well as selected training equipment and anatomic models, including the ZOE® gynecologic model and birthing simulators. Moreover, JHPIEGO worked with the midwifery schools to set up cost-sustainable systems to ensure re-supply of the student copies of the RH Guide. During implementation of the above strategies, two common programmatic approaches were consistently applied. The first approach was to involve stakeholders throughout the process in planning and crucial decision-making and to keep them informed of the progress of the program. Not only did this stakeholder involvement ensure that the program was culturally appropriate, it also ensured that the systems set up will continue to function after JHPIEGO is no longer involved. The second approach was the use of a “peer system”—using human resources built up in Phase 1 to provide technical assistance to schools in Phase 2, and using the resources developed in Phase 2 to assist with technical assistance to Phase 3 schools. For example, the strongest tutors and preceptors from Phase 1 schools, who were trained and followed up by national trainers, provided technical assistance to train and follow up the tutors and preceptors in Phase 2. This system not only reduced costs by decreasing both travel and use of higher paid national-level consultants, it also fostered a sense of “sisterhood” among schools. Moreover, it encouraged a productive exchange of experiences across the schools because effective problem solving at one school could be replicated in others. Objective of the Evaluation In 2001, the Research and Evaluation (R&E) Office at JHPIEGO/Baltimore and the JHPIEGO/Ghana Office began planning a matched case-control study to determine if Phase 1 school graduates performed better in knowledge scores and selected skills than non-Phase 1 school graduates once they were in the workforce. We hypothesized that the improved training program experienced by Phase 1 midwives would result in better performance of midwifery skills. METHODOLOGY Procedure JHPIEGO/Ghana staff and consultants located 92 of 110 Phase 1 graduates (intervention group) and 280 of 295 graduates of the 10 non-Phase 1 schools (control group) by telephone or site visit, and verified that they were working as midwives in FP/RH (midwives working in other areas such as pediatrics were excluded from the study). Graduates from the control group were then matched with intervention group midwives by province, location (urban, rural), and facility type (e.g., hospital, polyclinic, health post). In total, 158 midwives (79 from the intervention group, 79 from the control group) were informed about the evaluation; 142 of these participated in the study, 72 in the intervention group and 70 in the control group (90% overall response rate). Selected clinical skills evaluation checklists used in the RH Guide were reviewed and adapted for use in the evaluation by JHPIEGO master trainers, consultants, and JHPIEGO field staff in Ghana. Skills were chosen to reflect four major components of the RH Guide (FP, antenatal care, L&D, and postnatal care), and focused on essential midwifery skills (e.g., FP counseling, infection prevention [IP]) as well as skills that midwives may perform on an emergency basis JHPIEGO Technical Report 5 (e.g., episiotomy repair, newborn resuscitation). An interview questionnaire was developed that focused on midwives’ preservice education, current work environment, job satisfaction, level of supervision, and other variables hypothesized to affect skill performance. The skills checklists and interview questionnaire, along with the knowledge questionnaire and consent forms, were reviewed and approved by an independent institutional review board. HRDD sent all district health officers a letter requesting their cooperation by permitting midwives in their district to attend the evaluation. Six weeks before the assessment, letters were delivered to each midwife selected, requesting her participation in the evaluation and asking her to participate on a specific date. A followup letter from JHPIEGO was delivered to the midwives 3 weeks before the start of the study. Two weeks before the assessment began, a 6-day evaluation training was conducted by staff from the R&E and Learning and Performance Support Offices and with program staff from Ghana. The main foci of the training were to train evaluators in the systematic and objective collection of data and to standardize their skills to achieve observer or interviewer comparability. The principles of inter-rater reliability were reviewed and participants who were to conduct observations at clinical skills stations spent a day and a half practicing the skills that they would perform at the evaluation. The non-clinical evaluators were trained on the delivery of a standardized interview and reviewed all the knowledge and interview questions. They then used role plays to conduct interviews until they were familiar with the instrument. Twelve clinical evaluators (trainers who were assigned to assess clinical skills performance at one of the four skills stations), nine non-clinical evaluators (who would conduct interviews and supervise the knowledge questionnaire), and three staff from the NMC attended the training. All evaluation instruments were reviewed and revised, as needed, based on participant feedback during the training. At the end of the training, the clinical evaluators determined by consensus which steps on each observation checklist would be considered “critical”—that is, which steps must be performed correctly by the provider to ensure the safety of any patient or the integrity of a skill (e.g., provider must use a speculum to do a vaginal examination). The clinical evaluators agreed that 100% of critical steps needed to be done correctly and 65% of other steps must be done correctly for the provider to be considered competent in the skill, and that the passing grade for the knowledge questionnaire would be 75%. The study was conducted from 27 May–1 June 2002 at one site in the city of Kumasi, which is a transportation hub for the entire country. The study site was a large hotel where two conference rooms and four empty suites were set up to accommodate four skills stations equipped with anatomic models and other equipment or supplies needed to perform each clinical skill assessed.4 There was also a large room available in which to administer the knowledge questionnaire, and lounge areas in which to conduct the interviews in privacy. On each day of the evaluation, a new group of 24 midwives was welcomed by the JHPIEGO/Ghana staff and given a brief orientation to the day’s schedule. Throughout the day, participants rotated through four 30-minute skills assessment stations, and completed the interview and the knowledge questionnaire, each of which took approximately 1 hour. All midwives completed each section of 4 Copies of the skills checklists are on file at the JHPIEGO/Ghana Office and the JHPIEGO R&E Office in Baltimore, Maryland. 6 JHPIEGO Technical Report the evaluation. At the end of the day, the clinical evaluators provided a 90-minute skills update to the midwives. Midwives were not paid for their participation in the study; however, they were reimbursed for transportation, hotel, and meal costs and they received a free copy of the World Health Organization manual Management of Complications in Pregnancy and Childbirth, a pregnancy wheel, and a certificate of participation. Assessments and Measures Table 1 lists and defines the knowledge and skills scales and provides the number of items in each scale. Table 1. Knowledge and Skills Scale Information Source Definition Number of Questions Total FP services 20 Total partograph case study 14 Total childbirth care 9 Total antenatal care 9 Total postpartum care 6 Total postpartum hemorrhage 6 Total IP 4 Total sexually transmitted infection (STI)/HIV/AIDS services 2 Knowledge Questionnaire Total knowledge score: 8 subscales 70 Abdominal palpation 14 Wearing gloves 11 Handwashing 10 Vulval swabbing 9 Insertion of a speculum 8 Removal of a speculum 6 Glove removal 4 Total clinical skills Station 1 62 Skills Assessment Checklists Station One: Antenatal Care Total critical steps Station 1 10 JHPIEGO Technical Report 7 Table 1. Knowledge and Skills Scale Information (continued) Source Definition Number of Questions Controlled cord traction 9 Decontaminant preparation 1 Instrument decontamination 3 Cleaning instruments 8 Total clinical skills Station 2 21 Skills Assessment Checklists Station Two: L&D Total critical steps Station 2 9 Medical history 11 FP method information 7 Total clinical skills Station 3 24 Station Three: Family Planning Total critical steps Station 3 5 Episiotomy repair 25 Newborn resuscitation 15 Total clinical skills Station 4 40 Station Four: Postpartum Total critical steps Station 4 12 Total clinical skills all stations 147 Overall Total critical steps all stations 36 Knowledge Questionnaire: The 70-item scale tested midwives’ knowledge in eight areas: FP services, partograph interpretation, childbirth care, antenatal care, postpartum care, postpartum hemorrhage care, IP, and STI/HIV/AIDS services. The questionnaire was self-administered and consisted of a series of true/false, multiple choice, and short answer/comprehension questions about case studies. Subscale and total scale scores were calculated. Incountry trainers set a goal of 75% correct for knowledge score achievement. Skills Assessment: At each of the four skills stations, a clinical observer (a trainer) read a brief scenario to the midwife who was then instructed to proceed on a set of tasks as she would at her job site, demonstrating skills with anatomic models or through role play with a study observer. The observer used a checklist to assess midwives’ skills at each of the stations. Station 1 assessed proper handwashing technique, abdominal palpation, gloving technique, vulval swabbing, insertion and removal of a speculum, and glove removal. Station 2 assessed placenta delivery by controlled cord traction, decontaminant preparation, instrument decontamination, and cleaning of instruments. Station 3 assessed the quality of FP counseling. Station 4 assessed episiotomy repair and newborn resuscitation. Subscale and total scale scores were calculated. Incountry trainers set a goal of 65% correct for skill score achievement and identified a set of 36 “critical skills” that should be performed correctly by all midwives. Interview: The interview questions were read to each midwife. The interview collected midwives’ background (e.g., age, number and age of children, marital status, religion) and professional information (e.g., years working as a nurse before midwifery school, type of nurse, 8 JHPIEGO Technical Report length of midwifery training, type of facility, primary current job responsibility). Information on the facility where the midwife worked—such as availability of services, FP methods, and procedures—was also collected. Midwives reported on supervision quality and type, inservice training, and availability and use of the National RH Policies and Protocols. Finally, midwives reported on job satisfaction and their town’s resources, such as availability of electricity, potable water, and roads. Clinic Statistics: A standardized form to collect clinic statistics was sent to each midwife who was asked to return it on the day of the evaluation. This survey form asked participants to provide the following data on FP clients seen at the facility: followup visits for each of 12 methods in each of the previous 3 months; availability of methods; and whether the clinic ran out of any of these methods in the previous month. Midwives also recorded total number of childbirths and antenatal care attendance at their facility in each of the previous 3 months. Statistical Methods Univariate statistics were performed on all variables of interest. Analysis of variance (ANOVA) was used to compare the average knowledge and skills scores of midwives in the intervention and control groups. Because the original hypotheses were one-tailed—that is, midwives in the intervention group were hypothesized to perform better on knowledge and skills assessments—p<.10 was chosen as the study significance level. For all other analyses, however, where no a priori hypotheses existed, p<.05 was used to judge statistical significance. Chi-square tests were used to determine bivariate associations between independent variables and group membership. Independent variables associated with group membership were used as covariates in analyses of covariance (ANCOVA) using the same knowledge and skills outcomes. RESULTS The two groups were compared on a variety of variables related to demographic characteristics, preservice experience, professional and practice information, facility resources, service provision variables, training, and supervision (see Tables 2–7). Demographics Midwives trained at the Phase 1 facilities were on average younger (42 years versus 46 years) and had fewer children (2.5 versus 3.2 children; see Table 2). Overall, most participants were married (75%) and Christian (91%). JHPIEGO Technical Report 9 Table 2. Demographic Information by Group Category Values Groups Intervention (n=72) Control (n=70) Age Average 42 46* Marital Status % Single 8 10 % Married 79 70 % Divorced 7 9 % Separated 3 1 % Widowed 1 10 % Not married, living with partner 1 0 Number of Children Average 2.5 3.2* Religion % Muslim 5 11 % Christian 93 89 % Other 1 0 * p<.05 Training and Preservice Experience Midwives in the two groups reported similar training and preservice experiences. The exception to this, as expected, was that the RH classroom and clinical activity guide was used significantly more often by the intervention group (Table 3). Table 3. Training and Preservice Experience by Group Category Groups Intervention (n=72) Control (n=70) % Satisfied with midwifery training 99 100 % Reported clinical skills were assessed with checklist 97 84 % Used RH classroom and clinical activity guide 94 28* * p<.05 Overall, nearly all of the participants reported they were either satisfied or very satisfied with their midwifery training. The overwhelming majority in both groups reported that their clinical skills were assessed with checklists. Professional and Practice Information Overall, the professional and practice information for the two groups was very similar except in two areas: intervention group midwives had worked on average fewer years as nurses before midwifery training (15 years versus 18 years), and more of them were SRNs (61% versus 13%; 10 JHPIEGO Technical Report see Table 4). This latter difference was expected because of the differences in entrance criteria between the 1-year and 2-year midwifery programs (i.e., the 1-year programs accept SRNs). The percentages cited clearly reflect that, as previously mentioned, 1 of the two Phase 1 schools has a 1-year program, and only 1 of the 10 other schools has a 1-year program. Because both of these differences could have an effect on skills performance (especially since ENs tend to have more limited academic skills than the SRNs), they were included as covariates in analyses testing the impact of training on skills and knowledge.5 Table 4. Professional and Practice Information by Group Category Values Groups Intervention (n=72) Control (n=70) Years of work Average number 15 18* Type of nurse before midwifery EN 38 79* training (%) SRN 61 13 Community health nurse 1 16 Primary responsibility (%) Clinical services 99 100 Other 1 0 Type of facility (%) Public 88 87 Other 12 13 Hospital setting (%) Yes 90 89 % Providing service daily during L&D 85 91 busy season Postpartum FP counseling 67 57 Antenatal care 56 51 Postnatal care 54 46 Incomplete abortion referral 44 43 Supply FP methods 33 30 STI treatment 29 34 HIV/AIDS counseling/education 32 29 STI testing 15 11 5 Described in further detail in the Results section. JHPIEGO Technical Report 11 Table 4. Professional and Practice Information by Group (continued) Category Values Groups Intervention (n=72) Control (n=70) Midwives (%) who, since training, Repaired episiotomy > 10 times 60 52 had: Resuscitated newborn > 10 times 51 61 Performed speculum examination > 10 times 6 3 Ever inserted IUD 8 10 Inservice training since graduation (%) None 63 56 Used National RH Policies and Protocols (%) Yes 68 62 * p<.05 For nearly all participants, their primary responsibility was clinical services, and most worked for public facilities (88%) and in a hospital setting (90%). The amount of time midwives spent in their practice providing a variety of services was also similar between groups. Overall, L&D was the service provided daily by the largest percentage of participants (88%), followed by postpartum FP counseling (62%), antenatal care (54%), and postnatal care (50%). Since training, more than half of the midwives had both repaired episiotomies and resuscitated newborns more than 10 times. Facility Resources and Supervision The ability to perform on the job depends not only on the training, skills, and knowledge of the midwives, but also on institutional factors such as facility resources and supervision. To provide a broader understanding of the institutional setting in which the midwives worked, during the interviews, the evaluation team collected data on facility resources and supervision practices at their institutions (see Tables 5–7). In general, the two groups of midwives reported similar resource and supervision situations. Most respondents reported that their facilities’ waiting areas had a roof (99%), adequate lighting (89%), a toilet (80%), and patient education materials (76%). Nearly all of the midwives (97%) believed the facility needed additional staff, and close to two-thirds reported adequate seating for all patients (62%) (Table 5A). 12 JHPIEGO Technical Report Table 5A. Facility Resources Variables Percentage (n=142) % Reported facility needs additional staff 97 % of midwives whose facilities’ waiting areas had: A roof 99 Adequate lighting 89 A toilet 80 Patient education materials 76 Adequate seating 62 Videocassette player for patient education 18 The most available method to treat incomplete abortions was dilatation and curettage (D&C), which was available at 78% of facilities. Providing patients with oxytocin, antibiotics, and a referral to another facility was reported by 40% of the respondents. Only 1% of midwives reported that nevirapine was available on-site for HIV-infected women during labor (Table 5B); an additional 23% did not know whether it was available. Table 5B. Facility Resources Variables Percentage (n=142) % Reported these methods were available for incomplete abortions: D&C only 48 Manual vacuum aspiration (MVA) and D&C 30 MVA only 2 Oxytocin, antibiotics, and referral 40 Other 9 % Reported nevirapine is available 1 % Reported no inservice training 59 All midwives stated that medical records were kept for patients in their facilities, but not all sites had records for all FP visits. More than 80% of midwives in both groups reported having access to an operating theater and rooms for private counseling. In the labor and lying-in wards, 47% and 45% reported sufficient beds, respectively; 82% reported having water available daily. Respondents reported occasional supply stockouts in the previous month, particularly for IP supplies. In the previous month, protective clothing (e.g., utility gloves and aprons) was unavailable in more than one-third of facilities in both groups, and chlorine bleach was unavailable in 15% of facilities (Table 5C). However, malaria medication, folic acid, iron tablets, and multivitamins were available in more than 96% of sites. JHPIEGO Technical Report 13 Table 5C. Facility Resources Variables Percentage (n=142) % of midwives whose facilities had: Medical record for each patient 100 Laboratory space 93 Records for all FP visits 89 Storage cabinet for contraceptives 85 Access to operating theater 86 Rooms for private counseling 85 Water available daily 82 Supply shortages that affect job 63 Sufficient beds in labor ward 47 Sufficient beds in lying-in ward 45 % Reported stockouts in the previous month of: Protective clothing 35 Chlorine bleach 15 A vaccine 15 Drugs for pain management 10 Regarding the availability of FP methods, participants reported that, overall, a wide range of FP methods were available and were provided at their facilities (Table 6). The most commonly provided FP methods (averaged over the 3 previous months) were male condoms, oral contraceptives, and injectables, whereas female sterilization and natural FP methods were rarely provided. FP methods most commonly available at sites were oral contraceptives (94%), followed by male and female condoms (93%), and injectables (93%). In fact, with the exception of spermicide and the diaphragm, all methods were offered by at least 83% of facilities. Fortunately, stockouts of the most commonly used contraceptive methods were less frequent than for less popular FP methods. Percentage stockouts in the previous month were 11% for male condoms, 11% for oral contraceptives, and 6% for injectables. More than 20% of midwives reported facility stockouts in the previous month for diaphragms, Norplant® implants, and spermicide (Table 6). 14 JHPIEGO Technical Report Table 6. FP Method Use and Availability at Participants’ Sites Variables Average Monthly Returns per Site* Percentage of Sites Offering Percentage of Stockouts Previous Month Male condoms 747 93 11 Oral contraceptives 236 94 11 Injectables 176 93 6 Foam 137 90 10 Lactational amenorrhea method 115 89 N/A Spermicide 101 57 24 Diaphragm 93 15 39 Female condoms 57 93 7 Norplant implants 11 86 25 IUD 10 87 15 Female sterilization 7 86 N/A Natural methods 3 83 N/A * Averaged over the past 3 months at each site where method was offered. Of the variables examined for facility resources, five variables revealed significant differences between groups. Midwives in the intervention group were more likely to work in sites where both MVA and D&C were available; where oxytocin, antibiotics, and referral were used for postabortion care; and where spermicides were available. The facilities of midwives in the control group were more likely to have D&C only for postabortion treatment, and were more likely to have had a vaccine stockout in the previous month. Although these are important characteristics of practice, it was not believed that these differences would have a significant impact on midwives’ performance on the knowledge and skills assessments in this evaluation. Sufficient, high-quality supervision is also required for maximum on-the-job performance (Caiola and Sullivan 2000). Midwives in both groups reported similar quality and quantity of supervision and support. Nearly all midwives (94%) reported having a supervisor on site, and 74% received job performance feedback from this person (Table 7). However, 10% said that they did not receive much support from their supervisor, and 13% said the supervisor did not solve problems very well. Table 7. Supervision Quality Variables Percentage % with on-site supervisor 94 % who received feedback from supervisor 74 % with a lot of support from supervisor 42 % whose supervisors solved problems very well 42 JHPIEGO Technical Report 15 Knowledge Average scores on the knowledge questionnaire were compared for the intervention and control group participants. As expected, intervention group midwives had higher knowledge scores overall, with an average of 77% correct compared to 72% correct for midwives in the control group (Table 8). The intervention group scored higher on each subscale and significantly higher on half of the subscales—antenatal care, FP services, partograph analysis, and IP. In fact, the average score for the intervention group was more than 75% for six of the eight subscales; in the control group, only four of the eight subscales were 75% or higher. For both groups, IP and postpartum care were the topics with the most incorrect answers. Table 8. Percentage of Correct Knowledge Subscale Scores by Group Group Category Intervention (n=72) Control (n=70) F p value STI/HIV/AIDS services 97 95 .46 .499 Postpartum hemorrhage 88 84 2.46 .199 Antenatal care 79 75 2.79 .097 FP services 78 70 14.32 .000* Childbirth care 77 76 .02 .878 Partograph case study 77 73 3.16 .077 IP 61 48 12.01 .001* Postpartum care 42 41 .66 .420 Total knowledge score: 8 subscales 77 72 12.9 .000* * After adjustment for significant covariates, p value remained less than .20. Because there were some imbalances between the groups in terms of age, number of years on the job, and level of training before midwifery school, analyses were re-run with these variables as covariates. The aim was to determine whether these variables (and not differences in midwifery training) were responsible for the differences in knowledge between the groups. The pattern of differences between the two groups remained the same after the covariates were included—midwives in the intervention group still had higher knowledge scores overall than those in the control group. Skills Intervention and control group performance in the skills stations was compared. The intervention group again performed better than the control group in several areas. Specifically, intervention group midwives performed significantly better on total skills assessed in Stations 1 and 2, scoring more than 70% correct on each (Table 9). Intervention group midwives performed significantly better on the following subscales: handwashing, abdominal palpation, vulval swabbing, removing gloves, controlled cord traction, decontaminant preparation, and instrument cleaning. 16 JHPIEGO Technical Report Intervention group midwives also performed better than control group midwives on one of the two subscales in Station 3—intervention group midwives excelled at providing information and counseling on an FP method (i.e., describing how the method works, effectiveness, side effects) compared to the control group (Table 9). In Station 3, it initially appeared that control group midwives performed reproductive skills and took medical history more adeptly than intervention group midwives (Table 9). Further analyses were performed to determine the reasons for this unexpected difference. From previous analyses, it appeared that the groups were similar in terms of experience providing medical histories and access to resources. Analyses revealed, however, that one of the three observers rating the skills of Station 3 provided significantly lower scores than the other two raters; she also observed 61% of the intervention group midwives performing these skills. When observer was used as a covariate in a test of the differences between mean medical history scores, the groups were no longer significantly different. In fact, for that observer, intervention group midwives’ scores were on average higher than control group scores. Table 9. Percentage of Correct Skill Scores by Group Group Category Intervention (n=72) Control (n=70) F p value Station One: Antenatal Care Handwashing 91 81 11.18 .001 Abdominal palpation 48 37 7.38 .007 Wearing gloves 88 84 2.54 .113 Vulval swabbing 71 57 10.28 .002* Insertion of a speculum 63 59 .90 .345 Removal of a speculum 72 79 2.18 .142 Glove removal 79 71 3.31 .071* Total clinical skills Station 1 71 64 9.49 .002* Total critical skills Station 1 66 64 .32 .575 Station Two: L&D Controlled cord traction 69 58 15.75 .000* Decontaminant preparation 65 34 14.87 .000* Instrument decontamination 85 88 1.11 .294 Cleaning instruments 82 70 13.02 .000* Total clinical skills Station 2 76 66 19.94 .000* Total critical skills Station 2 71 60 14.34 .000* JHPIEGO Technical Report 17 Table 9. Percentage of Correct Skill Scores by Group (continued) Group Category Intervention (n=72) Control (n=70) F p value Station Three: FP Medical history 58 81 16.11 .000** FP method information 41 28 11.22 .001* Total clinical skills Station 3 50 55 1.32 .249 Total critical skills Station 3 61 68 2.79 .097 Station Four: Postpartum Care Episiotomy repair 49 47 .37 .543 Newborn resuscitation 41 47 2.09 .151 Total clinical skills Station 4 46 47 .015 .902 Total critical skills Station 4 40 39 .098 .754 Overall Total clinical skills all stations 61 58 2.78 .099 Total critical skills all stations 58 55 2.25 .136 * After adjustment for significant covariates, p value remained less than .20. ** After adjustment for observer, no difference between groups was found. Finally, intervention group midwives performed better on total clinical skills overall, doing 61% of the skills correctly, compared to 58% for control group midwives (Table 9). No one performed all of the 36 critical steps correctly, and the majority of midwives missed one-third of the critical steps. A closer look at the missed critical steps revealed that only about half of the midwives from both groups performed newborn resuscitation critical steps correctly. As with the knowledge score analyses, because there were some imbalances between the groups in terms of age, number of years on the job, and level of training before midwifery school, analyses on each of the skill subscales were re-run with these variables as covariates. Similar to the knowledge scale ANCOVAs, the pattern of differences between the two groups remained the same after the covariates were included. Again, midwives in the intervention group had higher skills assessment scores overall than those in the control group. DISCUSSION The intervention group, whose preservice education was based on the updated RH Guide and improved training methods and materials, had exceptional knowledge and skills 2 years after graduation compared to matched midwives who graduated in the same year, but used the standard training curriculum. For example, in the area of FP, intervention midwives had significantly higher knowledge scores in FP services and were more skilled at providing appropriate FP counseling (such as describing the side effects and effectiveness of a FP method, and describing how the FP method works to prevent pregnancy). The evaluation also demonstrated the role that training materials—in this case the RH Guide—can play in 18 JHPIEGO Technical Report standardizing practical training and providing more structure to this crucial period of a student’s education. It is also important to note that the work under Phases 2 and 3 is expected to have even better results. This is because Phase 2 and 3 implementation included integration of a finalized version of the RH Guide as well as use of the performance improvement approach to support clinical training sites to meet predetermined standards. Although the effect of these inputs could not be captured under this assessment because they affected graduates after 2000, the assumption is that the same if not higher results were achieved under Phases 2 and 3. Before the evaluation, clinical skills evaluators identified a target level of 75% for knowledge score achievement. The intervention group achieved this level, answering on average 77% correctly on the knowledge questionnaire. In fact, on six of the eight subscales, the average intervention group score exceeded 75%. Considering that the evaluation took place 2 years post-graduation, these scores are quite strong. On the other hand, the control group did not achieve the desired level overall, scoring 72% for total knowledge. Additionally, only four of the eight subscales exceeded 75% for the control group. Similarly, study evaluators identified a target level of 65% of skills correct overall. Of the 14 subscales considered,6 the intervention group exceeded the target in nine areas, whereas the control group exceeded the target in six areas. Midwives in both groups performed fairly well on simulations of standard antenatal care visits and management of the third stage of labor. This evaluation also suggested performance areas in which skills were not well retained, and areas that might require additional training or on-the-job supervision and support. For example, the average scores for the two Station 4 subscales—modeling episiotomy repair and newborn resuscitation—were less than 50%. Further, as stated previously, no one performed 100% of the critical steps correctly, and half of the failed critical steps addressed newborn resuscitation. (One possible reason for this finding is that evaluators learned from study participants that the Ambu bags for resuscitating the newborn model in Station 4 were not available at many midwives’ facilities.7 Although the midwives were trained in school to use the bag, they had never used one since then.) Overall, these findings suggest that further exploration is needed to determine how the skills in newborn resuscitation and episiotomy repair were taught, and whether the schools tested competency of the midwives in this area before they graduated. This information may in turn suggest the need for possible changes either at the schools or worksites to support skill learning and maintenance. This example also illustrates two broad, recurring issues that affect midwives in both the intervention and control groups. These issues will need to be further examined and discussed by Ghanaian stakeholders, particularly as JHPIEGO’s role in strengthening preservice education in Ghana draws to a close. These two broad issues are: ♦ The need to ensure consistency across clinical training sites: Although use of the mastery learning approach is common among the midwifery tutors and preceptors, the findings from this study suggest that the clinical portion of midwifery education continues to need strengthening to ensure consistency across sites. This is an important issue because the number of students has been on the rise and not all of the clinical training sites seem able to accommodate the students they already receive. Moreover, it is important for these 6 Medical history subscale was not included because scoring was inconsistent across observers. 7 The site where the evaluation tools were field-tested had Ambu bags, but is apparently exceptional in this respect. JHPIEGO Technical Report 19 sites to simulate what the students will experience at their worksites by using appropriate technology. JHPIEGO worked with NMC and HRDD in Phase 2 of this program to set performance standards for clinical training sites. These should continued to be used by HRDD and NMC to ensure consistency across clinical training sites. ♦ The need to address the environment in which midwives perform after they graduate: The performance of midwives in both the intervention and control groups indicated that even if they are more competent and knowledgeable as a result of improvements in preservice education, their performance ultimately also depends upon the environmental factors that they encounter when they work at the health facilities to which they are deployed. For example, some sites lacked adequate resources (e.g., enough beds for patients, not enough Ambu bags for infant resuscitation), and/or experienced supply stockouts (particularly for IP supplies), and nearly 20% did not have water available daily. Clearly, these constraints create challenges for midwives who are trying to perform their jobs and maintain their skills. Moreover, they clearly had an effect on ultimate performance in this evaluation. CONCLUSION The findings from this study indicate that 2 years after graduation, midwives trained at schools using the revised and updated training methods and curriculum were more knowledgeable and had better clinical skills compared to midwives trained in other schools throughout the country. This was true despite the challenging environmental circumstances that both groups of midwives often faced in their work. In some content areas, these differences were quite dramatic. In the area of FP, the intervention group excelled, scoring higher in FP knowledge and performing more competently in FP counseling skills compared to the control group. It is expected that the same (if not higher) results were achieved under Phases 2 and 3. These results demonstrate that this training approach should be adopted by other technical areas or for other cadres of providers in Ghana, and be applied to other preservice education institutions in the region. Within Ghana, this program approach was expanded to Phase 2 and 3 midwifery schools. JHPIEGO has also been adapting this approach to strengthening preservice education at six Community Health Nursing Schools to integrate content for community health officers under the CHPS Project (Community-Based Health Planning and Services) into community health nursing education. Moreover, the experiences in Ghana are being shared with its neighboring countries, most notably at a regional meeting on preservice education in West and Central Africa, held in Cotonou, Benin, in September 2002. Attended by representatives from preservice institutions in 18 countries, this meeting was an opportunity to showcase the results of this approach as applied in the context of the midwifery schools in Ghana, and for other schools in the region to learn how they might apply this approach in their own contexts. 20 JHPIEGO Technical Report JHPIEGO Technical Report 21 REFERENCES Brechin SJG et al. 2000. Strengthening Preservice Midwifery Education in Ghana: Achievements and Phase 2 Expansion Plans. JHPIEGO Technical Report JHP-13. JHPIEGO Corporation: Baltimore, MD. Caiola N and R Sullivan. 2000. Performance Improvement: Developing a Strategy for Reproductive Health Services. Strategy Paper #8. JHPIEGO Corporation: Baltimore, MD. Ghana Statistical Service Division and Macro International, Inc. 1999. Ghana Demographic and Health Survey 1998. Ghana Statistical Service Division and Macro International: Calverton, MD. Ghana Statistical Service Division and Macro International, Inc. 1994. Ghana Demographic and Health Survey 1993. Ghana Statistical Service Division and Macro International: Calverton, MD. McDermott J et al. 1999. MotherCare/Indonesia Training Evaluation Report. MotherCare/John Snow, Inc.: Arlington, VA. Population Reference Bureau. 2002. World Population Data Sheet. Population Reference Bureau: Washington, DC. Sacknoff D. 2002. Trip Report: Ghana, 30 March–10 April 2002. JHPIEGO Corporation: Baltimore, MD. Schaefer L et al. 2000. Advanced Training Skills for Reproductive Health Professionals. JHPIEGO Corporation: Baltimore, MD. Spratt K, C Ruparelia and D Sacknoff. 2002. Trip Report: Ghana, 8 May–4 June 2002. JHPIEGO Corporation: Baltimore, MD. 22 JHPIEGO Technical Report