March 2014 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Pamela Putney, Sean Drysdale, and Roy Mutandwa through Social Impact, Inc. End of Project Evaluation of the Maternal and Child Health Integrated Program (MCHIP) in Zimbabwe ACKNOWLEDGEMENTS The Evaluation Team would like to thank all the staff at the MOHCC (Ministry of Health and Child Care) in Harare and Manicaland who took time out of their busy schedules to talk to us about the MCHIP program; the dedicated USAID Zimbabwe staff who provided on-going feedback and support throughout the evaluation, especially Jo Keatinge, Matthews Maruva and Ioli Filmeridis; all the health workers who patiently and kindly answered our questions and filled out a lengthy survey, despite long lines of patients waiting for their attention; and the Village Health Workers who traveled long distances and often waited hours to speak to us so enthusiastically about their role in improving health in their communities and then sang us songs in extraordinary harmonies about their work. END OF PROJECT EVALUATION OF THE MATERNAL AND CHILD HEALTH INTEGRATED PROGRAM (MCHIP) IN ZIMBABWE Final Evaluation Report March 5, 2014 AID-613-TO-13-00002 under Contract AID-RAN-I-00-09-00019 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. CONTENTS Acronyms .....................................................................................................................................................1 Executive Summary ....................................................................................................................................3 Introduction and Background ...............................................................................................................3 Evaluation Purpose, Team, Methodology ............................................................................................3 Findings and Conclusions .....................................................................................................................4 Recommendations ..................................................................................................................................7 Evaluation Purpose & Evaluation Questions ...........................................................................................9 Evaluation Purpose ................................................................................................................................9 Evaluation Questions ...........................................................................................................................10 Project Background ..................................................................................................................................11 Context ...................................................................................................................................................11 Project Overview ...................................................................................................................................13 MCHIP Project Vision, Goal and Objectives ......................................................................................15 Evaluation DESIGN, Methods & Limitations ..........................................................................................17 Focused Document Review .................................................................................................................18 Qualitative Data .....................................................................................................................................18 Key Informant Interviews .....................................................................................................................18 Focus Group Discussions ...................................................................................................................19 Quantitative Data ..................................................................................................................................20 Health Worker Survey ..........................................................................................................................20 Quality Assurance and Limitations of Data Collection .....................................................................20 Findings, Conclusions & Recommendations.........................................................................................22 Findings and Conclusions ...................................................................................................................22 Recommendations ................................................................................................................................42 1 ACRONYMS ANC Antenatal Care ARK Absolute Return for Kids Project CBO Community-Based Organization CBT Competency Based Training CHERG Child Health Epidemiology Reference Group cHMIS Community Health Management Information System cMNCH Community Maternal Neonatal Child Health cPQI Community Performance Quality Improvement DEC Development Exchange Clearing House DFID Department for International Development (United Kingdom) DH District Hospital DNO District Nursing Officer EGPAF Elizabeth Glaser Paediatric AIDS Foundation EHO Environmental Health Officer EmONC Emergency Obstetrics and Newborn Care EPI Expanded Program for Immunization EU European Union FANC Focused Antenatal Care FGD Focus Group Discussion FP Family Planning HBB Helping Babies Breathe HF Health Facility HMIS Health Management Information System HR Human Resources HTF Health Transition Fund HW Health Worker IMNCI Integrated Management of Newborn and Child Illnesses IP Infection Prevention IPTp Intermittent Presumptive Treatment of Malaria in Pregnant Women ISP Integrated Support Program IRC International Rescue Committee IYCF Infant and Young Child Feeding JHPIEGO Johns Hopkins Programme for International Education in Gynaecology and Obstetrics JSI John Snow, Inc. KI Key Informant KII Key Informant Interview KMC Kangaroo Mother Care LBW Low Birth Weight LiST Lives Saved Tool LWA Leader with Associates MCHIP Maternal & Child Health Improvement Project MDG Millennium Development Goal M&E Monitoring and Evaluation MMR Maternal Mortality Ratio MNCH Maternal, Newborn, and Child Health 2 MNH Maternal and Newborn Health MOHCC Ministry of Health and Child Care MPMA Maternal Perinatal Mortality Audit MSI Management Services International NGO Non-Governmental Organization NMCP National Malaria Control Program NIHFA National Integrated Health Facility Assessment PCV Pneumococcal Vaccine OPHID Organization for Public Health Interventions and Development ORT Oral Rehydration Therapy PATH Program for Appropriate Technology in Health PA Pharmacy Assistant PCN Primary Care Nurse PCV Pneumococcal Vaccine PMTCT Prevention of Mother to Child Transmission PNC Post-natal Care PPH Post-Partum Hemorrhage PQI Performance Quality Improvement PSI Population Services International PSZ Population Services Zimbabwe QA Quality Assurance QI Quality Improvement QoC Quality of Care RBF Results Based Financing RDT Rapid Diagnostic Testing RED Reaching Every District RGN Registered Graduate Nurse RH Reproductive Health RHC Rural Health Center RHF Rural Health Facility SBM-R Standards Based Management and Recognition SI Social Impact, Inc. SRH Sexual and Reproductive Health SS Supportive Supervision TWG Technical Working Group UNICEF United Nations Children Fund UNFPA United Nations Population Fund USAID United States Agency for International Development USG United States Government VHW Village Health Worker WEI World Education, Inc. WHO World Health Organization WRA White Ribbon Alliance ZACH Zimbabwe Association of Church Hospitals ZMPMS Zimbabwe Maternal and Perinatal Mortality Study 3 EXECUTIVE SUMMARY INTRODUCTION AND BACKGROUND The Maternal and Child Health Integrated Program (MCHIP) is the United States Agency for International Development (USAID) global flagship maternal and child health program implemented by a consortium of US-Based non-governmental organizations (NGOs). In 2008, Zimbabwe’s socioeconomic situation had deteriorated to the point that health services had all but ceased to function. As health indicators declined to critical levels, USAID identified maternal and child health as a significant area of need in Zimbabwe and requested MCHIP to undertake a rapid assessment in 2009, followed by the development of a strategy to provide technical assistance in maternal, newborn and child health, including immunization and post-partum family planning. The MCHIP/Zimbabwe project was launched in 2010 with Johns Hopkins Program for International Education in Gynecology and Obstetrics (JHPIEGO), John Snow, Inc. (JSI), Program for Appropriate Technology in Health (PATH), and Save the Children as principal implementing partners. The first phase was completed in September 2013 and the project has recently been extended to February 2014. The goal of MCHIP project in Zimbabwe is to contribute to accelerated and sustainable improvement in maternal, newborn, and child health (MNCH) through the scaling up of evidence-based, high impact, and integrated interventions. The project aims to contribute to reductions in maternal and child mortality and thus support Zimbabwe’s progress towards Millennium Development Goals (MDGs) four and five on reducing child mortality and improving maternal health. MCHIP works with the Ministry of Health and Child Care (MOHCC) and a number of strategic donor and NGO partners in Zimbabwe at the national level, and in selected districts of the Manicaland Province – which has the worst MNCH indicators nationally – implementing activities in the Provincial Hospital, district hospitals, rural health centers and communities. At the national level, the project assists the development of national policies, strategies and guidelines. EVALUATION PURPOSE, TEAM, METHODOLOGY The end of project evaluation of the USAID Zimbabwe MCHIP was carried out between October and November of 2013. The purpose of the evaluation was to: ascertain the extent to which the MCHIP intervention contributed to overall learning and innovation in MNCH care in Zimbabwe at national, district and health facility and community levels; provide detailed analysis of determinants of program performance; and provide critical information and document lessons learned for future USAID/key stakeholder project design and evaluation. The evaluation addresses the following key and sub questions: 1. How did MCHIP contribute to overall learning and innovation in MNCH care in Zimbabwe?  What innovative processes and products did MCHIP support or implement?  What factors may affect the feasibility of scaling up these innovations and how?  What was the nature of relations between MCHIP and key MNCH stakeholders and how did the relations contribute to achievement of results?  What challenges or barriers to achievement of results did MCHIP experience in Zimbabwe? 2. How effective is the Standards Based Management and Recognition (SBM-R) approach in improving MNCH care in MCHIP supported health facilities in Zimbabwe? 4  What proportion of MCHIP supported facilities is achieving a minimum set of MNCH care standards?  How are changes in standards of care influencing health outcomes e.g., in early newborn mortality, maternal mortality, obstetric and newborn complications?  How acceptable is the SBM-R approach to service providers, policy makers and other MNCH stakeholders in Zimbabwe?  Describe and analyze factors contributing to the effectiveness or ineffectiveness of the SBM￾R approach?  What factors may affect the feasibility of scaling up this approach and how can the SBM-R approach be adopted to increase potential for successful nationwide scale up? The team included two international professionals: a nurse midwife with broad global MNCH experience and a physician with extensive expertise in MNCH, health systems and health financing, including many years in Zimbabwe; a local monitoring and evaluation specialist and a local logistics and data entry professional. The evaluation methodology used a combination of quantitative and qualitative techniques, including: a focused document review; key informant interviews (KIIs) in Harare and Manicaland province; a health worker survey using a self-administered questionnaire and focus group discussions (FGDs) with formal health workers (HWs) and village health workers (VHWs) at health facilities (HFs) in the districts of Mutare and Chimanimani where MCHIP implemented key MNCH interventions. Due to the lack of valid control groups, a non-experimental performance evaluation design was utilized focusing on the key evaluation questions. The evaluators conducted a briefing and debriefing with the USAID mission and provided weekly written updates. Limitations included the possibility of recall bias; the subjectivity of self-reported data; and logistical and time constraints to conduct the field visits and collect, organize and analyze the sizeable amount of data. FINDINGS AND CONCLUSIONS 1) How did MCHIP contribute to overall learning and innovation in MNCH care in Zimbabwe? (i) What innovative processes and products did MCHIP support or implement? To innovate means to introduce something new or to change something. MCHIP promoted and implemented an integrated package of globally accepted MNCH interventions, many of which were not necessarily new, though the program’s innovative approach was found to have resulted in change, modernization and renewal and even a transformation in the way staff in targeted facilities operate. The approach used: a focus on quality of care using health service building blocks such as small but significant improvements in human resources, logistics and supplies and infrastructure simultaneously; “standard operating procedures” for life-saving clinical interventions such as Emergency Neonatal and Obstetric Care (EmNOC), Helping Babies Breathe (HBB), Kangaroo Mother Care (KMC) and Integrated Management of Newborn and Child Illness (IMNCI) that were more detailed than most guidelines and protocols that taught health workers step by step how to detect and manage complications; staff self￾assessment both at the individual and facility level, with supportive supervision provided by peers where supervisory visits became an activity to look forward to rather than a source of anxiety. The resulting improved morale and ownership of quality of care led to local homegrown solutions such as a Newborn Resuscitation Stand to save newborns with asphyxia or premature/low birth-weight (LBW), built by the community served by the Chakohwa Rural Health Center. 5 Competency based training, with regular post training follow up and provision of essential small supplies such as buckets for infection prevention, ensured health workers were able to apply their new skills. Concurrently, MCHIP worked at the policy level to facilitate an environment that enabled the adoptions of best practices locally and likelihood of acceptance at the national level. For example, KMC was pioneered in Zimbabwe in the 1980’s but had all but disappeared until MCHIP and its partners successfully reintroduced it. During FGDs HWs stated that previously, families with LBW and premature newborns were reluctant to be referred due to financial, geographic and transportation barriers, whereas KMC at the health centers was found to be highly acceptable to mothers and families. (ii) What factors may affect the feasibility of scaling up these innovations and how? The factors most commonly cited by key informants and HWs that may affect the feasibility of scaling up MNCH innovations include: buy-in from the government and MOHCC; scarcity of human resources, as well as acceptance of MNCH innovations among staff; resources (primarily financial) and availability of funding; poor infrastructure; supply of adequate equipment; coordination structures, reported to need improvement both within government and between partners; availability of adequate technical expertise and support in MNCH-especially at peripheral levels; community engagement and participation; stability of socio-economic circumstances in communities; the ability of Results Based Financing (RBF) activities to complement MCHIP’s efforts; alignment and harmonization of donor initiatives; availability of mentors; and adaptation of standards. (iii) What was the nature of relations between MCHIP and key MNCH stakeholders and how did the relations contribute to the achievement of results? The majority of key informants (18 of 21) including Ministry, donor or NGO partners reported feeling positive or very positive about their relationship with MCHIP, and 11 of 13 Key Informants (KIs) reported that good relations were a major factor in MCHIP’s success. The health workers in all focus groups were strongly positive about their relationship with MCHIP, and the VHWs were also universally positive. Strong relationships with most stakeholders allowed MCHIP to overcome substantial barriers, and MCHIP’s influence can be seen at the national, provincial, and district and community levels. MCHIP skillfully cultivated and utilized strategic partnerships and as a result global best practices such as KMC, HBB, EmONC, and IMNCI are now accepted and promoted by the MOHCC and MNCH donor partners. (iv) What challenges or barriers to achievement of results did MCHIP experience in Zimbabwe? Barriers and challenges cited by key informants during interviews and HWs and VHWs during the FGDs include: human resource shortages and a lack of skilled manpower such as midwives and nurses; competing demands for participation of HWs in different programs that target the same health professionals at the same time; low morale and poor motivation was reported as endemic; inadequate supervision resulted in a lack of accountability and inadequate infrastructure and resources provided a ready excuse for poor standards of care; coordination throughout MOHCC and absence of effective platforms for coordination among partners; resistance from some religious sects; inadequate supplies, drugs and materials such as antibiotics, zinc, disinfectants for infection prevention (IP), and gloves; and lack of diagnostic technologies such as sonograms or dopplers to assess fetal well-being during labor was reported as a barrier to diagnosing complications requiring more referrals. 2) How effective is the SBM-R approach in improving MNCH care in MCHIP supported health facilities in Zimbabwe? 6 (i)What proportion of MCHIP supported facilities is achieving a minimum set of MNCH care standards? MCHIP started working on maternal and newborn health (MNH) standards in 17 facilities in Nov 2010. After orientation, a baseline assessment was carried out, before implementation proper began. At baseline 14 of the 17 participating health facilities in the two learning districts scored “0” in meeting MNH-related performance standards; the highest score was 50 percent. As of September 2013, 12 of the 17 facilities have achieved over 80 percent of MNH performance standards, a significant improvement since no health facilities had reached this level at baseline The most frequently cited areas in which SBM-R was reported to have improved quality of care were related to antenatal care, delivery and essential newborn care, postnatal care, emergency obstetric/neonatal care, and MNH service management. The activities were deemed to be least effective in improving immunization services and IMNCI. However in interpreting this result the comments made earlier relating to the duration of the support should be borne in mind. It is worth noting that many respondents in FGDs commented on the value of the Oral Rehydration Therapy (ORT) corners in managing diarrheal illness, which was a stated focus of MCHIP. (ii) How are changes in standards of care influencing health outcomes e.g., in early newborn mortality, maternal mortality, obstetric and newborn complications? Based on available data, there is no strong evidence of changes in overall maternal and newborn health outcomes. However, it is clear from the responses in the HW survey that provider behavior has changed. In spite of the absence of hard data – due to inadequate monitoring and evaluation (M&E) and the short time span of the interventions – as well as a valid control group against which to compare health outcomes, qualitative findings indicate that SBM-R has contributed to a decrease in maternal, and especially, neonatal morbidity and mortality among clients of MCHIP-supported health facilities. HWs stated repeatedly that adverse outcomes have been avoided and lives that previously would have been lost were saved due to the new practices. A frequent comment in the FGDs was “now we don’t panic when complications or emergencies occur.” Anecdotal reports of improved survival of premature and low birth weight infants were frequent and the team found three case reports of infants weighing <1,000gm at birth surviving to at least 6 months, with only KMC. (iii) How acceptable is the SBM-R approach to service providers, policy makers and other MNCH stakeholders in Zimbabwe? Of the 12 key informants who offered an opinion, four said the SBM-R was very acceptable and eight said it was acceptable. In FGDs, HWs were positive about SBM-R and its impact on their capacity to provide good quality care. In spite of the length of the tool, which they found intimidating at first, HWs reported that it was comprehensive, told them exactly what to do and gave them greater confidence to undertake procedures; comments such as, “now we know what quality care means” were common. However, in FGDs, numerous HWs noted that the SBM-R scoring system, which requires a 100% compliance with criteria to achieve a standard, is demoralizing. (iv) Describe and analyze factors contributing to the effectiveness or ineffectiveness of the SBM-R approach? Limited human resources and lack of training materials for health providers at all levels will be significant barriers for future scale-up. HWs respondents agree that scale-up is necessary for consistency across facilities, but recognize that further assessment is necessary to determine the resources and modifications required to scale up SBM-R nationally. The SBM-R tool will require further modifications, including a reduction in length and change in scoring system to fully engage service providers nationwide. 7 RECOMMENDATIONS The key recommendations of the evaluation team are: Scaling-Up • Buy-in from the MOHCC has been identified as a critical issue for successful scale-up. It is recommended that MCHIP continue to play a leadership role in improving quality of care and continue policy dialogue with the MOHCC and key stakeholders to support the identification of a single national approach to quality improvement (QI). This should be followed by collaboration and support to develop an implementation strategy for QI. • MCHIP should continue to strengthen the harmonization and collaboration with funding partners – engaging and enmeshing the Health Transition Fund (HTF); the Integrated Support Program (ISP) on Sexual and Reproductive Health (SRH); MCHIP; and other European Union (EU), Development for International Development (DFID), and US Government (USG) projects and programs. This could go some way towards addressing the inadequate resources, infrastructure and funding identified as obstacles to scaling up. • The use of a single approach to quality improvement and greater harmonization and collaboration with partner programs should be pursued in order to alleviate the human resources (HR) shortage. Cross-program support at local levels should be explored as an option. • High-impact MNCH interventions and activities such as KMC, HBB, EmNOC, supportive supervision and self-assessment should be prioritized during scale-up • At the provincial level, scale-up activities should start with a review of mortality and morbidity (near misses) as well as maternal-neonatal mortality to identify districts and facilities that should be prioritized for the focus activities mentioned above. • Competency based training (CBT) as an approach to training should be advocated, emphasizing the key parts played by follow-up and on-the-job training and support. • In order to overcome the shortage of suitable on-site supporters, MCHIP should consider using champions (leaders) from currently supported facilities to introduce SBM-R into and mentor at new sites. • MCHIP should further strengthen communication and project visibility SBM-R • Rural health facilities function as a unit, with everyone from the Nurse-in-Charge to the gate guard and general hand playing a part in provision of high quality care; this is especially the case in smaller facilities where there may only be 1 or 2 clinical staff. It is recommended therefore that all staff are oriented in the SBM-R approach and given appropriate training and support. This applies especially to nurse aides who provide a significant percentage of the day to day care in health facilities • It is recommended that increased attention is given to referral hospitals in the roll-out process. • Recognition should be built in to earlier parts of the process and as a high priority the achievements of VHWs and HWs at HFs should be recognized in an appropriate manner. • It is recommended that the tool be reviewed to see if the number of criteria can be reduced without compromising the resulting quality of care. • The scoring system used during assessments came in for severe criticism. The main objection was to the “all-or-nothing” approach. It is recommended that a review of the criteria be undertaken, that steps critical to safety or quality of care be identified and scoring be restricted to the resulting critical pathways. Community Performance Quality Improvement (cPQI)  The intervention with VHWs has been running for less than a year and it is possibly too early to make recommendation. However, the sheer enthusiasm and commitment of the VHWs 8 interviewed, leads the team to recommend that the program be continued and further assessed in perhaps 6-12 months. It is further recommended that the same approach be adopted in other rollout districts. 9 EVALUATION PURPOSE & EVALUATION QUESTIONS EVALUATION PURPOSE The end of project evaluation of the USAID Zimbabwe Maternal and Child Health Integrated Program (MCHIP) was carried out between October and November of 2013. The evaluation was contracted to Social Impact, Inc. (SI), who conducted the evaluation in collaboration with Management Systems International (MSI). The four-person team comprised Pamela J. Putney, a nurse midwife with extensive experience in the design, implementation and evaluation of MNCH programs, Dr. Sean Drysdale, a physician with broad expertise in MNCH, health systems and health financing, Roy Mutandwa, a local Monitoring and Evaluation specialist with experience in evaluations of public health programs, and Charity Tinofirei, who provided logistics and administrative support. The purpose of the evaluation was to: • Ascertain the extent to which the MCHIP intervention contributed to overall learning and innovation in MNCH care in Zimbabwe at national, district and health facility and community levels. • Provide detailed analysis of determinants of program performance • Provide critical information and document lessons learned for future USAID/key stakeholder project design and evaluation The evaluation findings will be shared with implementing partner organizations, the host government and other relevant national stakeholders to maximize opportunities to leverage other donor resources to support scale up of proven maternal and child health interventions. This will make MCHIPs investments catalytic and complementary to those of other donors. Lessons learnt from the MCHIP supported districts will inform the Zimbabwe Ministry of Health and Child Care (MOHCC) decision making regarding MNCH quality of care (QoC) improvement efforts. It is anticipated that the evaluation will be useful to multiple audiences and stakeholders including: • MOHCC– national, provincial, and district health offices; • Other USAID supported projects and partners – Elizabeth Glaser Pediatric AIDS Foundation (EGPAF), Organization for Public Health Interventions and Development (OPHID), Population Services International (PSI), Population Services Zimbabwe (PSZ), John Snow International DELIVER (JSI Deliver); • NGOs and Community Based Organizations – Zimbabwe Association of Church Hospitals (ZACH), Save the Children, Plan International, Absolute Return for Kids (ARK), Cordaid, International Rescue Committee (IRC) and others; • Other technical partners – United Nations Children Fund (UNICEF), United Nations Population Fund (UNFPA) and World Health Organization (WHO); • Technical Working Groups – Maternal and Newborn Health Steering Committee, Child • Survival Task Force, Immunization Interagency Coordinating Committee, Prevention of Mother￾to-Child Transmission of HIV, Nutrition Task Force, Health Management Information Systems, and others. The report will be disseminated widely among relevant stakeholders and project beneficiaries as well as 10 submitted to the Development Exchange Clearing House (DEC). EVALUATION QUESTIONS The final evaluation of MCHIP addressed the following key and sub questions: 1. How did MCHIP contribute to overall learning and innovation in MNCH care in Zimbabwe?  What innovative processes and products did MCHIP support or implement?  What factors may affect the feasibility of scaling up these innovations and how?  What was the nature of relations between MCHIP and key MNCH stakeholders and how did the relations contribute to achievement of results?  What challenges or barriers to achievement of results did MCHIP experience in Zimbabwe? 2. How effective is the SBM-R approach in improving MNCH care in MCHIP supported health facilities in Zimbabwe?  What proportion of MCHIP supported facilities is achieving a minimum set of MNCH care standards?  How are changes in standards of care influencing health outcomes e.g., in early newborn mortality, maternal mortality, obstetric and newborn complications?  How acceptable is the SBM-R approach to service providers, policy makers and other MNCH stakeholders in Zimbabwe?  Describe and analyze factors contributing to the effectiveness or ineffectiveness of the SBM￾R approach?  What factors may affect the feasibility of scaling up this approach and how can the SBM-R approach be adopted to increase potential for successful nationwide scale up? 11 PROJECT BACKGROUND CONTEXT In 2008, Zimbabwe’s socioeconomic situation had deteriorated to the point that health services had all but ceased to function: drugs and medicines were not available; equipment was not serviceable; infrastructure had deteriorated to a critical level; and a large percentage of health workers had either emigrated to other countries to work or left the health sector for better paid employment. Health workers who remained in the system faced almost insurmountable challenges to provide adequate care. Financing for health, including donor funding was limited and MNCH partners were working primarily in Prevention of Mother to Child Transmission (PMTCT). In-service training in MNCH was almost non￾existent, and the training that was taking place was fragmented and not standardized. As health indicators declined to critical levels, in 2008, USAID identified maternal and child health as a critical area of need in Zimbabwe and, in 2009, requested MCHIP/DC to undertake a rapid assessment, followed by the development of a strategy and initial work-plan to provide technical assistance in maternal, newborn and child health, including immunization and post-partum family planning. The approach was to address core challenges in the sector including the human resource crisis and the poor state of health infrastructure. The MCHIP situation analysis reported high levels of maternal and under-5 child mortality that had been rising since 1994. Table 1. Selected MNCH Indicators as Reported in MCHIP Situation Analysis Indicator Rate Source Maternal Mortality Rate (Deaths /100,000 live births) 725 Zimbabwe MPNMS 2007 Children Under 5 year Mortality Rate (Deaths /1,000 live births) 94 MIMS 2009 Infant Mortality Rate (Deaths/1,000 live births) 67 MIMS 2009 Perinatal Mortality Rate (Deaths /1,000 live births) 30 MIMS 2009 The analysis noted the main causes of maternal and neonatal mortality as reported by Child Health Epidemiology Reference Group (CHERG) in 2008 and the timing of neonatal deaths as reported in the Zimbabwe Maternal and Perinatal Mortality Survey conducted in 20071 as: Maternal Mortality: HIV/AIDS (36%); Postpartum Hemorrhage (19%); Pre-eclampsia/eclampsia (18%); Sepsis (11%); Malaria (8%) and Abortion Complications (8%). Neonatal Mortality: Prematurity (33%); Asphyxia (24%); Infection (23%); Congenital Malformations (9%); Tetanus (2%); Diarrhea (2%) and other (7%). The majority of newborns were dying on the first day of life as illustrated in the chart below. 1 Zimbabwe Maternal and Perinatal Mortality Survey 2007. MOHCW 2009. 12 Figure 1. Timing of Neonatal Death in 506 Infants Identified During Zimbabwe PMNMS 2007 The authors of the situational analysis used the Lives Saved Tool (LiST) program to identify the potential to save lives using a combination of interventions. They recommended a package at District Hospital (DH) and high-volume Rural Health Centers (RHCs). The proposed goal of the program was “to contribute to improved maternal, newborn and child health outcomes by helping to rebuild the capacity of Zimbabwe’s public health system to deliver high-impact MNCH/FP interventions.” The proposed approach would support the MOHCC to implement policies and strategies known to have a direct positive impact on maternal and neonatal mortality, and work closely with MOHCC and other partners to implement the selected package of interventions including: Focused Antenatal Care (FANC) with PMTCT; Basic Emergency Obstetric and Newborn Care (BEmONC); Immediate Postnatal Care; KMC; and Post￾partum Family Planning. The National Integrated Health Facility Assessment (NIHFA) was carried out with support from MCHIP between January-December 2010. A total of 1,375 health facilities were surveyed, including 125 hospitals, 6 Central Hospitals; 8 Provincial Hospitals; 50 District Hospitals; 46 Mission Hospitals; and 15 others, some private institutions. The survey reported: Table 2. Summary of NIHFA Report for Maternal and Neonatal Deaths Maternal Deaths Neonatal Deaths  A total of 905 maternal deaths were reported  68% of deaths occurred at the hospital level  28% occurred in the community  3.8% occurred at Level 1 facilities (majority Rural Health Center)  The majority of maternal deaths (48%) reported by hospitals occurred at the central hospital level  4,380 neonatal deaths were reported in 2010  Hospitals reported the highest number of neonatal deaths (92%)  8% of neonatal deaths occurred at a Level 1 facility 0.0 10.0 20.0 30.0 40.0 50.0 60.0 ≤ 1 Day ≤ 2 Days ≤ 3 Days ≤ 4 Days ≤ 5 Days ≤ 6 Days ≤ 7 Days 8-28 Days % Timing of Neonatal Death 13 Two points are striking: the reported number of maternal deaths suggests a maternal mortality ratio (MMR) below 300, which is considerably less than the accepted figure; and although almost 20% of deliveries took place at home, no community neonatal deaths were reported. PROJECT OVERVIEW The MCHIP/Zimbabwe project was launched in 2010 and the first implementation phase was completed in September 2013; the project has recently been extended to February 2014. MCHIP is a centralized Leader with Associates (LWA) cooperative agreement and it is the USAID global flagship maternal and child health program implemented by a consortium of US-based NGOs including JHPIEGO, JSI, Save the Children, PATH, Population Services International (PSI) and others. In Zimbabwe, JSI is the lead organization, with JHPIEGO, PATH, and Save the Children. JHPIEGO supports maternal and newborn activities; Save the Children supports newborn health and MNCH community activities; PATH supports nutrition activities; and JSI supports immunization and child health activities, including pediatric HIV prevention, care and treatment. The goal of MCHIP project in Zimbabwe is to contribute to accelerated and sustainable improvement in MNCH through the scaling up of evidence-based, high impact, and integrated interventions. The project aims to contribute to reductions in maternal and child mortality and thus support Zimbabwe’s progress towards MDGs four and five on reducing child mortality and improving maternal health. Table 3. MCHIP Works at the National, Provincial, District and Community Levels National Provincial District Community Supporting development & review of policy, strategy guidelines, curricula, MNCH studies & assessments Scaling up routine immunization & new vaccine introduction Scaling up high impact MNCH interventions Implementing SBM-R at health facility level Supporting community MNCH MCHIP works in Zimbabwe at the national level, and in selected districts of the Manicaland Province – which has the worst MNCH indicators nationally – implementing activities in the Provincial Hospital, district hospitals, rural health centers and communities. At the national level, the project assists the development of national policies, strategies and guidelines. At the Provincial, district and health facility level, the project provides direct technical support to implement MNCH interventions. These activities are mainly implemented in two districts of Manicaland province, Mutare and Chimanimani, although other project activities extend to all the seven districts of the province. In the two operating districts – see figure 2 – the project has a critical focus on improving the quality of maternal, newborn and child health services provided by the 2 district hospitals and 71 Rural Health Centers (RHC) through training, supervision and mentorship of health workers. However, the project is also working, in the other districts in the Province, to increase routine immunization coverage, using the Reach Every District approach (RED) – see figure 2. Within the two focal districts, MCHIP is supporting the two district hospitals and 20 health facilities (12 in Mutare and 8 in Chimanimani) to use new, self-administered SBM-R tools to identify and address important gaps in service delivery. The project provides support to the national quality assurance unit within the MOHCW to adapt the SBM-R tools so the approach can become institutionalized on a wider scale. 14 At community level, MCHIP is supporting improvements in the coverage and quality of high-impact MNCH interventions provided by VHWs. This is done through provision of both formal and on-the-job training and supervision in the integrated management of maternal, newborn and childhood illnesses. The project is developing integrated training materials and supervisory tools to carry out close supervision by the MCHIP district coordinators. Regular update and joint review meetings for health workers are held to encourage greater peer-to-peer support and utilization of data for decision-making. An important aspect of this work is support to VHWs and promotion of linkages between the community and the formal public health system. Figure 2. Where MCHIP Works2 At the time of the evaluation, MCHIP has implemented SBM-R in the health facilities listed in the box below. Table 14. MCHIP SBM-R Facility Sites for MNCH MUTARE District CHIMANIMANI District Mutare Provincial Hospital Sakubva District Hospital Marange Rural Hospital St. Andrews Mission Hospital Gutaurare Rural Health Centre Zimunya Rural Health Centre Odzi Rural Hospital Bezeley Bridge Rural Health Centre Sakubva Health Centre-Polyclinic Dangamvura Polyclinic Mutambara Mission Hospital Rusitu Mission Hospital Nyanyadzi Rural Hospital Biriri Rural Hospital Chimanimani Hospital Chakohwa Rural Health Centre Mutsvangwa Rural Health Centre Additional Health Facilities for Child Health Additional Health Facilities for Child Health Mt. Zuma Rural Health Centre Chewere Rural Health Centre Burma Valley Rural Health Centre Nyahode Rural Health Centre Chayamiti Rural Health Centre 2 MCHIP works in all the districts in Manicaland to improve immunization coverage. 15 MCHIP PROJECT VISION, GOAL AND OBJECTIVES MCHIP/Zimbabwe’s vision is to significantly contribute to accelerated and sustainable improvement in maternal, newborn, and child health in Zimbabwe through scaling up evidence-based, high‐impact, integrated public health interventions. MCHIP/Zimbabwe’s goal is to assist in scaling up evidence‐based, high impact maternal, newborn, and child health interventions and thereby to contribute to significant reductions in maternal and child mortality and progress towards MDGs 4 and 5. MCHIP’s objectives are: 1. Support the MOHCC to formulate national health policies, strategies and programs that increase the population’s access to affordable, evidence‐based, high impact maternal, newborn, and child health (MNCH) interventions; 2. Improve the quality of maternal and newborn health services provided by District Hospitals and high volume RHCs; 3. Improve the coverage and quality of high‐impact MNCH interventions provided by PCNs in RHCs and by VHWs in communities; and 4. Increase routine immunization coverage, focusing on those districts with large numbers of unimmunized children, and successfully obtain and introduce pneumococcal vaccine by 2013. MCHIP works with a number of partners at the national, provincial and district levels including:  The MOHCC at all levels  USAID supported projects and partners such as EGPAF, OPHID, World Education, Inc. (WEI), PSI, PSZ, JSI/DELIVER and others  NGOs and Community-based organizations (CBOs) such as ZACH, Save the Children, Plan International, ARK, Cordaid, IRC and others  Multilateral partners such as UNICEF, WHO, UNFPA  Technical Working Groups such as Reproductive Health, Health Management Information System, Quality Assurance/Quality Improvement, Nutrition Task Force, and others Gender Gender issues contribute directly and indirectly to health outcomes. MCHIP is committed to gender neutral (do no harm) or gender transformative outcomes (change outcomes). In its activities MCHIP seeks to: address barriers to the use of MNCH/FP/RH services and behavior change such as gender biases and inequalities; and facilitate constructive male involvement; incorporate messages into training/health promotion activities that promote gender sensitivity and gender equality; and facilitate “mainstreaming” of gender themes into service provision activities. Technical Areas MCHIP’s key technical areas and goals for each are outlined in the chart below: Table 4. Key Technical Areas Maternal health, postpartum FP, Prevention of mother to child transmission of HIV (PMTCT) To reduce morbidity and mortality associated with pregnancy, labor and delivery and the postpartum period, as well as the transmission of HIV from mothers to their children through implementation Newborn health To reduce illness and death associated with newborn asphyxia, prematurity and LBW, and infection through the support of evidence-based interventions, such as HBB and KMC 16 of a quality improvement approach in hospitals and health centers Child health To reduce morbidity and mortality associated with the most common causes of childhood illness such as pneumonia and diarrhea, through support of the IMNCI (Integrated Management of Newborn and Childhood Illness) approach among other activities Immunization To reduce illness and death in children associated with vaccine-preventable diseases such as measles and polio, through support to improve activities to improve routine immunization coverage in Manicaland as well as national introduction of new vaccines Malaria To reduce illness and death in pregnant women and children caused by malaria, through support to the MOHCW’s national malaria control program Nutrition To help reduce stunting and underweight in children, through support for the MOHCWs infant and young feeding initiatives Monitoring & Evaluation/HMIS To facilitate the availability of high quality, complete, and timely health data for use in planning, monitoring, and evaluation of health services Quality of care improvement To help increase stakeholder awareness of quality of care issues and to support MOHCW development of national quality of care policies and standards 17 EVALUATION DESIGN, METHODS & LIMITATIONS The evaluation used a combination of quantitative and qualitative techniques, including a focused document review, a health worker survey using a self-administered questionnaire, key informant interviews, and focus group discussions with formal health workers and village health workers. Due to the lack of valid control groups, a non-experimental performance evaluation design was utilized focusing on the key evaluation questions. The evaluation team purposively selected sites to visit from a list of project sites provided by MCHIP; selection was made in consultation with USAID, MCHIP and MOHCC staff. Site visits were determined according to a set of established criteria, such as health facility performance ratings (gathered from project documents), size of the facility and the length of time key project MNCH interventions had been implemented in each facility, in order to include data from a range of MCHIP-supported health facilities. Logistical considerations were also included in the criteria for selection given the length of travel time and poor road conditions requiring a four-wheel vehicle. Key informants were similarly identified through discussions with USAID and MCHIP. Official permission to carry out the evaluation and collect data was obtained prior to the start of fieldwork from the MOHCC. This was passed down to lower levels of the system in writing and local clearance was subsequently obtained for the team to operate. All respondents provided formal consent to being interviewed, having being informed that participation in the FGDs, KIIs and HW survey was completely voluntary and that individuals were free not to answer any of the questions if they chose not to. Figure 3. Data Triangulation Summary Key Informant Interviews 17 in Harare and 7 in Manicaland Focus Group Discussions 80 VHW in 8 groups 37 HCWs in 7 groups Health Worker Survey 72 HCWs completed a self￾administered questionnaire with 47 closed and open-ended questions Data Triangulation 18 FOCUSED DOCUMENT REVIEW Table 5. Focused Document Review • The MCHIP Program description • MCHIP annual work plans • MCHIP annual and quarterly progress reports • The MCHIP Performance Management Plan • SBM-R baseline and subsequent assessments • MCHIP technical reports including, the Village Health worker baseline report, National Integrated Health Facility Assessment (NIHFA) report, Infant and Young Child Feeding Assessment report, and the Head Count of Children under 5 Years Report and the Situational Analysis • Report of 2010 • Facility level data reported through the National Health Management Systems (HMIS) A formal document review was conducted prior to the start of fieldwork, and a desk review outcome matrix3 was created to include the key findings gleaned from project documents and areas identified for further inquiry. Additional documents gathered over the course of the evaluation were reviewed on an on￾going basis. 4 QUALITATIVE DATA Table 6. Qualitative Data Key Informant Interviews Focus Group Discussions • Semi Structured • Explore attitudes on stakeholder relationships • Challenges and barriers to achievement of results • Wide range of managers implementers and other stakeholders in Harare and Manicaland • Triangulated with quantitative data to assess project performance • 8 MCHIP-supported facilities • Experiences with project interventions • Acceptability and feasibility of SBM-R • Managers and service providers • Village Health Workers KEY INFORMANT INTERVIEWS A list of key informants was compiled in discussions with USAID and MCHIP and 24 were subsequently interviewed, 17 in Harare and 7 in Manicaland. KIIs were semi-structured and included a wide range of managers, implementers, and other stakeholders in Harare and Manicaland Province. The key Informant Interviews provided qualitative information on how MCHIP has contributed to improvements in MNCH care at the national, provincial, and district levels. KIIs were useful in exploring attitudes on stakeholder relations and challenges to MCHIP results achievement. Key informants interviewed included:5  USAID and MCHIP staff;  MOHCC national, provincial, and district level staff;  Other USAID supported projects and donor partners such as DFID and EU; and 3 See Annex IV.E for Desk Review Matrix 4 See Annex IV.D for List of Documents Reviewed 5 See Annex IV.A for list of KIIs conducted 19  NGOs and CBOs such as Save the Children, Plan International, ARK, Cordaid, and other technical partners such as UNICEF, UNFPA and WHO FOCUS GROUP DISCUSSIONS Two sets of FGDs were conducted: one with formal health workers and the other with trained Village Health Workers, who are unpaid volunteers receiving only a monthly stipend from the MOHCC. Two local experienced data note takers and at least one facilitator conducted all the focus groups and in addition, a team member with extensive technical knowledge in MNCH was present during the HW FGDs to ensure key responses to priority questions were consistently elicited. Formal Health Workers The evaluation team conducted 7 focus group discussions (FGDs) with 37 health care workers at MCHIP￾supported health facilities to gain an in-depth understanding of their experiences with project interventions. The FGDs were useful in supplementing KIIs and quantitative data by investigating the acceptability of the SBM-R approach among service providers, as well as the factors that may affect the feasibility of scaling up the SBM-R approach. The FGDs also helped to elucidate the effectiveness of training activities supported by the project and how these activities have led to changes in provider behaviors and improvements in MNCH care and outcomes. Additionally, they provided an insight into the degree to which the project interventions had been adopted, internalized and revealed participants very strong opinions on their effectiveness. Table 7. Break down of HW Participants by Cadre (33 Female and 4 Male) Registered Graduate Nurse Midwife Registered Graduate Nurse (RGN) Sister-in￾Charge State Certified Mental Nurse Primary Care Nurse (PCN) General Hand Upgraded Primary Care Nurse Nurse Midwife Pharm Asst. (PA) 2 9 1 2 8 1 1 4 1 Village Health Workers Eight FGDs were conducted with 80 trained VHWs who work in communities surrounding the facilities visited by the team. Findings from these FGDs provided insight into the effectiveness of trainings and how newly acquired skills in integrated management of maternal newborn and childhood illnesses have been utilized in the community. Table 8. Distribution of VHW Focus Group Discussion Participants by Location and Gender Facility Name Male % Female % Total VHWs % Burma Valley Rural Hospital 1 2 3 Mutambara Mission Hospital 2 12 14 Chakohwa Clinic 0 6 6 Bezeley Bridge 0 2 2 Biriiri Rural Hospital 0 15 15 Odzi Rural Health Centre 4 6 10 Rusitu Mission Hospital 1 14 15 St Andrews 3 12 15 20 Total Number of VHW FGD 11 14% 69 86% 80 100 QUANTITATIVE DATA The evaluation team collected and analyzed primary and secondary quantitative data through the use of a survey administered among health workers, as well as project documents. Table 9. Quantitative Data The health worker questionnaire: • Was self-administered • Contained 47 closed and open-ended questions • Sought to gain additional data on the acceptability and effectiveness of SBM-R • Had 72 respondents from various health facilities in Manicaland Province Secondary sources include: • MCHIP quarterly and annual reports • Zimbabwe DHS reports • HMIS reports • National Integrated Health Facility Assessment Report • SBM-R Assessment reports HEALTH WORKER SURVEY In order to gain quantitative data in addition to secondary data from project documents, the evaluation team developed a self-administered questionnaire to gauge the acceptability and effectiveness of the SBM-R approach among health workers. The survey was developed in close consultation with USAID and comprised 47 closed and open-ended questions. Respondents were selected from the same sites that were visited for FGDs. Where possible, health workers other than those participating in the FGDs were asked to complete the questionnaire, but where this was not feasible – due to small numbers – the same health workers participated in both FGDs and questionnaires. In addition to the HWs, managers at district and provincial level also completed the questionnaire. Due to the absence of a complete sampling frame and logistical constraints, random selection from the universe of HWs was not possible. Instead, convenience sampling was conducted from among HWs available for interview on the day and time of the team’s visit. Of the 72 surveys completed by health workers and managers in Mutare and Chimanimani, 95 percent of respondents were female and the average age was 56 years. Approximately 32 percent of respondents reported being in charge of a facility or a department. Respondents reported an average of ten years working at the facility level and an average of eight years spent at the post. The following table shows a breakdown of respondents according to staff cadre: Table 10. Survey Respondent Roles Staff Category Freq. % Nurse/Midwife 43 60.56 Registered Graduate Nurse 12 16.90 Primary Care Nurse 12 16.90 EHO/T 1 1.41 Pharmacist /Pharm Tech /Asst. 0 0.00 Laboratory Tech/Assistant 0 0.00 Health Administrator 2 2.82 QUALITY ASSURANCE AND LIMITATIONS OF DATA COLLECTION 21 To ensure the collection of the highest quality data feasible, the team collaborated with USAID and MCHIP project staff as well as global monitoring and evaluation experts at SI in the development of the tools and selection of sites. Data collected was entered, organized and cross checked as it was collected by the team and sent to SI for feedback throughout the evaluation. Regular updates and collaboration through a weekly memo, phone conversations and e-mails were carried out with USAID. MCHIP project staff in Harare and Manicaland provided data and logistic support as needed and requested by the team. Limitations in the data collection included: the possibility of recall bias among key informants and focus groups; the subjectivity of self-reported data; the lack of availability of several of the key informants proposed were not available - particularly the MOHCC; the length of the HW Survey tool which required approximately one hour for completion time and was comprised of 47 questions, 31 of which had an open￾ended component which was challenging for HWs to fill out when they were often taking time from attending to a long line of patients waiting to be seen; constraints to carry out field work and enter, organize data; logistical challenges with distances between sites; the lack of an adequate space to conduct FGDs for both VHWs and HWs at many health facilities (in all cases, FGDs could not be recorded due to excessive background noise); and long lines of patients waiting for care at the health facilities requiring flexibility and additional time for the team to collect data in the field. With regard to the open ended responses for the HW survey, the evaluation team noted a degree of “collusion” within facilities, as in some cases, several consecutive responses were identical, such that its subsequent analysis represents more of a collective response rather than one of a collection of individual responses. Furthermore, every respondent did not answer every question, while others would have been counted twice, or more, if they mentioned two or more factors; the reported number of mentions for any factor must therefore be considered simply as a frequency. 22 FINDINGS, CONCLUSIONS & RECOMMENDATIONS FINDINGS AND CONCLUSIONS 1. How did MCHIP contribute to overall learning and innovation in MNCH care in Zimbabwe? (ii) What innovative processes and products did MCHIP support or implement? FINDINGS The evaluation team notes two points to be considered in addressing this question. Firstly, the term “innovative” does not simply imply “new”; rather, to innovate means to introduce something new or to modernize, transform, update, renovate, renew or remodel. Secondly, MCHIP was tasked with assisting in scaling up evidence-based, high-impact interventions; by definition these interventions cannot be new in the broader sense, although they could conceivably be new to Zimbabwe. MCHIP promoted and implemented an integrated package of globally accepted MNCH interventions. While many of the interventions were not new, the evaluation demonstrates that the program has resulted in certain change, modernization and renewal and even, it could be suggested, a transformation in the way staff in targeted facilities operate. The most notable innovations found by the evaluation team were: 1. Rather than focus on delivery of a single service and simply count numbers of clients who received the service, MCHIP focused on quality of care, which required paying attention to a number of health service building blocks simultaneously: human resources, supplies, infrastructure. This was in itself a change from the usual approach. 2. The introduction of, what are in effect, standard operating procedures (SOPs) for clinical interventions or procedures; the criteria used to assess each standard are essentially single steps in a production process – the product being high-quality care. These SOPs differed from existing guidelines or protocols, largely in their detail. Staff expressed appreciation and support of this and certainly saw it as something new: “We now know what to do – what is expected of us”; “no-one told us what to do before”. 3. The largely successful attempt to get staff to undertake self-assessment, both at the individual and facility level. The combination of self-assessment and supportive supervision provided by peers, “traditional” line managers, and MCHIP staff proved to be successful and innovative. Supervisory visits became welcome events whereas, prior to MCHIP’s intervention, supervision was a source of anxiety. “Before when the supervisors were coming you would be really shaking but now the supervision is friendly and the supervision has improved greatly.”6 Another aspect, the aforementioned transformation, was that the improved knowledge of staff and their greater confidence, when combined with the direct and practical support of MCHIP, raised morale and improved motivation. Even by themselves these two factors would have resulted in better quality of care; for example, VHWs and HWs reported in the FGDs that community members also noticed results such as increased awareness of danger signs, malaria detection and treatment and respectful care at birth (resulting in an increase in the number of facility births). A further indication of the improved morale and the ownership of quality instilled by MCHIP’s intervention was the manner in which staff developed locally supported home-grown solutions for their problems. An example is the Newborn Resuscitation Stand shown in the photo below, built by the community served by the Chakohwa Rural Health Center, to save newborns with asphyxia, or who are premature/LBW (low birth weight). 6 Nurse at Dangamvura Polyclinic 23 MCHIP’s competency based training approach (see below) coupled with the post-training follow-up and supervision was a more modern approach to adult education and training, and was certainly new to most health staff. The approach, particularly the follow-up, was undoubtedly critical in the success noted above. Some things were innovative in a more usual sense: introduction of new vaccines; SBM-R tools; Community Malaria Case Management (cMCM) – although it is accepted that this was a MOHCC initiative. Another aspect found innovative was the approach of working concurrently at policy and operational level. MCHIP sought to demonstrate clearly the operational effectiveness of their intervention, while at the same time working at policy and strategic level to facilitate an environment that enabled the adoption of their proposals locally and enhanced the likelihood of their consideration as a national approach. They increased the capacity of staff to provide services that facilitated the engagement of local communities and harnessed their support. From the stakeholders’ perspective, KII respondents noted 27 different processes or products they considered innovative. The most frequently cited were: • Introducing a Quality of care approach • Strengthening Emergency Obstetric Care (EmNOC) • Introducing a new curriculum called Helping Babies Breathe (HBB) • Revitalizing the Kangaroo Mother Care (KMC) approach • Introducing Competency Based Training (CBT) • Strengthening Integrated Management of Newborn and Child Illness (IMNCI) Some specific examples of MCHIP’s innovative work the broader sense follow. Competency Based Training (CBT) Respondents reported that prior to MCHIP, most training in Zimbabwe was perceived as workshop-based with little “hands on” or practical experience or post training follow up. MCHIP introduced CBT for important lifesaving interventions, such as EmNOC and HBB, which HWs frequently cited during the FGDs as having given them new confidence to manage obstetric and newborn emergencies and complications. A frequent comment by many respondents was, “Now we don’t panic; we just manage as we were taught”. The CBT approach used by MCHIP was participatory, practical and directly relevant to the development of skills and core competencies in MNCH in Zimbabwe that HWs need to perform in order to save lives and improve outcomes. HWs practiced on models during training sessions (e.g. HBB training sessions included extensive practice on life-like models of newborns) and skills development was a priority rather than theoretical knowledge. Skills acquisition was enhanced by not just anatomic models, but also by the use of virtual patients and simulation tools. An additional novel feature, for Zimbabwe, was MCHIP’s post-training follow-up. Theirs was not a fire-and￾forget approach; rather MCHIP made strenuous efforts to provide on-going support to HWs, in the form of further on-the-job training and support and essential equipment and supplies, to ensure staff were able to apply their new skills. Some staff comments relating to this include: “MCHIP trained us, supervised us and supported us”; and “they demonstrate procedures and skills and also provide essential small items such as scales, buckets, teaspoons and cups for cleaning and giving medications.” Nurses frequently mentioned during FGDs their consistent use of the partograph since receiving training, which they reported helped them to easily identify complications. This is in itself an achievement and an indicator of the success of the approach since encouraging HCWs to use the partograph is notoriously difficult. 24 Figure 4. “The follow up after training was novel and very important. Previously people went to workshops and then just went back to work as usual. MCHIP gave hands on assistance.” - Senior Nursing Staff Mutare Provincial Hospital Between October 2012 and September 2013 MCHIP trained a total of 263 VHWs: 162 in cHMIS (community Health Management Information System); 64 in cMNCH (community MNCH) refresher; 140 in Malaria testing using RDT; and 20 in Supportive Supervision. Kangaroo Mother Care (KMC) KMC, a low cost intervention known to improve the survival of low birth-weight and premature babies, was pioneered in Zimbabwe in the 1980s. However, its use had all but disappeared until its recent revival when MCHIP collaborated with other partners, including NGOs such as Absolute Return for Kids (ARK). This renewal of a practice that had fallen into disuse constitutes innovation in the wide sense. Figure 5. “KMC had been well forgotten but MCHIP reintroduced it.” - Deputy DNO Mutare Province “The renovations and provision of equipment for the KMC unit was a first.” - Senior Nursing Staff Mutare Provincial Hospital To support the revitalization of KMC at the national level, a training package for KMC has been finalized with MCHIP support. In Manicaland, eight KMC sites (four in Mutare and four in Chimanimani) have been established by MCHIP (outlined below). Table 12. Kangaroo Mother Care Unit Sites Mutare Chimanimani Mutare Provincial Hospital Sakubva District Hospital Marange Rural Hospital St. Andrews Mission Hospital Mutambara Mission Hospital Rusitu Mission Hospital Biriri Rural Hospital Chimanimani Hospital *Nyanyadzi Rural Hospital does not have a KMC Unit but practices KMC with MCHIP support The chart below shows the percentage of LBW (low birth weight) babies admitted into KMC in Mutare and Chimanimani between October 2011 and September 2013. The indicator increased from 15 to 28 percent in one year mainly due to the increase in the number of KMC facilities. The increase illustrates the recognition by health staff of the value of the intervention. Data from KMC registers show the units are now operating at full capacity, but the low percentage of LBW babies admitted to KMC units also highlights the distance still to be travelled to achieve full coverage. 25 Figure 6. LBW Baby Admittance KMC was frequently cited as having a significant impact on the survival rates of premature and low birth weight infants during KIIs, the FGDs and HW survey. There are three case reports of infants weighing <1,000gm at birth surviving to at least six months, where KMC was provided at remote rural health facilities with no referral to the larger district or provincial hospital. HWs stated that previously, families with LBW and premature infants were reluctant to be referred to secondary or tertiary institutions due to financial, geographic and transportation barriers, whereas KMC at the health centers was found to be highly acceptable to mothers and their families. Additionally, as one VHW said, “KMC has improved the attachment between preterm babies and mothers.” In spite of this anecdotal data, there is no hard evidence available to support this transformational change in behavior and potentially outcomes; this is clearly an area where data collection and monitoring could be improved, perhaps by focusing on intermediate outcomes rather than impact. Figure 7. “We saved a 700gm baby with KMC and now it is 6 months old and thriving. The mother was just in with the baby yesterday.” “We used to refer babies born who were less than 2 Kgs to Mutambara and that was a problem because the mothers didn’t want to go but now we don’t have to refer them because we have KMC. - Biriri Rural Hospital “A baby whose mother’s name is Musapatika, which translates loosely as ‘Don’t Panic!’ was born in June weighing 600gm. On 25th October the baby was doing well and weighed 3,100gms!” “Mothers appreciate the benefits of the unit as well as the staff so the community also can see the progress.” - Mutare Provincial Hospital Community Malaria Case Management (cMCM) Although the introduction of cMCM was strictly speaking a MOHCC initiative, MCHIP successfully leveraged the available funds to improve MNCH care and outcomes through the VHW training and quality improvement activities. MCHIP utilized the opportunity to train VHWs, not only in management of malaria, but also in those areas noted below. A further difference in the MCHIP approach was the post-training follow-up. This was novel since previously most training was conducted on a single-issue, workshop-based basis (e.g. PMTCT). Table 11. Trainings Received by VHWs as noted in FGDs (N=80) Training Total % Antenatal Care (ANC) 76 95 Postnatal Care (PNC) 79 98.75 Diagnosis & Treatment of Malaria 78 97.5 4% 8% 24% 26% 21% 24% 17% 41% 0% 10% 20% 30% 40% 50% 60% FY12 Q1 (Oct-Dec 2011) FY12 Q2 (Jan-Mar 2012) FY12 Q3 (Apr -Jun 2012) FY12 Q4 (Jul-Sep 2012) FY13 Q1 (Oct-Dec 2012) FY13 Q2 (Jan -Mar 2013) FY13 Q3 (Apr -Jun 2013) FY13 Q4 (Jul-Sep 2013) % of LBWs admitted in KMC Percentage of LBW newborns admitted in KMC % of LBW newborns admited in KMC Annual target (FY 12 and FY 13) 26 Training Total % Treatment of Diarrhoea 70 87.5 Management of Child with Breathing Difficulties 54 67.5 Home Based Care for Colds, Aches and Pains 72 90 Peer Supervision 48 60 Breastfeeding 76 95 Immunization 79 98.75 IYCF (Infant and Young Child Feeding) 79 98.75 In addition to the above, VHWs repeatedly mentioned they received additional trainings and support from MCHIP in: Exclusive Breast Feeding; IMNCI; KMC; weighing babies; they mentioned the provision of supplies such as soap, petroleum jelly, cotton wool, umbrellas and raincoats, scales for weighing babies, and ANC registers (as a part of the Community HMIS); and peer supervision and supervision from MCHIP. The introduction of cMCM and improvement in VHW capacity in other areas, allowing more treatment to be provided close to patients’ homes, is not only a more efficient use of resources but also increases the effectiveness of VHWs. Figure 8. “We are less burdened with malaria now due to the VHWs.” - Biriri Rural Hospital IMNCI (Integrated Management of Newborn and Child Illness) Key informants and HWs cited IMNCI as having made a significant difference in integrating care, identifying complications and decreasing the need for return visits and referrals. The staff at the Chakohwa Rural Health Center stated, “IMNCI had a big impact. We follow mothers and babies holistically now. Through IMNCI we learned a lot about the complaints of mothers and the child - we now join mothers and children together.” Although INMCI is a standardized and widely used approach, the capacity for follow-up within MOHCC is typically limited and therefore not systematized. By integrating IMNCI with other training and care giving (i.e. ANC, delivery, PNC, and a continuum-of-care approach), MCHIP created a systematic method to ensure follow up and supervised practice of IMNCI. This integrated approach also helped ensure that community workers, following the new community IMNCI approach, were following standard operating procedures known to and consistent with those used by formal HCWs. Integration of Care “One Stop Shop” Thus MCHIP was considered to have promoted the oft-referenced “one-stop-shop” approach to care that integrated mothers, babies, children and family members, resulting, anecdotally from HW FGDs, in fewer repeat visits and reduced workload for the staff, particularly in the case of malaria testing and treatment. Again, MCHIP’s innovation was to make the concept a reality and to enable patients to get all the care they required for all their ailments at one go; previously the one-stop-shop had addressed only one area of care (e.g. PMTCT). Figure 9. “With MCHIP we are using a holistic approach and assessing mother and baby together at each visit and it is reducing the workload.” “We care for the needs of the parent as well as the needs of the child at each visit.” St. Andrews - Mission Hospital Nurses CONCLUSIONS MCHIPs approach is found to be innovative in the wider sense of the term. Many of the individual interventions were not new, but the integrated manner in which they approached the situation was found to have transformed the delivery of not just MNCH services, but all other services in the targeted facilities. There is a widespread perception within the service that MCHIP’s support has led to the adoption of new practices and the revival of old ones and contributed to major improvement in the quality of care in a 27 number of key MNCH areas (See below). MCHIP utilized an innovative approach, which combined high-level advocacy and policy support with high￾quality, competency-based training in essential high-impact MNCH technical areas. Critically, this was followed by supportive supervision and on the job training. Nationally adopted policies and protocols developed in collaboration with MCHIP, along with job aides and other technical materials (e.g. client information materials, quality of care guides, detailed protocols (SOPs) for managing life-threatening complications) ensured an enabling environment for newly motivated staff. The culture of innovation spread to HWs and motivated them to come up with “home-made” solutions such as the Newborn Resuscitation Stand mentioned earlier Further, the consistent use of partographs demonstrates a key, traditionally difficult achievement 7 illustrative of MCHIP’s innovative and effective approach. Finally, newly skilled and motivated health workers were better able to engage with communities, garner their support and harness local resources to fill some of the gaps identified during the interventions. (iii) What factors may affect the feasibility of scaling up these innovations and how? FINDINGS The factors most commonly cited by key informants and HWs that may affect the feasibility of scaling up MNCH innovations include: • Buy-in from the government and MOHCC: The MCHIP approach is but one of several initiatives seeking to improve service delivery in Zimbabwe; other partners are using performance based incentives to try to improve quality of care. It would be better if the MOHCC were to select one approach, which may in effect be one that selects different elements from several initiatives, to implement nationally with support from all partners. Should the MOHCC not select the MCHIP approach, or elements of it, it will be very difficult to rollout the activities outside the already selected province because doing so would result in confusing HCWs with different approaches. • Alignment and harmonization of donor initiatives: As noted above, other donors, including the World Bank, are also engaged in initiatives to improve quality of care, e.g. RBF and HTF. These are implemented in geographical areas that overlap with those where MCHIP is working. Close coordination and cooperation will be necessary to avoid confusing staff and conflicts of interest, and to ensure the activities are complimentary. • Acceptance of MNCH innovations among staff: The main innovation that caused some difficulties with staff was the use of the SBM-R assessment tool. This was initially considered too long and bulky by HWs, but many subsequently came to see it as an invaluable aide to providing high quality care. A careful and measured approach would need to be used, with the assistance of “the converted” to facilitate widespread acceptance. • Scarcity of human resources: One of the key aspects of MCHIPs intervention was the extent and high quality of the post-training follow-up and on-the-job support provided. Whether there is adequate technical, hands-on expertise and support available, especially at peripheral levels, to support wide scale-up is debatable. • Mentoring: This is closely linked to the above. Staff from facilities where the intervention has been successful would ideally be used to mentor staff in new facilities. It is not certain that staff from these facilities have the necessary skills for this. Furthermore, such an approach may put an additional strain on already over-burdened clinical staff. • Resources (primarily financial) and availability of funding to extend the MCHIP approach. KIs expressed a general sentiment that expansion will require additional funds that are not available. • Inadequate infrastructure – availability of space and services to expand the number and size of KMC units is limited and could limit scale-up. • Inadequate supply of essential equipment at facilities limits staff ability to carry out the procedures in which they have been trained, including management of post-partum hemorrhage and eclampsia, 7 Globally, successfully getting HWs to consistently use the partograph to manage labor, let alone “like using it” is notoriously difficult and rare. 28 and neonatal resuscitation. • Coordination structures across the health sector were reported to need improvement both within government and between partners. • Community engagement, including knowledge of and participation in MCHIP activities, would be crucial to ensure widespread acceptance of the approach. • Stability of socio-economic circumstances in communities could prevent adoption of the approach. • Adaptation of standards; as noted earlier, the SBM-R standards and assessment criteria are long and time consuming. Adaptation of the tools to make them user-friendlier may facilitate scale-up, especially if the MCHIP presence is not as intense as in the current program. • At the time of the evaluation, the “R” part of the SBM-R approach had not been implemented; many respondents commented on the absence of the promised recognition. It would be important for scale up that the issue of recognition, or incentives is addressed earlier. This will be more the case if other interventions are clearly promising financial incentives for performance. CONCLUSIONS Widespread scaling up will require political will, substantial human and financial resources and adaptation of tools. Stronger coordination and improved management of existing resources will be essential to maximize impact. HWs and VHWs will need ongoing support, as well as a mechanism for providing professional recognition and incentives to sustain their quality of work. (iv) What was the nature of relations between MCHIP and key MNCH stakeholders and how did the relations contribute to the achievement of results? FINDINGS The majority of key informants (18 of 21) including Ministry, donor or NGO partners reported feeling positive or very positive about their relationship with MCHIP, and 11 of 13 KIs reported that good relations were a major factor in MCHIP’s success. On the contrary, informants of key donor partners DFID and EU stated they did not know much about MCHIP, indicating a potential weak relationship between MCHIP and specific stakeholders. Key informants noted that MCHIP staff were responsive, collaborative, supportive, communicated effectively, worked with partners as a team, and were technically knowledgeable. This high regard for MCHIP facilitated dialogue and made it much more likely that managers and facility staff would engage constructively with its staff, accept their advice and implement their recommendations. It also encouraged clinical staff to ask for assistance and enabled MCHIP staff to offer criticism without risk of jeopardizing the relationship. The resulting open, two-way communication was found to have contributed to the adoption of suggested innovations and the achievement of results. Figure 10. “There is an open door and we meet regularly. They have supported this office a lot.” “They are an asset to the MOH and we have not had any problems.” “It is a complementary relationship bringing technical expertise and finances.” “MCHIP communicated well with all the relevant offices and stakeholders.” - MOH KIIs “Our relationship is perfect. As an NGO we have a good relationship and appreciate their tools and knowledge.” 29 - NGO The health workers in all focus groups were strongly positive about their relationship with MCHIP, and the VHWs were also universally positive. The support provided by MCHIP, both technical and material, raised morale in a workforce long recognized to be struggling under the dual burden of trying to serve communities with high morbidity and with inadequate resources, and increased their motivation. It is crucial to note that, at the time of the evaluation, no incentives other than training, support, small items of equipment, and minor infrastructure repairs had been provided; nonetheless service quality undeniably improved. Figure 11. “A lot was done by MCHIP. When you call them they come! They are always welcome here and always come with a smile.” “We’ve had other partners but MCHIP is the best. They demonstrate to you, they are skilled and assist you and address whatever you ask.” - Chakohwa Rural Health Center Traditionally, resources provided for one program are typically guarded and not available for use by another. In the case of MCHIP, the excellent relationship between MCHIP and National Malaria Control Program (NMCP) reportedly allowed for the use of funding designated for malaria to be leveraged to improve MNCH outcomes. CONCLUSIONS MCHIP utilized good working relationships with the MOHCC and its implementing partners, to successfully collaborate on a large number of national policies, guidelines, protocols, national training curricula and surveys. These include:  Policies: Reproductive Health (RH) Policy, QA/QI Policy, IYCF (Infant and Young Child Feeding) Policy, Expanded Program on Immunization (EPI) Policy  Guidelines and Protocols: MPMA Guidelines, Supportive Supervision (SS) Guidelines, EmONC protocols, Mother’s booklet  National Training Curricula: National MNCH Training of Trainers, Malaria, EmONC, KMC, IMNCI, HBB, infection prevention (IP), RED, PCV, Rotavirus  Studies and Surveys: NIHFA/QOC, IYCF Program Review, IYCF formative research  Other: PCV vaccine introduction, MNCH related launches and health events, strengthening of MNCH coordination platforms Strong relationships with most stakeholders allowed MCHIP to overcome substantial barriers, and MCHIP’s influence can be seen at the national, provincial, and district and community levels. MCHIP skillfully cultivated and utilized strategic partnerships and as a result global best practices such as KMC, HBB, EmONC, and IMNCI are now accepted and promoted by the MOHCC and MNCH donor partners. However, while relationships at the higher levels within the health care system were clearly strong and productive, not all key stakeholders appeared to have strong working relationships with, or even knowledge of, MCHIP (DFID and EU). However, it is arguable whether cultivating relationships with other donors is an appropriate role for MCHIP staff; higher level relationships must be built and fostered at the level of USAID. Nevertheless, it is maintained that greater visibility amongst all other donors, but especially amongst those highly active in the health sector, is beneficial to MCHIP moving forward. At the operational level, MCHIP’s approach to building and maintaining good relationships enabled them to overcome initial reluctance, and even refusal, to engage and adopt the proposed interventions. Consequently staff at the hospital, clinic and community level has fully adopted the new ways of working and are independently spreading the word. For example, at Mutare Provincial Hospital, staff rotating into the maternity unit are routinely given an induction into the SBM-R approach and the associated standards 30 and criteria. The approach has produced the best possible result: knowledgeable, skilled, and self-motivated HWs whose success, both individually and as a team, contribute to on-going and sustained improvements in quality of care. (v) What challenges or barriers to achievement of results did MCHIP experience in Zimbabwe? FINDINGS Barriers and challenges cited by key informants during interviews and HWs and VHWs during the FGDs include: • Human resource shortages and a lack of skilled manpower such as midwives and nurses. • Competing demands for participation of HWs in different programs that target the same health professionals at the same time. One HW referred to a colleague who hadn’t been at her place of work for 3 consecutive weeks due to attendance at (mostly) workshops and meetings; by the time she returned to her station she had largely forgotten what the first workshop was about and therefore was unable to brief colleagues. A further issue is the multitude of reports required by the different programs, which results in an unwelcome and unhelpful burden of form filling. • Low morale and poor motivation was reported as endemic. Inadequate supervision resulted in a lack of accountability and inadequate infrastructure and resources provided a ready excuse for poor standards of care. A lack of community involvement also allowed for poor performance. MCHIPs approach reportedly addressed many of these failings and resulted in widespread improvements. • Coordination throughout MOHCC and absence of effective platforms for coordination among partners. For example, some technical working groups (TWGs) were reported not to function at an optimal level. • Leadership is critical at all levels; there have been reported vacancies for extended periods of time or frequent changes in leadership for key posts at the national, provincial and district levels. • Resistance from some religious sects to implement critical interventions such as immunization. • Lack of adequate infrastructure including sources of water at health centers and hospitals and inadequate number of available beds to provide quality care. • Inadequate supplies, drugs and materials such as antibiotics, zinc, disinfectants for infection prevention (IP), and gloves. • Lack of diagnostic technologies such as sonograms or dopplers to assess fetal well-being during labor reported as a barrier to diagnosing complications. Absence of this equipment at times requires unnecessary transfer of some patients to other facilities. CONCLUSIONS Ineffective and lack of coordination structures within the MOHCC are a challenge to effective implementation of programs. Chronic HR and equipment shortages impede the ability to reach MCHIP’s main objectives. 2. How effective is the SBM-R approach in improving MNCH care in MCHIP supported health facilities in Zimbabwe? i) What proportion of MCHIP supported facilities is achieving a minimum set of MNCH care standards? FINDINGS MCHIP started working on MNH standards in 17 facilities in Nov 2010. After orientation, a baseline assessment was carried out, before implementation proper began. Table 21 shows numbers of respondents whose facility had, at the time of the evaluation, received MCHIP support in different areas of SBM-R. 31 MCHIP was active in each of the 17 facilities, thus the figure reflects the percentage of staff aware of specific MCHIP activities within their facility. Table 21. Areas of SBM-R Support in Health Facilities Was the facility supported through SBM-R in the following areas? Area No. of Respons es Yes No Not Sure Freq. % Freq. % Freq . % Management of MNH services 67 62 92.5 3 4.5 2 3.0 MNH human resources 66 55 83.3 5 7.6 6 9.1 Physical and material resources for MNH 68 63 92.6 2 2.9 3 4.4 Health Education 67 66 98.5 0 0.0 1 1.5 Antenatal care 71 71 100 0 0.0 0 0.0 Normal labor and delivery and essential newborn care 71 71 100 0 0.0 0 0.0 Postnatal care 71 71 100 0 0.0 0 0.0 Emergency obstetric care 71 71 100 0 0.0 0 0.0 Emergency neonatal care 71 71 100 0 0.0 0 0.0 Infection prevention 70 67 95.7 1 1.4 2 2.9 Immunization (Reach Every District) 72 53 73.6 8 11. 1 11 15.3 IMNCI for 0 – 2 months 71 49 69.0 9 12. 7 13 18.3 IMNCI for 2 months – 5 years 71 49 69.0 9 12. 7 13 18.3 At baseline, 14 of the 17 participating health facilities in the two learning districts scored “0” in meeting MNH related performance standards; the highest score was 50 percent. By September 2013, 12 of the 17 facilities have achieved over 80 percent of MNH performance standards - a significant improvement from baseline, at which point no health facilities had reached this level. Figure 12 below highlights the proportion of facilities achieving a minimum set of MNH standards, both before and after the intervention, based on MCHIP assessments. 32 Figure 12. Proportion of MCHIP Supported Facilities Achieving Minimum Set of MNH Care Standards8 With the exception of one facility for which no baseline data was reported, all MCHIP-supported health facilities substantially increased their total performance scores, as measured by percent of MNH care standards met. On average, participating health facilities increased total performance score over the course of the project from approximately 25 percent to just over 80 percent. The picture on child health (CH) performance is indeed positive, but not quite as remarkable as the MNH picture. However, it should be noted that efforts to improve CH care started much later (Oct 2011) than MNH related activities (Nov 2010). The CH baseline assessment was conducted by clinic staff and is therefore not directly comparable to the latest assessment, which was carried out by MCHIP staff. The comparison below, Figure 13, is between the two available external assessments that were just less than a year apart. Any comments should, therefore, be considered with this in mind. At the time of the first external assessment, 4 of 22 facilities scored above the minimum 80%. A year later, this number had more than doubled to 9, with a further 4 facilities scoring more than 70%. The average score over this period rose from 40.7% to 74.2%. In the absence of a counterfactual, it is not possible to attribute this improvement solely to MCHIP’s intervention. Due to the limited scope of this evaluation, it was also not possible to conduct KIIs or gather information specific to facility, including those facilities that did not follow the general trend. The decrease in performance could be due to a number of factors, including HR and supply availability, minor decreases in scores that pushed HFs below the minimum for certain standards, staff turnover, improvements in data quality, improved capacity to maintain accurate data, or others. However, results of the surveys and interviews the team was able to conduct, confirm that MCHIP’s effect on standards adherence is indeed significant. 8 Does not include child health 0 20 40 60 80 100 120 Total performance score, % 2010 2013 33 Figure 13. Proportion of MCHIP Supported Facilities Achieving Minimum Set of CH Care Standards Table 22 below shows the proportion of respondents in the HW survey who felt SBM-R was effective in improving quality of care in different areas. Table 22. HW survey respondents who felt SBR-R was Effective (N=72) Was SBM-R Effective in Improving Quality of Care? Area Responses Yes No Not Sure Freq. % Freq. % Freq. % Management of MNH services 63 58 92.1 0 0.0 5 7.9 MNH human resources 57 41 71.9 5 8.8 11 19.3 Physical and material resources for MNH 63 57 90.5 1 1.6 5 7.9 Health Education 67 65 97.0 1 1.5 1 1.5 Antenatal care 66 65 98.5 0 0.0 1 1.5 Normal labor and delivery and essential newborn care 67 66 98.5 0 0.0 1 1.5 Postnatal care 66 65 98.5 0 0.0 1 1.5 Emergency obstetric care 64 62 96.9 0 0.0 2 3.1 Emergency neonatal care 63 61 96.8 0 0.0 2 3.2 Infection prevention 62 58 93.5 0 0.0 4 6.5 Immunization (Reach Every District) 64 40 62.5 8 12.5 16 25.0 IMNCI for 0 – 2 months 59 40 67.8 8 13.6 11 18.6 IMNCI for 2 months – 5 years 63 42 66.7 8 12.7 13 20.6 The most frequently cited areas in which SBM-R was reported to have improved quality of care were related to antenatal care, delivery and essential newborn care, postnatal care, emergency obstetric/neonatal care, and MNH service management. The activities were deemed to be least effective in improving immunization services and IMNCI. However in interpreting this result the comments made earlier relating to the duration of the support should be borne in mind. It is worth noting that many respondents in FGD commented on the value of the ORT corners in managing diarrheal illness, which was a stated focus of MCHIP. 0% 20% 40% 60% 80% 100% 120% % of CH standards fully achieved per Health Facility Axis Title 2nd Assessment (external assessment) (Sep/Oct 2012) 3rd Assessment(external assessment)(Aug 2013) 34 Table 23 gives the most common reasons for their conclusions; this was an open-ended question on the HW survey tool and the table gives a summary of collated comments. Table 23. Reasons for Respondents’ Conclusions Why was SBM-R Effective in Improving Quality of Care? Area Responses Comment Management of MNH services 9 Standards were provided 7 Mortality was reduced 5 Emergencies are better diagnosed and managed 4 Services are better managed MNH human resources 8 MCHIP came to help when staff were short 4 Staff skills improved 4 Staff were trained Physical and material resources for MNH 30 Materials/resources were provided 5 We were able to maintain quality Health Education 11 Equipment and materials were provided 9 Clients were better informed 6 Staff skills were improved 3 A work plan was developed Antenatal care Normal labor and delivery and essential newborn care 13 Able to resuscitate newborns using HBB 12 Better able to conduct normal deliveries 9 Procedures are standardized 8 Able to monitor delivery using the partograph 7 Early breastfeeding is implemented 6 Able to manage newborns using KMC 5 Mortality is reduced Postnatal care Emergency obstetric care 18 Able to manage emergencies promptly; PPH mentioned 8 times, PE/E 6 times, and HBB 4 times 8 New knowledge and skills were provided 5 Referrals were more appropriate and timely Emergency neonatal care 11 Emergencies managed promptly 5 Guidelines / standards were provided 4 Knowledge provided 4 Number of deaths reduced 4 Referrals were more appropriate and timely 4 Staff were more confident in managing cases 35 Why was SBM-R Effective in Improving Quality of Care? Area Responses Comment Infection prevention 13 Importance of hand washing highlighted and implemented 9 Knowledge was improved 7 Infection control procedures are now being followed 6 General cleanliness has improved 6 Sterile procedures now being observed 5 Awareness of post-exposure prophylaxis greater and it is available 4 Waste disposal has improved Immunization (Reach Every District) 8 Coverage has improved 5 Outreach is being done regularly 4 Supermarket approach / Fewer missed opportunities 3 Micro-planning has been implemented 3 VHWs being used to increase coverage 2 Registers have been introduced IMNCI for 0 – 2 months 8 Cases managed using protocols 5 Cases are better managed 4 We are using a supermarket / holistic approach 3 Registers were provided 2 Deaths have been reduced IMNCI for 2 months – 5 years 7 Cases are better managed 7 We are now using protocols 4 Cases now managed in OPD 3 Registers provided and being used 3 Supermarket approach being used 3 Fewer repeat visits 1 Mothers have been empowered Based on the above most frequently reported responses, health workers cited the most significant reasons for SBM-R effectiveness as related to improved service management, availability and use of innovative technologies, and general emphasis on adherence to clearly defined standards of care. CONCLUSIONS All MCHIP-supported health facilities have substantially increased their adherence to a minimum set of MNH standards, demonstrating the success of SBM-R rollout and implementation. In the area of child health, relative to maternal health, improvements are not particularly striking or universal. However, this can be explained by the fact that the duration of the CH improvement activities has been much less than MNH. An across the board improvement of 82% (from 40.7% to 74.2%) in a year, however is still a considerable achievement. MCHIP is perceived as overwhelmingly effective in its approach to improving the quality of maternal, child, and neonatal care. Adherence to MNH standards has increased the quality and efficiency of care, as reported by health 36 workers. Continual HW training in best practices is a key component of consistent health facility adherence to MNH standards. ii) How are changes in standards of care influencing health outcomes e.g., in early newborn mortality, maternal mortality, obstetric and newborn complications? FINDINGS Based on available data, there is no strong evidence of changes in overall maternal and newborn health outcomes. However, it is clear from the responses in the HW survey that provider behavior has changed. HWs stated repeatedly that previously absent SOPs (standard protocols) were now being used, that care was now standardized, and that the quality of care had improved. A frequent comment in the FGDs was “now we don’t panic when complications or emergencies occur.” In the case of infection prevention, HWs reported that they now recognize the importance of hand washing and practice it regularly before each procedure. HWs also stated that infection control and sterile procedures are being implemented and general cleanliness has improved. Another indication of this is the consistent use of the partograph, as mentioned earlier. HWs reported embracing the SBM-R concept with enthusiasm; the responses given during focus groups and to the HW survey indicate their behavior has changed and that the quality of care is now a clear priority. HWs stated repeatedly that adverse outcomes have been avoided and lives that previously would have been lost were saved due to the new practices. However, it would be very difficult to record an improvement in health outcomes, especially mortality at this stage due to: the relatively low number of deaths in participating health facilities prior to the start of the project, the short period the project has been operational, especially for CH; the fact that clinical care is only one part of the system required to prevent deaths – good communications and transport for example are others; and the absence of a control group (counterfactual). However, responses from MCHIP-supported health providers indicate that steps are being taken along the pathway from behavior change (improved attitude, performance and overall motivation) to improved MNCH outcomes. Rather than focus on measuring impact and final outcomes, the program should revise its M&E system to monitor intermediate outcomes. Anecdotal reports of improved survival of premature and low birth weight infants were frequent and the team found three case reports of infants weighing <1,000gm at birth surviving to at least 6 months, with only KMC. Figure 14. “There have been no maternal deaths in patients referred from SBM-R sites since the project began.” “Since the KMC Unit opened in April, there have been 50 admissions with only 2 deaths.” - Senior Nursing Staff Mutare Provincial Hospital “HBB has played a big role. In 2010/11 we had lots of stillbirths but not now.” - Biriri Rural Health Center Available data for early neonatal mortality, from the 2013 MCHIP Annual Report, is shown below in Figure 15. This demonstrates a decline in early neonatal mortality since 2010, correlated with significant MCHIP and MOHCC investments in facility-level quality improvement activities targeting the main causes of newborn mortality. However, as noted above, the leveling off through 2012 and into 2013 may reflect the multitude of factors affecting neonatal mortality. The clinic-based interventions of MCHIP cannot achieve the desired reductions in isolation - the remaing improvements must result from efforts to strengthen other parts of the health system. 37 Figure 15. Trends in intra-partum and very early neonatal mortality rate in MCHIP￾supported health facilities, aggregated by quarter for the period 2010-2013 (n=73)9 Changes observed by VHWs in the communities as a result of MCHIP Intervention VHWs consistently mentioned in the FGDs that there has been a noticeable community response to and increased awareness of their role. However there is no hard evidence to back up these assertions since there is no monitoring tool to confirm the first and the only way to confirm the second would be a community based survey, before and after the intervention. Noted frequently were:  Improved early ANC booking by pregnant mothers  Increased male participation and involvement in pregnancy, birth, and child health  Increased exclusive breastfeeding  Reduced work load on HWs as diagnosis and treatment of uncomplicated malaria is performed in the communities MCHIP Impact on Gender Issues There is no quantitative data to indicate the impact of the intervention on gender. However, data from FGDs and staff interviews reveals the following:  VHWs and HWs, the vast majority of which are female, are empowered by their new skills, capacity and leadership skills and note increased respect from communities  Men now frequently accompany their pregnant wives to clinics for consultations and delivery  ANC training has helped the VHWS to improve the attachment between fathers and mothers as fathers now attend most of the ANC visits with the mothers  Fathers/husbands are now accepting the role and interventions of VHWs more readily than previously CONCLUSIONS In spite of the absence of hard data – due to inadequate M&E and the short time span of the interventions – as well as a valid control group against which to compare health outcomes, qualitative findings indicate that SBM-R has contributed to a decrease in maternal, and especially, neonatal morbidity and mortality among clients of MCHIP-supported health facilities. 7Adapted MCHIP FY13 Annual Report: October 1, 2012 – September 30, 2013 0 50 100 150 200 Q4 2010 Q1 2011 Q2 2011 Q3 2011 Q4 2011 Q1 2012 Q2 2012 Q3 2012 Q4 2012 Q1 2013 Q2 2013 Q3 2013 Neonatal deaths 38 iii) How acceptable is the SBM-R approach to service providers, policy makers and other MNCH stakeholders in Zimbabwe? FINDINGS Of the 12 key informants who offered an opinion, four said the SBM-R was very acceptable and eight said it was acceptable. In FGDs, HWs were positive about SBM-R and its impact on their capacity to provide good quality care. In spite of the length of the tool, which they found intimidating at first, HWs reported that it was comprehensive, told them exactly what to do and gave them greater confidence to undertake procedures; comments such as, “now we know what quality care means” were common. Another frequent comment was along the lines, “we didn’t know before that we could ask the community for help; now we ask them for resources and they help us”. 100% of respondents in the HW survey said it should be rolled out to other facilities/districts. However, in FGDs, numerous HWs noted that the SBM-R scoring system, which requires a 100% compliance with criteria to achieve a standard, is demoralizing, can be complicated and time-resource intensive; paradoxically many said the tool should be shortened, but should remain comprehensive. Notwithstanding their reservations noted above, as illustrated in Figure 16, the vast majority of respondents found SBM-R activities to be very useful, while the remainder found them moderately useful. Figure 16. HW Responses to Usefulness of SBM-R Activities When asked for reasons they though it useful, respondents were able to give multiple reasons for each activity. The most common reasons given, across all activity areas, are summarized in Table 24. Detailed reasons are in Table 25. Table 24. Reasons Why SBM-R Was Considered Useful Reason Frequency It told us what to do 32 It standardized care 29 It improved quality of care 57 It identified gaps 61 Resources were provided 21 Staff capacity was increased/training was provided 33 It increased motivation 25 It encouraged self-assessment 6 0.0 10.0 20.0 30.0 40.0 50.0 60.0 70.0 80.0 90.0100.0 Orientation of all staff Setting of standards 1st external (Baseline) assessment Development of action plans Training and implementation of standards Self assessment Supportive Supervision 2nd external assessment Recognition for achievement of standards % of those responding in HW survey who found MCHIP led SBM-R activities very/moderately useful Very Useful Mod useful Not Useful 39 Table 25. Respondent Reasoning for Usefulness of SBM-R Why was this SBM-R Step Useful? No of Mentions Comment Orientation of all staff Setting of standards 19 It told us what to do 15 Procedures were standardized 8 It improves or ensured quality of care 1st external (Baseline) assessment 23 It identified gaps 16 It helped upgrade standards 6 It demoralized staff Development of action plans 13 Expectations were clarified 11 It identified or addressed areas for improvement 8 Resources were provided 6 Nursing care/standards were improved Training and implementation of standards 11 Workshops were held 10 Staff capacity was improved 8 Skills were improved 7 Quality of service was improved 6 Care was standardized Self-assessment 12 Helps improve quality of care 12 Identified areas in need of improvement 7 Provided new knowledge and skills 6 Helped us assess and improve each other 6 Created a desire to improve/build individual capacity Supportive Supervision 13 Helped address or fill gaps 8 Improved motivation and confidence 7 Helped identify gaps 7 Provided on-the-job training 7 Was participative and friendly 2nd external assessment 11 We got a better score than at 1st assessment 9 Identified areas for improvement 8 It improved standards 6 Quality improved 4 It provided encouragement Recognition for achievement of standards Not implemented at time of evaluation CONCLUSIONS The SBM-R approach is clearly very acceptable to service providers, policymakers and most other MNCH stakeholders who are familiar with the approach. There were very few negative comments, except in relation to the scoring system and the length of the tool. However, complaints regarding the length of the tool decreased as familiarity with the tool increased. The proof of acceptability is that in every facility visited staff used the tools on a daily basis, reported assessing each other and unanimously recommended 40 roll-out to other facilities and districts. While the SMB-R approach enhances the quality and efficiency of care, it is resource-intensive and requires adequate on-the-job training and support to ensure service providers at all levels, including nurses, have the appropriate knowledge and skills to carry out the activities required. iv) Describe and analyze factors contributing to the effectiveness or ineffectiveness of the SBM-R approach? FINDINGS Some of the factors contributing to the effectiveness or ineffectiveness of the SBM-R approach cited by KIs and HWs are listed below: Table 26. Factors Contributing to or Hindering Effectiveness of SBM-R Contributing Factors Hindrances Raised the issue of quality Length and complexity Communicates a sense of “can-do” All-or-nothing scoring system - demoralizing Identifies what is expected – from both supervisors and (V)HWs Perception of time required to use – especially in view of HR shortages Helps identify gaps in quality/service provision On-the-job training/follow-up Assists in making the most of available resources Comprehensive Strong support from MoHCC In FGDs it was frequently stated that all staff in a facility should be introduced to SBM-R and trained accordingly so they can play a full part in the quality improvement process; this particularly referred to Nurse Aides. At another level it was noted that hospital nursing-staff, especially at the provincial hospital, felt left out of the process. One participant noted that district staff had greater knowledge and skills than provincial staff and this could cause problems on referral. Figure 17. “MCHIP brought nursing services where they weren’t before-we now have the knowledge to improve quality.” - Chakohwa Rural Health Center Nurse Of the HWs surveyed, 97% were familiar with SBM-R. When asked about the purpose of SBM-R, the 70 (of 72) respondents mentioned improving the quality of care 27 times; assessing quality or standards was also mentioned 27 times, while reducing maternal and neonatal morbidity and mortality was mentioned 13 times. Improving health worker skills received nine mentions and identifying gaps and solutions was noted four times. Illustrated in the chart below, this indicates that HWs were well aware of the overall purpose of SBM-R although some saw it as a general quality improvement program, while others saw it as more narrowly focused. 41 Figure 18. Respondent Understanding of the Purpose of SBM-R Although SBM-R had been implemented in all facilities visited, staff were asked to specify which steps in the process had been implemented in their facility or area; the results are shown in Table 21 below. Table 27. MCHIP Activities Implemented in Health Facilities Did you participate in any of the following activities organized by MCHIP? No of Responses Participation Freq. % Orientation of all staff 66 57 86.4 Setting of standards 68 56 82.4 1st external (Baseline) assessment 67 54 80.6 Development of action plans 60 46 76.7 Training and implementation of standards 61 51 83.6 Self-assessment 67 54 80.6 Supportive Supervision 67 54 80.6 2nd external assessment 62 50 80.6 Recognition for achievement of standards 50 13 26.0 Figure 19. Contributing: “We have to plan now and we go back to our notes and do self-assessment and it makes us comfortable and happy.” - Biriri Rural Hospital “SBM-R helps to identify gaps. Before SBM-R we didn’t know there was a gap-we had no benches and patients used to queue standing up.” “Self assessments make the nurses do things right.” “When you follow the SBM-R tool it really gives you quality of care.” “Some problems are attitudinal. SBM-R helped improve attitudes.” Hindrances: “The scoring system is harsh. It is demotivating.” “One fails to meet standards if not supplied with resources and proper infrastructure.” - Biriri Rural Hospital CONCLUSIONS Effectiveness: SBM-R provided HWs and supervisors/managers with an effective framework for identifying and monitoring 0 5 10 15 20 25 30 To improve QoC Reducing Mortality and Morbidity Assessing quality Improving skills ID gaps Freq 42 gaps in care and improving quality of MNCH care. HWs feel more confident in their ability to detect complications and manage emergencies. The SBM-R tools were comprehensive and useful as guidelines/protocols for supervisors and as a self-assessment tool, which assisted HWs to develop solutions to problems using existing resources. Strong support by the MOHCC at all levels contributed to the effectiveness of SBM-R. Ineffectiveness: The length and complexity of SBM-R makes it challenging to implement where human resources are scarce. The SBM-R scoring system is perceived almost universally by HWs to be “harsh, demoralizing and demotivating.” The fact that not all staff in facilities were trained was considered a drawback to effectiveness, and not equally supporting all levels also raised issues in this regard. Results from the KIIs and FGDs indicate that wider based training is necessary for successful nation-wide scale-up. v) What factors may affect the feasibility of scaling up this approach and how can the SBM-R approach be adopted to increase potential for successful nationwide scale up? FINDINGS Respondents during the KIIs and FGDs regarding the feasibility for scaling up SBM-R stated: • There is a perception among service providers that SBM-R is “big” and “difficult” • A strategy or policy for scaling up nationally needs to be developed • There is a lack of clarity on how quality should be measured and defined • SBM-R should be sold well to staff in other facilities/districts and the results available should be disseminated, to demonstrate its effectiveness and encourage participation • The standards of care tool is too elaborate and its size should be reduced • The shortage of human resources able to provide the standard of on-the-job training and technical support needed will be a challenge to scaling up • There are a limited number of people in the MOH, i.e. outside facilities who can administer the training using the MCHIP/SBM-R approach • Better elaboration of training materials must be tailored to different settings and HW levels CONCLUSIONS Limited human resources and lack of training materials will be significant barriers for future scale-up. HWs respondents agree that scale-up is necessary for consistency across facilities, but recognize that further assessment is necessary to determine the resources and modifications required to scale up SBM-R nationally. The SBM-R tool will require further modifications, including a reduction in length and change in scoring system to fully engage service providers nationwide. RECOMMENDATIONS Overall, the evaluation acknowledges that the MCHIP approach and intervention has been very successful in improving the quality of care in the targeted facilities. MCHIP has cultivated excellent relationships with all stakeholders resulting in considerable successes at policy and strategic levels. It may be premature to record the impact of the program on final outcomes, but the high opinion of service managers at all levels is notable, and the improved morale and motivation of HWs and their obvious acceptance of and participation in the program are testament to its impact on processes if not outcomes. It is the team’s strong recommendation that MCHIP continue with their approach and roll out the program into additional facilities and districts. Specific key recommendations of the evaluation team are: Scaling-Up the MCHIP Approach to Quality Improvement using Standards Based Management and Recognition • Buy-in from the MOHCC has been identified as a critical issue for successful scale-up. It is recommended that MCHIP continue to play a leadership role in improving quality of care and continue policy dialogue with the MOHCC and key stakeholders to support the identification of a 43 single national approach to QI. This should be followed by collaboration and support to develop an implementation strategy for QI. • MCHIP should continue to strengthen the harmonization and collaboration with funding partners – engaging and enmeshing the HTF (Health Transition Fund), the ISP (Integrated Support Program), MCHIP, and other EU/DFID/USG projects/programs. This could go some way towards addressing the inadequate resources, infrastructure and funding identified as obstacles to scaling up. • The use of a single approach to quality improvement and greater harmonization and collaboration with partner programs should be pursued in order to alleviate the HR shortage. Cross-program support at local levels should be explored as an option. • High-impact MNCH interventions and activities such as KMC, HBB, EmNOC, supportive supervision and self-assessment should be prioritized during scale-up • ORT corners for management of diarrhea are clearly a success. For other CH interventions, it is too early to comment and it is therefore recommended that a further assessment of these takes place in perhaps 12 months’ time. It is our impression that MCHIP staff are more MNH oriented and it is therefore recommended that they pay particular attention to CH activities during facility visits. • At the provincial level, scale-up activities should start with a review of mortality and morbidity (near misses) as well as maternal-neonatal mortality to identify districts and facilities that should be prioritized for the focus activities mentioned above. • CBT as an approach to training should be advocated, emphasizing the key parts played by follow-up and on-the-job training and support. This advocacy should include discussions with the MOHCC to have this approach formally adopted nationally and other programs, funded by USAID or other donors, should be persuaded to adopt the approach. • In order to overcome the shortage of suitable on-site supporters, MCHIP should consider using champions (leaders) from currently supported facilities to introduce SBM-R into and mentor at new sites. These champions may need additional training on supervision and mentoring to enable them to do this work. It is recommended that this approach be investigated and an individual assessment of each champion be undertaken to, as far as possible, tailor-make a training program. • MCHIP should further strengthen communication and project visibility by • Reviewing their M&E activities and identifying intermediate outcomes that can demonstrate success in different elements of the program, e.g. staff knowledge and skills; management of resources; service management; community satisfaction surveys. • Choosing success stories for greater impact (e.g. KMC successes in saving newborns < 1000 grams in rural health facilities) and • Consider making a documentary DVD to inform other donors and (busy) key stakeholders • Develop a communication strategy to overcome potential initial resistance of HWs in new districts. • MCHIP should recognize, as a high priority, the achievements, to date, of VHWs and HWs at health facilities (HF). In rolling out the program MCHIP should look at ways to bring the recognition phase in earlier. • Rural health facilities function as a unit, with everyone from the N-i-C to the gate guard and general hand playing a part in provision of high quality care; this is especially the case in smaller facilities where there may only be one or two clinical staff. It is recommended therefore that all staff are oriented in the SBM-R approach and given appropriate training and support. This applies especially to nurse aides who provide a significant percentage of the day to day care in health facilities • After initial resistance, SBM-R was perceived to be fully adopted and owned by staff at the provincial hospital. Comments were made indicating that these staff felt, to a degree, neglected. They also noted that it caused difficulties if peripheral staff used more appropriate or up-to-date management than central staff. It is recommended therefore, that increased attention is given to referral hospitals in the roll-out process. • Numerous staff noted the lack of recognition, or incentives in the current parlance, in the program to date. As noted earlier, recognition should be built in to earlier parts of the process and as a high priority the achievements of VHWs and HWs at HFs should be recognized in an appropriate manner; this could be determined by surveying staff to find out what they would consider appropriate. 44 Changes Required to the Standards Based Management and Recognition Materials and Tools • The number of criteria in each standard was reported by many to be excessive; however paradoxically the same respondents praised the comprehensiveness of the SBM-R tool. In recommending a revision, they wanted the best of both worlds. It is recommended that the tool be reviewed to see if the number of criteria can be reduced without compromising the resulting quality of care. It is acknowledged that this will not be an easy task, and the reduction deemed feasible may be minimal. It is worth noting that after using the tool for a while, many respondents reported that the too “was not so bad”. • The scoring system used during assessments came in for severe criticism. The main objection was to the “all-or-nothing” approach. The team felt some sympathy with this view; if 19 out of 20 criteria are fulfilled and the missing one has little, or no, impact on the quality of care, it seems a bit harsh to get a score of 0. It is recommended therefore that a review of the criteria be undertaken, that steps critical to safety or quality of care be identified and scoring be restricted to the resulting critical pathways. E.g. during a simulated assessment, omitting to ask the “patient” the mannequin’s name should not result in a “fail”. Community Performance Quality Improvement  The intervention with VHWs has been running for less than a year. A report on a 2nd assessment of performance, mentioned in the FY13 Annual Report was not seen by the team; in any case it is possibly too early to make recommendation. However, the sheer enthusiasm and commitment of the VHWs interviewed, leads the team to recommend that the program be continued and further assessed in perhaps 6-12 months. It is further recommended that the same approach be adopted in other rollout districts. 45 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523 January 2014 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Pamela Putney, Sean Drysdale, and Roy Mutandwa through Social Impact, Inc. End of Project Evaluation of the Maternal and Child Health Integrated Program (MCHIP) in Zimbabwe ACKNOWLEDGEMENTS The Evaluation Team would like to thank all the staff at the MOHCC (Ministry of Health and Child Care) in Harare and Manicaland who took time out of their busy schedules to talk to us about the MCHIP program; the dedicated USAID Zimbabwe staff who provided on-going feedback and support throughout the evaluation, especially Jo Keatinge, Matthews Maruva and Ioli Filmeridis; all the health workers who patiently and kindly answered our questions and filled out a lengthy survey, despite long lines of patients waiting for their attention; and the Village Health Workers who traveled long distances and often waited hours to speak to us so enthusiastically about their role in improving health in their communities and then sang us songs in extraordinary harmonies about their work. END OF PROJECT EVALUATION OF THE MATERNAL AND CHILD HEALTH INTEGRATED PROGRAM (MCHIP) IN ZIMBABWE Final Evaluation Report Annexes January 27, 2014 AID-613-TO-13-00002 under Contract AID-RAN-I-00-09-00019 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. CONTENTS Acronyms ................................................................................................................................. 1 Annexes .................................................................................................................................... 2 Annex I: Evaluation Statement of Work .............................................................................. 2 Annex II: Evaluation Methods and Limitations ..................................................................13 Annex III: Data Collection Instruments ..............................................................................21 Annex IV: Qualitative Data ..................................................................................................36 Annex V: Sources of Information .......................................................................................50 Annex VI: Disclosure of any Conflicts of Interest .............................................................58 1 ACRONYMS ANC Antenatal Care ARK Absolute Return for Kids Project cHMIS Community Health Management Information System cMNCH Community Maternal Neonatal Child Health DFID Department for International Development (United Kingdom) DNO District Nursing Officer EHO Environmental Health Officer EPI Expanded Program for Immunization EU European Union FP Family Planning HBB Helping Babies Breathe HMIS Health Management Information System HTF Health Transition Fund HW Health Worker IMNCI Integrated Management of Newborn and Child Illnesses IP Infection Prevention IPTp Intermittent Presumptive Treatment of Malaria in Pregnant Women ISP Integrated Support Program IYCF Infant and Young Child Feeding LBW Low Birth Weight MNCH Maternal Neonatal Child Health MOHCC Ministry of Health and Child Care MPMA National Integrated Health Facility Assessment MSI Management Services International NIHFA National Integrated Health Facility Assessment PCV Pneumococcal Vaccine PPH Post-Partum Hemorrhage PQI Performance Quality Improvement QI Quality Improvement RDT Rapid Diagnostic Testing RED Reaching Every District SI Social Impact SS Supportive Supervision TWG Technical Working Group USAID United States Agency for International Development USG United States Government VHW Village Health Worker WRA White Ribbon Alliance 2 ANNEXES ANNEX I: EVALUATION STATEMENT OF WORK SECTION C – DESCRIPTION/STATEMENT OF WORK C.1 OBJECTIVE AND GENERAL DESCRIPTION a. Background Information 1. Overview of Statement of Work This performance evaluation will ascertain the extent to which the MCHIP project contributed to overall learning and innovation in MNCH care in Zimbabwe at the three levels at which MCHIP projects were conducted: national, district and health facility and community. By examining the various components of the MCHIP project at their respective levels, the evaluation will provide a detailed analysis of the determinants of project performance, thereby providing critical information that will be useful for future USAID project design and implementation. 2. Country Context and MNCH Sector Zimbabwe has endured a sharp economic decline over the last decade resulting in significant reductions in public funding for basic health care services and a deterioration of the health delivery system. By 2009, Government of Zimbabwe expenditures on health amounted to$15 million, only 10% of the estimated $150 million requirement for health programming in the country. Since 2009, however, Zimbabwe’s public health system has been slowly gaining functionality. Nevertheless the system is still plagued by human resource challenges, insufficient supplies of essential drugs and materials, dilapidated equipment and decaying infrastructure. For more than a decade, Zimbabwe has been experiencing an upward trend in maternal, newborn and child mortality rates and an overall decline in life expectancy at birth. The maternal mortality rate (MMR) rose from 555 in the 2005/06 Zimbabwe Demographic and Health Survey (DHS) to 725 according to a 2007 Maternal and Perinatal Mortality Study. The upward trend in MMR is alarming having more than doubled since 1994. Childbirth with HIV/AIDS, postpartum hemorrhage, infection and eclampsia are the most common causes of mortality. Although modern contraceptive use is fairly high, the contraceptive prevalence rate (CPR) has remained unchanged between 2005 and 2010 at 60%. Under-5 mortality has also increased from a level of 62 deaths per 1,000 during the late 19902 to 84 during the current decade. Most common causes of mortally are acute respiratory infection, diarrheal disease and HIV/AIDS. Approximately two-thirds of childhood deaths occur during infancy, with more than one third taking place during the first month of life. Basic vaccination coverage has increased by 11% since 2005 to 53% in 2010 but still remains well below desired levels. Overall, 12% of children have not received any vaccination, while full vaccination coverage ranges widely among the regions. The incidence of low birth weight, a proxy indicator of maternal under-nutrition has also increased over the years with an estimated 10% of newborns being too small according to the last DHS of 2010/11. Low birth weight is attributed to the poor nutritional status of the mother which can also manifest as anemia resulting from chronic infections and malaria. 3 3. USAID/Zimbabwe Health Portfolio USAID/Zimbabwe’s current health sector portfolio has been oriented around the country’s most important public health issues and supports national programs to mitigate those issues by health technical area. In maternal and child health, USAID leveraged funds for an MNCH program, the MCHIP Project to improve service delivery in two districts in the Manicaland province. The MCHIP project is described in detail in the next section. USAID/Zimbabwe has supported a range of interventions for the prevention, care and treatment for HIV/AIDS. These includes a variety of activities such as the provision and distribution of condoms, behavior change promotion, prevention of mother to child transmission (PMTCT), voluntary counseling and testing, anti-retroviral treatment, palliative treatment, strengthening laboratory systems and logistics system support. To support family planning (FP), the Mission utilizes resources to integrate family planning and reproductive health services into other health services, particularly PMTCT and HIV counseling and testing. The overall goal of FP integration is to reduce the spread of HIV/AIDS through the prevention of unintended pregnancies. Activities include training service providers, designing communication campaigns and building local capacity to deliver a range of FP services via outreach teams. Zimbabwe has the second highest Tuberculosis (TB) rate in the world and TB is the leading cause of death in HIV positive individuals. USAID has been providing to improve TB case identification and services and program management. Zimbabwe receives Malaria funding as part of the President’s Malaria Initiative. The Mission, in coordination with other national and international partners supports a package of prevention and treatment interventions within the high risk malaria districts. Recently in 2012, the Mission formulated its Global Health Initiative (GHI) strategy to identify effective approaches to accelerate progress in reducing preventable deaths and lessen the burden of disease in Zimbabwe, with a particular focus on women and girls. The strategy aims to play an important role to assist the country rebuilding its health services and systems. Development of the strategy considered the major causes of death in the country and new assistance platforms to improve the integration of health services and systems. 4. MCHIP Project Overview MCHIP project is a centralized leader with Associates (LWA) cooperative agreement and it is the USAID flagship maternal and child health program implemented by a consortium of US-Based NGOs including JHPIEGO, John Snow Inc., Save the Children, PATH, Population Services International (PSI) and others. Zimbabwe is of the 30 “priority countries”—which account for more than 70% of the world’s maternal, newborn and child deaths—the Project works in. The project was launched in 2010 in Zimbabwe and the first implementation phase will be completed in September 2013. The goal of MCHIP project in Zimbabwe is to contribute to accelerated and sustainable improvement in MNCH through the scaling up of evidence –based, high impact, integrated interventions. The project aims to contribute to reductions in maternal and child mortality and thus support Zimbabwe’s progress towards Millennium Development Goals four and five on reducing child mortality and improving maternal health. 4 MCHIP works in Zimbabwe at the national level, and in selected districts of the Manicaland Province, implementing activities in district hospitals, rural health centers and communities. At the national level, the project assists the development of national policies strategies and guidelines. Since the inception of the project, MCHIP has supported the Ministry of Health and Child Welfare (MOHCW); the Reproductive Health Policy; Reach Every District’ field guide; guidelines for the introduction of Pneumococcal Vaccine; Emergency Obstetric and Newborn care (EmONC) training package including Help Babies Breathe (HBB); Kangaroo Mother Care (KMC) training materials; and the revised Integrated Management of Newborn and Childhood Illnesses (IMNCI) training package. At a district and health facility level, the project provides direct technical support to implement MNCH interventions. These activities are mainly implemented in two districts of Manicaland province, Mutare and Chimanimani, but other project activities extend to all the seven districts of the province. The project has a critical focus on improving the quality of maternal, newborn and child health services provided by 73 district hospitals and Rural Health Centers (RHC) through training, supervision and mentorship of health workers. In the first two years of the project, 554 trainers and 1,441 health workers have been trained in different components of MNCH such as EmONC, and IMNCI. In addition, the project carries out minor refurbishments, equips facilities with some basic equipment and strengthens monitoring systems at district and site levels. For instance, seven KMC units have been refurbished since the project began and 294 low birth weight babies have been admitted to these units. In addition, MCHIP is supporting 19 district hospitals and health facilities to use new, self-administered Standard Based Management and Recognition (SBM-R) tools to identify and address important gaps in service delivery. The project provides support to the national quality assurance unit within the MOHCW to adapt the SBM-R tools so the approach becomes institutionalized on a wider scale. The project is also working to increase routine immunization coverage, focusing on those districts with large numbers of unimmunized children. In addition to the focus at district hospital and RHCs level, MCHIP is supporting improvements in the coverage and quality of high-impact MNCH interventions provided by Village Health Workers (VHWs) in communities. This is done through provision of both formal and on-the-job training and supervision in the integrated management of maternal, newborn and childhood illnesses. The project is developing integrated training materials and supervisory tools to carry out close supervision by the MCHIP district coordinators. Regular update and joint review meetings for health workers are carried out to encourage greater peer￾to-peer support and utilization of data for decision-making. An important aspect of this work is support to VHWs and promotion of linkages between the community and the formal public health system. In addition to the focus at district hospital and RHCs level, MCHIP is supporting improvements in the coverage and quality of high-impact MNCH interventions provided by Village Health Workers (VHWs) in communities. This is done through provision of both formal and on-the-job training and supervision in the integrated management of SI Employs USAID’s Evaluation Policy to Increase Evaluation Quality and Use In designing and conducting USAID evaluations, SI employs the quality standards required by USAID’s Evaluation Policy. These include:  Selection of the best (most accurate and credible) methods given time and budget constraints.  Tools and checklists to minimize bias and to improve the validity of evaluation designs.  Description of methodological strengths and weaknesses in evaluation reports.  A data analysis plan developed during the work planning period.  A 50 point Checklist to ensure the quality of our Final Evaluation Report.  Training/orientation packages for all Evaluation Team leaders and team members to increase their understanding and adherence to the new requirements. 5 maternal, newborn and childhood illnesses. The project is developing integrated training materials and supervisory tools to carry out close supervision by the MCHIP district coordinators. Regular update and joint review meetings for health workers are carried out to encourage greater peer-to-peer support and utilization of data for decision-making. An important aspect of this work is support to VHWs and promotion of linkages between the community and the formal public health system. In addition to working hand in hand with the MOHCW, MCHIP collaborates with the other USAID funded programs including the Elizabeth Glaser Pediatric AIDS Foundation, PSI, the DELIVER project as well as other international partners such as the World Health Organization, United Nations Children’s Fund (UNICEF), Unite Nations Population Fund (UNFPA), UK Department for International Development and the European Union. 5. Evaluation Design and Methodology a. Purpose and Key Evaluation Questions This performance evaluation will ascertain the extent to which the MCHIP project contributed to overall learning and innovation in MNCH care in Zimbabwe at the three levels at which MCHIP projects were conducted: national, district and health facility and community. By examining the various components of the MCHIP project at their respective levels, the evaluation will provide a detailed analysis of the determinants of project performance, thereby providing critical information that will be useful for future USAID project design and implementation. The evaluation team will respond to the following key and sub-questions questions: First, how did MCHIP contribute to overall learning and innovation in MCHN care in Zimbabwe?  What innovative processes and products did MCHIP support or implement?  What factors may affect the feasibility of scaling up these innovations and how?  What was the nature of relations between MCHIP and key MNCH stakeholders and how did the relations contribute to the achievement of results?  What challenges or barriers to achievement of results did MCHIP experience in Zimbabwe? Second, how effective is the SBM-R approach in improving MNCH care in MCHIP supported nineteen health facilities in Zimbabwe?  What proportion of MCHIP supported facilities is achieving a minimum set of MNCH care standards?  How are changes in standards of care influencing health outcomes e.g., in early newborn mortality, maternal mortality, obstetric and new-born complications?  How acceptable is the SBM-R approach to service providers, policy makers and other MNCH stakeholders in Zimbabwe?  Describe and analyze factors contribute to the effectiveness or ineffectiveness of the SBM-R approach  What factors may affect the feasibility of scaling up this approach and how can the SBM approach be adopted to increase potential for successful nationwide scale up? After reviewing the Program Description and other background documentation, the team may propose additional or alternate questions. The evaluation matrix presented in the RFTOP will be updated and submitted for approval following clarification and finalization of the key questions with USAID. 6 b. Evaluation Methodology The evaluation methodology will combine a review and analysis of quantitative data and application of qualitative techniques to obtain data from various counterparts, partners, beneficiaries and other stakeholders. By using a mixed-method approach to analyze variables corresponding with project inputs, outputs, and outcomes, the team will aim to glean objective insights on the performance of the MCHIP project interventions. Analysis of complementary qualitative and quantitative data will lead to findings that are based on facts and evidence, as well as reliable and valid conclusions. Specifically, the team will use: 1) secondary data and existing project information, such as quarterly and annual reports and other technical reports, SBM-R baseline assessment and subsequent assessments, and the National Health Information System (NHMIS); and 2) primary data collected through key informant interviews (KIIs), focus group discussions (FGDs), and a quantitative survey of MCHIP-supported health workers. Specific attention will be given to gender considerations while collecting and analyzing data. Where possible, the evaluation team will sex-disaggregate data for outputs and outcomes of interest, in order to determine the degree to which the project affected both women and men. c. Data sources, data collection instruments and analysis: Focused documentary review: Prior to arriving in-country, the evaluation team will conduct a detailed desk review of available project documents and data from sources provided by USAID/Zimbabwe and MCHIP, including:  The MCHIP Program description;  MCHIP annual work plans;  MCHIP annual and quarterly progress reports;  The MCHIP Performance Management Plan;  SBM-R baseline and subsequent assessments;  MCHIP technical reports including , the Village Health worker baseline report, National  Integrated Health Facility Assessment (NIHFA) report, Infant and Young Child Feeding  Assessment report, and the Head Count of Children under 5 Years Report and the Situational  Analysis Report of 2010; and  Facility level data reported through the National Health Management Systems (HMIS) As necessary, additional documentation identified for review while in-country will be requested by the team in order to perform the proposed analyses. Key informant interviews: The team will conduct semi-structured key informant interviews (KIIs) and with a wide range of managers, implementers, and other stakeholders in Harare and Manicaland Province. Interview guides, developed and customized for each key informant category, will be used to frame in depth discussions aimed at gleaning qualitative information on how MCHIP has contributed to MNCH care at the national, provincial, and district levels. KIIs will be particularly useful in exploring attitudes on stakeholder relations and challenges to MCHIP results achievement. Some of these interviews can be structured as group meetings where appropriate, e.g., with Technical Working Groups. Key informants will include, but will not be limited to, the following categories:  USAID and MCHIP staff;  MOHCW national, provincial, and district level staff;  Other USAID supported projects and partners such as Elizabeth Glaser Pediatric AIDS Foundation (EGPAF), Organization for Public Health Interventions and Development, PSI, Population Services Zimbabwe (PSZ) and John Snow DELIVER; 7  NGOs and CBOs such as Zimbabwe Association of Church Hospitals, Save the Children, Plan International, ARK, Cordaid, International Rescue Commission, and other technical partners such as UNICEF, UNFPA and WHO; and  Technical Working Groups such as Maternal and Newborn Health Steering committee, Child Survival task force, Immunization Interagency Coordinating Committee, Prevention of Mother￾to- Child Transmission of HIV, Nutrition task force and Health Management Information Systems. The team acknowledges two inherent biases associated with this type of data collection. One limitation is the possibility of recall bias amongst key informants. To reduce recall bias in the design phase, the team will frame interview guides using questions to aid accurate recall. Where possible, the team will use alternative sources to corroborate interview findings. A second bias potentially introduced in conducting KIIs is the subjectivity of self-reported data. In order to address this potential bias, the team will purposively recruit a diverse sample of informants and triangulate responses with dat1a collected from other methods, such as documentary sources. Biases will be accounted for during the data analysis phase and will be acknowledged in full in the final evaluation report. Focus Group Discussions: The evaluation team will conduct focus group discussions (FGDs) with various cadres of MCHIP stakeholders from a sample of 19 MCHIP-supported health facilities to gain an in-depth understanding of their experiences with project interventions. Due to their participatory and dynamic nature, FGDs will be useful for supplementing KIIs and quantitative data by investigating the acceptability of the SBM-R approach among service providers and other stakeholders, as well as the factors that may affect the feasibility of SBM-R approach. One set of FDGs will be conducted among facility managers, service providers and other staff trained through MCHIP working at district hospitals and RHCs. It is envisioned that one FGD will be conducted in each facility visited, totaling between 8- 10 FGDs. These FGDs will help elucidate the effectiveness of training activities supported by the project and how these activities have helped to improve service quality. Further, FGDs will inform the team of the effectiveness and the acceptability of the SBM-R approach. A second set of FGDs will be conducted with trained VHWs in communities surrounding the facilities visited by the team. An estimated 8-10 FGDs will be organized with VHWs. Findings will provide insight into perceptions of the SBM-R approach and the effectiveness of trainings from the perspective of VHWs and how newly acquired skills in integrated management of maternal newborn and childhood illnesses have been utilized in the community. FGDs are subject to similar biases as KIIs (i.e. recall bias and subjectivity), and have the added challenge of being dominated by the most powerful voices in a group. Power dynamics between individuals based on status and sex will be a key consideration for the SI team when constructing focus group samples. If deemed appropriate, the team may opt to conduct sex segregated focus group discussions to mitigate challenges of this kind. 1 SI is prepared to adjust the sample design to reduce the margin of error, pending discussions with USAID regarding the practical and budgetary implications of doing so. SI’s Experience Using Focus Groups SI has conducted over 100 focus group discussions (FGDs) as part of our field evaluations in development contexts. We are a leader in methodologies for planning, conducting, and analyzing FGD results. At the request of the Office for Learning, Evaluation, and Research (PPL/LER), SI recently provided a Technical Note on Focus Group Interviews which will serve as a major technical resource for evaluators and program managers. 8 Quantitative Survey of Health Workers: To gain additional data pertaining to the acceptability of the SBM-R approach and its effectiveness, the team will conduct a quantitative survey with health workers from MCHIP-supported health facilities. This survey will be used to triangulate with qualitative findings from KIIs and FGDs, allowing structured data to be collected from a larger number of project beneficiaries than through KIIs or FGDs. Assuming a 95% confidence level and 10% margin of error, a sample of 91 health workers (of 1,441 total) will be drawn using simple random sampling1. Health workers will be asked a series of closed-ended questions related to the factors perceived to contribute to the effectiveness of the SBM-R approach and what factors may affect the feasibility of its scale-up. Specifically, data on the prevalence of explicit health worker knowledge, attitudes, and behaviors will be solicited using rating scale questions, whereby responses are ascribed a score on either a Likert or semantic differential scale. Multiple choice questions will also be used to capture the degree to which the SBR-M approach is perceived to be effective and replicable at the national level. Systematic field testing of the survey instrument will be conducted through respondent focus groups to ensure that variations in language, terminology, or interpretation of questions and response options are accounted for prior to the tool’s use. Probability sampling of health workers poses the potential for introducing non-response bias, in which selected individuals unable or unwilling to complete the survey lead to observed values that deviate from the true population values. To mitigate this bias, the team will confirm the accuracy of the sampling frame, ensure the quality of enumerator training, and plan for focused follow-up attempts. Analysis: Data obtained through the SBM-R baseline assessment, as well as HMIS facility level data, will be used to analyze health outcomes before and after the implementation of project interventions. To the extent possible, data collected from KIIs and FGDs will also be subject to before and after analysis to determine the extent to which MCHIP has contributed to learning and innovation in MNCH. Analysis of survey responses will complement the analyses of KIIs and FGDs while enriching the evaluation findings and conclusions. The SI Team will use parallel analysis to examine the evidence from its document review, key informant interviews, focus group discussions, and survey. In this “methods triangulation” analytical approach, the team will analyze data related to an evaluation question and relevant indicators, obtained using different methods in parallel, and then across data collection methods, and against the “likely baseline,” as applicable. Observed outcomes will be analyzed for attribution to MCHIP activities as inferred through process tracing and contribution analysis. The team will also disaggregate data collected from different sources— e.g., health workers, health facility staff, government entities, and implementing partners. Finally, the SI Team will analyze findings across the data collected using different methods, different instruments within the same method(s), and different sources to develop higher-level findings. This method, source, and data triangulation will increase the reliability and validity of our findings and conclusions. 6. Evaluation Timeline and Team Composition The evaluation will be conducted by a team of three consultants during a three month period from September to January 2014. The team will comprise of a team leader and one international and one local technical expert, accompanied by one USAID/Zimbabwe employee. a. Evaluation Plan Team Planning Meeting (TPM): 9 Following award, the team will conduct a team planning meeting (TPM) in order to:  Clarify all team members’ roles and responsibilities, including drafting of report  Review and clarify evaluation questions  Review and finalize the timeline for discussion with USAID  Review and clarify any logistical and administrative procedures for the assignment  Establish a team atmosphere, share individual working styles, and agree on procedures resolving  differences of opinion  Develop a preliminary draft outline of the team’s report for submission to USAID  Assign drafting responsibilities for the final report. Any outstanding issues related to the TPM agenda will be completed once the team meets in Harare. Initial in-brief with USAID/Zimbabwe: The evaluation team will participate in an introductory team meeting with USAID/Zimbabwe to meet the USAID management team and clarify any issues related to the purpose, expectations and plans for the assignment. During the in-brief, the SOW, evaluation methodology and logistical and administrative procedures for the assignment, particularly field visits, will be clarified. The team and USAID management will confirm the evaluation workplan components and associated schedule (including milestones, deliverables and due dates), and a communication plan will be developed for weekly progress briefings and reports to the Mission. Initial Meetings with MCHIP Project staff and MOHCW staff: Employing a participatory approach, the team will work closely with MCHIP project staff and key partners, primarily the MOHCW during the evaluation. SI’s Team Leader Pinar Senlet will coordinate closely with the evaluation Contracting Office Representative (COR) to ensure that appropriate introductions and protocol preparations are made for the team’s collaboration with the Government of Zimbabwe entities. The team will discuss the SOW and expectations from the evaluation with MCHIP and the MOHCW, and share the agreed-upon work timeline. Preparation of final evaluation design, workplan, data collection tools and report outline: During the first week work in-country, the evaluation the team will finalize the evaluation design, evaluation matrix, workplan, data collection instruments, and outline for the final evaluation report. Data Collection: The team will collect data from key stakeholders in Harare during the first and second weeks of the evaluation. Data collection will then continue in Manicaland Province for two additional weeks, during which additional stakeholders and MCHIP beneficiaries will be interviewed and surveyed. Field visits will be arranged with the assistance of a local logistics coordinator. Field trips and site visits: The team will travel to Mutare and Chimanimani districts of the Manicaland province during the third and fourth weeks of the in-country evaluation. Depending on the magnitude of MCHIP interventions, another district may be added to the schedule to achieve a geographically representative sample of the district hospitals and rural health centers supported by MCHIP. The selection of facilities and communities to be visited during field trips will be determined in consultation with USAID and with inputs from MCHIP and the MOHCW. From a frame of 73 MCHIP-supported facilities, it is envisioned that 8-10 district hospitals and rural health centers will be sampled based on selection criteria developed by the evaluation team. 10 Criteria will include factors such as:  Duration and intensity of project support: Facilities receiving support of longer than average duration may provide more useful insights on the project’s effectiveness.  Use of SBM-R approach interventions: At least half of the selected facilities should have received project support in SBM-R in order to evaluate the effectiveness of the approach.  Performance of the individual facility: A selection of both high performing and low performing facilities will help to understand factors contributing to success and barriers to achievement.  Logistical convenience such as ease of traveling, location of the facility, security considerations, etc. During the field trips, the evaluation team members will divide tasks in order to interview key district and facility level managers, service providers, and other stakeholders, taking inventory of facilities and conducting focus group discussions. Staff of MCHIP and MOHCW may be requested to accompany the evaluation team during field visits to facilitate introductions assist with logistics; however, in order to maintain the objectivity and independence of the evaluation, these staff will not participate in interviews or discussions. b. Team composition The SI Team, composed of Team Leader, Ms. Pamela Putney2, MNCH Expert, Mr. Sean Drysdale3, and Local Evaluation Specialist, Mr. Roy Mutandwa will travel to Zimbabwe to conduct data collection and evaluation activities. All team members will be involved in the development of the evaluation methodology and instruments at the outset, and in subsequent data collection, analysis, preparation of findings and recommendations. The Team will be supported by a local logistician to assist in arranging meetings and travel. Throughout the duration of the evaluation, the team will be supported by Senior Technical Advisor, Mr. James Fremming, along with additional support from Program Manager, Ms. Sabreen Alikhan, and Program Assistant, Ms. Alexandra Ernst. 7. Institutional Capacity SI, 2012 USAID Small Business of the Year, holds several ongoing, global M&E contracts as a trusted evaluator of USAID programs. The SI Team is USAID’s major partner in providing evaluation training to USAID staff and implementing partners on a global basis. SI brings deep experience conducting evaluations in Sub-Saharan Africa and worldwide of health projects. C.2 REPORTS AND DELIVERABLES In addition to the requirements set forth for submission of reports in Sections F, and in accordance with AIDAR clause 752.242-70, Periodic Progress Reports, the Contractor shall submit reports, deliverables or outputs subject to the deadlines specified in Section F.4 of this TO, as further described below to the COR (referenced in Section G.2). The contractor will also be responsible for submitting the following deliverables: 2 Due to visa challenges, Ms. Pamela Putney replaced Ms. Pinar Senlet as Team Leader during the desk review process, prior to the beginning of field work. 3 Mr. Sean Drysdale replaced Ms. Pamela Putney as MNCH Expert prior to the beginning of field work. 11 The following deliverables and reports are required under the Task Order. All deliverables and reports will be in English unless otherwise noted. The Contractor and the Contracting Officer’s Representative (COR) have the authority to make small changes to the deliverables and reports specified below. Any such alteration must not change the basic substance of the deliverable, require funds beyond the amount obligated or exceed the firm fixed price or any budgetary limitation. Each deliverable shall conform to the performance standards as described in the Statement of Work, Section C. 1. Signed statements attesting to a lack of conflict of interest or describing an existing or potential conflict of interest relative to the project being evaluated by each evaluation team member. 2. Final evaluation design, work plan and timeline presented to USAID by the lead evaluator within two weeks of the award of the contract. The evaluation design will include a detailed evaluation design matrix (including the key questions, the methodology and data sources used to address each question), draft questionnaires and other data collection instruments, and known limitations to the evaluation design. The final design requires USAID approval. The work plan will include the anticipated schedule and logistical arrangements and delineate the roles and responsibilities of members of the evaluation team. 3. The evaluation team will hold an in brief meet with USAID upon arrival in Zimbabwe and review the evaluation work plan and timeline. 4. The Contractor shall provide progress briefings and reports to USAID Mission on a weekly basis. The Contractor shall use e-mail, phones and/or hard copies, as mutually agreed, in meeting this requirement. Minutes of the key reporting meetings will be recorded by the Contractor. USAID will concur on key issues after these meetings 5. The team will present preliminary findings in PowerPoint format to USAID and other key stakeholders in two separate meetings after completion of field work and initial data analysis. 6. Draft evaluation report (meeting all the criteria below) delivered to USAID for review within 11 business days from the time of return to their base offices. 7. The final report will be provided to the USAID/Zimbabwe in electronic form within 15 business days following receipt of comments from USAID. The evaluation report must address all evaluation questions included in the scope of work. It must represent a thoughtful, well-researched and well organized effort to address the evaluation purpose. Readers must have sufficient information about the body of evidence and how information was gathered to make a judgment as to its reliability, validity and generalizability. The final report should not exceed 30 pages (excluding appendices) and must include the following sections:  An executive summary: 3-5 page that summarizes the key points (project purpose and background, key evaluation questions, methods, findings, and recommendations)  Background information on the project  Purpose of evaluation  Evaluation team: must be described with particular reference to the existence or lack thereof real or potential conflicts of interest relative to the project being evaluated 12  Evaluation methods: must be explained in detail and limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.) must be disclosed in the report  Evaluation findings: must be presented as analyzed facts, evidence and data and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings must be specific, concise and supported by strong quantitative or qualitative evidence. When applicable, include statements regarding any significant unresolved differences of opinion on the part of funders, implementers and/or members of the evaluation team.  Recommendations: need to be supported by a specific set of findings and must be action oriented, practical and specific, with defined responsibility for the action  The final scope of work, evaluation tools and sources of information must be properly identified and listed in annex 8. All data and records from the evaluation must be submitted to USAID in an easily readable and organized electronic format along with the final report END OF SECTION C 13 ANNEX II: EVALUATION METHODS AND LIMITATIONS EVALUATION SCOPE AND FRAMEWORK Per USAID Evaluation Guidelines, a performance evaluation assesses the extent to which a program has achieved the targets set out at its inception. Due to the lack of valid control groups, a non-experimental performance evaluation will be employed for this evaluation. The approach will utilize a systematic and comprehensive review of program outputs and outcomes on beneficiaries, using mixed methods to investigate the effect of MCHIP activities on improvements in maternal, newborn and child health (MNCH) care in Zimbabwe. This evaluation is intended to measure quantitative and qualitative changes that have occurred in MNCH care in Zimbabwe including the collaborating organizations, their staff and broader communities. To the extent possible, the evaluation team will make before and after comparisons using the available data to measure progress towards established targets. The evaluation methodology will combine a review and analysis of quantitative data and application of qualitative techniques to obtain data from various counterparts, partners, beneficiaries and other stakeholders. By using a mixed-method approach to analyze variables corresponding with project inputs, outputs, and outcomes, the team will aim to glean objective insights on the performance of the MCHIP project interventions. Analysis of complementary qualitative and quantitative data will lead to findings that are based on facts and evidence, as well as reliable and valid conclusions. Specifically, the team will use: 1) secondary data and existing project information, such as quarterly and annual reports and other technical reports, SBM-R baseline assessment and subsequent assessments, and the National Health Information System (NHMIS); and 2) primary data collected through key informant interviews (KIIs), focus group discussions (FGDs), a quantitative survey of MCHIP-supported health workers and data on select indicators collected from SBM-R sites. Specific attention will be given to gender considerations while collecting and analyzing data. Where possible, the evaluation team will sex-disaggregate data for outputs and outcomes of interest, in order to determine the degree to which the project affected both women and men. Key Evaluation Questions The evaluation team will respond to the following key and sub-questions questions: First, how did MCHIP contribute to overall learning and innovation in MCHN care in Zimbabwe?  What innovative processes and products did MCHIP support or implement?  What factors may affect the feasibility of scaling up these innovations and how?  What was the nature of relations between MCHIP and key MNCH stakeholders and how did the relations contribute to the achievement of results?  What challenges or barriers to achievement of results did MCHIP experience in Zimbabwe? Second, how effective is the SBM-R approach in improving MNCH care in MCHIP supported nineteen health facilities in Zimbabwe?  What proportion of MCHIP supported facilities is achieving a minimum set of MNCH care standards? 14  How are changes in standards of care influencing health outcomes e.g., in early newborn mortality, maternal mortality, obstetric and new-born complications?  How acceptable is the SBM-R approach to service providers, policy makers and other MNCH stakeholders in Zimbabwe?  Describe and analyze factors contribute to the effectiveness or ineffectiveness of the SBM-R approach  What factors may affect the feasibility of scaling up this approach and how can the SBM-R approach be adopted to increase potential for successful nationwide scale up? An evaluation matrix outlining the key questions and the sub-questions, the measures/ indicators, and source of data for each question is depicted in Annex 1. The evaluation matrix will be updated and finalized based on the discussions with and guidance from USAID/Zimbabwe. SOURCES OF DATA AND DATA COLLECTION ACTIVITIES To address the key evaluation questions, the evaluation team will rely on a variety of data sources and data collection methods. The Evaluation Matrix in Annex 1 is organized around the key questions and provides a description of data collection methods to be used. The matrix further describes the variety of data to be extracted and analyzed, the source(s) of data and the types of analyses that will be undertaken to inform findings and conclusions. Selection of Project Sites Since 2010, the MCHIP project has been working to improve the quality of maternal, newborn and child health services provided by 73 district hospitals and Rural Health Centers (RHC) in Mutare and Chimanimani district of Manicaland province. From this frame of 73 MCHIP-supported facilities, 8-10 district hospitals and rural health centers will be sampled and will be visited by the evaluation team. The selection of facilities will be determined in consultation with USAID and with inputs from MCHIP and the MOHCW. Suggested selection criteria will include factors such as:  Duration and intensity of project support: Facilities receiving support of longer than average duration may provide more useful insights on the project’s effectiveness.  Use of SBM-R approach interventions: At least half of the selected facilities should have received project support in SBM-R in order to evaluate the effectiveness of the approach.  Performance of the individual facility: A selection of both high performing and low performing facilities will help to understand factors contributing to success and barriers to achievement.  Logistical convenience such as ease of traveling, location of the facility, security considerations, etc. Qualitative Methods Key Informant Interviews The team will conduct semi-structured key informant interviews (KIIs) and with a wide range of managers, implementers, and other stakeholders in Harare and Manicaland Province. Interview guides, developed and customized for each key informant category, will be used to frame in-depth discussions aimed at gleaning qualitative information on how MCHIP has contributed to MNCH care at the national, provincial, and district levels. Draft KII interview guides are found in Annex III, 1. KIIs will be particularly useful in exploring attitudes on stakeholder relations and challenges to MCHIP results achievement. Some of these interviews can be structured as group meetings where appropriate, e.g., with Technical Working Groups. Findings gleaned from KII will be triangulated with quantitative data to assess the project’s performance. Key informants will include, but will not be limited to, the following categories: 15  USAID and MCHIP staff;  MOHCW national, provincial, and district level staff;  Other USAID supported projects and partners such as Elizabeth Glaser Pediatric AIDS Foundation (EGPAF), Organization for Public Health Interventions and Development, PSI, Population Services Zimbabwe (PSZ) and John Snow DELIVER;  NGOs and CBOs such as Zimbabwe Association of Church Hospitals, Save the Children, Plan International, ARK, Cordaid, International Rescue Commission, and other technical partners such as UNICEF, UNFPA and WHO; and  Technical Working Groups such as Maternal and Newborn Health Steering committee, Child Survival task force, Immunization Interagency Coordinating Committee, Prevention of Mother￾to-Child Transmission of HIV, Nutrition task force and Health Management Information Systems. The team acknowledges two inherent biases associated with this type of data collection. One limitation is the possibility of recall bias amongst key informants. To reduce recall bias in the design phase, the team will frame interview guides using questions to aid accurate recall. Where possible, the team will use alternative sources to corroborate interview findings. A second bias potentially introduced in conducting KIIs is the subjectivity of self-reported data. In order to address this potential bias, the team will purposively recruit a diverse sample of informants and triangulate responses with data collected from other methods, such as documentary sources. Biases will be accounted for during the data analysis phase and will be acknowledged in full in the final evaluation report. Focus Group Discussions The evaluation team will conduct focus group discussions (FGDs) with various cadres of MCHIP stakeholders from a sample of 8-10 MCHIP-supported health facilities to gain an in-depth understanding of their experiences with project interventions. Due to their participatory and dynamic nature, FGDs will be useful for supplementing KIIs and quantitative data by investigating the acceptability of the SBM-R approach among service providers and other stakeholders, as well as the factors that may affect the feasibility of SBM-R approach. One set of FDGs will be conducted among facility managers, service providers and other staff trained through MCHIP working at district hospitals and RHCs. It is envisioned that one FGD will be conducted in each facility visited, totaling between 8-10 FGDs. These FGDs will help elucidate the effectiveness of training activities supported by the project and how these activities have helped to improve service quality. Further, FGDs will inform the team of the effectiveness and the acceptability of the SBM-R approach. A second set of FGDs will be conducted with trained VHWs in communities surrounding the facilities visited by the team. An estimated 8-10 FGDs will be organized with VHWs. Findings will provide insight into perceptions of the SBM-R approach and the effectiveness of trainings from the perspective of VHWs and how newly acquired skills in integrated management of maternal newborn and childhood illnesses have been utilized in the community. FGDs are subject to similar biases as KIIs (i.e. recall bias and subjectivity), and have the added challenge of being dominated by the most powerful voices in a group. Power dynamics between individuals based on status and sex will be a key consideration for the SI team when constructing focus group samples. If deemed appropriate, the team may opt to conduct sex-segregated focus group discussions to mitigate challenges of this kind. Draft FGD interview guides are depicted in Annex III.B. Quantitative Methods Focused Document Review 16 Prior to arriving in-country, the evaluation team will conduct a detailed desk review of available project documents and data from sources provided by USAID/Zimbabwe and MCHIP, including:  The MCHIP Program description;  MCHIP annual work plans;  MCHIP annual and quarterly progress reports;  The MCHIP Performance Management Plan;  SBM-R baseline and subsequent assessments;  MCHIP technical reports including , the Village Health worker baseline report, National Integrated Health Facility Assessment (NIHFA) report, Infant and Young Child Feeding Assessment report, and the Head Count of Children under 5 Years Report and the Situational Analysis Report of 2010; and  Facility level data reported through the National Health Management Systems (HMIS) As necessary, additional documentation identified for review while in-country will be requested by the team in order to perform the proposed analyses. Quantitative Survey of Health Workers To gain additional data pertaining to the acceptability of the SBM-R approach and its effectiveness, the team will conduct a quantitative survey with health workers from MCHIP-supported health facilities. This survey will be used to triangulate with qualitative findings from KIIs and FGDs, allowing structured data to be collected from a larger number of project beneficiaries than through KIIs or FGDs. Assuming a 95% confidence level and 10% margin of error, a sample of 91 health workers (of 1,441 total) will be drawn using simple random sampling4. Health workers will be asked a series of closed-ended questions related to the factors perceived to contribute to the effectiveness of the SBM-R approach and what factors may affect the feasibility of its scale-up, and effectiveness of MCHIP-supported training activities. Draft questionnaire for the quantitative survey of the health workers is in Annex III.C. Specifically, data on the prevalence of explicit health worker knowledge, attitudes, and behaviors will be solicited using rating scale questions, whereby responses are ascribed a score on either a Likert or semantic differential scale. Multiple choice questions will also be used to capture the degree to which the SBR-M approach is perceived to be effective and replicable at the national level. Systematic field testing of the survey instrument will be conducted through respondent focus groups to ensure that variations in language, terminology, or interpretation of questions and response options are accounted for prior to the tool’s use. Probability sampling of health workers poses the potential for introducing non-response bias, in which selected individuals unable or unwilling to complete the survey lead to observed values that deviate from the true population values. To mitigate this bias, the team will confirm the accuracy of the sampling frame, ensure the quality of enumerator training, and plan for focused follow-up attempts. Survey of SBM-R Sites Among the selected 8-10 facilities to be visited by the team, four to five will be facilities where MCHIP has been implementing the SBM-R approach. The team, in consultation with USAID/Zimbabwe and the 4 SI is prepared to adjust the sample design to reduce the margin of error, pending discussions with USAID regarding the practical and budgetary implications of doing so. 17 MCHIP project, will select a sample of priority indicators from the modules of the SBM-R assessment tools and collect data on these indicators in each SBM-R site. Data is envisioned to be collected by a local expert who has worked on previous SBM-R assessments and has experience in SBM-R assessment tools and data collection. Data from the survey of SBM-R sites will be analyzed in conjunction with the data collected from the SBM-R baseline survey in 2010 and subsequent assessments of 2011 and 2012 to measure changes in adherence to performance and quality standards over time in each facility. DATA ANALYSIS AND TRIANGULATION Data obtained through the qualitative and quantitative sources outlined above, as well as HMIS facility level data, will be used to analyze health outcomes before and after the implementation of project interventions. To the extent possible, data collected from KIIs and FGDs will also be subject to before and after analysis to determine the extent to which MCHIP has contributed to learning and innovation in MNCH. Analysis of survey responses will complement the analyses of KIIs and FGDs while enriching the evaluation findings and conclusions. The SI Team will use parallel analysis to examine the evidence from its document review, key informant interviews, focus group discussions, and survey. In this “methods triangulation” analytical approach, the team will analyze data related to an evaluation question and relevant indicators, obtained using different methods in parallel, and then across data collection methods, and against the “likely baseline,” as applicable. Observed outcomes will be analyzed for attribution to MCHIP activities as inferred through process tracing and contribution analysis. The team will also disaggregate data collected from different sources— e.g., health workers, health facility staff, government entities, and implementing partners. Finally, the SI Team will analyze findings across the data collected using different methods, different instruments within the same method(s), and different sources to develop higher-level findings. This method, source, and data triangulation will increase the reliability and validity of our findings and conclusions. Data analysis will commence as data is collected. The evaluation team leader will facilitate routine team discussions to ensure team collaboration during data analysis. The need for additional data, inconsistencies between sets of data collected, or any questions relating to the quality of data will be consulted with USAID and MCHIP on a timely basis to avoid delays in completion of tasks. To the extent possible, and as applicable, data will be sex-disaggregated to understand the effects of project interventions on women and men. QUALITY ASSURANCE In order to assure that data of the highest quality is collected and analyzed, the evaluation team will consult with the Mission and the MCHIP Project staff to determine the extent to which available data is complete and likely to be accurate. The identification of potential weakness in existing data at the onset of the evaluation will aid the team in focusing refining its data collection tools to ensure that data gaps are adequately addressed. The consistent triangulation of quantitative and qualitative data analysis phase will ensure that findings are drawn from evidence of the highest possible quality. SI employs a three-stage QA process for all of its evaluations to ensure high quality, evidence-based results that are useful for program improvement, accountability, and learning purposes. Each stage of the evaluation is reviewed and vetted through checklists and direct feedback is given to the Team Leader and field team. 18 Stage I: Work plan — The Senior Technical Advisor will review the feasibility and rigor of the proposed methodology and work plan and adequacy of the dissemination plan. Stage II: Draft Evaluation Report — Report structure and logical linkages among findings, analysis, conclusions, presentation of qualitative and quantitative data, and actionable recommendations will be assessed. Stage III: Final Report — A 40-point quality check of the executive summary, program, and methodology description; adequacy of findings analysis, conclusions, and final recommendations; full compliance with USAID evaluation policies; and report presentation, e.g. charts, graphs, and annexes will be conducted. REPORTING The team will prepare and maintain summaries of all KIIs and site visits conducted throughout the evaluation. Summary data will be submitted to USAID prior to departure from the country and annexed to the final report. Following the completion of site visits, the team will hold a mid-term briefing with USAID to share insights gained from key informant interviews and field trips and will a written summary of preliminary findings. This meeting will help the team to make any revisions to the evaluation plan, if necessary, and receive USAID’s guidance for the remainder of the assignment. The team will hold a final internal debriefing with USAID after the completion of data analysis to present the preliminary findings, conclusions and recommendations of the evaluation. The presentation will be delivered in PowerPoint format. A second presentation will be conducted for a larger audience of stakeholders including the MCHIP and MOHCW and others, following the internal debriefing with USAID. The team will incorporate responses to USAID comments and insights gained from the discussions with USAID into the presentation with the stakeholders. The draft evaluation report will be submitted for USAID’s review and comments by December 9, 2013. The final report will be provided to USAID no later than January 13, 2014. The final report will address all evaluation questions included in the scope of work and will not exceed 30 pages. It will include the sections outlined in the RFTOP and will adhere to the criteria set forth by the new USAID Evaluation Policy Guidelines. A draft outline of the evaluation report is depicted in Annex 5. EVALUATION WORK PLAN AND TASKS The overall evaluation work plan is shown in Annex 6. The team will carry out the following tasks throughout the life of the evaluation process: Task 1: Conduct Desk Review of Background Documents First week: the team will receive background materials from USAID and conduct a desk review. Task 2: Team Planning Meeting First week: the team will hold a planning meeting to introduce themselves to each other, discuss the SOW, clarify any questions, and agree upon the roles and responsibilities of team members. Task 3: Preparation of Evaluation Methodology and Work plan 19 First week: the team leader will develop a detailed evaluation methodology and a work plan in consultation with the team members and submit it to USAID’s review early in the second week of the evaluation. Task 4: Initial In-brief with USAID Upon arrival in Zimbabwe, the team will participate in an introductory team meeting with USAID/Zimbabwe to meet the USAID management team and clarify any issues related to the purpose, expectations and plans for the assignment. During the in-brief, the SOW, evaluation methodology and logistical and administrative procedures for the assignment, particularly field visits, will be clarified. The team and USAID management will confirm the evaluation work plan components and associated schedule (including milestones, deliverables and due dates), and a communication plan will be developed for progress briefings and reports to the Mission. Task 5: Finalize and Submit Final Evaluation Methodology and Work plan Based on the discussions with and feedback from USAID, the team leader will finalize and submit the final evaluation methodology and work plan for USAID’s approval at the end of the second week of the evaluation. Task 6: Conduct Interviews in Harare Data collection from key informants in Harare will take place during the second and third weeks of the evaluation. Task 7: Conduct Field Trips and Site Visits During the fourth and fifth weeks of the evaluation, the team will travel to Mutare and Chimanimani districts of the Manicaland province and continue data collection. Task 8: Mid-term Debrief with USAID Following the completion of site visits, the team will hold a mid-term briefing with USAID to share insights gained from key informant interviews and field trips and will provide a summary of preliminary findings. This meeting will help the team to make any revisions to the evaluation plan, if necessary, and receive USAID’s guidance for the remainder of the assignment. Task 9: Data Analysis Data analysis will commence as data is collected. The evaluation team leader will facilitate routine team discussions to ensure team collaboration during data analysis. The need for additional data, inconsistencies between sets of data collected, or any questions relating to the quality of data will be consulted with USAID and MCHIP on a timely basis to avoid delays in completion of tasks. To the extent possible, and as applicable, data will be sex-disaggregated to understand the effects of project interventions on women and men. Task 10: Final Debriefing with USAID The team will hold a final internal debriefing with USAID after the completion of initial data analysis to present the preliminary findings, conclusions and recommendations of the evaluation. The presentation will be delivered in PowerPoint format. Task 11: Debriefing with Stakeholders A second presentation will be conducted for a larger audience of stakeholders including the MCHIP and MOHCW and others, following the internal debriefing with USAID. The team will incorporate responses to USAID comments and insights gained from the discussions with USAID into the presentation with the stakeholders. 20 Task 12: Draft Evaluation Report The draft evaluation report will be submitted for USAID’s review and comments by December 9, 2013. Task 13: Final Evaluation Report The final evaluation report will be provided to USAID by January 13, 2014. 21 ANNEX III: DATA COLLECTION INSTRUMENTS A. Key Informant Interview Guide Interviewee Name: Organization: Location: Position: E-mail contact: Interviewer: Date: 1. How familiar are you with the USAID-MCHIP programme? Very Quite Moderately Not very Not at all 2. What innovative processes or products do you think MCHIP supported or implemented in Zimbabwe? (National policies, strategies, guidelines, training of health providers, SBM-R approach etc) 3. Do you think any of these interventions have had a positive effect on health outcomes? (please give examples if yes) 4. Do you think any of these interventions can be scaled up and if so what factors may be important in scaling up? 5. What was the nature of the relationship between MCHIP, your organisation / department and other stakeholders and do you think this influenced MCHIPs achievement of results? (please explain) 6. In your opinion, what challenges or barriers to achievement of results did MCHIP experience in Zimbabwe? 7. Do you think MCHIP should continue with their current approach / activities or do you think they should change the focus. If so how? 8. Are you familiar with the QI approach used by MCHIP (SBM-R)? Could you describe its main features? 9. How acceptable do you think this approach is to service providers, policy makers and other MNCH stakeholders in Zimbabwe? 10. Overall, from your own experience and at your level of the health system, would you say the approach was considered: Very acceptable Acceptable Neutral Unacceptable Very unacceptable 22 11. What factors do you think have contributed to the effectiveness or ineffectiveness of the approach? 12. How do you think the approach could/should be adopted to increase potential for successful nationwide scale-up? (if yes, please explain) 13. What factors do you think may affect the feasibility of scaling up this approach? 23 B. Health Worker Focus Group Discussion Guide Name of group (VHW, etc.) Date Form identification #: Site of FGD: Number of participants: Moderator: Note-taker: Start time: End time: Introductions (FG Facilitators and participants) Informed consent (discuss and sign forms) Brief description of purpose of FGD Ask if there are any questions/clarifications Draft prompts: PMTCT Postpartum FP KMC Breast feeding HBB Maternal interventions Newborn interventions Child health interventions Immunisation Malaria Nutrition SBM-R PQI Partnerships Supervision Do you think things have changed since the MCHIP Intervention started? What activities have had the greatest impact on MNCH care and outcomes? (Specific examples) What were the main barriers/difficulties in introducing/implementing these activities? Has the SBM-R approach been useful? Is it acceptable to (as appropriate) service providers, policy makers and other key MNCH stakeholders? What has been most effective about SBM-R? What are some of the problems/weaknesses with the SBM-R approach? What were the difficulties in introducing SBM-R? What would make the introduction of the SBM-R approach easier? 24 Do you have any other suggestions that would make introduction of any of the activities/interventions easier in other locations? 25 C. Health Worker Survey: HCW Cadre No Question Response 1 Name (Optional) 2 Age (years) 3 Sex M F 4 What is your professional background? 1. Nurse/Midwife 2. RGN 3. PCN 4. EHO/T 5. Pharmacist /Pharm Tech /Asst 6. Laboratory Tech/Assistant 7. Health Administrator 5 Are you in charge of the facility / department? Yes No 6 How many years have you been qualified? 7 How long have you been at this facility? 8 How long have you been in your current position? 9 Are you familiar with MCHIPs activities in this facility / district? If yes: please list: Yes No 10 Are any other organizations providing support to this facility / district? Yes No 11 Who are they and what support are they providing? Organization Support 12 Are you familiar with SBM-R? Yes No 26 13 What is the purpose of SBM-R? 14 Is the SBM-R approach being implemented in this facility? Yes No Not sure 15 In what month / year was SBM-R introduced? What steps of the SBM-R process have been implemented? Please indicate how useful or not these have been and explain briefly why. 15a Orientation of all staff Implemented? Yes No 15b If Yes, was it: Very useful (1) Mod useful (2) Not useful (3) 15c Explain: 16a Setting of standards Implemented? Yes No 16b If Yes, was it: Very useful (1) Mod useful (2) Not useful (3) 16c Explain: 17a 1 st external (Baseline) assessment Implemented? Yes No 17b If Yes, was it: Very useful (1) Mod useful (2) Not useful (3) 17c Explain: 18a Development of action plans Implemented? Yes No 18b If Yes, was it: Very useful (1) Mod useful (2) Not useful (3) 18c Explain: 19a Training and implementation of standards Implemented? Yes No 19b If Yes, was it: Very useful (1) Mod useful (2) Not useful (3) 19c Explain: 20a Self assessment Implemented? Yes No 20b If Yes, was it: Very useful (1) Mod useful (2) Not useful (3) 20c Explain: 27 21a Supportive Supervision Implemented? Yes No 21b If Yes, was it: Very useful (1) Mod useful (2) Not useful (3) 21c Explain: 22a 2 nd external assessment Implemented? Yes No 22b If Yes, was it: Very useful (1) Mod useful (2) Not useful (3) 22c Explain: 23a Recognition for achievement of standards Implemented? Yes No 23b If Yes, was it: Very useful (1) Mod useful (2) Not useful (3) 23c Explain: 24 What subject areas have been supported (at any time) by MCHIP through the SBM-R approach in this facility? a Management of MNH services Yes No Not sure b MNH human resources Yes No Not sure c Physical and material resources for MNH Yes No Not sure d Health education Yes No Not sure e Antenatal care Yes No Not sure f Normal labor & delivery and essential newborn care Yes No Not sure g Postnatal care Yes No Not sure h Emergency obstetric care Yes No Not sure i Emergency neonatal care Yes No Not sure j Infection prevention Yes No Not sure k Immunization (Reach Every District) Yes No Not sure l IMNCI for 0 – 2 months Yes No Not sure m IMNCI for 2 months – 5 years Yes No Not sure Do you think the SBM-R approach has been effective in improving the quality of care in (any of) the following subject areas? Please explain why you say this: 25a Management of MNH services Yes No Not sure 25b Explain: 26a MNH human resources Yes No Not sure 26b Explain: 28 27a Physical and material resources for MNH Yes No Not sure 27b Explain: 28a Health Education Yes No Not sure 28b Explain: 29a Antenatal care Yes No Not sure 29b Explain: 30a Normal labor and delivery and essential newborn care Yes No Not sure 30b Explain: 31a Postnatal care Yes No Not sure 31b Explain: 32a Emergency obstetric care Yes No Not sure 32b Explain: 33a Emergency neonatal care Yes No Not sure 33b Explain: 34a Infection prevention Yes No Not sure 34b Explain: 35a Immunization (Reach Every District) Yes No Not sure 35b Explain: 36a IMNCI for 0 - 2 months Yes No Not sure 29 36b Explain: 37a IMNCI for 2 months – 5 years Yes No Not sure 37b Explain: 38a Is the SBM-R process still being implemented? Yes No Not sure 38b If no, why not? 39 When was the last SBM-R assessment carried out? 40 Was this an external assessment or a self-assessment? External Self Not sure 41a How often are self-assessments being carried out? Monthly Quarterly Not sure 41b If this is less than the recommended frequency, why is this not happening more frequently? 42a Do all staff participate in the assessments? Yes No Not sure 42b Why / Why not? 43a Have the initial action plans been reviewed or revised? Yes No Not sure 43b If not, why not? 44 What do you think (if anything) should be changed to make SBM-R more “user-friendly” or acceptable? 45 SBM-R is currently being implemented in 2 districts only. Do you think it could / should be rolled out to other districts / Provinces? Yes No Not sure 46 Why do you say this? 47 What do you think (if anything) should be changed to facilitate the wider use of SBM-R? 30 D. Health Worker Survey: Management Cadre No Question Response 1 Name (Optional) 2 Age (years) 3 Sex M F 4 What is your professional background? 1. Nurse/Midwife 2. RGN 3. PCN 4. EHO/T 5. Pharmacist /Pharm Tech /Asst 6. Laboratory Tech/Assistant 7. Health Administrator 5 Are you in charge of the facility / department? Yes No 6 How many years have you been qualified? 7 How long have you been at this facility? 8 How long have you been in your current position? 9 Are you familiar with MCHIPs activities in this facility / district? Yes No 10 Are any other organizations providing support to this facility / district? Yes No 11 Who are they and what support are they providing? Organization Support 12 Are you familiar with SBM-R? Yes No 13 What is the purpose of SBM-R? Did you participate in any of the following activities organized by MCHIP? If so, please indicate how useful they were and briefly explain why. 31 14a Orientation of all staff Implemented? Yes No 14b If Yes, was it: Very useful (1) Mod useful (2) Not useful (3) 14c Explain: 15a Setting of standards Implemented? Yes No 15b If Yes, was it: Very useful (1) Mod useful (2) Not useful (3) 15c Explain: 16a 1 st external (Baseline) assessment Implemented? Yes No 16b If Yes, was it: Very useful (1) Mod useful (2) Not useful (3) 16c Explain: 17a Development of action plans Implemented? Yes No 17b If Yes, was it: Very useful (1) Mod useful (2) Not useful (3) 17c Explain: 18a Training and implementation of standards Implemented? Yes No 18b If Yes, was it: Very useful (1) Mod useful (2) Not useful (3) 18c Explain: 19a Self assessment Implemented? Yes No 19b If Yes, was it: Very useful (1) Mod useful (2) Not useful (3) 19c Explain: 20a Supportive Supervision Implemented? Yes No 20b If Yes, was it: Very useful (1) Mod useful (2) Not useful (3) 20c Explain: 32 21a 2 nd external assessment Implemented? Yes No 21b If Yes, was it: Very useful (1) Mod useful (2) Not useful (3) 21c Explain: 22a Recognition for achievement of standards Implemented? Yes No 22b If Yes, was it: Very useful (1) Mod useful (2) Not useful (3) 22c Explain: 23 Have the below subject areas been supported (at any time) by MCHIP through the SBM-R approach in your area of responsibility? a Management of MNH services Yes No Not sure b MNH human resources Yes No Not sure c Physical and material resources for MNH Yes No Not sure d Health education Yes No Not sure e Antenatal care Yes No Not sure f Normal labor & delivery and essential newborn care Yes No Not sure g Postnatal care Yes No Not sure h Emergency obstetric care Yes No Not sure I Emergency neonatal care Yes No Not sure j Infection prevention Yes No Not sure k Immunization (Reach Every District) Yes No Not sure l IMNCI for 0 – 2 months Yes No Not sure m IMNCI for 2 months – 5 years Yes No Not sure Do you think the SBM-R approach has been effective in improving the quality of care in (any of) the following subject areas? Please explain why you say this: 24a Management of MNH services Yes No Not sure 24b Explain: 25a MNH human resources Yes No Not sure 25b Explain: 26a Physical and material resources for MNH Yes No Not sure 26b Explain: 33 27a Health Education Yes No Not sure 27b Explain: 28a Antenatal care Yes No Not sure 28b Explain: 29a Normal labor and delivery and essential newborn care Yes No Not sure 29b Explain: 30a Postnatal care Yes No Not sure 30b Explain: 31a Emergency obstetric care Yes No Not sure 31a Explain: 32a Emergency neonatal care Yes No Not sure 32b Explain: 33a Infection prevention Yes No Not sure 33b Explain: 34a Immunization (Reach Every District) Yes No Not sure 34b Explain: 35a IMNCI for 0 – 2 months Yes No Not sure 35a Explain: 36a IMNCI for 2 months – 5 years Yes No Not sure 34 36b Explain: 37a Is the SBM-R process still being implemented? Yes No Not sure 37b If no, why not? 38 When was the last SBM-R assessment carried out? 39 Was this an external assessment or a self-assessment? External Self Not sure 40a How often are self-assessments being carried out? Monthly Quarterly Not sure 40b If this is less than the recommended frequency, why is this not happening more frequently? 41a Do all staff participate in the assessments? Yes No Not sure 41b Why / Why not? 42a Have the initial action plans been reviewed or revised? Yes No Not sure 42b If not, why not? 43 What do you think (if anything) should be changed to make SBM-R more “user-friendly” or acceptable to health workers? 44 SBM-R is currently being implemented in 2 districts only. Do you think it should be rolled out to other districts / Provinces? Yes No Not sure 45 Why do you say this? 35 46 What do you think (if anything) should be changed to facilitate the wider use of SBM-R? 36 ANNEX IV: QUALITATIVE DATA A. Health Worker Surveys Please see the accompanying PDF document, which is a compilation of the 72 health worker surveys 37 B. Key Themes from Health Worker Focus Group Discussions (FGDs) 9 FGDs with 37 Health Workers HW FGD Participant Sex Distribution Males Females 4 33 KMC KMC  They used to have more referrals to Mutambara Hospital of babies with weight of less than 2.5kg but there has been a great change as a result of KMC. A 700g baby survived and is now 6 months old  Workshops in KMC  Kangaroo care has improved baby care  Kangaroo baby unit proved to be very effective as MCHIP provided materials like towels and wrappers for use. A baby with a birth weight of 850g survived  KMC was the most significant change  MCHIP opened the KMC Unit for us  KMC has made the greatest difference, along with IMNCI  Training in KMC has assisted in managing premature babies  Cupboards for linen and uniforms for mothers in the KMC Unit are required  KMC- MCHIP provided all materials and they are no longer referring mothers with babies of low weigh to Mutambara Mission Hospital Frequency 10 Helping Babies Breathe (HBB) HBB  Resuscitation of newborn babies has improved greatly because of HBB and on job trainings (10)  Workshops in HBB  “HBB has played a big role. In 2010/11 we had lots of stillbirths but not now!”  HBB trained personnel started training others who have never gone for that training and as a result, nurses are now more knowledgeable about child delivery processes.  HBB made work easy in resuscitating babies.  HBB has had the most impact in reducing neonatal mortality  HBB has made a “great” contribution-before we were not quite clear what to do but now we are saving money because we use oxygen or tubes so rarely now!  Reduced neonatal deaths  Low mortality rate noted  There has been a sharp decline in neonatal mortality rate 38 Frequency 16 ANC ANC  We did not know about FANC  We follow mothers and babies holistically now during ANC  ANC has given us knowledge on maternal danger signs Frequency 3 Supervision Supervision  Through assessment the supervisors are able to take note of gaps/ weaknesses among nurses and plan for training  Supportive supervision is now given using the tool  The changes have been maintained by means of regular supportive supervision (4)  Supportive supervision-“Before when the supervisors were coming you would be really shaking but now the supervision is friendly and the supervision has improved greatly.”  We now assess each other daily! We sit down regularly and review the standards and discuss cases together and also strategies to improve  It provided more supervision e.g. review of work procedures and expectations and tutorials on quality are done every Wednesday in all departments  “MCHIP has trained us, supervised us and supported us!”(Biriri).  We have benefited a lot from the supervision  SIC now know the welfare of the staff; they have expanded their supervision and identified gaps and plan for trainings to improve their knowledge and services  Improved supervision was noted  MCHIP tools have helped us to overcome some of the challenges and we thank MCHIP for their support and supervision  Short-comings are being highlighted on daily basis and areas for improvement worked upon Frequency 9 IMNCI IMNCI  “IMNCI has been a big change. We now assess and check the baby and the mother at the same time-it is integrated.”  IMNCI helps a lot  IMNCI- babies are managed holistically i.e., focusing on the baby and mother during treatment  IMNCI also had a big impact. We follow mothers and babies holistically now during ANC, delivery and PP and follow up mothers with special needs. We know the danger signs now, do birth plans during ANC-it is a complete management. Through IMNCI we learned a lot about the complaints of mothers and the child-we now join mothers and child together 39  IMNCI and KMC have made the most difference  ANC, IMNCI, KMC, PNC, labor and delivery had greatest impact They argued that all interventions have yielded greatest impact as everything worked very well  IMNCI registers which is easier to fill out and they can now explore all areas easily  Participants were trained in IMNCI, resulting in reduction in frequent visits by mothers and their babies to the hospital. This in turn has reduced their workload  “IMNCI helps with children under 5 because we are no longer seeing so many repeat visits.”  There are less neonatal and under 5 deaths Frequency 9 Scoring System Scoring System  Scoring system is unfair and sometimes demotivating hence needs to be improved  Scoring system is unfair and discouraging. It is not fair to get a “0” when in L&D there are 20 standards and you have met 18 you shouldn’t get a 0  Low unfair scores discourages perceived hard work  The scoring system should be changed it is “discouraging” to use all or nothing  Improve the scoring system for SMB-R  Rather lengthy and tools are too long (2). A shorter checklist would be more appropriate  Scoring system demoralizing nursing staff (2)  Needs time and commitment  SBM-R scoring system is demotivating and not fair when you get a 0 when you’ve answered all the questions right except for one  Scoring system is unfair and discouraging. It is not fair to get a “0” when in L&D there are 20 standards and you have met 18 you shouldn’t get a 0.  Low unfair scores discourages perceived hard work Frequency 11 (9 cite scoring system as unfair!) Quality of Care Quality of Care  The SBM-R instrument ensures quality care  Quality delivery of service was observed through provision of resources and knowledge  SBM-R enhanced the quality of services  Quality of care has improved due to SBM-R and made us identify the gaps  “When you follow the SBM-R tool it really gives you quality of care.”  Standards of care on the newborn have improved tremendously.  Quality of care has improved due to SBM-R and made us identify the gaps 40 Magnesium Sulphate Magnesium Sulphate  “Before we didn’t have Mag Sulfate and now the nurses use it to stabilize women with pre-eclampsia.”  We have learned a lot! Now we know how to do something for patients before we transfer them. For example, we now give Mg Sulfate to patients with pre-eclampsia before we transfer them and the outcomes are better Frequency 2 IP (infection Prevention) IP  Assisted in infection control e.g. area 10 of SBM-R through a golden rule that of washing hands before treating patients, when you follow it, it reduces a lot of problems which were going to occur if not followed  We have improved IP and the disposal of wastes to prevent contamination. We do rapid cleaning and dust all the time Frequency 2 Danger Signs Danger Signs  We now know the danger signs and how to assess quickly and have the full knowledge  We are now trained in BEmOC and that has helped us identify the danger signs  ANC has given us knowledge on maternal danger signs Frequency 3 MCHIP Support MCHIP Support  “MCHIP has trained us, supervised us and supported us!”(Biriri)  “MCHIP has empowered the staff and we now are able to classify problems with babies.”  Now we have a proper delivery pack with 6 towels and a general exam pack  “MCHIP helped us identify our needs to improve the quality of care.”  Workshops in EmOC, RED, IMNCI, HBB, PMTCT, KMC  MCHIP renovated a ward and furnished with new equipment. They introduced the Kangaroo Mother Care System to replace incubators for premature babies was a very welcome move as it enables babies to gain more weight compared to the old system  They have provided delivery packs  IMNCI registers which are easier to fill out and they can now explore all areas easily  We have put up a temporary shelter for mothers through MCHIP  “A lot was done by MCHIP. When you call them they come! They are always welcome here and always come with a smile.”  Mother Care Waiting Unit opened 2 months ago  Materials to use i.e. standard delivery packs, scissors  Partograph- For managing normal deliveries 41  The pregnancy wheel is very useful and we now know what to do  “MCHIP was an eye opener to me and everyone.”  We now know the danger signs and how to assess quickly and have the full knowledge  We are now giving the full dose of DT  MCHIP was an eye opener in terms of infrastructure required- they have discussed with responsible authorities on what must be done to improve and helped us look for and advocate for more funds  “MCHIP helped us identify our needs to improve the quality of care.”  Now have proper standard delivery packs  VHWs received training which has improved their performance and resulted in less burden to nurses e.g in malaria treatment  MCHIP strategy was effective as it provided training to nurses, provide resources for use and let the nurses implement what they would have learnt  “Mothers are empowered by increased knowledge.”  Through MCHIP we identified lots of gaps. People had to stand waiting for services  “A lot was done by MCHIP. When you call them they come! They are always welcome here and always come with a smile.”  “We’ve had other partners but MCHIP is the best. They demonstrate to you, they are skilled and assist you and address whatever you ask.” Frequency 12 Fewer Repeat Visits Needed for Children Fewer Repeat Visits for Children  There are fewer burdens from repeat visits.”  “We are less burdened due to the VHWs-with malaria and the Palliative Care Team  Less referrals to major hospitals  Reduced hospital visits after teaching the mother in postnatal care. As a result, fewer drugs were supplied to patients  VHWs received training which has improved their performance and resulted in less burden to nurses e.g. in malaria treatment  Participants were trained in IMNCI, resulting in reduction in frequent visits by mothers and their babies to the hospital. This in turn has reduced their workload  “IMNCI helps with children under 5 because we are no longer seeing so many repeat visits.”  “We used to refer babies born who were less than 2 kgs to Mutambara and that was a problem because the mothers didn’t want to go but now we don’t have to refer them because we have KMC.” Frequency 7 Health Workers Empowered By New Knowledge and Skills HWs empowered by new knowledge  “MCHIP has empowered the staff and we now are able to classify problems with babies.” 42 and skills  Nurses now able to assess and classify patients' problems and needs more accurately  Nurses gained skills and ability to deal with problems of both the mother and baby holistically  MCHIP improved my skills in ANC and we are implementing with mothers and children and everyone  Prepared information made the nurses work very easily and prescribe correct drugs  Before RED we didn’t know how to make site visits and now we plan with the community to increase immunization coverage and it has gone up and we now know how to order sufficient vaccines and syringes.  On the labor ward we used to transfer complications but because of MCHIP we now manage them here. “Now we don’t panic” and just manage them as we were taught!  “Because of MCHIP we know how to work with the community and now we know we can ask the community for things and that has helped remove the financial barrier. The community has bought mattresses.”  Nurses became more conscious and knowledgeable in job requirements and procedures  We are now able to manage emergency complications through the knowledge we have received  Improvement in nurses’ ability to detect patients problems, treat and refer accordingly  Ability to undertake a holistic approach in patients’ examination/ diagnosis and offer appropriate intervention  “We now know how to manage clients with pre-eclampsia, PPH, resuscitate newborns and we don’t have to refer them and we often don’t have transportation so that has made a difference.”  The pregnancy wheel is very useful and we now know what to do  “MCHIP was an eye opener to me and everyone.”  We now know the danger signs and how to assess quickly and have the full knowledge  We are now giving the full dose of DT  We are now trained in BEmOC and that has helped us identify the danger signs  Resuscitation of newborn babies has improved because of HBB and on job trainings  Able to create a conducive environment (and rapport) for pregnant women to deliver well  Management of complications has improved e.g PPH, eclampsia  Diagnosis of IMNCI has improved  Nurses gained skills and ability to deal with problems of both the mother and baby holistically  We now know how to manage clients with pre-eclampsia, PPH, resuscitate newborns and we sometimes don’t have transportation for referrals  We can now classify diarrhea  We can now check for malnutrition and identify other problems that they come with 43  Assisted in infection control e.g area 10 of SBM-R through a golden rule that of washing hands before treating patients, when you follow it, it reduces a lot of problems which were going to occur if not followed  Simple things like washing hands are highlighted and we now know the important “Golden Rules” to prevent infection that can cost a lot to the center!”  The MCHIP trainings were very different and more practical than before and training of staff in management of obstetric care has contributed a lot  “Mothers are empowered by increased knowledge.” Decreased Workload for Nurses Due to VHWs Decreased workload for nurses due to VHWs  VHWs received training which has improved their performance and resulted in less burden to nurses e.g. in malaria treatment  There are fewer burdens from repeat visits.”  “We are less burdened due to the VHWs-with malaria and the Palliative Care Team  Less referrals to major hospitals  Reduced hospital visits after teaching the mother in postnatal care. As a result, fewer drugs were supplied to patients  VHWs received training which has improved their performance and resulted in less burden to nurses e.g in malaria treatment  Participants were trained in IMNCI, resulting in reduction in frequent visits by mothers and their babies to the hospital. This in turn has reduced their workload  “IMNCI helps with children under 5 because we are no longer seeing so many repeat visits.” Need for Recognition Need for recognition  Recognition must be taken into consideration and must be taken down to the villages to encourage other villages to follow suit  There is need for recognition of high performers to and motivate them accordingly through incentives Frequency 2 Nurses Aides Need Training Training of nurse aides  “Training of nurse aides is required to meet up with standards at work and they are not involved enough. They are doing the work and the VHWs are more advanced now than the nurses aides which is a big gap.”  To do more workshops to review all areas and refresher courses especially for junior nurses because when one compares VHWs and health workers, VHWs are more knowledgeable especially in malaria treatment  “Include nurse aides in some of the trainings.”  Nurse aides do not possess adequate knowledge about MCHIP expectations as compared to VHWs  Train staff in all areas and nurse aides must get the same training VHWs has obtained 44  If all service providers that include sister in charge, nurse aids and general hand worked receive the same training, the system will yield excellent results. We must have workshops to help service providers understand the paradigm shift in delivering services. We must have enough resources for the approach to be effective.  Train more personnel at all levels, i.e. nurse aids, general hand and sister in charge Frequency 8 Improved Communication Improved Communication  “There is improved communication between community and the hospital with the assistance of the VHWs.”  Communication has improved! One Stop Shop / Integrated Care One Stop Shop/Integrated Care  “IMNCI has been a big change. We now assess and check the baby and mother at the same time. It is integrated.”  Nurses gained skills and ability to deal with problems of both the mother and baby holistically  MCHIP improved my skills in ANC and we are implementing with mothers and children and everyone  “HIV positive mothers and children are now being identified and treated here and tubes are supplied for blood collection.”  IMCI- babies are managed holistically i.e focusing on the baby and mother during treatment.  IMNCI also had a big impact. We follow mothers and babies holistically now during ANC, delivery and PP and follow up mothers with special needs. We know the danger signs now, do birth plans, ANC-it is a complete management. Through IMNCI we learned a lot about the complaints of mothers and the child-we now join mothers and child together.  “We are a one stop shop now”  “With MCHIP we are using a “holistic approach” and assessing mother and baby together at each visit and it is reducing the workload.”  “We care for the needs of the parent as well as the needs of the child at each visit.” SBMR SBMR  Assisted in infection control e.g. area 10 of SBM-R through a golden rule that of washing hands before treating patients, when you follow it, it reduces a lot of problems which were going to occur if not followed  One fails to meet standards if not supplied with resources and proper 45 infrastructure  Scoring system is unfair and sometimes demotivating hence needs to be improved  As MCHIP started, ‘SBM-R has been an animal and attitude of nurses was negative towards it.’  SBM-R enhanced quality service  The SBM-R approach has been found useful as it provides continuous assessment and guidelines on work expectations  It also provides for clear supportive supervision between the supervisor and supervisee  SBM-R enhances efficiency in the execution of duties  Quality of patients care improved immensely  Provided an opportunity to identify areas of weaknesses and therefore work to improve on them  Nurses became more conscious and knowledgeable in job requirements and procedures  It provided more supervision e.g. review of work procedures and expectations and tutorials on quality are done every Wednesday in all departments Self-Assessments Self￾Assessments  “We have to plan now and we go back to our notes and do self-assessment and it makes us comfortable and happy!”  We regularly carry out self-assessments now and have a register where we record gaps and then go back and look at the outcomes to see if we are maintaining  We now assess each other daily! We sit down regularly and review the standards and discuss cases together and also strategies to improve. Challenges Challenges  Lack of transport to visit needy places e.g. Dembeza which is 20km from Biriri  Resistance from some religious sects to implement programmes  Lack of knowledge by nurse aides who ironically directly work with nurses  Shortage of space particularly consultation rooms. As a result, it takes time to do consultation with patients  Water problems at the institution since the engine to pump water outdated and frequently break down  Financial difficulties to provide basic items such as furniture, food  Shortage of nursing personnel  Limitations of time to perform all necessary tasks  Requires adequate staff to make the changes required by SBM-R  Existed challenges have not been resolved by MCHIP  Shortage of manpower 46  Insufficient training  Inadequate materials i.e. drugs, instruments, gloves, disinfectants for IP, wheelchairs  Nothing is currently being done by MCHIP to overcome the above barriers.  We often don’t have zinc or antibiotics  We don’t have doplers or sonograms so often have to transfer patients because we can’t diagnose the complication  Inadequate materials e.g. scissors, towels, dissecting instruments remain a handicap to proper discharge of duties.  There is also a limitation in trained human resources thereby compromising quality delivery and this became serious when government frozen some post. As a result workload was increased e.g. there might be one nurse on duty doing a lot of tasks at once like issuing drugs to patients, and feeding babies. Some instruments are old dated and makes work more difficult e.g. stitch holding forceps. Some who were not trained are still using the old systems which are somehow risky hence need to train all people that are involved in child delivery services.  No enough infrastructure to cater for all interventions in the program  Not everyone was trained by MCHIP to implement the practices  VHWs are more knowledgeable than the nurses on certain things such as the management of malaria and that is a problem Personnel Issues Personnel issues  Shortage of nursing personnel  Limitations of time to perform all necessary tasks  Requires adequate staff to make the changes required by SBM-R  Health personnel (nurses) need incentives  Shortage of manpower  Insufficient training  There is also a limitation in trained human resources thereby compromising quality delivery and this became serious when government frozen some post. As a result workload was increased e.g there might be one nurse on duty doing a lot of tasks at once like issuing drugs to patients, and feeding babies.  Not all personnel were introduced to the tools.  Too many programmes for one person at once which may affect the results at work.  Those who didn't go for trainings are against receiving a feedback only which is like a summary of things taught. They therefore require first hand information from the source.  Only few people were trained in the SBM-R approach.  Train more personnel at all levels, i.e. nurse aids, general hand and sister in charge.  If there is improvement in nurse-patient ratio, quality will also improve  The labor ward is understaffed and we have only two beds for patients in labor e.g. only two were available 47 Least Important Interventions Least Important Interventions  Activity they felt had least impact was the Reach Every Child Programme because the personnel could not access some places like Dembeza area which is 20km from Biriri Hospital. This was as a result of unavailability of transport  There were no activities that could be pointed out as having least impact on MNCH care and outcome  ANC, IMCI, KMC, PNC, labor and delivery had greatest impact. They argued that all interventions have yielded greatest impact as everything worked very well Recommendations Recommendati ons  Train religious sect members so that they train others in their sect  Recognition must be taken into consideration and must be taken down to the villages to encourage other villages to follow suit  “Training of nurse aides is required to meet up with standards at work and they are not involved enough. They are doing the work and the VHWs are more advanced now than the nurses aides which is a big gap.”  Provision of running water for 24hrs at the center needed  Sensitization of community about MCHIP activities should increase so that they are actively involved in a positive way  More involvement of nurses in training before SBM-R is embarked on  Improved scoring system on the SMB-R  MCHIP should be extended to other provinces and districts because the other health facilities are far behind  MCHIP should remain in place  Health personnel (nurses) need incentives  “We need some sort of recognition!”  SBM-R should be introduced in other provinces and districts to provide uniformity in the provision of quality work output  Training on the implementation of the instrument should cater for all health service providers  “SBM-R should be scaled up to other clinics so we all sing from the same hymn book.”  Train more personnel at all levels, i.e. nurse aids, general hand and sister in charge  If there is improvement in nurse-patient ratio, quality will also improve  If we get more BP machines adding to one that we have, there will be reduction in time spent on one patient.  The approach has to start at district level so that when there are referrals, it will be continuation of the process than starting from zero  Award incentives to best achievers e.g. monetary  More provision of resources both human and material for use at work  Community involvement needs to be enhanced. A bottom to top approach could be used to introduce some of the MCHIP activities that directly have 48 an effect on the community  An improvement on SBM-R scoring is called for so it isn’t so demoralizing  There is need for recognition of high performers to and motivate them accordingly through incentives Poor Infrastructure Poor Infrastructure  Infrastructure is a challenge. We lack space to implement what we were taught and patients have to wait too long  Inadequate materials i.e. drugs, instruments, gloves, disinfectants for IP, wheelchairs  If we get more BP machines adding to one that we have, there will be reduction in time spent on one patient.  Small kangaroo care room with inadequate facilities e.g. toilets and bathrooms  Shortages of resources e.g. there is one functional BP machine, no sonograms and they refer to Mutambara Mission RED RED  Activity they felt had least impact was the Reach Every Child Programme because the personnel could not access some places like Dembeza area which is 20km from Biriri Hospital. This was as a result of unavailability of transport  Before RED we didn’t know how to make site visits and now we plan with the community to increase immunization coverage and it has gone up and we now know how to order sufficient vaccines and syringes Identify Gaps Identify Gaps  Sister In charge now knows the welfare of the staff; they have expanded their supervision and identified gaps and plan for trainings to improve their knowledge and services  “MCHIP helped us identify our needs to improve  Through MCHIP we identified lots of gaps. People had to stand waiting for services  Continuous assessments are helping as we are identifying gaps and rectifying them Community Participation Community participation and issues  Provided with some scales, timers for VHW to use in the community, scales  “VHWs and the Village Health Committee now discuss health at the village level with people getting the services. The community now works together.”  “Our communities really appreciate MCHIP services and how the VHWs extended the services!”  Sensitization of community about MCHIP activities should increase so that they are actively involved in a positive way  “The community itself appreciates the services and the Community Quality Committee does exit interviews with clients!”  “Because of MCHIP we know how to work with the community and now we 49 know we can ask the community for things and that has helped remove the financial barrier. The community has bought mattresses.”  Community involvement needs to be enhanced. A bottom to top approach could be used to introduce some of the MCHIP activities that directly have an effect on the community Other Organizations Other Organizations  CARITAS provides HBC  UMCOR assisted with mosquito nets and mosquito repellent creams  EGPAF assisted in PMCTC programs  Tsuro DzeChimanimani ensures proper nutrition among the patients/sick  Other organizations that have been contributing to significant changes in the provision of sound health services include EGPAF and PLAN International  EGPAF, Plan and TB Group also working with staff  EGPAF offered them PMCTC Religious Sects Religious Sects  Resistance from some religious sects to implement programmes  Train religious sect members so that they train others in their sects Transport Challenges Transport Challenges  Lack of transport to visit needy places e.g Dembeza which is 20km from Biriiri 50 ANNEX V: SOURCES OF INFORMATION A. Key Informant Interviews Completed (23) Name Title Organization HARARE Dr. Jo Keatinge Health Officer USAID Dr. Christina Rawdon Country Director White Ribbon Alliance Dr. Edwin Mpeta Reproductive Health Manager UNFPA Arjanne Rietsema Country Director CORDAID Dr. Assaye Kassie Chief of Health UNICEF Dr. Vonai Teveredzi Program Director ARK Brian Hunter Country Director Save the Children Trevor Kanyowa National Program Officer WHO Zvidzai Chidhakwa Program Support Manager PLAN Paolo Borduogni Country Director EU Dr. Madzima Director, Family Health MOHCW Dr. Chiware Director, Quality Assurance MOHCW Margaret Nyandoro Reproductive Health MOHCW Regina Gerede Deputy Director Community Health Nursing MOHCW Frances Tain Former Deputy Director MCHIP Frank Chikata M&E MCHIP MUTARE Judith *** *** CORDAID Viola Kamuti Senior Nursing Officer Mutare Provincial Hospital (MPH) Sister Chimbetete Acting DNO Mutare Dr. Maphosa DMO Mutare Dr. Siamuchembu Acting PMD Manicaland Mrs. Mukotekwa Provincial Manager Manicaland Evelyne Muvirimi Newborn Health Coordinator MCHIP B. Focus Group Discussions with Village Health Workers (VHWs) Completed 76 VHWs (65 Female and 11 Male) Facility Name Number of VHWs Female Male Burma Valley Health Center 3 2 1 Rusitu Mission Hospital 15 14 1 Chakohwa Rural Health Center 6 6 Odzi Rural Hospital 10 6 4 Mutambara Mission Hospital 14 12 2 Biriri Rural Hospital 13 13 St. Andrews Mission Hospital 15 12 3 51 C. Focus Group Discussions with Health Workers Completed 9 FGDs (37 HWs, 35 females and 4 males) Facility Name Number of HWs Female Male Mutare Provincial Hospital 7 7 Rusitu Mission Hospital 4 2 2 Chakohwa Rural Health Center 5 3 2 Dangamvura Polyclinic 6 6 Mutambara Mission Hospital 6 6 Biriri Rural Hospital 5 5 St. Andrews Mission Hospital 4 4 D. Documents and Publications Consulted MOHCW. 2013. Kangaroo Mother Care Method – Facilitator Manual, Zimbabwe. MOHCW. 2013. Quality Assurance and Quality Improvement Policy. Ministry of Health and Child Care, Zimbabwe. 10 June 2013. MOHCW. 2013. Malaria Case Management in the Community: Facilitator’s manual for the training of Community-Based Health Workers. With support from USAID/MCHIP. MOHCW. 2013. Malaria Case Management in the Community: Participant’s manual for the training of Community-Based Health Workers. With support from USAID/MCHIP. MOHCW. 2013. Draft Quality Assurance and Quality Improvement Policy. Ministry of Health and Child Care, Zimbabwe. April 2013. MOHCW. 2013. Assessment of Health Data Quality in Manicaland Province. Commissioned by MOHCW with support from USAID/MCHIP, February 2013. MOHCW. 2013. Standards Based Management and Recognition – Improving quality of care for maternal & neonatal health in Mutare and Chimanimani districts, Manicaland, Zimbabwe, 2010-2013. Supported by USAID / MCHIP. MOHCW. 2013. Circles of Support for breastfeeding mothers – World Breastfeeding Week 2013. Pamphlet produced by the MOHCW Health Promotion Unit and Nutrition Unit with collaboration with WHO, EGPAF, Save the Children, FAO, UNICEF, World Vision and support from USAID/MCHIP MOHCW. 2012. Assessment of Health Worker Knowledge, Attitudes, and Beliefs towards Breastfeeding, 2008. Supported by USAID / MCHIP, UNICEF & Zvitambo. MOHCW. 2012. Final report – Zimbabwe National Integrated Health Facility Assessment, December 2011- January 2012. MOHCW. 2012. Child Health Needs Assessment – Findings from a baseline assessment in Mutare and Chimanimani Districts, January 2012. Supported by USAID / MCHIP. 52 MOHCW. 2012. Registers for community maternal newborn and child health. This document contains registers for use by community health workers providing maternal, newborn, and child health services at the community level. The registers are: Form C1: for use during home visits for ANC and PNC; Form C2: for use when treating a sick child; Form C5: for use as a summary sheet for consolidating key indicators and for reporting. MOHCW, Reproductive Health Unit. 2012. Basic Emergency Obstetric and Newborn care – Facilitator’s Manual. Supported by MCHIP, UNICEF, UNFPA and WHO. MOHCW, Reproductive Health Unit. 2012. Basic Emergency Obstetric and Newborn care – Participant’s Manual. Supported by MCHIP, UNICEF, UNFPA and WHO. MOHCW. 2011. Integrated Management of Neonatal and Childhood Illnesses (IMNCI)- Chart booklet. September 2011. Supported by UNICEF, LATH, MCHIP and WHO. MOHCW. 2010. Report on the head count exercise for children below the age of One year in Chimanimani district. Supported by USAID / MCHIP. MOHCW. Dzivirira mwana wako kuchirwere chmabayo. (Shona pamphlet – Protect your child from pneumonia). Produced by the MOHCW Health Promotion Unit in collaboration with MCHIP. MOHCW. Imi nepamuviri penyu. Zvamunofanira kuziva. (Shona pamphlet – What you need to know about your pregnancy). Produced in collaboration with UNICEF, WHO, UNFPA and with funding from USAID/MCHIP. MOHCW. IMNCI Assessment, Classification and Treatment Tool (T12) for managing sick children aged 0-2 months. (register) MOHCW. IMNCI Assessment, Classification and Treatment Tool (T12) for managing sick children aged 2 months – 5 years. (register) MOHCW. Improving the quality of maternal, Newborn and Child care in Zimbabwe: Standards-based Management and Recognition (SBM-R) – Standards for MNCH Services. With support from USAID/MCHIP. (CD) MOHCW. Ita Kangaroo Care: Kumwana abarwa asati asvika (gavamwedzi). (Shona pamphlet – KMC for pre-term babies). Produced in collaboration with USAID/MCHIP. MOHCW. Kangaroo Care Unit: Patient admission register. MOHCW. Kangaroo Care Unit: Patient follow-up register. MOHCW. Kangaroo Mother Care Method: Participant handbook, Zimbabwe. (CD) MOHCW. Maternal and Neonatal Health Record. MOHCW. Okumele ubekwazi ngomukuhlane we malaria nxa ungumama ozithweleyo. (Ndebele pamphlet on what you should know about malaria in pregnancy, produced in conjunction with UNDP and printed with support from USAID/MCHIP. 53 MOHCW. Protect your child from pneumonia. English pamphlet produced by the MOHCW Health Promotion Unit in collaboration with MCHIP. MOHCW. What you need to know about High Blood Pressure in Pregnancy (Pregnancy Induced Hypertension). English pamphlet produced by the MOHCW Health Promotion Unit and Reproductive Unit with support from USAID/MCHIP. MOHCW. What you need to know about Malaria. English pamphlet produced by USAID/MCHIP. MOHCW. What you need to know about Measles. English pamphlet produced by the MOHCW Health Promotion Unit in collaboration with MCHIP. MOHCW. What you should know about malaria in pregnancy. English pamphlet produced with support from USAID/MCHIP. MOHCW. Zvamunofanira kuziva nezvemanyoka muvana vadiki vari pasi pemakore mashanu (Shona pamphlet – What you need to know about infant diarrhoea in under 5s). Produced in collaboration with MCHIP. MOHCW. Zvamunofanira kuziva pamusoro pechirwre cheMalaria kana muchinge makazvitakura (Shona pamphlet – What you need to know about malaria I pregnancy). Produced in collaboration with the MOHCW National Malaria Control Program and printed with support from MCHIP. MOHCW. Zvamunofarira kuziva pamusoro pechirwere cheBhiipii (BP) kana muchinge makazvitakura. Shona pamphlet on what you need to know about High Blood Pressure in Pregnancy, produced by the MOHCW Health Promotion Unit and Reproductive Unit with support from USAID/MCHIP. MOHCW. Zvamunofanira kuziva pamusoro pegwirikwiti (Shona pamphlet – What you need to know about measles). Produced in collaboration with the MOHCW Health Promotion Unit in collaboration with MCHIP. USAID/MCHIP. 2012. Maternal and Child Health Integrated Program: Implementation Plan – October 2012- September 2013. Submitted to United States Agency for International Development by Jhpiego in collaboration with John Snow Inc., Save the Children, Macro International Inc., PATH, Institute of International Programs/Johns Hopkins University, Broad Branch Associates, Population Services International. USAID/MCHIP Zimbabwe. 2012. FY12 Quarterly Reports – October 1, 2011-December 31, 2011. Submitted to the USAID/Zimbabwe Mission by Jhpiego in collaboration with John Snow Inc., Save the Children, Macro International Inc., PATH, Institute of International Programs/Johns Hopkins University, Broad Branch Associates, Population Services International. USAID/MCHIP Zimbabwe. 2011. FY11 Quarterly Reports – October 1, 2010-December 31, 2010. Submitted to the USAID/Zimbabwe Mission by Jhpiego in collaboration with John Snow Inc., Save the Children, Macro International Inc., PATH, Institute of International Programs/Johns Hopkins University, Broad Branch Associates, Population Services International. January 31, 2011. USAID/MCHIP Zimbabwe. 2011. Maternal and Child Health Integrated Program, Zimbabwe. FY11 Workplan – October 2010-September 2011. Revised and submitted January 8, 2010 to the United States Agency for International Development by Jhpiego in collaboration with John Snow Inc., Save the Children, Macro 54 International Inc., PATH, Institute of International Programs/Johns Hopkins University, Broad Branch Associates, Population Services International. January 31, 2011 USAID/MCHIP. 2011. Maternal and Child Health Integrated Program: Implementation Plan – October 2011- September 2012. Submitted to United States Agency for International Development by Jhpiego in collaboration with John Snow Inc., Save the Children, Macro International Inc., PATH, Institute of International Programs/Johns Hopkins University, Broad Branch Associates, Population Services International. USAID/MCHIP. 2011. Maternal and Child Health Integrated Program, Zimbabwe. Technical brief: Malaria case management, 2011-2012. Submitted to the USAid/Zimbabwe Mission by Jhpiego in collaboration with John Snow Inc., Save the Children, Macro International Inc., PATH, Institute of International Programs/Johns Hopkins University, Broad Branch Associates, Population Services International. USAID/MCHIP. 2010. Maternal and Child Health Integrated Program, Zimbabwe: Proposal and Workplan – Detailed plan for first 18 months April 1, 2010-September 2013. Revised and submitted 9 June 2010 to United States Agency for International Development by Jhpiego in collaboration with John Snow Inc., Save the Children, Macro International Inc., PATH, Institute of International Programs/Johns Hopkins University, Broad Branch Associates, Population Services International. ZEPI/MOHCW. 2012. Implementing the Reaching Every District (RED) Approach – Field Guide. With technical support by UNICEF, USAID/MCHIP and WHO. 55 E. Document Review Matrix Evaluation Question Desk Review Findings Gaps to Fill from Field Work How did MCHIP contribute to overall learning and innovation in MCHN care in Zimbabwe? What innovative processes and products did MCHIP support or implement?  FY11: MCHIP support to the national immunization program saw the successful revitalization of the RED strategy and roll out in all 7 districts of Manicaland. MCHIP provided technical and material support for the revitalization and refurbishment of 2 MNCH training units and 4 kangaroo mother care units in district hospitals.  FY12: MCHIP continues to provide support to the MOHCW through various training of trainers (TOTs) activities.  FY12: MCHIP reached a breakthrough with its community MNCH activities in launching its community MNCH Performance Quality Improvement (PQI) activities with village health workers. Both Provincial and District-level Health Executives were sensitized on MCHIP’s cPQI approach. This formed the foundation for a VHW baseline assessment, in which both intervention and control sites were assessed in FY12 and FY13.  FY13: MCHIP captured synergies with international commemorations such as World Malaria day by distributing IEC materials and launching the “ZamZim” malaria cross-border initiative.  What constitutes “innovative” in the context of MNCH in Zimbabwe?  What were MCHIP’s contributions in areas that previous interventions failed to address? What factors may affect the feasibility of scaling up these innovations and how?  Challenges identified by stakeholders include limited blood supply and lack of availability of certain supplies. Also, low knowledge among nurses on the use of partographs was reported during FY12.  FY13: MCHIP does not currently routinely report on facility-based child mortality due to difficulties with collecting this data reliably from the routine HMIS.  Perceptions of scale-up feasibility among project staff, key stakeholders  Perspective of other similar interventions and potential effects/consequences of MCHIP scale-up on national and provincial levels  Administrative capacity to achieve successful scale-up What was the nature of relations between MCHIP and key MNCH stakeholders and how did the relations contribute to the achievement of results?  FY11: MCHIP secured a seat on the main national technical working groups, allowing for the setting of priorities for national programs and the development of national implementation plans. MCHIP also provided tech support to the MOHCW to develop/submit proposals to GAVI for the introduction of two vaccines. Through an MCHIP-supported multi-stakeholder workshop, the MOHCW Reproductive Health Unit led the process of adapting performance standards.  Stakeholder perceptions of MCHIP contributions to MNCH learning in Zimbabwe  The extent to which MNCH results can be attributed to MCHIP alone and what is due to confluence of stakeholder interests 56  FY12: MCHIP provides support to the Zimbabwe National Family Planning Council (ZNFPC), and held a national stakeholders’ meeting providing a forum for family planning discussions and the opportunity for MCHIP to advocate for strengthened efforts around post-partum family planning. Results from this meeting guide the design of ZNFPC’s strategy moving forward.  FY12: MCHIP’s support for the MOHCW’s Quality Assurance Unit to raise the profile of QA/QI concepts nationally have contributed to the development of a national QA/QI strategy and formal policy.  MCHIP supports provincial-level planning and review by participating in the Provincial Health Team (PHT) meeting, which presents an opportunity for stakeholders to gather to review past progress and make future plans.  FY13: MCHIP provides continued support to the EPI Unit through updating the IIP training module, with the draft training guide used to facilitate TOTs. Support has also been sustained at national level MNCH coordination mechanisms, such as various technical working groups.  MCHIP’s partnership with PSI allowed for consultation on excess reusable medical supplies/equipment from the male circumcision program and the procurement of additional supplies. What challenges or barriers to achievement of results did MCHIP experience in Zimbabwe?  FY11: Baseline SBM-R surveys showed major skills gaps among health care providers, which will be addressed through additional training activities.  Additional challenges, barriers as perceived by project stakeholders How effective is the SBM-R approach in improving MNCH care in MCHIP supported nineteen health facilities in Zimbabwe? What proportion of MCHIP supported facilities is achieving a minimum set of MNCH care standards?  100% of HFs reaching at least 80% of MNH standards (n=17 HFs in 2 learning districts)  70% of HFs reaching at least 60% of CH standards (n=21 HFs in 2 learning districts)  Data validation How are changes in standards of care influencing health outcomes, e.g. in early newborn mortality, maternal mortality, obstetric and newborn complications?  Number of facility-based early neonatal and intrapartum deaths per 1000 live births from October 2010 to June 2013 indicates an overall downward trend in the 17 SBM-R supported facilities over time.  Reported from Q1, most MCHIP-supported SBM-R facilities have shown marked improvements between baseline and most recent assessment in meeting performance standards for managing newborn complications.  The number of facility-based child deaths remained relatively consistent between this quarter and previous quarters, with a slight sharp increase in fatalities in May. The table below shows the breakdown of deaths by cause for each month from the 22 SBM-R sites. During Q3, 71 deaths were reported from the 22 SBM- R supported sites in total. Of these, 11 were due to pneumonia, 10 were as result of kwashiorkor, 6 were due to diarrhea, and the rest were due to other causes. Despite having a malaria outbreak in  How attributable are observed outcomes to MCHIP inputs alone? 57 Manicaland during the quarter, only four deaths were a result of malaria, implying that malaria case management may be improving. A discussion held with Pediatricians at MPH to find out the reasons for the relatively high U5 mortality, revealed that the underlying cause of most of the U5 deaths is the late seeking of health care by caregivers. They cited that most of the children admitted to MPH die either at admission or within 24 hours of admission. During an IMNCI inpatient practice session the MCHIP Child Health Coordinator witnessed one case of a child who was aged eighteen months who died a few hours after admission. The child was brought in very late and she/he died of severe pneumonia. There is need to strengthen the screening of sick children by VHWs, also emphasis should be on strengthening the key household practices in the community through the VHWs. How acceptable is the SBM-R approach to service providers, policy makers and other MNCH stakeholders in Zimbabwe?  Perceptions of SBM-R approach among key stakeholders, trainees, HCWs, VHWs  Willingness/efficacy of training uptake Describe and analyze factors contributing to the effectiveness or ineffectiveness of the SBM￾R approach.  FY13: MCHIP does not currently routinely report on facility-based child mortality due to difficulties with collecting this data reliably from the routine HMIS.  Districts are still far below reaching the annual target for the indicator “% low birth weight (LBW) babies admitted into KMC in Mutare and Chimanimani.” There is a decrease in the number of LBW babies receiving KMC in Q3 compared to Q2. The demand in the districts is still high compared to supply and opening of new KMC units in FY13 will reduce pressure on existing KMC units. In addition, it is noted that referrals of LBW babies from facilities without KMC units to facilities with KMC units may be low; there is a need to further strengthen the KMC referral system or increase the number of facilities with KMC units.  The malaria cases dropped by about 50% compared to the previous quarter. In quarter 2, there was a malaria outbreak which affected the province. Pneumococcal Conjugate Vaccine {PCV 13} was introduced 12 months ago in July 2012, but pneumonia remains the main cause of mortality and morbidity in children aged less than 5 years. The need to strengthen pneumonia prevention, promotion and protection intervention as well as early care seeking behaviours targeting under 5s is obvious. The role played by poor nutrition in this community needs to be explored further.  Perceptions of SBM-R approach among key stakeholders  Specific obstacles encountered by MCHIP staff in implementing SBM-R approach What factors may affect the feasibility of scaling up this approach and how can the SBM￾R approach be adopted to increase potential for successful nationwide scale-up?  At times, limited availability of specific resources: financial, technical and material  Perceptions of feasibility of SBM-R approach among key stakeholders 58 ANNEX VI: DISCLOSURE OF ANY CONFLICTS OF INTEREST 59 60 61 62 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523