MATERNAL, NEWBORN, AND CHILD HEALTH (MNCH) PROJECT MID-TERM EVALUATION DECEMBER 27, 2016 This publication was produced for review by the United States Agency for International Development. It was prepared by Jean Capps, Huma Qureshi, Aisha Zafar, and Yasir Waraich of Management Systems International, a Tetra Tech Company. MATERNAL, NEWBORN, AND CHILD HEALTH (MNCH) PROJECT MID-TERM EVALUATION Contracted under Order No. AID-391-C-15-00004 Performance Management Support Contract DISCLAIMER This report is made possible by the support of the American people through the United States Agency for International Development (USAID). The contents are the sole responsibility of the Management Systems International and do not necessarily reflect the views of USAID or the United States Government. MNCH PROJECT MID-TERM EVALUATION i ACKNOWLEDGMENTS The evaluation team wishes to thank the government officials, NGO implementing partners, PPHI, and stakeholders in Sindh Province who generously gave us their time and patiently answered our questions. Without their contributions the evaluation would not have been possible. We are also thankful to the head office and district representatives of PPHI who agreed to be interviewed, even though PPHI had withdrawn from the project. We would also like to thank representatives of private health facilities for contributing their perspectives. Finally, we would like to thank the community groups and women beneficiaries, health workers, and facility managers of the project for their participation in individual and group interviews conducted during the hottest time of the year during the month of Ramadan. MNCH PROJECT MID-TERM EVALUATION ii TABLE OF CONTENTS Acknowledgments.......................................................................................................................... i Acronyms and Abbreviations...................................................................................................... iv Project Summary .......................................................................................................................... 1 Executive Summary...................................................................................................................... 2 Evaluation Purpose and Questions.......................................................................................................................2 Project Background..................................................................................................................................................2 Key Findings and Conclusions...............................................................................................................................2 Changes to Improve Effectiveness........................................................................................................................3 Best Practices, Innovations, and Lessons Learned............................................................................................3 Evaluation Questions and Methods ............................................................................................. 4 Evaluation Purpose and Questions.......................................................................................................................4 Data Collection and Analysis.................................................................................................................................4 Limitations..................................................................................................................................................................5 Project Context............................................................................................................................. 6 Sindh MNCH Health Services Situation..............................................................................................................6 MNCH Project Design ............................................................................................................................................7 Findings and Conclusions.............................................................................................................. 8 Objective 1: Access to Family Planning and MNCH Services........................................................................8 Special Interventions..............................................................................................................................................15 Objective 2: Capacity of Health Workers........................................................................................................17 Objective 3: Referrals............................................................................................................................................19 Overall Project Outcomes...................................................................................................................................21 Project Implementation and Management........................................................................................................24 Conclusions .................................................................................................................................. 26 Question 2: Recommendations.................................................................................................. 28 Question 3: Best Practices and Lessons Learned..................................................................... 29 Annexes........................................................................................................................................ 31 Annex 1: Scope of Work......................................................................................................................................31 Annex 2: Assignment Work Plan........................................................................................................................35 Annex 3: Conflict of Interest Statements.........................................................................................................57 Annex 4: List of Persons Interviewed................................................................................................................58 Annex 5: Bibliography............................................................................................................................................59 Annex 6: Data Collection Instruments..............................................................................................................61 MNCH PROJECT MID-TERM EVALUATION iii LISTS OF TABLES AND FIGURES List of Tables Table 1: Project Summary................................................................................................................................................1 Table 2: Data Collection Summary by Stakeholder and Method...........................................................................5 Table 3: Maternal and Child Health (MCH) Program Components .....................................................................7 Table 4: Percentage of Facilities Providing Seven Signal Functions......................................................................10 Table 5: Change in MNCH Project Indicators, 2014–2016...................................................................................23 Table 6: Implementation Approaches.........................................................................................................................24 List of Figures Figure 1: Trends in Provision of Implants and IUCDs............................................................................................12 Figure 2: Change in QIPS Indicators in Phase A Districts......................................................................................22 Figure 3: Client Satisfaction With Services................................................................................................................22 Figure 4: Quality of ANC and PNC Services............................................................................................................23 MNCH PROJECT MID-TERM EVALUATION iv ACRONYMS AND ABBREVIATIONS ANC Antenatal Care AOR Agreement Officer’s Representative BCC Behavior Change Communication BEmONC Basic Emergency Obstetric and Newborn Care BHU Basic Health Unit CEmONC Comprehensive Emergency Obstetric and Newborn Care CHW Community Health Worker CHX Chlorhexidine CMW Community Midwife COR Contracting Officer’s Representative CSG Community Support Group DHQ District Headquarters DO Development Objective DoH Department of Health EmONC Emergency Obstetric and Newborn Care EPI Expanded Program on Immunization FP Family Planning FP/RH Family Planning and Reproductive Health GRD Government Rural Dispensary HBB Helping Babies Breathe HSS Health Systems Strengthening IEC Information, Education, and Communication IFA Iron/Folic Acid IPC Interpersonal Communication IR Intermediate Result IUCD Intrauterine Contraceptive Device JHUCCP John Hopkins University Center for Communication Program JSI John Snow Incorporated KMC Kangaroo Mother Care LHS Lady Health Supervisor LHV Lady Health Visitor LHW Lady Health Worker LTFP Long-Term Family Planning MNCH PROJECT MID-TERM EVALUATION v LUMHS Liaquat University of Medical and Health Sciences MCH Maternal and Child Health MCHIP Maternal and Child Health Integrated Program Miso Misoprostol MNCH Maternal, Newborn, and Child Health MOU Memorandum of Understanding NGO Nongovernmental Organization OB/GYN Obstetrics and Gynecology OJC On-the-Job Coaching OJT On-the-Job Training OPD Outpatient Department OT Operating Theater PAC Post-Abortion Care PATH Program for Appropriate Technology in Health PCPNC Pregnancy, Childbirth, Postpartum, and Newborn Care PDQ Partnership Defined Quality PMDC Pakistan Medical and Dental Council PMP Performance Management Plan PNC Postnatal Care PPH Postpartum Hemorrhage PPHI People’s Primary Healthcare Initiative PPIUCD Postpartum Intrauterine Contraceptive Device PWD Population Welfare Department QIPS Quality Improvement and Patient Safety QIT Quality Improvement Team QOC Quality of Care RFA Request for Application RHCs Rural Health Centers RSPN Rural Support Program Network SBA Skilled Birth Attendant SOW Scope of Work STTA Short-Term Technical Associate TB Tuberculosis THQ Tehsil Headquarters MNCH PROJECT MID-TERM EVALUATION vi TOT Training of Trainers USAID United States Agency for International Development WSG Women’s Support Group MNCH PROJECT MID-TERM EVALUATION 1 PROJECT SUMMARY Table 1 summarizes basic information about the Maternal and Child Health Integrated Program (MCHIP). 1 TABLE 1: PROJECT SUMMARY Title/Field Project/Activity Information Contract/agreement numbers AID-391-LA-13-00001 Contracting/Agreement Officer’s Representative (COR/AOR) Start date January 2013 Completion date September 2017 Location 16 districts of Sindh Province Implementing partner(s) A consortium of Jhpiego, Save the Children, and John Hopkin University Center for Communication Program (JHUCCP) USAID/Pakistan Mission Strategic Framework objectives addressed DO5: Improved MCH outcomes in target areas IR5.1: Increased access to integrated family planning and maternal and child health services Budget 1 The Maternal and Child Health Integrated Program (MCHIP) is a consortium of three organizations (Jhpiego, Save the Children, and JHUCCP) that implements the USAID/Pakistan Maternal, Newborn, and Child Health (MNCH) Services project. MNCH PROJECT MID-TERM EVALUATION 2 EXECUTIVE SUMMARY Evaluation Purpose and Questions The mid-term performance evaluation of the Maternal, Newborn, and Child Health (MNCH) Services Project examined the project’s effectiveness, with attention to opportunities to improve effectiveness in the remaining years of the project and on identifying innovations, best practices, and high-impact interventions. Recommendations will help USAID/Pakistan and the implementing partner improve effectiveness and impact during the remainder of the project and contribute to future programming in maternal and child health service delivery. The evaluation focused on answering three questions: 1. To what extent has the project been effective in meeting its major goal and objectives? 2. What changes could be made to increase effectiveness in the remaining years of the project? 3. What best practices, innovations, and lessons learned can be applied to existing or future MNCH service delivery programs? Project Background The MNCH project worked closely with the Sindh Department of Health (DoH) and the People’s Primary Healthcare Initiative (PPHI), the public-private partnership that manages primary healthcare facilities for the DoH in Sindh. It worked with DoH and PPHI facilities in 16 districts to 1) improve access to integrated family planning (FP) and MNCH services; 2) increase the capacity of health workers to deliver high quality FP and MNCH services; and 3) improve the referral network. Activities designed to address the first objective focused on strengthening health-related community mobilization structures and healthcare facilities. The project emphasized training to achieve the second objective of increasing the capacity of workers and trained transporters and established a referral system to address the third objective. Key Findings and Conclusions Overall, the implementing partners employed their technical expertise to introduce high-quality and effective interventions that substantially increased the capacities of skilled birth attendants (SBAs) and substantially increased access to MNCH services in rural areas. However, the project’s limited use of the vast amount of data it collected to inform decision-making represents a missed opportunity to improve performance. In 2016, Jhpiego transitioned interventions in PPHI-managed facilities to PPHI, and as a result, 442 health facilities were no longer involved the project. The project’s activities to increase the capacity of health workers were particularly effective, and the facilities and healthcare providers the project worked with overwhelmingly identified training as the most effective project activity. On-the-job coaching (OJC) and training (OJT) and facilitating community midwives’ (CMWs’) access to hands-on experience and mentoring were especially effective interventions. The fact that PPHI liked the training approach so well that it has adopted it in all of its facilities is a good indicator of its effectiveness and a key project achievement, especially in the context of sustainability. Activities geared to improving access to FP and MNCH services were particularly effective in training lady health workers (LHWs) and revitalizing the women’s support groups (WSGs) through which the LHWs work. This provided a platform for the project to distribute two evidence-based treatments MNCH PROJECT MID-TERM EVALUATION 3 (chlorhexidine [CHX] for umbilical cord care and misoprostol [miso] to prevent postpartum hemorrhage) proven to reduce neonatal and maternal mortality, respectively. Building the capacities of CMWs to improve clinics in rural areas was also a key intervention for improving access to services in remote, rural areas. Infection prevention techniques and helping babies breathe (HBB) were also high￾impact practices that the project successfully promoted. The project’s efforts to train transporters and establish a referral network have met with mixed success. The trained transporters have gained new skills in handling clients and they are confident that they can continue providing the service. However, clients note that the cost of the transporters is higher than other available alternatives, especially in urban and peri-urban areas. Neither the clients nor the transporters were using the referral slips the project needs to track referrals. Changes to Improve Effectiveness Some of the more important changes that could improve project effectiveness include:  Train training institute staff as master trainers to build facility capacity and promote sustainability.  Advocate with the DoH to find the resources to hire additional staff at basic emergency obstetric and newborn care (BEmONC) and comprehensive emergency obstetric and newborn care (CEmONC) facilities so they are capable of offering the full range of required services around the clock.  Work with the LHW program to scale up distribution of CHX/miso throughout Sindh. The distribution model leverages the reputation of the LHW program and its deep connections in rural communities to distribute an important treatment in a cost-effective manner.  HBB has undoubtedly resulted in fewer newborn deaths. In its remaining years, the project should scale up HBB within the 15 project districts. Future projects could scale it up throughout Sindh. Best Practices, Innovations, and Lessons Learned 1. The project demonstrated how targeted assistance and relatively simple interventions (e.g., providing equipment and supplies or facilitating a mobility allowance for vaccinators) can contribute significantly to improving access to and the quality of MNCH services. 2. Engaging with the established and trusted LHW program with its strong connections in rural communities was a best practice and helped the program distribute new, lifesaving treatments broadly within its target communities. 3. Assisting CMWs to establish and develop their own clinics is another best practice. The project has learned valuable lessons in helping CMWs become effective SBAs and MNCH service providers who are currently assisting the national MNCH program and could continue in the future. 4. Placing CMW coordinators in the training hospitals to facilitate training and mentoring was an important intervention that substantially improved the effectiveness of the CMW schools in building the skills of SBAs. MNCH PROJECT MID-TERM EVALUATION 4 5. The project’s emphasis on quality of care and introducing a culture of quality is a best practice and was appreciated by health providers. Assessing service quality, potentially using the quality improvement and patient safety (QIPS) assessment tool, can be a valuable tool to monitor quality of care and deliver targeted refresher training or coaching to address deficiencies. EVALUATION QUESTIONS AND METHODS Evaluation Purpose and Questions The mid-term performance evaluation of the Maternal, Newborn, and Child Health (MNCH) Services Project examined the project’s effectiveness from February 2013 through March 31, 2016, with attention to identifying innovations, best practices, high-impact interventions, and activities that are contributing to achieving results. The evaluation provides lessons learned and recommendations that can help USAID/Pakistan and Jhpiego improve effectiveness and impact during the remainder of the project and contribute to future programming in maternal and child health service delivery. The evaluation scope of work (SOW) (Annex 1) specified three key questions: 1. To what extent has the project been effective in meeting its major goal and objectives? 2. What changes could be made to increase effectiveness in the remaining years of the project? 3. What best practices, innovations, and lessons learned can be applied to existing or future MNCH service delivery programs? Data Collection and Analysis The evaluation team used a mixed methods approach employing qualitative and quantitative data collection techniques from primary and secondary sources. The evaluation team participated in a planning workshop to design a systematic and rigorous evaluation approach. The team met with USAID/Pakistan representatives, project staff, and other stakeholders to gain a thorough understanding of project objectives, implementation mechanisms, evaluation purpose, and context. The project rolled out implementation in phases starting with five (phase A) districts in 2013. It added the remaining ten districts (two phase B and two phase C) in 2015. To examine differences in outcomes by implementation phase, the evaluation purposively selected two districts from phase A (because these districts implemented pilot interventions and had the longest exposure to interventions introduced by the project), one district from phase B, and one from phase C. The team then randomly selected a cross section of 16 public and private health facilities, and associated community structures, for field visits. The team collected data from 185 individual respondents and 32 group interview participants from the 16 selected health facilities in the 4 districts (Table 2). Secondary data came from available project performance data, progress reports, research studies, and other sources. (Annex 5 contains a complete list of documents the evaluation team reviewed.) MNCH PROJECT MID-TERM EVALUATION 5 TABLE 2: DATA COLLECTION SUMMARY BY STAKEHOLDER AND METHOD Data Source Data Collection Method Total by Data Source Client Exit Interviews Group Interview Participants In-depth Interviews Observation Community/WSGs 37 22 17 - 76 Government - - 15 - 15 Healthcare providers - - 38 6 44 Health facility - - - 10 10 MCHIP consortium - - 2 - 2 MCHIP implementing partners - 0 6 - 6 MCHIP team - 0 40 - 40 PPHI - 4 - 4 QITs - 10 - - 10 Training institutes - - 6 - 6 USAID - - 2 - 2 USAID implementing partners - - 1 - 1 Working groups / meetings - - 1 0 1 Total by data collection method 37 32 132 16 217 The evaluation employed rigorous methods to analyze qualitative and quantitative data. Quantitative data provided evidence of what happened while qualitative data provided evidence of how and why the project functioned and produced, or failed to produce, results and whether it generated unanticipated results. The evaluation used thematic analysis and coding techniques to identify key themes in the qualitative data and used quantitative analysis (e.g., descriptive statistics and cross-tabulation) methods to identify patterns in the qualitative data and to explore relationships between the quantitative and qualitative findings. Limitations The evaluation methods facilitated a multi-tiered triangulation process, including both primary and secondary data sources, to ensure a reliable final set of findings and corresponding conclusions and recommendations. However, the practical considerations of conducting an evaluation introduced some limitations.  Due to issues beyond the evaluation team’s control, it was not possible to visit the People’s Primary Healthcare Initiative (PPHI) facilities. This substantially limited the sample universe for field data collection since PPHI manages all of the project-supported Basic Health Units (BHUs) which comprised 48 percent of the facilities with which the project worked. Additionally, MNCH PROJECT MID-TERM EVALUATION 6 community structures and health workers such as women’s support groups (WSGs), quality improvement teams (QITs), and lady health workers (LHWs) are based at the BHUs. To mitigate this limitation, to the extent possible, the evaluation team visited more non-PPHI facilities than planned. The evaluation team also interviewed staff at the PPHI head office in Karachi and service providers and district managers in sampled districts to get their perspective. Due to this limitation, the evaluation findings are not generalizable to all cadres of project￾supported facilities.  The project modified its implementation approach several times without updating its performance management plan (PMP). The lack of updated targets and inconsistent data on indicators limited the evaluation team’s ability to assess progress against targets to those indicators for which updated targets and consistent performance data existed.  The evaluation intended to rely on periodic population-based randomized household surveys to measure some results indicators. It conducted the baseline survey in 2013 and a follow-up survey in 2014.The summary report focused on selected indicators and showed no statistically significant changes. Changes in government registration requirements for field data collection prevented the project from conducting the scheduled survey in 2015 which was not done at the time of the evaluation. The evaluation team assessed project monitoring data but it had not been aggregated at the project level. The evaluators were unable to quantitatively estimate trends in MCHIP performance indicators from the year 1 PMP. PROJECT CONTEXT Sindh MNCH Health Services Situation Healthcare services in Sindh are provided by the Department of Health (DoH) and a variety of private providers. The DoH is the main provider of preventive care throughout the province and the major provider of curative services in most rural areas. In the public sector, health services are provided through a tiered referral system of healthcare facilities, with increasing levels of complexity and coverage from primary, to secondary, to tertiary health facilities. Primary care facilities include BHUs, rural health centers (RHCs), government rural dispensaries (GRDs), maternal and child health (MCH) centers, and tuberculosis (TB) centers. These facilities provide outpatient department (OPD) services eight hours a day/six days a week for preventive and a limited number of curative services, although RHCs provide a broader range of curative services, 24 hours a day, 7 days per week. Primary care facilities also provide outreach preventive services to communities through vaccinators, sanitary inspectors, and a sanitary patrol. At the community level, services are provided through facility￾based outreach health workers, lady health workers (LHWs), and community midwives (CMWs). The LHW program is the flagship program of the Department of Health (DoH) for community interventions but covers only 20 to 43 percent of the communities in certain districts, and technical knowledge and supervision overall is weak.2 Tehsil headquarters (THQ) hospitals and 17 district headquarters (DHQ) hospitals comprise the secondary tier and provide increasingly specialized secondary healthcare, while teaching hospitals form the tertiary level tier. Health indicators in the rural areas of Sindh are particularly poor, falling below the average for rural Pakistan. Coverage of maternal and child health services, contraception, vaccination, and communicable disease control is uneven due to poorly functioning basic and emergency services. Only 27 percent of 2 Sindh Health Sector Strategy, 2012-2020. Accessed from: http://www.trfpakistan.org/LinkClick.aspx?fileticket=1EyZSVfIMkg%3D&tabid=2618 MNCH PROJECT MID-TERM EVALUATION 7 deliveries take place in health facilities, merely 70 percent of children under one year of age are immunized for measles, and 11 percent of childbearing age couples practice contraception. Only 22 percent of Sindh’s population choose to use public health facilities, opting instead to visit private facilities. This is lower than the national average of 29 percent and reflects the relatively poor quality of public health services. 3 MNCH Project Design The MNCH project is one component of USAID/Pakistan’s integrated five-component maternal and child health (MCH) program (Table 3). Each project in the portfolio is responsible for specific activities and deliverables, but all partners are supposed to work in close coordination to achieve the same goal: improved health outcomes for newborns, children, and women. The MNCH project is implemented by a consortium of three organizations that includes Jhpiego, Save the Children, and JHUCCP. This consortium is commonly known as the Maternal and Child Health Integrated Program (MCHIP), but this report refers to the project as the MNCH project. TABLE 3: MATERNAL AND CHILD HEALTH (MCH) PROGRAM COMPONENTS Component Description Implementing Partners 1: Family Planning and Reproductive Health (FP/RH) Delivers FP/RH services and improves the quality of care in the public and private health sectors. Marie Stopes Society; Health and Nutrition Development Society; Creative Social Marketing; Marie Stopes International 2: Maternal Newborn Child Health (MNCH) Increases access to high-quality MNCH. Jhpiego; Save the Children; JHUCCP 3: Behavior Change Communication (BCC) Employs commercial marketing techniques and behavior change communications to promote healthy behaviors. JHUCCP 4: Health Commodities and Supply Chain Ensures the procurement and distribution of critical contraceptive and health commodities. John Snow, Inc.(JSI)/DELIVER; Chemonics 5: Health Systems Strengthening (HSS) Provides technical assistance to reform and improve service delivery in a post-devolution operating environment. JSI The MNCH project aims to prevent maternal, newborn, and child deaths by increasing access to high￾quality MNCH services. Its technical approach focuses on: i) increasing demand for high-quality services at the community level, ii) increasing the supply of high-quality skilled health providers at the facility level, and iii) strengthening the referral system. The primary strategies through which the project pursued these goals include: i) supporting all cadres of SBAs including doctors, lady health visitors (LHVs), and CMWs to ensure a full range of high-quality MNCH services at both public and private health facilities; ii) developing transport and communication systems to improve referral and transport in the event of complications; iii) mobilizing communities to create and sustain demand for high-quality MNCH services; iv) implementing specific technical interventions to reduce postpartum hemorrhage (PPH) and reduce birth asphyxia by providing SBAs 3 Sindh Health Sector Strategy, 2012-2020. Accessed from: http://www.trfpakistan.org/LinkClick.aspx?fileticket=1EyZSVfIMkg%3D&tabid=2618 MNCH PROJECT MID-TERM EVALUATION 8 with helping babies breathe (HBB) training and resuscitation equipment; and v) addressing bottlenecks and improving the functioning of the market system for MNCH services.4 In its first year, the project focused on improving the performance of CMWs by providing technical and business skills training, infrastructure, and equipment and supplies. In the second year, the project shifted its emphasis to improving the quality of care at public (DoH) facilities including those managed by PPHI, the semi-public entity contracted by the DoH to manage its facilities, as well as private facilities. By the third year of the project, it had significantly extended technical support to SBAs working at the facility and community levels. In 2016, Jhpiego transitioned interventions in PPHI-managed facilities to PPHI, and as a result, 442 health facilities were no longer involved in the project. However, the project works directly with the LHW program so it still maintains a working relationship with the LHWs and lady health supervisors (LHSs) and their associated community mobilization structures (WSGs). FINDINGS AND CONCLUSIONS This section presents findings for evaluation question 1—the extent to which the project has met its major goal and objectives. Questions 2 and 3 ask for best practices and lessons learned and recommendations, respectively, and the report addresses these in the conclusions and recommendations sections. This section presents findings and (when appropriate) conclusions by the three project objectives: 1) improving access to integrated FP and MNCH services; 2) increasing the capacity of health workers to deliver high-quality FP and MNCH services; and 3) improving the referral network, including community mobilization activities designed to increase health-seeking behaviors and demand. Objective 1: Access to Family Planning and MNCH Services To improve access to FP and MNCH services, the project worked to build the capacities of community service providers—LHWs and community health workers (CHWs)—and strengthen health facilities by refurbishing, supplying, and equipping CMW clinics and facilities providing basic emergency obstetric and newborn care (BEmONC) and comprehensive emergency obstetric and newborn care (CEmONC) services. Specific interventions included providing specialized training to strengthen CMWs, blood banks, and long-term family planning (LTFP), and revitalizing QITs to improve access to and quality of healthcare services. The section also presents findings on special and pilot interventions the SOW specified that the evaluation address. These include chlorhexidine (CHX)/misoprostol (miso), HBB, kangaroo mother care (KMC), nutrition, and immunization. CMW Clinics The project collaborated with the Sindh MNCH program to identify 186 CMWs in the project districts and build their capacities to become sustainable, business-oriented service providers. The project trained the CMWs to refresh existing skills and taught them new ones. It also provided training in business management and refurbished—when necessary—and equipped the CMWs’ clinics with specific instruments and supplies required for effective infection prevention practices and HBB. Project staff also provided follow-up on-the-job coaching (OJC) and on-the-job training (OJT) when necessary. The evaluation team interviewed two CMWs in each district (six in total) and observed their clinical set￾up. They also interviewed beneficiary women in the CMWs’ catchment communities. The six CMWs the 4 MNCH contract agreement and annual work plans MNCH PROJECT MID-TERM EVALUATION 9 evaluation team interviewed reported receiving training in nine topics and, except for training in diarrhea/pneumonia where two-thirds of trainees found the training useful, 100 percent said they found the training useful.5 All six CMWs reported receiving specialized business training and said that the training helped them in establishing their clinics and running them profitably. Half reported receiving equipment and all reported receiving supplies. The project also worked to strengthen referral linkages between LHWs and CMWs, and 83 percent of the CMWs the team interviewed reported increased referrals from LHWs. The project did not fund construction or rehabilitation of CMW clinics. However, because the condition of the clinics affects the quality of services they provide, the evaluation team collected data on several aspects of the clinics’ condition as indicators of the quality of services. The evaluation team’s observation found that two of the six CMW clinics it visited lacked electricity, two were not clean, one did not have a separate delivery area, and one lacked adequate storage space. All the CMWs interviewed reported improved clinical practice and technical skills, and they especially appreciated regular OJC and OJT visits by project staff. Basic Health Units The PPHI, a public-private partnership between the government of Sindh and the Sindh Rural Support Program, manages BHUs, which are the first-line facilities in rural areas. The project supported BHUs by building staff capacity, refurbishing labor rooms and patient toilets, and providing equipment and supplies. The evaluation team was not able to formally visit project-supported BHUs because the interventions had already been formally transitioned to PPHI oversight and management as of May 2016; these accounted for 48 percent of the facilities with which the project worked. BEmONC The project worked with public and private BEmONC facilities to build their capacities to more effectively provide the seven “signal functions” that treat the major causes of maternal and newborn morbidity and mortality. The project trained facility staff in these procedures and provided infection prevention-related supplies such as buckets, chlorine, and gloves; delivery sets and other instruments; and information, education, and communication (IEC) materials. Of the six BEmONC facilities the evaluation team visited, only one was able to provide all seven functions when the evaluation team visited the facility (Table 4).6 In the facilities where functions five and six were not available at the time of the team’s visit, the facility attributed it to the fact that the team visited late in the day when the doctor trained in these procedures was not on shift. At the time of the team’s visit, these facilities were being managed by SBAs who were paramedics. Although they were trained in pregnancy, childbirth, postpartum, and newborn care (PCPNC), they did not administer anti￾convulsants or practice vacuum extraction delivery. The doctor was available on call for complicated cases. 5 The team did not collect data on why the one CMW each did not find training in diarrhea/pneumonia useful. 6 A facility may have the capability to provide all 7 signal functions but not be able to provide them 24 hours per day, 7 days per week if it does not have sufficient relevant staff to cover all shifts. For this reason, not all BEmONC facilities the evaluation team visited were able to provide all seven signal functions at the time of the team’s visit. MNCH PROJECT MID-TERM EVALUATION 10 TABLE 4: PERCENTAGE OF FACILITIES PROVIDING SEVEN SIGNAL FUNCTIONS Function # BEmONC Seven Signal Functions Percentage of Facilities 1 Administer antibiotics to prevent puerperal infection 50% 2 Administer uterotonic drugs (e.g., oxytoxics) for postpartum hemorrhage 83% 3 Administer anticonvulsants to treat eclampsia and preeclampsia 67% 4 Manual removal of placenta 67% 5 Removal of retained conception products 17% 6 Assisted or instrumental vaginal delivery (vacuum extraction, forceps) 33% 7 Neonatal resuscitation (with bag and mask) 83% The team observed infection prevention chlorine corners and HBB posters and references in all six public and private health facilities it visited, and all paramedics the team observed were following the infection prevention procedures. The team observed partograph capabilities in 50 percent of the public health facilities, but its interviews with SBAs suggested that project clinical officers were not fully abreast of partographs and provided little or no coaching on this component. Some of the DoH facilities were also missing some basic components. For example, one of the two RHCs the evaluation team visited did not have pit holes for safely handing placentas over to families for burial. At the RHC in Subho Dero, a facility the project identified as a BEmONC facility, the doctor told evaluators that the center offered no BEmONC services because the facility did not have a gynecologist on staff. Conclusions: Project-wide progress towards BEmONC objectives is unclear since none of the facilities visited met the signal functions required of BEmONC facilities. Overall, a lack of sufficient human resources remains a challenge in BEmONC facilities. In many facilities services are provided through makeshift arrangements, i.e., engaging doctors for caesarian sections at limited times. Otherwise, patients have to go either to private sector BEmONC facilities or travel to adjacent districts that render BEmONC services on a 24/7 basis. CEmONC CEmONC facilities provide the basic services of BEmONC facilities but also have the capacity to provide blood transfusions, surgery (e.g., cesarean section), and neonatal intubation and advanced resuscitation (intubation and respirator available). To improve CEmONC capacities, the project trained SBAs and other staff and provided regular OJC. The project also provided labor room and operating theater (OT) equipment (e.g., infection prevention-related supplies such as buckets, chlorine, and gloves; theater kits and accessories; and instruments such as delivery sets necessary for cesarean sections), refrigerators for drug storage, infection prevention kits, and IEC materials. The evaluation team’s interviews and observations at 4 CEmONC facilities—3 public and 1 private— found that they all provided 10 of the 11 key maternal health services, and 3 of the 4 offered the 11th— availability of blood with a storage facility. All four of the CEmONC facilities the evaluation team visited had PPH trollies and HBB corners, infection prevention chlorine corners, and a partograph chart, all with prescribed posters mounted on the wall for quick reference. In three out of the four CEmONC facilities the evaluation team visited, the project had refurbished the labor room, delivery room OT, patient wards, and toilets. The two SBAs the evaluation team interviewed at the public CEmONC facility, i.e., the DHQ hospital in Tando Allahyar, reported that the project had improved adherence to infection prevention protocols. They explained that in the past, nobody, including patients’ relatives, respected the red line and the MNCH PROJECT MID-TERM EVALUATION 11 statement “no admission to unauthorized personnel” outside labor rooms and OTs. They reported that, after staff training and the assertive approach adopted by the project’s CEmONC supervisor, the red line is respected as the accepted norm. The project also appointed a CEmONC coordinator in each district tasked with visiting public and private CEmONC facilities to ensure adherence to infection prevention practices and extending continuous OJC to the OB/GYN clinical and non-clinical staff. The staff of all facilities the team visited said this continuous supportive supervision was very valuable in sensitizing the staff to the importance of observing infection prevention practices and bolstering technical skills. As one facility in-charge explained, the project’s support, especially equipment and supplies, has enhanced facilities’ capacity and extended its service area. “It is imperative to mention that in DHQ hospital Sukkur, [the project] made small but very important and most needed contributions in the CEmONC services rendered through the DHQ hospital. Their facilitation led to extending of catchment area of the DHQ hospital beyond the boundaries of Sukkur as patients were coming from adjacent districts for caesarian section. It needs to be highlighted that a year ago patients requiring natal and especially caesarian section were refused due to lack of equipment and other basic supplies. Currently, as most of the needs have been catered for, emergency and routine procedures are being accepted and there is no refusal. The OT is operational 24/7 with three shifts. All the patients are poor and from marginalized sector of the society. Patients are also accepted with complications generally from the private sector and these mainly include ruptured uterus.” – OB/GYN in-charge, Sukkur Conclusions: CEmONC facilities seem to be operating effectively. Even though providing equipment and supplies seems like a small intervention, it enabled some facilities to expand the services they offered and handle more complicated cases. Blood Bank In an effort to improve blood bank services, the project trained blood bank technicians from three CEmONC facilities on blood bank management and provided continuing OJC to ensure the quality of services. The project also provided some basic equipment to make the blood bank in Sukkur operational, but the bank has not yet been inaugurated or opened. All six blood bank staff the evaluation team interviewed said the project interventions had improved the quality of their work and improved blood banking services overall. They also noted that OJC in blood bank management had ensured adherence to infection prevention protocols. Postpartum Long-Term Family Planning (LTFP) One of the project’s key interventions was building SBAs’ capacities in postpartum LTFP, specifically in postpartum intrauterine contraceptive device (PPIUCD) insertion and implant removal since government of Sindh health officials identified this as a major skill gap. Project-supported training institutes employed comprehensive modules that covered counseling techniques and procedures, including volunteerism and informed consent. The training incorporated practice on models and on actual patients. MNCH PROJECT MID-TERM EVALUATION 12 FIGURE 1: TRENDS IN PROVISION OF IMPLANTS AND IUCDs After the training, the project oriented each participant to the online registration mechanism necessary to obtain contraceptives from the district warehouses. Project-reported data show a steady increase in the average number of implants and intrauterine contraceptive devices (IUCDs) inserted per supported facility between October 2014 and March 2016. However, the non-constant number of facilities reporting in each period makes it difficult to interpret the numbers (Figure 1). Lady Health Workers LHWs provide information and simple curative care in family planning, vaccination, diarrheal and other communicable disease control, and nutrition and water and sanitation practices. LHWs disseminate information largely through the WSGs they establish and facilitate.7 The project trained LHWs in interpersonal communication (IPC), support group methodology, and CHX/miso to revitalize WSGs. It reports having trained 3,206 LHWs and 161 CHWs in phase A districts. Project reports indicate that these trained LHWs and CHWs support the catchment areas of 306 of the 341 MNCH centers in the project districts.8 In consultation with the LHW program, the project also updated LHW job aids (e.g., flip chart book for community information and discussion). It also relied on LHWs to refer patients to project-supported SBAs, facilitate cooking demonstrations (a project nutrition activity), support project￾supported PPIUCD camps and advocacy events, and serve as focal points for distributing CHX/miso and iron/folic acid (IFA). The evaluation team interviewed 10 LHWs and 15 members of the WSGs they facilitated in Tando Allahyar and Khairpur.9 The LHWs reported receiving training in CHX/miso (60 percent of respondents), IPC (60 percent of respondents), and conducting WSGs (60 percent of respondents). All who received training on CHX/miso and WSGs reported using the skills they learned, while 50 percent 7 The LHW program uses the term community support groups (CSGs) which are equivalent to what the project calls women’s support groups (WSGs). The remainder of the report uses the term WSG. 8 MCHIP Year 3, Qtr-2 Report 9 In Sukkur, the project had not yet provided training to LHWs. MNCH PROJECT MID-TERM EVALUATION 13 reported using what they learned in the IPC training.10 The training is ongoing and the data do not necessarily reflect the percentage of LHWs the project planned to train over the life of the project. All the interviewees said that the package of trainings was relevant and has contributed significantly to improving the quality of WSGs. Sixty percent of LHWs explained that before the project, they conducted WSG meetings infrequently and that the meetings were poorly attended, often repeated topics, and involved little interactive discussion. In response to open-ended questions in group discussions about how the project had improved the function of WSGs, LHWs and WSG member participants explained that the groups were meeting more frequently, with greater regularity and function, and discussed a wider range of topics in more interactive discussions.11 All WSG members reported that the WSGs disseminated useful health information that they shared with their families, friends, and neighbors. All of the LHWs the evaluation team interviewed reported using the revised job aids and found them user-friendly and effective with new topics and updated information. The LHW provincial program representative said the materials were of high quality and has requested a sufficient supply to distribute throughout the province. The project also developed a “mother’s booklet” which the LHW program endorsed and distributed to 15,000 pregnant women in 15 districts.12 Pregnant WSG members the evaluation team interviewed in a group interview in Tando Allahyar reported receiving the book and found it informative. The WSG in Khairpur did not mention the mother’s booklet. The team did not conduct group interviews with WSGs in the other two districts because another MCH partner, JHUCCP, was conducting the community mobilization work in the other two districts (phases B and C). Conclusions: Interviews with LHWs and WSG members suggest that project interventions have revitalized WSGs in the project districts. The LHW program’s endorsement of the mother’s booklet will help disseminate project-supported practices more broadly throughout the LHW program. However, the loss of PPHI as a partner means that the project will need to engage LHWs directly through the LHW program and not through the BHUs. Community Health Workers13 The project engaged 161 CHWs to cover areas where there were no LHWs and trained them to perform selected LHW activities, including forming WSGs. The project expected that the provincial government would be willing to groom these CHWs and incorporate them into the regular LHW training program.14 Although the CHWs interviewed aspired to obtain LHW training, the LHW program representative the evaluation team interviewed said that the program does not plan to engage the CHWs after the project ends. All four CHWs the evaluation team interviewed reported forming WSGs and holding regular meetings with women who were eager to participate. They all wanted training in more topics and refresher training so they could better answer questions raised by WSG participants. 10 The three LHWs who said they did not use the IPC training did not provide any reasons for not using it. 11 Participants in both group discussions mentioned these improvements, but the analysis did not identify individual responses. 12 MCHIP Year 3, Qtr-2 Report. 13 The CHW nomenclature is an internationally recognized term usually associated with broader skills and responsibilities than those of project-supported CHWs whose roles and responsibilities coincide more closely with community health volunteers in other countries. This could cause confusion when the health workforce is mapped in the future. CHWs the team interviewed aspire to become LHWs, but the LHW program representative the evaluation team interviewed said that the program does not intend to take on the CHWs at the end of the project. 14 MCHIP Year 2- Work Plan. MNCH PROJECT MID-TERM EVALUATION 14 Quality Improvement Teams The project established QITs associated with CEmONC and BEmONC facilities. The QITs include representatives from the facilities and the communities and employ a partnership defined quality (PDQ) process to engage communities in improving the quality of services according to community-established criteria. The project reported establishing 220 QITs of a target of 250—54 in DoH facilities, 164 in PPHI facilities, and 2 in private facilities.15 The project also worked with PPHI to revitalize its facility-based teams, called community support groups (CSGs), which serve the same purpose as QITs.16 It also helped strengthen the capacity of PPHI social organizers to use PDQ principles to improve the quality of services. The evaluation team conducted two group discussions with QITs—one in Tando Allahyar and one in Khairpur (both phase A districts).17 Together, the QITs included 23 members, 47 percent of whom were community members, 53 percent of whom were facility staff, and 15 percent of whom were women. Project staff also reported their attendance at the meetings. The facility medical superintendent chaired both QITs, and members reported that as a result of transfers, facility staff members changed frequently. Group discussion participants reported meeting monthly with four to five members attending. They also reported that few community members participated because they found it cumbersome to attend these meetings. As the only community representative in the group discussion at Tando Allahyar said: “I have been a member of QIT for the last one and a half years, but have attended only three or so meetings. I rarely come due to my other engagements.” Group discussion participants described the purpose of the QITs as improving the condition of the facility and provision of services. None mentioned needs-based service demand, improved governance, or shared accountability. Participants in both group discussions explained that the role of facility members is to improve facility conditions while the role of community members is to mobilize communities and encourage them to utilize the facility’s MNCH services. None of the group discussion participants reported receiving training in PDQ, QIT purpose refreshers, or orientation. Both QITs the evaluation team interviewed had developed action plans for improving facility functioning and services. In Tando Allahyar, the current action plan is to address the issues of irregular staff, lack of a medical officer, planting trees around the facility (about which it recently met with the municipal administration), and the facility staff’s poor attitude towards patients. Its previous action plans addressed improving facility cleanliness, removing shrubs and weeds, improving water supply, and improving staff behavior towards patients. In Khairpur, the action plan focused largely on activities that were not MNCH-specific, such as inadequate water supply for planting trees and limited community awareness about available services. The team collected no information during a follow-up visit in Khairpur because the QIT was established very recently. The main challenges QIT members mentioned included the chairperson’s (medical superintendent) influence in its establishment, efficacy, and continuation. For example, the chairperson in Khairpur suspended the QIT for almost 18 months. Other reported challenges included members’ directing resources to their own areas, lack of community members’ active participation, and action points that require district or provincial level health officials’ support to resolve (e.g., increasing facility staff). 15 Year 3, Qtr 2 Report 16 Year 3, Qtr 1 Report 17 The evaluation team was unable to reach PPHI’s CSGs because PPHI was no longer part of the project. PPHI CSGs are different groups than the CSGs formed under the LHW program. MNCH PROJECT MID-TERM EVALUATION 15 “We are short of staff. Everyone knows this. What can we do? What can our medical superintendent do? He can only inform the district health officer about such an issue. He cannot solve it by himself.” – QIT participant in Tando Allahyar Despite these challenges, members of both QITs described it as an effective body that contributed to improved facility operation and services. Both QITs also believed that the QITs will continue to work even after the end of the project. Special Interventions The evaluation also examined pilot interventions including CHX, miso, and HBB and other specific interventions such as KMC, nutrition, and immunization. This section presents findings for these pilot and special interventions. Misoprostol and CHX The project promotes the use of misoprostol to prevent maternal deaths from postpartum hemorrhage, which account for 27 percent of maternal deaths in Pakistan.18 National policy already dictates that LHWs distribute misoprostol tablets, but effective implementation, supply, and monitoring have been problematic. The project also promotes the use of CHX for umbilical cord care, another evidence-based intervention that has been shown to reduce neonatal mortality by up to 30 percent in comparable settings in South Asia. The project has been involved in successfully advocating for the addition of CHX to the official national essential drug list—so it is available from sources other than the project—and testing various concentrations of the drug in the local environment. The project piloted distribution of CHX through LHWs and found that 97 percent of clients adhered to its recommendations.19 At the time of the evaluation, the project had not yet scaled up this component in the initial five districts. With project support, the LHW program is distributing a CHX/miso joint pack. In Khairpur and Tando Allahyar districts, district coordinators told the evaluation team that the project completed training of trainers (TOT) on miso and CHX, and started distributing CHX/miso packs. Sukkur district, on the other hand, has yet to start activities pertaining to this component. Sixty percent of the 10 LHWs and 67 percent of the 6 CMWs the evaluation team interviewed reported receiving and distributing miso or CHX and reported that all the clients who received miso or CHX used them— implying a 100 percent utilization rate among clients who received miso or CHX. This finding is consistent with the 97 percent utilization rate reported by the project, but the evaluation team did not interview users and cannot verify the reported utilization rate. Conclusions: The findings suggest that the miso/CHX component of the project is progressing well—it has trained LHWs on these treatments and successfully promoted their distribution and use. The project’s advocacy to get CHX added to the national drug list is a substantial achievement and will improve access to an evidence-based treatment for reducing neonatal mortality. Helping Babies Breathe (HBB) HBB is an evidence-based intervention that has been proven to reduce neonatal deaths related to asphyxia.20 The evaluation of the national MNCH program found that facilities were already practicing some components of resuscitation such as rubbing the babies’ backs and clearing the airway.21 However, 18 Lancet (2013) 19 MCHIP CHX Adherence Study 20 The American Academy of Pediatrics and Saving Newborn Lives are two major partners. 21 Term Evaluation of the National Maternal and Child Health Programme in Pakistan, Report. Accessed from: http://www.trfpakistan.org/LinkClick.aspx?fileticket=0WOWUr-6vQI%3D&tabid=2618 MNCH PROJECT MID-TERM EVALUATION 16 they lacked the equipment and training to practice ventilation using a bag valve mask (commonly referred to by the proprietary name “Ambu bag”). To strengthen the facilities’ capabilities for neonatal resuscitation, the project included HBB in all its SBA training, and equipped HBB corners (including providing Ambu bags and posters) in labor and delivery rooms. The project is collaborating with the HSS project to get the DoH to include relevant indicator(s) in the district health information system. In the meantime, the project provides stickers to place in individual medical records to document when a baby has been resuscitated. Two of the three pediatricians the evaluation team interviewed in public CEmONC facilities reported that the OB/GYN departments offered HBB. The evaluation team observed HBB posters and equipment in all facilities visited but was not able to directly observe the abilities of SBAs to use the equipment. Conclusions: The project’s contribution to increasing the availability of Ambu bags and providing related training has substantially increased access to an effective method for neonatal resuscitation and reduced the loss of life from a preventable cause of death. Kangaroo Mother Care KMC is an evidence-based, cost-effective, and easy-to-use care of preterm and full-term infants that promotes holding the infants in skin-to-skin contact with the mother or another caregiver. The project reported that it has conducted a situation analysis in two CEmONC facilities in Khairpur district and is working with DoH to select a district for the KMC pilot. Since the project has not started KMC activities, there was nothing for the evaluation team to observe. Nutrition For reasons that are unclear to the evaluation team, the project did not start implementing nutrition activities in the field until year three and did not follow many of the recommendations of a consultant hired by PATH, the project’s technical partner for nutrition. Starting in year three, the project trained 20 LHSs from 10 districts as trainers for LHWs using a nutrition module consisting of maternal, infant, and young child nutrition knowledge and practices. The project also trained 18 of the 186 CMWs involved in the project on lactation management and added a lactation management TOT in training institute curricula.22 The project also conducted 32 cooking demonstrations associated with nutrition activities, but project reports do not explain the rationale and objectives of these demonstrations. Nutrition counseling is a regular part of LHW and CHW practice, and the project provided IFA tablets—already widely available and used in Pakistan—to LHWs and CHWs to distribute to pregnant women. A large majority of the CMWs (83 percent of 6 CMWs) and all 10 LHWs the evaluation team interviewed reported distributing IFA to pregnant women. At the time of the evaluation, more than halfway through the project’s third year, nutrition interventions had not progressed sufficiently for the evaluation team to observe any activities. Immunization The project completed an immunization situation analysis in 2013 and, on the basis of the analysis, proposed activities for the first year’s work plan. Shortly afterward, however, USAID issued a separate RFA for immunization and asked the project not to spend the money originally set aside for immunization. However, USAID never awarded the separate immunization activity for programmatic reasons and, in September 2014, asked the project to resume immunization activities in FY 2015. 22 Yr 3. Quarter 2 Report, 2016. MNCH PROJECT MID-TERM EVALUATION 17 Project managers told the evaluation team that USAID recommended that Jhpiego use immunization funds to scale up “the successful approach used by the Health Systems Strengthening (HSS) component of the MCH program to four additional districts.” The approach involves contracting with the Rural Support Program Network (RSPN) to identify a cohort of newborns and support immunizers from the Expanded Program on Immunization (EPI) to go to communities and vaccinate each child. Project managers reported that RSPN completed a baseline study in Tando Allahyar district in 2016 and planned a baseline for Shikarpur district which had not yet started. RSPN trained 27 EPI vaccinators on cold chain management and registration and verification of children. The baseline found that EPI vaccinators were not able to travel because they were not provided a travel allowance to cover their costs. Associated with the baseline, RSPN calculated that it would cost very little to provide this allowance, a low enough cost to convince the district government to begin providing an allowance. The DoH manager the evaluation team interviewed in Tando Allahyar was happy with the baseline and especially the cost calculation that ultimately facilitated the mobility of the vaccinators. Conclusions: Because the project started activities so late, a complete cohort of newborns would need to be enrolled by the end of the first quarter of FY 2017 to allow sufficient time for all of them to be completely immunized by the end of the program. A very important initiative taken by RSPN is to address the longstanding issue of providing a travel allowance for vaccinators. Mother’s Support Card The project also supported the Mother’s Support Card initiative to facilitate marginalized and poor women’s access to services. The initiative subsidizes the cost of a package of MNCH services to encourage marginalized and poor pregnant women to access a complete package of care including attending ANC, delivery, and PNC at CMW clinics. Because the project did not begin the initiative until late in the project life, the evaluation team was not able to assess its effectiveness. Objective 2: Capacity of Health Workers The project addressed capacity building from two perspectives: establishing or strengthening the institutes and schools that trained health workers and conducting the training itself, both in the facilities and through follow-up OJC and OJT. This section presents findings separately for strengthening institutes and schools and the training itself. Strengthening Training Institutes and Schools To enhance training capacities, the project established or strengthened training institutes and CMW schools that provide pre-service and in-service training. Project support included providing supplies and equipment, refurbishing facilities, and minor revisions to curricula. MNCH and Family Planning Training Institutes To enhance the capacity of SBAs, the project established or strengthened training institutes in the OB/GYN departments of five government and private health facilities.23 Support to these institutes included minor updates to existing curricula and adding new modules on CHX cord care and HBB. The evaluation team examined the training materials and found that they included evidence-based maternal, newborn, and FP practices consistent with recognized international standards. In addition to classroom 23 The project worked with four existing institutes in Liaquat University of Medical and Health Sciences (LUMHS) in Hyderabad (government) and Lady Duffering Hospital in Karachi (private), and it established a new institute in the DHQ hospital in Sukkur (government). MNCH PROJECT MID-TERM EVALUATION 18 learning, the trainings also engaged trainees in practical clinical experience in the labor rooms, OTs, and wards of OB/GYN departments. Managers of all three training institutes the evaluation team visited saw value in the training.24 However, two mentioned that they had no memorandum of understanding or terms of reference to define the respective roles of the institutes and the project. They said they would have preferred a more formal agreement, and one that trained their qualified staff as master trainers to build ownership and sustainable capacity. Two of the three managers also said they would have preferred that the project use accredited curricula to enhance the capacities of their institutes. Project staff explained that meeting these requirements was beyond the scope of the MNCH project. To facilitate the training, the project also refurbished labor and seminar rooms and equipped skills laboratories with the necessary mannequins, supplies, and instruments. The project also hired around￾the-clock postpartum FP counselors to advise women about FP options at the time of delivery. Representatives of two of the three training institutes found these positions valuable and said they intended to continue them using their own funds after the project. Conclusions: Establishing formal training institutes in the OB/GYN departments was a valuable intervention in upgrading the knowledge and skills of SBAs in both the public and private sectors. The decision not to collaborate more formally with training institutes affected the departments’ sense of ownership and may have reduced prospects for sustainable results. CMW Schools The GoP’s 2006 national MNCH program introduced CMWs to extend MNCH services to poor and disadvantaged populations in remote areas that were not covered by LHWs and also established schools to train the CMWs. The training strategy provides institution-based training for a period of one year to cover the theoretical and practical aspects of the curriculum followed by six months of practical training at a DHQ or THQ hospital or an RHC. An evaluation of the national MNCH program identified inadequate engagement and mentoring in clinical training in labor rooms and wards as a weakness of the CMW program.25 To address this issue, the project placed a CMW coordinator in the hospitals to match CMWs with senior staff and ensure that they get the necessary hands-on experience, monitoring, and mentoring in managing patients. The evaluation team’s interviews with OB/GYN staff at all three hospitals with CMW coordinators revealed that mentoring increased the medical staff’s sense of ownership towards CMWs, the interest of CMWs in clinical work, and their skills. All three CMW schools the evaluation team visited also reported receiving some equipment and teaching aids from the project which facility staff reported contributed positively to the CMWs’ classroom training. Conclusions: Placing CMW coordinators in the training hospitals was an important intervention that substantially improved the effectiveness of the CMW schools in building the skills of SBAs. 24 The valuation team visited LUMHS in Hyderabad (government), the DHQ hospital in Sukkur (government), and Lady Duffering Hospital in Karachi (private). 25 Mid-Term Evaluation of the National Maternal and Child Health Program in Pakistan, Report. Accessed from: http://www.trfpakistan.org/LinkClick.aspx?fileticket=0WOWUr-6vQI%3D&tabid=2618 MNCH PROJECT MID-TERM EVALUATION 19 Training Facility-Based Service Providers The project reported training 13,341 facility-based SBAs (CMWs, LHVs, nurses, midwives, medical officers, and gynecologists) and other healthcare providers (LHWs, CHWs, anesthetists, pediatricians, and blood bank staff) in the project-supported districts between January 2013 and April 2016. It used project staff or consultants to administer the trainings and conducted them in facilities within the districts (e.g., hospitals, training institutes, CMW schools, government buildings, or hotels). The project provided a standard package of training that included pregnancy, childbirth, postpartum, and newborn care (PCPNC); post-abortion care (PAC); postpartum intrauterine contraceptive device (PPIUCD) insertion; implant insertion; HBB; miso; CHX; lactation management; infection prevention; and CMW business management. To all those trained in infection prevention and HBB, the project provided the instruments and supplies required to implement the practices (e.g., bag valve masks, commonly referred to by their proprietary name “Ambu bags”). It also provided posters and wall mounts pertaining to the labor room, pre- and post-surgery infection prevention, HBB, CHX, and other practices to all facilities. When asked to list the project’s contributions to improving MNCH services, DoH and PPHI managers, healthcare service providers, and project staff mentioned training more frequently than any other contribution (72 percent of 39 responses). Fifteen percent of responses mentioned providing infrastructure, supplies, and medicines, and 13 percent mentioned OJC. PPHI district managers (3) and facility in-charges (2) reported that project-supported capacity building (i.e., training and OJC) increased SBAs’ technical knowledge, skills, and practices in antenatal care (ANC), normal delivery, postnatal care (PNC), and newborn care, including HBB. They explained that, prior to the project, even though affordable infection prevention protocols existed, delivery room instruments were often rusty and cleaned only in plain water. The project also routinely monitored trainees after they returned to their facilities and provided OJC or OJT as necessary to refresh skills and improve practice. Quality of care (QOC) studies conducted by the project, and feedback from SBAs trained by the project, concluded that OJC and OJT had improved SBAs’ knowledge, skills, and practice. The project’s CEmONC supervisors are responsible for OJC and OJT in CEmONC facilities, while in BEmONC facilities, clinical officers, supervised by the project’s senior clinical supervisor, are responsible for OJC and OJT. In two of the three districts the evaluation team visited (Tando Allahyar and Khairpur), most clinical officers were paramedics and less experienced than the SBAs they were meant to coach. Consequently, the SBAs often did not welcome the clinical officers. The project’s senior clinical supervisors from these two districts reported that this situation affected the morale of the clinical officers. The senior clinical supervisors (doctors) tried bridging the gaps as they arose, but this is not a practical approach to coaching a large number of SBAs. PPHI district managers (3) and facility in-charges (2) reported that the project’s frequent monitoring support visits and OJC were particularly important in improving the quality of services, facility management, and adherence to infection prevention protocols. However, as the project progressed, its staff did not coordinate monitoring visits with PPHI local offices and facilities which disturbed the OPD and routine client services at the facilities. The PPHI and MNCH senior managers the evaluation team interviewed reported that PPHI has adopted the best practices provided in the training and SBA skills refreshers, and has appointed designated staff to ensure the continuity of training and OJC. Objective 3: Referrals The project addressed referrals by strengthening referral networks and training transporters. MNCH PROJECT MID-TERM EVALUATION 20 Referral Mechanism At the community level, LHWs and CMWs refer patients and pregnant women to other facilities. The project strengthened the referral system by training LHWs and CMWs when, where, and how to refer cases. The project also used referral slips to track referrals. All 10 LHWs the evaluation team interviewed reported distributing referral slips. Eighty percent reported that clients questioned the benefit of the extra effort required to use the referral slips since it makes little difference to treatment or follow-ups. Using referral slips is part of the LHWs’ routine activity, but they reported that inconsistent supply of slips, little utility to the patient, and loss or misplacement of slips by patients undermined the efficacy of the referral process. All LHWs the team interviewed reported that they frequently accompanied the patients themselves, especially for childbirth. All six CMW health facilities the team visited conspicuously displayed a chart showing the distance and travel time to the next level facilities. The CMWs, LHWs, and clients the evaluation team interviewed explained that the actual referral depends on travel distance and time, the condition of the patient, the services available, and personal preference. All six of the CMWs the evaluation team interviewed reported distributing referral slips. Transporters The project developed transporters’ networks to improve access to and timely availability of health services. It reported training 1,329 local drivers to handle transfer requests and clients, and provided transporters with maps to referral hospitals. It also provided a list of trained drivers to community healthcare providers. Project reports indicate that 2,056 women in 10 districts have used the service, an average of two women per transporter. The evaluation team conducted group interviews with transporters, and individual interviews with community level service providers (CHWs and LHWs) and beneficiary women to explore transporter services. Transporters in three group interviews were familiar with the public facility serving their catchment area, but reported that more than half of their clients prefer to go to private facilities. All of the drivers interviewed stated that the training has helped them handle patients better, as one transporter from Sukkur explained: “Before this training, we did not know how serious a delivery case was. We would ask clients to just sit in the car and then would take them to the hospital. Now we know which hospitals provide facilities for normal delivery and operation [C-section] and which hospitals are open for 24 hours or 12 hours.” All drivers interviewed reported that at times they were not allowed to enter patients’ homes or advise them of the best position in which to be transported. Six of the eight drivers the team interviewed reported receiving referral slip books and completing referral slips for clients to give to project staff. Of those who received referral slips, 90 percent stated that they had not returned the slips and infrequently share information on the phone with the project. Consequently the project has no information for monitoring the performance of transporters. Fifty-six percent of the beneficiary women and all CMWs and LHWs the team interviewed knew of the trained transporters; 30 percent of the 10 LHWs and 50 percent of the 6 CMWs reported referring clients to transporters. Clients cited the availability of transportation at their doorstep in emergency situations as the most important reason for using these services. They explained that the availability of other transportation options, including their own vehicles, was the most common reason for not using the service. LHWs (5 of 10) and CMWs (2 of 6) believed the high cost of trained transporters relative MNCH PROJECT MID-TERM EVALUATION 21 to other options presented a barrier to their use. The transporters charge for the trip to the client’s home and from there to the facility, as opposed to a normal taxi that charges only for the trip from the home to the facility. Half of the 16 clients interviewed noted that the transporters charged more than taxis. Because all transporters are established taxi drivers, they said they expected to continue providing transport services after the program ended. Overall Project Outcomes It is not possible to separate the contribution of activities grouped under each objective to increasing access to and quality of MNCH services. This section provides evidence of the overall outcomes associated with the complete package of project activities taken from the perspectives of healthcare providers, the project’s quality improvement and patient safety (QIPS) assessments, and the evaluation’s client exit interview data. Ninety percent of 39 DoH and PPHI managers and healthcare providers the evaluation team interviewed said that the project has been effective. When asked to identify the single most effective project component, 67 percent identified training, 13 percent cited the quality of interventions, and 10 percent noted the supplies and equipment the project supplied. Other interventions they mentioned included infrastructure and introducing new practices. The evaluators asked the same individuals about the sustainability of the project and which single component they thought was most sustainable. Eighty-three percent of respondents believed the skills learned in training were the most sustainable interventions, and 8 percent identified partnerships. The fact that PPHI has institutionalized the project’s training and capacity building components in all PPHI managed health facilities with its own resources demonstrates the sustainability of this component of the project. A comparison of QIPS scores between March 2015 and January 2016 shows substantial improvement in facility performance (Figure 2). 26 The project used its customized QIPS assessment tool to assess and improve the quality of care provided by project-supported MNCH facilities. The tool assesses the quality of care in seven core MNCH areas: i) ANC, ii) labor and delivery, iii) pneumonia and diarrhea, iv) PNC, v) postpartum family planning, vi) infection prevention, and vii) referrals and linkages. The project conducted a first-round QIPS assessment of 258 supported health facilities in phase A districts in March 2015 and a second round that included 256 of the first-round facilities in January 2016. The project has not implemented the QIPS assessment in phase B and C districts, so the evaluation team could not assess improvement in performance in phase B and C districts. The evaluation team’s client exit interviews also provide evidence of the QOC provided by project￾supported facilities—although not a measure of change in QOC. The data show a high level of client satisfaction with services and validate the QIPS assessment results. Figure 3 illustrates the percentage of 41 clients the evaluation team interviewed that responded positively to 11 aspects of care such as facility cleanliness, healthcare behavior and responsiveness, and measures undertaken to ensure privacy. It is important to note that 90 percent of the clients said they would visit the facility again and 85 percent said would refer someone else to the facility. 26 Final report on QIPS rounds 1 and 2 implemented by MNCH project. MNCH PROJECT MID-TERM EVALUATION 22 FIGURE 2: CHANGE IN QIPS INDICATORS IN PHASE A DISTRICTS FIGURE 3: CLIENT SATISFACTION WITH SERVICES The client exit survey also asked clients about the type of advice and counseling on FP, ANC, and PNC that they received from service providers. Results were mixed across the seven questions presented in Figure 4, but the data generally show a greater level of compliance with good patient care practices than noncompliance. MNCH PROJECT MID-TERM EVALUATION 23 FIGURE 4: QUALITY OF ANC AND PNC SERVICES The project’s reported performance data also show a substantial improvement in MNCH indicators. The MNCH project started working with 143 health facilities and gradually increased to 927 at the time of evaluation. The interventions in phase A, B, and C districts are at different stages of maturity and are not comparable. In consultation with the project team, the evaluators identified 121 phase A facilities on which the project had data from 2014 through 2016 to observe changes in key indicators. Table 5 shows a substantial change in most indicators of use of project-supported services. The data show larger increases for FP services such as IUCD and implant insertions that were not commonly available prior to the project than for more standard services such as tetanus injections. TABLE 5: CHANGE IN MNCH PROJECT INDICATORS, 2014–2016 Indicator Average Score Q3 FY 2014 Q2 FY 2016 Overall Change % Increase Antenatal care visits 139.8 173.66 33.86 24% Normal vaginal deliveries 56.57 80.31 23.74 42% Postnatal care 14.99 27.61 12.62 84% Tetanus injections 44.31 46.53 2.22 5% IUCD insertions 0.12 2.02 1.9 1,583% Implant insertions 0.25 5.2 4.95 1,980% Newborns resuscitated 0.19 0.85 0.66 347% Case management of pneumonia 24.78 43.13 18.35 74% Case management of diarrhea 88.86 126.25 37.39 42% MNCH PROJECT MID-TERM EVALUATION 24 Project Implementation and Management This section summarizes findings on implementation and management that are relevant to the evaluation’s conclusions. Separate sections address the project’s organizational arrangement with implementing partners, coordination and collaboration with DoH and PPHI, and monitoring and evaluation. Project Implementation Implementation varied by district. In eight districts, local implementing partners administered and implemented project activities at the field level and provided field staff at the community level while the technical partners (Jhpiego, Save the Children, and JHUCCP) provided technical direction. In the remaining seven districts, the project was responsible for all aspects of implementation (Table 6). MNCH project district managers in the three districts the evaluation team visited—all districts where implementing partners implemented the project activities—told the evaluation team that the administrative arrangement of working through implementing partners was not appropriate given the pace of project implementation. Therefore, in phase C districts, the project largely limited the role of implementing partners to providing office space and services and one assistant. In districts where implementing partners implement activities, the district teams include staff from three or four organizations, each with different responsibilities and levels of experience in MNCH programs. There are no direct lines of implementation authority or feedback mechanisms. For example, the senior clinical supervisor is a project employee, but is dependent on the clinical officer who reports to the implementing partner’s district coordinator, and not directly to the project district coordinator, and has no direct supervisory authority to question or supervise implementing partner staff. Similarly, the senior social mobilizers are Save the Children employees while the social mobilizers are implementing partner employees which creates similar challenges in lines of authority. TABLE 6: IMPLEMENTATION APPROACHES Implementation Approach Phase A Districts Phase B Districts Phase C Districts Total Districts Implemented by the MNCH project with implementing partner providing the office Tharparkar, Thatta, and Dadu N/A Ghotki, Matiari, Mirpurkhas, and Shikarpur 7 Implemented by implementing partners with technical oversight from the MNCH project Khairpur and Tando Allahyar Jaccobabad, Naushehro Feroze, Sanghar, Sukkur, and Umerkot Larkana 8 Project staff the evaluation team interviewed related that when the project expanded into phase B and C districts, it moved a lot of senior managers so they could work closer to their homes. This decision had the advantage of moving experienced staff to some new districts which should facilitate a rapid startup—although the evaluation team was not able to observe these effects. It had the drawback, however, of bringing into the phase A districts relatively inexperienced managers who required orientation and slowed implementation. MNCH PROJECT MID-TERM EVALUATION 25 At the time of the evaluation, the project had no written exit strategy. Project managers the evaluation team interviewed said that they have started thinking about sustaining results, and USAID has requested that the project include an exit strategy in the project’s FY 2017 work plan. Coordination and Collaboration with Government and PPHI Since the project worked within public and private health facilities, collaboration with government and PPHI at the provincial and district levels was crucial. The additional director of the DoH was relatively new to his position, but expressed concerns about implementation of the CHW component, thinking that the project was hiring and deploying LHWs. This suggests that the project had not effectively briefed the new additional director about its activities. Representatives of the DoH MNCH office whom the evaluation team interviewed were enthusiastic about the support they had received from the project, especially strengthening the CMW program. However, they also mentioned that project senior managers had not attended MNCH task force meetings for several months. The deputy program manager of the Sindh province nutrition program was also relatively new, but said that staff does not attend the nutrition program’s partner meetings. At the district level, the project and the implementing partners work closely with the DoH and PPHI. The project collaborates with DoH, PPHI, and the LHW and MNCH programs on joint events, e.g., health festivals, education sessions, recipe competitions, district level meetings, and other events. The district Population Welfare Department (PWD) officer said the project was not effectively collaborating with PWD. The project coordinated on province-wide capacity building and joint activities with the PPHI head office in Karachi, and even placed two staff at the PPHI head office. The PPHI district managers said there was little coordination and collaboration at the district level. They developed joint work plans, but project staff rarely followed the plans. Monitoring and Evaluation The project developed a PMP in the first year but altered its implementation approach several times without updating the PMP to reflect the changes. It submitted a revised PMP to USAID/Pakistan in 2016. The evaluation team was able to assess progress against targets only for the indicators on which the project reported updated targets and continued to collect data throughout the implementation period. The project planned to measure performance indicators from the annual survey data. It conducted a baseline in 2013 and a follow-up in 2014. Due to changes in government requirements for field data collection, the project was unable to collect data in 2015. Consequently, the evaluation team did not have the project data necessary to assess progress toward goal and outcome indicators. The evaluation team also noted inconsistencies within and between the project’s various databases. For example, the training topics in progress reports are different than the topics in the project’s online training dashboards. Project documentation was also poor. The project did not document substantial changes in implementation including changes in geographic focus, project interventions, indicators, and targets. Conclusions: The project collects and enters copious quantities of performance data through its routine monitoring processes. These data are available from various online and offline databases kept primarily at the head office. The evaluation team saw no evidence that the project used aggregated performance data to determine what the project had accomplished or for informed decision-making. MNCH PROJECT MID-TERM EVALUATION 26 The project headquarters short-term technical associate report also validates the evaluation team’s finding about the limited use of project data for decision-making. 27 CONCLUSIONS Conclusions with respect to question 1: To what extent has the project been effective in meeting its major goals and objectives? The project expects to measure the goal (i.e., reducing maternal and child mortality) at the end of the project and was not able to provide the evaluation team with evidence of progress towards the goal. The project was most effective in increasing the capacity of health workers to deliver high-quality FP and MNCH services (objective 2). Health facility managers and healthcare providers overwhelmingly identified training as the most effective project intervention. The project’s OJC and OJT were effective approaches to improve and refresh skills and enhance the quality of care in facilities. PPHI believed the training approach was successful enough that it adopted the approach throughout all its facilities. Establishing and strengthening (refurbishing, equipping, and supplying) training institutes and CMW schools enhanced the capacities of these institutions to implement their regular pre-service and in￾service training. The project was moderately effective in improving access to integrated FP and MNCH services (objective 1). Its support to LHWs revitalized WSGs and facilitated wide distribution of CHX and miso. The LHW program’s adoption of the mother’s booklet is also an important project achievement. The project’s support to CMWs (technical and business training, refurbishing and equipping clinics, providing supplies, and linking to LHWs for referrals) effectively increased access to FP and MNCH services in remote, rural areas. The project reported providing a great deal of support to BHUs, but because PPHI withdrew from the project just before the evaluation, the evaluation team could not collect information about the impacts of these activities. Project-supported CEmONC facilities were providing most of the required functions, while none of the BEmONC facilities was providing the full range of required functions. When the facilities could not provide a function, it was most often because they lacked the human resources to staff hospitals around the clock. The project’s efforts to improve the referral network (objective 3) met with limited success. The training improved transporters’ skills in handling clients, and transporters believed they would continue providing the service after the project closed, but few used the referral slips. Clients found the trained transporter service more expensive than alternatives and saw little value in using referral slips. Overall, the project provided high-quality technical expertise and effective interventions. Weaknesses in some areas of management, however, may have kept it from meeting its full potential. For example, the apparent failure to effectively utilize the data it collected from the field to inform decision-making is a missed opportunity that could have improved performance. The following component-specific conclusions provide more detail. Training Institutes: Strengthening existing and establishing new training institutes within OB/GYN departments was a valuable intervention that substantially upgraded the knowledge and skills of healthcare providers and the capacities of the institutes. The intervention could have been more 27 MCHIP Fogarty STTA Report, November 2015. MNCH PROJECT MID-TERM EVALUATION 27 effective and sustainable if it had engaged institute staff not only as training participants, but trained them as master trainers. CMWs: Support to CMWs to help them establish and improve their clinics has been effective. After PPHI-managed facilities, CMWs comprise the largest number of facilities with which the project engaged. Relevant training and OJC/OJT have been pivotal in making the CMWs operational and bringing SBA services closer to hard-to-reach and underserved women in poor communities. The project has taken strategic measures to meet the challenges to setting up effective CMW clinics by providing business skills, needs-based structural improvements, and equipment and supplies, and supporting referrals from LHWs. BEmONC and CEmONC: Limited human resources emerged as the primary constraint to providing consistent BEmONC and CEmONC services. It is a more binding constraint for BEmONC facilities, most of which could not provide the required seven signal functions at the time of the evaluation team’s visit. CEmONC facilities fared better but still lacked the necessary staff to provide two functions consistently. Immunization: USAID’s decision to issue a separate RFA for immunization in 2014 delayed the start of immunization activities by a year. Jhpiego then took another year to design the intervention and begin implementation. As a result, Jhpiego did not begin implementing immunization activities until the middle of 2016, making it difficult for the project to implement a comprehensive, effective strategy for immunization across all intervention sites. LHWs: Project-imparted trainings have appreciably improved the routine work of LHWs in conducting WSGs. Introducing new interventions (e.g., CHX and miso) through the LHWs was an effective method for improving awareness and generating demand for life-saving treatments. The 97 percent CHX adherence rate among LHW clients suggests that the approach is cost-effective and efficient. The geographic reach and reputation of the LHW program makes it an ideal vehicle for scaling up the CHX/miso intervention. The LHW program’s endorsement of the mother’s booklet is also a significant achievement that will facilitate dissemination of updated health-related information. CHWs: The project’s CHWs are registering women in communities and holding WSG meetings, but their training and technical skills are not comparable to LHWs’. Transporters: The transporters’ network with specialized training on patient handling is working well. The patient handling skills and linkages to facilities available to transporters will continue to facilitate access to transportation services well beyond the project. Higher fares, limited need in urban settings, social limitations on handling pregnant women, limited use of referral slips, and no mechanism for feedback on transporter services are the key barriers to greater utilization. QITs: The two QITs the evaluation team interviewed were meeting regularly. However, QIT members are not clear on their purpose, identify actions that are outside their scope of improving access to and quality of services, and identify gaps in service delivery that are beyond the control of the QIT—all of which contribute to frustration, loss of interest by members, and limited community ownership. Pilot Interventions: The project has demonstrated that high-impact, evidence-based MNCH interventions (HBB, CXH/miso, and infection prevention) can be operationalized in Sindh province with the full participation of public and private healthcare providers. The national government placed CHX on the national essential drug list in record time due to the project’s and partners’ advocacy. Reaching Poor and Marginalized Populations: The project worked with DoH and PPHI facilities that serve rural, poor, and marginalized communities. The project’s success in strengthening the MNCH PROJECT MID-TERM EVALUATION 28 capacities of these facilities implies that it improved access to and quality of services to the poor and marginalized. Ninety-seven percent of the DOH managers and healthcare providers the evaluation team interviewed reported that project-supported healthcare providers are reaching poor and marginalized populations. The Mother’s Support Card initiative also facilitates marginalized and poor women’s access to services. The initiative subsidizes the cost of a package of MNCH services to encourage marginalized and poor pregnant women to access a complete package of care including attending ANC, delivery, and PNC at CMW clinics. QUESTION 2: RECOMMENDATIONS Question 2: What changes could be made to increase the effectiveness in remaining years? Training Institutes: To ensure sustainability, the training institutes should be fully proficient as trainers for the project’s focus interventions. To support sustainable change, the project should focus on developing master trainers within the OB/GYN departments of the respective training institutes. BEmONC Health Facilities: The project should identify the gaps in BEmONC facilities’ capacities to render the seven “signal” functions. When human resources are the barrier, the project should advocate with the DoH to make the resources available. CEmONC Health Facilities: Through staff training and OJC, the project has improved the quality of services in blood bank management and infection prevention. To sustain these gains, the project will need to advocate with the DoH to provide the funding necessary to ensure an adequate quantity of the essential supplies and procure and maintain needed equipment. LHWs: As requested by the LHW program, the project should consider making the mother’s booklet available for distribution to other districts in Sindh. CHWs: The project should continue advocating for CHWs’ inclusion in the LHW program after the project ends. The government of Sindh’s health sector strategy (2012–2020) describes plans to increase LHW coverage from 45 percent to 80 percent, which implies increasing the number of LHWs.28 Since CHWs are from the communities, have received some training, and have started to build connections within the communities and conduct WSGs, they may have an edge over other applicants if the province does expand the LHW program. However, at this point in time, the project has no formal agreement with the provincial government to incorporate, or give preference to, CHWs when expanding the LHW program. CMWs: The project’s training and support to establish CMW clinics has been successful. With the resources freed up by PPHI’s departure, the project should consider supporting CMWs to establish more clinics. The project is now collaborating directly with midwifery schools, and this provides a direct link and opportunity to collaborate with the upcoming batch of CMWs. The project could offer CMW business training within the school curricula. Once CMWs graduate, the project can help them establish their facilities. Mother’s Support Card: The project should evaluate the Mother’s Support Card initiative at the end of the intervention to determine whether it is an effective model for wider application. QITs: Existing QITs need to have a clear understanding of their purpose, the objectives of their action plans, and follow-up mechanisms. They need to focus their activities on improving the demand for and 28 Sindh Health Sector Strategy, 2012-2020. Accessed from: http://www.trfpakistan.org/LinkClick.aspx?fileticket=1EyZSVfIMkg%3D&tabid=2618 MNCH PROJECT MID-TERM EVALUATION 29 quality of healthcare services and on activities they can realistically complete within their available resources. If the project, or a future project, works with QITs, it should provide the initial support necessary to firmly establish their function. Miso/CHX: Since the distribution of CHX/miso is an effective and successful model, the project should work with the LHW program to include CHX/miso distribution in LHWs’ regular work throughout Sindh. Other partners or future projects could support this through focused advocacy, ensuring availability, and other support until it is completely supported by public health policy. HBB: Overall, HBB is a well-managed and important component of the program, and it has undoubtedly resulted in fewer newborn deaths. The project should scale up HBB as much as possible within the 15 districts during the time remaining in the project. Future projects could scale it up throughout Sindh. To facilitate scale-up, all training institutes and CMW schools should have TOT capacity in HBB. Immunization: The project should provide a clear plan of action to ensure that at least one complete cohort of newborns is enrolled and begin its vaccinations in time for all the babies to receive their final first-year measles vaccination at around 9 months of age or to be completely vaccinated by their first birthdays. The project should ensure that performance indicators and measurements are implemented to determine whether the intervention has been effective in improving immunization coverage. Measurement and Documentation: To develop lessons learned about providing technical support to facilities, project reports should differentiate between aspects of the project that influence QOC and external factors over which the project has no control. The project should aggregate and analyze monitoring data at the project level against baseline conditions to identify quantifiable changes, especially if there is uncertainty about the value or impact of an intervention or activity. The project should complete all research activities and share results with stakeholders before the end of the project in September 2017. Exit Strategy: The project must develop a detailed exit strategy, especially given the large number of personnel in the project (over 200 project staff plus many implementing partner staff). This plan should be developed in consultation with the Sindh DoH (including technical line managers), other stakeholders, and project partners. QUESTION 3: BEST PRACTICES AND LESSONS LEARNED Question 3: What are the best practices, innovations, and lessons learned that can be applied to other existing or future programing in MNCH service delivery? The project demonstrated how targeted assistance and relatively simple interventions (e.g., providing equipment and supplies) can make significant contributions to improving MNCH services and saving lives. Furthermore, providing initial trainings at the training institute followed by regular OJC is an approach that can be institutionalized for all sorts of healthcare service delivery programs and is not limited only to MNCH services. Given that 80 percent of women who deliver with an SBA deliver in private facilities, improving practices in private facilities is essential to improving MNCH outcomes more broadly. Technical assistance to public sector facilities also benefits private facilities because many public sector health workers have second jobs in private facilities. MNCH PROJECT MID-TERM EVALUATION 30 Assisting CMWs to establish and develop their own clinics is another best practice. The project has learned valuable lessons in helping CMWs become effective SBAs and MNCH services providers who are currently assisting the national MNCH program and could continue in the future. Placing CMW coordinators in the training hospitals to facilitate training and mentoring was an important intervention that substantially improved the effectiveness of the CMW schools in building the skills of SBAs. The project’s emphasis on quality of care and introducing a culture of quality is a best practice and was appreciated by healthcare providers. Assessing service quality, potentially using the QIPS, can be a valuable tool. If used to measure program effect, it should use indicators that are within the target facilities’ ability to address. The value of facility self-assessments, followed by OJC and OJT, has been well documented; how they can be used as a motivational tool to achieve outcomes remains to be documented. Representatives of the OB/GYN department of the DHQ hospital in Sukkur told the evaluators that due to the establishment of the training institute, the department has been able to fulfill the requirements of the Pakistan Medical and Dental Council (PMDC) and the College of Physicians and Surgeons (CPSP) and become recognized as a training site for postgraduate fellowship OB/GYN trainees. This is an important achievement and, even though not a specific objective of the project, it indicates that professional organizations recognize the value of the training institute to improving quality of care. The department has already submitted a request to PMDC for an inspection so they can receive official recognition of this new capacity. Most private CEmONC facilities do not have an EPI center. To improve newborns’ access to their first immunizations, the project’s district coordinator and CEmONC coordinator encouraged the private Balquis Musaf Hospital in Tando Allahyar to use its own funds to establish an EPI center. The practice of providing a mobility allowance for DoH vaccinators to travel to their catchment areas may overcome a major barrier to delivering routine immunizations. The HSS project introduced the practice based on its calculations of the cost vaccinators would incur to reach the communities in their catchment area. RSPN is now providing one liter of petrol per day to each vaccinator. This allocation will permit vaccinators to perform their duties for about Rs. 400 per month. This small yet important intervention has the potential to increase routine immunization, assuming all vaccines and the cold chain remain available. The DoH liked the approach and believed it could easily operationalize its entire outreach program for routine immunization within its district health budget. MNCH PROJECT MID-TERM EVALUATION 31 ANNEXES Annex 1: Scope of Work MNCH PROJECT MID-TERM EVALUATION 32 MNCH PROJECT MID-TERM EVALUATION 33 MNCH PROJECT MID-TERM EVALUATION 34 MNCH PROJECT MID-TERM EVALUATION 35 Annex 2: Assignment Work Plan MNCH PROJECT MID-TERM EVALUATION 36 MNCH PROJECT MID-TERM EVALUATION 37 MNCH PROJECT MID-TERM EVALUATION 38 MNCH PROJECT MID-TERM EVALUATION 39 MNCH PROJECT MID-TERM EVALUATION 40 MNCH PROJECT MID-TERM EVALUATION 41 MNCH PROJECT MID-TERM EVALUATION 42 MNCH PROJECT MID-TERM EVALUATION 43 MNCH PROJECT MID-TERM EVALUATION 44 MNCH PROJECT MID-TERM EVALUATION 45 MNCH PROJECT MID-TERM EVALUATION 46 MNCH PROJECT MID-TERM EVALUATION 47 MNCH PROJECT MID-TERM EVALUATION 48 MNCH PROJECT MID-TERM EVALUATION 49 MNCH PROJECT MID-TERM EVALUATION 50 MNCH PROJECT MID-TERM EVALUATION 51 MNCH PROJECT MID-TERM EVALUATION 52 MNCH PROJECT MID-TERM EVALUATION 53 MNCH PROJECT MID-TERM EVALUATION 54 MNCH PROJECT MID-TERM EVALUATION 55 MNCH PROJECT MID-TERM EVALUATION 56 MNCH PROJECT MID-TERM EVALUATION 57 Annex 3: Conflict of Interest Statements The conflict of interest disclosures were removed to preserve the confidentiality of evaluation team members. They are available from PERFORM on request. MNCH PROJECT MID-TERM EVALUATION 58 Annex 4: List of Persons Interviewed The list of persons interviewed was removed to protect the confidentiality of respondents. MNCH PROJECT MID-TERM EVALUATION 59 Annex 5: Bibliography Agha, S. (2013). Quality of Care Study Research Plan. Agha, S. (2013). Quality of Care for prevention, identification, and management of common serious Maternal & early Neonatal Complications in Health Facilities in Sindh, Pakistan. Jhpiego. Agha, S., & Smith, J. (2015). Chlorhexidine and Misoprostol Research Plan. Currie, S. (2014). MCHIP Program Trip Report. Fitzgerald, L. (2013). MCHIP Program Trip Report. Fitzgerald, L. (2014). MCHIP Program Trip Report. Fitzgerald, L. (2015). MCHIP Program Trip Report. Fogarty, L. (2013). MCHIP Program Trip Report. Galloway, R. (2014). MCHIP Program Trip Report. Galloway, R. (2015). MCHIP Program Trip Report. GOP. (2015). Chlorhexidine Policy Brief. GOP. (2015). Government Notification on CHX. Government, Sindh. (2016). Minutes of Joint Monitoring Visits. Graham, K. (2013). MCHIP Program Trip Report. Hamid. (2015). MCHIP Program Trip Report. Hashim, A. (2016). MCHIP Program Trip Report. Jhpiego. (2016). District Based Organogram . Jhpiego. (2016). Headoffice Organogram. Jhpiego. (2016). Mid-term Evaluation. Jhpiego. (n.d.). Data Collection Tools. Jhpiego. (n.d.). MET Presentation for MCHIP Evaluation. KMC. (2016). HFA Findings in two CEmONC Hospitals. Lathrop, L. (2015). MCHIP Program Trip Report. MCHIP. (2013). Sampling Methodology for MCH HH Survey. MCHIP. (2014). Pre Award Assesments. MCHIP. (2014-2015-2016). Annual Workplans. MCHIP. (2015). MCH Program Indicator Survey Summary Card. MCHIP. (2015-2016). Quality Assurance Team Formatioan Reports. MCHIP. (2016). Brief on MCHIP's Initiative to Establish EPI CEnters in Private CEmONC Facilities through PPP. MCHIP. (2016). MCHIP Staff JDs. MCHIP. (2016). Non-Structural Adjustments. MCHIP. (2016). Sub Agreements. MNCH PROJECT MID-TERM EVALUATION 60 MCHIP. (May 18, 2016). MNCH Overall Presentation. MCHIP. (May 19, 2016). Community Mobilization & Newborn Health. MCHIP. (May 20, 2016). Field Program Overview. MCHIP. (May 20, 2016). M & E Presentation. MCHIP. (May 23, 2016). Grants and Finance Presentation. MCHIP. (May 23, 2016). Operation Presentation. MCHIP. (n.d.). QOC Pages with Selective Findings. MHCIP. (2016). Overall Project Presentation. MHCIP. (2016). Scope of Work for Operations Research. PPHI. (2014). MOU between Jhpeigo and PPHI. PPHI. (2016). PPHI Memo to MCHIP. Russo, S. (2013). MCHIP Program Trip Report. Smith, J. (2013). MCHIP Program Trip Report. Taylor, A., Tsega, D., & Khan, D. (2013). Immunization Planning Mission. Taylor, P. (2013). MCHIP Program Trip Report. USAID. (2013). Training Documents. MCHIP. USAID. (2013-2014). Maternal and Child Health (MCH) Household Survey Data set and Reports. MCHIP. USAID. (2013-2014-2015). Maternal, Newborn, and Child Health (MNCH) Services Component, Annual Report. MCHIP. USAID. (2014-2015-2016). Quarterly Reports. MCHIP. Varallo, J. (2015). MCHIP Program Trip Report. MNCH PROJECT MID-TERM EVALUATION 61 Annex 6: Data Collection Instruments Key Informant Interview with Training Institute on MCHIP Training Component Serial number Name of training institute Researcher name Start time: hh:mm (AM/PM): District name: End time: hh:mm (AM/PM): Tehsil name: Interview date: DD/MM/YY City /village name: UC name: Respondent name Respondent designation Respondent organization Contact number Email Location of interview Questions 1. Does the training institute have TORs or an MOU for collaborating with the MCHIP project? (Circle one number) 1 Yes 2 No Types of Training 2. What types of training does your institute offer to MCHIP? (Write a response) 3. Do you regularly provide refresher trainings? (Circle one number) 1 Yes 2 No MNCH PROJECT MID-TERM EVALUATION 62 Please explain (Write a response) 4. Do you give certificates for all training? (Circle one number) 1 Yes 2 No Please explain (Write a response) Selection of participants 5. Who selects participants? What different cadres are trained? (Write a response) 6. What criteria are used to select participants? (Write a response) 7. For which criteria would you suggest having some flexibility in applying the criteria? (Write a response) Curriculum / Content of the training 8. Are there specific modules for each training? (Circle one number) 1 Yes 2 No If no, what are the bases use for conducting the training? (Write a response) 9. In your opinion, are the curricula adequate to develop core competencies in CMWs? (Circle one number) 1 Yes 2 No MNCH PROJECT MID-TERM EVALUATION 63 Please explain (Write a response) 10. Are the curricula relevant to the scope of work of CMWs? (Circle one number) 1 Yes 2 No Please explain (Write a response) Methodology 11. In your opinion, was the classroom-based training approach effective? If not, why not? (Write a response) 12. In your opinion, was the clinic-based training approach effective? If not, why not? (Write a response) 13. In your opinion, what are the problems, if any, with the training methods? (Write a response) Infrastructure 14. Is available infrastructure adequate for training the required numbers of participants? (Circle one number) 1 Yes 2 No If “no”, please explain (Write a response) 15. What, if any, specific infrastructure can hamper the training quality and how? (Write a response) MNCH PROJECT MID-TERM EVALUATION 64 16. Are the clinical training sites generally adequate for their purpose? (Circle one number) 1 Yes 2 No Please explain (Write a response) 17. What contribution, if any, does MCHIP make to improving infrastructure at clinical training sites? (Write a response) 18. Do you have any suggestions for improving the skill training for MCHIP Project? (Write a response) Human Resources 19. Were master trainers trained for specific components? (Circle one number) 1 Yes 2 No If “yes”, which curriculum did you use to train them? (Write a response) 20. How many master trainers in the institute have received training under the TOT activity? (Enter number trained) ___________ Master trainers trained 21. How many master trainers are TOT qualified? (Enter number TOT qualified) ___________ Master trainers TOT qualified 22. Do you have any suggestions for making master trainers more effective? (Write a response) MNCH PROJECT MID-TERM EVALUATION 65 23. In your opinion, are the skills of the master trainers adequate to deliver effective trainings? (Write a response) Equipment, Supplies, and Logistics 24. Are the available equipment, supplies, and logistics adequate for training the required number of participants? (Circle one number) 1 Yes 2 No If “no”, please describe deficiencies? (Write a response) 25. In your opinion, how do these deficiencies negatively affect training quality? (Write a response) 26. What, if any, contribution has MCHIP made to improving equipment, supplies, and logistics? (Write a response) Examination System 27. Is the examination robust enough to pass only those who have acquired adequate competencies? (Circle one number) 1 Yes 2 No Please explain. (Write a response) 28. In your opinion, what are the strengths and weaknesses of the examination systems? (Write a response) MNCH PROJECT MID-TERM EVALUATION 66 29. What suggestions, if any, do you have for improving the examination system for MCHIP project participants? (Write a response) General Issues 30. In your opinion, what strategies, if any, are required to maintain the quality of services? (Probe: refresher courses, incentives, etc.) (Write a response) 31. If refresher courses are required? (Circle one number) 1 Yes 2 No If “yes”, how frequently? (Enter number and units, e.g., “2 times per month”) __________________________ If “yes”, how should candidates for refresher course be identified? (Write a response) 32. In your opinion, what are the main strengths and successes of the MCHIP project training to date? (Write a response) 33. What are your main concerns, if any, about the training process? (Write a response) 34. What practices would you like to see put in place to improve future training? (Write a response) Field Monitoring 35. Is the institute involved in monitoring trained participants? (Circle one number) 1 Yes 2 No MNCH PROJECT MID-TERM EVALUATION 67 Please explain (Write a response) 36. Is the institute involved in the On-Job-Training component of the MCHIP project? (Circle one number) 1 Yes 2 No Please explain (Write a response) Signature of interviewer: ______________________________________ MNCH PROJECT MID-TERM EVALUATION 68 Key Informant Interview Guide for District Level Managers MCHIP Serial number Start time: hh:mm (AM/PM): Interviewer name End time: hh:mm (AM/PM): District name Interview date: DD/MM/YY Respondent name Respondent designation Respondent organization Contact number Email Location of interview Questions / discussion points 1. What role, if any, have you played in implementation of MCHIP? What activities are being implemented and at what level? 2. What is the SOW of partner organizations with MCHIP in the district? 3. What other projects in the district are offering support for improving MNCH services? How, if at all, is MCHIP collaborating with these projects? 4. Has the MCHIP program been effective and, if so, how? (Has it worked)? Probe: Achievement of component outputs and objectives, on target? Changes in services (positive and negative)? Impact on service providers (preparedness)? Satisfaction with quality of outcomes, give examples? 5. Is the program being implemented efficiently (on time, on budget, and managed efficiently)? MNCH PROJECT MID-TERM EVALUATION 69 Probe. On time (disbursement and utilization of funds, supplies, equipment, etc.)? Do outputs justify costs (value for money)? Is TA appropriate (timely, adequate, needs based, quality)? Is support/logistics adequate (monitoring supervision, feedback, MIS, training and clinical support)? Integration/synergy with other programs (government and donor). Gaps and duplication in the program? Are there better options – best practices & innovations? 6. What are the mechanisms, if any, for enhancing collaboration and coordination with DoH, PPHI, and other stakeholders? Probe: Any joint committees, events, etc.? 7. Based on the MCIP activities to-date do you think the program is sustainable? If not what step should be taken to make it sustainable? Probe: What steps have been taken to support future activities (partnerships, networks and linkages)? 8. Do you think the MCHIP program has contributed towards improving MNCH in the district? Probe: What has happened as a result of the program (difference in lives of beneficiaries, poor and marginalized)? Progress with immediate outcomes? Roles of partners in realizing outcomes? 9. Is the program reaching those most in need? Probe: Poor and most marginalized, gender – rights based approach? 10. What lesson, if any, have been learnt to date? Probe: Planning, operational level, management and community level. MNCH PROJECT MID-TERM EVALUATION 70 11. What are the current issues, if any, facing implementation of the MCHIP project? What are the solutions? 12. Do you have any further comments or suggestions? Any best practices or innovations? Signature of interviewer: _______________________________ MNCH PROJECT MID-TERM EVALUATION 71 Key Informant Interview Guide CMW Training School Serial number Name of training institute Researcher name Start time: hh:mm (AM/PM): District name: End time: hh:mm (AM/PM): Tehsil name: Interview date: DD/MM/YY City /village name: UC name: Respondent name Respondent designation Respondent organization Contact number Email Location of interview Questions Does the training institute have TORs or an MOU for collaborating with the MCHIP project? (Circle one number) 1 Yes 2 No 1. What role have CMW training schools played in preparing the CMWs to practice in the communities? (Explain) Probe: What support is provided to CMWs after they graduate? (Explain) Probe: Is it addressing the needs at the community level? (Explain) MNCH PROJECT MID-TERM EVALUATION 72 2. What role, if any, has the MCHIP project played in supporting the training of CMWs? (Explain) Probes: o Theory, clinical support o Any addition in the curricula (HBB, Business Model, CHX)? o Standards to be followed 3. How, if at all, has MCHIP facilitated the CMWs to perform better at the community level? Probes: o Getting the clinic established o Providing supplies o Capacity building (HBB, business model, CHX) 4. Has MCHIP facilitated CMWs to develop linkages with other cadres at the community level? (Circle one number) 1 Yes 2 No If yes, how? (Explain) Probes: o Resolving any issues between different cadres? o How were they, or should they, be addressed? 5. What are the 3 biggest issues CMWs face? (Explain) Are these issues being addressed and are the steps taken to address the issues working? (Explain) MNCH PROJECT MID-TERM EVALUATION 73 If the issues are not being addressed, what do you believe needs to be done and by whom? (solutions) (Explain) Signature of interviewer: ______________________ MNCH PROJECT MID-TERM EVALUATION 74 Key Informant Interview Guide District Clinical Supervisor (MCHIP) Newborn & Child Health Coordinator (Save the Children) CEmONC Supervisor Serial number Start time: hh:mm (AM/PM): Interviewer name End time: hh:mm (AM/PM): District name: Interview date: DD/MM/YY Respondent name Respondent designation Respondent organization Contact number Email Location of interview Interview questions 1. What role, if any, have you played in implementation of MCHIP? What activities are being implemented and at what level? 2. What is your specific scope of activities for the MCHIP project? (training, monitoring, mentoring (OJT), etc. o At the community level? o CMW facility level? o BEmONC facility level? o CEmONC facility level? o At training institute? o At CMW school? 3. Do you have specific tools for supervising each level of the activities you are responsible for? If yes kindly share. How do you decide which trainings to offer at what level? How do you determine if these trainings have been effective? MNCH PROJECT MID-TERM EVALUATION 75 How do you decide when to offer OJT and/or clinical supportive supervision? How do you tell if it has addressed the reasons you decided to offer either one of them? 4. Is there a feedback mechanism for incorporating your findings in the district level MCHIP project plan? Do you share your findings with your DoH, PPIH or Private Facility managers? If not yet, could this be done? 5. For Save the Children representative only: Newborn and Child Health Coordinator To whom do you report (at district and headquarter level? How do you operate within you position? What is coordination mechanism with other staff present at district? Is it a one MCHIP team or different teams operating at same time? 6. In your opinion, what has the MCHIP program done that has been most effective in reducing mortality and morbidity for pregnant women, newborns and children? if yes, how? (Has it worked)? How can you tell they have been effective? If you do not believe some activities have been effective, why not? What could MCHIP do (within their responsibilities) that could make them more effective? If there are factors outside of the control of the project (i.e. the responsibility of MCHIP) that have an influence on MCHIP’s effectiveness (positive or negative), what are they? Probe: Achievement of component outputs and objectives, on target? Changes in services (positive and negative)? Impact on service providers (preparedness)? Satisfaction with quality of outcomes, give examples? 7. What and how timely the district staff is getting support from MCHIP headquarters? Are there issues if that support doesn’t come on time? If yes: How does it affect their relations with public sector authorities (including PPHI)? What are its effects on the program implementation? How often are supportive supervisory visits are done by the headquarter staff, etc.? What happens during these supervisory visits? Do you collect any information that is entered into any databases or monitoring done by MCHIP? If yes, what kind and how often? Probe. On time (disbursement and utilization of funds, supplies, equipment, etc.)? Is TA appropriate (timely, adequate, needs based, quality)? Is support/logistics adequate (monitoring supervision, feedback, MIS, training and clinical support)? Integration/synergy with other programs (government and donor). Gaps and duplication in the program? Are there better options – best practices & innovations? MNCH PROJECT MID-TERM EVALUATION 76 8. In your opinion, what activities that MCHIP is doing will be sustainable? (For each answer, ask why them the reason that they think it is sustainable. For each answer “no” ask them what can be done to make them sustainable. For yes and no questions, be specific. Probe: What steps have been taken to support future activities (partnerships, networks and linkages) 9. Do you think the MCHIP project overall has had an impact on delivery of high-quality MNCH, FP, nutrition, immunization (as applicable); how do you know? If not, why not? Do you think that these high-quality services will reach 80% of your district by end of September 2017? If not, will some services reach, but not others? (specify which ones) If yes, how will you know that this has been reached with regard to the services you are personally working on? If no, what could be done in the next 1-2 years to insure 80% are reached? Probe: What has happened as a result of the program (difference in lives of beneficiaries, poor and marginalized)? Progress with immediate outcomes? Roles of partners in realizing outcomes? 10. Is the program reaching the most in need? In your opinion, is the program reaching the “hardest to reach” or “most needy” mothers, children and families? If yes, please give examples, if not why not? For yes, how do you know? How are they being reached? If no, what needs to be done to reach them? Probe: Poor and most marginalized, gender? 11. Are you engaged in any of the research activities of the project? (HBB, CXT, Miso, IFA, KMC?) If yes, what is your role? With whom do you coordinate for the research activities? What lessons, if any, have been learnt to date regarding providing high quality integrated MNCH, FP, nutrition and immunization services in Sindh Province (address only those interventions that apply)? Probe: Planning, operational level, management, and community level? Demand MNCH PROJECT MID-TERM EVALUATION 77 Supply Strengthening Referral Services 12. What are the current issues, if any, you are facing regarding implementation of MCHIP project activities that are part of your job? What could be solutions in the next 1 -2 years that would make a difference? Are there some activities that you think could be discontinued 1) because they have already achieved the desired impact MCHIP efforts can be redirected to other activities? or 2) the approach being used is not the one needed to solve the problem? Or 3) other reasons (specify) 13. Do you have any further comments, suggestions? Signature of interviewer: _________________________________ MNCH PROJECT MID-TERM EVALUATION 78 Key Informant Interview Guide District Managers (DoH and PPHI); District Specific Program Managers (EPI, MNCH, Nutrition, DPWO) Serial number Start Time: hh:mm (AM/PM): Interviewer name End Time: hh:mm (AM/PM): District name Interview Date: DD/MM/YY Respondent name Respondent designation Respondent organization Contact number Email Location of interview Questions 1. Are you aware of or familiar with the MCHIP project? Probe: Does he/she know about the goal/objectives/role of MCHIP? Was he/she ever oriented regarding MCHIP? 2. What role, if any, have you played in operationalization of the MCHIP project? Probe: What projects and/or support are you offering for operationalization of MCHIP in the district? 3. Is MCHIP meeting its objectives effectively? Is MCHIP important for strengthening health facilities in the district? Probe Does it meet the needs of health professionals? Is MCHIP as a project relevant to strengthening health services, i.e. by providing in service training, capacity development, and introducing new interventions to make health facilities quality MNCH facilities? MNCH PROJECT MID-TERM EVALUATION 79 4. Has MCHIP been effective in strengthening the following health services at the facility and community level and, if yes, how effective? Probe: Changes in services (positive and negative), Impact on service providers (preparedness)? Satisfaction with quality of outcomes, give examples? o Immunization services ___________________________________________________ o CEmONC____________________________________________________________ o BEmONC____________________________________________________________ o Nutrition_____________________________________________________________ o Support to LHWs ______________________________________________________ o Support to CMWs _____________________________________________________ o Referal/Transporters____________________________________________________ o Family Planning ________________________________________________________ 5. Do you think MCHIP is working efficiently at present (activities as scheduled and on time, in line with requirements, and managed efficiently)? Probe. On time? (disbursement and utilization of funds, supplies, equipment, etc.)? Do outputs justify costs (value for money)? Is TA appropriate (timely, adequate, needs based, quality)? Is support/logistics adequate (monitoring supervision, feedback, training and clinical support)? Integration/synergy with other programs (government and donor)? Management and institutional arrangements since devolution (coordination)? Are there other better options available that can be replicated – best practices & innovations? 6. Do you consider MCHIP a sustainable intervention? Probe: Can the DoH/PPHI sustain MCHIP effectively and efficiently? 7. Do you think MCHIP interventions, i.e. trainings, have had any impact on overall health service delivery and, if yes, how? Probe: What has happened as a result of the program training, etc.? Progress with immediate outcomes? Roles of partners in realizing outcomes? 8. What are the gaps/issues, if any, from the MCHIP project to date? What are the possible solutions to the gaps/issues you identified? Probe: Planning, operational level, and management level? MNCH PROJECT MID-TERM EVALUATION 80 9. What MCHIP activities would you adopt as an integral part of the DoH/PPHI in the future? Probe: Planning, operational level, and management level? 10. Do you have any comments or suggestions? Signature of interviewer: _______________________________________ MNCH PROJECT MID-TERM EVALUATION 81 Key Informant Interview Guide Government and Stakeholder Interview Tool Provincial Level Serial number Start time: hh:mm (AM/PM): Interviewer name End time: hh:mm (AM/PM): District name Interview date: DD/MM/YY Respondent(s) name Respondent(s) designation Respondent organization Contact number Email Location of interview Introduction into purpose of the evaluation and request from respondent to address the following with regards to MCHIP MNCH project MNCH PROJECT MID-TERM EVALUATION 82 1. To the best of your knowledge, what is the purpose of the MCHIP MNCH project? (Check all that apply) 2. (Ask only if checked in question 1) Does your organization collaborate with MCHIP on this topic? (Check if “yes”) (Ask only if checked in question 2) 3. Please List areas of collaboration. (Ask only if checked in question 2) 4. Please explain who is managing this collaboration and at what level. a) Training in antenatal care   __________________________ __________________________ ____________________________ ____________________________ b) Training in normal delivery   __________________________ __________________________ ____________________________ ____________________________ c) Training in managing pregnancy complications   __________________________ __________________________ ____________________________ ____________________________ d) Training in postnatal care   __________________________ __________________________ ____________________________ ____________________________ e) Training in family planning (specify methods)   __________________________ __________________________ ____________________________ ____________________________ f) Training in care of the newborn   __________________________ __________________________ ____________________________ ____________________________ MNCH PROJECT MID-TERM EVALUATION 83 1. To the best of your knowledge, what is the purpose of the MCHIP MNCH project? (Check all that apply) 2. (Ask only if checked in question 1) Does your organization collaborate with MCHIP on this topic? (Check if “yes”) (Ask only if checked in question 2) 3. Please List areas of collaboration. (Ask only if checked in question 2) 4. Please explain who is managing this collaboration and at what level. g) Training in managing premature delivery   __________________________ __________________________ ____________________________ ____________________________ h) Care of the sick child (pneumonia & diarrhea)   __________________________ __________________________ ____________________________ ____________________________ i) LHW training in (list topics)   __________________________ __________________________ ____________________________ ____________________________ j) Health facility quality of care   __________________________ __________________________ ____________________________ ____________________________ k) Immunization   __________________________ __________________________ ____________________________ ____________________________ l) Health human resources (training skills)   __________________________ __________________________ ____________________________ ____________________________ MNCH PROJECT MID-TERM EVALUATION 84 1. To the best of your knowledge, what is the purpose of the MCHIP MNCH project? (Check all that apply) 2. (Ask only if checked in question 1) Does your organization collaborate with MCHIP on this topic? (Check if “yes”) (Ask only if checked in question 2) 3. Please List areas of collaboration. (Ask only if checked in question 2) 4. Please explain who is managing this collaboration and at what level. m) Supportive supervisions   __________________________ __________________________ ____________________________ ____________________________ n) Training in blood banking   __________________________ __________________________ ____________________________ ____________________________ o) Training in anesthesia   __________________________ __________________________ ____________________________ ____________________________ p) Training in AMSTL   __________________________ __________________________ ____________________________ ____________________________ q) Training in Misoprostol   __________________________ __________________________ ____________________________ ____________________________ r) Training in Chlorhexidine use   __________________________ __________________________ ____________________________ ____________________________ s) Training in HBB   __________________________ __________________________ ____________________________ ____________________________ MNCH PROJECT MID-TERM EVALUATION 85 1. To the best of your knowledge, what is the purpose of the MCHIP MNCH project? (Check all that apply) 2. (Ask only if checked in question 1) Does your organization collaborate with MCHIP on this topic? (Check if “yes”) (Ask only if checked in question 2) 3. Please List areas of collaboration. (Ask only if checked in question 2) 4. Please explain who is managing this collaboration and at what level. t) Women’s Support Groups   __________________________ __________________________ ____________________________ ____________________________ u) Quality Improvement Teams   __________________________ __________________________ ____________________________ ____________________________ v) PDQ   __________________________ __________________________ ____________________________ ____________________________ w) BCC (list topics and groups)   __________________________ __________________________ ____________________________ ____________________________ x) Maternal and child nutrition   __________________________ __________________________ ____________________________ ____________________________ y) Quality of care facility assessments   __________________________ __________________________ ____________________________ ____________________________ z) Strengthening referral systems   __________________________ __________________________ ____________________________ ____________________________ MNCH PROJECT MID-TERM EVALUATION 86 5. a. Do you participate in joint activities with the MCHIP project? (Please circle the correct number) Yes No 1 2 b. If yes, what are they? (Describe) c. Where do they take place? How often? (Describe) 6. If no, is there a reason you are not working with or collaborating with MCHIP? 7. Are there other opportunities where MCHIP could collaborate with you, but they are not currently doing so? 8. For MCHIP activities that you know, can you comment on the impact they have had or not had on the following topic areas in Sindh Province? (Please circle the correct number) Topic Area Has Impact No Impact If “impact”, please explain the impact Increasing population coverage in demand, supply or referrals for quality MNCH 1 2 Increasing population coverage in demand, supply or referrals for quality FP 1 2 Increasing population coverage in demand, supply or referrals for quality nutrition 1 2 MNCH PROJECT MID-TERM EVALUATION 87 Topic Area Has Impact No Impact If “impact”, please explain the impact Increasing population coverage in demand, supply or referrals for immunization services 1 2 9. a. Are you familiar with any local NGO partners that are part of the MCHIP program? (Please circle the correct number) Yes No 1 2 b. If yes, can you comment on the effectiveness of the activities of these local partners on the performance of the MCHIP program? 10. a. Do you think there will be any aspects of the MCHIP program that will be sustained after September 2017? Yes No 1 2 b. If yes, what do you think they will be? c. If not, are there changes MCHIP could make that could have a lasting impact on the quality and coverage of MNCH, FP, immunization or Maternal and Child nutrition? 11. Do you have any additional comments about the MCHIP MNCH program? Signature of interviewer: _______________________________________ MNCH PROJECT MID-TERM EVALUATION 88 Key Informant Interview Guide for BEmONC Health Facility Staff Serial number Facility name MCHIP facility ID Start time: hh:mm (AM/PM): Researcher name End time: hh:mm (AM/PM): District name: Interview date: DD/MM/YY Tehsil name: UC name: City /village name: Entered by: Respondent name Respondent designation Respondent organization Contact number Email Location of interview Interviewer: I have informed the respondent about the evaluation scope and s/he has consented to the interview. Interviewer signature: ___________________________________ 1. GENERAL INFORMATION Type of health facility: 1. DHQH____________________ 2. THQH ________________________ Name of health facility: Name of district: Name of respondent: Designation: MNCH PROJECT MID-TERM EVALUATION 89 2. STAFF Trained and Type of Training MNCH PROJECT MID-TERM EVALUATION 90 Questions 3. What role have you played in implementation of MCHIP? What activities are being implemented at this health facility and at what level? 4. What other projects in the district are offering support for improving MNCH services? How, if at all, is MCHIP collaborating with these projects? 5. Has the MCHIP program been effective and, if so, how, i.e., has it worked? Probe: Achievement of component outputs and objectives, on target? Changes in services (positive and negative)? Impact on service providers (preparedness)? Satisfaction with quality of outcomes, infrastructure, supplies, etc., give examples. 6. Is the program being implemented efficiently (on time, on budget, and managed efficiently)? Probe. On time (disbursement and utilization of funds, supplies, equipment, etc.)? Do outputs justify costs (value for money)? Is TA appropriate (timely, adequate, needs based, quality)? Is support/logistics adequate (monitoring supervision, feedback, MIS, training, and clinical support)? Integration/synergy with other programs (government and donor? Gaps and duplication in the program? Are the better options – best practices & innovations? 7. Do you think the program is sustainable? Probe: What steps have been taken to support future activities (partnerships, networks, and linkages) 8. What are the mechanisms, if any, for enhancing collaboration and coordination with DoH, PPHI, and other stakeholders? Probe: Any joint committees, events, etc.? MNCH PROJECT MID-TERM EVALUATION 91 9. Is the program reaching those most in need? Probe: Poor and most marginalized, gender? 10. Do you think the MCHIP program has had an impact on MNCH; how do you know? Probe: What has happened as a result of the program (difference in lives of beneficiaries, poor and marginalized)? Progress with immediate outcomes? 11. What lessons, if any, have been learnt to date? Probe: Planning, operational level, management, and community level? 12. What are the current issues facing implementation of the MCHIP project? What are the solutions? Do you have any further comments, suggestions? Signature of the respondent: ________________________ MNCH PROJECT MID-TERM EVALUATION 92 Key Informant Interview Guide for CEmONC Health Facility Staff Serial number Facility name MCHIP facility ID Start time: hh:mm (AM/PM): Researcher name End time: hh:mm (AM/PM): District name: Interview date: DD/MM/YY Tehsil name: UC name: City /village name: Entered by: Respondent name Respondent designation Respondent organization Contact number Email Location of interview Interviewer: I have informed respondent about the evaluation scope and s/he has consented to the interview. Interviewer signature: ___________________________________ 1. GENERAL INFORMATION Type of health facility: 1. DHQH____________ 2. THQH ___________ 3. Private: ___________ Name of health facility: Name of district: Name of respondent: Designation: MNCH PROJECT MID-TERM EVALUATION 93 2. STAFF Trained and Type of Training MNCH PROJECT MID-TERM EVALUATION 94 Questions 13. How Jhpiego has been helpful to you in improving availability and quality of service delivery at this particular health facility?’ what are the key inputs provided by Jhpiego?” 14. What other projects in the district are offering support for improving MNCH services? How, if at all, is MCHIP collaborating with these projects? 15. Has the MCHIP program been effective and, if so, how? (Has it worked)? Probe: Achievement of component outputs and objectives, on target? Changes in services (positive and negative)? Impact on service providers (preparedness)? Satisfaction with quality of outcomes, infrastructure, supplies, etc., give examples? 16. What are the activities implemented by MCHIP that are sustainable without their future support and what are the ones which cannot be sustained? 17. Is the program reaching those most in need? Probe: who are the prime beneficiary of this health facility? 18. Do you think the MCHIP program has contributed in improving MNCH; how do you know? Probe: What has happened as a result of the program (difference in lives of beneficiaries, poor and marginalized)? Progress with immediate outcomes? 19. What lesson, if any, have been learnt to date? Probe: Planning, operational level, management, and community level? MNCH PROJECT MID-TERM EVALUATION 95 20. What are the current issues, if any, facing the implementation of the MCHIP activities? What are the solutions? 21. Do you have any further comments or suggestions? Signature of interviewer: ______________________ MNCH PROJECT MID-TERM EVALUATION 96 Key Informant Interview for LHW/ LHS/ CHW Serial number Facility name MCHIP facility ID Start time: hh:mm (AM/PM): Researcher name End time: hh:mm (AM/PM): District name: Interview date: DD/MM/YY Tehsil name: UC name: City /village name: Entered by: Respondent name Respondent designation Respondent organization Education Year of experience Contact number Email Location of interview Interviewer: I have informed respondent about the evaluation scope and she/he has consented to the interview. Interviewer signature: ___________________________________ MNCH PROJECT MID-TERM EVALUATION 97 1. Please indicate the services you provide in the community (Circle the appropriate number for each service) Service Yes No Service Yes No Service Yes No ANC registration: 1 2 Pregnant/ lactating mothers counseling: 1 2 Provision of Misoprostol and/or CHX: 1 2 Nutrition counseling: 1 2 Cooking demonstration at household: 1 2 Provision of Iron Folic Acid tablets: 1 2 QIT facilitation: 1 2 QIT membership: 1 2 Referral for newborn care: 1 2 Issuance of referral slip: 1 2 Direct referral support: 1 2 Women support group: 1 2 Other services, give names: 1 2 (Skip the sections which are not applicable) 2. What trainings have you received in the past 3 years (focus MCHIP imparted trainings)? Trainings received from MCHIP Training received? (Circle one number in each row) If “Yes”, year(s) in which training received? (2013/2014/2015/2016) Yes No Cooking demonstration for community 20__ Mother and child nutrition 20__ Women support group (only in five districts) 20__ Interpersonal communication (IPC) tool kit training (only in five districts) 20__ Training on formation of quality improvement team (QIT) for limited staff of MCHIP & PPHI 20__ Others: MNCH PROJECT MID-TERM EVALUATION 98 3. Have you used the training related information and/or skills in your work? If yes, please give specific examples. (relate with previous Q-2 responses and enquire only the pertinent) Trainings Used training? (Circle one number in reach row) If “Yes”, give application/examples Yes No Cooking demonstration for community 1 2 Mother and child nutrition 1 2 Women support group (only in five districts) 1 2 Interpersonal communication (IPC) tool kit training (only in five districts) 1 2 Training on formation of quality improvement team (QIT) for limited staff of MCHIP & PPHI 1 2 4. How useful were the trainings? Have you received technical support from MCHIP after training? If you received support, please elaborate (what, how often). (relate with previous Q-2 responses and enquire only the pertinent) Trainings Usefulness? (Circle one number in each row) Post-training support (MCHIP) (Complete as applicable) Very useful Some￾what useful Not useful Not applic￾able Yes No Explain Cooking demonstration for community 1 2 3 4 1 2 Mother and child nutrition 1 2 3 4 1 2 Women support group (only in five districts) 1 2 3 4 1 2 Interpersonal communication (IPC) tool kit training (only in five districts) 1 2 3 4 1 2 Training on formation of quality improvement team (QIT) for limited staff of MCHIP & PPHI 1 2 3 4 1 2 MNCH PROJECT MID-TERM EVALUATION 99 5. Where do you refer pregnant women for ante natal checkup in your community? (Check all that apply) o Attached BHU/ RHC o Private clinic o CMW o Other 6. Do you distribute referral slips to pregnant women? If yes, about how many utilize the transport services offered by the transporters’ network made available? Distribute referral slips? (Circle one number) If yes, about how many utilize transport services? (Check appropriate box) Yes No  All 1 2  Most  Few  None 7. Based on actual client feedback, what are the key benefits, if any, of the transporter network/ contacts established by the MCHIP program and what are the major hindrances or challenges, if any, to utilization? o Benefits ________________________________________ o Challenges ________________________________________ 8. Do you have a women’s support group formed in your community, is it currently operational? Status of WSG (Circle one number) If yes, how many are formed & when? Yes No WSG formed 1 2 WSG operational 1 2 9. If there is a WSG, has there been an improved support group functioning in the last few years? (enhanced regularity, number of groups, sub-groups, frequency, increased participation, feedback incorporation, provision of informational material related to topics, guest speaker…etc.) MNCH PROJECT MID-TERM EVALUATION 100 (Pertinent to MCHIP activity support and training – revival – rejuvenation- technical information support …et.) Improved support group functioning (Circle one number) If yes, what? (Probe for detail) Yes No 1 2 10. What are the key discussion / information/ promotion topics covered in the WSG (related to the updated support provided by MCHIP only)? 11. What MCHIP-related nutrition support activities are part of your work, name the key activities? 12. Has anyone talked with you about the importance of focusing on the “1000 days”? If yes, where did you hear this? Are there any other projects, programs, NGOs working with you on maternal and child nutrition? If yes, who are they and what are they doing in this area? Received information about first 1000 days? (Circle one number) If yes, where did you hear about first 1000 days? Other organizations Yes No Name? Which component do they support? 1 2 MNCH PROJECT MID-TERM EVALUATION 101 13. Do you distribute IFA (Iron Folic Acid) to pregnant women? If “yes”, in which month? Distribute IFA (Circle one number) If yes, in which month of the pregnancy? If yes, how many tablets during the entire pregnancy? If yes, do women take them? (Circle one number) If not, why not? Yes No Yes No 1 2 1 2 14. Do you give Misoprostol + Chlorhexidine pack to pregnant women? If “yes”, in which month? Misoprostol + Chlorhexidine pack to pregnant women? (Circle one number) If “yes”, in which month of the pregnancy? Yes No 1 2 15. In your experience, do women use the Misoprostol + Chlorhexidine packs, if “yes”, both or one? If “no”, what are the reasons for non-compliance? Do women use Miso and/or CHX? (Circle one number) If yes, both or one (Circle one number) If “one”, which one (Circle one number) Yes No Both One Miso CHX 1 2 1 2 1 2 16. Where do you refer clients for family planning (postpartum IUCD, implant) and newborn and child health (diarrhea, pneumonia)? Has there been a recent change in referral/ services? If yes, please explain? MNCH PROJECT MID-TERM EVALUATION 102 Long acting reversible contraceptive (IUCD, implant) (Check if you refer to facility type) Newborn care (Check if you refer to facility type) Attached BHU/ RHC  Refer  Refer Private Clinic  Refer  Refer THQ/DHQ  Refer  Refer CMW  Refer  Refer Other  Refer  Refer Change in referral services? (Explain) 17. Do you distribute MCHIP printed information booklet to pregnant women in your community? (show the booklet to them), If “yes”, share the women’s feedback regarding whether it is useful and informative and its application. Distribute booklet? (Circle one number) If “yes”, was it useful and informative and did women apply information? Explain. Yes No 1 2 18. Share suggestion to improve MNCH service access and utilization at local MNCH center? (Focus on support by MCHIP) MNCH PROJECT MID-TERM EVALUATION 103 ONLY to be Enquired to LHWs / CHWs who response in affirmation – Relate to Question No.2 Responses 19. Have cooking demonstrations been held at WSG meetings? If “yes”, how useful did the women in community find them (those who actually applied the cooking information) Did women change their behavior in any way related to the introduction of complementary foods to 6- 23 month old children as a result of these demonstrations? If yes, how? Cooking demonstrations held? (Circle one number) Did women find the theme useful? Explain. Did women apply the knowledge/change feeding behavior? How? Yes No 1 2 MNCH PROJECT MID-TERM EVALUATION 104 Key Informant Interview with Community Midwives (CMWs) Serial number Facility name MCHIP facility ID Start time: hh:mm (AM/PM): Researcher name End time: hh:mm (AM/PM): District name: Interview date: DD/MM/YY Tehsil name: UC name: City /village name: Respondent name Age Education Year of experience as CMW Marital Status Use qualifier with marital status, married no children, unmarried, married with children under 5/ over 5 Contact number Completed CMWs training? Yes or No Name of institute of CMW training: Interviewer: I have informed the respondent about the evaluation scope and she has agreed to the interview. Interviewer signature: ___________________________________ 1. Do you have a functioning clinic set-up/ practice? (Circle one number) Yes No 1 2 If “no”, give reason. (Enter reason) MNCH PROJECT MID-TERM EVALUATION 105 If “yes”, where? (Circle one number) 1. House 2. Community 3. Other community 4. Other (specify) ___________________ If “yes”, number of years/months in operation? (Enter time) If “other”, please explain. (Circle one number) 1. Working at other health facility 2. Clinic under development 3. Temporarily closed 4. Other (specify) ___________________ 2. Please indicate the services you provide in the community. (Circle the numbers of all that apply) 1. ANC Checkup 2. Referral for complicated pregnancy/ delivery 3. Provision of Iron Folic Acid Tablets 4. QIT Participation (member) 5. Normal Vaginal Delivery 6. Nutrition Counseling 7. Provision of Misoprostol / CHX 8. Post Natal Check Up 9. Newborn Check Up 10. Tetanus vaccination of pregnant women 11. Other Services (specify): (Skip the sections in later questionnaire which are not applicable) 3. What trainings have you received in last 3 years (focus on MCHIP imparted trainings)? Trainings received: (focus in MCHIP training package) (Circle one number in each row) (Enter year) Yes No In what year did you receive the training? Pregnancy childbirth postpartum and newborn care (PCPNC) 1 2 20__ Post-abortion care (PAC) 1 2 20__ Postpartum intrauterine contraceptive device (PPIUCD) 1 2 20__ Client centered family planning - intrauterine contraceptive device (CCFP-IUCD) 1 2 20__ Helping Babies Breath (HBB) 1 2 20__ Lactation management training 1 2 20__ Immunization 1 2 20__ Other (specify) _____________________________ 1 2 20__ MNCH PROJECT MID-TERM EVALUATION 106 4. Have these trainings helped you perform better and, if so, and how? Trainings received: (focus in MCHIP training package) (Circle one number in each row) How have they improved your performance? Yes No Pregnancy childbirth postpartum and newborn care (PCPNC) 1 2 Post-abortion care (PAC) 1 2 Postpartum intrauterine contraceptive device (PPIUCD) 1 2 Client centered family planning - intrauterine contraceptive device (CCFP￾IUCD) 1 2 Helping Babies Breath (HBB) 1 2 Lactation management training 1 2 Immunization 1 2 Other (specify) __________________ 1 2 5. Have you received any training, coaching or other kinds of support from MCHIP and, if yes, what kind? Support received: (focus in MCHIP training package) (Circle one number in each row) What kind of technical support have you received? Yes No Pregnancy childbirth postpartum and newborn care (PCPNC) 1 2 Post-abortion care (PAC) 1 2 Postpartum intrauterine contraceptive device (PPIUCD) 1 2 Client centered family planning - intrauterine contraceptive device (CCFP￾IUCD) 1 2 Helping Babies Breath (HBB) 1 2 Lactation management training 1 2 Immunization 1 2 Other (specify) ___________________ 1 2 6. How many checkups are minimally advised in pregnancy and how many checkups your clients on an average come for? (Enter number of visits) ANC visits advised Usual ANC visits by pregnant women MNCH PROJECT MID-TERM EVALUATION 107 7. Where do you refer the complicated cases from your community? (Circle numbers of all that apply) 1. Attached BHU/ RHC 2. Private Clinic 3. Other (specify)_______________________________ 8. Rate the service delivery in the referral facility for the complicated case management, and has there been any recent improvement in the service delivery at the Health Facility? (probe for last 2-3 years and reasons) Quality of service delivery (Circle one number) 1. Excellent 2. Good 3. Fair 4. Poor How has service quality improved? (Circle one number) 1. No major change 2. Marked improvement If “marked improvement”, what is the reasons? (Explain) 9. Do you distribute referral slips for complicated case clients (or refer them to LHWs) and, if so, how many clients utilize the transport services offered by the transporters’ network? Referral of complicated cases (Circle all that apply) (If no options checked, skip question 10) 1. Distribute referral slips 2. Refer to CMW 3. Refer to LHW How many referred clients utilize transporter network service? (Circle one number) 1. All 2. Most 3. Few 4. None 10. Based on actual client feedback, what are the key benefits of the transporter network/ contacts and the major hindrances or challenges in the utilization? (Write an explanation) Benefits MNCH PROJECT MID-TERM EVALUATION 108 Hindrances/Challenges 11. Have you received any business development training from the MCHIP project? (Circle one number) Yes No (Go to question 13) If “yes”, when and where did you receive the training? (Explain) 12. Did you find the business training/information useful? (Circle one number) 1. Yes 2. No (Go to question 13) If “yes”, how did it help your business? (Explain) 13. Have you received any equipment and/or supplies from the MCHIP project? If so, what did you receive and how often? Received equipment or supplies? (Circle one number) 1. Yes 2. No (Go to question 14) List equipment received List supplies received Frequency (Circle one number for each row) Regularly One-time Needs based 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 MNCH PROJECT MID-TERM EVALUATION 109 14. Has the MCHIP project supported any non- structural development of the clinic? (Circle one number) 1. Yes 2. No (Go to question 15) If “yes”, please explain how? (Explain) 15. Have you been paid stipend from any source? If yes, from whom did you receive them, when and for how long? (Circle one number) 1. Yes 2. No (Go to question 16) If “yes”, when and for how long? (Explain) 16. Do you distribute IFA (Iron Folic Acid) for pregnant women or refer to LHW? (Circle one number) 1. Yes, I distribute IFA 2. No, I do not distribute IFA (Go to question 16) 3. I refer to LHW for IFA (Go to question 16) If “yes”, in which month of the pregnancy? (Explain) 16. Do you give misoprostol + Chlorhexidine pack to the pregnant women or refer them to the LHW? (Circle one number) 1. Yes, I distribute misoprostol + Chlorhexidine pack 2. No, I do not distribute misoprostol + Chlorhexidine pack (Go to question 17) 3. I refer to LHW for misoprostol + Chlorhexidine pack (Go to question 17) If “yes”, in which month of the pregnancy? (Explain) 17. In your experience, do women use the misoprostol + Chlorhexidine pack and, if yes, both or one? If No, what are the reasons for non-compliance? Do women use? (Circle one number) 1. Yes, both 2. Yes, one 3. No MNCH PROJECT MID-TERM EVALUATION 110 If, “yes, one”, reasons for selected compliance. (Explain) If “no”, reasons for non-compliance. (Explain) 18. Is there an operational QIT group in the community/facility? If yes, are you a member of that group? Yes No Operational QIT? (Circle one number) 1 2 (Go to question 19) Are you a member? (Circle one number) 1 2 19. Do you have any suggestions for improving MNCH service access and utilization in the community? (Focus on support by MCHIP) MNCH PROJECT MID-TERM EVALUATION 111 CHECKLIST for BEmONC HEALTH FACILITIES Serial number Type of health facility _____________ (BHU (DoH), BHU (PPHI), RHC, THQ, Private) Name of the health facility MCHIP facility ID Observation start time: hh:mm (AM/PM): Researcher name Observation end time: hh:mm (AM/PM): District name: Interview date: DD/MM/YY Tehsil name: UC name: City /village name: Entered by: Total catchment population: Total number of staff members: Total attached LHWs: Total attached CMWs: Opening timing of the facility Total number of villages in catchment: Interviewer: I have informed the BEmONC health facility in-charge about the evaluation scope and received his/her consent to fill this checklist. Observer signature: ___________________________________ 1. PHYSICAL INFRASTRUCTURE AND UTILITIES Instructions: Please check if the utility was facilitated by MCHIP or not. (Circle one number in each row) S. No DIMENSION Yes No 2.1 ACCESS / UTILITIES Facility sign board 1 2 2.2 2.3 Electricity 1 2 2.4 Telephone 1 2 2.5 WASTE DISPOSAL Rubbish pit (for refuse and medical waste) 1 2 2.6 Sewerage pipe from building to external sewer/drain 1 2 2.7 External sewer/drain merges into main drain outside boundary wall 1 2 MNCH PROJECT MID-TERM EVALUATION 112 S. No DIMENSION Yes No 2.8 ENTRANCE TO BHU BUILDING Space for registration 1 2 2.9 Waiting area for patients 1 2 2.10 Ramp for disabled/wheelchair 1 2 2.11 WAITING AREA Covered area to protect patients from sun, rain, and extremes of temperature 1 2 2.12 Functional ceiling fans at women’s waiting area 1 2 2.13 Separate women’s waiting area 1 2 2.14 Benches in women’s waiting area 1 2 2.15 Complaint/suggestion box 1 2 2.16 Separate function toilets for men and women 1 2 2.17 WATER SUPPLY Pipe with running water within facility 1 2 2.18 Storage tank within the facility 1 2 2.19 Protected water source within 200 meters of the facility (borehole, water tank, or protected spring) 1 2 2.20 EXAMINATION ROOM Examination room for women 1 2 2.21 Curtains/screens to ensure privacy 1 2 2.22 LABOR ROOM Well-lit 1 2 2.23 Ventilation 1 2 2.24 Attached toilet 1 2 2.25 Drinking water facility 1 2 2.26 Designated space for new-born care 1 2 2.27 GENERAL STORE Well-lit 1 2 2.28 Ventilation 1 2 2.29 Area for storage of sterile linen 1 2 2.30 Area for storage of common linen 1 2 MNCH PROJECT MID-TERM EVALUATION 113 S. No DIMENSION Yes No 2.31 Area for storage of other materials/drugs/consumables 1 2 2.32 Dispensing cum store area 1 2 2.33 Vaccine storage and immunization area 1 2 2.34 BCC and family planning counsel area 1 2 CLEANLINESS OF BUILDING S. No DIMENSION STATUS (Circle one number in each row) Satisfactory Unsatisfactory 2.35 Waiting area 1 2 2.36 Consultation rooms 1 2 2.37 Treatment/injection rooms 1 2 2.38 Delivery room 1 2 2.39 Main pharmacy / dispensing area 1 2 2.40 Toilets-patients 1 2 2.41 Store room 1 2 STATUS OF BUILDING S. No DIMENSION STATUS (Circle one number in each row) Satisfactory Un-satisfactory 2.42 Windows and doors 1 2 2.43 Floor 1 2 MNCH PROJECT MID-TERM EVALUATION 114 S. No DIMENSION STATUS (Circle one number in each row) Satisfactory Un-satisfactory 2.44 Roof condition 1 2 3. RANGE OF SERVICES HEALTH EDUCATION AND PROMOTION S. No DIMENSION STATUS (Circle one number in each row) 3.1 Yes No 3.2 HYGIENE PROMOTION, WATER AND SANITATION Hand washing with soap 1 2 3.3 Proper toilet use and hand washing practices 1 2 3.4 Problems related to open defecation 1 2 3.5 Advice on making water safe for drinking and storage 1 2 3.6 MNCH AND FAMILY PLANNING Importance of antenatal check-up 1 2 3.7 Tetanus Toxoid (TT) injections during pregnancy 1 2 3.8 Danger signs during pregnancy 1 2 3.9 Delivery through skilled birth attendance 1 2 3.10 Danger signs during labor 1 2 3.11 Importance of postpartum examination 1 2 3.12 Danger signs after delivery 1 2 3.13 Use of Chlorhexidine gel 1 2 3.14 Bathing the neonate 1 2 3.15 Early wrapping and keeping baby warm 1 2 MNCH PROJECT MID-TERM EVALUATION 115 S. No DIMENSION STATUS (Circle one number in each row) 3.1 Yes No 3.16 Early initiation of breastfeeding 1 2 3.17 Neonatal danger signs 1 2 3.18 Family planning methods especially (PPIUCD) 1 2 3.19 CHILD HEALTH & DEVELOPMENT Exclusive breast feeding up to 6 months 1 2 3.20 Continuation of breast feeding till 2 years of age 1 2 3.21 Child immunization 1 2 3.22 Managing diarrhea at home 1 2 3.23 Growth/development monitoring 1 2 3.24 NUTRITION Iron / folic acid supplementation of pregnant lactating women 1 2 3.25 Balanced diet for adolescents and adults 1 2 3.26 Weaning after 6 months of age under IYCF guidelines 1 2 3.27 Prevention of parasitic infections and deworming 1 2 CARE PROVISION S. No DIMENSION STATUS (Circle one number in each row) Yes No 3.28 MATERNAL HEALTH ANC (screening for high risk, TT vaccination) 1 2 3.29 ANC (dietary counseling) 1 2 3.30 NATAL CARE (normal delivery with forceps/MVA) 1 2 MNCH PROJECT MID-TERM EVALUATION 116 S. No DIMENSION STATUS (Circle one number in each row) Yes No 3.31 NATAL CARE (referral for complicated delivery) 1 2 3.32 NATAL CARE (ambulance services) 1 2 3.33 PNC (screening for risk/complications) 1 2 3.34 PNC (PPIUCD) 1 2 3.35 NEWBORN/ CHILD HEALTH Chlorhexidine use 1 2 3.36 Neonatal examination within 72 hours 1 2 3.37 EPI vaccination services – at health facility 1 2 3.38 EPI vaccination services – as outreach services 1 2 3.39 FAMILY PLANNING Provision of short term methods (condoms, pills) 1 2 3.40 Provision of short term methods (PPIUCDs, injectables, implants) 1 2 3.41 NUTRITION SERVICES Outpatient therapeutic program (treatment of moderate and severe acute malnutrition without complications) 1 2 3.42 Referral linkage with a stabilization center at THQ/DHQ 1 2 3.43 Provision of nutrition supplements to the malnourished children 1 2 4. EmONC SERVICES BASIC EmONC SERVICES Are the following obstetrics care services available in the facility? MNCH PROJECT MID-TERM EVALUATION 117 BASIC EmONC SERVICES STATUS (Circle one number in each row) Yes No 4.1 Administer IV, IM (Parenteral) antibiotics 1 2 4.2 Augmentation of labor by oxytocic drugs (Injection oxytocin, Injection ergometrine) 1 2 4.3 Management of pre-eclamsia and eclampsia by sedatives (by injection magnesium sulfate or injection diazepam 1 2 4.4 Manual removal of placenta 1 2 4.5 Removal of retained products (manual vacuum aspiration (MVA), without general anesthesia, and D&C) and referral of complicated cases 1 2 4.6 Assisted vaginal delivery (vacuum extraction, forceps) 1 2 4.7 Who conducts delivery at the health facility? (Circle one number) 1. WMO 1 2 2. LHV 1 2 3. Midwife 1 2 4. Dai 1 2 5. MO 1 2 NEWBORN CARE SERVICES Are the following new born care services available in the facility? NEWBORN CARE SERVICES STATUS (Circle one number in each row) Yes No 4.8 Neonatal resuscitation (with bag and mask) 1 2 4.9 Warmth (drying, wrapping the baby, and skin-to-skin contact) 1 2 4.10 Clean cord care with Chlorhexidine gel 1 2 4.11 Early initiation of breast feeding 1 2 MNCH PROJECT MID-TERM EVALUATION 118 NEWBORN CARE SERVICES STATUS (Circle one number in each row) Yes No 4.12 Helping Baby Breath 1 2 5. AVAILABILITY OF EQUIPMENT/IEC MATERIAL GENERAL S. No EQUIPMENT STATUS AVAILABLE (Circle one number in each row) FUNCTIONAL (Circle one number in each row) Yes No Yes No 1.1 Ambulance 1 2 1 2 1.2 Stethoscope 1 2 1 2 1.3 Fetal stethoscope 1 2 1 2 1.4 Steam inhaler 1 2 1 2 1.5 Nebulizer 1 2 1 2 1.6 X-ray view box 1 2 1 2 1.7 X-ray unit 1 2 1 2 1.8 ECG machine 1 2 1 2 1.9 Glucometer for blood sugar 1 2 1 2 1.10 Ultrasound 1 2 1 2 1.11 Protocols for pregnancy, child birth, postpartum, and newborn 1 2 1 2 MNCH PROJECT MID-TERM EVALUATION 119 S. No EQUIPMENT STATUS AVAILABLE (Circle one number in each row) FUNCTIONAL (Circle one number in each row) Yes No Yes No 1.12 Protocols for pregnancy complications 1 2 1 2 LABOR ROOM S. NoEQUIPMENT STATUS AVAILABLE (Circle one number in each row) FUNCTIONAL (Circle one number in each row) Yes No Yes No 1.13 Labor /delivery table with washable plastic cover 1 2 1 2 1.14 Macintosh/plastic apron 1 2 1 2 1.15 Delivery light 1 2 1 2 1.16 Normal delivery set 1 2 1 2 1.17 Standard surgical set (for minor procedures like episiotomy stitching) 1 2 1 2 1.18 Bulb sucker 1 2 1 2 1.19 Fetal heart detector (Fetoscope) 1 2 1 2 1.20 Examination light 1 2 1 2 1.21 Suction and evacuation set (SNE) 1 2 1 2 1.22 IUD insertion kit 1 2 1 2 1.23 Adult stethoscope 1 2 1 2 1.24 Bedpans 1 2 1 2 MNCH PROJECT MID-TERM EVALUATION 120 S. NoEQUIPMENT STATUS AVAILABLE (Circle one number in each row) FUNCTIONAL (Circle one number in each row) Yes No Yes No 1.25 Blood pressure apparatus 1 2 1 2 1.26 Adult Ambu bag and mask 1 2 1 2 1.27 Thermometer 1 2 1 2 1.28 Oxygen source (portable cylinder or central wall supply), with mask or nasal cannula 1 2 1 2 1.29 Baby weighing scale 1 2 1 2 1.30 Manual Vacuum Aspiration (MVA) 1 2 1 2 1.31 Suture needles 1 2 1 2 1.32 Partograph forms 1 2 1 2 1.33 Adult weighing scale 1 2 1 2 1.34 Protocols for managing pregnancy complications 1 2 1 2 1.35 Stethoscope 1 2 1 2 1.36 Ventilation bag mask 1 2 1 2 1.37 Suction device 1 2 1 2 1.38 Scissors 1 2 1 2 1.39 Ties 1 2 1 2 1.40 Gloves 1 2 1 2 1.41 Cloth 1 2 1 2 1.42 Head covering 1 2 1 2 1.43 Timer (clock, watch) 1 2 1 2 1.44 HBB poster 1 2 1 2 MNCH PROJECT MID-TERM EVALUATION 121 S. NoEQUIPMENT STATUS AVAILABLE (Circle one number in each row) FUNCTIONAL (Circle one number in each row) Yes No Yes No 1.45 Hand washing stations 1 2 1 2 1.46 Disinfectants 1 2 1 2 1.47 Boiler / autoclave 1 2 1 2 1.48 Disposable syringe cutter 1 2 1 2 1.49 Puncture resistant container for sharps disposal 1 2 1 2 1.50 Bucket for soiled pads and swabs 1 2 1 2 EPI UNIT 1.51 Vaccine refrigerator (ILR) 1 2 1 2 1.52 Thermometer for vaccine refrigerator 1 2 1 2 1.53 Temperature log 1 2 1 2 1.54 EPI cards 1 2 1 2 1.55 EPI schedule 1 2 1 2 6. QUALITY INSPECTION TEAM (QIT) STATUS (Circle one number in each row) Yes No 6.1 Does the facility have a functional QIT? 1 2 6.2 Record of SG/QIT meeting conducted in last 2 months is available 1 2 MNCH PROJECT MID-TERM EVALUATION 122 STATUS (Circle one number in each row) Yes No 6.3 Was an action plan developed based on the outcome of the meeting? 1 2 7. IMPORTANT DOCUMENTS S. No DOCUMENTS STATUS (Circle one number in each row) Yes No 7.1 Updated health facility action plan 1 2 7.2 Daily client register/OPD register maintained 1 2 7.3 Record of all cases referred maintained in referral register/DHIS register 1 2 7.4 Results of last QIPS assessments and action plans 1 2 7.5 Duty roster 1 2 7.6 Results of internal assessment maintained at facility 1 2 MNCH PROJECT MID-TERM EVALUATION 123 CHECKLIST HEALTH FACILITY CEmONC Serial number: MCHIP facility ID: Start time: hh:mm (AM/PM): Researcher name: End time: hh:mm (AM/PM): District name: Interview date: DD/MM/YY Tehsil name: UC name: City /village name: Entered by: 1. GENERAL INFORMATION Type of health facility: 1. DHQ___________ 2. THQ___________3. Private__________ Name of health facility: Name of District: Total catchment population: What is the working timing of the facility? (Check one)  All week - 24 hrs  7 days - day time only (8 am to 2 pm)  Six days 24 hrs  Six days - day time only (8 am to 2 pm)  Not regular  Other (specify)______________________ Interviewer: I have informed the CEmONC health facility in-charge about the evaluation scope and he/she has consented to conduct the fill this checklist. Observer signature: ___________________________________ MNCH PROJECT MID-TERM EVALUATION 124 2. PHYSICAL INFRASTRUCTURE AND UTILITIES S. No DIMENSION STATUS (Circle one number in each row) Yes No 2.1 Access / utilities Metalled access road 1 2 2.2 Facility sign board 1 2 2.3 Electricity 1 2 2.4 Electric wiring concealed 1 2 2.5 Functioning telephone 1 2 2.6 Dedicated mobile phone for communication regarding referral care 1 2 2.7 Ob/Gyn OPD room Consultation area 1 2 2.8 Examination area 1 2 2.9 Screens/curtains for privacy of examination area 1 2 2.10 Hand washing area 1 2 2.11 Ob/Gyn ward Patient area 1 2 2.12 Nursing station 1 2 2.13 Store for general items/drugs 1 2 2.14 Functional attached wash room 1 2 2.15 Pediatric nursery Patient area 1 2 2.16 Nursing station 1 2 2.17 Store for general items/drugs 1 2 2.18 Changing room 1 2 2.19 Feeding area 1 2 2.20 Day care area 1 2 2.21 Delivery room 1 2 MNCH PROJECT MID-TERM EVALUATION 125 S. No DIMENSION STATUS (Circle one number in each row) Yes No 2.22 Labor room Preparation / stage room 1 2 2.23 Scrub area 1 2 2.24 Staff duty room 1 2 2.25 Store for general items/drugs/equipment 1 2 2.26 Staff wash room 1 2 2.27 Patient wash room 1 2 2.28 Blood bank Blood collection room 1 2 2.29 Staff duty room 1 2 2.30 Store for equipment / reagents 1 2 2.31 Functional attached washroom 1 2 REPAIR REQUIREMENTS OF BUILDING S. No DIMENSION STATUS (Circle one number in each row) Few or no repairs needed Many repairs needed Not present 2.32 Windows and doors 1 2 3 2.33 Interior paint 1 2 3 2.34 Facility interior walls 1 2 3 2.35 Facility exterior walls 1 2 3 2.36 Floor 1 2 3 2.37 Roof condition 1 2 3 2.38 Fence/wall 1 2 3 2.39 Women’s waiting area 1 2 3 MNCH PROJECT MID-TERM EVALUATION 126 S. No DIMENSION STATUS (Circle one number in each row) Few or no repairs needed Many repairs needed Not present 2.40 Consultation rooms 1 2 3 2.41 Treatment/injection rooms 1 2 3 2.42 Pediatric ward 1 2 3 2.43 Obstetric ward 1 2 3 2.44 Delivery room 1 2 3 2.45 FP services room 1 2 3 2.46 Pharmacy 1 2 3 2.47 Toilets- female patients 1 2 3 2.48 OT and washing area 1 2 3 2.49 Nursery 1 2 3 3. RANGE OF SERVICES AT FACILITY S. No DIMENSION STATUS (Circle one number in each row) Yes No 3.1 MATERNAL HEALTH ANC (screening for high risk) 1 2 3.2 ANC (TT vaccination - static center) 1 2 3.3 ANC (dietary counseling) 1 2 3.4 NATAL CARE (normal delivery 24/7 with forceps/MVA) 1 2 3.5 NATAL CARE (C-section) 1 2 3.6 NATAL CARE (functional ambulance services) 1 2 3.7 PNC (screening for risk/complications) 1 2 MNCH PROJECT MID-TERM EVALUATION 127 S. No DIMENSION STATUS (Circle one number in each row) Yes No 3.8 CHILD HEALTH EPI vaccination – static center 1 2 3.9 Intensive care / nursery with incubators 1 2 3.10 Counseling of mothers (exclusive breast feeding and complimentary feeding) 1 2 3.11 Stabilization center – linkage with CMAM program 1 2 3.12 FAMILY PLANNING FP counseling of MWRA 1 2 3.13 Provision of short term methods (condoms, pills) 1 2 3.14 Provision of short term methods (IUDs, injectables, implants) 1 2 3.15 Provision of permanent methods (vasectomy, tubal ligation) 1 2 3.16 LAB SERVICES Blood profiles (CBC, LFTs, RFTs) 1 2 3.17 Hepatitis B screening 1 2 3.18 Hepatitis C screening 1 2 3.19 Stool R/E 1 2 3.20 Blood sugar 1 2 3.21 DIAGNOSTIC SERVICES CT scan 1 2 3.22 ECG 1 2 3.23 Ultrasonography 1 2 3.24 Blood grouping and cross matching 1 2 3.25 Blood screening for HBs Ag 1 2 3.26 Blood screening for Anti-HCV 1 2 3.27 Pregnancy testing 1 2 3.28 Hemoglobin 1 2 3.29 Urine R/E 1 2 MNCH PROJECT MID-TERM EVALUATION 128 S. No DIMENSION STATUS (Circle one number in each row) Yes No 3.30 Other (Specify) 1 2 4. COMPREHENSIVE EmONC SERVICES EMERGENCY OBSTETRIC SERVICES Are the following obstetrics care services/ inputs available in the facility? S. No EMERGENCY OBSTETRIC SERVICES STATUS (Circle one number in each row) Remarks Yes No 4.13 IV, IM (parenteral) antibiotics 1 2 4.14 Oxytocic drugs (Injection oxytocin, Injection Ergometrine) 1 2 4.15 Injection magnesium sulfate 1 2 4.16 Injection diazepam 1 2 4.17 Manual removal of placenta 1 2 4.18 Removal of retained products (D&C) 1 2 4.19 Assisted vaginal delivery (vacuum extraction, forceps) 1 2 4.20 Caesarean-Section 1 2 4.21 Availability of blood with storage facility 1 2 4.22 Functional ambulance service 1 2 MNCH PROJECT MID-TERM EVALUATION 129 NEWBORN CARE SERVICES Are the following newborn care services available in the facility? S. No EMERGENCY OBSTETRIC SERVICES STATUS (Circle one number in each row) Remarks Yes No 4.23 Basic newborn resuscitation 1 2 4.24 Phototherapy 1 2 4.25 Clean cord care with Chlorhexidine gel 1 2 4.26 Incubator 1 2 4.27 1 2 4.28 Paediatric nursery 1 2 4.29 Health promotion for early and exclusive breast feeding 1 2 4.30 Birth weight 1 2 4.31 Suction through sucker machine 1 2 4.32 Functional oxygen cylinder 1 2 4.33 Do you see any neonate, child <5 or obstetric case after the working hours, if there is an emergency? 4.34 If yes, who attends? 1.Specialist 1 2 2. Doctor 1 2 3. Nurse 1 2 4. LHV/midwife 1 2 5. Dispenser 1 2 99. Other (Specify)________ 1 2 MNCH PROJECT MID-TERM EVALUATION 130 5. AVAILABILITY OF EQUIPMENT AND MEDICINES GENERAL HOSPITAL S. No EQUIPMENT ITEM QUANTITIES (Enter “0” when an item is not available or not functional) Number or items AVAILABLE Number of items FUNCTIONAL Number of items NOT FUNCTIONAL 5.1. Functional ambulance 5.2. Defibrillator 5.3. Wheel chair 5.4. Stretcher 5.5. Generator 5.6. UPS power supply GYNAE/OBS OPD ROOM S. No EQUIPMENT ITEM QUANTITIES (Enter “0” when an item is not available or not functional) Number or items AVAILABLE Number of items FUNCTIONAL Number of items NOT FUNCTIONAL 5.7. Weighing machine (adult) 5.8. Infant weighing machine 5.9. Screen folding (complete) 5.10. Ultrasound machine 5.11. Examination lamp 5.12. Stethoscope (adult size) MNCH PROJECT MID-TERM EVALUATION 131 S. No EQUIPMENT ITEM QUANTITIES (Enter “0” when an item is not available or not functional) Number or items AVAILABLE Number of items FUNCTIONAL Number of items NOT FUNCTIONAL 5.13. B.P apparatus desktop type 5.14. Office chair 5.15. Office table with drawers 5.16. Patient stool 5.17. Patient waiting bench 5.18. Examination couch PEDATRIC WARD AND NURSERY S. No EQUIPMENT ITEM QUANTITIES (Enter “0” when an item is not available or not functional) Number or items AVAILABLE Number of items FUNCTIONAL Number of items NOT FUNCTIONAL 5.19. Suction machine 5.20. Infant B.P apparatus (cuff 2.5 cm) 5.21. Stethoscope, pediatric Littman type 5.22. Nebulizer 5.23. Oxygen cylinder complete with trolley 5.24. Emergency medicine trolley 5.25. Patient's attendant bench MNCH PROJECT MID-TERM EVALUATION 132 S. No EQUIPMENT ITEM QUANTITIES (Enter “0” when an item is not available or not functional) Number or items AVAILABLE Number of items FUNCTIONAL Number of items NOT FUNCTIONAL 5.26. Fowler bed (iron) 5.27. Screen folding (complete) 5.28. Functional air conditioner in pediatric nursery LABOR ROOM S. No EQUIPMENT ITEM QUANTITIES (Enter “0” when an item is not available or not functional) Number or items AVAILABLE Number of items FUNCTIONAL Number of items NOT FUNCTIONAL 5.29. Electric instrument sterilizer (12”x6”) 5.30. Episiotomy instruments set (complete) 5.31. D&C instruments set (complete) 5.32. Delivery forceps set 5.33. Infant Ambo bag 5.34. Portable light & rechargeable batteries 5.35. Sterilizing drum 5.36. Vacuum extractor 5.37. Delivery table MNCH PROJECT MID-TERM EVALUATION 133 S. No EQUIPMENT ITEM QUANTITIES (Enter “0” when an item is not available or not functional) Number or items AVAILABLE Number of items FUNCTIONAL Number of items NOT FUNCTIONAL 5.38. Infusion / drip stands 5.39. Oxygen cylinder complete with trolley OPERATING THEATER S. No EQUIPMENT ITEM QUANTITIES (Enter “0” when an item is not available or not functional) Number or items AVAILABLE Number of items FUNCTIONAL Number of items NOT FUNCTIONAL 5.40. Operating theater ceiling light 5.41. Operating theater table 5.42. Functional air conditioner 5.43. General surgery instrument set 5.44. Obstetrics instrument set 5.45. Sterilizer 5.46. Autoclave (steam sterilizer) 5.47. Oxygen cylinder (large size with regulator) 5.48. Receptacles / containers of different colours for different types of solid wastes MNCH PROJECT MID-TERM EVALUATION 134 ANAESTHESIA S. No EQUIPMENT ITEM QUANTITIES (Enter “0” when an item is not available or not functional) Number or items AVAILABLE Number of items FUNCTIONAL Number of items NOT FUNCTIONAL 5.49. Oropharyngeal airway (adult size) 5.50. Anesthetic face masks 5.51. Anesthesia machine 5.52. Laryngoscopes 5.53. Epidural sets 5.54. Endotracheal tube INFECTION CONTROL S. No EQUIPMENT ITEM QUANTITIES (Enter “0” when an item is not available or not functional) Number or items AVAILABLE Number of items FUNCTIONAL Number of items NOT FUNCTIONAL 5.55. Hand washing stations 5.56. Disinfectants 5.57. Boiler / autoclave 5.58. Disposable syringe cutter 5.59. Puncture resistant container for sharps disposal MNCH PROJECT MID-TERM EVALUATION 135 S. No EQUIPMENT ITEM QUANTITIES (Enter “0” when an item is not available or not functional) Number or items AVAILABLE Number of items FUNCTIONAL Number of items NOT FUNCTIONAL 5.60. Bucket for soiled pads and swabs 6. AVAILABILITY OF MEDICINES AND SUPPLIES SUPPLIES Medicines / Supplies Available (Circle one number in each row) Expired (Circle one number in each row) Stock out during last 3 months (Circle one number in each row) Yes No Yes No Yes No 6.1 Antiseptic solutions (cetrimide, savalon, bleach, povidone, cidex, chlorhexidine, gluconate, iodine, solution or iodophors, "spirits") 1 2 1 2 1 2 6.2 Bandages 1 2 1 2 1 2 6.3 Gauze 1 2 1 2 1 2 6.4 Surgical cotton 1 2 1 2 1 2 6.5 Adhesive tape 1 2 1 2 1 2 6.6 Syringes 1 2 1 2 1 2 6.7 Hypodermic needles and syringes (10-20 cc) 1 2 1 2 1 2 6.8 IV infusion set and fluids (tubing, needles) 1 2 1 2 1 2 6.9 Scalpel blades 1 2 1 2 1 2 MNCH PROJECT MID-TERM EVALUATION 136 Medicines / Supplies Available (Circle one number in each row) Expired (Circle one number in each row) Stock out during last 3 months (Circle one number in each row) Yes No Yes No Yes No 6.10 Soap 1 2 1 2 1 2 6.11 Spare bulb and spare batteries for room light and flashlight 1 2 1 2 1 2 6.12 Suture and suture needles 1 2 1 2 1 2 6.13 Urethral catheter and bag 1 2 1 2 1 2 6.14 Surgical gloves 1 2 1 2 1 2 6.15 Utility gloves 1 2 1 2 1 2 6.16 Bucket with chlorine (0.5%) for decontamination 1 2 1 2 1 2 6.17 Sharps-disposal containers 1 2 1 2 1 2 6.18 Waste buckets (for medical and other waste 1 2 1 2 1 2 6.19 Surgical clothing 1 2 1 2 1 2 6.20 Scrub brushes 1 2 1 2 1 2 6.21 Hand washing sink with running water 1 2 1 2 1 2 6.22 Wheel chair 1 2 1 2 1 2 6.23 Stretcher 1 2 1 2 1 2 MNCH PROJECT MID-TERM EVALUATION 137 MEDICINES Medicines / Supplies Available (Circle one number in each row) Expired (Circle one number in each row) Stock out during last 3 months (Circle one number in each row) Yes No Yes No Yes No 6.24 Injection Dexamethasone 1 2 1 2 1 2 6.25 Injection Adrenaline 1 2 1 2 1 2 6.26 Injection Atropine 1 2 1 2 1 2 6.27 Injection Ampicillin 1 2 1 2 1 2 6.28 Cap Amoxicillin 1 2 1 2 1 2 6.29 Syrup Amoxicillin 1 2 1 2 1 2 6.30 Tablet Cotrimaxazole 1 2 1 2 1 2 6.31 Syrup Cotrimaxazole 1 2 1 2 1 2 6.32 Tablet Metronidazole 1 2 1 2 1 2 6.33 Syrup Metronidazole 1 2 1 2 1 2 6.34 Syrup Antihelminthic 1 2 1 2 1 2 6.35 Tablet Iron 1 2 1 2 1 2 6.36 Tablet Folic Acid 1 2 1 2 1 2 6.37 Tetracycline ointment 1 2 1 2 1 2 6.38 Injection Oxytocin 1 2 1 2 1 2 6.39 Injection Magnesium Sulfate 1 2 1 2 1 2 6.40 IV solutions 1 2 1 2 1 2 6.41 Syrup Salbutamol 1 2 1 2 1 2 6.42 Tablet Chloroquine 1 2 1 2 1 2 6.43 Syrup Choloroquine 1 2 1 2 1 2 MNCH PROJECT MID-TERM EVALUATION 138 Medicines / Supplies Available (Circle one number in each row) Expired (Circle one number in each row) Stock out during last 3 months (Circle one number in each row) Yes No Yes No Yes No 6.44 Syrup Zinc Sulfate 1 2 1 2 1 2 6.45 Syrup Paracetamol 1 2 1 2 1 2 6.46 Injection Diclofenac Sodium 1 2 1 2 1 2 6.47 Tablet Diclofenac 1 2 1 2 1 2 6.48 Ready to use therapeutic food (RUTF) 1 2 1 2 1 2 6.49 Therapeutic milk (F-75 and F-100) 1 2 1 2 1 2 VACCINES Medicines / Supplies Available (Circle one number in each row) Expired (Circle one number in each row) Stock out during last 3 months (Circle one number in each row) Yes No Yes No Yes No 6.50 BCG 1 2 1 2 1 2 6.51 OPV 1 2 1 2 1 2 6.52 Penta-valent 1 2 1 2 1 2 6.53 Measles 1 2 1 2 1 2 6.54 Heptatitis-B vaccine 1 2 1 2 1 2 6.55 Tetanus Toxiod 1 2 1 2 1 2 MNCH PROJECT MID-TERM EVALUATION 139 FAMILY PLANNING COMMODITIES Medicines / Supplies Available (Circle one number in each row) Expired (Circle one number in each row) Stock out during last 3 months (Circle one number in each row) Yes No Yes No Yes No 6.56 Condoms 1 2 1 2 1 2 6.57 Combined oral contraceptive (COC) pills 1 2 1 2 1 2 6.58 IUCDs 1 2 1 2 1 2 6.59 Injection DMPA 1 2 1 2 1 2 6.60 Implants 1 2 1 2 1 2 MNCH PROJECT MID-TERM EVALUATION 140 CHECKLIST FOR COMMUNITY MIDWIFE (CMW) HEALTH FACILITY Serial number MCHIP facility ID Start time: hh:mm (AM/PM): Researcher name End time: hh:mm (AM/PM): District name: Interview date: DD/MM/YY Tehsil name: UC name: City /village name: Entered by: BACKGROUND Name of CMW: Name of BHU/RHC attached with: Number of registered households: Registered population size: When did you graduate: From where: How long have you been practicing: How long have you been working in this location: Interviewer: I have informed the CMW about the evaluation scope and she has given her consent to fill this checklist. Observer signature: ___________________________________ 1. AVAILABILITY/FUNCTIONALITY OF SUPPLIES AND EQUIPMENT S. No SUPPLIES/EQUIPMENT STATUS (Circle the one number in each row corresponding to the status of item) Available and functional Available but not functional Not available 1.1 Safe Delivery Kit 1 2 3 1.2 Weighing machine (Adult type) 1 2 3 1.3 BP Apparatus 1 2 3 1.4 Baby weighing machine 1 2 3 MNCH PROJECT MID-TERM EVALUATION 141 S. No SUPPLIES/EQUIPMENT STATUS (Circle the one number in each row corresponding to the status of item) Available and functional Available but not functional Not available 1.5 Stethoscope 1 2 3 S. No SUPPLIES/EQUIPMENT STATUS (Circle the correct number corresponding to the status of each supply/equipment) Available Not Available 1.6 Chlorhexidine Gel 1 2 1.7 Kit Bag with functional items (check availability of sufficient NEW gloves) 1 2 1.8 Partographs 1 2 1.9 MNCH Cards 1 2 1.10 Daily Register 1 2 1.11 Referral slip 1 2 1.12 Monthly report 1 2 1.13 CMW database form 1 2 1.14 Stock register 1 2 S. No SUPPLIES/EQUIPMENT STATUS OF THE FOLLOWING (Circle one number in each row corresponding to the observed condition of the item) Satisfactory Unsatisfactory 1.15 Overall cleanliness and safety of facility 1 2 1.16 Privacy for consultation and also examination/service delivery 1 2 1.17 Safe disposal of medical waste 1 2 1.18 Hand washing station (soap AND water available) 1 2 MNCH PROJECT MID-TERM EVALUATION 142 S. No SUPPLIES/EQUIPMENT STATUS OF THE FOLLOWING (Circle one number in each row corresponding to the observed condition of the item) Satisfactory Unsatisfactory 1.19 Other remarks 2. AVAILABILITY OF MEDICINES/IEC S. No MEDICINES AVAILABLE (Circle one number in each row) WITHIN EXPIRY (Circle one number in each row) Yes No Yes No 2.1 Tab Paracetamol 1 2 1 2 2.2 Tab Misoprostol 1 2 1 2 2.3 Injection Diclofenac Sodium 1 2 1 2 2.4 Injection Oxytocin 1 2 1 2 2.5 Ferrous Fumerate 150mg + Folic Acid 0.5mg 1 2 1 2 2.6 BCC material/Protocols etc. displayed 1 2 1 2 2.7 Contraceptives (by type) 3. SERVICE RELATED DATA S. No SERVICE Write as appropriate (ask if number is increasing, decreasing or is the same as before MCHIP or when she began practicing.) 3.1 How many deliveries did you conduct during last month? (Enter number) 3.2 How many ANC (Antenatal Care) visits did you make during the last month? (Enter number) 3.3 How many PNC (Postnatal Care) visits did you make during the last month? (Enter number) MNCH PROJECT MID-TERM EVALUATION 143 S. No SERVICE Write as appropriate (ask if number is increasing, decreasing or is the same as before MCHIP or when she began practicing.) 3.4 How many pregnant women were referred for facility based care? (Enter number) 3.5 How many pregnancies are currently registered with you? (Enter number) 3.6 How many pregnant and lactating women received iron / folic acid tablets? (Enter number) (Circle one number for each row) Yes No 3.7 Are you aware of Jhpiego’s eMentoring Supportive Supervision? 1 2 3.8 If yes are you part of this project intervention? 1 2 3.9 Are you executing the CMW business model of Jhpiego? 1 2 3.10 If Yes, is your income increasing? 1 2 3.11 Are you part of the voucher scheme introduced by Jhpiego? 1 2 4. AVAILABILITY/FUNCTIONAL HBB AREA S. No MEDICINES/EQUIPMENT AVAILABLE? (Circle one number in each row) FUNCTIONAL? (Circle one number in each row) Yes No Yes No 4.1 Stethoscope 1 2 1 2 4.2 Ventilation bag mask 1 2 1 2 4.3 Suction device 1 2 1 2 4.4 Scissors 1 2 1 2 4.5 Ties 1 2 1 2 4.6 Gloves 1 2 1 2 4.7 Cloth 1 2 1 2 4.8 Head covering 1 2 1 2 MNCH PROJECT MID-TERM EVALUATION 144 S. No MEDICINES/EQUIPMENT AVAILABLE? (Circle one number in each row) FUNCTIONAL? (Circle one number in each row) Yes No Yes No 4.9 Timer (clock, watch) 1 2 1 2 4.10 Reference guide 1 2 1 2 5. TRAINED AND TYPE OF TRAINING (Circle one number in each row) Yes No a. Were you trained by MCHIP? If “yes”, go to question 5.2. If “no”, go to question 6 1 2 b. Did you get OJT by MCHIP? If “yes”, go to question 5.3. If “no”, go to next section (i.e., immediately after question 5.4 1 2 c. Did you get any reference material after the training? 1 2 d. Did you get any reference material after the OJT? 1 2 If you were trained by MCHIP, please select the topics on which you received the trainings (Please circle the number next to the topic that applies) 1. BEmONC Training (I & II) 2. Misoprostol & Chlorhexidine 3. FP counseling 4. Immunization 5. Infection Prevention 6. PPIUCD 7. HBB 8. Implant 9. Pnemonia and Diarrhea 10. Referrals 11. Record keeping 12. Enhancing Business Skills 13. Providing immunization at birth 14. (Essential) Newborn Care (ENC) 15. FP service delivery 16. Maternal nutrition 17. Breastfeeding support 18. Maternal immunization (TT) MNCH PROJECT MID-TERM EVALUATION 145 If you were trained by MCHIP, please select the topics on which you received the trainings (Please circle the number next to the topic that applies) 1. Focus ANC 2. Normal child birth 3. Birth preparedness & complication readiness 4. Postpartum care 5. PPH 6. Active management of 3rd stage of labour 7. Vaginal bleeding during pregnancy 8. Management of Pre-eclampsia /eclampsia 9. Rapid initial assessment & management of shock 10. Newborn care 11. Newborn sepsis 12. Breast feeding 13. Use of Partograph 14. Management of PPH 15. Infection Prevention 16. Supporting Maternal Nutrition 17. FP counseling 18. FP service delivery 19. Normal child birth - Beneficial Practices 20. Referrals 21. Others ____________________ 6. Please also indicate here if the CMW received training on the topics above but not from MCHIP. CMW received trainings from other than MCHIP (Circle one number) Yes No 1 2 Additional comments or observations MNCH PROJECT MID-TERM EVALUATION 146 Group Discussion with Women’s Support Group (WSG) Members Serial number Facility name MCHIP facility ID Start time: hh:mm (AM/PM): Researcher name End time: hh:mm (AM/PM): District name: Interview date: DD/MM/YY Tehsil name: UC name: City /village name: Entered by: Interviewer: I have informed WSG members about the evaluation scope and they have given their consent to conduct the interview. Moderator signature: ___________________________________ Note taker signature: ___________________________________ Timelines:  Start-up: (Time – 5 mins)  Discussion: (30-40 mins)  Wrap up and thanks – (3-5 mins) Process:  Introduction of team and purpose - ice breaker  Group demographic and background data reported on a simple sheet Respondents’ Profile 1 2 3 4 5 6 7 8 Name Age a. Less than 18 b. 18 -31 c. 32 -45 d. Above 45 Occupation MNCH PROJECT MID-TERM EVALUATION 147 Respondents’ Profile 1 2 3 4 5 6 7 8 Duration of engagement with WSG (in months) Marital status (Use qualifier as unmarried, pregnant, lactating, mothers with children under 5, mothers..etc.) Questions / discussion points for group interviews 1) 1.a.How often does the WSG meet in general? 1.b. For how long has this specific group been meeting? 1.c.Do group members remain the same or does the composition of the group change over time? (Check if other friends/ community members/ family members of WSG members join the some session – based on relevance) 1.d. How many pregnant women or women with young children are members of your group, even if they are not here today? 2) 2.a. What motivated you to become a member of the WSG? MNCH PROJECT MID-TERM EVALUATION 148 3) What are the major roles and responsibilities/functions of the WSG? 4) What are the health related topics most often discussed in the WSG? (Check all that apply) Twice or more in last 3 months ANC   MC nutrition (what mothers should eat and how often when pregnant and lactating)   PNC   Family planning (include implant or IUCD)   Newborn care   Child nutrition (breastfeeding, complementary feeding (including type, time of introduction, frequency, feeding the sick child)   Cooking demonstration   Care seeking for Pneumonia and Diarrhea   Misoprostol and how to use it?   CHX gel and how to use it?   Other (please specify)   5) Since last Ramadan, (June last year), have you perceived any changes in the health topics the WSG discusses? If yes, please describe the changes you have seen. (Note to interviewer: Relate to what MCHIP is working on in that particular district & facility) Specifically: i) Mother and child nutrition, (topics) ii) Immunizing your child, (topics) iii) Going for ANC, PNC? (topics) iv) What to do when your child is sick with diarrhea or fever/fast breathing? (topics) v) Going to health facility or CMW for delivery? Going for newborn checkup? (topics) vi) Getting transport for mother or child emergencies? (topics) vii) Family planning (topics, listen for implant or IUCD after delivery) 6) Have you recently received any materials about these topics (mother’s book, etc.)? (Show them a copy of the mother’s book from MCHIP). If yes, have all of the topics in the (book, poster, etc.) been discussed in your WSG? MNCH PROJECT MID-TERM EVALUATION 149 7) Is the health information the group acquires at WSG spread to the community at large? f yes, please explain how. If no, why not? 8) Have you been given information on Misoprostol & CHX? If yes, please explain how the community women in your circle have received the message (in case Miso and CHX was mentioned in list) Ask separately- check for women who actually used, or know women who have used it after their babies were born. (this will only be women who delivered in past few years) 9) In your opinion, does the WSG make a positive difference in the health of mothers and children in your community? If “yes”, how? If “no”, why not? Please give examples. 10)What are the major hindrances & challenges to health promotion by the WSG in your community? What possible measures can be taken to address these effectively? 11) Do mothers and families follow the health promotion recommendations related to mother and child health and nutrition or family planning? Do you have any suggestion to make it more effective and useful? Note Compilation and Observations: MNCH PROJECT MID-TERM EVALUATION 150 Transporters Group Interview Guide Serial number Start time: hh:mm (AM/PM): Researcher name End time: hh:mm (AM/PM): District name: Interview date: DD/MM/YY Tehsil name: UC name: City /village name: Entered by: Interviewer: I have informed transporter group interview members about the evaluation scope and they have consented to the interview. Moderator signature: ___________________________________ Note taker signature: ___________________________________ Group interview process:  Start-up: 5 mins  Group Demographic – introduction  Ice breaker Transporters’ Profile (Use additional sheets if necessary) 1 2 3 4 5 Name Age Sex Occupation Duration of Engagement Trained on referral mechanism (Y/N)? MNCH PROJECT MID-TERM EVALUATION 151 Group Interview Questions 1. What role have you played in implementation of MCHIP? 2. What institutional arrangements, if any, do you have with MCHIP in the district? 3. Are you charging the MCHIP referred patients? If yes, do you charge the market rate or a subsidized rate? 4. Were you trained in supporting MCHIP? (Circle one number) Yes No 1 2 If yes, can you describe the specific things you were trained to do? (Circle the number for all that apply) De-sensitization 1. Timely referral 2. To save the mother and baby, it is vital for the transporter to take the pregnant patient to the hospital/CMW at an opportune time, when her life can still be saved. Every second counts! 3. Be available on the phone. 4. An emergency transporter must be available via cell phone at all times and must receive all calls. 5. Keep phone charged with credit. 6. Can receive a call from a patient, LHW/CHW, SBA, or the AMAN tele-health agent. 7. Know local area, hospitals, and keep a small diary with local hospital addresses and numbers of the area. 8. Stay connected with AMAN tele-health. 9. Stay connected with the local LHW and CMW/SBA. 10. Fees and Payment: Payment will be decided by you and the patient’s family. Transporting and Positioning a Pregnant Patient: Log Rolling 11. Movement of a supine/prone pregnant lady. 12. Keep the woman lying on her left or right side with legs straight. 13. Keep a halfway folded bed sheet tucked under the woman as close to her body as possible. 14. Get one person each to kneel near her head, waist, and leg end (include yourself) 15. Grasp the woman and on the count of three, gently turn her onto the sheet. 16. Go to the other side and unfold the sheet, the sheet should now proportionately be under the woman. Now fold the sheet inwards to prevent sagging. 17. Utilizing the sheet to lift the women from head, waist, and leg end on count of three and place her on available transport. MNCH PROJECT MID-TERM EVALUATION 152 5. What precaution needs to be taken to avoid the following scenario: o A pregnant patient is lying flat on her back which is very harmful since it can reduce blood flow to the baby. 6. How do you find out about women in need of transport (who contacts you and how?) 7. Have you had the opportunity to transport any women using the training you received? If so, how many? (Circle one number) Yes No 1 2 If “yes”, how many? (Enter number) 8. Has anyone from MCHIP visited you for a follow up since you were trained? (Circle one number) Yes No 1 2 9. If you transport a woman, do you give a report to anyone? (Circle one number) Yes No 1 2 10. Will you continue to do this transportation work after the MCHIP project is over? If no, why not? MNCH PROJECT MID-TERM EVALUATION 153 Group Interview Guide for Beneficiary Pregnant Women and Mothers of Children under 1 Year Age – Non Facility Based Serial number Nearby facility name Nearby MCHIP facility ID Start time: hh:mm (AM/PM): Researcher name End time: hh:mm (AM/PM): Note taker name Interview date: DD/MM/YY District name: UC name: Tehsil name: City /village name: Interviewer: I have informed group members about the evaluation scope and they have consented to the interview. Moderator signature: ________________________________ Signature of interviewer: _____________________________ Timelines:  Start-up: (Time – 5 mins)  Discussion: (30-40 mins)  Wrap up and thanks – (3-5 mins) Process:  Introduction of team and purpose - ice breaker  Group demographic reported on a simple sheet Group Demographic Name Age Pregnant / young mother? MNCH PROJECT MID-TERM EVALUATION 154 Questions / discussion points (Record responses separately for each question and each woman interviewed) 1. a) Are / were you registered as a pregnant woman by the local LHW? If yes, in which month? Registered as a pregnant woman? (Circle one number) If “yes”, in which month of the pregnancy? Yes No 1 2 b) Did the LHW refer you to a health facility? If yes, to which type of facility were you referred for ANC and subsequent delivery? LHW referred? (Circle one number) If yes, to which type of facility were you referred? (Circle the number of all that apply) Yes No 1. BHU 1 2 2. RHC 3. Private 4. CMW 2. a) Has anyLHW or health facility staff member given you information or supplies for nutrition and feeding of a child? Did you receive CHX/Misoprostol pack and/or pregnant woman hand book from an LHW? Received information or supplies? (Circle one number) If yes, what information or supplies? (Circle the number of all that apply) Yes No 1. Your nutrition and feeding of the child 1 2 2. CHX/Misoprostol pack 3. Pregnant woman handbook MNCH PROJECT MID-TERM EVALUATION 155 b) Do you / did you go for ante-natal checkup (ANC), if yes, how many visits do you plan to/did you have in the total course of pregnancy? Had ANC? (Circle one number) If yes, number of visits? (Enter number) Yes No 1 2 c) If yes, where do/ did you go for ante-natal checkups? Facility for ANC (Circle one number in each row) If “yes”, please explain why you chose this facility and your experience with the ANC services Yes No Public facility/CMW clinic 1 2 _____________________________________ Private facility 1 2 If “yes”, please explain why you did not choose a public facility/CMW clinic? ______________________________________ 3. Where do you plan to/did you go for delivery? Facility for delivery (Circle one number in each row) If picked “Yes”, please explain why you chose this facility and your experience with the delivery services Yes No Public facility/CMW clinic 1 2 _______________________________________ Private facility 1 2 If picked “Yes”, please explain why you did not choose a public facility/CMW clinic. _______________________________________ MNCH PROJECT MID-TERM EVALUATION 156 4. Do /did you experience any complication/ issues related to pregnancy, childbirth, or neonate/ infant? If yes, where did you go for appropriate care/ services? Do /did you experience any complication/issues related to pregnancy, childbirth or neonate/ infant? (Circle one number) If yes, where did you go for the appropriate care/ services? Yes No 1 2 5. Did you require caesarian section? If yes, where? Did you require a caesarian section? (Circle one number) If yes, where did you go? Yes No 1 2 6. Do/ did you know about the referral transport facility available? If yes, do you plan to/ did you use it? If not, why? 7. Where do you plan to/ did you go for newborn care? If you chose a public facility/ CMW clinic, why do/ did you choose it & how did you find the child under 1 year health services? If you opted for a private facility, explain the reason for not choosing a CMW clinic or public health facility for delivery. Facility for newborn care (Circle one number in each row) If “yes”, please explain why you chose this facility and your experience with the newborn services. Yes No Public facility/CMW clinic 1 2 MNCH PROJECT MID-TERM EVALUATION 157 Facility for newborn care (Circle one number in each row) If “yes”, please explain why you chose this facility and your experience with the newborn services. Yes No Private facility 1 2 If “yes”, please explain why you did not choose a public facility/CMW clinic. 8. Have you heard of/ experienced any improvement in the service delivery at the public facility in the last 2-3 years? If yes, in which services? Explain the possible reasons for improved services. 9. In your opinion, how can service delivery be improved at local BHUs/ RHCs? What changes do these facilities require? What issues do you see when striving for improved service delivery? 10. Other remarks? MNCH PROJECT MID-TERM EVALUATION 158 QIT Group Interview Guide and Check List Serial number Facility name MCHIP facility ID Start time: hh:mm (AM/PM): Researcher name End time: hh:mm (AM/PM): District name: Interview date: DD/MM/YY Tehsil name: UC name: City /village name: Entered by: Interviewer: I have informed QIT members about evaluation scope and they have consented to conduct the interview. Moderator signature: ___________________________________ Note taker signature: ___________________________________ BACKGROUND – fill from QITs record Sr. # Question Response 1.1. Name of QIT focal person 1.2. Designation 1.3. QIT formulation Males # Females # Total # Health Facility Members # Community Members # (Request for updated list) 1.4. Is QIT Functional? (Circle one) 1. Yes 2. No 1.5. If yes, frequency of meeting MNCH PROJECT MID-TERM EVALUATION 159 Sr. # Question Response 1.6. If no, give reason ________________________________________ 1.7. Frequency with which MCHIP staff attend meeting (recall, last 3 meetings)) a) 3 b) 2 c) 1 d) 0 1.8. QIT Register available (Circle one) 1. Yes 2. No 1.9. Check for a random month entry 1.10. QIT formation and partnership defined quality based findings available 1.11. Action plan present (mounted – graphed?) (Circle one) 1. Yes 2. No 1.12. Follow-up plan available (Circle one) 1. Yes 2. No QIT Group Interview Guide  Start-up: 5 mins  Group Demographic and back ground data reported on a simple sheet – introduction  Ice breaker QIT Members’ Profile (Use additional sheets if necessary) 1 2 3 4 5 Name Age Sex Occupation Duration of engagement Trained on PDQ approach (Y/N)? MNCH PROJECT MID-TERM EVALUATION 160 1 2 3 4 5 Understand purpose of PDQ? Participant of initial action plan exercise(Y/N)? 1. What motivated you to become a QIT member? Did you volunteer or were you selected? If selected, by whom? 1 2 3 4 5 Motivation Volunteer Selected (by whom) 2. What do you do as a QIT member? 3. Have you received any QIT work related training since 2013? If yes, what training have you received (listen for PDQ or a description of what it is)? From whom, when, and where? Respondents 1 2 3 4 5 NoYes What When By Whom Where 4. How often does the QIT meet? Where? Who calls the meetings? Do you receive any support for attending meetings (e.g., transport or stipend)? o How often o Called by MNCH PROJECT MID-TERM EVALUATION 161 o Where o Stipend/ support (name) 5. Do you have an Action Plan? If yes, have you taken steps to implement the plan? Please share the process of how you work to achieve your action plan (lead, support, follow￾up, etc.)?  NO (give number of respondents)  YES o Steps o Lead o Support o Follow-up 5.a. Do you have any discussions in your meetings about the following? General information about helping to increase access to quality healthcare for mothers, newborns and children such as: i. Encouraging or helping the mother and/or child to go for checkups or when they are having health problem? If yes, please provide examples? ii. Improving the quality of the facility for mothers and babies’ healthcare? iii. Knowing when to refer a mother and/or baby somewhere for care? iv. Helping the mother and/or baby to get care (identifying when there are danger signs or complications, obtaining transport, knowing where to go, etc, encouraging community members to take action to get care? ) v. Do you undertake any advocacy with the HF for increase supply of services such as (more staff, different type of providers, longer hours, etc?) If yes, where do you direct your input(s)? (feeds into question 6) 6. Please describe the ways in which you interact with health facility healthcare decision makers, if at all? How often does this take place and at what level (facility/ district/ other)? o Frequency of interaction with decision makers o Decision maker level (facility, district…etc.) 7. How much progress has the QIT made on implementing the Action Plan so far? What are the main supporting / hindering elements? Supporting elements Hindering elements MNCH PROJECT MID-TERM EVALUATION 162 8. Do you modify the Action Plan with changing needs/ requirements with time? If yes explain with experience, if any? If no, why not?  NO Reason  YES Explain 9. Do you find QIT meetings useful in improving the quality of care at your health facility? If not, why not? Do you have ideas on how it can be made more effective?  NO Reason  YES Explain Suggestions 10. In your opinion, what are the main contributions of the QIT (facility, service, client satisfaction levels, etc.). Explain the key or most effective MNCH related contributions in detail. If the QIT does not contribute, can you give reasons why?  NO Reason  YES Details In your opinion, can the QIT become effective at enhancing the accountability of the service provider/facility regarding MNCH, FP on a regular basis without any outside support (from an NGO, project, etc.)? If yes, please give your ideas for how the QIT can become sustainable. 11. Do you receive any support from MCHIP (they may say Save the Children) for organizing the work of the QIT? If yes, please describe the support, i.e., the activities they do with you. (Listen for MNCH-specific topics) 12. How often, if at all, does someone from the MCHIP project visit you? Please be specific. MNCH PROJECT MID-TERM EVALUATION 163 13. What information or data, if any, do they collect from you? Do you use any of this information in planning the activities of the QIT? 14. Do you receive any feedback from the MCHIP staff member about the work you are doing and the information you collect in the QIT? If yes, please give examples. 15. Additional comments arising from the discussion: 16. Do you have any other comments? U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523