Expanding Maternal and Neonatal Survival in Indonesia July 2017 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Management Systems International. FINAL EVALUATION OF THE INDONESIA EXPANDING MATERNAL & NEONATAL SURVIVAL PROGRAM July 2017 Contracted under AID-497-C-16-00006 COVER PHOTO: In Indonesia, Neng Supartini experienced a safe and healthy delivery in Ciparay Health Center thanks to USAID’s Expanding Maternal and Neonatal Survival (EMAS) program. DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM i TABLE OF CONTENTS Expanding Maternal and Neonatal Survival in Indonesia ............................................... i Acronyms, Abbreviations and Definitions ..................................................................... iii ACKNOWLEDGEMENTS .............................................................................................. vi Executive Summary.......................................................................................................... 1 Evaluation Purpose and Evaluation Questions.........................................................................................1 Project Background........................................................................................................................................1 Evaluation Methods and Limitations...........................................................................................................2 Lessons Learned........................................................................................................................................... 11 Findings and Conclusions..............................................................................................................................2 Recommendations..........................................................................................................................................9 I. Evaluation Purpose & Evaluation Questions ............................................................. 13 Evaluation Purpose...................................................................................................................................... 13 Evaluation Questions.................................................................................................................................. 13 II. Project Background.................................................................................................... 14 Maternal Newborn Health in Indonesia................................................................................................. 14 The EMAS Program..................................................................................................................................... 14 III. Evaluation Methods & Limitations........................................................................... 16 Site Selection ................................................................................................................................................ 16 Key Informant Interviews.......................................................................................................................... 16 Key Documents and Data Sources......................................................................................................... 17 Data Analysis................................................................................................................................................ 17 Limitations of the Evaluation Methodology........................................................................................... 18 Ethical Considerations................................................................................................................................ 19 IV. Findings & Conclusions ............................................................................................. 19 Question 1: Improved outcomes and impact of project approaches............................................. 19 Question 2: Explanations and evidence of successful approaches................................................... 19 Question 3: M&E, use of evidence and reporting................................................................................ 22 Question 4: Lessons learned & opportunities for future................................................................... 23 V. Recommendations...................................................................................................... 48 Question 1: Improved outcomes and impact of project approaches............................................. 48 Question 2: Explanations and evidence of successful approaches................................................... 49 Question 3: M&E, Use of evidence and reporting............................................................................... 50 Question 4: Lessons learned & opportunities for future................................................................... 50 FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM ii VI. LESSONS LEARNED................................................................................................ 52 Annexes............................................................................................................................ 54 Annex I: EMAS Final Evaluation Statement of Work .......................................................................... 54 Annex II: Evaluation Methods and DATA COLLECTION TOOLs ................................................ 62 Annex III: Sources of Information: KIIs and documents reviewed .................................................. 69 Annex V: RESULTS AGAINST TARGETS: EMAS REPORTED & EValuation team reanalysis. 83 Annex VI: REVIEW OF PMP INDICATORS......................................................................................... 91 Annex VII: INTERNATIONAL STANDARDS for QOC Indicators............................................... 94 Annex VIII: FACILITY BASED Quality of CARE.................................................................................. 95 FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM iii ACRONYMS, ABBREVIATIONS AND DEFINITIONS APBD Local Government Annual Budget APH Antepartum Hemorrhage BAPPEDA Development Planning Agency at Sub-National Level (Badan Perencanaan Pembangunan Daerah) BEmONC Basic Emergency Obstetric and Newborn Care BPJS Kesehatan National Health Care Security (Badan Penyelenggara Jaminan Sosial Kesehatan) BUKR Bina Upaya Kesehatan Rujukan Bupati Head of District CEmONC Comprehensive Emergency Obstetric and Newborn Care CFR Case Fatality Rate C/S C-Section CSO Civil Society Organization CSR Corporate Social Responsibility Dana DEKON Deconcentration Fund DAK (BOK) Special Allocation Fund (Health Operational Assistance) DEC Development Exchange Clearinghouse DHO District Health Offices DHS Demographic and Health Survey DOH Department of Health EMAS Expanding Maternal and Neonatal Survival EmONC Emergency Obstetric and Neonatal Care ENAP Every Newborn Action Plan (National) FMM Civic Forum (Forum Masyarakat Madani) FP Family Planning GOI Government of Indonesia GP General Practitioner HMIS Health Management Information System FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM iv HR Human Resources IBI Indonesian Midwives Association IDAI Indonesian Pediatric Society IDHS Indonesia Demography Health Survey IDI Indonesian Medical Association (Ikatan Dokter Indonesia) IT Information Technology Jhpiego Johns Hopkins Program for International Education in Gynecology and Obstetrics JKN National Universal Health Insurance Program KARS Commission of Hospital Accreditation (Komisi Akreditasi Rumah Sakit) KII Key Informant Interview LKBK Lembaga Kesehatan Budi Kemuliaan M&E Monitoring and Evaluation MCH Maternal and Child Health MESP Monitoring and Evaluation Support Program MgSO4 Magnesium Sulfate MKIA Maternal Child Health Motivators MN Maternal and Newborn MNH Maternal and Newborn Health MOH Ministry of Health MOU Memorandum of Understanding MPA Maternal Perinatal Audit OB/GYN Obstetrician/Gynecologist PE/E Pre-Eclampsia/Eclampsia PHO Provincial Health Office PMP Performance Monitoring Plan POGI Indonesian Society of Obstetrics and Gynecology Pokja Working Group FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM v PPH Post-Partum Hemorrhage PROKESGA Healthy Family Program (Program Keluarga Sehat) PUSDATIN Center for Data and Information (MOH Indonesia) Puskesmas Health Center at Sub-District Level (Pusat Kesehatan Masyarakat) QOC Quality of Care RFP Request for Proposals RTI Research Triangle Institute TB Tuberculosis SijariEMAS Referral Exchange System SOP Standard Operating Procedure SPGDT Integrated Emergency Response System (Sistem Penanggulangan Gawat Darurat Terpadu) SPSS Statistical Package for the Social Sciences UHC Universal Health Care USAID United States Agency for International Development WHO World Health Organization FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM vi ACKNOWLEDGEMENTS The EMAS Evaluation Team is extremely grateful to the more than 100 individuals who generously gave their time and detailed, frank and fruitful insights to this report. Throughout this evaluation, the team benefited from the contributions of dozens of Ministry of Health (MOH) directors, district health managers and health care providers, as well as many other central and district authorities in Indonesia. As such this report represents a collective effort and incorporates opinions and observations of a large community of key stakeholders and influencers in the future efforts to improve maternal newborn health. The team would especially like to thank the Zohra Balsara, Sigit Sulistyo and Maria Pinto from the United States Agency for International Development (USAID)/Indonesia, whose unfailing support, guidance, direction and dedication to this work and future efforts in maternal and child health (MCH are laudable. The team is also very appreciative of the time and assistance received from EMAS partners in document and data sharing, numerous interviews and meetings and notably, the former field staff who spent valuable time ensuring our field visits were well organized and fruitful despite the fact they were no longer working for the project. The ongoing technical guidance and faultless logistical support provided by David Callihan, Jonathan Simon, Retno Sri Handini and Nuri Kusuma Wardani from Management Systems International’s Monitoring and Evaluation Support Project were invaluable to the team throughout this work. To all who are committed to improved maternal newborn service delivery and outcomes in Indonesia, we offer these observations, analyses, and recommendations in the confidence that further progress will be realized through collective efforts. FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 1 EXECUTIVE SUMMARY Evaluation Purpose and Evaluation Questions The purpose of the evaluation of the Expanding Maternal and Neonatal Survival (EMAS) program in Indonesia was to learn from program successes and challenges. The main objectives of this performance evaluation were: • To build on the work in the midterm evaluation by identifying what factors and interventions contributed to the successes and shortcomings of EMAS, as well as identify and document opportunities for further improvement. • To help inform future maternal and child health (MCH) interventions in Indonesia, especially given the context of universal health care (UHC), persistently high maternal and newborn mortality rates and continuing devolution of health management and service delivery authorities. The following key questions guided this performance evaluation: • Question 1: Did EMAS’s primary approaches, as implemented, lead to improved outcomes, and did they have the desired magnitude of impact? • Question 2: What are the explanations for why these approaches worked or did not work, and what evidence supports these explanations (e.g., facilitation, limitation, context, continuum of impact)? • Question 3: How did EMAS learn and use evidence to make adjustments in program approaches during implementation? How can USAID internal management processes better support its programs and the Ministry of Health (MOH)? • Question 4: What has been learned that can inform future priorities, approaches and activity processes? Key findings, conclusions, and recommendations for each question are presented below, followed by several lessons learned. Project Background Despite a long history of social programs and a strong policy environment in support of maternal, newborn and child health, Indonesia’s health indicators remain poor, with reports of stagnant and at times increases in mortality rates: the most recent Demographic and Health Survey (DHS), from 2012, reports a maternal mortality ratio of 359 deaths per 100,000 live births, and a newborn mortality rate of 19 deaths per 1,000 live births. The majority of births (73 percent) are assisted by a skilled provider, yet a large percentage of maternal deaths occur in facilities. Women and newborns often receive substandard maternal and newborn emergency care, even in hospitals. To improve maternal and neonatal health in the country, the Government of Indonesia (GOI) has taken steps that include development of a national strategy to address maternal mortality, development of a national child health policy and national action plan to improve newborn health and devolution of authority to local governments. The Expanding Maternal and Neonatal Survival (EMAS) Program was USAID/Indonesia’s flagship MCH effort to support the GOI in its efforts to reduce maternal and newborn mortality by FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 2 focusing on three major objectives: 1) improved quality of emergency obstetric and neonatal care (EmONC) services in hospitals and community health centers; 2) increased efficiency and effectiveness of referral systems between community health centers and hospitals; and 3) strengthened accountability among government, community and health systems. The EMAS Consortium consists of five partners: Johns Hopkins Program for International Education in Gynecology and Obstetrics (Jhpiego, the prime), Budi Kemuliaan, Muhammadiyah, Save the Children and the Research Triangle Institute (RTI). Over its five years, EMAS worked with GOI agencies (national, provincial and district), civil society organizations (CSOs), 123 public and private hospitals and 287 community health centers in 30 districts and cities in six provinces. Evaluation Methods and Limitations A team of five consultants conducted this evaluation between January 10 and April 20, 2017. It covers the period from the midterm evaluation in 2014 through March 2017. The evaluation used a mix of qualitative and quantitative data collection and analysis, including: extensive document review; review and analysis of progress toward 10 key indicators; analysis of other results that were separate from the performance monitoring plan (PMP); and 100 key informant interviews (KIIs) with more than 220 respondents in Jakarta and across 10 districts. The team encountered delays in receipt of routine monitoring data from the project and, most critically, faced limitations in assessing the program’s effectiveness due to the quality of data received. Notably: • Substantial delays in receipt of data. It took over six weeks for the evaluation team to receive the complete routine monitoring data as requested. • Constraints and quality limitations of data received. Notably, gaps, duplications, inconsistencies and misreported data across multiple data sets and bases. Selection of districts, facilities and participants used non-probability sampling methods, so the evaluation team was not able to generate findings that statistically represented the larger population from which they were drawn. Since key informants constituted the primary source of information, interview data was subject to personal bias, opinion and recollection, which the team mitigated by meeting frequently to compare and analyze findings as a group. Findings and Conclusions I. Did EMAS primary approaches lead to improved outcomes and desired impact? The projects PMP included 22 indicators, of which three were outcome measures: one outcome measure of case fatality rate (CFR) for maternal mortality, and two outcome measures for newborn mortality. While EMAS achieved notable results regarding MCH processes, institutions, and practices in supported locations – as described in program reports and this evaluation – looking exclusively at these outcome measures, the project did not achieve targets set in the program design and SOW. Specifically EMAS did not achieve the outcome target for CFR in any program phase or year. Also, though no targets were set for newborn mortality, reported rates were stagnant over the life of project. On average, CFR reported (range: 1.4 to 2.7 percent) was almost double the target (1.2 to less than 0.1 percent) in all years. The project developed its own definition of CFR1 which resulted in 1 The project’s definition of CFR included a number of co-morbidities, but not causes of death nor in line with WHO standard categories use to calculate CFR. FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 3 a significantly lower reported CFR compared to the rate calculated using the globally accepted World Health Organization-defined (WHO) CFR. The team re-analyzed quarterly data to calculate the proportion of hospitals that achieved CFR <0.1 percent as a proxy measure of outcome improvements in project facilities: In aggregate, facilities with CFR <0.1 percent increase over three years. However, results by phase are mixed and the inconsistent quality of the data restrict a more robust or conclusive analysis. The team identified other discrepancies between reported results and those calculated from quarterly data notably the percentage of facilities conducting maternal audits and those with 80 percent standard operating procedure (SOP) compliance. The project’s mix of 22 indicators was, on one hand, too high-level (three impact mortality measures); and on the other hand, too low-level, with 7 process and 12 activity-level indicators. The project’s PMP lacked mid-level outcome measures that would more clearly link results of project activities with morbidity level outcomes. For example, measures of total caseload of post-partum hemorrhage (PPH) and pre-eclampsia more closely correlate with project activities and measures of provision of uterotonic to prevent PPH, and Magnesium Sulfate (MgSO4) to stabilize or treat eclampsia. They also have higher denominators and far less confounding factors than those in calculations of CFR. Team results of correlations for selected activities with mid￾level outcomes found: • A higher percentage of uterotonic given in the third stage of labor significantly reduced the total caseload for PPH; and • A higher percentage of corticosteroid given to women in premature labor (24 to 36 weeks) demonstrated a lower neonatal asphyxia fatality rate and a reduction in total cases of birth asphyxia, though the correlation was low. Given the constraints in data, results of these correlations should be taken with caution; for each of the positive correlations presented above, the team found an equal or greater number of negative or inconclusive correlations. As such, presentation of these is primarily illustrative, demonstrating how a more robust indicator mix would better serve the project and its key stakeholders. Results of baseline and endline studies were not included in this evaluation, given a variety of constraints in the methodology, site selection and sample size, timeline and presentation of preliminary findings. Conclusions: The project did not achieve its three outcome indicator targets, despite measurable results and successes from approaches in several areas. However, no certain or direct correlation can be drawn or made regarding the impact of project activities on maternal newborn outcomes. The project’s indicator mix was not sufficiently robust to measure mid-level outcomes linked to project activities to better assess project results. The quality of data provided precluded this team from reliably and confidently re-analyzing the data to draw any further conclusions regarding the results of EMAS approaches in improving outcomes or impact. 2. What are the explanations and evidence for why approaches did or did not work? Analysis and key findings follow, presented by each of EMAS’s three objectives. These include findings related to health centers at sub-district level called puskesmas, which are found throughout Indonesia. Objective 1: Improved quality of emergency obstetric and neonatal care (EmONC) services in hospitals and puskesmas. The program’s first objective used peer-based clinical mentoring to strengthen clinical governance to improve the quality of maternal and newborn emergency care in health facilities. Clinical mentoring focused on improved data collection and use (through the FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 4 introduction of the new streamlined maternal register and use of dashboards), increased compliance and use of SOPs to improve emergency clinical care, introduction of emergency drills and facilitating death and near miss reviews. Over the life of project, clinicians from more than 180 facilities (50 hospitals and 133 puskesmas) served as mentors to more than 120 hospitals and nearly 290 puskesmas. In Year 5, EMAS reported that 20 of 30 EMAS districts had – with support from the pokjas – used district funding to expand various EMAS interventions to more than 70 hospitals and 500 puskesmas beyond those served directly by EMAS. The most commonly replicated interventions were clinical mentoring and rotations, performance tools, new maternal registers and emergency trolleys. Accomplishments reported by KIIs include: improved adherence with SOPs; better preparation for accreditation; use of standardized maternal registers, dashboards and job aids; improved communication between facilities; increased deliveries managed at the puskesmas level and fewer referrals; and staff better able to respond to emergencies. The “near-miss audits” introduced were perceived as less threatening than “death audits.” Constraints to mentoring identified included “watering down” of quality of many mentors in phases 2 and 3; too narrow a focus on emergency response; the need to update newborn SOPs; insufficient emphasis at puskesmas to help midwives follow SOPs; and only 30 percent of KIIs being able to provide examples of changes resulting from audits. Objective 2: Increased efficiency and effectiveness of referral systems between community health centers and hospitals. EMAS’s second objective was a set of inter-related interventions designed to increase the efficiency and effectiveness of the referral system. Long identified as a critical constraint/challenge to reducing maternal and newborn mortality, the new “SijariEMAS” referral exchange system was widely praised as the first comprehensive referral system developed and implemented in Indonesia and successfully addressed many of the previous system failures. Increased efficiency and effectiveness reported by KIIs include: improved communications and coordination between health facilities during the referral process; stabilization by midwives; readiness of referral network hospitals to deal with incoming cases; relationships and communications between hospitals and puskesmas; and the monitoring and evaluation (M&E) system to track referral data. All districts visited had – with support from the pokjas - committed funds to replicate/sustain the referral system and memoranda of understanding (MOU). SijariEMAS served as the foundation for other or further developed emergency services. Some districts have begun integrating SijariEMAS and 119 services. Constraints noted in KIIs include: text-based communication was difficult to do during emergencies and phone reception was often poor; inefficiencies in time and cost with call centers placed at district health offices (DHOs); lack of long-term planning for the information technology (IT) system; and too little consideration of newborn referrals. Objective 3: Strengthen accountability among government, the community and health system. The third objective of EMAS, governance accountability, was achieved through establishing the pokja, a technical working group of key representatives from government agencies, professional associations and health providers to generate political support for maternal and newborn survival. In all 30 EMAS districts, pokjas were their first intervention to create a legal basis to authorize quality of care improvement such as the development of a referral protocol for maternal and newborn health (MNH) emergencies; participation in facility maternal-perinatal audits (MPAs); assessments of compliance with clinical SOPs; and facility service charters to inform the public about facility services, performance and fees. All pokjas were able to secure funding from the local government annual budget (APBD). Constraints identified include limited evidence of participation in district death audit processes and the need for ongoing technical assistance noted in the majority of KIIs. EMAS also established the Civic Forum in all 30 districts, FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 5 as the community coordinating bodies for channeling MNH feedback to local government, Parliament, health facilities and providers. Selected Civic Forums successfully partnered with the private sector to raise funds to inform and promote MNH services in communities. Civic Forums were expected to advocate for new regulations and promote a culture of power-sharing among partner CSOs, mobilizing support for the new referral system and promoting maternal and newborn (MN) services and facility-based births. However, EMAS reported that only 50 percent of Civic Forums took action to leverage private sector funds, and KIIs’ results confirmed that none of the Civic Forums were able to secure long-term funding from the private sector. Pokjas were effective in securing financial resources and advocating for local regulations (e.g., the Bupati’s decree in support of facility-based births). In terms of replication and sustainability, district authorities perceive Civic Forums as a potential threat (“too much” accountability to the public) and they have been the least replicated component within EMAS. Conclusions: Reported EMAS results, replication of approaches within EMAS districts and beyond and key findings from KIIs provide substantial evidence of successes across various EMAS approaches in all three objectives. Accomplishments reported by KIIs include: • Objective 1: Mentoring has made significant progress in capacity building and skills development of key cadre of staff at both the hospital and puskesmas levels. KIIs report increased technical capacity of health workers as a result of mentoring, supervision, SOPs, clinical tools and emergency drills. Key accomplishments beyond the program’s mandate include increased efficiency and ability to meet criteria for accreditation and improved communications and knowledge sharing between specialists and midwives. • Objective 2: Vastly improved efficiency, access and quality of emergency MN services through the combined efforts of EMAS in the design, introduction and expansion of the country’s first dedicated, IT-based maternal newborn referral system. • Objective 3: Increased political and financial support for MN issues as a result of the pokjas and increased community engagement and awareness of MN issues and services through Civic Forums, but limited ability to communicate and advocate for funding with district officials. Constraints and challenges were also noted across all interventions, although the examples and evidence of accomplishments provided by KIIs far outweighed the constraints. Replication of EMAS approaches and commitment of funds in 2017 and 2018 are further evidence of successes across the various approaches. Unfortunately, routine data collection and reporting did not include any analysis or correlation with the data collected by partners in mentoring, referrals or development of local governance entities; without this, available evidence is insufficient to analyze and quantify the relative achievements of each approach and provide solid evidence as to whether and to what extent the various approaches are effective in improving MOH systems, QOC and maternal newborn outcomes. 3. How did EMAS use evidence to adjust program approaches in implementation? How can USAID internal management better support its programs and the MOH? USAID: The request for proposals (RFP) and original program design were ambitious, with targets of 25 percent reduction in national MN mortality. Yet the geographic coverage was limited – both in number of districts/provinces (only 30 districts and six provinces) as well as in the number of facilities to be included (estimated to be 25 percent of facilities in project areas, but ultimately, on average only 14 percent). The technical scope and scale for the program were narrowly defined, targeting emergency QOC issues at the facility level, alongside a list of areas in which the program was restricted from working. This list included gaps in human resources FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 6 (HR), notably obstetricians/gynecologists (OB/GYNs), arguably the biggest challenge to improve MN outcomes. The program was also restricted from working with private midwives and from addressing pre-service education, despite the estimated 30 to 40 percent of births performed by private midwives and multiple studies that note the critical lack in skills to manage and refer complicated cases. Alongside ambitious impact level targets and the narrowly focused targeted approach defined, the RFP also states that the program should “assess the problems that need to be addressed to improve QOC of MN complications. For every system failure that impacts maternal health, technical strategies will be identified to address these.” In sum, the goal and objective of the program were conflicting and unclear; the program was intended to measure successes of a discrete set of emergency QOC interventions in selected facilities but also tasked with addressing all system failures and, in five years, reducing mortality rates by 25 percent. KIIs noted the anticipated delay between the close of EMAS and execution of the new contract and start-up time for the follow-on project. These delays strain continuity of selected activities identified as needing on-going technical assistance beyond that which the MOH is able to provide. Respondents also noted the need for better alignment and coordination with USAID’s portfolio of MCH activities and a broader need for improved donor coordination to serve priority MOH needs. One of the most critical issues that KIIs noted, and the management review documented, was the need to improve USAID oversight and management of its programs, particularly in M&E, and in ensuring responsiveness to MOH needs. While MOH respondents praised USAID for the Agency’s coordination and engagement with the MOH during project design, respondents indicated insufficient engagement during implementation; repeated requests for improved reporting, information-sharing and presentation of data that was sufficiently detailed and in a useful format never resulted in any change. EMAS: As noted under the Question 1 findings, the project’s indicator mix was not sufficiently robust to measure mid-level outcomes linked to project activities, or to use as an evidence base to shift approaches and activities as needed. A baseline study was done in Year 4 and the endline in Year 5, with comparative data from one year of implementation in phase 3 districts only; both too late and too short a timeframe to provide any robust analysis of the project’s impact over five years and three phases. As a result, the program was not sufficiently able, over its life, to analyze correlations between project inputs and the resulting outcomes or impact. In the 2014 management assessment and midterm evaluation, seven of 19 recommendations addressed issues in M&E and communications. The program addressed some aspects of selected recommendations and did not address others. Both assessments outlined the need for deeper and better analysis and presentation of the linkages between activities and changes in morbidity and mortality. After the midterm evaluation, EMAS revised its PMP and a few indicators were dropped. Aside from this, the project made no major shifts in indicators, nor did it conduct a reanalysis of data. In 2105–2016, EMAS published two maternal and perinatal death reviews. Both identified the same critical constraint in QOC, linked to inadequate patient monitoring. Routine patient monitoring was not a focal area included in the RFP and results of these reviews were published in the project’s last year, too late to feasibly shift its approaches. Still, conclusions and recommendations in these reports do not highlight this issue as a priority focus nor make recommendations for the future. KIIs with the MOH at all levels stated that EMAS reporting hampered the ability of key stakeholders to meaningfully use the results and data as presented. EMAS reports focused on results against targets set in its PMP (as required for USAID), with little reporting on results not related to EMAS or on constraints in achieving its targets. EMAS reports included selected results of disaggregated CFRs (such as PPH and eclampsia) where the project appears to have had an impact, without due diligence to all-cause mortality, potentially misleading readers to FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 7 wrongly assume that all-cause CFR was dropping in project areas. EMAS reports used terminology such as “full coverage” and “full support” of project districts, when in reality, average facility coverage was 14 percent. The definition of “full support” and ”total coverage” included “limited support” to some puskesmas – which is confusing. EMAS annual results were reported by phase, but aggregate annual results of all three phases were never reported, even at end of the project. Results presented by phase were an aggregate of selected facilities and districts, and results by phase compared without adequately noting changes in total facilities and districts within and between phases, or explaining the shifts in denominators, making reports of limited or no use to individual DHOs and facilities. Districts have a fixed number of facilities and the denominator for all indicators should have fixed definitions and set denominators, regardless of expansion or drop of facilities over the life of project. Finally, KIIs noted that Jhpiego did not openly share data, despite repeated requests from the MOH at all levels, EMAS partners and field staff and USAID for improved data sharing and access. Conclusions: Design aspects in the EMAS RFP and limited oversight from USAID during implementation contributed to the limited M&E, reporting and use of evidence over the program’s life. One of the most critical issues noted in KIIs and the management assessment report was the need for improved USAID oversight and coordination of its programs, particularly in M&E, to ensure responsiveness to MOH needs. The project’s indicator mix was not sufficiently robust to measure mid-level outcomes linked to project activities, nor was the wealth of data collected by the project analyzed or used as an evidence base to shift approaches as needed. M&E and reporting focused primarily on showcasing selected positive program results and for USAID reporting purposes, with too little analysis and use of data either to shift approaches or assist the MOH and other stakeholders to identify gaps and needs beyond the program’s mandate. As a result, EMAS did not use evidence to shift approaches, and reports and presentation of data were of limited use to entities outside the program, including the MOH. 4. What has been learned to inform future priorities, approaches and activities? The team identified 11 key factors to inform future priorities, approaches and activities. Of these, four are “external” factors, or constraints to MN outcomes, which are beyond the scope or influence of USAID programs and pertain to GOI and MOH legislation and regulations. Four are “missed opportunities” or areas beyond the scope of EMAS as defined in the RFP, and three are lessons learned from interventions within the EMAS program. Four external factors are major constraints to MN outcomes: 1. Many facilities certified in basic emergency obstetric and newborn care (BEmONC) cannot offer two of seven WHO-defined signal functions, manual/curette aspiration and vacuum/forceps-assisted deliveries; and midwives cannot perform a third, intravenous antibiotics, without a doctor present. In most countries, midwives provide these services after receiving training. Many hospitals certified in comprehensive emergency obstetric and newborn care (CEmONC) cannot consistently provide two WHO-defined signal functions, reliable availability of blood for transfusions and timely provision of caesarian sections. II. Only OB/GYNs are authorized to perform cesarean sections, but a huge shortage of OB/GYNs and anesthesiologists preclude timely provision of both cesarean sections and many other life-saving services. III. By increasing universal health coverage, the national universal health insurance program (JKN), should help lower maternal and newborn morbidity/mortality. FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 8 However, some of its reimbursement rules, e.g. capped payments for newborn complications and estimated population based payments to puskesmas, are likely to have unintended consequences leading to both increased costs secondary to increased maternal/newborn morbidity and to compromised access and equity. IV. Midwife pre-service education requirements are weakest in the single most important area: hands-on experience in performing many deliveries. Midwives graduate having observed or performed too few deliveries. Such an education system is a betrayal to these graduates themselves, but more so to the women, babies and families of Indonesia. Four key missed opportunities in clinical QOC were identified: I. The project focused almost exclusively on emergency responses, as per the project design. However, death audit reviews revealed that failure to follow facility SOPs throughout the hospital stay was a significant contributor to both maternal and newborn deaths. II. Private midwives perform 30 to 40 percent of deliveries in Indonesia. Many have had little hands-on experience in normal deliveries, let alone high-risk pregnancies and deliveries. Missed opportunities to upgrade midwife skills include: WHO-supported tools to support midwife care such as job aids and checklists, and increased staff rotations for non-hospital midwives in hospitals with extensive as well as specialists at lower-level facilities that were an underused training resource within EMAS. III. An increased focus on newborns is needed. The project and the MOH rely too heavily on the assumption that better neonatal outcomes will follow improved maternal outcomes. The newborn death audit report revealed too many deaths in categories where one would expect deaths to be rare: full-term babies, normal birth-weight babies and newborns with high initial Apgar scores. 2 Studies are needed to elucidate the underlying causes of such deaths. Even this evaluation team lacks a newborn expert. IV. Community-level needs: The MOH’s Family Health Program monitors 12 indicators, including family planning (FP), place of delivery, basic immunization, exclusive breastfeeding, under-5 growth, tuberculosis (TB), hypertension, mental health, smoking, JKN coverage and clean-water latrines. Of note, family planning’s role in lowering both maternal and infant morbidity and mortality is long established. However, USAID’s Acting on the Call (2012) reports that only 45 percent of Indonesia’s target contraceptive prevalence rate was achieved. Opportunities exist to link with and exploit this program. One-third of JKN’s budget is earmarked for public health promotion and prevention initiatives; pokjas, Civic Forum Forum Masyarakat Madani (FMMs) and maternal and child health motivators (MKIAs) could advocate for use of these funds for maternal and newborn survival. Three macro-level lessons emerged from EMAS: I. The program relied almost exclusively on mentoring to improve QOC. However, mentoring is an approach, not a strategy. It serves as a complement to pre-service and formal training, job aids, external inspections, disciplining outliers, etc. By itself, it cannot solve issues such as specialist shortages, lack of blood banks, etc. Also, clinicians became mentors without the required deep understanding of the mentoring process, and the 2 Apgar scores summarize an infant’s health immediately after birth. A higher score indicates a healthier baby. FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 9 mentor’s facility was not required to meet high standards worthy of emulation. Lastly, mentors reported not always knowing how to achieve critical buy-in from leadership. II. Understanding the utility of audits: Establishing a culture of constructive criticism and holding each other accountable are behavior change and cultural issues not easily achieved. Audits have not yet proven effective in holding specialists accountable. As a companion to facility audits, WHO recommends the use of confidential inquiries 3 , whose results are then published and can be used to advocate publicly for increased accountability. III. The team heard repeatedly, and at all levels, about the critical nature of “strong district-level governance” in reducing maternal and newborn mortality. However, the definition and components of “good governance” were rarely articulated, except in a few of the districts visited. Identifying aspects of good governance and means to assess and monitor performance over time would be useful in elucidating key aspects of governance and to enhance accountability to the public. Pokjas and Civic Forums have the potential to play a crucial role in defining and monitoring governance. Conclusions: Until external constraints are addressed by the GOI and MOH, future projects will likely be limited in their ability to substantially lower maternal and newborn mortality in project areas. That said, a project with a comprehensive approach that includes referrals, upgrading providers’ knowledge/skills, facility compliance with SOPs, specialist accountability, audits, confidential iniquities, governance and community education and participation has potential to improve quality of care and outcomes. Recommendations Question 1: Improved outcomes and impact of project approaches In the short-term, USAID should: • Conduct a technical audit of all EMAS databases to reanalyze and present more robust findings for the MOH and future projects. • Continue to collect EMAS routine data to monitor results and sustainability. • Design an impact evaluation and collect baseline data alongside other studies as needed so that rigorous and conclusive data will be available to determine the effectiveness of USAID-supported MCH programs, and to provide the GOI with the information required to make evidence-based decisions to improve the quality of care. • Design a robust logical framework with key indicators for the future project. Question 2: Explanations and evidence of successful approaches Future project activities focused on emergency QOC activities should: • Utilize mentors from high-performance facilities and focus mentoring activities on death and near-miss audits, as well as introduce confidential inquiries. • Review and revise national SOPs and other MOH tools. • Expand approaches to include job aids, staff exchanges and consulting call centers. Future project activities in referral systems should: • Identify improvements and means to adapt and integrate the existing SijariEMAS system into larger MOH and emergency systems. 3 WHO monitoring quality of care, 2009 FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 10 • Adapt the IT system to measure timelines and delays of referrals at varying levels. Future activities with pokjas and Civic Forums should: • Ensure political and financial support of both entities through district-level MOUs and project funding for Civic Forums as needed. • Task pokjas with supporting Civic Forums to help identify MN issues they can address at the community level and in providing accurate information on QOC facilities. • Task pokjas with advocating for and monitoring WHO-recommended coverage of BEmONC and CEmONC facilities. Question 3: M&E, use of evidence and reporting Future USAID program design and M&E systems and reporting should: • Focus district selection on those where EMAS worked and/or replication districts with proven strong governance • Ensure district-wide coverage of all approaches, including private hospitals and private midwife clinics and practices. • Measure and correlate the results of activities at output and outcome levels, and at the facility and DHO-levels. • Report on all causes of death and monitor external constraints to MN outcomes. • Conduct an annual stakeholder’s meeting to present and review results, successes and challenges of the project, as well as external factors that assist or constrain improved outcomes. Key stakeholders include but are not limited to: USAID and its future MCH project (or portfolio projects), key GOI counterparts e.g. the MOH, professional associations such as Indonesian Society of Obstetrics and Gynecology (POGI), Indonesian Pediatric Society (IDAI), Indonesia Midwives Association (IBI), JKN, the private hospital association, and other key donors and MCH programs. Question 4: Lessons learned and future opportunities The future project should take a more comprehensive QOC approach, including contracting a newborn expert to identify key constraints to newborn outcomes, as well as to identify key indicators and high-impact interventions to address them. Assist the MOH in operational research for future systems and health financing, such as: • Accreditation: Establish criteria to improve MN outcomes and monitor the percentage of BEmONC and CEmONC facilities that maintain WHO standards. • Performance-Based Incentive System: Pilot a system to reward improved quality of care. • JKN Policies: Assess adverse effects on equity and maternal newborn outcomes, e.g. capped payments for newborn complications and to puskesmas. • Licensing and Compliance of Midwives: Monitor compliance of private midwives and help the Indonesian Midwives Association (IBI) strengthen criteria and stringency of licensing. • Task Shifting: Use WHO approaches to assess gaps in OB/GYNs and advocate for a pilot program in task-shifting, tracking the outcomes related to availability of OB/GYNs; • Cost-Effectiveness & Cost-Benefit Studies: Provide evidence of cost savings of improved MN outcomes to ensure ongoing political and financial support of the GOI. FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 11 Lessons Learned The key role of data and evidence: One of the most critical findings and lessons learned from this evaluation and noted throughout the report, were the constraints and limitations in data quality and the project’s PMP and indicator set, which posed challenges both to the program and stakeholders during implementation, and to this team in assessing successful approaches. Future projects with a baseline study conducted at inception would be better positioned to use quantitative evidence from this and other relevant sources to develop a clear theory of change and program framework; and a robust PMP and set of indicators, as well as routine monitoring and evaluation plans then designed to assess both the program’s progress and assumptions. All project innovations and approaches should generate evidence sufficient to assess whether and which interventions should be adjusted or abandoned versus those to be sustained and scaled. Annual portfolio and project reviews with key stakeholders would also improve communications and ensure coordination of efforts – both internal to the project and factors external to the project mandate and influence. Achieving impact at scale: Where the aim is to achieve impact at scale, an initial plan and clear pathway for achieving and measuring impact at scale, as well as a learning agenda to document and guide decision-making, need to be defined during the project design phase and progress towards this with adjustments as needed also reviewed annually. Key findings and lessons from EMAS illustrate that if these are not required at the time of project formation, the ability of the project to understand and respond to these needs later will be limited or absent. Replication, scale-up and sustainability: EMAS routine data collection and reporting did not include analysis or correlation with the data collected by partners working in the field – in mentoring, referrals or development of local governance entities; without this, the evidence available is insufficient to analyze and quantify the relative achievements of each approach and whether and to what extent the various approaches are effective in improving MOH systems, quality of care (QOC) and maternal newborn outcomes. An alternative plan for data presentation to the Ministry of Health and stakeholders, for example, one that includes trend￾analysis of routine monitoring indicators, would provide a meaningful basis for future political and financial commitment to replicate and sustain project activities, as well as an adequate basis for system-wide reform developed by the MOH and other GOI institutions. Design considerations: While USAID-supported activities cannot and should not seek to address all factors contributing to mortality, a periodic re-assessment of the selected strategy and initial assumptions about key leverage points to be addressed to achieve mortality reduction would strengthen project and MOH efforts and outcomes. Future projects should have the flexibility to adapt to new information built in to the design, rather than continuing with the initial approaches and interventions based on initial assumptions, as was the case of EMAS. EMAS demonstrates that if this is not built into a program from design and selection, it is unlikely that the project will develop the ability to learn and adapt later. Assessment of determinants: In the future, an assessment of determinants to check assumptions should be included from the start – in project design, development of a framework, approaches and interventions. These should be monitored as part of the PMP and M&E plan, including activities of key stakeholders (such as the GOI, MOH and professional organizations) as periodic checks to inform programmatic adjustments to project priorities and activities. The death reviews are an excellent example of information that, if available at project start up, could have significantly influenced initial project priorities as well as relationships and interactions with other stakeholders better positioned to address some key determinants. As above, USAID FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 12 projects cannot address all determinants, but should operate effectively with all stakeholders to make unique, important contributions to the broad spectrum of activities being implemented by, and with partners. Ultimately, in doing so, future programs are more likely to successfully improve outcomes and impact in project areas (both technical and geographical) and the MOH and GOI would be better positioned to respond to needs outside the project’s mandate. FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 13 I. EVALUATION PURPOSE & EVALUATION QUESTIONS Evaluation Purpose The purpose of this Expanding Maternal and Neonatal Survival (EMAS) program final performance evaluation, as stated in Annex I: Evaluation Statement of Work, is to learn from program successes and challenges. This evaluation was designed to build on the work in the midterm evaluation and identify key factors and interventions that contributed to both the successes and shortcomings of EMAS, as well as to identify and document opportunities for further improvement so as to inform future maternal and child health (MCH) interventions in Indonesia, especially given the context of universal health care (UHC), persistently high maternal and newborn mortality rates, and continuing devolution of health management and service delivery authorities. Overall, this evaluation focused on: • Improving quality of MCH services in facilities. • Improving the effectiveness of emergency referrals. • Governance practices, including policy relationships with the district, provincial and central level counterparts in the Ministry of Health (MOH) and other agencies. • Scale and sustainability. The target audiences include: 1. GOI stakeholders at the central, provincial and district levels. 2. Other local stakeholders, such as the OB/GYN association POGI, the pediatric association IDAI, private sector partners, CSOs, etc. 3. USAID and other development agencies in Indonesia. All evaluation materials will be posted to USAID’s Development Exchange Clearinghouse (DEC). Evaluation Questions The following key questions guided this evaluation, as outlined in the scope of work (Annex I): 1. Did EMAS primary approaches as implemented lead to improved outcomes, and did they have the desired magnitude of impact? 2. What are the explanations for why these approaches worked or did not work, and what evidence supports these explanations (e.g., facilitation, limitation, context, continuum of impact)? 3. How did EMAS learn and use evidence to make adjustments in program approaches during implementation? How can USAID internal management processes better support its programs and the MOH? 4. What has been learned that can inform future priorities, approaches, and activity processes? FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 14 II. PROJECT BACKGROUND Maternal Newborn Health in Indonesia The Republic of Indonesia, the fourth most populated country in the world, comprises more than 17,500 islands. Despite a long history of social programs and a strong policy environment in support of maternal, newborn and child health, Indonesia’s health indicators remain poor, with reports of stagnant, and at times increased, mortality rates. The most recent Demographic and Health Survey (DHS) in 2012 reports a maternal mortality ratio of 359 per 100,000 live births, and a newborn mortality rate of 19 deaths per 1,000 live births.4 A skilled provider assists the majority of births (73 percent), yet a large percentage of maternal deaths occur in facilities. Women and newborns often receive substandard maternal and newborn emergency care, even in hospitals. Almost 70 percent of all maternal deaths and 75 percent of newborn deaths occur in Java and Sumatra alone, most from preventable causes. Sixty percent of newborn deaths occur in the first month of life. To improve health in the country, the Government of Indonesia (GOI) has taken steps that include development of a national strategy to address maternal mortality, development of a national child health policy and national action plan to improve newborn health, and devolution of authority to local governments. The EMAS Program The Expanding Maternal and Neonatal Survival (EMAS) program was USAID/Indonesia’s flagship MCH endeavor, designed to support the GOI in its efforts to reduce maternal and newborn mortality by focusing on three major objectives: 1. Improved quality of emergency obstetric and neonatal care (EmONC) services in hospitals and community health centers; 2. Increased efficiency and effectiveness of referral systems between community health centers and hospitals; and 3. Strengthened accountability among government, community and health systems. The EMAS Consortium consists of five partners: Jhpiego (prime), Budi Kemuliaan, Muhammadiyah, Save the Children, and Research Triangle Institute (RTI). Over its five years, EMAS worked with GOI agencies (national, provincial and district), civil society organizations (CSOs), 123 public and private hospitals and 287 community health centers in 30 districts and cities in six provinces. The EMAS Results Framework follows as Figure 1. 4 Indonesia DHS 2012. FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 15 FIGURE 1: EMAS RESULTS FRAMEWORK FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 16 III. EVALUATION METHODS & LIMITATIONS The evaluation was conducted between January 10, 2017, and April 20, 2017. It covered the period from the EMAS midterm evaluation through the end of the program in March 2017. 5 The evaluation team included five consultants: Jennifer Peters, team leader; Dr. Meiwita Budiharsana, health systems and governance specialist; Dr. Kelly O’Hanley, MCH specialist; Daniel Sahanggamu, evaluation specialist; and Rosa Devitha Ayu, evaluation specialist. Dr. Detty Siti Nurdiati joined in initial team meetings and fieldwork before moving to the USAID MCH portfolio review team. Retno Sri Handini, USAID/Indonesia Monitoring and Evaluation Support Program (MESP) specialist, provided field visit oversight and overall administrative support of the evaluation. Nuri Kusuma Wardani, MESP research assistant, helped with all logistics and administrative issues and coordination with Jhpiego implementing partners and USAID. MESP Chief of Party Jonathan Simon provided technical support and direction for the evaluation team, as did MSI headquarters staff in Washington. A literature review began in January before the team commenced in-country activities. Team planning meetings began January 23, 2017, to develop the evaluation design and methodology and data collection tools (see Annex II: Evaluation Methods & Data Collection Tools). Fieldwork, including KIIs at the central level and in selected districts and facilities, took place over three weeks from January 30 through February 18. Preliminary findings of the evaluation were presented to USAID and EMAS partners on March 1, 2017; key findings were presented again to the EMAS partners and the Indonesian Ministry of Health (MOH) later in March. Site Selection Final district selection included 10 districts in five provinces: three in Central Java (Kudus, Pekalongan, Kota Semarang); two in West Java (Karawang, Bandung); two in South Sulawesi (Bulukumba, Gowa); two in North Sumatra (Deli Serdang, Serdang Bedagai); and one in Banten (Tangerang). Site selection was based on the criteria detailed in Table 1. TABLE 1: INCLUSION AND EXCLUSION CRITERIA Inclusion Criteria Exclusion Criteria MOH priority district, USAID selected EMAS endline district Mix of phases 1, 2, 3 & MOH replication Not part of future USAID program High and low performance Similar outcomes Strong and poor governance / commitment Similar geography, context High/low birth volume Sites selected for MCH portfolio team Urban versus rural Travel time/distance Key Informant Interviews In-depth interviews or group discussions were conducted with key informants at central and provincial levels. The team also conducted KIIs with provincial health organizations (PHOs), district health organizations (DHOs), bupati, pokjas, Civic Forums and professional associations, as well as with various cadres of health staff at vertical public, district and private hospitals, as well as at puskesmas. (See Annex III for a complete list of key informant interviews). Table 2 shows the total KIIs conducted at all levels. 5 Initially, EMAS was to close out at the end of 2016; later, it received an extension through March 2017. FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 17 TABLE 2: SUMMARY OF KIIs AT CENTRAL, PROVINCIAL & DISTRICT LEVELS Organization # of KIIs, # of respondents Central Level MOH 5 KIIs, 8 respondents USAID 6 KIIs, 5 respondents POGI 1 KII, 1 respondent EMAS Partners 10 KIIs: 15 respondents Jhpiego 5 KIIs: 10 respondents Save the Children 3 KIIs: 3 respondents Lembaga Kesehatan Budi Kemuliaan (LKBK) 2 KIIs: 2 respondents Research Triangle Institute (RTI) 1 KII: 1 respondent Total, Central Level 22 KIIs, 29 respondents Provincial & District Levels PHO, Team 21 4 KIIs,10 respondents DHO, Pokja, FMM, IBI, IDAI, POGI 28 KIIs, 51 respondents BAPPEDA 5 KIIs, 11 respondents Bupati 2 KIIs, 3 respondents District & Private Hospitals 16 KIIs, 74 respondents Puskesmas 14 KIIs, 37respondents Ex-EMAS District & Province Level 9 KIIs, 9 respondents Total, Provincial & District Levels 78 KIIs, 195 respondents Grand Total, All KIIs 100 KIIs, 224 respondents Key Documents and Data Sources The evaluation team conducted a comprehensive literature review of pertinent documents and data sources to respond to the evaluation objectives and questions: • Background documents: The team reviewed the original USAID request for proposals (RFP) and EMAS grant agreement, EMAS annual and quarterly reports, the performance monitoring plan (PMP) and indicator tables, project technical reports and briefs, project and external evaluation reports and assessments. The team also reviewed WHO and other international reports on best practices and GOI and MOH national guidelines, strategic documents, plans and policies. • Secondary data analysis: The evaluation team made extensive efforts to use any and all data that was available and pertinent to the evaluation questions. This included the routine databases received from the project, as well as data collected from facilities and districts during site visits. Data in MOH reports and the Indonesian Demography Health Survey (IDHS) were also reviewed. Data Analysis Several types of data analysis were conducted based on the evaluation design: • Document review: Content and trend analysis was performed on narrative reporting, PMP and indicator data, with comparisons between the program design, implementing partner (IP) agreements, project reports and results achieved against PMP targets. FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 18 • Qualitative data analysis: A framework for data analysis was designed prior to data collection and was refined at the end to identify key topics to be analyzed and reported for each evaluation question. The framework was also used to summarize interview notes and ensure that field interviews covered all pertinent points at the end of fieldwork. • Quantitative data analysis: Clinical and district-level data from the second half of the project was used to analyze EMAS progress toward indicators and compared with data reported, as well as with district- and facility-level data. Wherever possible, validation and triangulation of the qualitative and quantitative findings was completed. Limitations of the Evaluation Methodology The team encountered delays in the receipt of the routine monitoring data from the project, and there were issues regarding the quality and utility of the data received. Specifically: • Delays in receipt of routine monitoring data: Substantial delays occurred from the first request on January 13 through March 29, when the last set of data was received. This six-week process for receiving data limited the evaluation team’s ability to analyze data as required to respond to the objectives of this evaluation. • Constraints in data received: In addition to the limited annual data sets initially received, the team received 11 databases with 64 sets of routine data collected, many of which were not formatted or arranged consistently. As a result, the team had to recreate a master database and request more assistance from the M&E team in assuring accuracy and completion of quarterly data in a Statistical Package for the Social Services (SPSS) useable format. Data received for the SijariEMAS referral system was not sent directly from the IT source, but from the project’s internal system; it included only hospital data. • Quality issues of data received: Both annual and quarterly data included gaps, inconsistencies, duplication, and misreported data across different data sets and databases. For example, in addition to the initial annual data set received, another 11 databases with 64 sets of quarterly data were received in varying formats and arrangement, requiring recoding and reformatting by the team before analysis. Comparisons in data between the annual data set and quarterly data sets received revealed discrepancies such as no data for three districts on maternal newborn complications, deaths, and audits, although this data was included in the quarterly data sets received. Nine hospitals were listed in the wrong district and one facility was included in one district in quarterly data and a different district in the annual data. Three facilities were not included in one quarterly data set and thus could not be merged. Overall, gaps in quarterly data varied among facilities, due to variations in closeout time, graduation and phase, constraining the team’s ability to verify and analyze results from data received. • Issues of data analysis and results reporting: Results of some indicators were calculated incorrectly, e.g., including all pre-referral stabilization data from facilities district-wide in the calculation of percentage of EMAS referrals stabilized. As stated, a series of emails from Jhpiego included varying start and end dates for facility data collected based on the project phase. As a result, some facilities do not have any data for a given year; in some reporting periods, numbers are very small. In other districts, facilities were included across phases, and reporting within the district was split by phase, adding to the shifts in denominators and limiting the ability to assess achievements by phase, by facility or by district-level results. Service statistics were collected and reported from Year 3, whereas other data were collected and reported earlier. Five indicator data sets were noted as derived from data collected outside the project’s standard M&E, notably referral data, leaving the team without ability to verify and validate this from its source. For three FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 19 indicators, the definition was reduced and “terminology dropped,” though this is not noted clearly in the PMP or reports. As non-probability sampling methods were used for selecting districts, facilities, and participants, the evaluation team was unable to generate findings that statistically represented the larger population from which they were drawn. As key informants constituted the primary source of information, interview data was subject to personal biases, opinions, and recollection. The evaluation team mitigated interviewer bias by meeting frequently to compare transcripts and analyze data as a group. Ethical Considerations Verbal consent from interview participants was obtained per USAID guidelines. All interview data was kept confidential and used for evaluation purposes only. IV. FINDINGS & CONCLUSIONS Question 1: Did EMAS primary approaches as implemented lead to improved outcomes, and did they have the desired magnitude of impact? PMP and Indicators Of the project’s 22 indicators, three are outcome level measures, 7 are output/process level and the remaining 12 (more than half) are activity level. The project’s indicator mix is, on the one hand, too high-level (impact level, i.e., mortality), such that even if project activities do have some efficacy, the efficacy is not easy to detect due to that fact that so many confounding factors contribute to maternal newborn mortality at both the facility and district levels. On the other hand, the indicator mix is disproportionately low-level; it lacks measures of mid-level outcomes that might have more clearly demonstrated linkages between project activities and outcomes as well as provided a more robust evidence base to reassess and shift project approaches/activities, as needed. The team selected 10 key indicators and analyzed annual and quarterly results and trends against targets set for Years 3 to 5. Below is a summary of EMAS reported annual results against targets for the 10 indicators in Years 3 to 5, followed by key findings from the team’s reanalysis of quarterly data for selected indicators. EMAS Reported Results Against Targets Table 3 summarizes reported results against targets of the 10 key indicators in Years 3 to 5. The project reports four targets achieved for all three phases in all three years, including: • Percentage of severe pre-eclampsia and eclampsia (PE/E) cases managed with MgSO4); • Percentage of deliveries given at least one dose of uterotonic in third stage of labor; and • Two measures of death audits, one for stillbirths (>2000g) and one maternal near-miss conducted. Three targets were reported “almost achieved,” two were achieved in seven of eight (88 percent) reporting phases and one was achieved in five of eight (62.5 percent) phases: • Percentage of pre-term deliveries (24–36 weeks) with at least one dose of corticosteroids; and FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 20 • Two measures of death audits conducted, one for neonatal and one for maternal deaths. Three key indicators were reported as “not achieved,” with none of the three achieved in any phase or year. Notably, two of the three were the project’s only two measures of puskesmas￾midwife stabilization interventions. Results of both are significantly and consistently below targets for all years and phases. Results for pre-referral stabilization of eclampsia were, on average, 15 to 25 percent below target annually (highest target set: 75 percent in Year 5). Results for pre-referral antibiotics given in newborn suspected sepsis cases fared worse, on average, reported results were 16 to 28 percent below annual targets, against far more modest targets, with the highest target being 50 percent in Year 5. Critically, the third “not achieved” indicator in any phase or year was the program’s only outcome measure for maternal mortality, CFR for post-partum hemorrhage (PPH), PE/E and sepsis (“Category C” in the PMP). On average, the CFR reported (range: 1.4 to 2.7 percent) was almost double the target (1.2 percent to <0.1 percent) and relatively static across the three phases and years. A more detailed analysis of CFR follows; see Annex IV for further analysis of results against targets. FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 21 TABLE 3: RESULTS AGAINST TARGETS OF 10 KEY INDICATORS, YEARS 3 TO 5 PMP # Performance Indicator Year 3 (2 phases) Year 4 (3 phases) Year 5 (3 phases) # phases achieved % phases achieved Target Result Achieved Target Result Achieved Target Result Achieved 3 % severe PE/E cases managed with MgSO4 at EMAS facilities 90% 84-88% 2/2 95% 87-96% 3/3 100% 91-98% 3/3 8/8 100% 4 % deliveries receive 1+ uterotonic in 3rd stage labor at EMAS facilities 90% 95-97% 2/2 95% 96-100% 3/3 100% 100% 3/3 8/8 100% 6 % deliveries 24-36wks in EMAS hospitals receive 1+ dose corticosteroid 50% 65-70% 2/2 70% 59-80% 2/3 80% 76-89% 3/3 7/8 88% 7 % EMAS-facilities conduct regular reviews; fresh stillbirths >2000 grams 15% 42-78% 2/2 40% 36-70% 3/3 50% 58-75% 3/3 8/8 100% 8 % EMAS-facilities conduct regular reviews; neonatal death >2000 grams 15% 66-79% 2/2 40% 22-91% 2/3 50% 74-89% 3/3 7/8 88% 10 % EMAS hospitals conduct regular near miss reviews 15% 23-59% 2/2 50% 53-97% 3/3 60% 85-89% 3/3 8/8 100% 9 % EMAS facilities conduct 100% maternal death reviews w/in 24 hours 25% 36-59% 2/2 40% 26-86% 2/3 100% 69-97% 1/3 5/8 63% 11 Obstetric case fatality rate, Category C (hospital) 1.20% 2.0-2.7% 0/2 1.10% 1.8-2.4% 0/3 1.00% 1.4- 2.6% 0/3 0/8 0% 17 % severe PE/E cases receive 1+ MgSO4 pre￾referral to EMAS hospitals 40% 19-30% 0/2 65% 29-52% 0/3 75% 53-68% 1/3 1/8 13% 18 % newborn infections receive antibiotic pre￾referral to EMAS hospitals 30% 14-14% 0/2 40% 15-22% 0/3 50% 22-23% 0/3 0/8 0% FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 22 Team Analysis of Results Against Targets Team Analysis: CFR (Category C) The WHO-recommended target of <0.1 percent for obstetric CFR was the project’s Year 5 target for all phases. Results from the team’s analysis of quarterly data confirm that the annual reported CFR was not achieved. Unfortunately, phase-based implementation of activities did not allow annual comparative analysis, due to changes in total facilities covered in each year and phase of implementation.6 Though not reported this way by the program, the team calculated the proportion of hospitals that achieved a CFR <0.1 percent in Years 3 to 5. Figure 2 shows a summary of results in aggregate, and by phase. FIGURE 2. PROPORTION OF HOSPITALS WITH CFR < 0.1%, YEARS 3 TO 5 As a proxy measure of improvements at the outcome level in project facilities, the aggregate percentage of facilities with CFR < 0.1 percent increases from Year 3 to Year 5, which could indicate some contribution of program achievements. However, by phase, the results are mixed: Phase I shows steady improvements, whereas Phase II results fluctuate and in Phase III results decline. In Phase III, CFR (in its initial Year 4) is higher than in any other phase or year of the project, which would indicate that facilities were already providing fairly high QOC for maternal newborn complications. Comparisons in outcomes for only one year are of limited use. However, the highly variable results by phase, limited number of project activities in selected areas of maternal and newborn complications and numerous confounding factors that affect CFR rates in facilities are limitations to confidently correlating project inputs with improved CFR outcomes. Team Analysis: CFR (EMAS-Defined Category B) The WHO definition for case fatality identifies eight primary causes of death: hemorrhage, hypertensive diseases, abortion, sepsis/infections, obstructed labor, ectopic pregnancy, embolism and anesthesia-related death. In Indonesia, as in many countries, the most common causes of maternal death and those used to calculate CFR are: PE/E, PPH, sepsis and “other obstetric and non-obstetric causes.” However, EMAS included other causes of death when it calculated “Category B” CFR: antepartum hemorrhage (APH), ruptured membrane and moderate eclampsia. These are co-morbidities but not causes of death. The result of including these 6 Expansion to “total coverage” of referrals to include non-EMAS facilities affects data reported from EMAS-facilities only. 39.1 22.7 47.6 44.2 27.3 40.9 62.1 50.4 41.2 44.2 56.7 0 20 40 60 80 All phases Phase 1 Phase 2 Phase 3 Percentage (%) CFR for PPH, severe PE/E and Sepsis Year 3 Year 4 Year 5 FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 23 morbidities is a drastically reduced case fatality rate reported at EMAS facilities. This is both incorrect and misleading. These common non-fatal morbidities account for only a handful of deaths but greatly increase the denominator of CFRs. As result, CFR appears much lower. For example: in Year 5, Phase I, EMAS reports a total CFR of 0.52 percent and “Category B,” including non-WHO co-morbidities, at 0.67 percent. Per WHO definitions, total CFR was actually 0.77 percent and Category B CFR (PPH, PE/E and sepsis, without co-morbidities) was actually 1.76 percent. Redefining categories for cause of death is not in line with WHO and permits a false impression of progress in CFR. In response to the evaluation team bringing this to EMAS’s attention, EMAS claimed that the MOH collects data for all of these categories and that the program’s reporting is consistent with MOH. However, the team did not find evidence of incorrect MOH calculations for CFR. Regardless, incorrect calculations and reporting by EMAS are misleading, and where errors are identified in MOH systems, it is the responsibility of the program to assist in correcting this practice, rather than replicate or follow it. Team Analysis: Death Audits & SOP Compliance The team’s analysis of EMAS quarterly data identified results that contradicted those reported by the project. Two examples are: • Percentage of EMAS-supported facilities (hospitals and puskesmas) that conduct audits on all maternal deaths within 24 hours of occurrence. The analysis of this indicator used EMAS quarterly data from each facility – different and significantly lower results in seven of eight phases reported in Years 3 through 5. While the number of maternal audits improved, results of the team’s analysis found an average of 50 percent of Year 5 targets achieved in the three phases, versus 70 to 97 percent achieved across phases 1 through 3, as reported by the program. (See Annex IV for further analysis of maternal near-miss and neonatal death audits). • Percentage of EMAS-supported facilities that achieved 80 percent compliance with EmONC SOPs. Re-analysis of results using quarterly data identified some discrepancies with results reported. For example, in Year 5, the project reports that 100 percent of Phase 2 hospitals and puskesmas achieved 80 percent compliance with maternal, neonatal and infection prevention SOPs. Results from the team found that 93 to 95 percent of Phase 2 hospitals achieved compliance across the three SOPs and 92 to 97 percent of puskesmas achieved 80 percent compliance with MNH and infection prevention SOPs. Key Findings: Team Analysis of Indicator Mix At the outcome level, the project reported on maternal CFR, fresh stillbirth and early newborn mortality rate and newborn mortality rate. The project activities focused on three interventions targeting the two most common causes of maternal mortality, which account for a combined 55 percent of all deaths: PPH (20 percent) and PE/E (35 percent). No interventions focused on sepsis (5 percent) or “other obstetric and non-obstetric causes,” although this accounts for 40 percent of mortality.7 Similarly, the project reported on three newborn interventions: two for prevention of respiratory distress and sepsis, which combine to account for nearly 40 percent of newborn deaths.8 However, no interventions or indicators addressed stabilization or treatment of 7 Program Brief: Maternal Mortality Review of 112 Deaths (2015). 8 Program Brief: Neonatal Mortality Review of 76 Deaths (2016). FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 24 premature and low birth-weight babies, which account for more than 44 percent of newborn mortality. The project focused on selected maternal interventions in PPH and PE/E. The PMP includes one activity measure for prevention of PPH: “at least one dose of a uterotonic provided in [the] third stage [of] labor.” Two indicators measure activities to treat PE/E: pre-referral administration of MgSO4 at puskesmas and administration of MgSO4 at hospitals. However, EMAS did not calculate intermediate outcome indicators. For example, the rate of PPH is an ideal intermediate outcome measure, as it has a much larger denominator than CFR compared to PPH. Measuring the rate of PPH would have helped the program to readily determine adequacy and quality of care or lack thereof in administration of uterotonic to prevent PPH. If the rate of PPH had not decreased, this would alert the program to investigate dose/timing of administration of uterotonic, as well as to investigate other factors that contribute to PPH – e.g., technique for delivery of the placenta; extent of prolonged third stage of labor, etc. Taking it a step further, the project might have sought interventions (e.g., administration of additional uterotonic medications, use of balloon tamponade, etc.) to treat PPH to lower the case fatality rate from PPH. As with PPH, no intermediate outcome measure existed for PE/E – only provision of MgSO4 and case fatality from PE/E. Intermediate outcome measures (e.g., progression of pre-eclampsia to eclampsia; timeline from diagnosis of PE/E to initiation of MgSO4; maintenance of blood pressure in target range) might have helped identify solvable problems in clinical management. In fact, a maternal death audits’ report from the first half of 2015 concluded that the activities focused on PPH and PE/E were not sufficient to improve maternal outcomes. “By the end of Year 3, at least one dose of a uterotonic in the third stage of labor was provided in nearly all deliveries (>97 percent) and 94 percent of women with pre￾eclampsia/eclampsia were treated with MgSO4. … Despite high coverage of these lifesaving interventions, overall, virtually no changes have occurred in case fatality rates in these facilities.” — Maternal Mortality Review, 2015 For newborns, the project included two outcome-level measures of newborn mortality: newborn mortality rate and fresh stillbirths and very early neonatal death rate. Three indicators measured activities targeting newborns: administration of antenatal steroids during premature labor (to prevent respiratory distress syndrome in the newborn), pre-referral provision of antibiotics to prevent/treat sepsis and breastfeeding within one hour. As with maternal measures, there were no intermediate indicators – e.g., rate of development of respiratory distress syndrome or rate of development/adequacy of response to treatment for sepsis – that might have alerted the program to adequacy of interventions and possible needed changes in strategies. Team Analysis: Mid-level Outcome Measures Table 4 presents selected mid-level outcomes from the team’s re-analysis of data. Results indicate that in Years 3 through 5, facilities with higher percentages of uterotonic given to women in the third stage of labor had significantly reduced total caseloads for PPH in both hospitals and puskesmas. However, the correlation is low. Facilities with a higher percentage of corticosteroid given to women in premature labor (24 to 36 weeks) tend to demonstrate a lower neonatal asphyxia fatality rate. The correlation between women with preterm deliveries given corticosteroids and a reduction in total cases of birth asphyxia was identified, but did not FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 25 show a significant correlation; this is likely due to due to the broad definition of neonatal asphyxia. TABLE 4. CORRELATION OF MATERNAL NEWBORN INPUTS & OUTCOMES Variables Year 3 Year 4 Year 5 % women given uterotonic at third stage of labor vs. total PPH cases (hospitals & puskesmas) R = - 0.266 R = - 0.162 R = - 0.200 p<0.001, n= 259 p=0.002, n= 377 p<0.001, n= 388 % premature labor (24-36 wks) cases given corticosteroids vs. neonatal asphyxia fatality rate (hospitals) R = - 0.182 R = - 0.018 R = 0.056 p=0.153, n= 63 p=0.866, n= 93 p=0.535, n= 123 % premature labor (24-36 wks) cases given corticosteroids vs. total asphyxia cases (hospitals) R = 0.197 R = 0.180 R = 0.053 p=0.119, n= 64 p=0.082, n= 94 p=0.557, n= 125 Given the constraints in data, results of these correlations should be taken with caution; for each of the positive correlations presented above, the team found an equal or greater number of negative or inconclusive correlations. For example, results of the correlation between treatment of PE/E with MgSO4 in hospitals negatively correlates with CFR for PE/E in project areas. While there are many confounding factors that contribute to this outcome, without mid￾level measures, a project is unable to identify and investigate these nor to address them. Constraints in Baseline/Endline Studies Due to a number of constraints and limitations in the study design, site selection, timing of base and endline studies, sample size, and presentation of results by the project, the evaluation team did not attempt to validate or include reported findings in this evaluation. Constraints and limitations noted include: • Study design: Clinical observations conducted by trained midwives in hospitals in all districts for four weeks, except one in which only two weeks were conducted. Concerns regarding clinical observations include: o The act of observing reveals to the facility the items of interest being evaluated. o If the midwives were employed by the facility, this would be a confounder. o If indices used to measure QOC had not been used before, their validity and reliability are questionable. o Indices for a facility were summed and an average calculated, as noted in an EMAS PPP: “Index items include a mix of topics which are the direct or indirect focus of EMAS activities. The composition of a given index will influence the extent to which it can change over time.” • Site selection: Selection of districts was based on population density within the same province. The team cannot assess the extent to which comparison districts and facilities are well-matched, representative control sites. The entire premise of the study rests on this. • Sample size: The study design for the cohort included non-comparative sample sizes for referral data. Sample sizes differed for EMAS and non-EMAS sites and from base to end line – EMAS districts at baseline = 56; End line = 65; Comparison, at baseline = 43; End line = 16. • Timeline: Baseline conducted in Year 4 and no comparative data is available for Phases 1 and 2. For Phase 3, one year is an insufficient time frame to observe changes. The intervention in one district in Phase 2 started a year earlier than in the other districts. FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 26 • Access to Reports & Data Sets: No access to the baseline and end line data sets were provided to the evaluation team. The baseline report received was incomplete and the endline report was not yet available at the time of this evaluation. Conclusions: Although the project reports achieving the majority of targets for its 22 indicators, more than half of these are activity level measures and only three at outcome level. Only one of three outcome measures had targets set – maternal CFR, none of which were achieved in any year or phase. No targets for the two newborn mortality indicators were set; nonetheless no significant change in newborn mortality was reported. In sum, while EMAS achieved notable results across MCH processes, institutions, and practices (as detailed below), these results of the project’s activities did not lead to improved outcomes or the intended impact as defined and measured per the PMP. The project’s indicator mix was not sufficiently robust to measure and/or correlate the results of project activities and interventions with mid-level outcomes. Disaggregated data presented by the project and reanalysis and correlations calculated by this team suggest that the project’s approaches may have contributed to improved MNH outcomes in selected project areas. However, discrepancies between the results of this team’s analysis and those reported, alongside constraints and limitations in data quality received, limit the team from confidently drawing any conclusions regarding the results of EMAS approaches in improving outcomes or the level of impact. Due to a number of constraints and limitations in the study design, site selection, sample size, timing and presentation of results of the project’s base and endline studies, the team did not attempt to validate or include reported findings in this evaluation. Question 2. What are the explanations for why these approaches worked or did not work, and what evidence supports these explanations? Objective 1: Improved quality of emergency obstetric and neonatal care (EmONC) services in hospitals and puskesmas. EMAS’ first component used peer-based clinical mentoring to strengthen clinical governance as a means to improve the quality of maternal and newborn emergency care in Indonesian health facilities. EMAS felt that strengthening clinical governance would catalyze doctors, midwives, nurses and managers within the health facility to change the culture of clinical care and to make health workers more accountable for providing quality care. To do this, EMAS’ peer-based clinical mentoring focused on: i) improving the recording and use of service statistics to track trends and inform decisions in puskesmas and hospitals; ii) introducing the use of SOPs to assess and improve emergency clinical performance; iii) identifying emergency teams and performing emergency drills; iv) establishing and facilitating death and near miss reviews; and v) establishing the use of clinical dashboards to monitor performance. EMAS also enlisted POGI and IDAI to perform one-time external death audit studies. EMAS’ process of on-site, peer-to-peer clinical mentoring was intended to be more effective than prior MOH EmONC trainings. The peer-to-peer form of mentoring (OB/GYN to OB/GYN; midwife to midwife) was selected with the intention of creating a safe and collegial relationship for learning; establishing a cultural aspect of mentoring visits was thought to be key. The mentoring institution’s staff delegation and the host institution staff were expected to show mutual respect and friendliness, traits associated with conducive two-way learning. An unprecedented example of peer-to-peer clinical mentoring was undertaken at LKBK, a private hospital. EMAS aimed to create a break-through to the decades-long poor communication between high-performing hospitals and lower-performing hospitals, and between district FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 27 hospitals and puskesmas staff. The repetitive contacts in a mentoring cycle were intended to instill strong and accountable leadership to self-analyze performance. Clinical mentoring was implemented in three phases. Initially, LKBK was the only institution designated to mentor facilities in Phase 1 through May 2012. LKBK and Phase I facilities mentored new facilities in Phase 2 through September 2013. Phase 3 facilities were mentored by LKBK and Phase 1 & 2 facilities and the Muhammadiyah network of faith-based hospitals began mentoring within its network. In total, across all EMAS six provinces, 183 facilities – 50 hospitals and 133 puskesmas – had mentors. Facility selection criteria evolved by Phase 2 and included: large caseloads, commitment, and potential for influence beyond the facility, major referral hospitals, and cities where vertical, provincial or influential Muhammadiyah hospitals are located. Phase 3 districts, selected at the same time as Phase 2 districts, did not commence activities until one year later. EMAS reports that this gap lead to better preparation in the facilities because it allowed them to allocate budget line items early to fund EMAS-related activities. The readiness criteria for a health facility to begin mentoring other facilities included: whether they had achieved an average of ≥ 80 percent performance standards, if referral system components are in place, facility leadership/DHO is supportive of mentoring, the availability of staff to mentor, and the availability of resources and local government support for mentoring team members to travel. One standard mentoring cycle consisted of two visits by mentee facilities to the mentor hospital to observe effective clinical governance practices, and four on￾site mentoring visits to help with self- assessments, emergency drill simulations, development of dashboard indicators, facilitated near-miss audits and case reviews, and action plan development. As needed, follow up visits by smaller mentoring teams or EMAS staff in between mentoring visits were made to monitor progress on action plans. Impacting quality within so many facilities (including large, complex hospitals) required significant human resources, management, and logistics from LKBK. Of the 93 puskesmas mentored in Phase 1, not all became mentors. At the end of Phase 1, 6 of 22 hospitals did not become mentoring facilities because of limited human resources, especially specialists. The puskesmas with higher caseloads were more able to meet criteria to become mentors. EMAS reports that in some facilities, clinical performance or motivation dropped over time, as indicated by clinical performance scores lagging or declining. To address this, LKBK continued periodic mentoring until facilities achieved mentoring criteria or were determined unable to meet them. By Year 4, EMAS was working with 123 hospitals (both public and private) and 287 community health centers (puskesmas). At the end of Year 5, EMAS reported that all Phase 1 & 2 hospitals and puskesmas were using the standardized maternal registers; the majority of hospitals were routinely using dashboards and most puskesmas were using job aids. These results indicate a strong effort to improve clinical governance. Sustainability: In Year 5, EMAS reported that of the 30 EMAS districts, 20 had used district funding to expand EMAS interventions and approaches outside of EMAS facilities to over 70 hospitals and 500 puskesmas. The most common interventions replicated at facility levels were clinical mentoring and rotations, performance tools and emergency trolleys. Many DHOs have also replicated and expanded the use of the EMAS register book within and beyond EMAS districts. EMAS activities were discussed with the MOH Director General of Community Health, the MOH BUKR, KARS related stakeholders to explore the idea that EMAS approaches be incorporated into national accreditation tools. Provincial mentoring mechanisms have been finalized in 5 of 6 provinces; PHOs became more directly involved in coordinating/facilitating the mentoring process and preparing new mentors. POGI and IDAI ongoing involvement as mentors is pending final MOH approval after EMAS close-out. FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 28 Key Findings from KIIs: KIIs in all 10 districts and from varying cadres of health staff said that mentoring was very useful in improving adherence with SOPs by improving knowledge, confidence and skills. Mentoring was cited as extremely helpful in preparing for and achieving accreditation, most notably at puskesmas level (14 of 14, or 100 percent of key informant interviews). Other examples of accomplishments from mentoring include: improved communication between/among facilities (both between specialists and between nurses/midwives); increased numbers of deliveries managed at the puskesmas level and fewer referrals – attributed in part to improved knowledge/skills for management of complications; KIIs with mentors identified the emergence of champions who gained an increased sense of empowerment, and were thus more motivated to ownership and responsibility for QOC at both higher levels of administration and at other facilities. “Mentoring and self-assessment helped us to negotiate with upper management to obtain equipment and increased staffing and to reallocate hospital spaces.” – KII, District Hospital Constraints to mentoring were identified across all cadres of respondents in all districts. Key examples include: human resources and time constraints; too narrowly focused on emergency response and too limited in other aspects of QOC; the need to update newborn SOPs to equal the detailed level of those for maternal care; insufficient emphasis at the puskesmas level, where midwives lacked confidence to follow SOPs for fear of a bad outcome; and the resulting blame and repercussions in the community. Respondents in all 10 districts reported that the new and unified maternal registers introduced, and implementation of emergency drills, were the most useful tools from EMAS. All eight district hospitals praised the new registers and KIIs at hospitals and puskesmas cited decreased paperwork and increased efficiency in data entry as a result of the new registers. Similarly, the majority (over 90 percent) of respondents at hospitals and puskesmas in all 10 districts stated that emergency drills were extremely useful in increasing both skills and confidence to improve compliance with SOPs and that they continued to implement drills regularly and after EMAS ended. However, some noted the need for increased drills on newborn complications and to address a broader range of newborn stabilization and care. Dashboards were reported to be useful, however, in eight EMAS districts visited, the team found decreased and inconsistent updates to these tools, and only 25 percent (two of eight) district hospital were still using these after EMAS; respondents stated this was due to their low perceived utility. In all 10 districts visited, 80 percent of respondents reported conducting increasing numbers of death audits and doing so more regularly. The introduction of the near miss review by EMAS was viewed as an important new activity as it is perceived to be less threatening than death audits, where fear of blame is still a considerable constraint. As a result, more focus on near miss audits was deemed more likely to improve knowledge and use of findings to improve QOC and systems in the short-term and lead to better uptake of death audits in the long-term. While 80 percent of respondents reported increased numbers of audits conducted, very few (less than 30 percent) could provide examples of specific changes in systems or practices that resulted from these. More technical assistance is needed to overcome the constraint in conducting audits and/or in fully disclosing results for fear of blame. Respondents also noted the need to update newborn audit guidelines with detail comparable to those for maternal audits. FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 29 Objective 2: Increased efficiency and effectiveness of referral systems between community health centers and hospitals. EMAS’ second component was a set of inter-related interventions to increase the efficiency and effectiveness of the referral system. The referral system has been long identified as a critical constraint/weakness in reducing maternal and newborn mortality. EMAS’ approach was comprised of five inter-related interventions: 1) referral performance standards and tools to be monitored and improved by the DHOs; 2) referral network MOUs to strengthen the referral network, primarily between puskesmas and hospitals; 3) the automated SijariEMAS to improve communication and coordination of emergency referrals between puskesmas’ midwives and referral hospitals; 4) routine implementation of the 2010 National Maternal Perinatal Audits (MPAs) Guidelines to improve the quality of audits and secure needed budget at the district level; and 5) maternal and child health motivators (MKIA) to promote facility-based delivery, the uptake of social health insurance (Jamkesda) to reduce financial barriers to emergency health care, the benefit of JKN for maternal and newborn survival at the village level, and the early detection of high-risk cases. EMAS also used peer-to-peer mentoring to expand the referral system, staff from high performing facilities/entities (DHO, Facilitative Supervision teams, MPA teams, pokjas and Civic Forums) mentored their counterparts in other districts in how to implement EMAS referral interventions. These interventions were implemented in the following order: i) establishment of pokjas; ii) use of the referral performance standards tools to assess the critical elements of the referral system in order to identify areas for action; iii) development of an MOU before SijariEMAS was initiated; and iv) use of the MPA teams to identify weaknesses in the emergency referral system as a whole. Within four years, more than 1,430 facilities were linked into the SijariEMAS network. EMAS also helped to: a) establish procedures for incoming referral notification; b) establish availability of services and transport 24/7; c) create treatment protocols for referrals; and d) orient the MPAs’ and the puskesmas’ for village and private midwives for emergencies and referrals. EMAS and the MOH then compiled referral performance monitoring tools (one for hospitals and another for puskesmas) to quantitatively measure the effectiveness of the referral system, and to identify gaps; results were discussed in the pokja to develop an action plan. The Referral Operational Guidelines provided guidance on how to strengthen the identified gaps. Scores were compared over time to measure progress at both individual facility level and district level, with a score of 80 percent or more considered satisfactory. These tools have been well received by DHOs, perceived to be user-friendly, and outlined a clear pathway to improve referral system performance. In short, DHOs have been receptive to the process of strengthening the referral system. SijariEMAS, the computerized referral exchange system to improve communication between puskesmas and hospitals, was created to reduce delays in seeking care and referring patients and providing care. SijariEMAS automated the referral process by linking all types of health facilities in its network, leading to improved responsiveness to referrals within a district. The system was equipped with a detailed mapping of all public and private health facilities and an agreed set of referral pathways that were formalized in the MOUs. SijariEMAS collected four sets of key data: 1) referral case data (patient’s vital signs, diagnosis, and insurance scheme; 2) a summary of treatment provided to patients pre and post-referral; 3) communication between health facilities regarding the referral case, including treatment advice from the hospital to the puskesmas; and 4) referral case response time (less/more than 10 minutes). SijariEMAS also ensured that referrals were routed to a hospital in an efficient manner to prevent patients being taken to multiple hospitals before treatment was received. SijariEMAS encouraged feed-back about patient care and status to the facility where the patient would receive care post-discharge, although this was not measured. The most common interventions FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 30 replicated were SijariEMAS at the hospital and puskesmas levels and referral system MOUs in districts. The pokjas are comprised of key influential individuals and districts heads, such as the Bupati, the Development Planning Agency at Sub-National Level (BAPPEDA), DHO, and representatives from the private sector, professional organizations and civil society. The supporters were key to the successful implementation of referral system strengthening. The broad pokja membership helped enable them to resolve issues identified by communities, or supply-side barriers to service provision, including those outside of the direct control of health facilities. Pokjas were able to facilitate the drafting of local policies that support MNH, development of referral network MOUs, and integrated referral strengthening activities into districts’ annual plans and budgets. Other lessons were that all interventions (described above) had to work together to have the greatest effect on strengthening emergency referrals, the more facilities involved in the network the better, linking pokja and Civic Forums was critical. Further, developing strategies to utilize Government/MoH’s policies, structures and funding helped to promote sustainability. Sustainability: In 2016, EMAS reported a significant interest and commitment among DHOs and PHOs to strengthen the referral system and replicate SijariEMAS. As part of the business plan development, data were collected to assess which districts have already budgeted line items to support SijariEMAS in 2017. Additionally, many PHOs, DHOs, and hospital management teams were continuing to intensify their routine monitoring and analysis of SijariEMAS data using dashboards. EMAS finalized a SijariEMAS “business plan” that outlines the options for handing SijariEMAS over to a private enterprise to maintain and manage SijariEMAS. In Year 5, EMAS reported working together with the new head of the Center for Data and Information (MOH Indonesia – PUSDATIN) to build an integration model for SijariEMAS ensuring that opportunities to expand utilization of SijariEMAS are not lost. EMAS reported that its proposal to do a pilot integration of SijariEMAS with the 911 National Integrated Emergency Response System (Sistem Penanggulangan Gawat Darurat Terpadu or SPGDT) was already approved. Key Findings from KIIs: This was the first time a comprehensive referral system was developed for use in Indonesia. The majority (80 percent) of respondents in all districts – and at all levels from DHOs, pokjas and Civic Forums to hospitals and puskesmas – noted improved efficiency/effectiveness of the system. In two districts, respondents noted an improved balance of caseloads between puskesmas and hospitals (JKN also played a part in this) and also, its utility as the foundation for future/other emergency response systems. A key accomplishment of the new SijariEMAS system was the development of district-wide mapping and networking and coordination among facilities (8/8 EMAS Districts). The KIIs reiterated this point through the following points: 1. Improved networking and collaboration at all levels and across districts. Vastly improved one-to-one relations were achieved between hospitals and puskesmas/midwives as a result of having worked together (whether or not emergency/case ultimately referred). 2. More efficient routing of referrals. Referrals were routed faster, more efficiently and to correct hospitals without delay at puskesmas or delays from multiple referrals/refusals. Improved hospital readiness was a key achievement noted in almost 100 percent of KIIs. 3. Improved stabilization. Eighty percent of KIIs with midwives reported that although rates of stabilization were still low, SijariEMAS improved the confidence and ability of midwives to recognize/stabilize and refer complications. They also noted faster and improved communication flow between hospitals and puskesmas; necessary for managing complications. Midwives also noted a better flow of data from hospital to referring midwives/puskesmas for post-delivery follow up. FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 31 4. Improved communications and data for M&E. Overall, 100 percent of respondents noted a comprehensive and vastly improved system to track data, more robust data for use in death audits, better ability for DHOs to assess responsiveness of hospitals as well as effectiveness and challenges within referral system and increased data sharing of referrals en route. In all 10 districts visited, the most often cited constraint (90 percent of respondents from DHOs, hospitals and puskesmas) was the text-based design for midwives to initiate referrals due to: internet/network coverage issues, no automated response to confirm receipt of text and the challenge for midwives to enter data when simultaneously trying to manage patients during emergencies. Perceived benefits of SijariEMAS depended on a mix of urban/rural settings of cities/districts within an area. In two districts, respondents indicated that the “10 minutes response time” was the only timeline measured in urban areas. They also noted that for vertical hospitals, a 10-minute response time was too long and added a delay. Call centers placed at the DHO with the intention of improving governance was also flawed. In 80 percent of districts this was identified as an added delay and added substantial costs; Placing the call center at the primary referral hospital was perceived as more efficient, and 50 percent of districts had already moved this to the hospital.9 Long-term planning for management and sustainability of the IT system was noted by four DHOs: In addition, SijariEMAS is another vertical system and the MOH already has too many Health Management Information Systems (HMIS) to manage. Two districts also noted the need to revise the original MOU over medium- and long-term. KIIs also noted that the design didn't adequately address newborns, while one district reported a significant increase in newborn referrals, a significant proportion (40 percent) of KIIs stated that the system was designed primarily for maternal complications, not newborns. For example, the system prompts hospitals to send postpartum updates back to puskesmas and referring midwives, but does not for newborns. In four districts visited, the initial coverage of SijariEMAS did not include private facilities, private midwives and cross border referrals. This prevented the system from achieving full impact and the DHO from measuring and correlating outcomes. In three districts, the system was expanded to cover 100 percent of facilities district-wide and includes cross border referrals. In addition to replication and expanded coverage noted in many districts, 16 regional MOUs were developed to assist with cross-district referrals via SijariEMAS. Of importance, 100 percent of districts had committed funds to replicate/sustain the process in the future. Roughly 25 percent of districts visited maintained/budgeted for DHO call centers. The remainder moved to hospitals that were more cost-efficient. SijariEMAS was noted in three districts and one provincial hospital as the foundation for other/further development of emergency services. Selected districts have already begun integrating elements of sijariEMAS and 119 services. Objective 3: Strengthen accountability among government, the community and health system. Over the past decade, Indonesia has adopted a decentralized system that devolves significant governing authority from the central to district level. District governments now have real power to respond to the needs of the people. EMAS’ third component, Governance Accountability, was included with the recognition that maternal and newborn survival is not merely a function of the health system but also relies on both broad political support and community involvement. EMAS helped develop two governance mechanisms, pokja (working group) and Civic Forums, to address these components, respectively. 9 EMAS developed three levels for its referral system; the one with call center at the DHO is the most advanced and expensive option. FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 32 A pokja is an inter-sectoral group composed of individuals from government agencies, professional associations (POGI, IDAI, IDI, IBI), Red Cross, BPJS for health, CSOs (or the Civic Forums), as well as other public and private health providers. Pokjas were created to facilitate collaboration and mutual accountability among the various stakeholders to improve MN health services. Once a pokja is established in a district, it has the legal basis to both secure funding from the APBD and to advocate for, and initiate draft regulations to support maternal and neonatal death reduction in the district. The primary responsibilities of a pokja include: a) advocating for allocation of financial resources; b) advocating for policy changes; and c) collaborating with the Civic Forum to improve community engagement. Regular pokjas meetings are hosted by the DHO and its budget comes from the district via the DHO. A pokja can advocate directly to the District Head and the Local Working Unit who, together, have significant authority to plan, budget and implement programs tailored to meet local needs while conforming to national standard for services. The District Secretary and the District Head appointed members of the pokja. Subsequently, the District Head issued a decree that established the pokja as a legal entity. In all 30 districts, the pokjas have implemented the three main activities: (1) development of a referral protocol for MNH emergencies; (2) build a consensus to conduct and participate in the MPA’s in the health facilities, carry out performance assessments using Clinical Performance Standard Assessment modules, bring findings from clinical mentoring, performance assessment, and MPA results to the quarterly meeting; and (3) support implementation of facility service charters meant to inform the public about the types of services available, facility performance and fees. Evidence from EMAS reports suggest that these approaches would have worked if the district had: 1) political commitment to encourage action on both health sector and non-health sectors; 2) strong political support in mobilizing engagement of the stakeholders; 3) strong leadership by senior DHO staff; 4) clarity of roles and responsibility among pokja members to achieve agreed￾upon results; and 5) continued effective communication within the pokja and between pokja and the Civic Forum. EMAS helped establish the Civic Forum to channel civil society feedback to the health care providers of MNH services. The strategy faced considerable challenges given the low level of knowledge in communities about specific causes and conditions contributing to high rates of maternal and newborn deaths, and because in most communities, there were no existing CSOs targeting MN issues. The Civic Forum was expected to fulfill three roles: i) channeling MNH related feedback to local government, district parliament, health facilities and health care providers, especially community concerns regarding access and quality of emergency maternal/neonatal care; ii) partnering with the private sector to benefit from corporate social responsibility (CSR) and encourage involvement in MNH; and, iii) promoting MNH service information within the community. EMAS’ support was targeted primarily at the district level, although in practice, Civic Forum members expanded their networks to include volunteers at sub-district and village levels. These larger networks enabled the Civic Forum to collect information of pregnant women’s perceived needs at the grass-root or village level. At the village level, the Civic Forum helped establish the MKIA, or volunteers, to gather information on problematic maternal and newborn cases, issues, and barriers identified as having negative impact on maternal and newborn survival. The Civic Forum also employed the MKIA volunteers to encourage pregnant women to use MNH services. By Year 5, EMAS successfully created Civic Forums in each of the 30 intervention districts. Although the project did not report on collaboration with MKIAs, each district Civic Forum has at least one MKIA at the village level. Thus, indirectly, the Civic Forums helped promote and increase access to hospitals and puskesmas. (It is unknown how to separate this claim from the impact of the newly launched National Health Insurance.) FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 33 Challenges: 1. At the beginning, EMAS encountered difficulties in identifying CSOs interested or engaged in MNH issues, and thus recruited members from all CSOs available in a district. After EMAS ended, many of these CSOs left the Civic Forum because MNH was not their main focus. 2. EMAS’ partner, Muhammadiyah, contributed significantly to the development and advancement of the Civic Forums and when EMAS support ended, the existing Civic Forums had to rely heavily on its support. As a tradeoff, Muhammadiyah appointed a new position, the Coordinator of the Civic Forum, to its cadre of staff. This is a problem, given that the Civic Forum should be an independent entity that answers to the public, not to Muhammadiyah. 3. Fact-findings revealed that Civic Forums did not achieve financial independence, because none of the Civic Forums were able to secure long-term funding from local private companies. EMAS reports that 50 percent of Civic Forums did not take any actions to leverage private sector funds. 4. Civic Forum members have limited knowledge about maternal and newborn health and the local CSOs failed to expand their knowledge of the most common causes of maternal and neonatal deaths (e.g. PE/E, PPH, sepsis). This lack of knowledge was a challenge in communicating with other pokja members and in holding the MNH system accountable to the public. 5. External factors, such as the high turnovers of heads of district, DHOs and MCH section heads, also undermined Civic Forums’ abilities to partner with and demand support from the pokjas. EMAS defined sustainability of pokjas’ as having mentor pokjas’ providing technical assistance to pokjas in new districts and the emergence of a champion to motivate and strengthen commitment within the existing pokjas. All 30 pokjas with a legal basis to function and secure funding will have the ability, at least in principle, to facilitate change. Most of the Civic Forums have been successful in advocating to local parliament and district officials, and in demanding the issuance of District Head Decrees to support key district-level regulations about MNH priorities (e.g., 100 percent decreeing that all deliveries must take place in a facility), and in promoting a culture of power-sharing and equal voice among CSOs participating in the Civic Forum. Key Findings from KIIs: In all eight (100 percent) EMAS districts visited, pokjas were effective in securing financial resources for 2017 and 2018, advocating for local regulations (e.g. the Bupati’s decree in support of facility-based births), and in five (5/8-62.5 percent) improved collaboration with Civic Forums over the years. All eight had referral protocols in place and had agreed/committed to the death audit process. Yet, the team did not see evidence of participation in audit processes or evidence that this or other EMAS approaches were used at pokja meetings. Between Year 4 and 5, pokjas reported decreased membership and attendance and longer intervals between meetings, after EMAS. All pokjas requested ongoing technical assistance and guidance, indicating institutionalization of pokjas in this area is not yet sufficiently strong. Frequent PHO and DHO re-organization, new staffing and the re-election of districts heads negatively affected long-term sustainability. More than half (5/8, 62.5 percent) of Civic Forums helped improve linkages with civil society, and had established feedback mechanisms with health facilities. Notably all 8 succeeded in mobilizing support for the new referral system, and had collaborated well with MKIAs to promote facility-based births and other MN services. Civic forums were less effective in dialoguing with private health care facilities. Although all eight conducted regular meetings with FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 34 pokjas and had advocated for increased MN budgets with district and parliament officials. Still, all Civic Forums reported lack of ability to engage in meaningful communications with pokjas or to provide meaningful feedback to facilities due to lack of training on MNH issues. No Civic Forums were successful in leveraging resources, none had established long-term private sector partnerships and only three (40 percent) had short-term agreements with local media and NGOs for specific events. None had tried to communicate or collaborate with the managing body of JKN. Civic Forums face considerable challenges in replication and sustainability because they are not legal entities and can only become a legal entity three years after their establishment. Also, they can only obtain district funding for a maximum of one year. Civic Forums are perceived by many district authorities as a potential threat because they offer “too much” autonomy to the pubic; thus, it was the least replicated component within EMAS. Conclusions: EMAS reported results, replication of approaches within EMAS districts and beyond, and key findings from KIIs provide substantial evidence of successes across various EMAS approaches in all three objectives. Accomplishments reported by KIIs include: • Objective 1: As established by Budi Kemuliaan, and later expanded to vertical and district hospitals as well as Muhammadiyah, has made significant progress in clinical governance through capacity building and skills development of key personnel at both hospital and puskesmas levels. KIIs report increased life-saving technical capacity of health workers resulted from mentoring, supervision, SOPs, clinical tools and emergency drills. Key accomplishments beyond those within the project’s mandate include: increased efficiency and ability to meet criteria for government accreditation of hospitals and puskesmas and improved communications and knowledge sharing between specialists and midwives and between primary and secondary levels of care. • Objective 2: Vastly improved efficiency, access and quality of emergency referral services through EMAS efforts specifically in the design, introduction and expansion of the country’s first dedicated IT based maternal newborn referral system. • Objective 3: Increased district-level supportive policies for MN issues initiated by the pokjas and increased community engagement, community feedback and household-level awareness of MN issues and services through Civic Forums. But, there has been limited ability to communicate and advocate for funding with district officials. Future constraints and challenges were also noted across all interventions, though on balance, the examples and evidence of accomplishments provided by KIIs far outweighed the constraints. Replication of EMAS approaches and commitment of funds in 2017 and 2018 are further evidence of successes across the various approaches. Unfortunately, routine data collection and reporting did not include any analysis or correlation with the data collected by partners in mentoring, referrals or development of local governance entities. Without this, there is insufficient evidence available to analyze and quantify the relative achievements of each approach and to provide solid evidence as to whether and to what extent the various approaches are effective in improving MOH systems, QOC and maternal newborn outcomes. Without this evidence base, the long-term political and financial commitment to maintain and sustain these activities is significantly compromised, and unlikely to provide an adequate basis for system wide reform by the MOH and other GOI institutions. FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 35 QUESTION 3. How did EMAS learn and use evidence to make adjustments in program approaches during implementation? How can USAID internal management processes better support its programs and the MOH? USAID: RFP & Project Design The RFP and program design were ambitious, with targets of 25 percent reduction in national maternal and neonatal mortality. In contrast, the geographic coverage was limited to two districts in each of the six provinces for a total of 30/540 (roughly 5.5 percent) districts in 6/34 (17.5 percent) of provinces and targeted a total of 150 hospitals and 300 puskesmas across the 30 districts. This further restricted the project’s geographic coverage and scale. In addition, the technical scope for the project was narrowly defined, targeting emergency QOC issues at facility level only; therefore, and resulting from the midterm evaluation, USAID approved impact measure modifications to state: “contribute to reduced mortality” and not to hold the project directly accountable. The technical focus and scope of project interventions were designed to focus on QOC of emergency response at facility level, both district hospitals and puskesmas, but restricted from activities with private and community midwives below the puskesmas level. The RFP includes relatively more emphasis on management of complications at hospital level, compared to the isolated interventions for midwives at puskesmas to identify, stabilize and refer complications. At the start of EMAS, USAIDs focus in prior programs on improved midwifery training and skill combined with lacking government registration and total numbers of private midwifery practices, resulted in the exclusion of private midwives who perform an estimated 30-40 percent of all births. In hindsight, their exclusion from the project presented a considerable constraint in increasing early recognition, stabilization and referral of complications. “When we talk about reductions in MN mortality, a selection of interventions focused at only one level is not enough, it has to be comprehensive.” – KII, central MOH The RFP did identify the need to address a number of other key constraints, including: “Improved … regulatory environments which support the maternal clinical services, including licensing and accreditation practices;” “Support… the MOH in increasing the numbers of certified BEmONC and CEmONC facilities, that are delivering care according to standards;” and, “At puskesmas level: to prevent maternal complications through increased screening, active management of third stage, etc. leading to increased ability to [manage] complications and reduce their escalation.” Yet later, the RFP lists a number of areas in which the project was 0 500 1000 1500 North Sumatera Banten West Java Central Java East Java South Sulawesi EMAS Puskesmas Coverage Supported Not Supported FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 36 restricted from working. These include: gaps in distribution of human health resources, notably OB/GYNs, which is arguably the biggest challenge to improve MN outcomes, restrictions in working with private midwives and from addressing pre-service education. For decades, the Indonesian health system has ignored and/or discounted the existence and importance of private health care providers despite the estimated 30-40 percent of births performed by private midwives and multiple studies and sources that note the critical lack in skills and capacity to recognize, manage and refer complicated cases. The project was also restricted from revision of SOPs, which in hindsight was mistaken, as KIIs noted the need for this, particularly, SOPs for newborn complications. Although the RFP recognizes the need to increase the number of BEmONC/CEmONC facilities, it wrongly assumes that those already certified by the MOH can maintain and perform all WHO-defined signal functions. On the other hand, the RFP also states: “A fundamental part of EMAS is assessing in greater detail the problems within Indonesia’s hospital system that need to be addressed… to improve management of obstetric and neonatal complication. For every identified system failure that impacts maternal health… technical strategies will be identified to address these.” (RFP, pg 16) In sum, the goal and objective of the project design were conflicting and unclear. On one hand, the project was intended to target a discrete set of emergency QOC interventions in selected facilities; yet, simultaneously it was tasked with addressing every system failure identified and in five years, to reduce national mortality rates by 25 percent with the assumption that SOPs and facilities met WHO standards and requirements. USAID: KII findings Key findings noted USAID delays in execution of new project agreements and contracts, in project start-up and the anticipated long gap between EMAS and the follow-on project. All of these delays strain the continuity of activities identified in KIIs as needing ongoing technical assistance beyond that which the MOH can provide. Both USAID/Health and the MOH experienced changes in leadership from the start through the midterm of EMAS implementation. Last minute requests and re-alignment of project and USAID global indicator requirements were also a challenge. Ideally, this could be better clarified at the start of the project. Respondents also noted a need for better alignment and coordination between and among USAID’s portfolio of MCH activities; as well as the broader need for improved donor coordination between USAID, the UN and other agencies, to better serve the priorities and needs of the MOH and reduce duplication of efforts. One of the most critical issues noted in KIIs and the Management Review was the need to improve USAID oversight and management of its programs, particularly in M&E and in ensuring responsiveness to MOH needs. The Management Assessment states: “Some of the recommendations regarding EMAS require leadership for action to be taken by USAID. USAID in the Co-operative Agreement had reserved the right to take certain actions. At this time in the growth of EMAS, it is essential that USAID take the lead on actions that will assist EMAS in developing its foundational capacity to perform. Only USAID is positioned and has the capacity to do so. Neither Jhpiego nor EMAS leadership are positioned to take the lead on these actions.” While MOH respondents praised USAID for their coordination and engagement with the MOH during project design, respondents indicated the desire for more engagement during FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 37 implementation. Of note, repeated requests for improved reporting and information sharing, and presentation of data that is sufficiently detailed and in a format useful to them, never resulted in any change. One respondent stated, “Projects should share data that is detailed enough to assist the MOH to identify results and also challenges and constraints. We don't just want the positive project-related results. We also need to know if we want to replicate and sustain activities, what are the bottlenecks and challenges?” Another also identified the constraint to sustainability, noting, “For the considerable effort in implementation combined with the budget, was it worth it? [The MOH] recognizes some improvement, but it’s very hard to tease out clearly. This is a challenge to sustainability.” “M&E is an integrated component of the entire EMAS program, cutting across all strategic areas. M&E data will inform program management decisions by capturing program successes and challenges. M&E will be used to determine whether program activities are executed as planned, identify program constraints and barriers, assess the program’s success against expected results and draw lessons learned for continuous program improvements. Given program innovations within EMAS, a program learning strategy will be developed to inform and document impact of innovations on health outcomes.” - RFP Page 4 EMAS: PMP & Indicator Mix The project’s PMP was developed in 2011 and originally included 30 indicators. This was revised five times in the first two years of the project, resulting in a total of 22 indicators. In response to the midterm evaluation, the impact level target was revised to “contribute to,” and not be responsible for a 25 percent reduction in mortality. In addition, some indicators were dropped in recognition of their limited contribution while other indicator definitions were revised. No further strategic review of the project’s PMP and indicator mix or more in-depth analysis and revision of indicators was done by either USAID or EMAS after the midterm evaluation. As noted in key findings for Question I above, the project’s indicator mix was not sufficiently robust to measure mid-level outcomes linked to project activities, nor to use as an evidence base to shift approaches and activities as needed. Project Baseline/Endline Although a baseline was part of the original EMAS project design, baseline data collection was limited to facility level QOC data but had to include district wide data for the expanded coverage of the SijariEMAS referral system. Macro-level data collection to better measure outcome and impact level change over the program’s life was not done until Year 4 and only in response to the midterm. This was too late to provide any analysis of impact in Phase 1 and 2 districts, and allowed only one year of implementation before the endline study was done. Endline data was collected late in Year 5, and the results were not available to the evaluation team so no comparisons can be drawn on this. Overall, the project and its evaluation study design were restricted from being able to analyze the links between approaches and outcomes. This lack of correlation in EMAS processes and performance indicators and MN outcomes constrained EMAS from making use of data and shifting focus in any robust fashion over the five years of the program. (See Findings for Q1 and Annex V for a more detailed review of baseline and endline studies.) FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 38 EMAS: Response to Midterm & Management Recommendations In early 2014, a management assessment was conducted followed shortly by the EMAS midterm evaluation. Both processes identified several areas for improvement, including M&E, data analysis and reporting, and improved communications. The Management Assessment had 17 conclusions and 19 recommendations, of which seven are M&E and communications related. The midterm evaluation included 17 recommendations in four thematic areas, of which 7 are M&E related. The project did not address some recommendations adequately and only addressed selected aspects of others. Examples of management assessment recommendations insufficiently explained include; (i) the clarity of purpose and outcomes within EMAS and for its primary customers; (ii) improving relations at all levels; (iii) reporting relationships; and (iv) internal and external communications with stakeholders – especially access to information. “EMAS does not have a knowledge management system… for people in the field to easily access information… useful to them in supporting improved clinical, referral, and governance processes. Without [this] a significant valuable resource… in the field is missing. Access to the knowledge should be easy for all partners… using a platform that all can get into. If EMAS wants to be a learning organization, it must benefit from its knowledge in a timely way.” (Conclusion 15, Management Assessment Report) Examples of M&E recommendations and project follow-up include: • “Ensure mentoring by new mentors is “applied effectively.” No quality of mentoring measures was introduced; • “Increase efforts to link EMAS indicators and data with facility management.” No change in reporting or assistance to DHOs & facilities in use of EMAS data was made • “Apply a causal pathway analysis to critically examine the “moving parts” of each approach to identify those that support implementation and sustainability.” EMAS developed causal pathways in narrative form but did not develop a logframe nor apply any quantitative correlation analysis. EMAS M&E feedback (to the reviews and this evaluation) confirmed that the process to develop pathways was useful, but its utility ended there – there was no improved M&E analysis tested or used. • “Define and track connections between project inputs and…changes in intermediate outcomes and consequent reduction in mortality… in a broader district context.” EMAS responded to this recommendation by asking POGI and IDAI to conduct two death audit reviews, but as findings were only reported in Year 5, no change in project approach based on results was adopted, nor did EMAS assist in identifying and recommending future needs. Although, the midterm evaluation and management assessment both clearly outlined the need for deeper and better analysis and presentation of the linkages between activities and processes implemented/reported and any changes in morbidity and mortality, EMAS M&E continued to track and report data as before, with no major shifts in indicators, no further analysis or attempt to link approaches (processes) with performance and outcomes. “A major issue is that data available from EMAS and from health services where it is working do not allow connection of the observed and reported changes in processes with changes in health worker or system performance, nor with MN mortality” – KII, central MOH FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 39 EMAS: Use of Evidence from MN Death Reviews In 2015 and 2016, EMAS, through POGI and IDAI, conducted and published findings from the review of 112 maternal deaths and 76 neonatal deaths in selected facilities in project areas. The below summarizes selected key findings from both reviews: Neonates: 43% inadequate clinical monitoring and inadequacy of treatment, most common contributors to death; 68.5% after 2-7 days in hospital; 53% at term, 36% had Apgar scores 7-10 after 5 minutes; 51% referrals and 28% pre-referral stabilization contributed; and 70% of deaths deemed preventable. Mothers: 47% delay in care; poor patient monitoring and follow-up; 30% - after 48 hours in hospital; 53% - poor clinical decision-making; 40% - “Other” cause of death; 35% PE/E; 16% PPH; 38% - late referrals and only 9% - received any re-referral stabilization; and 72% of deaths deemed preventable. Both reviews identified the same critical constraint or gap in the QOC of patient monitoring and follow-up. Among neonates, 43 percent of deaths were linked to inadequacy of patient monitoring, and 68.5 percent died after spending two to seven days in the hospital. Similarly, 47 percent of maternal deaths were linked to poor patient monitoring and follow-up; nearly one-third (30 percent) died after 48 hours in the hospital. Notably, routine patient monitoring was not a focal area originally included in the RFP, nor was it an agreed upon part of EMAS’s approaches. And, as both reviews were conducted relatively late in the program (Years 4 and 5), this presents an added challenge to shifting the focus of mentoring and other interventions as needed. Nonetheless, the conclusions and recommendations for these reports do not highlight this issue as one of the priority needs for the future, nor do they suggest that the MOH, professional organizations or future USAID programming prioritize these elements moving forward. And while both reports conclude that at least 70 percent of deaths were deemed preventable, neither report includes a set of clearly defined, recommended next steps to address findings beyond stating that the professional organizations involved will continue to “to focus on examining cases of maternal death,” (Program Brief: Maternal Deaths, page 8) and to “determine what additional… inputs and innovations are required,” (Program Brief: Neonatal Deaths, page 9). “Learning lessons and acting on the results is the whole purpose of using [death audit review] approaches. There is no point in committing valuable resources to collecting information that just gathers dust on shelves. Action is the ultimate goal… There should be an objective method built into the system to monitor how recommendations are being implemented.” –WHO Beyond the Numbers Furthermore, in Year 5, EMAS reported that audit forms were shortened and simplified. The objective stated for this was to increase the number/volume of audits completed, but not necessarily used. The evaluation team did not have any evidence of similar focus on increased use of audit findings and as the team did not witness any audits finished, no assessment of the quality of these new audit forms can be made. Nonetheless, it is worth highlighting again that the FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 40 number of audits completed is not a meaningful or useful measure of QOC improvements, however, use of audit findings, and improvements in quality measures derived from audits, is. Attempts to simplify or streamline the process risk losing valuable data, input and time invested to properly learn and improve. The priority focus of conducting these reviews to reach targets rather than learn from and identify linkages was a missed opportunity to learn from findings to better inform EMAS activities and/or key stakeholders for future. Issues in EMAS Reporting EMAS reporting focused on results against targets set in its PMP required for USAID. There was little or no comprehensive reporting on non-EMAS related results, achievements or constraints identified by the program. There was limited reporting on reimbursement mechanisms, governance, and other factors outside of EMAS that affected maternal and newborn health which might have helped set a broader context, useful to the MOH and others. EMAS reported on selected disaggregated definitions for CFR, such as PPH and eclampsia, to showcase achievements without due diligence to all-cause mortality, potentially misleading the reader to believe CFR was dropping in project areas. Selected reporting leaves the MOH and other stakeholders without critical information needed if they are to address MN challenges beyond the scope of the project. EMAS narrative reports stated results as “achieved” instead of including the actual percent achieved versus the percent target set. For example, with a target of 60 percent of facilities to achieve 80 percent compliance with SOPs, the report simply states “achieved.” Without careful review this could easily be interpreted as 100 percent of facilities and/or 100 percent compliance.10 The EMAS team in charge of M&E and reporting did not collapse Phase 1 and 2 data at the end of Phase 2. Likewise, it did not collapse data from all 3 phases at the end of the project. Data was collected and reported by phase and as an aggregate total of selected districts and facilities. Presentation of data in this format made it impossible to discern changes in individual districts or facilities, or identify what improved and where more work was needed. Without further assistance from external data analysts to reanalyze and present this data, EMAS reports are of limited use for DHOs and facilities. In addition, EMAS engagement in districts changed over time, with some starting or ending earlier/later than others as the project expanded to new facilities and terminated work in others between reporting periods. EMAS reporting compared results per phase quarterly and annually, without adequately noting these changes or explaining the shifting denominators and length of intervention time. This resulted in further challenges in either identifying results within the progress, or for use by other key stakeholders. Reporting by phase was, ultimately, only meaningful to EMAS. KIIs with the MOH and USAID confirmed their frustration with this. A USAID-funded study identified the mixed interpretation in certain terminology used in EMAS reports such as “total coverage” and “full support” of the districts in which it worked, when in reality, the average facility coverage was 14 percent across the 30 districts, and in some, as little as 3 percent. The definition of “full support/total coverage” included “limited support” in some puskesmas, which again, is confusing and seemingly contradictory. In conclusion, “full support” does not necessarily mean “full coverage.” This interpretation causes confusion because none of the districts the team visited were able to replicate or cover all facilities district-wide. EMAS M&E changed denominators throughout its reporting, making comparisons in results across years inaccurate. For example, in Year 4, the denominator, or total number of Phase I facilities who conducted one maternal and two newborn death audits was 116 at baseline and in 10 EMAS Year 4 Annual Report, pg 34 FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 41 Year 2. In Years 3-4, the denominator for these three indicators drops significantly – ranging from 13 to 34 – and was also inconsistent across each of the three indicators in both years. Results reported for SijariEMAS were also unclear. In Year 5, the percent of referrals using SijariEMAS included in the denominator the total number of all referrals district-wide. However, in the calculation of the percent of SijariEMAS referrals with a 10-minute response time, both the numerator and denominator were higher than the reported total number of SijariEMAS￾based referrals in the district, with no explanation given. It is difficult, if not impossible, to report a higher number of SijariEMAS-based referrals answered in 10 minutes than the total number of SijariEMAS referrals. Ultimately, districts have a fixed number of facilities and therefore the denominator for referrals and all indicators should be fixed, regardless of expansion or drop of facilities over the life of project. Over the life of the project, EMAS’s central management and M&E unit did not openly share datasets with its implementing partners, EMAS staff in the provinces/districts, or with USAID the MOH or DHO. EMAS staff report receiving printed tables of selected PMP results against targets across reporting periods but lacked access to datasets to cross-validate these. KIIs indicate that MOH staff (at all levels), partners and field staff made repeated requests for better sharing and clearer presentation of data and reports. The midterm evaluation, management assessment, and this evaluation team all identified this important issue. “From the data [EMAS] collected, we cannot evaluate the impact derived from the various processes measured. This was discussed many times but never adequately addressed.” - KII, MOH Conclusions: In the RFP for the EMAS project, the goal and objective was contradictory and unclear. On one hand, the project was intended to target a discrete set of emergency QOC interventions in selected facilities and to measure and report on achievements of these for future replication and scale up; yet, simultaneously it was initially tasked with addressing every system failure identified, and in five years, to reduce national mortality rates by 25 percent. The inclusion of ambitious national impact level targets alongside too narrowly defined technical, geographical and facility coverage are confusing and also led the project to focus on monitoring interventions and outcomes at too high or too low levels, with limited flexibility to adapt or realign the objectives and activities as outlined. The most critical issue noted in KIIs and the Management review was the need for improved USAID oversight and management of its programs, particularly in M&E and to ensure responsiveness to MOH needs. While respondents praised USAID coordination and engagement with the MOH during project design, they also noted insufficient engagement during implementation; repeated requests for improved reporting and presentation of data that were sufficiently detailed and in a useful format, never resulted in any change. The project’s indicator mix was not sufficiently robust to measure mid-level outcomes linked to project activities, nor to use as an evidence base to shift approaches and activities as needed. Overall, M&E data as presented to the evaluation team could not be used as a tool for learning, evidence based decision-making, or strategic planning. EMAS placed too much emphasis on tracking and reporting of activities and too little investment in data analysis, sharing, use or reporting of data to identify mid-level morbidity outcomes, to link project inputs with outputs, or to receive feedback and assist the MOH and other stakeholders in identifying key constraints to maternal newborn health beyond those in the project’s scope. A broader mix of indicators and more robust analysis, presentation and reporting of data – such as that from the death audit reviews – would have aided the project to identify successes and constraints during implementation as well as those beyond its scope, for the MOH to address. FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 42 Issues with quality data collection11 and use have hampered the EMAS implementation. Too much focus on reporting requirements, coupled with too little or no attention to broader issues, resulted in two critical constraints: 1) in accurate or misguided information about progress; and 2) inability of other stakeholders to address issues beyond those in EMAS’s mandate. Ultimately, it was confusing and difficult for stakeholders – MOH most of all – to understand the failure to achieve impact/outcomes despite achieving project targets. QUESTION 1V. What has been learned that can inform future priorities, approaches, and activity processes? The team identified 11 key factors to inform future priorities, approaches and activities. Of these, four are “external” factors, or constraints to MN outcomes, which are beyond the scope or influence of USAID programs and pertain to GOI and MOH legislation and regulations. Four are “missed opportunities” or areas beyond the scope of EMAS as defined in the RFP, and three are lessons learned from interventions within the EMAS program. External Constraints Beyond USAID & Program Mandates BEmONC & CEmONC Certifications Various reports/studies12 have identified that many of the MOH-certified BEmONC puskesmas and CEmONC hospitals do not meet WHO standards for either certification. KIIs noted that CEmONC face challenges with blood banks, and restrictions on midwives’ performing various functions in BEmONC facilities. The team found that: none of the 7 certified puskesmas we visited could perform 2 of the 7 BEmONC signal functions: MVA and vacuum/forceps assisted deliveries. At BEmONC facilities, midwives cannot provide three of the seven signal functions: intravenous antibiotics (without a doctor present), forceps-/vacuum-assisted deliveries and sharp/vacuum aspiration curettage. In other countries, midwives, once trained, do provide these services. Many hospitals cannot consistently provide two of the CEmONC signal functions: complete blood banks for timely availability of blood for transfusion, and caesarian sections, due to the shortage and the limited timely availability of OB/GYNs and anesthesiologists (See below for more on HR Gaps/OB/GYNS). Per Monitoring Obstetric Emergency Care – A Handbook, given staff changes, and other changes that affect services, WHO also recommends re-assessing BEmONC/CEmONC facilities quarterly, to maintain a current picture of the extent to which each/all facilities are able to provide their signal functions. (See Annex VII, Sections A and C for more on these issues.) ObGyn Shortages & Accountability Current legislation allows specialists, including OB/GYNs, to work in as many as 3 facilities – public sector, private practice, and a private hospital. Specialists are contracted and allocated to district facilities from the central level, thus limiting the ability to oversee or sanction specialists who are late or skip shifts. Private hospitals also have limited authority over specialists due to specialist shortages. KIIs confirmed that QOC and maternal outcomes are threatened by the lack of available OB/GYNs, reluctance of hospital management to reprimand OB/GYNs for fear of losing them entirely, and by the limited DHO authority over specialists. KIIs said: “The single most critical thing needed is 24/7 coverage by OB/GYNs,” and “If the OB/GYN does not show 11 Limitations in the type of data – mostly binary – collected and low quality of data cleaning went undetected prior to this evaluation and the end of project. 12 “And Then She Died” World Bank et al, 2010; “USAID EMAS Midterm Evaluation” 2014; “Maternal and Neonatal Health Care System Strengthening in West Java Province” CFW, University of Indonesia, 2011. FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 43 up for work at a public hospital – nobody complains. Staff on labor and delivery learn to accommodate – they are helpless to do anything about lack of responsiveness from OB/GYNs.” They also stated, “Specialists protect their privilege… they have more control than the hospital director. They have their own kingdom. Private hospitals also cannot enforce their contracts with OB/GYNs.” Similarly, KIIs commented on lack of accountability of private midwives (who perform a significant percent of all deliveries), noting that although there are laws specifying staffing, equipment, medications, referral requirements, etc., that these laws are variably enforced. KIIs report that it is very rare for clinicians (midwifes or specialists) – even those who are outliers in terms of performance – to be placed on probation or be otherwise disciplined. Improved oversight and systems that both sanction and reward health staff are effective mechanisms used in many countries. Accreditation is a form of accountability, and KIIs note that this new system may help address some aspects of staffing accountability. pokjas and Civic Forums are a possible means of increasing accountability were they provided with necessary information – e.g., require all OB and newborn charts to record information such as time of arrival to facility; time first seen by specialist, all times seen by specialist, etc. and provide this information to pokjas and Civic Forums. Task shifting, endorsed by WHO, is not endorsed by POGI, and previous attempts to advocate for this strategy failed. At present task shifting is not a legal option. However, task shifting – including performance of cesarean sections by other cadres of heath staff - is endorsed by WHO and is a one of the only proven effective means of filling critical service gaps- in the short term until more/sufficient numbers of OB/GYNs are trained. See Annex VII, Section C for more on these issues. JKN Reimbursement JKN is helping bring coverage to many who previously lacked access to care. JKN is designed to expand access and equity while containing costs: However some of its reimbursement rules have unintended consequences that may increase costs, lower quality of care and/or limit access and equity. Illustrative examples identified in KIs include: • Over performance of C-Sections (C/S) at private facility to get more money from JKN. • Reimbursement for only four prenatal visits. This is not compliant with WHO’s recommended eight visits and one ultrasound. Many complications present late in pregnancy; and not screened/identified as result. More focus on the need for frequent ANC visits close to term are needed. • JKN does not pay or reimburse costs of referrals from midwives; only for referral costs by ambulance – this constrains midwives from making timely referrals. • Payments to Puskesmas is based on capitation, where puskesmas serves a large population, it may not receive enough money to provide the most basic services or even maintain its building. Rural populations are most disadvantaged, as challenges in ability/access to JKN registration too; this is an equity issue for rural populations. • Payments are capped – leading hospitals to refuse to receive complicated referrals, and for newborns. No distinction is made between complications, leaving many newborns at risk and/or hospitals to bear the costs. (See Annex VII for further description on possible unintended consequences.) To date, no study has been conducted, nor routine monitoring done, to assess the effect and impact of JKN as it continues to expand and increase its coverage. Without this, JKN, the MOH and all partners and FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 44 stakeholders are limited in their ability to work well within the system, or to assist in identifying rules that contain costs without jeopardizing QOC or inadvertently limiting access to services. Pre-Service Training and Re-Licensure of Midwives Midwife pre-service education requirements are weakest in the single most important area: hands￾on experience – most notably the performance of a large number of deliveries. Midwives can graduate having performed only a handful of deliveries. Such an education system not only fails the graduates themselves, but more importantly fails the women, babies, and families of Indonesia. No key informants stated that they would want their own female relatives delivered by a midwife with little hand-on experience. For re-licensure, only 12 deliveries over five years are required. In its KII with IBI, the Team noted that this requirement was not actually verified on re-licensure application. Missed Opportunities & Lessons Learned for the Future Missed Opportunities A. Comprehensive Approach beyond Emergency Response Maternal and Newborn Audit Review Results: As the maternal and newborn audit reports make heartbreakingly clear, many deaths are not due to inadequate emergency responses. Many deaths occur later during the hospital stay. In many instances, inadequate clinical monitoring contributed to the cause of death. KIIs also identified the need for a more comprehensive approach to address more than just emergency response issues. Selected findings from the maternal death review (2015) show that almost half (47 percent) of the cases are associated with delay in decision-making and care, and with poor patient monitoring and follow-up, and clinical decision-making was identified as inappropriate in over half (53 percent) of cases. In sum, the report concluded that 72 percent of maternal deaths reviewed were preventable. Notably, a reported 40 percent of maternal deaths were coded as “other,” the highest category contributing to maternal deaths (followed by 35 percent severe pre-eclampsia/eclampsia, 16 percent PPH and 5 percent sepsis). Deaths can be miscoded as “other” for any number of reasons. Yet knowing the cause of death – particularly in a category with such a disproportionately high and alarming contribution to maternal mortality rates, is a critical cornerstone to lowering maternal mortality and an important issue to monitor in any maternal survival program. The newborn death review reports poor/incorrect clinical decision-making and delays in initiating treatment contributed to 55 percent of cases. Patient monitoring and follow-up was cited as a problem in almost half (43 percent) of cases. Over two-thirds (68.5 percent) of newborn deaths occurred after two to seven days in the hospital. Seventy percent of cases were cited as preventable. The report identified the three leading causes of newborn death as asphyxia (33 percent), sepsis (22 percent) and respiratory distress syndrome (17 percent). The Quality and Access to Perinatal Hospital Care at the District Level Study (2010) assessed compliance with the Indonesia National Standards of Maternal and Neonatal Care. Its conclusions are similar. The study identified major gaps between national standards of obstetric and neonatal care and actual hospital practice. Among the shortfalls in compliance with maternal SOPs noted were: antepartum hemorrhage (44 percent), postpartum hemorrhage (51 percent), pre-eclampsia/eclampsia (56 percent), abortion (44 percent), monitoring during hospital stay (20 percent) and eclampsia (56 percent), with a median score for checking vital signs every hour and keeping blood pressure below 160/110 (0 percent). Compliance with newborn SOPs was worse: overall compliance with neonatal SOPs was 39.2 percent, compared with 59 percent FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 45 compliance with maternal SOPs. KIIs also expressed the need for more emphasis on all aspects of stabilization of newborns not just resuscitation. Currently, accreditation ensures that SOPs are in place, but it does not measure compliance with SOPs. Health systems in other counties have utilized various performance-based incentives to motivate staff. Performance-based incentives might be an ideal intervention to help staff comply with facility SOPs. If incentives are given to all staff in a department, they can hold each other accountable. “A number of maternal health interventions… implemented in recent years… were unsuccessful in reducing maternal deaths. While interventions are different in many aspects… these were similar in their inability to demonstrate significant impacts on maternal mortality. This situation can be interpreted as "no one size fits all", implying the need for a comprehensive strategy. The new strategy should be able to overcome the main barriers to expand [quality] services related to the scarcity of skilled health providers and… to sub-standard quality of care. Malaysia and Sri Lanka, two countries that have successfully reduced maternal mortality ratio, focused their strategies on [many] aspects... and changed strategies several times.” (Final Report, CFWUI, June 2011 pg 4). B. Increased Focus on Newborns Focus on newborns is lagging compared to the focus on mothers. Fewer newborn audits were performed compared to maternal. One key informant noted that the central MOH has over￾relied on the assumption that improved newborn outcomes would closely follow improved maternal outcomes. The newborn death audit report revealed many deaths within many categories where one would expect deaths to be rare; notably, 53 percent if cases were at-term babies, 67 percent of cases were normal birth-weight (>2,500 grams) babies and 36 percent of deaths occurred in newborns with high initial Apgar scores.13 One key informant, a specialist at a higher-level referral hospital with a neonatal intensive care, expressed the need for studies to elucidate the underlying causes of neonatal deaths as a basis for planning subsequent interventions. (The EMAS evaluation team did not include an expert in newborn morbidity/mortality). C. Upgrading Knowledge and Skills of Nurses, Midwives and Physicians Since the 1990s, USAID and MOH have been targeting interventions and programs for midwives to improve knowledge and skills. Consequently, USAID felt that upgrading midwives’ knowledge/skills should no longer be the primary focus – that a new facility level approach targeting selected midwives and focusing on emergency issues for nurses/midwives/specialists should be a primary strategy for EMAS. However, it is clear from the results of this evaluation that midwives still need considerable technical assistance. Midwives perform 30-40% of births, the majority of them in private hospitals and private midwife facilities. Key informants state that midwives are not currently trained in the use of balloon tamponade, a potentially life-saving technique that is supported by WHO. Inter-facility one-to-one exchanges for all cadres of clinicians are a missed opportunity. The large caseloads at the higher-level facilities are an untapped resource, as is the more highly trained/skilled staff in these institutions. The staff at lower level facilities reported they too seldom get the opportunity to spend time at a higher-level facility to acquire knowledge/skills – and the desire to do so more frequently, and for a longer time (1-2 weeks, versus 2-3 days) was 13 Apgar scores summarize an infant’s health immediately after birth. A higher score indicates a healthier baby. FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 46 expressed by key informants. Similarly, KIIs noted that staff from higher-level facilities also spend insufficient time at lower level facilities as teachers and mentors and requested more technical assistance from specialists at puskesmas as well as for private and community midwives. While EMAS supported clinical rotations ranked very high among KIIs and are reported as one of the most replicated interventions, a longer-term approach to inter-facility exchanges – swapping staff for several weeks and between low and high level facilities – intermittently, on an ongoing basis, would allow those spending time at the higher-level facilities to learn by emersion. Further, that those spending time at lower level facilities could serve as de facto mentors, with no staff shortages at either end. Results of SijariEMAS confirm that increased linkages and better communications between cadres/facilities not only improved referrals, but also improved overall knowledge sharing and communications between midwives and specialists in general, and as an extension of the improved communications and collaboration created through SijariEMAS. EMAS reported that establishing inter-facility relationships greatly encouraged puskesmas midwives to feel comfortable about making referrals and seeking advice for stabilizing. Still, too many midwives reported that when they do not know how to manage a patient, they primarily rely on referring the patient to a higher-level facility. Informants reported that although communications have improved between selected puskesmas and private midwives and referral hospitals, they still have too little opportunity for real-time consultation. The key informants stated that general practitioners (GPs), who are tasked with providing back-up, often know little more about MN health than the midwives do. Few midwives are trained to perform three of the BEmONC signals functions. In addition to conventional training, several strategies exist for addressing the need for improved knowledge/skills of nurses/midwives/specialists. For example, KIs stated that more job aids are desired. (See Annex VII Section D for more on job aids, including checklists.) The MOH-MCH Handbook needs updating to be compliant with WHO and international standards. (See Annex VII Section H for more on this.) D. Community Needs KIIs identified the need to include private/community levels, including FP - especially for younger women and to address critical issues of equity and access to FP. The role of family planning in lowering both maternal and infant morbidity/mortality is long established. However, family planning services are currently somewhat neglected in Indonesia. According to USAID’s Acting on the Call, as of 2012, Indonesia had achieved only 45 percent of the targeted contraceptive prevalence rate. KIIs identified other needed community level components such as early screening of pregnancies and improved nutrition in pregnancy. The Family Health Program Approach (PERMENKES No 39/2016) aims to improve public health through community health efforts and empowerment, supported with financial protection and equitable health services. It relies on the census of all households and includes 12 indicators: FP, place of delivery, basic immunization, exclusive BF, under-five growth, TB, hypertension, mental health, smoking, JKN, clean water latrines. Opportunities exist to link with and exploit this program. The PROKESGA budget comes from various sources: APBN, APBD, Dana DEKON, DAK (BOK). One-third (35 percent) of JKN’s budget is earmarked for community outreach initiatives under this approach, though this has not happened yet; pokjas, FMMS and MKIAs could advocate for use of these funds. FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 47 Lessons Learned A. Mentoring To improve QOC at the facility level, EMAS utilized mentoring to motivate and model the conducting of ongoing cycles of: self-assessment → action plan → execution of plan → reassessment. Mentees became mentors but within inadequate understanding of or training on the philosophy and fundamentals of mentoring. KIs reported, “Mentors (second generation) do not understand the difference between mentoring and training.” Also, subsequent generations of mentors were affiliated with facilities that could not, in such a short time, achieve a standard of excellence worthy of emulation comparable to the standards of the facilities to which the original mentors (BK) were affiliated. Mentoring is needed not for a set time period, but as long and until the results are achieved and sustained. Mentoring is an approach – not a strategy. It serves as a compliment to pre-service training, training, job aids, external inspection, inspections, disciplining outliers, etc. By itself, there are many issues it cannot solve – e.g., specialist shortages, lack of blood for transfusions, etc. KIIs reported “Mentoring cannot change things without the buy-in from leadership.” For mentoring to “get in the door” and then be sustained governance is crucial. Mentors reported not always being clear how to achieve such buy-in. There was no evidence of plans within or beyond EMAS to expand self-assessment beyond maternal/newborn emergency response. In principle, the mentees themselves could have expanded the self-assessment cycle focus beyond emergency response – but this was not part of the original model or design for mentoring and the team found no examples in the field of an expanded scope for mentoring to include a more comprehensive approach. B. Audits Key informants provided a number of insightful comments regarding constraints to death audits. Examples include: “There are no incentives to pull the cover off of underlying problems;” “Specialists are not taking leadership;” “Audit discussions often focus on cases that were referred in – so as not to focus on non-referred cases where it is more difficulty to shift the blame;” and “It is difficult to break the wall of silence.” KIIs also reported ongoing challenges with audits: • Newborn audit guidelines need more detail comparable to those for maternal audits; • Hospital-based review teams do not always disclose findings; lack of transparency can limit their impact on improving QOC; • No confidential inquiries are routinely performed; and • Indonesian culture can be averse to open criticisms. This is an area where external mentors are particularly important to help facility participants grasp the utility audits and to free themselves of fears and protectionism. (See Annex VII for more in-depth information on audits.) Understanding of the utility of audits, establishing a culture of open critique, holding each other accountable are mindsets and a culture not easily established. Though audits are important learning tools and a cornerstone in identifying key constraints to improving MN outcomes, given the many challenges identified by KIIs here, it is not logical to expect audits to have significant impact on QOC over the short term. Nonetheless, more assistance in this area, notably in the application and use of key findings and not just on the number/volume of audits conducted, should, over the medium to long-term, prove beneficial. WHO, in Beyond the Numbers, Reviewing Maternal Deaths and Complications to Make Pregnancy Safer, recommends the institution of the confidential inquiry, in addition to audits. Importantly, because the results of confidential inquiries are usually published and FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 48 available to a wide public, they can be used for advocacy and to increase accountability to the public. C. Governance The decentralization of power to the local district level in the last 10 years means that district governments are now crucial to maternal and newborn survival. The Team heard frequent references from KIIs about the critical nature of “strong district-level governance” – vis-a-vis reducing maternal and newborn mortality/morbidity. However, the components and aspects of “good governance” were never articulated. EMAS reports that pokjas and Civic Forums have the potential to advance MN health. However, they are a work in progress. A KI stated, “For these activities to be sustained, substantial advocacy and commitment at all levels would be required.” One hundred percent of KIIs with both entities reported the need for more technical assistance in the future. Moreover, some district health officials expressed a reluctance to continue with Civic Forums because they have the potential to make the district MOH more accountable to the public. Identifying aspects of good governance and ways to assess and monitor governing performance in an ongoing fashion would be useful for elucidating key aspects of governance and for enhancing accountability to the public. Conclusions: Until external constraints are addressed, any future project will likely be limited in its ability to substantially lower maternal and newborn morbidity/mortality within its coverage areas. That said, a project that focuses on referrals, upgrading providers’ knowledge/skills, facility compliance with SOPs, specialist accountability, audits, confidential iniquities, community education and participation, as well as governance does have the possibility to improve quality of care and outcomes. V. RECOMMENDATIONS Question 1: Improved outcomes & impacts of project approaches USAID should: • Conduct a technical audit of all EMAS databases to assist in analyzing more robust findings, including but not limited to, all databases from routine monitoring, mentoring, SijariEMAS, baseline, and endline. • Continue to collect EMAS routine data to monitor results and sustainability of project activities after EOP in some or all districts, including replication districts. • Prior to the start of the follow-on project: o Design an impact evaluation and collect baseline data so that rigorous and conclusive data will be available to determine the effectiveness of USAID-support MCH programs, and to provide the GOI with the information required to make evidence-based decisions to improve the quality of care. Special attention should be given to WHO recommendations and guidelines. o Conduct other studies as needed to assess QOC for PE/E and PPH: adequacy of prevention measures (PPH only), timeliness of diagnosis, timeliness/adequacy of stabilization and referrals, of initial management, and of postpartum care. o Design a robust logical framework from the evidence-base and identify key indicators that the project will be required to present MOH, districts, facilities and USAID. FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 49 Question 2: Explanations and evidence of successful approaches Future project activities focused on emergency QOC activities should: • Conduct a technical review all EMAS mentoring and monitoring tools and systems against international best practices (WHO and others) and adapt, expand, edit as needed. • Based on results of the technical audit of EMAS data, identify approaches and activities that should be continued. • Utilize mentors who come from facilities with high performance standards and who are fully oriented to the mentoring process. • Provide ongoing mentoring for death and near-miss audits. • Introduce Confidential Inquiries and make results public – per WHO recommendations. • Review national SOPs and other MOH tools (including the MCH handbook) and systems in line with WHO standards to expand technical focus of future projects beyond QOC in emergencies. • Consider both tried-and-true and novel approaches to support high quality clinical care, including: job aids, staff exchanges between higher and lower level facilities, and call centers that provide 24/7 consulting services. Future project activities focused on referral systems should: • Undertake a comparative study of the various emergency services being developed and implemented across districts, including SijariEMAS, 119 services, Wahidin referral system and others. • Identify improvements and means to adapt and integrate MN referral systems into larger MOH and emergency systems, including availability of ambulances. • Review and revise the IT system, data needs and collection, analysis and reporting according to district and central MOH needs. • Adapt the system to measure timelines and delays of referrals at varying levels. • Assess aspects of sustainability beyond just financing, e.g.; institutionalization, health impact, efficiencies, costs and adaptability to and emergence of new technologies and IT systems. To increase sustainability and impact of future pokjas and Civic Forums’ activities: • Include both pokja and FMM in project-district MOUs to ensure that they are supported both politically and - where feasible – financially. • USAID should fund the costs to establish Civic Forums, including: o A baseline needs assessment study to identify needs and focus of activities; o Development of an evidence-based, strategic design for key interventions, activities, and a monitoring and reporting plan (to the Bupati, DHO and project); and o Operating costs (e.g. transport, communications – village level, meeting costs). • Task Civic Forums with ongoing assessment of the strength of local governance vis-a-vis supporting MN health. • Task pokjas with supporting Civic Forums to help identify MN issues and improve their technical knowledge of QOC issues and ability to communicate these at community level. • Task pokjas with advocating for WHO-recommended coverage of BEmONC/CEmONC facilities (with all required signal functions maintained at each). FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 50 Question 3: M&E, use of evidence and reporting USAID should work with the MOH to: • Focus district selection on EMAS and/or replication districts only and not expand beyond these, to build on and not lose gains made in the previous five years. • Select districts with proven strong governance, accountability and dedication to MN issues, as demonstrated by the signing of MOUs that mandate key aspects of project approaches. • Include a broad range of stakeholders in the design of district-level activities, e.g.: DHO, Bupati, BAPPEDA, POGI, IDAI, IBI, private facilities and CSOs. Ensure adequate scale and coverage of interventions in all facilities district-wide, including private hospitals and private midwives, clinics and practices. • Design its M&E and reporting to respond to DHOs, PHOs and MOH needs first, with USAID reporting requirements also designed to support those of the MOH, and where possible, with limited USAID-specific indicators and other requirements. • Set a fixed number of output and mid-level outcome indicators that the project is required to monitor and report, to measure and correlate results of its activities, but allow flexibility at activity level to enable shifts in interventions as challenges and gaps are identified. • As possible, harmonize the selection of indicators, definitions, calculations and presentation of results with those most useful to the MOH, DHOs and facilities to present project data and results, accomplishments and challenges in a clear and consistent format. • Track and report on all causes of death; provide assistance to improve rigor used in assigning of cause of death. • For transparency, report on why results are/are not achieved, and on challenges/constraints without fear of criticism or the appearance of failure. • Document external and internal factors that limit the ability to achieve results and impact. • Document successful interventions that are scalable and replicable, with the aim of increased sustainability over long-term. • Conduct an annual stakeholder meeting to present and review results, successes and challenges of the project, as well as external factors that assist or constrain improved outcomes. Key stakeholders include but are not limited to: USAID and its future MCH project (or portfolio projects), key GOI counterparts e.g. the MOH, professional associations such as POGI, IDAI, IBI, JKN, the private hospital association, and other key donors and MCH programs. Question 4: Lessons learned & opportunities for future The future project should take a comprehensive approach to QOC in MN services: • Conduct a global literature review of global MN best practices which may be applicable to Indonesia, including: o Examples of success from Vietnam and Malaysia in lowering maternal mortality; o Examples of successful task-shifting, e.g., to utilize non-OB/GYN cadres to perform C/S and midwives to perform more functions; and o Performance Based Incentive programs to enhance facility-based quality of care. FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 51 • Contract a newborn expert to: o Provide examples of global best practice in improving QOC for newborns; o Suggest needed studies to identify key constraints in relation to the National Every Newborn Action Plan (ENAP); and o Use this evidence base, to develop robust indicators and high-impact interventions. Assist the MOH to identify and undertake operational research on key external constraints to better inform policies and systems, such as: Accreditation • Seek to identify and modify key criteria that will affect MNH outcomes such as availability of blood banks and OB/GYNs 24/7 at a CEmONC. • Conduct a baseline survey to assess the percentage of BEmONC/CEmONC certified facilities that do/do not meet WHO standards and monitor changes over life of project. • Work with DHOs to address gaps in WHO-recommended coverage of BEmONC and CEmONC facilities. JKN and reimbursement • To identify and report on policies which adversely affect maternal/newborn outcomes, e.g.: o Reimbursements to midwives for referrals, o Caps on reimbursements for newborn complications, and o Equity and access among the poor, disadvantaged and young women. Licensing and compliance of private midwives and facilities • Conduct baseline survey and routine spot checks of compliance of private midwifery facilities with regulations over provision of services. • Advocate IBI for increased stringency and technical criteria for licensing of midwives – especially in the number of required pre-service deliveries witnessed and performed; and for changes in regulations to allow trained midwives to perform all BEmONC signal functions. Task shifting • Use WHO approaches to assess needs for task shifting and approaches to achieve buy￾in by central GOI and other stakeholders. Consider novel approaches such as co￾management. • Consider piloting a system to track presence/absence, active involvement, response￾time of specialists to document shortages and to increase accountability. Make this information available to the public. Cost-effectiveness and cost-benefit studies • Conduct, as appropriate, cost-effectiveness and cost-benefits studies of each approach to provide evidence of cost-savings that result from improved QOC to JKN, accreditation bodies, the MOH, Bupati, BAPPENAS and other GOI institutions to inform future policies and health financing decisions and ensure ongoing support by GOI. FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 52 • Conduct baseline, mid and endline studies to measure the impact of any changes in outcomes at district-level; preferably, in comparison to non-project “control” districts. • Test performance based incentives (to be awarded to entire department staff) as a means to improve compliance with facility SOPs. VI. LESSONS LEARNED A summary of over-arching lessons learned from EMAS is presented below. The key role of data and evidence: Among the most critical findings and lessons learned from this evaluation were the constraints and limitations in data quality and the EMAS PMP and indicator set, which posed challenges both to the program and stakeholders during implementation, and to this team in assessing successful approaches. Future projects with a baseline study conducted at inception would be better positioned to use quantitative evidence from this and other relevant sources to develop a clear theory of change and program framework. A robust PMP and set of indicators, as well as routine monitoring and evaluation plans designed to assess both the program’s progress and assumptions could also help reduce constraints. All project innovations and approaches must generate evidence sufficient to assess whether and which interventions should be adjusted or abandoned versus those to be sustained and scaled. Annual portfolio and project reviews with key stakeholders would also improve communications and ensure coordination of efforts – both internal to the project and factors external to project mandate and influence. Achieving impact at scale: Where the aim is to achieve impact at scale, an initial plan and clear pathway for achieving and measuring impact at scale, as well as a learning agenda to document and guide decision-making, need to be defined during the project design phase and progress towards this with adjustments as needed also reviewed annually. Key findings and lessons from EMAS illustrate that if these are not required at the time of project formation, the ability of the project to understand and respond to these needs later will be limited or absent. Replication, scale-up and sustainability: EMAS routine data collection and reporting did not include analysis or correlation with the data collected by partners working in the field – in mentoring, referrals or development of local governance entities. Without this, the evidence available is insufficient to analyze and quantify the relative achievements of each approach, and whether and to what extent the various approaches are effective in improving MOH systems, QOC and maternal newborn outcomes. An alternative plan for data presentation to the MOH and stakeholders, for example, one that includes trend-analysis of routine monitoring indicators would provide a meaningful basis for future political and financial commitment to replicate and sustain project activities, as well as an adequate basis for system-wide reform developed by the MOH and other GOI institutions. Design considerations: While USAID-supported activities cannot and should not seek to address all factors contributing to mortality, a periodic re-assessment of the selected strategy and initial assumptions about key leverage points to be addressed to achieve mortality reduction would strengthen project and MOH efforts and outcomes. Future projects should have the flexibility to adapt to new information built in to the design, rather than continuing with the initial approaches and interventions based on initial assumptions, as was the case of EMAS. EMAS demonstrates that if this is not built into a program from design and selection, it is unlikely that the project will develop the ability to learn and adapt later. FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 53 Assessment of determinants: In the future, an assessment of determinants to check assumptions should be included from the start – in project design, development of a framework, and approaches and interventions. These should be monitored as part of the PMP and M&E plan, including activities of key stakeholders (such as the GOI, MOH and professional organizations) as periodic checks to inform programmatic adjustments to project priorities and activities. The death reviews are an excellent example of information that, if available at project start up, could have significantly influenced initial project priorities as well as relationships and interactions with other stakeholders better positioned to address some key determinants. As above, USAID projects cannot address all determinants, but should operate effectively with all stakeholders to make unique, important contributions to the broad spectrum of activities being implemented by, and with partners. Ultimately and in doing so, future programs are more likely to successfully improve outcomes and impact in project areas (both technical and geographical) and the MOH and GOI also better positioned to respond to needs outside the project’s mandate. FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 54 ANNEXES Annex I: EMAS Final Evaluation Statement of Work I. Introduction USAID/Indonesia’s Expanding Maternal and Neonatal Survival (EMAS) Project is a five-year activity that supports the Government of Indonesia (GOI) in its efforts to reduce maternal and newborn mortality. To inform future USAID investments in maternal and neonatal health (MNH), USAID/Indonesia is commissioning the Monitoring & Evaluation Support Project (MESP) to perform a final performance evaluation of EMAS. II. Activity Description EMAS is a five-year activity (September 2011 – March 2017) to support the GOI in reducing maternal and newborn mortality. EMAS, implemented by JHPIEGO, works with GOI agencies (national, provincial and district), civil society organizations (CSOs), hospitals and health centers. Over its five years, EMAS has worked with 150 public and private hospitals and more than 300 community health centers across 30 districts and cities in 6 provinces: North Sumatra, Banten, West Java, Central Java, East Java, and South Sulawesi. EMAS is also emphasizing scale-up and sustainability in order to impact districts and provinces outside its 30 target districts. Intervention districts have been added to the activity across three intervention phases. Phase 1 began in 2012, Phase 2 began in 2013 and Phase 3 in January 2015, with an overall budget of $55 million. EMAS received a six-month no-cost extension in September 2016, to continue activities until December 2016, to be followed by a three-month close out period until March 31, 2017. As the Mission’s flagship MCH activity, EMAS seeks to contribute to reduced maternal and newborn mortality by focusing on three major objectives: 1. Improved quality of emergency obstetric and neonatal care (EmONC) services in hospitals and community health centers. 2. Increased efficiency and effectiveness of referral systems between community health centers and hospitals. 3. Strengthened accountability among government, community, and health systems. III. Background and Development Hypothesis The design of EMAS is built on two decades of MCH program experience, and is based on four key assumptions. 1. To drive down mortality, services need to be improved at the facility level because women are increasingly delivering at facilities. A particular emphasis on improving emergency services is also needed to drive down mortality, in addition to improving routine delivery. 2. Referral systems are fraught with problems, which contribute to women not reaching services in a timely manner and receiving the right level of care they need. Improving such systems is a crucial part of the puzzle to improve key MCH metrics. 3. Maternal and newborn survival is no longer solely a function of the health system, and any future interventions should have a governance aspect that brings in community organizations and engages local governments to create participation across the stakeholder spectrum and increase accountability. FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 55 4. Decentralization is a driving force within the government and district based counterparts need to be the primary agents of change. The EMAS Results Framework is presented below: IV. Existing Information Sources USAID suggests the following materials for the EMAS final evaluation desk review: 1. EMAS Grant Agreement and technical program description from the Project Agreement 2. EMAS Performance Monitoring Plan and indicator data tables 3. EMAS Midterm Evaluation, Month, Year 4. EMAS Midterm Management Review, Month, Year 5. EMAS Outcome Evaluation Data and Report (due November 2016) 6. EMAS studies on maternal and neonatal deaths in projects areas 7. National Academy of Sciences 2013 monograph on reducing maternal and neonatal mortality in Indonesia 8. EMAS Annual and Quarterly Progress Reports 9. Information from the Evidence Summit 10. Other EMAS technical materials (e.g., case studies, death reviews, annual report executive summaries, factsheet, infographics, profiles, program briefs and technical briefs) V. Evaluation Purpose, Audience, and Intended Uses Contribute to Reduced Maternal and Newborn Mortality Improved quality of emergency MNH services Increased coverage of life-savingMNH Interventions Increased efficiency and effectiveness of referral system Quality improvement - Strategic leadership & shared vision - Clinical performance standard - Provider behavioral intervention - Staff from puskesmas complete rotations - Death and Near Miss Audits - Dashboard Referral strengthened - Referral performance standards - Network MoUs - Sijari EMAS (referral exchange) - Maternal Perinatal Audits at District Level (Pathway audit) - Equitable access/ Social insurance - Dashboard Strengthening Accountability for Improved Policies and Resources - Civil engangement through MKIA - Pokja - multi sector TWG - Civil Forum Community M e n t o r i n g P r o c e s FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 56 Purpose The purpose of the EMAS evaluation is to learn from program successes and challenges. USAID will use the findings from this evaluation to inform future MCH portfolio decisions. This final evaluation is not intended to be an implementing partner performance evaluation, but should build on the work done in the midterm evaluation by identifying what factors and interventions contributed to the successes and shortcomings of EMAS, as well as identify and document opportunities for further improvement. Audiences There are three distinct groups that should receive and make use of evaluation findings. The first is the GOI stakeholders at the central, provincial, and district levels. The second is other local stakeholders such as POGI (Ob-Gyn Association), IDAI (Pediatric Association), private sector partners, CSOs, etc. The third group includes USAID and other development practitioners in Indonesia. Intended Uses This evaluation should help inform future MCH interventions in Indonesia, especially given the context of universal health care (UHC), persistently high maternal and newborn mortality rates, and continuing devolution of health management and service delivery authorities. Overall, learning from the EMAS evaluation should focus on: • Improving quality of MCH services in facilities • Improving the effectiveness of emergency referrals • Governance practices, including policy relationships with the district, provincial and central level counterparts in the Ministry of Health and other agencies • Scale and sustainability Evaluation findings should be widely disseminated and discussed with high-level GOI policy makers, including in the Ministry of Health, Ministry of Home Affairs, and Bappenas, in order to begin a process of building a consensus around addressing the evaluation’s findings and recommendations. Stakeholders at the province and district levels should also be able to review evaluation findings and discuss how they can be incorporated into future practices. All evaluation materials will be posted to the USAID Development Exchange Clearinghouse (DEC). VI. Evaluation Questions To guide this evaluation, USAID has identified key questions (below) and lines of inquiry for each question (Annex B). 1. Did EMAS primary approaches as implemented lead to improved outcomes, and did they have the desired magnitude of impact? 2. What are the explanations for why these approaches worked or did not work, and what evidence supports these explanations (e.g., facilitation, limitation, context, continuum of impact)? 3. How did EMAS learn and use evidence to make adjustments in program approaches during implementation? 4. What has been learned that can inform future priorities, approaches, and activity processes, including internal USAID programmatic and bureaucratic processes? VII. Gender Considerations FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 57 In accordance with USAID’s Automated Directive System (ADS) 201 point 7, the research design for this evaluation will consider gender-specific and differential effects of the EMAS program. The evaluation team will explore gender aspects of the activity per the questions and data sources in Annex C. VIII. Data Collection and Analysis Methods The evaluation team will propose a comprehensive design for addressing the evaluation questions and will address how choices will be made related to selecting country/geographic focus and intervention activities/results, to maximize the ability of the evaluation to develop findings based on evidence and that will be relevant to the broadest possible range of activities. Within this design, the evaluation team will propose the best and most rigorous methods for data collection appropriate to address the evaluation questions. The evaluation team will conduct Team Preparation Meeting to discuss further detailed of data collection and analysis methods. Annex D provides an illustrative “Illustrative Analytical Framework for the Evaluation” matrix, including a range of potential data collection methods that may be suitable for each evaluation question. This matrix will be further developed by the evaluation team in the Evaluation Design Proposal. Data Analysis Methods Data analysis methods to be proposed by the evaluation team will follow closely from the methods used to collect each type of data needed to answer the evaluation questions. Whatever data analysis methods are chosen for this evaluation, they should be justified in terms of their fit with the data collected for each question and the types of answers that USAID seeks. Time and cost considerations are also important in this area. The evaluation team’s proposed design for this evaluation should consider the most appropriate sequence for answering each question, based on the data that will be collected and analyzed for each question. IX. Deliverables The evaluation team will be responsible for the following deliverables. Specific due dates will be proposed in the evaluation team’s Evaluation Design, following the TPM. Deliverable Estimated Due Date 1. Evaluation Design draft, including detailed research methodology, drafts of data collection instruments, sampling plan, and implementation plan & schedule. 1. One week after completing the TPM in Jakarta and associated meetings with USAID and the EMAS implementing partner. 2. Final Evaluation Design. 2. One week after receiving USAID and any GOI comments on Design Proposal. 3. Draft Evaluation Report w/ draft set of infographics highlighting EMAS results. 3. Initial draft within three weeks after completion of fieldwork. 4. Together with Draft Evaluation Report. 5. Together with Draft Evaluation Report. 4. Oral presentation(s) of key findings, etc. on draft findings and recommendation to USAID – 2 versions – 1 for internal USAID consumption and 1 for external use. 5. PowerPoint presentation on how USAID programmatic and administrative 6. Within four weeks after completion of fieldwork. 7. Together with oral presentation to FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 58 Deliverable Estimated Due Date processes could be improved for future MCH activities (e.g., start-up, management, leadership support, and technical intervention design. USAID. 6. Oral presentation(s) of key findings, etc. on evaluation findings and recommendation to the MOH. 7. Presentation on comparative review case study examples addressing clinical care, emergency referrals, and local-level accountability mechanisms from other middle- and upper-income countries. (This presentation can be combined with deliverable #8, per USAID & GOI schedules.) 8. Following presentation to USAID and USAID approval of GOI presentation materials. Timing TBD based on GOI availability. 9. Following presentation to USAID and USAID approval of GOI presentation materials. Timing TBD based on GOI availability. 8. Final Evaluation Report w/ final set of infographics. 9. Detailed case study paper regarding program approaches addressing clinical care, emergency referral, and local-level accountability mechanisms in other middle- and upper-income countries, to be submitted to the GOI for possible follow￾up. 10. Detailed outline and initial draft text for article(s) to be submitted to international and Indonesian journals regarding EMAS about valuable experiences and lessons from EMAS implementation, outcomes, and adaptive management, based on evaluation findings. 11. Draft OpEd article for USAID Mission Director re: Evaluation findings, in English and Bahasa Indonesia. 10. Two weeks after receiving last USAID comments on the Draft Evaluation Report and associated materials. Note: there may be more than one revision before the Final Report is completed. Est’d end May 2017. 11. Together with Final Evaluation Report. 12. Together with Final Evaluation Report. 13. Together with Final Evaluation Report. All documents and reports will be provided electronically to USAID no later than the dates indicated in the approved Evaluation Design Proposal. All qualitative and quantitative data will be provided in electronic format to USAID in a format consistent with Automated Directives System (ADS) 579 requirements. All debriefs will include a formal presentation with slides delivered both electronically and in hard copy for all attendees. X. Reporting and Dissemination The format of the evaluation report should follow USAID guidelines set forth in the USAID Evaluation Report Template (http://usaidlearninglab.org/library/evaluation-report-template) and the How-To Note on Preparing Evaluation Reports (http://usaidlearninglab.org/library/how￾note-preparing-evaluation-reports). Evaluation team members will be provided with the USAID’s mandatory statement of the evaluation standards they are expected to meet (see Annex A) FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 59 Dissemination actions are expected to include some or all of the following items, per agreement with USAID: • Executive Summary, summarizing EMAS, evaluation methodology, and main findings and recommendations. • Developing a list of supporting materials required for communications through consultation with the USAID activity manager and USAID communications team. • Developing a final version of a power point presentation for USAID internal use, summarizing EMAS, the evaluation methodology, and evaluation findings, and recommendations. • Developing a final version of a power point presentation for external use, summarizing EMAS, the evaluation methodology, and evaluation findings, and recommendations. • Working in close collaboration with USAID, organize a dissemination event where the final external power point and other methods, e.g. a panel discussion, are presented and used to disseminate findings and recommendations to a broad range of stakeholders, including the GOI, professional associations, NGO's and civil society organizations working in MNH, academia, private sector organizations, and media are represented. • Developing a press release for this external dissemination event. • Developing a two-page summary fact sheet with a summary and key findings and recommendations, suitable for use for other dissemination activities (these activities are to be discussed, but may include further media events, social media, newspaper articles and/or Op-Ed pieces (draft), etc.). • Infographics summarizing the results • Other activities as identified with USAID such as – article to be published in the MOH newsletter to the field, article for MOH magazine, etc. XI. Team Composition The ideal composition of the evaluation team would be six people, at least four of them Indonesian nationals: • Team Leader/Evaluation & Health Expert, international • MCH Expert, international • Health Systems/Governance Expert, Indonesian • Evaluation Specialists (2), Indonesian • Research Assistant, Indonesian Within this team, one international and one Indonesian expert will develop the outline for journal articles based on evaluation findings (see Table of Deliverables, above), and will work for one additional month beyond the approval of the Final Evaluation Report by USAID, to write that article(s) and submit to journals as agreed by USAID. In addition to this core evaluation team, one international MCH Expert will work in parallel to the team for approximately one month to produce the comparative review deliverables detailed above. Selection criteria for team members include the following: • Strong knowledge of Indonesia, both the health sector and governance structures • Knowledge of USAID programming practices • For the team leader, prior successful experience in leading teams FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 60 • Knowledge of qualitative and quantitative evaluation practices. • Ability to communicate in Bahasa Indonesia. • One clinical specialist with a background in both maternal and neonatal health. • Excellent writing and inter-personal communication skills. • For the MCH comparative review expert, extensive experience working as a technical specialist within a government health system or with a consulting/international organization advising government health systems, including upper and middle income countries (e.g., Thailand, China, Malaysia, Australia, UK, Germany). Exposure to Indonesia’s health systems is preferable. XII. USAID Participation Regular communication between the evaluation team and the designated USAID Activity Manager will be essential to the successful execution of the EMAS evaluation. The evaluation team will keep USAID apprised of changes and developments that necessitate any significant decision-making or modification of the approved evaluation design. Possible USAID participation in the data collection phase of the evaluation will be discussed in the TPM, prior to the start of fieldwork. XIII. Scheduling and Logistics EMAS will close its activities in March 2017, with programmatic activities will end in December 2016. There is the anticipated release of the report of the internal outcome evaluation by November 2016, which is crucial documentation for this overall final evaluation. The optimal timing would be to begin the desk work in early November, extend field work through mid￾March (given the scope of the evaluation) and have a final report due the first week of April. The Mission has access to an evaluation project managed by MSI who will be able to finalize the scope of work and manage the logistics. Budget considerations for the evaluation will depend on which timing option is chosen, how big the team is, how much travel will be done in country and how many consultative meetings are held. Estimated EMAS Final Evaluation Timeline (2016-2017) Task/Delive rable Dec 2016 Jan 2017 Feb Mar Apr May June Evaluation SOW Evaluation Preparation and TPM TPM Evaluation Final Design Desk Review FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 61 Fieldwork In-Country USAID Debrief(s) Analysis & Report Writing Draft Eval Report & Deliverables Final EvalReport& Deliverables XIV. Budget A full detailed budget will then be prepared and included in the Design Proposal for USAID’s proposal FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 62 Annex II: Evaluation Methods and Data Collection Tools MSI’s Evaluation Standard Operating Procedures (SOPs) are being applied to the EMAS evaluation to ensure that the evaluation team gets off to an effective start and is able to focus on analytical issues from the outset. The SOPs define step-by-step timelines for the pre-planning work that is required to manage an effective evaluation process. Ensuring the application of the SOPs will be the responsibility of the MESP COP and task owner. The SOPs identify the actions to be completed in Indonesia prior to an evaluation team’s mobilization, including the completion of a background literature review, identification of key stakeholders, and the development of an evaluation schedule. The evaluation schedule will include a list of focus activities to be evaluated, budget allocations and a geographic profile of the areas to be analyzed. This will enable MESP to recommend and plan the evaluation field schedule prior to the team starting work, and also permits the distribution of notification and scheduling letters to IPs and government officials ahead of time. The Team Planning Meeting (TPM) process was initiated on 23 January at the MESP offices in Jakarta. All evaluation members, relevant MESP staff and representatives of USAID’s Program and Technical Offices participated. Representatives of the MOH also attended sessions. The EMAS evaluation team continued with TPMs throughout the week, presenting final site selection and travel plans to USAID and the MOH on 27 January at MSI. TPM meetings covered best evaluation practices, such as: the usage of MSI’s Getting to Answers Matrix as part of a structured process for developing the evaluation’s detailed methodology; the importance of findings, conclusions and recommendations matrices; the need to update the analysis on an on-going basis; and a clarification of team roles and responsibilities. The evaluation team plans to use both quantitative and qualitative methodologies to address the evaluation questions, including (but not limited to) document and literature review, analysis of secondary data, key informant interviews and site visits. Following fieldwork, the data will be synthesized and analyzed using the framework of the Getting to Answers Matrix (Annex C) to answer the evaluation questions and sub-questions. The evaluation team will develop detailed data analysis and report writing plans (with draft report outlines), and share these with USAID teams for initial comments. METHODOLOGY The following are anticipated elements of the methodology: 1. Document and literature review: The evaluation team is conducting a comprehensive literature review of pertinent documents including study and assessments regarding EMAS quality services in facilities, EMAS effectiveness of emergency referrals, EMAS governance practices and EMAS scale and sustainability, including but not limited to: • List from SOW, USAID, & IP. • EMAS Grant Agreement and technical program description from the Project Agreement • EMAS Performance Monitoring Plan and indicator data tables • EMAS Midterm Evaluation, 2014 • EMAS Midterm Management Review • EMAS Outcome Evaluation Data and Report (due November 2016) • EMAS studies on maternal and neonatal deaths in projects areas FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 63 • National Academy of Sciences 2013 monograph on reducing maternal and neonatal mortality in Indonesia • EMAS Annual and Quarterly Progress Reports • Information from the Evidence Summit • Other EMAS technical materials (e.g., case studies, death reviews, annual report executive summaries, factsheet, infographics, profiles, program briefs and technical briefs) • Other relevant documents. A summary of the literature review is now being completed. The literature review will include overview of EMAS, key approaches/activities used, key stakeholders involved, as well as other policies that may affect the implementation of EMAS. 2. Collection and analysis of secondary data: The evaluation team will make extensive use of any and all data available that is pertinent to the evaluation questions. These include: • The evaluation team will collect secondary data from various sources which are data from PMP (Performance Monitoring Plan) data, annual report, quantitative and qualitative data from Midterm Evaluation, data in clinical assessment database, IDHS (Indonesia demographic health survey), standardized EMAS registry book. • These data will be used and analyzed to answer evaluation questions and provide preliminary findings before team goes to the fieldwork. • Preliminary findings will be used for comparative study (with other middle-income and from upper-income countries) and identifying the potential area/topic for published journal 3. Key Informant Interviews: In-depth interviews or group discussions will be conducted with key informants at central, provincial, district and site levels. The following are proposed as key stakeholders, and important informants. The list of key informants and any needed interview protocols (with interview guides) will be finalized with USAID/Indonesia prior to the fieldwork. • USAID/Indonesia: MCH team and PRO team. • Implementing partners: JHPIEGO (prime), the Budi Kemuliaan, Muhammadiyah, Save the Children and Research Triangle Institute (RTI). With each partner the team will interview: − Chief of Party (COP), Deputy COP, program manager, technical coordinators and M&E Staff • Central level stakeholders potentially include: − Ministry of Health, Ministry of Home Affairs and Bappenas • Local Stakeholders include: − Dinas Kesehatan (PHO &DHO) − Service providers including district health centers (Puskesmas), referral hospitals, private hospitals, clinicians and mentors (e.g. OB/GYNs, nurses, midwives), hospital administration staff (systems and data managers), provincial and district health authorities, professional associates − MKIA, Pokja, Civil Forum Community 4. Direct site visits and observations: The fieldwork will include site visits, interviews and direct observations of facilities and related activities in selected districts in North Sumatra, Banten, West Java, Central Java, and South Sulawesi. The list of facilities and community places to be visited is in development and will be finalized by the evaluation team, in FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 64 consultation with EMAS and USAID. Province, district, and facility selection was based on the following set of inclusion and exclusion criteria: Inclusion Criteria Exclusion Criteria MOH Priority District, USAID selected EMAS end line district MOH Replication High and Low Performance Similar outcomes Phases 1, 2, & 3 Strong and Poor Governance/Commitment Not part of future USAID program High/low birth volume Similar geography, context Urban versus rural Travel time/distance Private sector Final district site selection includes: • Central Java: Kudus, Pekalongan and Kota Semarang • West Java: Karawang & Bandung • South Sulawesi: Bulukumba & Gowa • North Sumatra: Deli Serdang & Serdang Bedagai • Banten: Tangerang • East Java: The EMAS Final Evaluation team will not visit East Java. However, the MCH Portfolio Review team will visit EMAS activity sites and stakeholders as part of its fieldwork in Feb-Mar 2017 and will share the findings with the EMAS evaluation team. DATA ANALYSIS Several types of data analysis will be conducted based on the evaluation design. These include: Document review: • Comparisons may be made between the original program design, the IP agreements, PMPs and implementation work plans to identify potential strengths, weaknesses and modifications of the program. These will be compared to the strategic design of the current RFP. • EMAS program reports and documents: Annual reports, technical briefs, audit reports and other programmatic tools and standards. • International reports and documentation on best practices (WHO, etc) and national guidelines, strategic documents, plans and policies. Qualitative data analysis: A framework for data analysis will be designed before data collection begins and refined as needed at the end of fieldwork. This will specify the main topics to be analyzed as organized by the evaluation questions. It is also be used to summarize interview notes and assure that the field interviews are covering all pertinent points at the end of each week of field work. Quantitative data analysis: • Comparison of clinical and district level data at the start and end of EMAS • Analysis of EMAS progress towards indicators from reports and PMP • Triangulation: wherever possible we will attempt to validate the qualitative findings with quantitative data. DATA COLLECTION INSTRUMENTS FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 65 GOVERNANCE KII INSTRUMENT: POKJA (PHO & DHO, BAPPEDA) AND FMM (CSOS, LOCAL COMMUNITIES AND CATCHMENT GROUPS). Introduction: Provide a brief introduction to the evaluation, main objectives, names and roles of the team members present. 1. Did EMAS approaches lead to improved outcomes, and did they have impact? Possible probes: - Examples of outcomes that did/not improve; • Concrete examples that POKJA/Civic Forum improve accessibility and/or quality of maternal and newborn care (others: policy issues; UHC; sanction and reward) • Frequency of meetings conducted by POKJA/Civic Forum during the past 12 months • What were the topics discussed by the POKJA/Civic Forum during the past 12 months (e.g. availability of key personnel 24/7; blood banks; death/near miss audit; case fatality and near miss rate; adequacy of referral; private midwives) 2. Please give evidence to support why these approaches worked or did not work? Possible probes: - Examples of action taken as a result of direction from POKJA - Examples of action taken as a result of inputs from - Gaps, challenges, needs (e.g. scope, & scale, context, replication, ownership) 3. Did EMAS make adjustments in program approaches during implementation? Possible probes: • Were decisions made based on data/evidence (adequacy and accuracy of data) • Service charter • Adjustment in mentoring, referral, other • Utilization of suggestion box (Kotak Saran) – Citizen feedback 4. What did you learn through EMAS? What are your future priorities and strategies? Possible probes: • What is the future agenda of POKJA/Civic Forum • Examples of action taken as a result of direction from POKJA • Any plan to involve additional stakeholder especially private sector (companies, pharmaceuticals) • Support to other key stakeholders- private companies, private practitioners • Other innovations- media/social media; branding and advocacy; certification, licensing for practice, accreditation; insurance/reimbursement; performance-based incentives; pre￾service training; new technologies (ICT); new health technology; local sourcing • Equity – gender & most-at-risk FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 66 FACILITY LEVEL KII INSTRUMENT: PUBLIC & PRIVATE HOSPITALS, PUSKESMAS (RESPONDENTS INCLUDE: OBS, MIDWIVES, NURSES, MENTORS/MENTEES). Introduction: Provide a brief introduction to the evaluation, main objectives, names and roles of the team members present. 1. Did EMAS approaches lead to improved outcomes, and did they have impact? Possible probes: - Examples of outcomes that did/not improve; • Critical aspects of QOC that improved for mothers, newborns? Why/How? • Critical aspects still problematic? - Examples of improved QOC – SijariEMAS? • What can be improved? - Examples of improved QOC – Death audits? • Maternal and newborn? • What can be improved? 2. Please give evidence to support why these approaches worked or did not work? Possible probes: - Explanation and evidence of how outcomes improved due to; • Mentoring, referrals, governance, other - Suggestions to tackle persistent poor outcomes; • Mentoring, referrals, governance, other - Gaps, challenges, needs (e.g. scope, & scale, context, replication, ownership) 3. Did EMAS make adjustments in program approaches during implementation? Possible probes: - Examples of shifts – useful/helpful or not? - Ways the program could have changed/adapted better? 4. What did you learn through EMAS? What should be future priorities and strategies for QOC? Possible probes: - Summary of best practices/key challenges; - What should continue; - What could be improved; - What should be added; - Institutionalization & sustainability FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 67 SYSTEMS KII INSTRUMENT: HMIS & REFERRALS STAFF AT THE DHO, UHC, PUBLIC & PRIVATE HOSPITALS AND PUSKESMAS. Introduction: Provide a brief introduction to the evaluation, main objectives, names and roles of the team members present. 1. Did EMAS approaches lead to improved outcomes, and did they have impact? Possible probes: - Explanation and evidence of how HMIS improved due to; • Data capture tools (register, dashboard, etc.) • TA in data systems, analysis, use • SijariEMAS system data tracking & use • Data – accuracy, quality, usefulness • EMAS data- shared? Used? Helpful? - QOC improvements 2. Please give evidence to support why these approaches worked or did not work? Possible probes: - Explanation and evidence of how outcomes improved; • HMIS system versus EMAS system(s) - Suggestions to tackle persistent poor outcomes; • How to improve HMIS; integrate EMAS referrals, other - Gaps, challenges, needs (e.g. scope, & scale, context, replication, ownership) 3. Did EMAS make adjustments in program approaches during implementation? Possible probes: - Examples of shifts – useful/helpful or not? - Ways the program could have changed/adapted better? 4. What did you learn through EMAS? What are your future priorities and strategies? Possible probes: - Summary of best practices/key challenges; - What should continue; - What could be improved; - What should be added; - Institutionalization & sustainability FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 68 CENTRAL KII INSTRUMENT: USAID, EMAS, MOH, MOP, UNICEF, WORLD BANK, IDAI, IBI, ETC. Introduction: Provide a brief introduction to the evaluation, main objectives, names and roles of the team members present. 1. Did EMAS approaches lead to improved outcomes, and did they have impact? Possible probes: - Examples of outcomes that did/not improve; - Strategy, approach; 2. Please give evidence to support why these approaches worked or did not work? Possible probes: - Explanation and evidence of how outcomes improved; - Explanation and evidence of poor outcomes; - Gaps, challenges, needs (e.g. scope, & scale, context, replication, ownership) 3. Did EMAS make adjustments in program approaches during implementation? Possible probes (For EMAS, USAID and MOH): - EMAS shifts in programmatic priorities: indicators, districts, facilities; - Responses to midterm evaluation - Change in strategy/implementation approach from Phase 1 to 3 - Management and coordination at central, provincial and district levels; - Gaps, challenges, needs (outcome, intervention, scope, scale) 4. What are key lessons learned from EMAS? What should be USAID future priorities and strategies to best support the MOH? Possible probes: - Summary of best practices/key challenges; - What should continue; - What could be improved/added; - Institutionalization & sustainability - New innovations: performance based financing; private companies, private practitioners; certification, licensing for practice, accreditation; insurance/reimbursement; performance-based incentives; pre-service training; new technologies (ICT); - Equity – gender and most-at-risk FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 69 Annex III: Sources of Information – KIIs & Documents Reviewed I. KEY INFORMANT INTERVIEWS FOR THE EMAS FINAL EVALUATION PROVINCE DATE of INTERVIEW DISTRICT/ CITY INSTI-TUTION NAME OCCUPATION DKI Jakarta 30 Jan 2017 Jakarta MoH Dr. Anung Sugihantoro, M.Kes Director of General of Public Health Dr. Eni Gustina, MPH Director of Family Health Dr. Lukas C. Hermawan, M.Kes Sub-Head Directorate of Maternal& Neonatal Health Dr. Nida Rohmawati, MPH Head Section of Neonatal Health EMAS Trinawati Referral Team Leader JHPIEGO Kristina Program Manager 31 Jan 2017 POGI dr. Nurdadi Saleh, SpOG Chairman of Supreme Council 1 Feb 2017 MoH Drg.Oscar Primadi, MPH Head Communication and Public Service Bureau EMAS Lilik Clinical Team Leader 2 Feb 2017 JHPIEGO Maya Tholandi Monitoring, Evaluation & Research Systems Team Leader JHPIEGO Nancy Caiola Field Operations Team Leader EMAS Dr Dwirani Amelia SpOG EMAS Clinical Governance Advisor EMAS Dr. Trisna EMAS Component-2 Team Leader EMAS EMAS & M&E TEAM 3 Feb 2017 Save The Children Dr. Pancho Kaslam, SpA. Advisor of EMAS MoH Boga Central Data and Information (Pusdatin) JHPIEGO Ali Zazri EMAS M&E West Java 6 Feb 2017 Karawang DHO Dr. Yuska Head of DHO Karawang Rina Listiana Data person Midwives Staff of DHO/IBI member Dr.Rashim POKJA member Dr.Yayuk POKJA member BAPPEDA Eka Sanata Head of BAPPEDA RS Citra Sari Husada Karawang Dr.Tresna, SpKK Director of RS.Citra Sari Husada Annisa Coordinator Midwives Ely Coordinator Midwives Doctor Obgyn RSUD Karawang Arief Gunawan Deputy Director Fitri Head of Delivery room FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 70 PROVINCE DATE of INTERVIEW DISTRICT/ CITY INSTI-TUTION NAME OCCUPATION Dr.Unggul, SpOG POGI of Karawang Dr. Didi Sukandi, SpA IDAI of karawang Puskesmas Curug Asep Gunawan SKM,M.Kes Head of Puskesmas Dr Budi Suharyanto GP IntanPratiwi Responsible for PONED TutiTurkawati Midwife Puskesmas Klari Dr.Dini Head of Puskesmas Klari Bd. Cucu Mentor at Puskemas Klari - GP 7 Feb 2017 Bandung DHO Drg.Pradiba Public Health of DHO Bandung/ POKJA member Nonong Head of IBI chapter District of Bandung Sri Endarsih Data Person Aam uamar FMM Eka MKIA RSUD Majalaya Dr.Yuli Head of Medical Service - Midwife - Obgyn Puskesmas Cikalong Drg.Novita Head of Puskemas Cikalong Ety Coordinator of midwives Puskesmas Ciparay Trijeni Fitriani, Amd.Keb Head of Puskemas Ciparay Lilis Mentor/midwife 8 Feb 2017 PHO Dr.Djoko EMAS Province Team Leader dr. Tini Sudartini Head of Community Health-West java PHO dr. Yuanita Head Sectoin of Family Health drg. Marion Sub-Head of Helath Effort Uus Sukmana Taskforce For Saving Mothers and Newborn Coordinator Central Java 9 Feb 2017 Semarang DHO Dr. Widoyono,MPH Head of DHO Ir. Sri Purwanti Referral Person/POKJA Member M.Juma’i FMM RSUD Tugurejo Dr.Yuswanti,MH.Sc Deputy Director Dr. Retno Wuryani Head of Hospital Ward and PONEK Technical Coordinator FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 71 PROVINCE DATE of INTERVIEW DISTRICT/ CITY INSTI-TUTION NAME OCCUPATION Dr.Agus,SpA IDAI member Dr.Taufik, SpOG EMAS coordinator - Midwives - Nurses IBI Central Java Province Secretariat Ristyah Deputy IBI Head Yenny Secretary of Bidan Delima RSUP Kariadi Dr.Darwito Medical Nursing Director Tim Mentor Team Mentor RSUP Kariadi Puskesmas Halmahera Dr.Hidayanto Head of Puskesmas Halmahera Tri Suksesi PONED midwives Coordinator 10 Feb 2017 PHO Dr.Hartanto EMAS PTL Kudus DHO Djoko Dwi Putranto SH, MM Head of DHO Endang Triastuti Sub-Head of Family Health and Nutrition Emlia Suryani Staff of Family Health – Kudus DHO RSUD Dr.Loekmono Dr.Budi Santosa,SpRM Deputy Director Dr.Abdul hakam IDI Dr.Aditya,SpA Member of IDAI Dr.Hari Head of PONEK RS Aisiyah Dr.Hilal Hariadi,,Mkes Director - Midwife Puskesmas Jekulo Dr.Enny Runayah Head Of Puskesmas Jekulo Dr Deni Witjaksono GP Drg. Kurnia GP Puskesmas Dawe Darsono Head of Puskemas Dawe - Coordinator midwives 11 Feb 2017 Pekalongan DHO Dr.Sutanto Setiabudi Head of DHO Budi Darmoyo Head section of Community Health - BAPPEDA staff - Coordinator midwives/Member of IBI - Head NCD Division - Other DHO Staff Dr.Ridho Ex-Quality Control of EMAS RSUD Kajen Istikharoh Head Sub Medical Service RSI Muhammadiyah Dr.Widjidan Kadir Deputy Director Dr.Arif Darmawan, SpOG,MKes POGI member FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 72 PROVINCE DATE of INTERVIEW DISTRICT/ CITY INSTI-TUTION NAME OCCUPATION Dr.Soraya Doctor Sukisto,S.Kep Nurse Puskesmas Karanganyar Bambang Rusmanto, SKM.MKes Head of Puskesmas Dr.Suryadi GP Umi Nadhiroh, AmdKeb Coordinator midwives of Mother and Child Health Esty , Amd.Keb Coordinator midwife of Delivery Puskesmas karangdadap Drg.Asmuni Head of Puskesmas - Coordinator midwife South Sulawesi 13 Feb 2017 Gowa Asri pati Roy Ex-EMAS (SijariEMAS) Bulukumba Bakri FMM coordinator 14 Feb 2017 Gowa DHO Armin Secretary of DHO, POKJA Rumaisah Head Sub of Binkesmas Rossalina Head Sub of Health Resources Nurmiyati POKJA- referral Rosalina Ketua cabang IBI Hasni Hayat FMM member Rosna MKIA RSUD Syekh Yusuf Dr.H.Solahuddin, M.Kes Director Dr.Syamsul Pediatric Doctor Obgyn Hasriwati Midwife Mardiwati Midwife Masna Midwife Dr.Maya Deputy Director of Medical Service Dr.Ummu Salamah Head Sub of medical Service Puskesmas Kampili Dr.Aswan Head of Puskesmas - Coordinator midwife of PONED Puskesmas Pallangga Dr.Gaffar Head of Puskesmas - Coordinator midwife of PONED Bulukumba Bupati A.M Syukri Sappewali Head of District DHO Abd Gaffar, M.Epid Head of DHO Rukiyah Head Section of Family Health and Nutrition/ IBI member Dr. Hj.Wahyuni Ex- RSUD Director Kustigawati Sub-Head of Human Resources Zufri MKIA RSUD Sulthan Radja Daeng Dr. A. taufan Vice Director of Medical Service Dr. Amrullah IDI Head Nuraeni, SKM Post-Partum Midwife FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 73 PROVINCE DATE of INTERVIEW DISTRICT/ CITY INSTI-TUTION NAME OCCUPATION Astuti Ahmad Midwife Asmawati Perinatology BAPPEDA Agus Salim Ishak Head Sub of Social Culture Puskesmas Bontobangun Dr. Marwah Head of Puskesmas Yuliana, S.ST Coordinator of midwves 15 feb 2017 Makassar PHO Yani Runde Head Section of Family Health Dr.Nurdin Perdana Ex-EMAS Province Team Leader RSUP Dr.Wahidin Sudirohusodo (Vertical Hospital) Dr.dr.Khalid Saleh, SpPD Director Dr.Ema Head of IDAI South Sulawesi Dr. Umar, SpOG POGI member Mulyati Head of IBI South Sulawesi Prida Staff of IBI/ midwife North Sumatera 16 feb 2017 Deli Serdang DHO Nuraisyah IBI member/ Head of Referral Service Evi Barus Ketua Penyelia Fasilitatif EMAS Emy Herati Head Section of Family Health Syahrani Clinical Referral Taufik Hidayat Ex-EMAS of Deli Serdang Susi M New in EMAS Eldina FMM Mariani Sembiring MKIA Puskesmas Aras Kabu Nancy Herlina Midwife Coordinator 2016 Rohana Head of SP2TP and midwife Coordinator Kasinem Midwife Dr. Henny Emergency Unit and MCH Coordinator Puskesmas Galang Drg. Rita Head of Puskesmas Midwife Coordinator midwife of PONED RSUD Deli Serdang Herna julianty Head of Delivery Room Sylvi Head of Caring Room Delyunmas Ziraluo KAsi pelayanan medik Rosmary Sembiring Kasi Penunjang Medis Marthalena Staff Ruang VK Midaretha Head of Neonatal Room Dewi Head of NICU RS Grand Medistra - Director - Head sub of Medical Service - Midwife Medan PHO/TEAM 21 Dr.Aswan TEAM 21 North Sumatera Rosidah Berutu TEAM 21 North FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 74 PROVINCE DATE of INTERVIEW DISTRICT/ CITY INSTI-TUTION NAME OCCUPATION Sumatera Rusdin Pinem TEAM 21 North Sumatera Arjuna TEAM 21 North Sumatera 17feb 2017 Serdang Bedagai BUPATI Ir.H. Soekirman Bupati (Head of District) Darma Wijaya, SE Vice Bupati BAPPEDA Ronny Gunawan Sub-Head of Social & Culture Romli Simbolon Head Sub￾Directorate Social &culture- Serdang Bedagai DHO Dr. Helmy Noor Sinaga Secretary of DHO Wahdini Head Section of Family Health Sri Herawati MCH Staff MEDAN PHO Agustama Head of PHO North Sumatera Fatni Sulani EMAS Province Team Leader (PTL) North Sumatera RSUP Adam Malik Dr. Iman Helmi, SpOG POGI member/ mentor of vertical Hospital Idau Ginting Head of IBI North Sumatera Dr. Bugis Mardina Lubis, SpA IDAI member/ Mentor of Vertical hospital Prof.Munar Lubis Head of IDAI North Sumatera Prof. Delfi Lutan,SpOG Head of POGI North Sumatera Dr.Khairani Sukatendel (HOGSI) Head of HOGSI North Sumatera DKI Jakarta 21 Feb 2017 Jakarta EMAS Broto Advisor of EMAS Banten Tangerang DHO Dr. Manik Kusmayoni Head Sub of Public Health Dr. Sri Indriyani Head Section of Family Health Deni Septiawan FMM (FOPKIA) Joehairudin FMM (FOPKIA) Wiwik MKIA BAPPEDA Cucu Heri Sub-Head Sosial Culture Muh. Nabawi Bappeda Staff Kirwan Bappeda Staff Agus Husen S Bappeda Staff Ahmad Hifni Bappeda Staff Desi Riani Dwi P Bappeda Staff RSU Kab.Tangerang Dr. Sherly Deputy Director of Medical Service - Obgyn - Pediatrician FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 75 PROVINCE DATE of INTERVIEW DISTRICT/ CITY INSTI-TUTION NAME OCCUPATION - Midwives DKI Jakarta 24 Feb 2017 Jakarta BAPPENAS Entos Zainal Head Section of Empowerment and Public Health Nutrition II. KEY DOCUMENTS REVIEWED # Topic Description Contributors Timeline 1 EMAS Maternal Death Report Maternal Mortality: A Retrospective Chart – Review of 112 Deaths in Twelve Hospitals in Indonesia POGI supported by USAID & EMAS January – June 2015 2 EMAS Baseline Report • 2015 Cross-sectional data collection for Phase 1 & 2 Districts • 2015 Baseline data collection for Phase 3 & Comparison Districts EMAS & USAID March 2016 3 EMAS Baseline Report Baseline Report; Annexes EMAS & USAID March 2016 4 EMAS Expansion and Replication Report Expansion and Replication of EMAS Programme Approaches EMAS & USAID 2016 5 Annual Reports Year 4 & Year 5 Annual Reports EMAS & USAID October 2014 – September 2016 6 WHO Monitoring Quality of Care Monitoring Emergency Obstetric Care – a handbook WHO 2009 7 Interfaith Network Muhammadiyah’s experience connecting Faith-Based Organizations to Improve Maternal and Newborn Care and Advocacy EMAS & USAID 2015 8 Technical report - Data System Programme Strengthening Facility – Based Measurement of Maternal and Newborn Health EMAS & USAID October 2015 9 Indonesia Maternal Health Assessment And Then She Died World Bank Supported by MoH, BAPPENAS, DFID Februari 2010 10 Program Brief Retrospective Review of Cases and Factors Associated with 76 Neonatal Deaths in six EMAS￾supported Hospitals IDAI & EMAS January 2016 FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 76 11 Program Brief Retrospective Review of Cases and Factors Associated with 112 Maternal Deaths in 12 Hospitals POGI & EMAS May 2015 12 Technical Report - Clinical Mentoring Connecting providers and Facilities to Improve Maternal and Newborn Care In Indonesia EMAS & USAID October 2015 13 Technical Report – Civic Forums Fostering Civil Society Engagement to Increase Maternal & Newborn Survival in Indonesia EMAS & USAID May 2016 14 Technical Report – POKJA Improving Collaboration To Increase Maternal & Newborn Survival : District- Level Inter sectoral Working Groups In Indonesia EMAS & USAID May 2016 15 Technical Report – Referral System Strengthening Referral System for Maternal and Neonatal Survival: Connecting Facilities To Improve Emergency Care EMAS & USAID January 2016 16 Technical Report – SijariEMAS SijariEMAS Referral Exchange System: Improving the Referral System For Maternal and Newborn Emergencies in Indonesia EMAS & USAID July 2015 17 National Academy of Science 2013 Reducing Maternal and Neonatal Mortality in Indonesia: Saving Lives, Saving the Future AIPI (Indonesian Academy of Sciences) 2013 18 Clinical governance case study Description of EMAS supported activity to strengthen clinical governance EMAS Published in 2015, World Health Population FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 77 19 Maternal death audit What are the characteristics of the women with a maternal death in the 12 hospitals in Indonesia during the study period? What is the primary cause (i.e., obstetric, non-obstetric) and the final cause of Maternal death in 12 hospitals in Indonesia during the study period? What are the contextual factors associated with maternal death (From the perspective of 3 delays)? LKBK and POGI supported by EMAS; Led by Stephanie Suhowatsky & Lilly Amelia Expected completion in March 2017 20 Newborn death audit What are the characteristics of newborn deaths reviewed through the newborn death audit conducted by EMAS? IDAI & EMAS; led by Anne Hyre Expected completion in March 2017 21 MNH HIS strengthening activities How did the EMAS program improve the ability to examine relevant MNH measures over time including the coverage of evidence-based practices and mortality rates in Indonesian’s hospitals? EMAS Led by Maya Submitted and rejected by WHO Bulletin in 2016; resubmission planned for March 2017 22 Evaluation study of EMAS interventions on referrals for obstetric complications from public health centers to hospitals in 6 provinces in Indonesia Findings from the quasi￾experimental program evaluation of the EMAS Program *communication *referral timeliness *facility emergency readiness Led by EMAS Eval Team Analysis underway – Target: March-April completion 23 Hospital Quality of Care for Emergency Complications: PPH and newborn resuscitation *Examine complications which occur during the delivery process in terms of treatment to standard (PPH and newborn resuscitation) Led by EMAS Eval Team Analysis underway – Target: March-April completion FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 78 Annex IV: Details of Data Constraints Quarterly monitoring data on the following were received from EMAS M&E team: 1. Number of women delivering 2. Number of deliveries at 24-36 weeks 3. Number of deliveries at 24-36 weeks that receive dexametazone 4. Number of maternal referral cases 5. Number of newborn referral cases 6. Number of women who receive a uterotonic in AM3TL 7. Number of live births 8. Number of live births > 2000 grams 9. Total number of maternal deaths 10. Total number of maternal deaths reviewed 11. Total number of maternal deaths reviewed within 24 hours of occurrence 12. Number of severe PE/E cases that receive MgSO4 13. Number of referred PEB/E cases that receive MgSO4 before referral 14. Number of live births with IMD started within 1 hour after delivery 15. Number of IUFD 16. Number of intrapartum deaths 17. Number of intrapartum deaths > 2000 grams 18. Number of intrapartum deaths > 2000 grams that are reviewed 19. Newborn Sepsis- Referred- Case 20. Total number of newborn deaths age 0-28 days 21. Number of newborn deaths > 2000 grams 22. Number of neonatal deaths > 2000 grams that are reviewed 23. Number of very early neonatal deaths 24. Number of very early neonatal deaths that are reviewed 25. Number of early neonatal deaths age 0-7 days 26. Number of late neonatal deaths age 8-28 days 27. Number of referred newborns who received antibiotic before referral 28. Number of newborns severe sepsis who received antibiotic before referral 29. Number of near miss cases 30. Number of near miss cases reviewed 31. APH Case 32. PPH Case 33. Infection/Sepsis Case 34. PE Moderate Case 35. PEB/E Case 36. Obstructed Labor Case 37. Ruptured Membrane Case 38. Others Maternal Complications Case 39. Total Maternal Complication Case 40. Maternal deaths associated with PEB/E 41. Maternal death associated infection/Sepsis 42. Maternal death associated PPH 43. Maternal death associated APH 44. Maternal death associated Others 45. PEB/E referred -Case 46. Number of Newborns with Asphyxia 47. Number of Newborns with Low Birth Weight 48. Number of Newborns with Sepsis 49. Number of newborn with "Others" complication 50. Number of Asphyxia associated with deaths 51. Number of BBLR associated with death 52. Number of Newborn sepsis associated with death FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 79 53. Number of Acute respiratory associated with death 54. Number of Newborn "Others" associated with death 55. Number of PEB/E referred 56. Total Neonatal Complication 57. Number of Still births (IUFD + Intrapartum Deaths) 58. Number of death associated with early membrane rupture 59. Number of death associated with obstructed labor 60. Number of death associated with Moderate Eclampsia 61. 123_Number of Newborns with Acute Respiratory Infection 62. PPH- Referred- Case 63. Infection/sepsis maternal- Referred- Case Initially, the above list of data were not provided in workable and transferable format for statistical software; later, the EMAS team sent the data again using the provided template. The available quarterly data varies between facilities. Some have until Yr5 Q4, or Yr5 Q3, or Yr5 Q2, or Yr5 Q1; some only have until Yr4 due to graduation or variability of close-out time, including variability of starting time by phases and even within the same phase. The referral data needed from Sijari EMAS were requested as follows: 1. % maternal referral cases originated from EMAS supported facilities 2. % newborn referral cases originated from EMAS supported facilities 3. % of PEB/E cases originated from EMAS supported facilities that receive MgSO4 before referral 4. % of referred newborns originated from EMAS supported facilities who received antibiotic before referral 5. % of newborns severe sepsis originated from EMAS supported facilities who received antibiotic before referral 6. % newborn Sepsis Referred Case originated from EMAS supported facilities However, the data received were (all of which are hospital data that originated from EMAS or non-EMAS facilities): 1. Number of maternal referral cases 2. Number of newborn referral cases 3. PEB/E referred case 4. Total Maternal Referred Case 5. Sepsis referred cases 6. Total Neonatal Complication Referred Case 7. Number of referred PEB/E cases that receive MgSO4 before referral 8. Number of PEB/E referred The data calculated from the above list, assuming that (for example) all PEB/E referred cases that receive MgSO4 before referral were originated from EMAS facilities (which is not the case), or Number of newborns suspected of having severe sepsis who received antibiotic before referral are all originated from EMAS facilities (which is also not the case, given the total approach of referral system that includes all facilities in the respective district.) Below is the list of data received that were not from the SijariEMAS database: a. % of PEB/E cases originated from EMAS supported facilities that receive MgSO4 before referral (V_56 / V_131) b. % of referred newborns originated from EMAS supported facilities who received antibiotic before referral (V_100 / V_07) c. % of newborns sepsis originated from EMAS supported facilities who received antibiotic before referral (V_100 / V_71) d. % newborn Sepsis Referred Case originated from EMAS supported facilities (V_71 / V_83 --> sepsis referred case / neonatal complication) FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 80 Below are the codebook that shows V_100 or V_71 refers to the same data in the list of quarterly monitoring data: Var_01 1_Number of women delivering Var_04 4_Number of delivering at aged 24-36 weeks Var_05 5_Number of deliveries aged 24-36 weeks that receive dexametazone Var_06 6_Number of maternal referral cases Var_07 7_Number of newborn referral cases Var_08 8_Number of women who receive a uterotonic in AM3TL Var_09 9_Number of live births Var_11 11_Number of live births more 2000 gram Var_52 52_Total number of maternal deaths Var_53 53_Total number of maternal deaths reviewed Var_54 54_Total number of maternal deaths reviewed within 24 hours of occurrence Var_55 55_Number of severe PE/E cases that receive MgSO4 Var_56 56_Number of referred PEB/E cases that receive MgSO4 before referral Var_58 58_Number of live births with IMD started within 1 hour after delivery Var_59 59_Number of IUFD Var_60 60_Number of intrapartum deaths Var_61 61_Number of intrapartum deaths more 2000 gram Var_62 62_Number of intrapartum deaths more 2000 that are reviewed Var_71 71_Sepsis- Referred- Case Var_87 87_Total number of newborn deaths age 0-28 days Var_91 91_Number of newborn deaths more 2000 g Var_92 92_Number of neonatal deaths more 2.000 g that are reviewed Var_94 94_Number of very early neonatal deaths Var_95 95_Number of very early neonatal deaths that are reviewed Var_96 96_Number of early neonatal deaths age 0-7 days Var_97 97_Number of late neonatal deaths age 8-28 days Var_99 99_Number of referred newborns who received antibiotic before referral Var_100 100_Number of newborns severe sepsis who received antibiotic before referral Var_101 101_Number of near miss cases Var_102 102_Number of near miss cases reviewed Var_105 105_APH Case Var_106 106_PPH Case Var_107 107_Infection/Sepsis case FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 81 Var_108 108_PE Moderate case Var_109 109_PEB/E case Var_110 110_Obtruct Labor case Var_111 111_Rupture Mambran case Var_112 112_Others Maternal complication case Var_113 113_Total Maternal Complication case Var_114 114_Maternal deaths associated with PEB/E Var_115 115_Maternal death associated infection/Sepsis Var_116 116_Maternal death associated PPH Var_117 117_Maternal death associated APH Var_118 118_Maternal death associated Others Var_119 119_PEB/E referred case Var_120 120_Number of Newborns with Asphyxia Var_121 121_Number of Newborns with Low Birth Weight Var_122 122_Number of Newborns with Sepsis Var_123 123_Number of Newborns with Acute Respiratory Infection Var_124 124_Number of newborn with "Others" complication Var_125 125_Number of Asphyxia associated with deaths Var_126 126_Number of BBLR associated with death Var_127 127_Number of Newborn sepsis associated with death Var_128 128_Number of Acute respiratory associated with death Var_129 129_Number of Newborn "Others" associated with death Var_131 131_Number of PEB/E referred Var_133 133 Total Neonatal Complication Var_134 134 Number of Still births (IUFD + Intrapartum Deaths) Var_141 141_Number of death associated with early membrane rupture Var_142 142_Number of death associated with obstruct labor Var_143 143_Number of death associated with Moderate Eclampsia There are 3 facilities below that were not included in the Clinical Performance Assessment data set, and therefore cannot be merged with the database developed by the evaluation team: South Sulawesi Bulukumba 2 Klinik Bersalin Daffiku Central Java Grobogan 3 Kradenan II Central Java Kota Pekalongan 3 RSUD Kraton In the annual data set under the file Service Status Facility Name 23Feb2017.xlsx, the following issues were identified: FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 82 • In the file ServiceStatusFacilityName23Feb2017.xlsx, there are no Deli Serdang district, Sidoarjo district and hospital in Tegal, in the annual data set under tab Maternal complication death, Newborn complication death, near miss cases and Maternal very early newborn death, different with their quarterly data set. • There are hospitals with incorrect district names: 3 in Deli Serdang, 3 in Sidoarjo, 3 in Tegal and RSUD Waled in Cirebon (in annual data set it's in Bandung) • There are 2 entries with similar facility name (RSUD Kraton) that have different data. As result, the team is uncertain which facility the data belongs to. The evaluation team decided to do the analysis using quarterly data sets, and found that obstruct labor cases have similar data with maternal death associated with obstruct labor as if all cases are fatal (100%). But this seems to be an error in entering the data into the template, which was then revised in the last data set received on 27 March 2017. The team also identified the referral cases for maternal was only severe PE/E, and then asked the EMAS M&E team whether they have the data on referral for PPH and Sepsis. They indicated that do have the data but only share it if requested to do so. This approach seemed consistent with the repeated requests that the evaluation team, MESP, and USAID had to make to Jhpiego in order to receive the full sets of requested data for evaluation. Below is the list of other data set received but with different format and arrangements, leading to the evaluation team having to identify and code the facilities in order to merge the data into the developed database: • Number of Maternal deaths reported • Number of Maternal deaths audited using AMP process • Total number of neonatal deaths reported during to DHO (0 to 28 days) • Number of early neonatal (0 to 7 days) deaths reported to DHO • Number of late neonatal (8 to 28 days) deaths reported to DHO • Total number of neonatal deaths audited during neonatal period (0 to 28 days) • Number of early neonatal (0 to 7 days) deaths audited using the MPA process • Number of late neonatal (8 to 27 days) deaths audited using the MPA process • Referral database that shows the score of referral standard achievement • Percentage of EMAS-supported hospital referral cases managed using SijariEMAS (different data structure) • Percentage of referral cases with a hospital response occurring within 10 minutes upon receipt of SijariEMAS notification (different data structure) FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 83 Annex V: Results Against Targets – EMAS Reported and Evaluation Team Re-analysis EMAS REPORTED RESULTS: YEARS 3 - 5 PMP # Performance Indicator YR 3 Targe t YR 3 Result s YR 4 Targe t YR 4 Result s YR 5 Targe t YR 5 Result s 2 % EMAS- supported facilities that achieved 80% EmONC standards Phase 1 40% ?? 60% ?? 80% ?? Phase 2 20% ?? 60% ?? 80% ?? Phase 3 NA NA 30% ?? 60% ?? 3 % severe pre- eclampsia/eclampsia cases managed with MgSO4 at EMAS- supported facilities 90% 0.88 95% 100% Phase 1 90% 88% 95% 96% 100% 98% Phase 2 90% 84% 95% 93% 100% 98% Phase 3 90% NA 95% 87% 100% 91% 4 % deliveries that receive 1+ dose of uterotonic in third stage labor at EMAS- supported facilities 90% 95% 100% Phase 1 90% 97% 95% 100% 100% 100% Phase 2 90% 95% 95% 99% 100% 100% Phase 3 90% NA 95% 96% 100% 100% 6 % deliveries 24 to 36 weeks in EMAS￾supported hospitals who receive 1+ corticosteroid doses 50% 70% 80% Phase 1 50% 70% 70% 80% 80% 89% Phase 2 50% 65% 70% 80% 80% 89% Phase 3 50% NA 70% 59% 80% 76% 7 % EMAS-supported facilities that conduct regularly scheduled death reviews of fresh stillbirths (intrapartum deaths) > 2000 grams 15% 40% 50% Phase 1 15% 78% 40% 46% 50% 64% Phase 2 15% 42% 40% 70% 50% 75% Phase 3 15% NA 40% 36% 50% 58% 8 % EMAS-supported facilities conduct regularly scheduled reviews -neonatal death > 2000 grams 15% 40% 50% Phase 1 15% 79% 40% 85% 50% 79% Phase 2 15% 66% 40% 91% 50% 89% Phase 3 15% NA 40% 22% 50% 74% 9 % EMAS-supported facilities conduct 100% maternal death reviews in 24 hours of occurrence 25% 40% 100% Phase 1 25% 36% 40% 64% 100% 69% Phase 2 25% 59% 40% 86% 100% 97% Phase 3 25% NA 40% 26% 100% 77% 10 % EMAS-supported hospitals conduct regularly scheduled near miss reviews 15% 50% 60% Phase 1 15% 59% 50% 93% 60% 85% Phase 2 15% 23% 50% 97% 60% 89% FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 84 PMP # Performance Indicator YR 3 Targe t YR 3 Result s YR 4 Targe t YR 4 Result s YR 5 Targe t YR 5 Result s Phase 3 15% NA 50% 53% 60% 89% 11 Obstetric case fatality rate (hospital) 1.2% 1.1% < 1.0% Phase 1 1.2% 2.7% 1.1% 2.4% < 1.0% 2.6% Phase 2 1.2% 2.0% 1.1% 1.8% < 1.0% 1.8% Phase 3 1.2% NA 1.1% NA < 1.0% 1,4% 12 Newborn mortality rate (facility) Annual targets will be developed in year 3 when mortality information will be more reliably recorded and collected 13 Fresh stillbirth and very early neonatal death rate (hospital) Annual targets will be developed in year 3 when mortality information will be more reliably recorded and collected 17 % women with severe PE/E referred to EMAS￾supported hospitals that receive at least one dose of MgSO4 before referral 40% 65% 75% Phase 1 40% 30% 65% 52% 75% 68% Phase 2 40% 19% 65% 40% 75% 60% Phase 3 40% NA 65% 29% 53% 18 % newborns with suspected severe infection referred to EMAS-supported hospitals that receive at least 1+ antibiotic dose before referral 30% 40% 50% Phase 1 30% 14% 40% 20% 50% 22% Phase 2 30% 14% 40% 15% 50% 22% Phase 3 30% NA 40% 22% 50% 23% FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 85 EVALUATION TEAM RE-ANALYSIS OF RESULTS: YEARS 3 - 5 # Perf. Indicator Year 3 Year 4 Year 5 # Perf. Indicator Year 3 Year 4 Year 5 # % N D % N D % N D 2 % EMAS facilities achieved 80% EmONC standards Phase 1 (1) hospital maternal standard 77% 17 22 100% 22 22 100% 17 17 (2) hospital newborn 50% 11 22 77% 17 22 82% 14 17 (3) hospital IP 36% 8 22 96% 21 22 100% 17 17 (4) hospital clinical governance 36% 8 22 64% 14 22 88% 15 17 (5) puskesmas MNH 70% 63 90 89% 81 91 93% 70 75 (6) puskesmas IP 14% 13 90 91% 83 91 97% 71 73 Phase 2 (1) hospital maternal standard 51% 22 43 92% 43 47 95% 40 42 (2) hospital newborn 21% 9 43 75% 35 47 95% 40 42 (3) hospital IP 9% 4 43 83% 39 47 93% 39 42 (4) hospital clinical governance 21% 9 43 62% 29 47 88% 37 42 (5) puskesmas MNH 30% 35 117 75% 91 122 97% 111 115 FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 86 # Perf. Indicator Year 3 Year 4 Year 5 # Perf. Indicator Year 3 Year 4 Year 5 # % N D % N D % N D (6) puskesmas IP 7% 8 117 66% 80 122 92% 104 113 Phase 3 (1) hospital maternal standard NA 7% 2 30 60% 40 67 (2) hospital newborn NA 3% 1 30 51% 34 67 (3) hospital IP NA 13% 4 30 63% 42 67 (4) hospital clinical governance NA 3% 1 30 45% 30 67 (5) puskesmas MNH NA 11% 8 71 81% 58 72 (6) puskesmas IP NA 11% 8 71 68% 49 72 3 % severe pre￾eclampsia/ eclampsia cases managed with MgSO4 at EMAS facilities (hospitals & puskesmas) Phase 1 89% 4036 4554 96% 4714 4903 98% 2662 2722 Phase 2 87% 4860 5608 93% 9624 10312 98% 8194 8388 Phase 3 NA 88% 1515 1714 91% 8114 8916 FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 87 # Perf. Indicator Year 3 Year 4 Year 5 # Perf. Indicator Year 3 Year 4 Year 5 # % N D % N D % N D 4 % deliveries that receive 1+ dose of uterotonic in third stage labor at EMAS facilities (hospitals & puskesmas) Phase 1 97% 47480 48912 100% 53233 53421 100% 2171 3 2172 7 Phase 2 97% 53287 54928 99% 93150 93640 100% 6367 8 6393 0 Phase 3 NA 97% 18316 18860 100% 9509 4 9548 9 6 % deliveries 24 to 36 weeks in EMAS hospitals who receive 1+ corticoster oid doses (hospitals only) Phase 1 70% 1988 2827 80% 1528 1906 89% 733 819 Phase 2 68% 2633 3869 80% 4009 4995 89% 2900 3275 Phase 3 NA 59% 400 680 76% 2477 3249 7 % EMAS facilities conduct regularly scheduled death reviews, fresh stillbirths & intrapartu m deaths >2000 gms FE Team unable to calculate this indicator from the data set without clear definition of "regularly scheduled reviews" FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 88 # Perf. Indicator Year 3 Year 4 Year 5 # Perf. Indicator Year 3 Year 4 Year 5 # % N D % N D % N D 8 % EMAS facilities conduct regularly scheduled reviews - neonatal death > 2000 grams FE Team unable to calculate this indicator from the data set without clear definition of "regularly scheduled reviews" 9 % EMAS facilities conduct 100% maternal death reviews <24 hours (hospitals & puskesmas) Phase 1 0% 0 23 32% 8 25 42% 5 12 Phase 2 19% 7 36 40% 14 35 57% 20 35 Phase 3 NA 26% 5 19 48% 21 44 1 0 % EMAS hospitals conduct regularly scheduled near miss reviews FE Team unable to calculate this indicator from the data set without clear definition of "regularly scheduled reviews" 1 1 Obstetric case fatality rate (hospital) - Category C % maternal death associated with PPH, Severe PE/E and Sepsis Phase 1 2.7% 114 4280 2.4% 107 4549 2.6% 62 2422 Phase 2 2.0% 119 5972 1.8% 186 10198 1.8% 138 7838 Phase 3 NA 1.2% 21 1760 1.4% 130 9423 1 2 Newborn mortality rate (hospital) FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 89 # Perf. Indicator Year 3 Year 4 Year 5 # Perf. Indicator Year 3 Year 4 Year 5 # % N D % N D % N D Phase 1 37 1339 35958 37 1374 37576 37 545 1479 4 Phase 2 43 1786 41634 37 2509 68514 31 1399 4532 5 Phase 3 NA 26 370 14022 26 2077 8062 3 1 3 Fresh stillbirth and very early neonatal death rate (hospital) The number of fresh stillbirths (intrapartum deaths) and very early neonatal deaths per 1000 live births Phase 1 7 253 34048 6 221 35575 7 98 1389 2 Phase 2 8 296 39133 5 349 64805 4 173 4280 9 Phase 3 NA 6 82 13426 4 322 7753 2 1 7 % women with severe PE/E referred to EMAS hospitals that receive one+ dose of MgSO4 pre￾referral (hospital) The results may not be comparable to EMAS annual report Year 5 since there is no identifier for facilities that receive “full support” or “limited support”. Phase 1 30% 948 3115 52% 1587 3029 69% 1166 1702 Phase 2 20% 738 3607 39% 2317 5973 59% 2969 5074 Phase 3 NA 32% 381 1182 46% 2657 5793 FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 90 # Perf. Indicator Year 3 Year 4 Year 5 # Perf. Indicator Year 3 Year 4 Year 5 # % N D % N D % N D 1 8 % newborns with suspected severe infection referred to EMAS hospitals that receive 1+ antibiotic dose pre￾referral (hospital) The results may not be comparable to EMAS annual report Year 5 since there is no identifier for facilities that receive "full support" or "limited support" Phase 1 10% 225 2296 13% 406 3228 14% 178 1280 Phase 2 8% 334 4272 8% 365 4301 7% 193 2577 Phase 3 NA 7% 50 734 7% 301 4218 FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 91 Annex VI: Review of EMAS PMP Indicators OBJECTIVE 1: HIGH-IMPACT, LIFE-SAVING CLINICAL INTERVENTIONS FACILITY LEVEL MEASURES 1. Vanguard facilities: • Activity level indicator, of little no use beyond this. • The definition of Vanguards was reduced to equal that of mentors. • No measures of the QOC and outcomes achieved in mentor sites – if these were poor, mentors can't reasonably be expected to mentor and improve QOC in other sites. • Criteria is activity level, and not as a measure of the QOC e.g. case fatality <1%, PPH incidence reduced through compliance with active management of third stage of labor (provision of uterotonic during 3rd stage, not waiting for PPH to treat). • No dashboards, death audit compliance, or feedback mechanisms required for mentors. • Dashboards displayed and “current” is equally meaningless – measures of achievements in compliance – at 100% and not just 80%, would be a better measure of vanguard. 2. % facilities who achieve 80% compliance with EmONC standards: • Results of this indicator are self-reported. • It remains unclear if the project measures % hospitals who meet criteria for clinical standards or % who follow SOPs to managed cases. • When cross-check between indicators, it is difficult to reconcile how e.g. 89% of puskesmas in phase 1 have achieved MNH standards if only 59% of women with eclampsia are stabilized with mg so4 and 12% of newborns receive antibiotics. The only rationale for this is that achievement of a standard is based on the presence of critical equipment and medicines, not their application or use. • Targets set at 80% SOP compliance - reported as “benchmark globally used [sic] by quality improvement programs (supported by jhpiego)”; yet no evidence that 80% compliance has statistically significant impact on outputs and outcomes. • IP is of limited relevance and value in lowering maternal neonatal mortality; the team noted fewer than half of bathrooms visited had soap. Though an ineffective indicator for MNH, it is an important aspect of QOC facility-wide and based on this this and not statistically significant sample set, over 50% of facilities visited by this team failed. MATERNAL INTERVENTION MEASURES 3. % of eclampsia cases treated with mag so4: 4. % of deliveries treated with uterotonic in 3rd stage: • Both are good activity measures. Yet without a more robust analysis of how results of activities link to co-morbidities and correlate with mid-level outcomes, it is just an activity. • Treatment with MGSO4 should decrease total deaths and near miss cases for PE/E. • Uterotonic in third stage labor should decrease PPH cases and deaths though the drop is likely less dramatic.14 • Provision of MGSO4 and uterotonics at puskesmas should increase the total deliveries managed without need for referral. 14 Reduced caseload of PPH is likely to be contaminated by unsafe abortions – which may not be recorded as such. FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 92 NEWBORN INTERVENTION MEASURES 5. % of live births breast-fed within one hour: 6. % of deliveries between 24-34 weeks who receive 1+ antenatal steroids: • The measure was limited to hospitals, but should include puskesmas and private facilities. • Unknown if puskesmas conduct these currently, though they should. MATERNAL PERINATAL DEATH REVIEWS 7-10. % death newborn and maternal audits & near miss reviews: • Conducting audits is an activity, and more in line with needs of JHPIEGO to present results against targets: • Measuring use of audits to improve QOC- is a better outcome measure, and should include measures at both referral hospitals and from referring bodies; • Audit reviews should include health staff from all levels to increase ownership and responsibility amongst all individuals across various cadres of health staff and not simply point blame elsewhere. • Definition for near miss should be done according to WHO standards, and not only for purposes of reporting targets achieved. • Cases selected for review can’t be based solely on prevalence or those included in project interventions – e.g. “other” accounts for 1/3 of deaths but was never included in the facilities visited. Less common categories may also never be reviewed. MATERNAL NEWBORN DEATH RATES 11. Obstetric case fatality rates: • WHO definition for case fatality defines 7 categories as: hemorrhage, hypertensive diseases, abortion, sepsis/infections, obstructed labor, ectopic pregnancy, embolism, anesthesia-related death. • Definition category B also: defined on PPH, PE/E, sepsis and “EMAS other”: APH, ruptured membrane, moderate E, obstructed labor- which account for the least number and % of deaths, but have a huge denominator, which skews the ratio. • Conversely, they omit the “other” undefined category, which accounts for 1/3 of all deaths, and a higher proportion of deaths to the denominator, resulting in misreporting and/or incorrect analysis of MMR, leading to inability to recognize and appropriately address all causes of mortality. • Indicator definition sheet indicates that the program should report on category A, but they report on B. Furthermore, they report on arbitrary breakout between A and B, excluding critical causes of mortality and including categories that are neither leading causes nor WHO approved. 12-13. Newborn & Fresh stillbirth & very early neonatal death (>2000) mortality rates: • Hospital-based only – why exclude puskesmas, community, etc.? • Did not include referral cases – why? Its critical as otherwise this rate is meaningless? This definition chooses to exclude complicated cases, which skews the number. OBJECTIVE 2: MORE EFFECTIVE AND EFFICIENT REFERRAL SYSTEMS FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 93 ✓ Effectiveness: is defined as increased ability in reaching the correct facility, correct stabilization en route, improved communications between referee and referral, hospital preparedness. ✓ Efficiency: Faster identification of referral facility, reduced delays in transport times, reduced delays in identification of complications & decision to refer and faster initiation of treatment. REFERRAL STANDARDS & SIJARI MANAGED REFERRALS 14. % of referral standards achieved by EMAS supported networks: 15. % of EMAS supported hospitals referrals managed using sijariEMAS: ✓ Similar to clinical standards, targets set at 60-80% facilities achieve 80% compliance, making the %achieved further reduced. 16. % of referral cases with hospital response within ten minutes of sijariEMAS notification: ✓ Ten minutes was widely criticized as too long. ✓ This is the only timeline measured, no other measures to assess reduce delays resulting from the new system were included. ✓ Reported total referral cases responded to in under ten minutes was at times a higher number than the total referrals managed through sijari and the total number of referrals. PRE-REFERRAL STABILIZATION RATES 17. Pre-referral stabilization with MgSO4: 18. Pre-referral stabilization of newborns with antibiotics: ✓ This is fine as a measure, but correlation with outcomes is more robust. Also measuring reductions in the denominator of cases. ✓ Midwives are not legally allowed to provide this without a doctor’s consent, so it's an ineffective indicator and sets the project up to fail. DISTRICT LEVEL MATERNAL PERINATAL AUDITS 19. % of districts conducting MPA process: ✓ As above, district level audits are not done only for reporting and achievement of targets. Measures of changes resulting from the use of findings from these should be the objective. ✓ WHO terminology and processes, including confidential inquiries, should be adhered to. OBJECTIVE 3: IMPROVED GOVERNANCE (POKJAS AND CIVIC FORUMS) 20-22. % of districts implementing citizen feedback, with Vanguard POKJAS and civic forums ✓ Measures of activities only, little no correlation with improved health outcomes. ✓ Citizen feedback activities were dropped in recognition of their limited use. FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 94 Annex VII: International Standards for QOC Indicators WHO identified 11 key indicators for maternal, newborn and child health: ✓ Maternal mortality ratio and rate ✓ Newborn mortality rate ✓ % of demand for family planning satisfied (met need for contraception) ✓ Antenatal care coverage (at least four times during pregnancy) ✓ Skilled attendant at birth ✓ Postnatal care for mothers and babies within two days of birth ✓ Exclusive breastfeeding for six months (0–5 months) ✓ Antibiotic treatment for suspected pneumonia/infection, newborns WHO reports and policies on MN and QOC indicators include: 1. “WHO: Standards for QOC of M&N Care in Facilities”, 2016. 2. “WHO: Facility Level QOC Measures for MNC”, 2013. 3. “WHO: Evaluating Quality of Care of Severe Pregnancy Complications”, 2011. 4. “WHO: Monitoring Quality of Care”, 2011. 5. “Every Woman, Every Child: Indicator and Monitoring Framework for Women’s and Children’s Health: 2016-2030”, 2016. www.everywomaneverychild.org/hub/ Examples of other key maternal quality of care indicators for future projects: ✓ Manual Removal of Placenta Rate ✓ Correlation between Manual Removal of Placenta and PPH ✓ PPH Rate ✓ Rate of Conformance to Management of PPH, PE/E, Sepsis, per National Standards ✓ Rate of Conformance to National Standards for Assessing/Recording Patient Clinical Status – antepartum, intrapartum, postpartum ✓ Number of Units Blood per Case of PPH And Time of Transfusion From Time of Diagnosis ✓ Rate of Ruptured Uterus by Contributing Cause – Prior C/S; No Prior C/S ✓ Late Umbilical Cord Clamping Rate ✓ Birth Trauma Rate – Injury to Neonate ✓ Obstetric Trauma Rate – vaginal delivery with instrument ✓ Obstetric Trauma Rate – vaginal delivery without instrument ✓ Transfusion reaction count ✓ Meconium Staining Rate ✓ Frequency of Fetal Heart Rate Checks During Active Labor ✓ Average time from “Falling off Partograph Curve” to augmentation of labor or C/S ✓ Percent Receiving Prophylactic Antibiotics for Prolonged Rupture of Membranes ✓ Percent Receiving Prophylactic Antibiotics before C/S (correct antibiotic/dose/timing) Examples of other key newborn quality of care indicators for future projects: ✓ Neonatal Mortality Rate – by gestation, weight, APGAR score, and age in hours/days ✓ Neonates That Become Hypothermic During Hospital Stay Rate ✓ Mortality Rate of Babies Resuscitated ✓ Neonatal Blood Stream Infection Rate ✓ Accidental Puncture or Laceration Rate ✓ Pressure Ulcer Rate ✓ Sepsis Rate ✓ Transfusion Reaction Count International standards & toolkits at: www.qualityindicators.ahrq.gov/modules/iqi_overview.aspx FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 95 Annex VIII: Facility-based Quality of Care A. WHO Guidance and Additional Approaches in Establishing Key Baselines At the district level, the next project could establish district-wide baselines and conduct other analyses to identify resources as well as gaps as they relate to the health care system’s provision of care and the patient/community’s utilization of care. WHO’s Monitoring Obstetric Case offers one methodology of assessment. Indicators it suggests assessing include availability and geographic distribution of EmOC facilities; proportion of births in EmOC facilities; met need for EmOC facilities; C/S a proportion of all births; direct obstetric case fatality rate; intrapartum and very early neonatal death rates; proportion of deaths due to indirect causes in EmOC facilities. The assessment determines degree to which acceptable levels of each key indicator has been achieved. This assessment can be used in conjunction with, WHO’s Evaluating the Quality of Care for Severe Pregnancy Complications, the WHO Near-Miss Approach for Maternal Health. B. Death & Near-miss Audits, Confidential Inquiries & Other Quality Assurance Mechanisms The purpose of audits is to learn from complications and errors, to modify behavior and judgment based on previous experiences, and to prevent repetition of errors leading to complications. Death and near￾miss audits are important for identifying systems issues (e.g., outdated policies, changes in patient identification procedures, arithmetic errors, etc.), which affect patient care. Several KIs reported that near miss audits were less threatening than death audits – and therefore created a better opportunity to identify needed changes in facility processes or clinical management with the idea of implementing those changes. However, just because audits are held certainly does not automatically mean that problems will be adequately identified or that quality will improve. To be of true utility, those understandings must deep. Therefore, for those relatively new to the concept and process, initial and periodic training, as well as ongoing mentoring (when possible) is crucial. Some KIs did report specific changes in the layout of department services and other changes attributed to near miss audits. But KIIs revealed challenges: most KIs could not cite any changes in practices/policies that emerged as a result of audits; audits happened much less in private hospitals; their use trailed off over the course of the project; and audits occurred much less for neonatal cases. Worse, many KIs reported a central problem - stonewalling: “OBGYNs are not taking leadership.” “It is difficult to bread the wall of silence.” “There are no incentives to pull the cover off of the underlying problems.” Cases were often chosen for review that were unlikely to cast an unfavorable light on the hospital or specialist, e.g. referred cases in with mother arrived in critical condition. Understanding of the utility of audits, establishing a culture of open critique, holding each other accountable are mindsets and a culture not easily established. Given the Indonesian culture, described as having a tendency to shy away from open criticism, it may be slightly more challenging. It will take time and extensive mentoring. Physicians participating in audits need to better understand the philosophy/use/utility of audits. Also the learning curve might be accelerated by the establishment of various incentives/disincentives, monitoring by the administration of changes in practices/policies resulting from mentoring, strong encouragement and support from the administration, etc. Other questions about audits that will need attention in the following project include: Are the audits confidential? Is confidentially protected by law? Are the appropriate cadres attending? Is the name of patient (and possible identifiers) kept anonymous? Is the name of the physician (and other staff who managed the near-miss case) kept anonymous except to the person responsible for selecting and presenting the cases? Are the cases chosen with a view to identifying underlying problems? Are there a sufficient number of peers in attendance to form a critical mass for critique? Are audits held regularly? The medical records department can automatically pulls the medical records for all deaths and for near￾miss, all potentially problematic cases for that time period based – criteria for pulling charts for near-miss should be established. Examples of criteria for pulling are admission to the ICU, extended hospital stay, etc. A lead physician (for example Vice Chairman of the Department) should have the ongoing responsibility of looking over all pulled charts and selecting those in which he thinks there could be any question about quality/appropriateness of physician care or interesting learning points. He should present FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 96 the case. The name of the patient and the names of those who managed the case should not be identified. The attendees should discuss the case and decide if there was/was not a failure to meet standards of care (as defined by such things as - position papers of the professional associations; papers with high level of evidence, published in peer-reviewed journals; etc.). Recommendations should be generated – as appropriate, and these should be followed up by a designated party. For additional insights, refer to articles/sites such as Transforming the Morbidity and Mortality Conference into an Instrument for System wide Improvement that share experiences of institutions that have held audits (referred in many countries as mortality/morbidity conferences, or M&Ms) for many years and are still exploring way to make the audits more useful to the institution. https://www.ahrq.gov/downloads/pub/advances2/vol2/advances-deis_82.pdf, http://www.gynob.emory.edu/education/conferences/mm.html In addition to near-miss and death audits, WHO, in Beyond the Numbers, Reviewing Maternal Deaths and Complications to Make Pregnancy Safer, recommends the institution of the confidential inquiry. Confidential inquiry is a systematic multi-disciplinary anonymous investigation of all or a maternal deaths occurring at an area, regional representative sample of (state) or national level. The confidential inquiry identifies the numbers, causes and avoidable or remediable factors associated with them. Confidential inquiry requires the existence of either a functioning statistical infrastructure (vital records, statistical analysis of births and deaths, human resources, recording clerks, etc.) or nominated professionals in each facility to regularly report maternal deaths to the enquiry. The confidential inquiry can make recommendations of a more general policy nature. Importantly, because the enquiry is usually published and available to a wide public, it can be used for advocacy tool (e.g. by pokjas) to press for improvements in the quality of care. In addition to audits, and confidential inquiries, there are many additional mechanisms to safeguard and advance quality of care. These include, but are not limited to: • Licensure Board(s) – ensures that clinicians meet continuing medical education (CME) requirements as well as demonstrate ongoing active practice of clinical medicine unless a special constricted practice license is granted; disciplines clinicians conducting malpractice or having a problem with substance abuse; disciplines clinicians investigates complaints from the public; • Separately regulated authority for controlling the prescription of medicines • Departmental educational conferences • Hospital credentialing units that rigorously assesses and verifies: schooling/training, hospital privileges at prior hospitals (and problems), any time gaps in career of practicing medicine, etc. • Fear of lawsuit - can help decrease both negligence and mismanagement. In conclusion, a system to ensure quality should be composed of many components, with death and near￾miss audits being just one. Audits alone are not a magic bullet. But if audits are to improve quality of care, a deep understanding of “why and how” is need by all those who participate. Moreover, confidential inquiries add public accountability and an ancillary process to identify areas for improvement. C. WHO Guidance and Task Shifting Task shifting is a well-recognized strategy to address shortages. As WHO states in its publication, WHO Recommendations: Optimizing Health Worker Roles to Improve Access to Key Maternal and Newborn Health Interventions through Task Shifting. “Human resource shortages in the health services are widely acknowledged as a threat to the attainment of the health related Millennium Development Goals (MDGs). Attempts to optimize the potential of the “Learning lessons and acting on the results is the whole purpose of using [death audit review] approaches. There is no point in committing valuable resources to collecting information that just gathers dust on shelves. Action is the ultimate goal… There should be an objective method built into the system to monitor how recommendations are being implemented.” – WHO Beyond the Numbers FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 97 existing health workforce are therefore crucial. A more rational distribution of tasks and responsibilities among cadres of health workers is seen as a promising strategy for improving access and cost effectiveness within health systems. For example, access to care may be improved by training and enabling ‘mid-level’ and ‘lay’ health workers to perform specific interventions that might otherwise be provided only by cadres with longer (and sometimes more specialized) training. Such task shifting strategies might be particularly attractive to countries that lack the means to improve access to care within short periods of time.” CEmONC and C/S WHO supports the strategy of task shifting to address human resource shortages – including to help in the provision of C/S. Given the shortage of OBGYNs there is a great need for an additional cadre to perform C/S – one of the signal function of a CEmONC facility. However, inevitably political/professional push-back can be expected. There are as many reasons for protecting the status quo as there are people. One reason may be a sincere belief that a non-OBGYN cannot technically perform C/S as safely as an OBGYN. That said, Indonesian women, newborns, and families have a strong interest in improving access to C/S. One novel idea, offered by the Team, is to pilot the concept of co-management. The idea is to train a cadre to perform C/S and then utilize OBGYNs to their highest value by having them serve as “attending physicians” to that cadre. Midwives are the obvious cadre to perform C/S because they already have a basic knowledge/experience in obstetrics. They can “talk real shop” with the OBGYN. As soon as it is recognized that a case does or might need C/S, the OBGYN is contacted by the midwife. The OBGYN then directs the midwife in management and makes final determination about need for C/S. The OBGYN also determines the likelihood that the C/S case might be surgically or obstetrically high risk. The OBGYN comes in to manage and perform C/S on all high risk cases. The OBGYN is on call to come quickly even for what are believed to be low risk cases. Were the midwife to perform in this capacity, she would be functioning as an amalgam of midwife/surgical assistant/OBGYN. She is the “hands” of the OBGYN. The more the OBGYN is familiar with the surgical skills/judgement of the midwife (which are hopefully ever improving), the more the OBGYN can rely on the midwife as an OBGYN “extender”. The Team does not know the extent to which the C/S load can be taken off the OBGYN with this model. It is imagined that a significant portion of C/S could be performed by the non-OBGYN, including most repeat C/Ss. Indonesia is in an advantageous situation in terms of training a new cadre. First, it already has the cadre of midwives – some of whom are well trained and experienced. Second, unlike many/most countries, Indonesia has a large concentrated caseload available for training. In some hospitals, as much as 50% of all cases are performed by C/S. In order to achieve competency in C/S, a minimal number of cases must be performed – and even then, competency needs to be confirmed by some process agreed upon by the training group/institution. Other countries, including the United States have articulated case-load standards. For example, according to the American Association of Family Practice, the number of C/Ss performed by residents in training range from 25 to 100 with the average being 46. The American Board of Physician Specialties verifies training in obstetrics through the Board of Certification in Family Medicine Obstetrics (BCFMO) (24). To sit for the Board, a family medicine obstetrics applicant must have satisfactorily completed 100 vaginal deliveries and 50 cesarean sections. With the caseload available in Indonesia, a midwife could be trained quite rapidly – at least as measured by caseload. Moreover, the achievement of competency is likely to be greater in a training setting where there are many cases packed into a shorter time-frame compared to a setting where the cases are stretched out over a longer period of time. In terms of outcomes, a 2013 study in compared 250 cesarean sections performed by family physicians and OB/GYNs (Homan FF, Olson AL, Johnson DJ: A Comparison of Cesarean Delivery Outcomes for Rural Family Physicians and Obstetricians. J Am Board Fam Med 2013; 26:366-372.) There were no differences in intraoperative or infectious complications; however, there were fewer complications in the cesarean sections performed by family physicians including fewer blood transfusions and readmissions. Neonatal outcomes were the same. Additional literature speaks to good outcomes of for C/S performed by non-OBGNs. D. WHO Guidance and Job Aids - Including Checklists FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 98 Job aids, including checklists, algorithms, etc. help guide clinical management and are important for both midwives and specialists. WHO is becoming a big proponent of checklists: “In medicine, as in aviation, checklists can help ensure consistency and completeness in carrying out complex tasks.” WHO Patient Safety is currently developing a framework for identifying a range of clinical care processes where checklists would lead to an improvement in patient safety and clinical outcomes. With the help of other collaborating departments within WHO, we are already developing pilot checklists in a range of disciplines, including labor and delivery, neonatal care, and trauma care. The WHO Surgical Safety checklist and other checklists have improved reliability and helped to standardize care for thousands of patients globally. Checklists allow complex pathways of care to function with high reliability by giving users the opportunity to pause and take stock of their actions ensuring that nothing has been omitted before proceeding to the next step. The checklist approach has the same potential to save lives and prevent morbidity in medicine that it did in aviation over 70 years ago by ensuring that simple standards are applied for every patient, every time.” WHO and other excellent resources found at: www.who.int/patientsafety/implementation/checklists/background/en/ and at: reprolineplus.org/resources/job-aids-pre-eclampsia-eclampsia-prevention-detection-and-management￾toolkit. E. Postpartum Hemorrhage: Prevention, Diagnosis, and Treatment Prevention: EMAS focused primarily on the administration of uterotonic to prevent PPH. However, a KI stated that retained placenta is a huge contributor to PPH. Dr. O’Hanley, evaluation team member, has personally witnessed in other countries the problem of midwives not having adequate technique in delivering placentas – leading to prolonged third stage of labor and increased risk for PPH. Dr. O’Hanley did not have a chance to observe midwife deliveries in Indonesia – but the problem may likely exist in Indonesia as well. Diagnosis: One delay in the diagnosis and treatment of PPH likely results from an underestimation of blood loss at delivery. Many studies found that visual estimates of blood loss are notoriously inaccurate. Cochrane is currently collecting more data on the topic – stay tuned. http://onlinelibrary.wiley.com/doi/10.1002/14651858.CD010980/pdf Treatment: Midwives have not been trained in the use of balloon tamponade for treatment of PPH. This technique is easy to learn and it can be lifesaving. WHO supports its use (2012): “The use of balloon tamponade was considered by the GDG to be a measure that can potentially avoid surgery or as a temporizing measure while awaiting transfer to a higher level facility. The GDG acknowledges that balloon tamponade can be obtained with specific devices as well as with lower cost adaptations, including those based on the use of condoms and surgical gloves.” http://apps.who.int/iris/bitstream/10665/75411/1/9789241548502_eng.pdf More recent data from 2014 further supports the utility of balloon tamponade: 51 cases of postpartum hemorrhage, bleeding decreased or stopped after balloon insertion in 28/28 cases of atony, 7/8 cases of adherent placenta, 9/9 cases with both atony and adherent placentation, and 6/6 cases with neither atony nor adherent placentation (Dildy GA, Belfort MA, Adair CD, et al. Initial experience with a dual-balloon catheter for the management of postpartum hemorrhage. Am J Obstet Gynecol 2014; 210:136.e1.) Possible interventions: Conduct a study of midwife practices of placenta delivery. Introduce and train on methods to estimate blood loss. Introduce and train midwives on use of balloon tamponade. F. Midwife Preservice Training and Re-Licensure According to KIs and other reports, partly in response to a recognized midwife shortage, shortage, graduation requirements from midwife schools were lowered so as to be able to produce graduates more quickly. That midwife shortage no longer exists. There is such an oversupply that KIs report that new graduates, unable to find work, now sometimes volunteer at puskesmas just to get more experience to improve likelihood of employment. A particularly critical aspect of midwife competence is the number of deliveries performed. Countless studies in the general medical literature correlate the volume of procedures performed with outcomes. Currently in Indonesia, there is no standard that specifies the FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 99 number of deliveries required for graduation from midwife school. Midwives can graduate having performed no deliveries. In the past midwives trained in Indonesia were required to perform many deliveries. In the US, 30 observed deliveries and 30 performed deliveries is a typical requirement. In Indonesia, for re-license (required every 5 years), only 12 deliveries over the 5 years are required although many hours of lots CME re required. CME improves knowledge but not skills. In a KII it was noted that an applicant for relicensure had received a “pass” without any documentation of having performed deliveries. Possible interventions: Convene key decision makers and stakeholders, including leaders in the Indonesian government in health and education, the nursing organization, and patient/society advocates to reassess all graduation and relicensure requirements, with a special focus on number of required deliveries. Also, conduct a study to identify if there are pervasive re-licensure problems. G. Call Centers GPs supervise midwives at Puskesmas level, but in fact their primary role, vis-à-vis the midwives, is often just to be the physician of record. Many do not have more knowledge about OB than do the midwives. Consequently, midwives in private practices and at Puskesmas have inadequate opportunity to consultation. They need ready access to consulting advice to help manage patients in a timely manner and also to reduce the need for formal consults requiring the patient to travel to a higher level facility. The Team suggests a novel solution: consider establishing a telemedicine consulting service/team, available by phone 24/7 (or only during clinic hours – depending on perceived need and anticipated utilization). There are countless examples in high income countries of call centers (utilized by group practices and hospitals) that are accessible directly to patients to answer questions and manage various medical problems over the phone – often following a protocol/management algorithm. Example: https://www.mdanderson.org/for-physicians/clinical-tools-resources/clinical-practice-algorithms/clinical￾management-algorithms.html. However, the Team is unfamiliar with the existence of call centers to support clinicians. The consultant service/team could be composed of both midwives and an OBGYN. If a large number of districts shared this service and shared costs, a relatively small contingent of consultants, forming the service/team could serve a very large number of puskesmas midwives. H. MCH Handbook for Midwives The MOH’s Maternal and Child Health Handbook describes itself as a “tool for an early detection of any mothers and children’s health problems, a communication tool as well as counselling tool with important information for mothers, families and public”. However, in the prenatal care section, it lacks: a column to record fetal movement; a column to record urine protein screening; probes for problems with prior pregnancies or most recent delivery. [Note: screening for urine protein helps identify preeclampsia – hopefully when it is still mild preeclampsia. Having the mother and clinician monitor fetal movement is a tool to help lower rates of stillbirths and newborn deaths.] The educational information for pregnant women inadequately addresses many important maternal problems/concerns. A possible solution would be to convene decision makers at MOH to consider revising the Maternal and Child Health Handbook. If changes were made to the handbook, staff using the handbook would need to be oriented/trained about the changes. One could gather (and adopt as appropriate) examples from other countries for educational messaging packages for pregnancy women - for example Kaiser Permanente has excellent one-page educational handouts for each prenatal visit. I. JKN JKN is designed to expand access and equity while containing costs. Some of its rules may have the unintended impact of limiting access or unwittingly increasing costs (e.g. indirectly morbidities with the attendant costs of treating them). Illustrative examples offered by KIs include: • Reimbursement for only four prenatal visits. This is not compliant with WHO’s recommended eight prenatal visits and one ultrasound. Many problems, including severe preeclampsia usually do not present until late in pregnancy and are more likely to be detected with relatively frequent prenatal visits close to term. • Hospitals sometimes refuse JKN referrals because reimbursement often does not cover true costs. FINAL EVALUATION OF THE USAID/INDONESIA EMAS PROGRAM 100 • In some districts, JKN forces midwifes to refer only to a Level 2 hospital - even if mother or newborn actually need to go immediately to a Level 3 hospital. This leads to unnecessary delays that can result in increased morbidity/mortality. • JKN does not reimburse adequately to cover costs for stabilization. • Some private hospitals provide referral incentives to midwives (IDR 300 – 350K) which can result referral to inappropriate facility or over referral. • JKN reimburses at a higher rate for cesarean section than for vaginal delivery. Some private facilities over perform cesarean sections to collect the much higher reimbursement compared to vaginal delivery. • JKN does not cover non-ambulance referral costs – patients accompanied by an MKIA cadre not reimbursed therefore wait for money. • Patients need to wait for processing of KIS/JAMPERSAL/JAMKESDA/SKTM before receiving treatment at hospital. • Payments to Puskesmas is based on capitation, but if a puskesmas serves a small population, it may not receive enough money to provide the most basic serves or even maintain its building. This is equity for rural populations. • Payments are capped – leading hospitals to refusal to receive complicated referrals, allowing some newborns to die to avoid providing expensive care, etc. The next project should consider conducing on behalf of JKN, maternal and newborn cost benefit, cost and efficacy, access/equity studies, and quality of care studies, to help JKN re-evaluate and possibly revise some of its policies.