USAID-UNICEF Maternal and Child Health Integrated Malaria Control program in Eastern Indonesia 2   ACKNOWLEDGMENT This report was made for review by the United States Agency for International Development and prepared by Lucas Pinxten, Siti Nurul Qomariyah, and Irma Anintya Tasya for the USAID Indonesia Mission under the contract number AID-497-O-14-00097. The following individuals have contributed to the preparation of this report: John Rogosch, Masee Bateman, Maria Pinto, Edhie S. Rahmat, Mildred Pantouw, Ratna Suwandono of the United States Agency for International Development (USAID), Robin Nandy, Bill Hawley, Ferdinand Laihad, Karina Widowati, Budhi Setiawan, Olivi Silalahi, Hellen Parrera, Sudhir Kanal, Jana Fitria Kartika Sari, Ratih Woelandaroe, Nurlely Bethesda Sinaga of the United Nations Children’s Fund (UNICEF), Asik Surya, Erna Muliati, Lukas Hermawan, Riskiyana Sukandhi Putra of the Indonesia Ministry of Health (MoH) Meike Pontoh (PHO Maluku) Aloysius Giyai (PHO Papua) Victor Eka Nugrahaputra (PHO West Papua) Bitzael S. Temmar (Maluku Tenggara Barat(MTB)/ West South-East Maluku district). Special thanks for the warm welcome and input from the Head of DHO and Staff, District Hospital Staff, Community Health Center Staff and clients, and the community leaders at MTB District, Jayapura District and Sorong District. Contact persons: Lucas Pinxten, MD, MPH, Global Health Management, Maastricht, The Netherlands. (lpinxten@gmail.com) and Siti Nurul Qomariyah, MD, PhD, Center for Family Welfare, Faculty of Public Health, University of Indonesia.(snqomariyah@gmail.com) USAID-UNICEF Maternal and Child Health Integrated Malaria Control program in Eastern Indonesia 3   Content page A. Acronyms and Abbreviations 4 B. Executive Summary 6 1. Introduction and background 11 1.1. Introduction 11 1.2. Background 15 1.3. Scope of Work 18 2. Methodology 20 2.1. Overarching framework 20 2.2. Time line design, selection of sites and informants 21 2.3. Data collection and instruments 23 2.4. Data process and interpretation 24 3. Findings 25 3.1 Progress on objectives against combined MIP / ACHIEVE Indicators 25 3.1.1. Objective I: Improve service quality 27 3.1.2. Objective II: Improve health management 29 3.1.3. Objective III: Use of evidence for implementation 30 3.1.4. Project-specific objectives 30 3.1.5. Outcome Indicators 31 3.2. UNICEF inputs and results to strengthen MoH Malaria and MCH program 33 3.3. Program focus and coherence 36 3.4. Coordination and harmonization with other (non-) USAID funded activities 37 3.5. Integration and synergy MIP and ACHIEVE 38 3.6. Health system challenges and adaption to MoH policy changes 39 3.7. Monitoring and evaluation 40 3.8. Sustainability and scalability 41 4. Recommendations 43 4.1. Recommendation for the remaining years 43 4.2. Observations and recommendations for the future programming 46 4.3. Measuring programs contributions towards USAID’s higher-level health indicators 47 4.4. Lessons learned 49 List of Annexes: Deliverables, Timeline, Itinerary and list of informants, Overview structural interview detailed questionnaires, National Performance on MCH/ICM indicators, Task division, Matrix overview of interviews, focus group discussions with Clients/Patients List of MCH indicators, Recommendation and Priority Matrix, List of documents/literature, and statement regarding conflict of interests. USAID-UNICEF Maternal and Child Health Integrated Malaria Control program in Eastern Indonesia 4   A. ACRONYMS and ABREVIATIONS ACT Artemisinin-based Combination Therapy ANC Antenatal Care ABER Annual Blood Examination Rate AKBID Akademi Kebidanan: Midwifery Academy APBD Anggaran Pendapatan dan Belanja Daerah (District Revenue and Expenditure Budget) API Annual Parasites Incidence ART Antiretroviral Therapy ATM AIDS, Tuberculosis and Malaria Center BAPPEDA Badan Perencanan Pembangunan Daerah (Local Development Planning Board) BEONC Basic Emergency Obstetric and Neonatal Care BPS Badan Pusat Statistik (National Statics Office) BOK Biaya Operational Kesehatan, or Operational Health Funds Bapelkes Provincial Health Training Center Bupati Head of district Cadre Community volunteer CCM Country Coordinating Mechanism CDC Centers for Disease Control and Prevention CHC Community Health Center (Puskesmas) CEONC Comprehensive Emergency Obstetric and Neonatal Care DHO District Health Office DHS Demographic and Health Survey EMAS Expanding Maternal and Neonatal Survival, a USAID program EPI Expanded Program on Immunization GFATM Global Fund to Fight AIDS, Tuberculosis and Malaria GOI Government of Indonesia NGO Non-governmental Organization IBI Ikatan Bidan Indonesia (Indonesian Midwives Association) IDAI Ikatan Dokter Anak Indonesia (Indonesian Pediatricians Association) IMP Integrated Microplanning IPT Intermitted Preventive Treatment Jamkesmas Jaminan Kesehatan Masyarakat (Community Health Insurance) Jampersal Jaminan Persalinan (Delivery Insurance) JKN Jaminan Kesehatan Nasional (Universal Health Coverage) KIE Knowledge, Information and Education LLIN Long-lasting insecticide treated bed net MCH Maternal and Child Health MDG Millennium Development Goal MIP Malaria in Pregnancy MNCH Maternal, Neonatal and Child Health MCH/IMC Maternal and Child Health / Integrated Malaria Control MMR Maternal Mortality Rate NMR Neonatal Mortality Rate MOH Ministry of Health USAID-UNICEF Maternal and Child Health Integrated Malaria Control program in Eastern Indonesia 5   MTB Maluku Tenggara Barat District (district in eastern Indonesia) MWH Maternity Waiting Home NMCP National Malaria Control Program NTB Nusa Tenggara Barat, (West Nusa Tenggara, a province in Eastern Indonesia NTT Nusa Tenggara Timur (East Nusa Tenggara), a province in Eastern Indonesia PERDA Peraturan Daerah (District or province regulation) PHO Province Health Office PKK Pembinaan Kesejahteraan Keluarga (Family Welfare Movement) PLA Participatory Learning and Action PMTCT Prevention of Mother-to-Child Transmission Polindes Pondok Bersalin Desa (Village delivery post) Poltekkes Politeknik Kesehatan (Polytechnic health academy) Posyandu Pos Pelayanan Terpadu (Integrated Health Service Post) Puskemas Pusat Kesehatan Masyarakat (Community Health Center) Pustu Puskemas Pembantu (Sub-CHC) RDT Rapid Diagnostic Test RSCM Rumah Sakit Dr. Cipto Mangunkusumo RSUD Rumah Sakit Umum Daerah / District Hospital SBA Skilled Birth Attendant SB Sekolah Bidan: midwifery school SOP Standard Operating Procedure TB Tuberculosis TBA Traditional Birth Attendant TWG Technical Working Group UNDIP Diponegoro University UNHAS Hasanuddin University UNICEF United Nations Children’s Fund US United States USAID United States Agency for International Development WHO World Health Organization USAID-UNICEF Maternal and Child Health Integrated Malaria Control program in Eastern Indonesia 6   B. Executive Summary Introduction Indonesia has a population of 238 million: the fourth largest population in the world and is the largest economy in South East Asia. Despite impressive macro economic growth figures, significant health challenges remain: over 40% of the population or around 105 million are living on less than 2 US$ a day and the progress on maternal health (MDG 5) has slowed down: maternal mortality rate (MMR) (number of deaths among pregnant women / within 42 days after delivery per 100,000) declined from 390 / 100,000 in 1991 to 228 / 100,000 in 2007 and was in 2012 pegged at 359 / 100,000. Infant and child mortality (MDG 4) is currently 32 respectively 40 per 1,000 live births. Especially the reduction of neonatal mortality rate (NMR) has stalled: still at the 2007 level: 19 deaths per 1,000 births.1 This performance is, compared to other Asian socio-economic less developed countries, rather poor and Indonesia will probably not meet these MDG targets next years.2 East Indonesia is a high Malaria transmission area because it has ideal mosquito breeding grounds: dense tropic forests, swamps, marshes and shallow water basins. This part of Indonesia also has a myriad of remote islands in Maluku, an extremely difficult highland terrain in Papua, and is home to the 4 provinces with the highest percentage of relative poverty3,4. Malaria is important in endemic areas, like Eastern Indonesia and can be responsible for maternal deaths both during acute malarial episodes and afterward because of the effects of the disease on maternal anemia5 . In East Indonesia the 3 leading causes of maternal mortality are: 1. Haemorrhage: 34% (including: post- and antenatal haemorrhage, placenta previa, and premature separation of placenta). This is higher compared to other Indonesian areas. 2. Eclampsia: 26% (including: edema, proteinuria, and hypertensive disorder). This is lower compared to other Indonesian areas. 3. Infection and puerperium problems, including Malaria (30%) (One of the highest compared to other Indonesian areas)6 . Neonatal mortality also in Indonesia is typically attributed to one of three major causes: infection, asphyxia, or                                                                                                               1 Indonesia - Demographic and Health Survey 20122013; Statistic Indonesia, National Population and Family Planning Board, Ministry of Health, Measure DHS, ICF International: Accessed from: http://dhsprogram.com/publications/publication-FR275-DHS-Final￾Reports.cfm. 2 Issue Briefs: Maternal and Child Health [database on the Internet]. UNICEF Indonesia,. 2012 [cited August 7, 2014]. Accessed from: http://www.unicef.org/indonesia/A5-_E_Issue_Brief_Maternal_REV.pdf. 3 World Bank. Indonesia Economic Quarterly - Slower Growth; High Risks2013: Accessed from: http://www.worldbank.org/content/dam/Worldbank/document/EAP/Indonesia/IEQ-Dec13-ENGLISH.pdf. 4 North Maluku, Maluku, West Nusa Tenggara, East Nusta Tenggara, West Papua and Papua - Poverty - Percentage of Poor People, Rural [database on the Internet]. knoema.com. [cited September 8, 2014]. Accessed from: http://knoema.com/atlas/embed/Indonesia/North-Maluku/topics/Living-Conditions/Poverty/percent-of-Poor-People￾Rural?compareTo=ID-MA,ID-PB. 5 Ibid. 6 Joint Committee on Reducing Maternal and Neonatal Mortality in Indonesia. Reducing Maternal and Neonatal Mortality in Indonesia : Saving Lives, Saving the Future. 2013. USAID-UNICEF Maternal and Child Health Integrated Malaria Control program in Eastern Indonesia 7   prematurity. Indonesia relies on national IDHS data. The IDHS National NMR: 20/1000, which is about 50% of the Infant Mortality rate (which was according to IDHS 2012 45/1000 live births). According to IDHS 2012 NMR per province: Papua: 27, West Papua 35, NTT 26, Maluku 24, North Maluku 27/1000 live births.1 Most if not all of these maternal and neonatal deaths are preventable. About 50% of maternal deaths are managed by a health professional before death7 while in Malaria endemic situations up to 25% of the maternal deaths can be attributed to Malaria.8 Socio-economic, educational, cultural factors and the poor access to services, due to the extreme geographical settings, are problems beyond short time interventions. In addition delays in recognition of a maternal/neonatal emergency, decision making and transport/referral, and the quality of services: the classic "three delays and lack of quality of services" can be targeted as we can learn from other countries who managed to reduce both maternal and neonatal mortality to the MDG targets in a rather short period of time span (5-10) years. The MCH/MIP program approach of using Malaria prevention, diagnoses and treatment as a vehicle to strengthen the complex system- prevention- and care interventions is an innovative approach that will lead to professional synergy9 and is planned to reduce malaria in pregnancy and maternal and neonatal death3 . The current USAID-funded and GoI/UNICEF implemented Maternal Child Health and Integrated Malaria Control in Eastern Indonesia project (MCH/MIP) covers past years activities from two separate projects: Malaria in Pregnancy (MIP) and the ACHIEVE: the former MNCH project. This integrated project started on October 1, 2010, will expire on September 29, 2015, and is funded through a five-year grant, which is as MCH Umbrella Grant awarded to UNICEF/NY Health Section for health and immunization response support. The program is in sync with both USAID and UNICEF’s overall goal to support GoI in attaining the MDG 4, 5 and 6, and is designed to implement activities to improve MCH through utilizing evidence-based activities and practices, to improve the continuum of care as it links to communities, primary level facilities, and hospitals in Eastern Indonesia10. The purpose of the mid-term evaluation was to assess UNICEF’s progress towards their stated objectives and indicators in this project, and to make specific recommendations to improve the project’s performance and future programming.                                                                                                               7 Ronsmans C, Scott S, Qomariyah SN, Achadi E, Braunholtz D, Marshall T, et al. Professional assistance during birth and maternal mortality in two Indonesian districts. Bull World Health Organ. 2009 Jun;87(6):416-23. 8 Steketee RW, Nahlen BL, Parise ME, Menendez C. The burden of malaria in pregnancy in malaria-endemic areas. Am J Trop Med Hyg. 2001 Jan-Feb;64(1-2 Suppl):28-35. 9 WHO Bulletin. Editorial. 2005 April;83(4). 10 Maternal and Child Health and Integrated Malaria Control in Eastern Indonesia revised 3-08-2013. USAID project document regarding incremental funding to immunization in collaboration with CDC. Jakarta2013. USAID-UNICEF Maternal and Child Health Integrated Malaria Control program in Eastern Indonesia 8   Methodology The evaluators used a mixed quantitative-qualitative design. For the quantitative analysis the evaluation team compared numerical aggregated provincial data from the year 2013 on MCH/MIP program progress on targets/indicators, against 2010 Baseline MCH/MIP program targets/indicators data. The team performed an overall progress on program targets/indicators and an analysis of progress on MCH/MIP program targets/indicators per province. For the qualitative analysis the evaluation team performed a desk study of available MCH/MIP program documentation and conducted structural interviews and focus group discussions with informants. For this purpose a structural overall interview guideline was developed of which subsequently 5 specific interview guidelines were derived for interviewing stakeholders form MoH national, UNICEF Country Office, DHO/PHO, UNICEF Provincial Offices and for focus group discussion with clients. A result matrix was developed containing detailed questions on: progress towards the project’s objectives, target, focus and scalability of the project to support GoI investments in MCH/MIP, appropriateness, accuracy and specificity of indicators, donor coordination and harmonization and the integration of the two formerly-separate MIP and ACHIEVE projects. The informants were: national and local level government staff working at both policy and technical levels, USAID, UNICEF and CDC professional staff, a representative from Global Fund for Aids Tuberculosis and Malaria (GFATM), representatives from two other USAID funded programs related to mother and neonatal health care: Kinerja and EMAS, UNICEF MCH/MIP program staff and clients/patients in 3 of the 5 Eastern Indonesian provinces: Maluku, Papua, West Papua. Due to time limits the evaluation team, together with USAID/UNCEF Country offices, selected the above-mentioned provinces. The quantitative analysis covered all 5 provinces of Eastern Indonesia: Maluku, North Maluku, Papua, West Papua and Nusa Tenggara Timor (NTT). The qualitative analysis covered the above-mentioned 3 provinces. The evaluation lasted from August 18-28,2014. Findings Comparing the MCH/MIP program baseline targets/indicators from 2010 against the 2013 performance indicators, it was concluded that over all provinces and over all indicators 46% reached 100% or beyond the targets/indicators and 78% of the targets/indicators were reached at 50% or beyond. Most targets seemed to be set realistic and program progress was good. Some specific selected findings are listed below: 1. Improvement of district health system (objective I of the MCH/MIP program) was measured by the number of Health Centers reporting no stock outs of essential items for the integrated Malaria program supplies: artemisinin-based combination therapy (ACT), rapid diagnostic test (RDT) and long-lasting insecticide treated bed nets (LLIN) or maternal USAID-UNICEF Maternal and Child Health Integrated Malaria Control program in Eastern Indonesia 9   health program: magnesium sulphate, oxytocin, intravenous antibiotics (MCH/MIP indicators 5-10). It proved that no data at all were collected for MCH indicators (8, 9, 10), while for the Malaria indicators, NTT province did not report and Maluku province reached 86% on the stock of LLIN distribution (indicator 7) while all other provinces (Maluku Utara, Papua, West Papua and NTT reached 100% or more on indicators covering the stocks of LLIN, ACT and RDT (ind.5, 6, 7). 2. To measure service quality the proportion of pregnant women attending first ANC contact who receive screening and appropriate treatment (indicator 12 & 13) were recorded: Maluku Utara reached on both the Malaria and MCH target 84%. West Papua reached only 30% of its Malaria target while Papua did not report at all on the MCH indicator (13). Maluku and NTT Province reached respectively 52% and 72% on both the Malaria and MCH indicators. Papua reached 58% on Malaria Indicator (12) and West Papua 57% on the MCH Indicator. 3. In order to measure increase of coverage of the integrated MCH/MIP program through advocacy and leveraging of local Government/GFATM funds (Malaria Objective 3) the number of districts with budget allocation for integrated MCH /MIP program that is at least equal to the total contribution of the GFATM and UNICEF, was recorded (Indicator 18). Analysis of the routine reported data revealed that only Maluku Utara and Papua reached 100% or more on the targets of this indicator. Maluku and West Papua showed no progress and NTT did not report on this indicator. 4. East Indonesia combines a very poor referral system, due to extreme geophysical setting with a low professional density: of the 1700 OB&GY’s, 14 work in Papua and only 9 in Maluku. Also neonatology and anesthesiology are underdeveloped in this part of Indonesia. 5. In order to ensure that all women and newborns receive comprehensive and quality care during pregnancy, delivery and postnatal periods the proportion of delivery assisted by skilled birth attendance was recorded. The results showed that Maluku Utara and Maluku reached respectively 75% and 53% of their targets, while Papua, West Papua and NTT reached over 100% of their targets. Recommendations 1. To counter the regular Malaria and MCH stock-outs it is recommended that UNICEF advocates for a buffer stock at DHO-PHO level for Oxytocin, Magnesium sulphate, RDT and ACT and ILLNs, and increases its efforts to strengthening the referral system and the quality of BEONC sites whereby action is required from UNICEF and the District and Provincial Health offices (DHO/PHO). 2. The successes of the ILLN, Malaria screening and treatment, in terms of increased first ANC visit and increased immunization figures, probably does not contribute much to USAID-UNICEF Maternal and Child Health Integrated Malaria Control program in Eastern Indonesia 10 increase the proportion of deliveries assisted by skilled midwives. Beside the development of buffer stocks it is recommended to use the synergy obtained by the integrated program to inform expecting mothers of the benefits of the JKN health insurance scheme, and to promote skilled delivery at an official BEONC site whereby action is required from UNICEF and DHO/PHO level. 3. The integrated MCH/MIP program has proven to be successful in leverage of other funds like APBD (Local Government) funds in Maluku and North Maluku for Malaria screening, RDT and ILLNs for pregnant women and children and GFATM funds for scaling-up the integrated MCH/MIP approach. However there is no indicator to measure this success. For future and current program planning and to strengthen monitoring and evaluation, it would be advisable adding strong output indicators for leverage, such as ensuring sustainable funding from the local government to support the program whereby action is required from UNICEF and USAID. 4. With the new budget allocation based on Village Law, villages in the remote area are encouraged to secure some funds to support transportation to refer emergency patients. Further formalizing the referral system at all levels is needed to ensure the roles of each level to stimulate their ownership through actions by UNICEF and DHO/PHO. 5. Given the fact that over 30% of the neonatal mortality happens at delivery and about 50% of maternal deaths are managed by a health professionals before death it is highly recommended that UNICEF develops a stronger focus on strengthening the basic integrated maternal/neonatal health services including safe deliver practices and neonatal resuscitation at Health Center/BEONC level instead of trying to cover the whole spectrum of Maternal and Neonatal health: Action required at UNICEF Country office level. Selected lessons learned 1. Midwives at BEONC centers had a patient caseload below 100 deliveries per year. Below this international recognized standard, especially emergency delivery skills are eroding. As a result of the low caseload of midwives, a trained attendant is not necessarily a skilled attendant and there is no guarantee for a safe delivery / appropriate neonatal care. 2. When the under-funded MoH system is faced with budget cuts it is the supervision budget at all levels that seems to be sacrificed first. Lack of quality monitoring / supervision / feedback mechanism results, not only in low motivation of field staff and reduced service quality, but also in poor data collection which hampers evidence-based policy making and planning: these budget cuts come at a very high cost. 3. Integrating the MIP guidelines and introduced to the curriculum of medical/midwifery schools is an excellent way to ensure the continuation transfer of learning / knowledge related to the integration and production of high quality of health service providers. USAID-UNICEF Maternal and Child Health Integrated Malaria Control program in Eastern Indonesia 11 1. Introduction and background 1.1. Introduction Indonesia has a population of 238 million; is the fourth largest population in the world and is the largest economy in South East Asia. Extreme poverty fell from 23.4% in 1999 to 11.4% in 2013, while over 40% of the population or around 105 million are living on less than 2 US$ a day.11 The Eastern Indonesian provinces rank among the poorest in Indonesia. Papua is the poorest of all Indonesian provinces ranking at 33, followed by West Papua (32), Maluku (31), East Nusa Tenggara (30) and West Nusa Tenggara (29); North Maluku is the exception in Eastern Indonesia ranking at 14 but still with about 10% of the population living below the poverty line.12 From 1990 to 2007 Indonesian reduction of Maternal Mortality (MDG 5) and of U￾five mortality (MDG 4) figures showed a steady progress with having the largest reductions happening before the financial crisis and the decentralization. The hasty decentralization in 2001 has increased the difficulty of establishing coordinated national health programs and accounting for health funds and as a result may have contributed to the slowdown of U-five mortality and especially maternal mortality (MMR) reduction. Despite last years and current strong economic growth, significant health challenges remain: the progress on maternal health (MDG 5) has slowed down maternal mortality rate (MMR) (number of deaths among pregnant women / within 42 days after delivery per 100,000) declined from 390 / 100,000 in 1991 to 228 / 100,000 in 2007 and was in 2012 pegged at 359 / 100,000. Infant and child mortality (MDG 4) is currently 32 respectively 40per 1,000 live births. Especially the reduction of neonatal mortality rate (NMR) has stalled: still at the 2007 level: 19 deaths per 1,000 births. 13 This performance is, compared to other Asian countries, rather poor and as a result Indonesia will probably need a few more years to meet these MGD targets. 14                                                                                                               11 World Bank. Indonesia Economic Quarterly - Slower Growth; High Risks2013: Accessed from: http://www.worldbank.org/content/dam/Worldbank/document/EAP/Indonesia/IEQ-Dec13-ENGLISH.pdf. 12 North Maluku, Maluku, West Nusa Tenggara, East Nusta Tenggara, West Papua and Papua - Poverty - Percentage of Poor People, Rural [database on the Internet]. knoema.com. [cited September 8, 2014]. Accessed from: http://knoema.com/atlas/embed/Indonesia/North-Maluku/topics/Living-Conditions/Poverty/percent-of-Poor-People￾Rural?compareTo=ID-MA,ID-PB. 13 Indonesia - Demographic and Health Survey 20122013; Statistic Indonesia, National Population and Family Planning Board, Ministry of Health, Measure DHS, ICF International: Accessed from: http://dhsprogram.com/publications/publication-FR275-DHS-Final￾Reports.cfm. 14 Joint Committee on Reducing Maternal and Neonatal Mortality in Indonesia. Reducing Maternal and Neonatal Mortality in Indonesia : Saving Lives, Saving the Future. 2013. USAID-UNICEF Maternal and Child Health Integrated Malaria Control program in Eastern Indonesia 12 In Indonesia, availability and quality of basic emergency obstetric and neonatal care (BEONC) is below standard: according to the 2011 Health Facility Review, by WHO and World Bank, about 60% of districts have less than the government’s recommended 4 Health Centers providing BEONC. Shortages of drugs and supplies at Health Center level are common also for key commodities to address maternal emergencies. Both health professionals and families have to deal with the harsh reality of poor BEONC services leading to maternal and neonatal death. According to the above￾mentioned survey almost 25% of the Health Center in-patients, including patients in BEONC Health Centers, had no transport for referral. Service quality and rates of maternal and neonatal death are the poorest in the provinces of Eastern Indonesia and among the poorest women and children. East Indonesia has a myriad of remote islands in Maluku, an extremely difficult highland terrain in Papua, and has a huge variety in quality of healthcare due to differing levels of commitment, availability and level of skills of health professionals in each district. Maternal and child mortality was, compared to the national average, always higher in Eastern Indonesia because of lower demand, a weaker health system, the geophysical challenges and the higher burden of infectious diseases, including Malaria. In East Indonesia the 3 leading causes of maternal mortality are: 1. Haemorrhage: 34% (including: post- and antenatal haemorrhage, placenta previa, and premature separation of placenta). This is higher compared to other Indonesian areas. 2. Eclampsia: 26% (including: edema, proteinuria, and hypertensive disorder). This is lower compared to other Indonesian areas. 3. Puerperium problems, mostly infections including Malaria (30%) (one of the highest compared to other Indonesian area's)15. Malaria is important in endemic areas, like Eastern Indonesia and can be responsible for maternal deaths both during acute malarial episodes and afterwards because of the effects of the disease on maternal anemia.16 Most if not all of these maternal and neonatal deaths are preventable. About 50% of maternal deaths are managed by a health professional before death17 while in a Malaria endemic situation up to 25% of the maternal deaths can be attributed to Malaria.18,19 The medical factors that cause                                                                                                               15 Ibid. 16 Ibid. 17 Ronsmans C, Scott S, Qomariyah SN, Achadi E, Braunholtz D, Marshall T, et al. Professional assistance during birth and maternal mortality in two Indonesian districts. Bull World Health Organ. 2009 Jun;87(6):416-23. 18 Steketee RW, Nahlen BL, Parise ME, Menendez C. The burden of malaria in pregnancy in malaria-endemic areas. Am J Trop Med Hyg. 2001 Jan-Feb;64(1-2 Suppl):28-35. 19 Schantz-Dunn J, Nour NM. Malaria and pregnancy: a global health perspective. Rev Obstet Gynecol. 2009 Summer;2(3):186-92. USAID-UNICEF Maternal and Child Health Integrated Malaria Control program in Eastern Indonesia 13 maternal, fetal (stillbirth), and neonatal deaths are often the same and the causes of increased maternal, fetal, and neonatal mortality include poor nutrition, living in poverty, poor access to care, and poor quality of care. High risk of maternal death is strongly related to high risk of both stillbirth and neonatal death and interventions aimed at reducing maternal mortality will frequently reduce stillbirths and many neonatal deaths.20 Socio-economic, educational and cultural factors and the poor access to services due to the geographical setting, are problems beyond short time interventions. However, delays in recognition of a maternal/neonatal emergency, decision making and transport/referral: the classic "three delays" and the quality of services, can be targeted as we can learn from other countries who managed to reduce both maternal and neonatal mortality to the MDG targets in a rather short period of time span (5-10) years.21,22 East Indonesia has vast tropical forests, swamps, coastal marshes and shallow lakes and as a result this is a high Malaria transmission area (>1 case per 1000 population) and annual parasites incidence (API) between 10% (Lesser Sunda Island) and 15% (Papua).23,24 Pregnancy weakens the pregnant women’s immunity, increasing the risk of infection, severe anemia and death. Malaria in pregnancy results in hemolytic anemia and is often the cause of maternal death. Also acute renal insufficiency, spontaneous abortion and stillbirth are related to Malaria in pregnancy while Malaria infection of the placenta leads to low birth weight.25 Globally Malaria case fatality rate in pregnant women varies from 13% in a stable epidemiological setting to 71% in a Malaria outbreak situation.26 Malaria contributes at about 8% to the child's mortality27 while fetal loss or neonatal death rate in a Malaria outbreak situation can be up to 67%. 28 In Africa where Malaria is endemic (Malaria Plasmodium only) it may directly contribute to about 25% of all maternal deaths.29 Sub￾Saharan African (SSA) interventions illustrate that 90% of the maternal Malaria burden can                                                                                                               20 Joint Committee on Reducing Maternal and Neonatal Mortality in Indonesia. 21 Freedman LP, Graham WJ, Brazier E, Smith JM, Ensor T, Fauveau V, et al. Practical lessons from global safe motherhood initiatives: time for a new focus on implementation. Lancet. 2007 Oct 13;370(9595):1383-91. 22 Lazanoet. Progress towards Millennium Development Goals 4 and 5 on maternal and child mortality: an updated systematic analysis. The Lancet. 2010;378:113-6. 23 WHO. International Malaria profiles: Indonesia Accessed from: http://www.who.int/Malaria/publications/country￾profiles/profile_idn_en.pdf. 24 Elyazar IR, Gething PW, Patil AP, Rogayah H, Kusriastuti R, Wismarini DM, et al. Plasmodium falciparum malaria endemicity in Indonesia in 2010. PLoS One. 2011;6(6):e21315. 25 Manson's Tropical Diseases. 23 ed. Jeremy Farrar et all, editor. London: Elsevier Saunders; 2014. 26 Wickramasuriya. Malaria and Ankylostomiasis in the Pregnant Woman. Oxford.: Oxford University Press; 1937. 27 Hammerich A, Campbell OM, Chandramohan D. Unstable malaria transmission and maternal mortality--experiences from Rwanda. Trop Med Int Health. 2002 Jul;7(7):573-6. 28 Steketee RW, Nahlen BL, Parise ME, Menendez C. The burden of malaria in pregnancy in malaria-endemic areas. Am J Trop Med Hyg. 2001 Jan-Feb;64(1-2 Suppl):28-35. 29 Schantz-Dunn J, Nour NM. Malaria and pregnancy: a global health perspective. Rev Obstet Gynecol. 2009 Summer;2(3):186- USAID-UNICEF Maternal and Child Health Integrated Malaria Control program in Eastern Indonesia 14 be reduced by a combined approach: preventive chemoprophylaxis/ intermittent preventive treatment (IPT) with Sulphadoxine-pyrimethamine (SP), and the use of long lasting insecticide treated bed nets (LLINs).30 However the traditional IPT utilized in SSA is not applicable to Indonesia, because of the SP resistance and the fact that Indonesia has four species of human malaria parasites: P. Vivax is the predominant species except in Papua where P. Falciparum slightly predominates. P. Malariae and P. Ovaleare mostly found in the eastern part of Indonesia, Nusa Tenggara Timur and Papua. The current national treatment policy is Artemisin Combination Therapy (ACT) for pregnant women except for 1st trimester infections. According to the 2012 IDHS in Maluku 87%, in North Maluku 90%, in West Papua 87% and in Papua 58%, received at least one check-up from public doctors, nurses, or trained midwives.31 With this coverage level strengthening ANC service delivery through increased coverage of interventions like LLINs and IPT with ACT, will be an incentive for the use of other services such as delivery with the assistance of a skilled birth attendant. According to B. Hawley, a CDC specialist on Malaria in Indonesia "Combining the vertical Malaria program stimulates like a wedge the horizontal MCH program."32 Integration of MCH/MIP also has a positive synergetic effect on the health professionals involved: they are stimulated to cooperate and to work as a team.33 According to the WHO: using Malaria in pregnancy as an entry point to comprehensive maternal and child health services, seems to be a very sound approach because it will not only reduce the burden of Malaria during pregnancy but will also improve MCH outcome.34 The Government of Indonesia introduced in 2014 a national health insurance scheme, Jaminan Kesehatan Nasional (JKN), aiming to achieve universal health coverage by the end of 2019. In 2015-16, substantial financial resources will be channelled to over 68,000 villages through arrangements under the new Village Law (800 million to 1.4 billion Rupiah per village). Coming years the village will have authority over a much larger budget, which will lead to increased accountability and inclusive decision-making. Especially point 3 of article 74 of the Village Law: improving public services for village citizens in order to                                                                                                               30 WHO. A Strategic Framework for Malaria Prevention and Control in Africa: Accessed from: http://www.who.int/malaria/publications/atoz/afr_mal_04_01/en/. 31 Indonesia - Demographic and Health Survey 2012. 32 Personal comment during discussion with UNICEF country office staff, August 8 2014. Jakarta. 33 WHO Bulletin. Editorial. 2005 April;83(4). 34 WHO Global Malaria Programme. Malaria: Global Fund proposal development (Round 11) - WHO Policy Brief2011: Accessed from: http://www.who.int/Malaria/publications/atoz/Malaria_gf_proposal_dev_who_policy_brief_201106.pdf. USAID-UNICEF Maternal and Child Health Integrated Malaria Control program in Eastern Indonesia 15 accelerate the realization of public welfare, offers opportunity to advocate for local funding of village level, strengthening of LOCAL maternal and neonatal preventive and care services.35 1.2. Background In October 2010, USAID decided through a five-year grant and under the UNICEF Umbrella Grant, (Agreement GHA-G-00-07-00007), to continue support for control of Malaria in Pregnancy (MIP) in 5 provinces in eastern Indonesia. The intent of the program is to reduce Malaria-related maternal and child mortality in provinces facing the highest burden of Malaria in Indonesia and home to 15 million people living on about 700.000 km2 and a population density of 0.05 persons per km2.. Also in October 2010, the new maternal and child health (MCH) UNICEF project was funded to accelerate progress in reducing maternal mortality in 4 of the provinces supported by the MIP program. In 2012 the two separate projects: Malaria in Pregnancy (MIP) and the MCH (ACHIEVE) were combined into one project titled: Maternal and Child Health and Integrated Malaria Control in Eastern Indonesia (MCH/MIP).This program expires in September 29, 2015. The MCH/MIP program is in sync with both USAID and UNICEF’s overall goal in supporting GoI in attaining the MDG 4,5 and 6.This integrated project covers past years activities of two separate projects and is measured against the same indicators. The integrated project is designed to enhance efficiency and alignment in the joint effort to strengthen GoI systems for greater coverage and quality of integrated MCH/MIP services in Eastern Indonesia. The medical events related to maternal, fetal (stillbirth) and neonatal deaths are mostly strongly related but the factors behind the causes include poor nutrition, living in poverty, poor access to care, and poor quality of care.36 As stated above Malaria is endemic in Eastern Indonesia with particularly high rates in the provinces of Papua, West Papua, Maluku, North Maluku and East Nusa Tenggara. These provinces have 8% of Indonesia’s population but over 70% of its Malaria cases.37 In these specific target provinces, people still lack basic preventive measures, receive poor                                                                                                               35 Republic of Indonesia, Village Law Number 6/2014, accessed from http://menpan.go.id/jdih/perundang-undangan/undang-undang. 36 Joint Committee on Reducing Maternal and Neonatal Mortality in Indonesia. 37 Maternal and Child Health and Integrated Malaria Control in Eastern Indonesia revised 3-08-2013. USAID project document regarding incremental funding to immunization in collaboration with CDC. Jakarta2013. USAID-UNICEF Maternal and Child Health Integrated Malaria Control program in Eastern Indonesia 16 diagnosis and inappropriate treatment and are highly mobile and Malaria is an important factor in maternal and newborn health. Survey data estimate about 20,000 deaths annually, although officially reported, deaths due to Malaria are relatively small.38 The USAID-funded MCH/MIP project uses the single disease approach of Malaria, through leverage of Global Fund, to strengthen the efforts to improve MCH and health systems in the remote areas of Eastern Indonesia, supports and facilitates the MoH Malaria and maternal health sub￾directorate’s implementation of integrated antenatal care. Using Malaria control as a wedge to force improvement to health systems in the most remote parts of the archipelago is an effort to maximize indirect beneficial effects of the primary Malaria prevention tool the LLIN, to increase demand for both antenatal health care services and a kind of "quick fix" for rapidly decreasing the MMR in the target provinces.39,40 The Overall goal of the MCH/MIP program is "to accelerate Indonesia’s progress towards attaining MDG5 in Eastern Indonesia, using Malaria control as a wedge to force improvement to health systems in the most remote parts of the archipelago". This goal is consistent with discussions with the Ministry of Health (MoH) on goals for the next several years, and is in line with both the US Government’s Global Health Initiative and UNICEF’s Country Program Action Plan, which aims for development of integrated Malaria control in all highly Malaria-endemic areas of the entire country of Indonesia and reductions in maternal and neonatal mortality in all communities. In order to counter major access and quality of service, monitoring, planning and evaluation, human resources challenges and the lack of community participation and accountability of service providers, the MCH/MIP project supports MoH at sub-national level to implement district/provincial-wide Malaria in pregnancy program by Malaria screening, IPT and distribution of LLINs to pregnant women visiting the firsts ANC, by improving reporting on maternal and neonatal mortality, the referral system and the quality of Basic and Comprehensive Emergency Obstetric and Neonatal Care and referral.41 The specific objectives of the MCH/MIP program are: Objective I: Improve service quality.                                                                                                               38 UNICEF Indonesia. Maternal and Child Health and Integrated Malaria Control in Eastern Indonesia (MIP and ACHIEVE) - Seventh Progress Report (November 2013 - April 2014) May 2014. 39 Ibid. 40 USAID. Report on a joint evaluation of UNICEF's USAID-funded program to control malaria in pregnancy in Eastern Indonesia. Jakarta May 13, 2009. 41     UNICEF Indonesia. Maternal and Child Health and Integrated Malaria Control in Eastern Indonesia (MIP and ACHIEVE) - Sixth Progress Report (May 2013 - October 2013) November 2013. USAID-UNICEF Maternal and Child Health Integrated Malaria Control program in Eastern Indonesia 17 This covers the following original ACHIEVE and MIP objectives: • Objective 1.(MIP): Malaria in Pregnancy (MIP) services provided by midwives and nurses improved. • Objective 2.(ACHIEVE): Ensure that all women and new-borns receive comprehensive and quality care during pregnancy, delivery and postnatal periods. Objective II: Improve health management. This covers the following original ACHIEVE and MIP objectives • Objective 2.(ACHIEVE): Improve district health system management for maternal and neonatal health. • Objective 3.(MIP): Quality monitoring and evaluation, and supervision at provincial, district, and health center level are implemented. Objective III: Use of evidence for implementation. This covers the following original ACHIEVE and MIP objectives: • Objective 4.(ACHIEVE): Enhance management and coordination for policy advocacy and sustainable and effective program outcomes through sharing of experiences and good practices. • Objective 4.(MIP): Support operational research in Indonesia related to the control of Malaria in pregnancy. Project-specific objectives There are two project specific objectives which are not re-formulated into new objectives of the project. They include objective 3 of MIP and objective 1 of ACHIEVE. • Objective 3. (MIP): Integrated Malaria program coverage is increased within and among districts through advocacy and leveraging of local government/GFATM funds. • Objective 1. (ACHIEVE):Improve the district based referral system to ensure accessibility of quality emergency care for women and newborns. There has been no indicator set for this specific objective. This reflects that improving the referral system has not been a focus for UNICEF in this project period. For the remaining years it is expected that the USAID-funded and UNICEF implemented Maternal and Child Health and Integrated Malaria Control (MCH/MIP) in Eastern Indonesia project will continue the following activities.42                                                                                                               42 Maternal and Child Health and Integrated Malaria Control in Eastern Indonesia revised 3-08-2013. USAID project document regarding incremental funding to immunization in collaboration with CDC. USAID-UNICEF Maternal and Child Health Integrated Malaria Control program in Eastern Indonesia 18 1. To support and sustain the control of Malaria in pregnant women and young children by strengthening management, coordination, advocacy, sustainable and effective program outcomes through sharing of experiences and best practices and replication/scaling up of integrated Malaria program coverage among districts through advocacy and leveraging of local Government/GFATM funds and government schemes. 2. To provide technical assistance to improve district-based referral systems to ensure accessibility of quality emergency care for women and newborns and to improve documentation and advocacy for needful policy and program formulation based on the evidence of current and upcoming initiatives. 3. To ensure that all women and newborns receive comprehensive and quality care including interventions to improve nutrition status of children and women during pregnancy, delivery and postnatal periods. For the detailed list of expected project activities: see ANNEX I: Scope of Work, Purpose and Objectives of the Evaluation. 1.3. Scope of Work The purpose of this mid-term performance evaluation is to assess UNICEF’s progress towards their stated objectives and indicators in this project, and to make specific recommendations to improve the project’s performance and to guide future programming. Given the fact that the integrated MCH/MIP program is at a mid-term, there is a need to assess the project's progress; whether the program is well targeted and focused; in-sync and in line with current GoI strategies and actions towards the MDG's; increased its effectivity and finally to assess whether the program indicators still cover program activities, input, process and outcome. USAID's and UNICEF's overall goal is to support the GoI's efforts regarding the MDG’s 4,5 and 6. The results of the mid-term evaluation will provide these three key partners with information on the current "state of the project", which allows midcourse corrections and changes when needed and might guide future programming. The specific objectives of this mid-term evaluation are: 1) to assess the performance of the project against indicators and targets, 2) to make recommendations for the remaining years, 3) and to make observations and recommendations for current efforts or future programming with specific focus on the program indicators. Detailed expected project activities, USAID-UNICEF Maternal and Child Health Integrated Malaria Control program in Eastern Indonesia 19 statement of work, purpose and objectives of the mid-term evaluation are listed in: ANNEX I. These three specific objectives are translated into the following overarching evaluation questions: 1) What progress has been made towards the project’s objectives of improving the quality of Maternal and Child Health and Malaria in Pregnancy programs, and replicating good quality programs broadly? How have UNICEF’s inputs and efforts contributed to the Government of Indonesia’s investments in Maternal and Child Health in Eastern Indonesia, such as the Island Cluster Approach? 2) Is the project appropriately targeted and focused in the breadth of what it does, and how it supports the GoI’s efforts to improve Maternal and Child Health in Eastern Indonesia? 3) How scalable is the impact of UNICEF’s interventions, and how replicable is their approach? 4) Are the project indicators accurately and sufficiently capturing the full scope of the project’s impact? Are the indicators sufficiently specified and appropriately designed to measure the project’s impact? If not, how can they be improved? 5) How successfully have the management and implementation of the two formerly￾separate MIP and ACHIEVE projects been integrated into a single coherent effort, with a unified set of indicators and performance measures? How effectively do the two components achieve synergies and/or leverage each other for greatest impact? USAID-UNICEF Maternal and Child Health Integrated Malaria Control program in Eastern Indonesia 20 2. Methodology 2.1. Overarching framework The CDC Working Group on Evaluation has developed a comprehensive and step-by-step evaluation framework to improve and account for public health and curative actions as pictured in Fig 1 below.43 Figure 1. CDC Evaluation framework for public health and curative actions. Evaluation in general involves procedures that are useful, feasible, ethical, and accurate: this CDC Framework for Program Evaluation in Public Health is a practical, non-prescriptive tool, which summarizes and organizes the steps and standards of an effective program evaluation. We use the above-mentioned CDC framework as a guideline for this mid-term evaluation process and to assess the MCH/MIP progress towards the program objectives. The Maternal and Child Health and Integrated Malaria Control in Eastern Indonesia (MIP and ACHIEVE) has three objectives, which are related to both MIP and ACHIEVE original                                                                                                               43 Framework for Program Evaluation in Public Health. CDC; 1999 [cited 2014 August 2]; Accessed from: http://www.cdc.gov/EVAL/framework/. USAID-UNICEF Maternal and Child Health Integrated Malaria Control program in Eastern Indonesia 21 objectives. In addition, there are two project-specifics objectives: all summarized in Table 1. To asses whether these objectives are met, UNICEF in consultation with the partners (i.e. the government of Indonesia and USAID) set 22 performance indicators. The evaluation team involved UNICEF and stakeholders from province and district levels in the data collection process. This approach may introduce bias to the data collection, as beneficiaries might not want to inform the real situation in the presence of UNICEF and other health offices staff. However, by informing the stakeholders on the objectives and the bottom-up approach of the evaluation, which includes getting their recommendation and creating commitment for the future programming, we assume that the bias is limited. New objectives MIP objectives ACHIEVE Objective Objective 1: Improve Service Quality Objective 1: Malaria in Pregnancy (MIP) services provided by midwives and nurses are improved. Objective 2: Ensure that all women and newborns receive comprehensive and quality care during pregnancy, delivery and postnatal periods. Objective 2: Improve Health Management Objective 2: Quality monitoring and evaluation, and supervision at provincial, district, and health center level are implemented. Objective 3: Improve district health system management for maternal and neonatal health. Objective 3: Use of evidence for implementation. Objective 4: Support operational research in Indonesia related to the control of malaria in pregnancy. Objective 4: Enhance management and coordination for policy advocacy and sustainable and effective program outcomes through sharing of experiences and good practices. Project-specific objectives Objective 3: Integrated malaria program coverage is increased within and among districts through advocacy and leveraging of local government/GFATM funds. Objective 1: Improve the district based referral system to ensure accessibility of quality emergency care for women and newborns. Table. 1.Objectives of the Maternal and Child Health and Integrated Malaria Control in Eastern Indonesia (MIP and ACHIEVE). 2.2. Time line, design, selection of sites and informants The assignment started with the formation of the team and introduction of the team at USAID offices on August 8, and the final report was due on September 16, 2014. The overview of the Time Line and Deliverables is attached as ANNEX II, the Itinerary is USAID-UNICEF Maternal and Child Health Integrated Malaria Control program in Eastern Indonesia 22 attached as ANNEX III and the list of informants as ANNEX IV. Discussions were conducted with UNICEF staff, as well as the PHO/DHO staff after field visits to clarify findings and to extract further information related to the issues discussed. The discussions were also used to share lessons learned right after the data collection in each site. For efficiency reasons the team sometimes split up some evaluation tasks. The team approach characterized this evaluation while team members had a specific task division as documented in ANNEX V. The evaluation team combined quantitative and qualitative approach including data set comparison, documents review, interviews and focused group discussion as explained in the following chapters: data collection/instruments (2.3) and data process/interpretation (2.4). In sequence, the approaches were conducted as follow: A. Desk-base study: review of project reports/documents data sets and other relevant documentation/literature before the start of the mid-term evaluation and during the first week in order to be introduced to the MCH/MIP program. B. Interviews in Jakarta with MoH officials, UNICEF Country Office staff and representatives of other (non) USAID-funded programs in order to understand the context depth of the program and to assess progress, scalability and sustainability of the MCH/MIP program and the coordination/collaboration with other (non) USAID-funded programs. Site visits to three project areas during week 2 and 3 of the evaluation. A) The evaluation started with a desk-study of relevant program documents. The team reviewed documents provided by USAID/Indonesia, and UNICEF/Indonesian Country Office, including the project SOW, routine UNICEF reports and other relevant documents related to specific Indonesian and global MCH and Malaria programs. The main objective of this document review was to compare program progress against the program indicators and targets, which is the first evaluation question. The documents review was also used to develop the data collection instruments for the site visits and interviews in Jakarta. B) Discussions at UNICEF country office were held to get the overall picture of the program implementation, including the challenges faced, efforts taken to overcome challenges, and the program progress against the program indicators/targets. These discussions were also instrumental for the site selection. Interviews were also conducted with related MOH officers at national level to explore UNICEF inputs to the government USAID-UNICEF Maternal and Child Health Integrated Malaria Control program in Eastern Indonesia 23 of Indonesia (GoI) and to assess whether these inputs are in sync and in line with the GoI policies. In addition, the discussion was also used to gather information on challenges for program implementation as well as the program sustainability. The final site selection was conducted in consultation with UNICEF country office as well as with USAID staff. This selection was not at random, but it was designed purposively to get the specific information needed for the evaluation process. For example Maluku was selected to get specific information on the cluster islands approach, which was developed and implemented in Maluku. Papua and West-Papua were also selected for specific reasons, such as to understand the challenges faced by the program due to geographical barriers. The visit to Sorong was planned by the evaluation team to get a good picture of the program implementation in an area with high a prevalence of HIV/AIDS. The selection of sites to be visited took also in consideration the approach being used by UNICEF in providing the specific technical assistance like the use of the "positive deviance approach": model sites developed in some focused districts/health centers to be further adapted and implemented in other districts/health centers. Once the sites were selected, the selection of informants to be interviewed by the evaluation team was in consultation with UNICEF field staff. Considering the limited time available, again, this also was a purposively selection approach. This may also introduce bias to the data collection process. However the data triangulation, by which the evaluation team gathers and combines quantitative and qualitative data from available data sources and from informants at all levels assures that the evaluation team gets the best possible picture. The team interviewed 97 informants (as listed in ANNEX IV) including national and local level government staff, working at both policy and technical levels and USAID staff, UNICEF project staff, other (non) USAID program staff working in MCH and Malaria in Eastern Indonesia. The team also had discussions with 21 Health Center clients (in 3 focused group discussion of 5-8 clients). Overall, the Evaluation team met about 200 people both through the interviews and FGDs, and at the island of Nyafar Nifmas, (Maluku) where the team had a focus group discussion with 8 villagers and conducted an extended village gathering with 85 villagers. 2.3 Data collection and instruments Based on the five core evaluation questions, as presented in the background chapter (1.3) an overall structural questionnaire (ANNEX VI) and specific interview guidelines, for MoH, USAID-UNICEF Maternal and Child Health Integrated Malaria Control program in Eastern Indonesia 24 UNICEF Country Office, DHO/PHO, UNICEF Provincial Offices and clients were developed (ANNEX VII). Informants for each evaluation question were defined based on their relevancy to the questions; while methods of data collection were defined based on the possible number of informants willing to be consulted for the question. The first evaluation question (the program progress against program indicators), for example, was answered by comparing the MCH/MIP program progress against Baseline 2010 and "National 2013 Performance" (meaning a compilation of routine quantitative performance data on the integrated MCH/MIP program indicators in all 5 provinces in Eastern Indonesia) and combined with results from interviews of UNICEF staff at country and provincial office level as well as discussion with beneficiaries in supported districts (qualitative data). The other evaluation questions will be answered based on the results from interviews with stakeholders using the specific interview guidelines (qualitative data) and combined with / compared against the results of the Baseline 2010-Results 2013 comparison. 2.4. Data process and interpretation For the quantitative evaluation: measuring the MCH/MIP program progress on the objectives against established program indicators, the evaluation team selected the best available routine data set. In order to measure this progress the evaluation team opted for maximal reliability by comparing the most recent available performance data (National Performance 2013) against the 2010 baseline indicators: covering a timespan of over 3 years. As a result the evaluation team first developed an overall overview "Progress on Program indicators 2010 baseline - 2013 results" spread sheet (see ANNEX X), to facilitate the analyses of total progress over all indicators per province and the progress per specific indicator per province. For the qualitative analysis: all interviews and focused group discussion, when possible and after consent, were audio-recorded. In addition, a matrix (by informant by question) was developed to put brief notes based on the finding from the interviews and discussions (see ANNEX VIII). Completion of the matrix was as much as possible conducted on a daily basis to avoid missing information. The findings presented in this report are based on the quantitative analysis and qualitative analyses of both the matrix and audio-records. USAID-UNICEF Maternal and Child Health Integrated Malaria Control program in Eastern Indonesia 25 3. Findings In order to fulfill the first specific objective of this evaluation: to assess the performance of the MCH/MIP project and the related overarching core questions (1.3): we start with paragraphs on progress against objectives and indicators followed by the paragraph on the UNICEF inputs and efforts. The next paragraphs describe the program focus and coherence, coordination and harmonization, integration and synergy of MIP and ACHIEVE, followed by the advocacy and leverage of GFATM and other funds, health system challenges and adaptation to MoH policy changes, monitoring and evaluation and ends with a description of sustainability and scalability. 3.1. Progress on objectives against combined ACHIEVE/MIP Indicators For the quantitative assessment of the program, the evaluation team was supplied with 3 data sets: the first set included the combined ACHIEVE/MIP (MCH/MIP) program 2010 baseline/ target indicators 2011-2015, per province and two sets "National Performance 2012 and 2013": covering MCH/MIP program target indicators, results and percentages reached in the 5 Eastern provinces as compiled by UNICEF. However, comparing the most recent available performance data (national performance 2013) against 2010 baseline indicators posed a challenge because of the missing values in both data sets. In order to increase the strength (validity and reliability) of the comparison between the 2010 baseline and the 2013 performance indicators, missing values of the 2010 baseline indicators were imputed using agreed 2011/2012 program target indicators (as stipulated in ANNEX X). As for the "National Performance 2012" this data set has, compared to the 2013 data, besides missing values, also data reporting formats that do not allow comparison with the 2010 baseline. Table 2 offers a summary of progress on overall indicators in 4 quartiles; respectively >/= 100%, 75-100%, 50-75% and /= 100% of target reached 10 7 12 13 9 75-100% 3 1 3 1 2 50-75% 6 7 1 0 3 ancy Proportion of pregnant women attending first: ANC contact who .---rve an insecticide treated bed net Number of Orstricts with budget: a l location for integrated MCH program that is at le ast equal to the total contrbution of 'the GFATM and UNICEF Proportion of pregnant women attending four a ntenatal visit during , i.e;r pcep>ancy Proportion of delivery assisted by skilled birth attendance Proportion of newborns and mothers who received a dlec:k up by ..deified health provider within 2 d ays of delivery ( l/ = 1009' of t arget reached 7 5-100"6 C'JL.75" .ance 3 7% 47 57 21" 172 NA -239' 97 157 62" 977 NA USAID-UNICEF Maternal and Child Health Integrated Malaria Control program in Eastern Indonesia 89 Annex XI Recommendations and Priority Matrix Priority Recommendations Action by MOH UNICEF USAID Nat PHO DHO For the remaining years 1 UNICEF advocates for a buffer stock at DHO-PHO level for Oxytocin, Magnesium sulphate, RDT and ACT and ILLNs, and increases its efforts to strengthening the referral system and the quality of BEONC sites √ √ √ 2 To use the synergy obtained by the integrated program to inform expecting mothers of the benefits of the JKN health insurance scheme, and to promote skilled delivery at an official BEONC site √ 3 To put strong output indicators for leverage, such as ensuring sustainable funding from the local government to support the program √ √ 4 To improve the current routine data collection system and development of specific indicators to measure the results of the integrated of MCH/MIP program √ √ √ 5 To reduce the intake of student at the midwifery school and the intake of new midwives through a "numerous clausus" system, to re-certify midwives having a caseload lower than 100 deliveries per year and to promote "4 hands delivery" √ (with IBI) 6 UNICEF to strengthen its evidence￾based health programming and documentation by increasing operations research (plus budget), systematic documentation and sharing √ USAID-UNICEF Maternal and Child Health Integrated Malaria Control program in Eastern Indonesia 90 of qualitative and quantitative results on the contextual conditions and the effect of intervention programs. 7 UNICEF to support the GoI in selecting appropriate quality of care indicators and finding the mechanism to measure them √ √ √ √ 8 refresher and new training of health professionals to counter staff rotations and to improved facilitative supervision by MoH staff to maintain these skills and ensure quality of services √ 9 To add outcome indicators, based on agreed national indicators for both malaria and MCH √ √ √ 10 To add output indicator for integrated MCH/MIP i.e.: proportion of villages reducing their endemicity level’ and number/percent of pregnant women with presumed/confirmed and treated malaria √ √ √ 11 Follow up the progress of MWH try out in Banten Province √ 12 To align Kinerja and UNICEF specific activities √ For future programming 1 UNICEF to have a stronger focus on strengthening the basic integrated maternal/neonatal health services including safe deliver practices and neonatal resuscitation at Health Center / BEONC level instead of trying to cover the whole spectrum of Maternal and Neonatal health √ 2 To strengthen the monitoring and evaluation with a combination of integrated planning and facilitative supervision to improve quality of care and to guide program planning √ √ √ √ Annex XII 54                                                                                                               54Source:  http://www.who.int/maternal_child_adolescent/documents/measuring-­‐care-­‐ quality/en/ 92 USAID-UNICEF Maternal and Child Health Integrated Malaria Control program in Eastern Indonesia 93 USAID-UNICEF Maternal and Child Health Integrated Malaria Control program in Eastern Indonesia 94 USAID-UNICEF Maternal and Child Health Integrated Malaria Control program in Eastern Indonesia ANNEX XIII List of documents and references (in alphabetical order) 1. Achadi E, Scott S, Pambudi ES, Makowiecka K, Marshall T, Adisasmita A, et al. Midwifery provision and uptake of maternity care in Indonesia. Trop Med Int Health. 2007; 12(12):1490-7 Accessed from: http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Citatio n&list_uids=18076557 2. Australia Support to Primary Health Care Strengthening and Maternal Newborn Health (PERMATA) Program Design. 2014. 3. Badan Perencanaan Pembangunan Nasional, Badan Pusat Statistik, United Nations Population Fund. Proyeksi Penduduk Indonesia (Indonesia Population Projection) 2010 - 2035. Jakarta: Badan Pusat Statistik; 2013. 4. Brabin BJ, Hakimi M, Pelletier D. An analysis of anemia and pregnancy-related maternal mortality. J Nutr. 2001; 131(2S-2):604S-14S; discussion 14S-15S Accessed from: http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Citatio n&list_uids=11160593 5. Brabin BJ, Premji Z, Verhoeff F. An analysis of anemia and child mortality. J Nutr. 2001; 131(2S-2): 636S-45S; discussion 46S-48S Accessed from: http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Citatio n&list_uids=11160595 6. Brownson RC, Fielding JE, Maylahn CM. Evidence-based public health: a fundamental concept for public health practice. Annu Rev Public Health. 2009; 30:175-201 Accessed from: http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Citatio n&list_uids=19296775 7. DFAT Concept Note Accessed from: http://aid.dfat.gov.au/business/Documents/mnch￾conceptnote-indonesia.pdf 8. Elyazar IR, Gething PW, Patil AP, Rogayah H, Kusriastuti R, Wismarini DM, et al. Plasmodium falciparum malaria endemicity in Indonesia in 2010. PLoS One. 2011; 6(6):e21315 Accessed from: http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Citatio n&list_uids=21738634 9. Evi Martha and team. A Review on Maternity Waiting Homes Effectiveness in 5 Provinces in Indonesia. Depok 2013. 10. Freedman LP, Graham WJ, Brazier E, Smith JM, Ensor T, Fauveau V, et al. Practical 95 USAID-UNICEF Maternal and Child Health Integrated Malaria Control program in Eastern Indonesia lessons from global safe motherhood initiatives: time for a new focus on implementation. Lancet. 2007 Oct 13; 370(9595):1383-91 Accessed from: http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Citatio n&list_uids=17933654 11. Framework for Program Evaluation in Public Health. CDC; 1999 [cited 2014 August 2]; Accessed from: http://www.cdc.gov/EVAL/framework/. 12. From Reformasi to Institutional Transformation - A Strategic Assesment of Indonesia's Prospects for Growth, Equity and Democratic Governance: Harvard Kennedy School - ASH Center for Democratic Governance and Innovation; 2011. 13. GoI MoH. Pertemuan Kajian Efektivitas Rumah Tunggu Kelahiran: Accessed from: http://www.gizikia.depkes.go.id/pertemuan-kajian-efektivitas-rumah-tunggu￾kelahiran/?print=pdf. 14. Hammerich A, Campbell OM, Chandramohan D. Unstable malaria transmission and maternal mortality--experiences from Rwanda. Trop Med Int Health. 2002 Jul. ;7(7):573- 6 Accessed from: http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Citatio n&list_uids=12100439 15. Hammerich A, Campbell OM, Chandramohan D. Unstable malaria transmission and maternal mortality--experiences from Rwanda. Trop Med Int Health. 2002; 7(7):573-6 Accessed from: http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Citatio n&list_uids=12100439 16. Heywood P, Nida P Harahap, Mimin Ratminah, Elmiati. Current Situation of Midwives in Indonesia: Evidence from 3 districts in West Java Province. BMC Research Notes. 2010; 3:287 Accessed from: http://www.biomedcentral.com/1756-0500/3/287 17. Hodge A, Firth S, Marthias T, Jimenez-Soto E. Location matters: trends in inequalities in child mortality in Indonesia. Evidence from repeated cross-sectional surveys. PLoS One. 2014. ;9(7):e103597 Accessed from: http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Citatio n&list_uids=25061950 18. Indonesia - Demographic and Health Survey 20122013; Statistic Indonesia, National Population and Family Planning Board, Ministry of Health, Measure DHS, ICF International: Accessed from: http://dhsprogram.com/publications/publication-FR275- DHS-Final-Reports.cfm. 19. Indonesia GHI Country Strategy - Improved Health Impact through Collaboration 2011. 96 USAID-UNICEF Maternal and Child Health Integrated Malaria Control program in Eastern Indonesia 20. Indonesia - Neonatal and Child Health Profile2013: Accessed from: http://www.who.int/maternal_child_adolescent/epidemiology/profiles/neonatal_child/idn. pdf?ua=1. 21. Maternal and Child Health and Integrated Malaria Control in Eastern Indonesia revised 3-08-2013. USAID project document regarding incremental funding to immunization in collaboration with CDC. Jakarta2013. 22. Innovative Approaches to Maternal and Newborn Health Compendium of Case Studies, 2013: Accessed from: http://www.unicef.org/health/files/Innovative_Approaches_MNH_CaseStudies-2013.pdf. 23. Joint Committee on Reducing Maternal and Neonatal Mortality in Indonesia. Reducing Maternal and Neonatal Mortality in Indonesia : Saving Lives, Saving the Future. 2013. 24. Kementerian Kesehatan RI. Buku Saku Pelayanan Kesehatan Neonatal Esensial - Pedoman Teknis Pelayanan Kesehatan Dasar. 2010. 25. Kementerian Kesehatan RI. Assesment GAVI - HSS 2010 - 2011 Direktorat Jenderal Bina Gizi dan KIA: Laporan Akhir Provinsi Papua: Kementerian Kesehatan RI, GAVI Alliance, Universitas Cenderawasih2011. 26. Lazanoet. Progress towards Millennium Development Goals 4 and 5 on maternal and child mortality: an updated systematic analysis. The Lancet. 2010; 378:113-6. Accessed from: 27. Makowiecka K, Achadi E, Izati Y, Ronsmans C. Midwifery provision in two districts in Indonesia: how well are rural areas served? Health Policy Plan. 2008 Jan. ;23(1):67-75 Accessed from: http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Citatio n&list_uids=17971368 28. Manson's Tropical Diseases. 23 ed. Jeremy Farrar et all, editor. London: Elsevier Saunders; 2014. 29. Maternal mortality in 2005: estimates developed by WHO, UNICEF, UNFPA, and The World Bank 2007. 30. Miranti R, Vidyattama Y, Hansnata E, Cassells R, Duncan A. Trends in Poverty and Inequality in Decentralising Indonesia. OECD Social, Employment and Migration Working Papers. 2013. ;148 Accessed from: http://dx.doi.org/10.1787/5k43bvt2dwjk-en 31. North Maluku, Maluku, West Nusa Tenggara, East Nusta Tenggara, West Papua and Papua - Poverty - Percentage of Poor People, Rural [database on the Internet]. knoema.com. [cited September 8, 2014]. Accessed from: 97 USAID-UNICEF Maternal and Child Health Integrated Malaria Control program in Eastern Indonesia http://knoema.com/atlas/embed/Indonesia/North-Maluku/topics/Living￾Conditions/Poverty/percent-of-Poor-People-Rural?compareTo=ID-MA,ID-PB. 32. OUTREACH: Development of Indonesia's Outlying Area -- Anti-Malaria Squads. TEMPO. 2012 19 February:1-7. 33. Papua at a glance: Province - Demography and Health Status 2012. 34. Personal comment during discussion with UNICEF country office staff, August 8 2014. Jakarta. 35. Republic of Indonesia, Village Law Number 6/2014, accessed from http://menpan.go.id/jdih/perundang-undangan/undang-undang. 36. Ronsmans C, Scott S, Qomariyah SN, Achadi E, Braunholtz D, Marshall T, et al. Professional assistance during birth and maternal mortality in two Indonesian districts. Bull World Health Organ. 2009;87(6):416-23 Accessed from: http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Citatio n&list_uids=19565119 37. Rukhsana Ahmed, Din Syafruddin, Jeanne Poepoprodjo, Puji Budi Asih, Endang Sumiwi, Faragher B, et al. Intermittent Screening and Treatment or Intermittent Preventive Therapy for Control of Malaria in Pregnancy in Indonesia: 'STOPMIP' Study. DFID/MRC/Wellcome Trust Joint Global Health Trial Scheme. 2011. 38. Schantz-Dunn J, Nour NM. Malaria and pregnancy: a global health perspective. Rev Obstet Gynecol. 2009; 2(3):186-92 Accessed from: http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Citatio n&list_uids=19826576 39. Soto EJ, Firth S, La Vincente S, Morgan A, Bauze A, Dayal P. Developing an Investment Case for Financing Equitable Progress towards MDGs 4 and 5 in the Asia Pacific Region. Phase 1: Strategic Report 2009. 40. Steketee RW, Nahlen BL, Parise ME, Menendez C. The burden of malaria in pregnancy in malaria-endemic areas. Am J Trop Med Hyg. 2001; 64(1-2 Suppl):28-35 Accessed from: http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Citatio n&list_uids=11425175 41. Supargiyono S, Bretscher MT, Wijayanti MA, Sutanto I, Nugraheni D, Rozqie R, et al. Seasonal changes in the antibody responses against Plasmodium falciparum merozoite surface antigens in areas of differing malaria endemicity in Indonesia. Malar J. 2013;12:444 Accessed from: http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Citatio 98 USAID-UNICEF Maternal and Child Health Integrated Malaria Control program in Eastern Indonesia n&list_uids=24321092 42. Titaley CR, Dibley MJ, Agho K, Roberts CL, Hall J. Determinants of neonatal mortality in Indonesia. BMC Public Health. 2008; 8:232 Accessed from: http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Citatio n&list_uids=18613953 43. Titaley CR, Hunter CL, Dibley MJ, Heywood P. Why do some women still prefer traditional birth attendants and home delivery?: a qualitative study on delivery care services in West Java Province, Indonesia. BMC Pregnancy Childbirth. 2010; 10:43 Accessed from: http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Citatio n&list_uids=20701762 44. Titaley CR, Hunter CL, Heywood P, Dibley MJ. Why don't some women attend antenatal and postnatal care services?: a qualitative study of community members' perspectives in Garut, Sukabumi and Ciamis districts of West Java Province, Indonesia. BMC Pregnancy Childbirth. 2010; 10:61 Accessed from: http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Citatio n&list_uids=20937146 45. UNICEF. Annual Report 2012 for Indonesia, EAPRO: Accessed from: http://www.unicef.org/about/annualreport/files/Indonesia_COAR_2012.pdf 46. UNICEF. Sharpening the equity focus: Selected Innovations and lessons learned from UNICEF-assisted programmes 2009-20102011. 47. UNICEF Indonesia. Integration of Malaria Control with Maternal and Child Health in Eastern Indonesia – Second Progress Report (1 June - 30 November 2011) 30 November 2011. 48. UNICEF Indonesia. ACHIEVE - Reducing Maternal and Neonatal Mortality in Indonesia: The Pathway to Achieving MDG 4&5 - Second Progress Report (1 May - 31 October 2011) 30 November 2011. 49. UNICEF Indonesia. Maternal and Child Health and Integrated Malaria Control in Eastern Indonesia (MIP and ACHIEVE) - Third Progress Report (November 2011 - April 2012) May 2012. 50. UNICEF Indonesia. Integration of Malaria Control with Maternal and Child Health in Eastern Indonesia - First Progress Report (October 2010 - April 2011) 31 May 2011. 51. UNICEF Indonesia. Integration of Malaria Control with Maternal and Child Health in Eastern Indonesia - End of year report (1 January 2010 to 31 December 2010) 17 December 2010. 99 USAID-UNICEF Maternal and Child Health Integrated Malaria Control program in Eastern Indonesia 52. UNICEF Indonesia. Indonesia - Annual Report 2013. 53. UNICEF Indonesia. Maternal and Child Health and Integrated Malaria Control in Eastern Indonesia (MIP and ACHIEVE) - Seventh Progress Report (November 2013 - April 2014) May 2014. 54. UNICEF Indonesia. Maternal and Child Health and Integrated Malaria Control in Eastern Indonesia (MIP and ACHIEVE) - Fifth Progress Report (November 2012 - April 2013) May 2013. 55. UNICEF Indonesia. Maternal and Child Health and Integrated Malaria Control in Eastern Indonesia (MIP and ACHIEVE) - Sixth Progress Report (May 2013 - October 2013) November 2013. 56. UNICEF Indonesia. Joint Progress Report on Australia-UNICEF Education Assistance to Papua and West Papua and Striding towards Equity in Education in Papua and West Papua (STEP) - Second Progress Report (1 March 2011 - 31 March 2012) 20 April 2012. 57. UNICEF Indonesia. ACHIEVE - Reducing Maternal and Neonatal Mortality in Indonesia: The Pathway to Achieving MDG 4&5 - First Progress Report (October 2010 - April 2011) 31 May 2011. 58. UNICEF Indonesia. DEMOGRAPHY: Demographic profile, inequality characteristics and mortality in Papua and West Papua provinces. MICS Further Analysis Series. 2011. Accessed from: 59. UNICEF Indonesia. EDUCATION: Disparities in educational outcomes in West Papua and Papua. MICS Further Analysis Series. 2011. Accessed from: 60. UNICEF Indonesia. THE GAPS: Social Disparities in West Papua and Papua. MICS Further Analysis Series. 2011. Accessed from: 61. Issue Briefs: Maternal and Child Health [database on the Internet]. UNICEF Indonesia, 2012 [cited August 7, 2014]. Accessed from: http://www.unicef.org/indonesia/A5- _E_Issue_Brief_Maternal_REV.pdf. 62. USAID. Report on a joint evaluation of UNICEF's USAID-funded program to control malaria in pregnancy in Eastern Indonesia. Jakarta May 13, 2009. 63. USAID. How-To-Note: Preparing Evaluation Reports in Publisher. Number 1, Version 1.02012. 64. USAID. Investing in Indonesia: A stronger Indonesia advancing national and global development - Country Development Cooperation Strategy for Indonesia 2014 – 2018: USAID2013. 100 USAID-UNICEF Maternal and Child Health Integrated Malaria Control program in Eastern Indonesia 65. WHO. A Strategic Framework for Malaria Prevention and Control in Africa: Accessed from: http://www.who.int/malaria/publications/atoz/afr_mal_04_01/en/. 66. WHO Bulletin. Editorial. 2005 April; 83(4) 67. WHO Global Malaria Programme. Malaria: Global Fund proposal development (Round 11) - WHO Policy Brief 2011: Accessed from: http://www.who.int/Malaria/publications/atoz/Malaria_gf_proposal_dev_who_policy_bri ef_201106.pdf. 68. WHO Jakarta. Review of the National Malaria Control Programme of the Republic of Indonesia. 201; (September): Accessed from: http://apps.who.int/iris/bitstream/10665/94367/1/2012- 08_SEARO.ReviewMalariaControl%20Prog.Indonesia%20201.pdf. 69. WHO. International Malaria profiles: Indonesia Accessed from: http://www.who.int/Malaria/publications/country-profiles/profile_idn_en.pdf. 70. Wickramasuriya. Malaria and Ankylostomiasis in the Pregnant Woman. Oxford.: Oxford University Press; 1937. 71. World Bank. Indonesia Economic Quarterly - Slower Growth; High Risks2013: Accessed from: http://www.worldbank.org/content/dam/Worldbank/document/EAP/Indonesia/IEQ￾Dec13-ENGLISH.pdf. 72. World Malaria Report - Indonesia2013: Accessed from: http://www.who.int/malaria/publications/country-profiles/profile_idn_en.pdf?ua=1. 101 USAID-UNICEF Maternal and Child Health Integrated Malaria Control program in Eastern Indonesia Statement regarding conflict of interest No conflicts of interest exist among the investigators of this Mid-Term Evaluation of the USAID-funded UNICEF Maternal and Child health and Integrated Malaria Project in Eastern Indonesia. Lucas Pinxten Team Leader Siti Nurul Qomariyah Irma Anintya Tasya