This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Social Impact, Inc. THE USAID JALIN PROJECT MIDTERM EVALUATION FINAL REPORT Final Report PHOTO: DE FOR USAID JALIN (DELETE THIS BLANK PAGE AFTER CREATING PDF. IT’S HERE TO MAKE FACING PAGES AND LEFT/RIGHT PAGE NUMBERS SEQUENCE CORRECTLY IN WORD. BE CAREFUL TO NOT DELETE THIS SECTION BREAK EITHER, UNTIL AFTER YOU HAVE GENERATED A FINAL PDF. IT WILL THROW OFF THE LEFT/RIGHT PAGE LAYOUT.) DISCLAIMER: The authors' views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. THE USAID JALIN PROJECT MIDTERM EVALUATION FINAL REPORT Program Title: The USAID Jalin Project USAID Office: USAID/Indonesia Contract Number: AID-497-C-17-00001 Contractor: DAI Global, LLC Evaluation Conducted By: The Developmental Evaluation of the USAID Jalin Project Social Impact, Inc. Contract Number: 72049718C00001 Date of Submission: Final Report This publication was produced at the request of the United States Agency for International Development. i | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV ABSTRACT This report presents the results of a midterm evaluation of the USAID/Indonesia Jalin Project, conducted by Social Impact, Inc. The evaluation addresses questions about progress toward work plan objectives, regional activities, monitoring and evaluation, project design, stakeholder needs and priorities, and project management. The evaluation team answered these questions using a combination of remote data collection methods, including remote key informant interviews, virtual workshops, and reviews of existing data, during the Coronavirus Disease 2019 pandemic, which prohibited in-person data collection. USAID Jalin has made limited progress toward its objective of contributing to Indonesia’s national goals of accelerated maternal and newborn mortality reduction in its three years of implementation. The Project’s primary successes have been engaging subnational actors in cocreating local solutions and partnerships to improve maternal and newborn health outcomes and collaborating with the Ministry of Health through a secretariat embedded there. Findings do not suggest that USAID Jalin’s challenges are inherent to its design, which features an innovative cocreation approach that is based on lessons learned from past projects and the needs of the Indonesian health sector. This report offers recommendations for course corrections in the Project’s remaining base period and for its option years if exercised. ii | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV TABLE OF CONTENTS ABSTRACT .......................................................................................................I TABLE OF CONTENTS.................................................................................II TABLES AND FIGURES............................................................................... III ACRONYMS AND ABBREVIATIONS .......................................................IV EXECUTIVE SUMMARY ............................................................................VIII INTRODUCTION............................................................................................1 USAID JALIN BACKGROUND.....................................................................................1 EVALUATION PURPOSE AND QUESTIONS ...........................................................2 USERS & UTILIZATION................................................................................................2 METHODOLOGY............................................................................................3 EVALUATION DESIGN.................................................................................................3 METHODS.......................................................................................................................3 DATA ANALYSIS AND LIMITATIONS ......................................................................4 FINDINGS ........................................................................................................4 EQ 1: PROGRESS TOWARD OBJECTIVES................................................................4 EQ 3: REGIONAL ACTIVITIES...................................................................................15 EQ 2: STAKEHOLDER PRIORITIES...........................................................................23 RECOMMENDATIONS ................................................................................25 ANNEX 1: JALIN PROJECT COMPONENTS...........................................29 ANNEX 2: PRIOR DE ACTIVITIES FOR PAST OBSERVATIONS........30 ANNEX 3: DOCUMENTS REVIEWED ......................................................32 ANNEX 4: USAID JALIN PROGRESS TOWARD YEAR 3 WORK PLAN MILESTONES.................................................................................................34 ANNEX 5: KEY ACHIEVEMENTS IN YEAR 1 ANNUAL REPORT ......38 OVERALL PROGRESS .................................................................................................38 SUCCESSES ..................................................................................................................38 ANNEX 6: KEY ACHIEVEMENTS IN YEAR 2 ANNUAL REPORT ......40 ANNEX 7: LOCAL SOLUTION DATABASE ...........................................43 ANNEX 8: USAID JALIN PLANS FOR TRANSITIONING AND SCALING LOCAL SOLUTIONS....................................................................................52 ANNEX 9: USAID JALIN FEEDBACK ON THE MIDTERM FINAL EVALUATION REPORT ..............................................................................56 iii | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV TABLES AND FIGURES Table 1: Midterm Evaluation Limitations and Mitigations Strategies.............................................................. 4 Table 2: Years 1 and 2 Intended Results............................................................................................................... 5 Table 3: USAID Jalin Performance Indicator Results according to Annual Reports.................................. 6 Table 4: Progress on Year 3 MELA Plan Performance Indicators................................................................... 9 Table 5: USAID Jalin Local Solutions Status Over Time .................................................................................17 Table 6: Local Solutions by IRs..............................................................................................................................17 Table 7: Opportunities to Better Integrate Local Solutions with the new Results Framework ...........18 Table 8: Links Between Local Solutions National Programs and Regulations............................................21 Table 9: Key Results and Challenges Reported by Regional Teams in Workshops.................................22 Figure 1: USAID Jalin Results Framework ............................................................................................................ 2 Figure 2: No. of Participants in KIIs (blue) and Workshops (red).................................................................. 3 Figure 3: Indicators to Achieve or Exceed Annual Targets in Year 1............................................................ 6 Figure 4: Indicators to Achieve or Exceed Annual Targets in Year 2............................................................ 7 Figure 5: Indicators to achieve or exceed targets ............................................................................................10 Figure 6: Map of MOH Priority Districts, USAID Jalin local solutions, and field visits by the DE for USAID Jalin.................................................................................................................................................................16 iv | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV ACRONYMS AND ABBREVIATIONS AMMDes Alat Mekanis Multiguna Pedesaan (Rural Multipurpose Mechanical Vehicle) AMP Audit Maternal dan Perinatal (Maternal and Perinatal Audit) AMP-SR Audit Maternal dan Perinatal Surveilans dan Respons (Maternal and Perinatal Audit Surveillance and Response) ANC Antenatal Care APINDO Asosiasi Pengusaha Indonesia (Indonesian Employers Association) ARSSI Asosiasi Rumah Sakit Swasta Indonesia (Indonesia Private Hospitals Association) BAPPEDA Badan Perencanaan Pembangunan Daerah (Local Development Planning Agency) BAST Berita Acara Serah Terima (Handover Certificate) BOK Biaya Operasional Kesehatan (Health Operational Cost) BPJS Badan Penyelenggara Jaminan Sosial (Indonesian Social Insurance Administration Organization) CDCS Country Development Cooperation Strategy COP Chief of Party COVID-19 Coronavirus Disease 2019 CSO Civil Society Organizations CSR Corporate Social Responsibility DAI Development Alternatives, Inc. DAK Dana Alokasi Khusus (Special Allocation Fund) DCOP Deputy Chief of Party DE Developmental Evaluation DG Directorate General DHO District Health Office DO Development Objective DPMD Dinas Pemberdayaan Masyarakat dan Desa (District Village and Community Empowerment Office) EMAS Expanding Maternal and Newborn Survival Project EMNC Every Mother and Newborn Counts EQ Evaluation Question FOPKIA Forum Peduli Kesehatan Ibu dan Anak (Forum for Maternal and Newborn Health) GIS Geographic Information System GOI Government of Indonesia GP2SP Gerakan Pekerja/Buruh Perempuan Sehat Produktif (Healthy and Productive Women Workers Movement) HQ Headquarters IBI Ikatan Bidan Indonesia (Indonesian Midwives Association) ICT Information and Communication Technology IDAI Ikatan Dokter Anak Indonesia (Indonesian Pediatricians Society) IDHS Indonesia Demographic Health Survey IDI Ikatan Dokter Indonesia (Indonesian Doctors Association) IMAN Integrated Mentoring and Accreditation National v | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV IP Implementing Partner IR Intermediate Result J2SR Journey to Self-Reliance Jamilah Program Jemput Antar Ibu Hamil Bermasalah (Shuttle Service for Pregnant Women with Complication) Jampersal Jaminan Persalinan (Insurance for Delivery) JKN Jaminan Kesehatan Nasional (National Health Insurance) JNPK Jejaring Nasional Pendidikan Kesehatan (National Network for Health Education) KEMENDESA Kementerian Desa, Pembangunan Daerah Tertinggal, dan Transmigrasi (Ministry of Villages, Development of Disadvantaged Regions, and Transmigration) KESGA Direktorat Kesehatan Keluarga Kementerian Kesehatan (MOH Family Health Directorate) KESJAOR Direktorat Kesehatan Kerja dan Olahraga (MOH Directorate of Occupational Health and Sport) KIA Kesehatan Ibu dan Anak (Maternal and Newborn Health) KII Key Informant Interview KM Knowledge Management M&E Monitoring and Evaluation MAB Multi-stakeholder Advisory Body MELA Monitoring, Evaluation, Learning, and Adaptation MERLA Monitoring, Evaluation, Research, Learning, and Adaptation MMR Maternal Mortality Rate MNH Maternal and Newborn Health MOH Ministry of Health MOU Memorandums of Understanding MPDN Maternal and Perinatal Death Notification MPDSR Maternal and Perinatal Death Surveillance and Response MUYAN Mobil Unit Kendaraan Keluarga Berencana (Family Planning Contraception Services Mobile Unit) NGO Non-Governmental Organization NMR Neonatal Mortality Rate P4K Program Perencanaan Persalinan dan Pencegahan Komplikasi (Birth Preparedness and Complication Readiness Program) PASTI Pekerja Aktif Sehat Bugar & Produktif (Productive, Active and Fit Workers Program) PE/E Pre-Eclampsia/Eclampsia PENAKIB Forum Penurunan Kematian Ibu dan Bayi (Forum for the Reduction of Maternal and Newborn Mortality) PERBUP Peraturan Bupati (District Regulation) PERDATIN Perhimpunan Dokter Spesialis Anestesiologi dan Terapi Intensif (Association of Anesthesiologist and Intensive Therapy) PERSI Perhimpunan Rumah Sakit Seluruh Indonesia (Hospital Association of Indonesia) PHO Provincial Heath Office PMB Praktik Mandiri Bidan (Private Midwife Practice) PMP Performance Management Plan vi | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV PNC Postnatal Care POGI Perkumpulan Obstetri dan Ginekologi Indonesia (Indonesian Obstetrics and Gynecology Association) PONEK Pelayanan Obstetrik Neonatal Emergensi Komprehensif (Comprehensive Emergency Obstetric and Newborn Care) PPJK Pusat Pembiayaan Jaminan Kesehatan (Center for Health Insurance and Financing) PPSDM Badan Pengembangan dan Pemberdayaan Sumber Daya Manusia (Human Resources Development and Empowerment Agency) PROMKES MOH Health Promotion Directorate PSE Private Sector Engagement PT. KMWI PT. Kreasi Mandiri Wintor Indonesia PT. STI PT. Sijarimas Teknologi Inovasi PUSDATIN Pusat Data dan Informasi Kementerian Kesehatan (MOH Data and Information Center) PUSKA-UI Pusat Penelitian Keluarga Sejahtera Fakultas Kesehatan Masyarakat Universitas Indonesia (Center for the Study of Family Welfare, University of Indonesia) Puskesmas Pusat Kesehatan Masyarakat (Primary Health Care Centre) RAKERKESNAS Rapat Kerja Kesehatan Nasional (National Health Work Planning Meeting) RF Results Framework RFP Request for Proposal RPJMN Rencana Pembangunan Jangka Menengah Nasional (National Midterm Development Plan) RSUD Rumah Sakit Umum Daerah (Regional General Hospital) SI Social Impact, Inc. SiCantik Sidoarjo Cegah Angka Kematian Ibu dan Anak (Sidoarjo Prevents Maternal and Children Mortality Rate) SijariEMAS Sistem Informasi dan Komunikasi Jejaring Rujukan Maternal & Neonatal (Information and Communication System for Maternal and Neonatal Referral Network) SISRUTE Sistem Rujukan Terintegrasi (Integrated Referral System) SIPB Surat Izin Praktik Bidan (Midwife Practice License) SOP Standard Operating Procedure SOW Statement of Work STRB Surat Tanda Registrasi Bidan (Midwife Registration Certificate) STTA Short-Term Technical Assistance TBD To be determined TKKSD Tim Koordinasi Kerjasama Daerah (Districts/Municipalities Cooperation Coordination Team) TOC Theory of Change TOPDeAMOR Tukang Ojek Puskesmas dengan Elektronik Ambulans (Puskesmas Ojek Driver with Motorcycle Ambulance) TOR Terms of Reference UHC Universal Health Coverage USAID United States Agency for International Development USG United States Government vii | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV VFC VF Corporation VoIP Voice over Internet Protocol WEALTH Women’s Health and Wellbeing project WHO World Health Organization ZOOM Zero Mother Mortality Preeclampsia viii | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV EXECUTIVE SUMMARY The United States Agency for International Development (USAID) Indonesia tasked Social Impact, Inc. (SI) with conducting a midterm evaluation of the USAID Jalin Project. USAID Jalin supports the Government of Indonesia (GOI) Ministry of Health (MOH) to accelerate the reduction of preventable maternal and newborn deaths during a period of performance from September 2017 to March 2021. This evaluation was undertaken through SI’s Developmental Evaluation (DE) of USAID Jalin and its team of embedded evaluators, who have conducted continuous data collection and learning for USAID, the MOH, and the Project during a period of performance from June 2018 to September 2021. This evaluation’s purpose is to identify how USAID, the implementing partner (IP), and other project stakeholders can improve USAID Jalin in the future and to document achievements and lessons learned midway through its implementation by addressing the following evaluation questions (EQs): EQ1: To determine how to improve USAID Jalin in the future, to what extent did the Project achieve objectives from its previous work plans? And, based on available data, to what extent is USAID Jalin likely to achieve its current work plan’s objectives and contribute to key MOH maternal and newborn health (MNH) strategies? EQ2: Does USAID Jalin’s programming meet the needs and priorities of key project stakeholders (including but not limited to MOH, provincial and district health offices, professional associations, project regional and national partners, and other relevant national GOI ministries and institutions)? EQ3: How should USAID Jalin adapt its regional activities to best contribute to the new results framework in its current work plan? And, which local solutions should the Project continue in the future? To improve this report’s logic and narrative, the EQs do not appear in numerical order (see below). EVALUATION DESIGN AND SCHEDULE The evaluation used a mixed-methods approach with a document review, key informant interviews (KIIs), and lessons learned workshops. The DE also drew on two years of past observations from being embedded with USAID Jalin. All data were collected remotely as the evaluation coincided with the global Coronavirus Disease 2019 (COVID-19) pandemic, prohibiting in-person data collection. The DE reviewed 381 plans, reports, and other documents, conducted 105 remote KIIs with 106 respondents, and held six workshops with 24 total participants from USAID Jalin’s six regional teams. In April 2020, USAID issued the Statement of Work (SOW) for this evaluation, and the DE began designing data collection instruments and reviewing documents and conducting KIIs. It continued KIIs and held workshops in May before finishing data collection and presenting preliminary findings in June. FINDINGS AND CONCLUSIONS EQ 1 – PROGRESS TOWARD OBJECTIVES USAID Jalin made limited progress toward its Year 1 and 2 work plan objectives: “Contribute to national goals to reduce maternal and newborn mortality and provide catalytic support to improve essential maternal and newborn services reaching the poorest and most vulnerable.” ● USAID Jalin produced two key successes during Years 1 and 2 that continued into Year 3: o The engagement of over 300 subnational actors through at least 71 local solutions to improve MNH outcomes developed by the Project’s six regional teams in the provinces of Banten, Central Java, East Java, North Sumatera, South Sulawesi, and West Java. Engaging subnational actors supports the Mission’s anticipated 2020-2024 Country Development Cooperation Strategy (CDCS). ix | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV o Strengthened coordination and collaboration with the MOH fostered through the Jalin Secretariat embedded within the Ministry. ● Implementation in Year 3 has been affected by many of the challenges from Years 1 and 2, a short period of performance thus far, and the COVID-19 pandemic. ● USAID Jalin is unlikely to achieve most of its Year 3 Work Plan results by December 2020, the end of the work plan period, or by March 2021, the end date of the Project’s base period. o Mentoring guidelines, a deliverable for the Work Plan’s Intermediate Result (IR) 1, were behind schedule as of this evaluation. This had delayed the MOH’s national mentoring program. o The Project’s IR 2 work on Maternal and Perinatal Death Surveillance and Response (MPDSR) has produced an MPD notification application. It had not yet begun involving regional teams in Audit Maternal and Perinatal Surveillance and Response (AMP-SR), which the MOH requested. o Designing an interoperable information and communication technology (ICT) referral system under IR4 was behind schedule as of this evaluation. Furthermore, it is unclear that developing an interoperable platform alone will significantly improve the referral system. o Through local solutions like the Rural Multipurpose Mechanical Vehicle (Alat Mekanis Multiguna Pedesaan/AMMDes) Ambulance Feeder, Gerai Kesehatan Ibu dan Anak (KIA), Floating Ambulance, Village Funds and Cross-Border Referrals, the Project had made progress under its IR 5, strengthening emergency referral networks. ● USAID Jalin is not comprehensively supporting the 65 MOH priority districts. The Project has local solutions in 15 of these districts, which do not support all elements of MOH’s strategy. EQ 3 – REGIONAL ACTIVITIES ● USAID Jalin did not comprehensively integrate local solutions into the Year 3 Work Plan and new results framework. ● Regional teams have not begun to implement new national programs from this Work Plan due to a lack of clear and actionable guidance from the IP’s Jakarta headquarters (HQ). ● The MOH expects USAID Jalin to rapidly begin supporting the national GOI priorities including in the Year 3 Work Plan IRs 1-5 at the regional level. ● The Project began handing over local solutions to partners and stakeholders in 2020. Stakeholders prefer a more gradual transition than what is occurring with ongoing support for facilitating partnerships, monitoring and evaluation (M&E), and national-level advocacy. ● If local solutions are transitioned as envisioned, local partners will sustain activities, but replication and scaling are unlikely to occur in most cases. ● USAID Jalin’s plans to scale local solutions focus on basic activities, like developing guidelines and holding workshops, which are unlikely to lead to sustainable MNH impact. The Project has plans to replicate around ten local solutions in new districts. EQ 2 – STAKEHOLDER PRIORITIES ● USAID Jalin has improved its engagement and collaboration with the MOH since Year 1. ● The Project improved its relationship with the MOH by better explaining how its approach supported GOI priorities through new staff who previously worked at MOH and by establishing the Jalin Secretariat (see EQ 1). ● The MOH has unfulfilled expectations for USAID Jalin’s regional support for national programs under IRs 1-4 of the Year 3 Work Plan. This report’s EQ 1 Findings Section describes MOH expectations for each IR and makes related recommendations in the Recommendations Section. x | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV ● The Project has adequately addressed the needs and priorities of counterparts and stakeholders at the provincial and district levels, including provincial health offices (PHOs), district health offices (DHOs), other local government actors, and professional actors. ● USAID Jalin’s Multi-Stakeholder Advisor Body (MAB) has been underutilized as a source of advice, networking and advocacy. RECOMMENDATIONS USAID Jalin should immediately take the following steps through March 2021: • Rapidly accelerate mentoring activities – a potential Project legacy – to initiate activities in vertical hospitals in October 2020 and in provincial and district hospitals in November 2020. • In line with MOH expectations, ensure that the mentoring program is designed to be carried out by local governments in the future. • Commence regional support for the Audit Maternal Perinatal (AMP) in line with the MOH’s expectations, ensuring that an AMP team is fully functioning in priority districts. • Regional teams should ensure that an AMP team is functional within their areas of jurisdiction and capable of ensuring accurate and timely audits of maternal and newborn deaths. • Work with the Indonesian Midwives Association (Ikatan Bidan Indonesia/IBI) to increase the availability of practical guidance for midwives to meet the Social Security Agency (Badan Penyelenggara Jaminan Sosial/BPJS) requirements for a midwife to have access to the BPJS payment system. • Complete a rapid strategic assessment on existing ICT platforms and applications by March 2021. • Readjust and disseminate new work plan timelines for implementing and transitioning local solutions, acknowledging that COVID-19 and other delays have put the Project behind schedule. • Design and implement responsible transition guidelines for each local solution with instructions at the provincial level on who should be involved, what technical components and areas should be included, and financial aspects including budget allocations and unit costs. • Develop and implement rapid plans to scale the AMMDes Ambulance Feeder, MNH in the Workplace, Village Funding, and Cross-Border Referral local solutions, which closely align with key MOH priorities and can be completed within six months. The Recommendations Sections identifies national programs and first steps for USAID Jalin to take to leverage these at scale. USAID Jalin may also consider the following steps through March 2021: • Facilitate increased coordination on the national mentoring program between key MOH counterparts at the Family Health Directorate (KESGA) and the Directorate of Health Services for Quality and Accreditation. • At the subnational level, develop mechanisms for supervision and monitoring for mentoring and practical guidance explaining guidelines and individual steps for mentors and mentees hospitals. • Provide the Project’s regional teams with updates on KESGA’s expectations for supporting AMP, guidance on best practices for AMP, and technical support on the MPDN application. • Foster engagement with government institutions, financing agencies, BPJS, the Indonesian Obstetrics and Gynecology Association (Perkumpulan Obstetri dan Ginekologi Indonesia/POGI), and the Indonesian Pediatricians Society (Ikatan Dokter Anak Indonesia/IDAI) to gain support for improvements in the quality of care and services of private midwives. • Encourage IBI to empower midwives to comply with administrative requirements and develop real-time monitoring systems on their registrations and license statuses. xi | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV • Design an advocacy strategy for ensuring hospitals’ commitment for investing in human resources for managing information and providing real-time information for referral systems. • Deliver a final report on each local solution being implemented (as opposed to those that the Project discarded or never developed beyond ideas) as part of transitioning these activities. • Continue to collect M&E data on local solutions and implement as many planned assessments as possible to generate evidence to support replication and scaling. • Prepare a systematic review of each USAID Jalin province that captures lessons learned from implementing regional activities over the last three years. 1 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV INTRODUCTION The United States Agency for International Development (USAID) Indonesia tasked Social Impact, Inc. (SI) to conduct a midterm evaluation of the USAID Jalin Project (hereafter “USAID Jalin” or “the Project”). USAID Jalin is implemented by Development Alternatives, Inc. (hereafter “DAI” or “the IP”) and its consortium of partners – IntraHealth, Vital Strategies, and MarketShare Associates – to support the Government of Indonesia (GOI) Ministry of Health (MOH) to accelerate the reduction of preventable maternal and newborn deaths. This midterm evaluation was undertaken through SI’s Developmental Evaluation of USAID Jalin (DE), where SI’s team of embedded evaluators conduct continuous data collection and provide innovative M&E services to USAID, the MOH, and the Project. These M&E services generate lessons learned and support rapid course corrections to help achieve USAID Jalin’s objectives. While the evaluation team is embedded within the Project to support real￾time learning and adaptive management, the DE is a separate contract from USAID Jalin to maintain objectivity. For the IP’s feedback on this midterm evaluation, please see Annex 9. USAID JALIN BACKGROUND In 2017, USAID/Indonesia initiated USAID Jalin, which means “intertwined” in Bahasa Indonesia, because the Project intends to weave together GOI, private sector, civil society, and other stakeholders to create sustainable solutions for improving MNH. USAID Jalin has a three-year base period from September 8, 2017, to March 7, 2021 with two potential option years through September 7, 2022.1 The Project’s goal is to contribute to Indonesia’s national goals to reduce maternal and newborn mortality and provide catalytic support to improve essential MNH services reaching the poorest and most vulnerable. Its current theory of change is: The Project is aligned with USAID’s global priority initiative Preventing Child and Maternal Deaths and USAID/Indonesia’s Country Development Cooperation Strategy (CDCS). It supports Development Objective (DO) 2 from the 2014-2019 CDCS: Essential human services to target populations improved and sustained.2 USAID Jalin provides national MNH support through its HQ office in Jakarta and local assistance through six regional offices in the provinces of Banten, Central Java, East Java, North Sumatera, South Sulawesi, and West Java.3 USAID Jalin has undergone several iterations in design and implementation since its inception. In September 2019, USAID reduced the total estimated cost from $55 million to $35 million and revised the SOW to focus on the primary causes of preventable maternal and newborn mortality. The Project’s current results framework (RF) is: 1 In May 2020, USAID granted DAI a no-cost extension, extending USAID Jalin’s three-year base period end date from September 7, 2020 to March 7, 2021. As of this mid-term evaluation in July 2020, USAID had not decided whether to exercise the Project’s first option year. 2 USAID Jalin will support the Mission’s new CDCS 2020-2024. 3 USAID removed the provinces of Papua, West Papua, Maluku, and north Maluku from the Project and added Banten through a contract modification. IF we strengthen both the use of evidence and reporting and accountability systems to inform targeted 'whole-of-market' responses relating to maternal and newborn deaths, AND IF in particular, we strengthen public and private facilities to provide high quality MNH services including identifying and treating obstetric and newborn complications, THEN the emergency referral systems and the quality of care for mothers and newborns will improve and the number of deaths will drop. 2 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV Goal: Contribute to national goals to accelerate maternal and newborn mortality reduction. Result 1: Improved MNH quality of care in public and private primary and secondary facilities, at scale Result 2: Improved efficiency and effectiveness of the emergency referral system at community and facility levels, at scale IR 1: Sustained improvements in practice by clinical hospital teams in treating obstetric and newborn complications IR 2: Stakeholder strategies more focused on main causes of maternal and neonatal death IR 3: Sustained improvements in quality of care by private midwives IR 4: Integrated ICT referral platform scaled IR 5: Improved referral networks scaled Figure 1: USAID Jalin Results Framework In response to the spread of novel COVID-19 to Indonesia, USAID approved an Addendum to USAID Jalin’s Year 3 Work Plan outlining emergency response activities to prevent excess maternal and newborn morbidity and mortality. EVALUATION PURPOSE AND QUESTIONS The purpose of this midterm evaluation is to identify how USAID, the IP, and stakeholders can improve USAID Jalin and to document learning and achievements. This midterm evaluation fulfills USAID Operational Policy requirements mandated by the Project’s contract. This midterm evaluation delivers findings, conclusions, and recommendations that address the following EQs: EQ 1: To what extent did Jalin achieve objectives from its previous work plans? And, based on available data, to what extent is Jalin likely to achieve its current work plan’s objectives and contribute to key MOH MNH strategies? EQ2: Does Jalin’s programming meet the needs and priorities of key project stakeholders (including but not limited to MOH, provincial and district health offices, professional associations, project regional and national partners, and other relevant national GOI ministries and institutions)? EQ3: How should Jalin adapt its regional activities to best contribute to the new results framework in its current work plan? And, which local solutions should Jalin continue in the future? To improve this report’s logic and narrative, the EQs do not appear in numerical order. This report presents EQ 1 first to address USAID Jalin’s overall progress and then EQ 3 on its regional activities. The report then assesses EQ 2 on USAID Jalin’s alignment with stakeholder priorities. USERS & UTILIZATION The primary users of this evaluation are the USAID/Indonesia Health Office, the MOH, and USAID Jalin. These users may apply its findings, conclusions, and recommendations to inform current programming and the design of future MNH activities. USAID Jalin’s successes and challenges may also contribute to learning across USAID/Indonesia’s portfolio. Secondary users include USAID/Washington and USAID Jalin’s partners and stakeholders at the national and regional levels as they reflect on their progress. The DE intends to promote the utilization of findings and recommendations through its ongoing M&E and learning activities with USAID, MOH, USAID Jalin, the Project’s MAB, and other stakeholders. 3 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV METHODOLOGY EVALUATION DESIGN The DE used a mixed-methods approach for the evaluation, using qualitative and quantitative data collection methods to answer the EQs. These data collection methods included a desk review, with quantitative analyses, remote KIIs, and virtual lessons learned workshops with USAID Jalin’s regional teams. The DE drew on its two years of past observations prior to this evaluation and answered the EQs with a combination of these methods, conducting content analysis and triangulating evidence to confirm and corroborate findings. ADAPTATIONS FOR COVID-19 The COVID-19 pandemic constrained the midterm evaluation of USAID Jalin, because public health guidelines and measures taken by USAID, the DE, and the IP to protect staff prevented in-person data collection, travel, and site visits. The DE overcame these constraints by adjusting the evaluation design so that all data were collected remotely. Adaptations for remote data collection allowed the team to collect relevant and timely data while maintaining public health standards. METHODS Past Observations: The DE applied past observations gathered through two years of DE data collection prior to this evaluation (see Annex 2 for a list of DE’s prior activities). Many findings emerged from the DE’s regional travel and site visits to USAID Jalin activities during 2018-2020 (see the map in EQ 3), which COVID-19 prevented during this midterm evaluation. Desk review: The DE conducted a desk review of 381 assessments, plans, reports, studies, and other documents related to USAID Jalin and its activities (see Annex 3 for a list of reviewed documents). It collected, organized, and synthesized this information, which included quantitative analysis of performance data and publicly available budget data. Remote KIIs: The SOW for this evaluation anticipated that the DE would complete 75 KIIs. Despite challenges associated with COVID-19, the DE exceeded this target, conducting 105 remote KIIs with 106 respondents, including USAID staff, MOH officials, current and former Project staff (termed Jalin KIIs), and national and regional partners, and stakeholders (see graphic right). Forty-five percent of respondents were female, and 55% were male. Five planned respondents were unavailable for interviews. The evaluators used non￾probability purposive and snowball sampling to identify and interview respondents who were most knowledgeable about USAID Jalin. The DE used Voice over Internet Protocol (VoIP) services, including WhatsApp and Microsoft Teams, and telephone calls to conduct remote KIIs. Evaluators administrated four KII protocols tailored to respondent groups to explore interviewee’s experiences through open-ended and probing questions. The DE conducted KIIs in English or Bahasa Indonesia depending on respondent preference. In most cases, a two-person team conducted KIIs, with one person interviewing and another taking notes. In cases where one evaluator was present, the evaluator recorded the interview to increase notetaking accuracy. Regional Workshops: The DE conducted virtual lessons learned workshops with each of USAID Jalin’s six regional teams to review activity progress and identify strengths, weaknesses, and opportunities for implementation and learning. The workshops were held using Microsoft Teams to enable staff to participate remotely. The DE circulated agendas to regional teams in advance and used Figure 2: No. of Participants in KIIs (blue) and Workshops (red). 4 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV PowerPoint presentations to guide the workshops. One evaluator facilitated sessions, and another took notes on participants’ responses. DATA ANALYSIS AND LIMITATIONS The DE applied sequential data analysis, first analyzing data from the desk review and past observations, and then using this to inform the design of KII instruments and workshop guides. The DE triangulated data from primary data collection using desk review and past observation data to validate qualitative data. The DE applied content analysis in an Excel-based framework, coding and tallying qualitative data from KIIs and workshops to determine themes and their relative strength across respondent groups. The DE used a matrix to organize the data and outline this report’s structure. Table 1 describes this evaluation’s limitations and the mitigation strategies employed to overcome them. FINDINGS EQ 1: PROGRESS TOWARD OBJECTIVES This section evaluates USAID Jalin’s overall progress toward objectives in the following work plans: 1. Year 1 Work Plan for 2018, approved by USAID in April 2018. 2. Year 2 Work Plan for 2019, approved by USAID and MOH in January 2019. Table 1: Midterm Evaluation Limitations and Mitigations Strategies LIMITATIONS MITIGATION STRATEGIES Limited time for USAID Jalin to implement its Year 3 Work Plans: USAID Jalin had approximately eight months to implement its Year 2 and Year 3 work plans between when USAID approved it in November 2019 and the conclusion of data collection for this evaluation in June 2020. This report’s assessment of the Year 3 Work Plan only refers to this eight-month period. This evaluation acknowledges this limitation and qualifies the related findings under EQ 1 on the Year 3 Work Plan accordingly. The DE identified the overall trends of USAID Jalin’s implementation under this workplan and believes its analysis is as precise as the timeframe allows. COVID-19 interruptions to Year 3 implementation: COVID-19 began to affect USAID Jalin in March 2020, hindering its ability to meet in-person with partners. Some counterparts lacked time and resources to engage with the Project as they focused on the pandemic. This slowed some activities, and the Project began additional MNH emergency response tasks. This midterm evaluation acknowledges this limitation and qualifies the related findings under EQ 1 and 3 accordingly. The DE notes how COVID-19 affected USAID Jalin and identifies successes and challenges during this period to help improve future programming. COVID-19 effects on this evaluation: COVID- 19 public health guidelines and travel restrictions prevented in-person data collection through KIIs and workshops. In-person data collection can yield more data than remote data collection, because it allows evaluators to observe body language and interactions between participants. The DE implemented these methods remotely using VOIP services and telephone calls. The DE also used past observation data, many from site visits, to ensure that field data were considered in this evaluation. Attribution issues: USAID Jalin does not engage in service delivery activities. Many of its programs involve facilitating MNH partnerships. This role could complicate attributing health outcomes directly to the Project’s interventions. This evaluation combines its four methods of a desk review, KIIs, workshops, and past observations to corroborate findings to strengthen its ability to identify potential causal links between the Project and health outcomes. Recall bias: This midterm evaluation covers Jalin’s design through implementation from 2014 to 2020. Participants in KIIs and workshops may experience differences in accuracy and completeness of recollections when describing events and perceptions over this long period. Questions within data collection instruments made explicit references to the specific period in question to help informants frame their responses. In addition, evaluators triangulated KII and workshop data with the desk review and past observations, which are unaffected by recall bias. USAID and Jalin staff turnover: Both USAID, MOH, and the IP have experienced staff turnover since the Project began in September 2017. The DE overcame this through snowball sampling, asking respondents to refer other interviewees who were involved in the early periods of USAID Jalin. 5 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV 3. Year 2 Modified & Year 3 Work Plan (hence forth called the Year 3 Work Plan) for September 2019-December 2020, approved by USAID in November 2019 and MOH in December 2019. 4. An Addendum to the Year 3 Work Plan for COVID-19 Activities for April-September 2020, approved by USAID in April 2020. USAID Jalin made limited progress toward its Year 1 and 2 work plan objectives and is unlikely to achieve most of its Year 3 Work Plan key results (see Annex 4) by December 2020, the end of the work plan, or March 2021, the end of the Project’s base period. The Project’s slow progress in Years 1 and 2 was the result of challenges developing and securing approval for work plans, using and generating evidence, engaging the private sector, and managing and coordinating regional teams. Despite these challenges, USAID Jalin produced two key successes during Years 1 and 2 that continued into Year 3: the engagement of subnational actors by regional teams through local solutions to improve MNH outcomes; and engagement with MOH fostered through the Jalin Secretariat embedded there. The Project’s implementation in Year 3 has been affected by a short period of performance and the COVID-19 pandemic. YEARS 1 AND 2 PERFORMANCE Objectives: The objective of USAID Jalin’s Year 1 and Year 2 work plans was to: “Contribute to national goals to reduce maternal and newborn mortality and provide catalytic support to improve essential maternal and newborn services reaching the poorest and most vulnerable.” The Year 1 Work Plan describes its results in terms of components and changes in behaviors and processes, and the Year 2 Work Plan included four results (see Table 2 below). Table 2: Years 1 and 2 Intended Results YEAR 1 WORK PLAN YEAR 2 WORK PLAN RESULTS BEHAVIORS AND PROCESSES RESULTS 1. Partnership process managed to generate, implement, evaluate, and scale successful local solutions and utilize critical evidence to result in sustainable MNH impact, which has two sub- components: 1.1 MNH evidence available and compellingly communicated. 1.2 Constructive and inclusive partnerships, solutions, and advocacy developed. 2. Technical assistance deployed effectively to support impact at scale. 3. Catalytic financing efficiently disbursed to support the incubation and implementation of local solutions. Multi-stakeholder actors demand and utilize MNH evidence. Collaboration occurs across multi-stakeholder actors as evidenced by co-investment and coordinated implementation. Multi-stakeholder groups actively co-invest to improve MNH outcomes. The Government of Indonesia integrates MNH evidence into programming that targets the poorest and most vulnerable. The Government of Indonesia and private sector providers deliver quality, evidence-informed MNH programming that reaches the poorest and most vulnerable. Communication channels use MNH evidence and priorities to increase public awareness. Multi-stakeholder actors adopt and influence evidence-based system changes. More Indonesians utilize MNH evidence-informed clinical and non-clinical services. Multi-stakeholder actors organize into new networks and take on leadership roles within the system to improve MNH outcomes. Evidence on MNH in Indonesia collected, synthesized, and made available in a variety of formats, and communicated to a variety of audiences, raising awareness, and yielding more clearly defined problems with actionable areas for solution development; A broader knowledge base and more effective policy and program implementation within the MOH; Increased mobilization of domestic public and private resources for MNH; and Locally sustainable, evidence-informed solutions and partnerships developed and scaled. 6 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV Year 1 Activities: The Year 1 Work Plan envisioned the Project partnering with “with multi-stakeholder actors to co-create and co-invest in sustainable evidence-based solutions aimed at reducing maternal and newborn mortality” via “local solutions across six technical focus areas.” Its components focused on a partnership process for sustainable MNH impact, technical assistance to support impact at scale, and catalytic financing. Year 2 Activities: The Year 2 Work Plan expounded on the Year 1 vision for cocreation by mandating the development of problem statements and seeking to target fewer, more strategic stakeholders in partnerships to “generate local solutions with maximum potential to positively impact MNH outcomes.” It intended to produce “more actionable problems and promising local solutions” to be “replicated and/or scaled-up during Year 3.” Lack of Results: USAID Jalin reported a lack of results from its Years 1 and 2 activities according to its M&E plans. The Project used a Year 1 Monitoring, Evaluation, Research, Learning, and Adaptation (MERLA) Plan, approved by USAID in June 2018, and a Year 2 MELA Plan, approved by USAID in March 2019. The Project did not achieve the annual targets in Year 1 and 2 for most performance indicators (see table below). It achieved the annual targets for more output indicators than outcome indicators and did not report on impact indicators (see text box above and below on the next page). INDICATORS THAT ACHIEVED OR EXCEED ANNUAL TARGETS IN YEAR 1 (RESULT / TARGET) • MNH Knowledge Management (KM) Strategy in Place: Yes • MNH evidence disseminated (articles, etc.): 275 / 8 • Synthesize latest MNH Evidence from Evidence Summit, GOI evidence, Jalin lessons, and other USAID MNH partners: Yes • MNH evidence slide deck available for use in co￾creation, design, and TA: Yes • Events and targeted outreach held to share evidence and engage multi-stakeholder actors: 67 / 10 • Problem statements defined: 65 / 12 • Co-creation events held: 14 / 8 • Co-creation design sessions via targeted partnerships and outreach: 39 / 5 • Solutions or advocacy initiatives identified because of co-creation events or targeted outreach: 21 / 5 • Number of innovations (solutions) supported through United States Government (USG) assistance: 8 / 6 • Number of civil society organizations (CSOs) receiving USG assistance engaged in advocacy interventions: 6 / 4 • MAB meetings held quarterly: 4 / 4 Figure 3: Indicators to Achieve or Exceed Annual Targets in Year 1 Table 3: USAID Jalin Performance Indicator Results according to Annual Reports Year 1 MERLA Plan Indicators No. to achieve or exceed annual targets in Year 1 Percentage to achieve or exceed annual targets Impact 4 0 0% Outcome 3 0* 0% Output 18 7** 39% Milestone 8 5 63% Subtotal Year 1 33 12 36% Year 2 MELA Plan Indicators No. to achieve or exceed annual targets in Year 2 Percentage to achieve or exceed annual targets Outcome 12 2 17% Output 12 5 42% Subtotal Year 2 24 7 29% Total for Years 1 and 2 57 19 33% *This does not count three indicators that reported 0 toward a Year 1 targets of 0. ** This does not count two indicators that reported 0 toward a Year 1 target of to be determined (TBD). 7 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV YEARS 1 AND 2 CHALLENGES USAID Jalin faced challenges in Years 1 and 2 developing and securing approval for work plans, using and generating information and evidence, engaging the private sector, and managing and communicating with regional teams. Deliverable Development: The Project required a significant amount of time to develop its work plans, MELA plans, and other deliverables, including data management and communications plans, in Years 1 and 2. For example, it required eight months, from September 2017 to April 2018 to develop and approve the Year 1 Work Plan. Evidence: USAID Jalin encountered difficulties generating and applying evidence to inform its programming. Past observations showed that the Project did not consistently socialize sources for this information like the MNH Evidence Summit, Rapid Evidence Synthesis, Banten II study, and IDHS. Regional workshops mentioned that the Project did not use evidence from Every Mother and Newborn Counts (EMNC) survey in cocreation, because the results arrived after they designed most activities in 2018. Private Sector Engagement: The Project also did not engage private sector partners to secure co-financing in Years 1 and 2. To date, Jalin has leveraged approximately USD 240,000 from the private sector, according to USAID/Indonesia’s Annual Report 2019. This includes VF Corporation (VFC) approved funding of approximately USD 175,000 for work in nine VFC-affiliate factories. According to the desk review, most of USAID Jalin’s activities do not contain co-financing commitments, and the Year 1 Annual Report stated: “appealing to the private sector as a development partners is difficult.” USAID Jalin’s private sector engagement (PSE) focused on corporate social responsibility (CSR) rather than investment. REGIONAL ENGAGEMENT OF SUBNATIONAL ACTORS USAID Jalin’s primary success has been the engagement of subnational actors by regional teams to improve MNH outcomes, which began in Year 1 and continued through Year 3. Engagement in Cocreation: Regional teams engaged 311 different stakeholders at the national, provincial, and district level in cocreating local solutions through 536 meetings and event and fulfilled 18 directives from PHOs, as of September 2019. This engagement and the local solutions that it generated comprise most of the Project’s work to date that directly affect mothers and newborns. According to past observations, USAID Jalin generated 71 local solutions in these six regions between July 2018 and August 2019. In 2019, the DE found that the Project was implementing 17 of these, designing another 17, and had discarded or postponed 37 ideas for solutions. Stakeholder Support: Counterparts and stakeholders believe that regional teams are the Project’s key feature. The MOH said USAID Jalin’s key feature was its regional teams’ ability to support GOI national programming at the district level in meetings with the DE in September and December 2019. A DHO KII respondent said, “I respect the Jalin Program because it [involves] other agencies at the district level such as Local Development Planning Agency (Badan Perencanaan Pembangunan Daerah/Bappeda), the Office of Information and Communication, the Port Administration, and others… making it easier for us to plan” (see EQ 2 for more on stakeholder’s needs and priorities). Strategic: USAID Jalin’s engagement of subnational actors and regional programs advance the Mission’s anticipated 2020-2024 CDCS, according to a presentation by USAID to IPs in January 2020. This CDCS will support “geographic targeting” and “subnational development” in eight “high-priority provinces,” including Banten, East Java, and South Sulawesi, and other “Urban/Health focus provinces,” including North Sumatera, West Java, and Central Java. USAID Jalin’s engagement of subnational actors INDICATORS THAT ACHIEVED OR EXCEED ANNUAL TARGETS IN YEAR 2 (RESULT / TARGET) • # of MNH evidence products disseminated: 142/84 • % of USAID Jalin technical assistance provided to stakeholders that receive positive feedback: 100% / 85% • # of MAB members participating in USAID Jalin-related activities: 14 /12 • # of stakeholders involved in co-creating solutions to improve MNH outcomes: 1,284 / 90 • # of Universal Health Coverage (UHC) areas supported by USG investment: 2 /2 • # of innovations supported through USG assistance: 18 / 12 • # of actors investing in USAID Jalin facilitated/ promoted solutions to improve MNH outcomes: 79 / 60 Figure 4: Indicators to Achieve or Exceed Annual Targets in Year 2 8 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV and regional programs also align with the Indonesia’s RENCANA AKSI NASIONAL (National Action Plan 2020-2024) and National Midterm Development Plan (Rencana Pembangunan Jangka Menengah Nasional/RPJMN) 2020 – 2024. JALIN SECRETARIAT USAID Jalin’s other key success has been engagement with the MOH fostered through the Jalin Secretariat embedded with KESGA. MOH KIIs suggested that the Jalin Secretariat has improved MOH coordination with USAID and the Project. An MOH respondent said, “we are grateful to USAID and Jalin because… we can have any information needed any time through the secretariat.” Another interviewed said, “Because of the Secretariat, good communication has been established.” Support for MOH: The desk review indicated that a prominent component of the Secretariat’s work is administrative, including “securing letters of support” and “the submission and approval of all outstanding USAID health portfolio Handover Certificate (Berita Acara Serah Terima/BAST) reports” (Year 2 Annual Report). MOH KIIs and the desk review mentioned that KESGA utilizes the Secretariat to oversee administrative and financial matters when conducting meetings in the absence of their MOH administrative staff. The desk review and Jalin KIIs showed that the Secretariat supported the first meeting of the US Ambassador and the new Minister of Health and meetings between USAID and the Directorate General (DG) of Public Health that positioned USAID Jalin to support the GOI President’s priorities. The Secretariat also situated the Project to assist the MOH MNH Working Group established in 2019. MOH Expectations: However, the MOH expects Jalin Secretariat’s to improve its coordination between different divisions in MOH, including between KESGA and Health Services Quality and Accreditation Directorate, Referral Health Services Directorate, and Primary Health Services Directorate. KIIs with the Center for Health Financing and Insurance (Pusat Pembiayaan Jaminan Kesehatan/PPJK) suggested that the Jalin Secretariat lacks the fulltime personnel to handle additional coordination tasks. One respondent said, “Even though the funds for Jalin are under KESGA. PPJK is in a position to be involved in its activities… Jalin should report well to PPJK as well.” The Directorate for Quality and Accreditation suggested that the Secretariat facilitate its collaboration with UN agencies and Non-Governmental Organizations (NGOs), but Jalin KIIs indicated that Secretariat staff are fully applied just with their current responsibilities. Strategic: Like Jalin’s regional engagement, the Jalin Secretariat is well situated to support the 2020- 2024 CDCS, which aims to “bolster host country engagement throughout program cycle.” The Jalin Secretariat consists of only two persons: its Government Relations Specialist who also works as the Project’s Portfolio Manager for Quality of Care, and its Office Administrator. As this evaluation was being revised in August 2020, the DE observed that USAID Jalin did not renew multiple staff contracts, including those working at the Secretariat. It was unclear at the time if the Project intended to continue or disband the Secretariat. If not carefully managed, these changes had the potential to disrupt coordination with the MOH because these staff had built relationships with the counterpart. YEAR 3 WORK PLAN PROGRESS TO DATE USAID Jalin’s Year 3 Work Plan revised its objective to “Contribute to national goals of accelerated maternal and newborn mortality reduction.” It also revised its results to prioritize the technical focus areas of quality of care and the emergency referral system. Lengthy Redesign: The process of redesigning these results and technical approach lasted approximately eight months, beginning with the Mission’s Portfolio Review in May 2019 and concluding with the approval of the Year 3 Work Plan by USAID in November and MOH in December 2019. During this time, KIIs and workshops revealed that regional teams continued implementing local solutions, while senior HQ staff and short-term technical assistance (STTA) spent a significant amount of time responding to a Request for Proposal (RFP) issued by USAID and designing the new work plan. Status of Regional Activities: USAID and the IP communicated extensively about regional 9 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV programs. According to past observations, the IP proposed eliminating regional teams and activities in the redesign. A Jalin KII said, “We thought we were going only national. Then local solutions came back due to the DE.” USAID Jalin’s Year 3 Work Plan shifted “from a bottom approach to top down,” according to a USAID KII. A Jalin respondent said, “Jalin now finally has a national program to take to the regions, but it depends on the MOH and Jalin's support.” According to a MOH KII, this “Formulation of the new framework of Jalin was developed in coordination with MOH and related stakeholders.” Limited Progress Thus Far: USAID Jalin has made limited progress toward its Year 3 Work Plan objectives and is unlikely to achieve most key milestones by December 2020, the end of the work plan, or March 2021, the end of the Project’s base period. The Project has not achieved annual targets in Year 3 for most performance indicators (see table below); however, please note that USAID Jalin has only reported data through June 2020, and the annual targets are set for September 2020. Please see the text box on the next page for the five indicators that have achieved their targets. Staff turnover: As noted in the Jalin Secretariat Section above, the DE observed that USAID Jalin did not renew multiple staff contracts in June-August 2020 and hired new positions. This turnover has the potential to further delay Year 3 Work Plan implementation, because those whose contracts were not renewed include the Deputy Chief of Party for technical activities, all staff at the Jalin Secretariat, and some personnel responsible for IR 1-3. IR 1: SUSTAINED IMPROVEMENTS IN PRACTICE BY CLINICAL HOSPITAL TEAMS IN TREATING OBSTETRIC AND NEWBORN COMPLICATIONS USAID Jalin has proposed an approach combining mentoring and accreditation to improve the quality of maternal and newborn services. This new mentoring approach aims to coach front-line hospitals and health centers to follow standard operating procedures (SOPs) and move toward compliance with technical guidance. Combining mentoring with improving clinical services and accreditation, called IMAN (Integrated Mentoring and Accreditation National), requires reconciling concepts from KESGA on clinical technical guidelines and from the Directorate of Health Services for Quality and Accreditation on aspects associated with hospital and health services facility accreditation. Table 4: Progress on Year 3 MELA Plan Performance Indicators Year 3 MELA plan indicators No. to achieve or exceed targets for year 3 (sept. 2020) by June 2020 * Percentage to achieve or exceed annual targets Result 4 0 0% Intermediate result 9 1 11% Output 9 2 22% USAID performance management plan (PMP) 5 0 0% USAID MCH 4 0 0% USAID COVID-19 5 2 40% Total 36 5 14% *This does not count three indicators that reported 0 toward Year 3 targets of 0. 10 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV The Project has encountered challenges integrating these different needs and perspectives on mentoring and accreditation from KESGA, Health Services Quality and Accreditation Directorate, and professional associations. One MOH KII respondent, said “Jalin needs to synthesize different concepts of mentoring guidelines” from these actors and “needs to build communication between them.” The time necessary to synthesize different concepts has delayed finalizing mentoring guidelines, a key deliverable for IR 1. The Year 3 Work Plan anticipates concluding this task in March 2020. But, as of this evaluation, KESGA and the Director of Health Services Quality and Accreditation had not finalized the guidelines. The delay finalizing the mentoring guidelines has caused corresponding delays initiating MOH-led mentoring at vertical and private hospitals and rolling the program out to hospitals in priority districts across Indonesia, which has not started as of this evaluation. Furthermore, USAID Jalin’s regional teams and stakeholders have not yet begun work on the mentoring program. The Year 3 Work Plan also aims to initiate mentoring in seven vertical hospitals by March 2020, which remains pending as of this evaluation. The MOH has requested that USAID Jalin immediately “begin the process of refocusing regional activities to support mentoring, such as recording the condition of health facilities in their area.” Regional teams reported not having received sufficient, clear, and actionable information on the mentoring programs since a visited by the Chief of Party (COP) in December 2019, according to regional workshops. Regional support for IMAN is critical because “the guidelines compiled by the MOH are only a reference. Regions and districts need to adapt these based on their conditions” per a PHO respondent. MOH Expectations for IR 1: • The mentoring program needs to begin at the provincial and district levels as soon as possible. Pushing the program from the national level to the regional level requires support from USAID Jalin and its regional teams. • USAID Jalin needs to facilitate the involvement of the Directorate of Health Services for Quality and Accreditation in addition to KESGA in order to advance the program. • The Project needs to deliver clear and precise guidelines with detailed steps and explanations of the activities that hospital mentors and mentees will perform. • In the future, the MOH envisions that mentoring can be run by local governments with little or no involvement from the central government. • At the subnational level, USAID Jalin needs to strengthen mentoring supervision and monitoring to ensure that improvements in clinical and managerial governance comply with SOPs. IR 2: STAKEHOLDER STRATEGIES MORE FOCUSED ON MAIN CAUSES OF MATERNAL AND NEONATAL DEATHS USAID Jalin intends to strengthen the MPDSR intervention promoted by the World Health Organization (WHO) and partners since 2013 through a MPDN application and by supporting data use to enhance accountability. MPDN provides an initial notification of a maternal death, which is an integral part of the AMP-SR. AMP-SR involves a clinical audit of deaths and action based on those results to improve policy and quality of care. USAID Jalin developed the MPDN application and commenced piloting on schedule by March 2020. However, COVID-19 has delayed field testing the application. Most work on MPDSR by USAID Jalin has been conducted by its HQ rather than regional teams. Successful MPDN implementations will strengthen AMP-SR implementation at the district level; INDICATORS TO ACHIEVE OR EXCEED TARGETS FOR YEAR 3 (SEPT 2020) BY JUNE 2020 (RESULT / TARGET) • % of Jalin priority districts in which at least one agreement is in place to implement transport solutions: 3.08% / 2% • % of target health facilities in (65) Jalin priority districts reporting maternal and perinatal deaths through MPDN application: 30.8% / 25% • Number of mothers using the supported transport solutions, accessing BPJS and/ or village funds for the journey: 133 / 100 • Number of innovations supported through USG assistance: 22 / 22 • # of mechanism to facilitate two-way communication with affected communities: 2 / 2 • # of health workers who received COVID-19-related training in IPC and/or WASH: 194 / 194 Figure 5: Indicators to achieve or exceed targets 11 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV however, additional challenges are confronting AMP-SR at the field level. In KIIs, a PHO respondent said, “Some of districts are discipline to submit their reports, but there are still undisciplined.” Regional workshops found that of the 24 districts in South Sulawesi only 10 implemented AMP, and in West Java, only 10 percent of maternal deaths had been audited. A stakeholder noted the importance of involving local branches of professional associations, such as POGI, saying “out of the 70 cases audited, lessons learned from the AMP have had no impact on service delivery.” Most of Jalin efforts on IR 2 have focused on developing a MPDN application, new AMP-SR guidelines, and reviewing indicators associated with the Maternal Mortality Rate (MMR) and Neonatal Mortality Rate (NMR). There is a need for activities by USAID Jalin to support implementing MPDN and AMP￾SR regionally at the field level. In KIIs, the MOH indicated that they expect USAID Jalin to use regional teams to support AMP-SR. One MOH respondent described the next steps for regional teams as “participating in monitoring AMP implementation which is carried out regularly every three months.” MOH Expectations for IR 2: • USAID Jalin needs to empower its regional teams to support the implementation of AMP-SR at the district level. • Regional support by the Project is necessary for its work on MPDN, AMP-SR guidelines, and indicators to have practical impact on maternal and neonatal mortality. PHO and other subnational government actor expectations: • USAID Jalin needs to strengthen and support MPDN at the local level given how recently it has been developed. • The Project should build understanding of the AMP to reinforce zero tolerance on preventable deaths and the need to review and learn from AMP results regularly to achieve this objective. • USAID Jalin should play a role in assessing and strengthening district AMP teams. The Neonatal Intensive Care Unit of Ciawi Hospital in Bogor, West Java, a USAID Jalin partner in its cross-border referral local solution (Photo: DE). IR 3: SUSTAINED IMPROVEMENTS IN QUALITY OF CARE BY PRIVATE MIDWIVES The Project works with private midwives by partnering with IBI to better deliver maternal and newborn services and to identify, manage, and refer emergency complications. USAID Jalin achieved strong engagement with several of IBI’s provincial branches in 2018 and 2019 and nationally in 2020 12 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV per KIIs. The Project completed a review of the Bidan Delima program by March 2020, a key work plan milestone. USAID Jalin is working to strengthen the link between Bidan Delima and access to credit. Yet, stakeholder KIIs indicated that access to the BPJS payment system is a higher priority. A BPJS respondent said, “The status of Bidan Delima is not the main thing. But, recommendations from the DHOs and IBI are an important part for the BPJS’s requirement.” Midwifes must have a network with health facilities, clinics, or other health service providers for BPJS access, which USAID Jalin’s current program does not address. BPJS requirements on midwife registration certificate (Surat Tanda Registrasi Bidan /STRB) and the midwife practice license (Surat Izin Praktik Bidan/SIPB), corresponds to MOH Regulation No 28/2017 on the License and Practice of Midwives. Human Resources Development and Empowerment Agency of MOH (Badan Pengembangan dan Pemberdayaan Sumber Daya Manusia /PPSDM) estimated that there are around 800,000 midwives and only 400,000 have this certificate. Others either do not have the certificate or their certificate has expired. USAID Jalin, using local national STTA in July-October 2019, assessed access to credit for Private Midwife Practice (Praktik Mandiri Bidan/PMB) to improve skills and clinical equipment. This assessment reported that PMB’s clinical facilities as a business institution have limited management capabilities and lack financial management. These conditions will limit access to credit to improve facilities’ quality, construction, and upkeep. The PMB needs to work with financial institutions in both business administration and in repayment capabilities. PMB could potentially access special loan funds to improve their quality of services if there is policy support from key stakeholders. IBI considers that this policy support would be necessary from the MOH, IBI, and BPJS at the central level, and PHOs and DHOs at the subnational level according to IBI KIIs. As with IR 1 and 2, most work on IR 3 to date has been conducted by HQ rather than regional teams. The Year 3 Work Plan included an access to finance activity for private midwives and clinics in North Sumatera. However, HQ took over responsibility for this activity from the regional team. Much of the Project’s COVID-19 work focuses on increasing midwife capacity for providing MNH services through webinars. The Project has a related performance indicator on whether these webinars increase midwives’ skills, but it had not reported on the indicator as of this evaluation. MOH Expectations for IR 3 • KESGA expects USAID Jalin to provide comprehensive updates on each activity conducted under IR 3. MOH notes that the Project’s work with private midwives will influence the achievement of increasing access for MNH services under the MOH Policy and Action Plan 2020-2024. • PPSDM-MOH expects a solution to the problem that many midwives have not been certified (STRB and SPIB) or that their STRB has expired. IBI Expectations for IR 3: • The Project should engage and gain the support of government institutions, financing agencies, BPJS, and professional organizations like POGI and IDAI for improvements in the quality of care and services of private midwives. IR 4: INTEGRATED ICT REFERRAL PLATFORM SCALED USAID Jalin supports the development of an integrated MOH ICT platform to improve referral systems to reduce the possibility for multiple referrals and delayed emergency responses. The Year 3 Work Plan anticipates having an ICT platform reviewed with recommendations shared with the MOH and an interoperable system designed by March 2020. These milestones remain pending as of this evaluation. Jalin’s activities to develop an interoperable ICT platform are unlikely to resolve existing systems’ low utilization and the causes of inefficient multiple referrals. According to past observations and KIIs, district-level referral systems and the nationwide Integrated Referral System (Sistem Rujukan 13 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV Terintegrasi/SISRUTE) prove inefficient because they often lack an administrator in each health facility to provide real-time updates on readiness. Moreover, this system is more applicable for general medical emergencies than pregnant women with emergency complications. A stakeholder KII stated, “there is often a need for emergency services from the puskesmas to the referral hospital, but it does not receive a response from the hospital… because there was no admin to forward the information.” A PHO respondent said, “multiple referrals often occur due to limited and emergency information facilities in the districts of Lebak, Serang and Pandeglang because of internet connection.” Multiple and delayed referrals are also caused by a limited number of competent health facilities and workers to handle emergencies. A MOH KII stated, “the delay in referrals relates to the limited facilities and personnel who are ready when needed.” Other factors could contribute to USAID Jalin’s integrated ICT platform being underutilized such as uneven hospital distribution, patients without identification or health insurance, the health insurance reimbursement system especially for private hospitals, and limited investment for computers and other devices. The East Java regional team met its work plan target of piloting the Sidoarjo Prevents Maternal and Children Mortality Rate (Sidoarjo Cegah Angka Kematian Ibu dan Anak/SiCantik) application for managing health records on schedule. MOH Expectations for IR 4: • If USAID Jalin designs a new program to strengthen the existing system, it should not duplicate or overlap with the existing system. • MOH Data and Information Center (Pusat Data dan Informasi Kementerian Kesehatan /PUSDATIN) stressed that the Project should consider integrating additional systems in its interoperable ICT needs assessment, such AMP and statistical systems noting births and deaths. IR 5: IMPROVED REFERRAL NETWORKS SCALED These activities intend to strengthen emergency referral networks by facilitating formal agreements between facilities and improving transport to referral facilities for emergency cases. Many local solutions developed in Years 1 and 2 and implemented by regional teams are included in IR 5 of the Year 3 Work Plan. These have made progress, notably including: ● Cross-border referrals (West Java): the regional team mapped health facility capacities, and memorandums of understanding (MOUs) between health facilities were signed on schedule. ● Ambulance Feeder (Banten): the regional team successfully advocated for the Head of Lebak District to allocate funding for ten ambulance feeders. They deployed two of these to the Puskesmas of Bojongmanik and according to the local puskesmas KII, “The AMMDes ambulance feeders significantly help the referral process for pregnant women especially from villages with poor infrastructure.” The MOH Directorate of Health Service Facilities included the AMMDes Ambulance Feeder in its Technical Guideline for Ambulances in 2019. ● Gerai KIA (Banten): USAID Jalin has found that this local solution “increases the number of antenatal care (ANC) visits by pregnant women, although it needs further study.” While the DHO and Forum for Maternal and Newborn Health (Forum Peduli Kesehatan Ibu dan Anak/FOPKIA) have taken ownership of implementing this activity, the Banten regional team intends to continue some involvement to monitor it performance. ● Floating Ambulance (South Sulawesi): The Governor of South Sulawesi has demonstrated ownership of the floating ambulance in Pangkep District, replicating it in the Pare-Pare and Sinjai districts. According to a PHO KII, plans exist to expand it to Luwuk Timur, Selayar, and Makassar districts. Past observations indicate that, while the cost of the floating ambulance and its paramedic crew is high, the local government has found funding for its operation. ● Village Funds (Central Java, South Sulawesi, Banten, North Sumatera, and West Java): Prior to COVID-19, the regional teams had success advocating local governments to allocate village funding for MNH expenses. Per one PHO, a “District Regulation (Peraturan Bupati/PERBUP) 14 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV has been issued confirming village funds will support the MMR and NMR reduction programs.” However, KIIs also indicated that local government were redirecting MNH funding to COVID-19 prevention and response and cash assistance to affected communities. MOH Expectations for IR 5: • USAID Jalin should integrate local solutions with national programs where possible, such as joining Jamilah with Desa Siaga or formally including MNH in the Workplace across all Healthy and Productive Women Workers Movement (Gerakan Pekerja/Buruh Perempuan Sehat Produktif/GP2SP) programs nationally. Please see EQ 3 for more information. PHO and DHO Expectations for IR 5: • USAID Jalin’s MNH in the Workplace, Jamilah-AMMDEs, Cross Border Referrals, Utilization Village Funding, and SOP for ANC and postnatal care (PNC) local solutions can be scaled. • The Project should replicate local solutions in other provinces or districts with similar MNH conditions, especially those with the support of key stakeholders. Subnational government actors indicated that the Floating Ambulance had support from multiple PHOs, POGI supported the Eradication of Pre-Eclampsia/Eclampsia (PE/E), and communities backed Puskesmas Ojek Driver with Motorcycle Ambulance (Tukang Ojek Puskesmas dengan Elektronik Ambulans/TOPDeAMOR). USAID Jalin designed the AMMDes Ambulance Feeder activities to accelerate referrals in remote and hard to access areas (Photo: DE). MOH PRIORITY DISTRICTS The Year 3 Work Plan intends to support 65 of the 120 priority MOH districts for accelerating the reduction of maternal and neonatal mortality under RPJMN 2020-2024 (see the map in EQ 3). Sixty￾five of the 120 districts are in USAID Jalin provinces: one in South Sulawesi, four in North Sumatera, seven in Banten, 14 in Central Java, 18 in East Java, and 21 in West Java. MOH has mandated that these districts undertake activities for improving access to and quality of MNH services, strengthening referral networks including SISRUTE, increasing community empowerment, increasing capacity through training, mentoring or others method, improving governance, and increasing utilization of 15 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV insurance for delivery (Jaminan Persalinan/Jampersal) and Special Allocation Funds (Dana Alokasi Khusus/DAK). The Project has local solutions in 15 of the 65 priority districts. COVID-19 WORK PLAN In April 2020, USAID Jalin began implemented an Addendum to its Year 3 Work Plan for COVID-19 Activities during April – September 2020. The Addendums activities intent to ensure access to emergency MNH services, provide advocacy and evidence generation, support IBI to address the pandemic, and improve referral pathways and networks. By the time of this evaluation, the Project had developed a webinar series with IBI on the pandemic’s impact on pregnancy, delivery and neonates, and infection prevention measures. In addition, USAID Jalin contributed a draft to Presidential Staff Office (Kantor Staf President/KSP) for the Protocol Petunjuk Praktis Layanan Kesehatan Ibu dan Bayi Baru Lahir selama Pandemi Covid 19 No: B-4 (05 April 2020) (Protocol for Practical Guidelines for Maternal and Newborn Health Services during the Covid-19 Pandemic). EQ 3: REGIONAL ACTIVITIES USAID Jalin did not comprehensively integrate regional activities and local solutions into the Year 3 Work Plan and new results framework, and regional teams have not begun to implement the new national programs in the Year 3 Work Plan. The Project began handing over local solutions (see the map below) to local partners and stakeholders in 2020, which prefer a more gradual transition of activities. USAID Jalin created plans to scale and replicate some local solutions, which focus on relatively basic activities, like developing guidelines and conferences, and are unlikely to lead to sustainable MNH impact. Opportunities exist to quickly link several regional activities to national programs to accelerate the Project’s process of leveraging change at scale. 16 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV Figure 6: Map of MOH Priority Districts, USAID Jalin local solutions, and field visits by the DE for USAID Jalin 17 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV INTEGRATION WITH THE YEAR 3 WORK PLAN & RESULTS FRAMEWORK USAID Jalin did not comprehensively integrate local solutions and regional activities into the Year 3 Work Plan and new results framework. Local Solutions Over Time: Please note that estimates of the local solutions have varied. The DE considers USAID Jalin to have developed 71 local solutions. USAID Jalin’s estimates have varied slight depending on the time period and the report. Please see Annex 7 for a Database of Local Solutions which tracks the local solutions since September 2019 and what the Project has done with them since according to Jalin's reports (see Table 5 below for a summary). USAID Jalin decided to implement several activities that were being designed that supported the Year 3 Work Plan and new results framework and discarded many others. In early 2020, the Project’s HQ directed regional teams to stop work and “transition” any other ideas being designed, according to its Quarter 1 2020 report. NO. OF LOCAL SOLUTIONS IN SEPTEMBER 2019 (DE) NO. OF LOCAL SOLUTIONS IN JUNE 2020 (DE) NO. OF LOCAL SOLUTIONS AS OF FEB. 2020 (USAID JALIN) Implementation 17 22 20 Design 17 0 0 Discarded or Postponed 37 46 8 Transitioned 0 2 1 Merged with Another Solution* 0 1* 1* Total 71 71 30 *USAID Jalin’s West Java team merged the MUYAN local solution with GP2SP in 2020. Results Framework Alignment: In the Year 3 Work Plan, USAID Jalin characterized 12 of its 30 local solutions as supporting IR 5 (see table 6), while it believed at least six were outside the results framework. Jalin and USAID KIIs differ on whether this is an accurate reflection of how these activities support Project goals. This evaluation identified several opportunities for USAID Jalin to better align local solutions with the new results framework by integrating them with new activities in the Year 3 Work Plan (see table below). Adapting local solutions to support these activities will help advance multiple IRs and may provide opportunities to begin fostering systemic change. Table 5: USAID Jalin Local Solutions Status Over Time Table 6: Local Solutions by IRs IR No. of Local Solutions by IR according to USAID Jalin IR 1 2 IR 2 4 IR 3 2 IR 4 2 IR 5 12 Enablers 2 “Others” 6 Total 30 18 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV LOCAL SOLUTION CURRENT IR OPPORTUNITIES TO INTEGRATE THE LOCAL SOLUTION WITH OTHER IRS Blood Donation Emergency Response IR 2 This local solution can be integrated with IR 4, the 4.1.3.1 activity to develop an integrated ICT platform. Blood donor availability is a part of an effective ICT referral system. The local solution in Bulukumba has developed an Android/web application for managing blood inventory and donor database to improve the Blood Transfusion Unit services. Its design can inform the ICT platform and its integration with the system can provide information needed in MNH emergencies on the availability of blood. MOU with CSR Forum to Improve MNH Enabling Environment This has potential linkages with IR 3 activities, including Activity 3.4.2 to foster engagement with midwives with private sector providers. The CSR Forum is a hub of private sector social responsibility activities at the provincial level. This local solution can be utilized to support the channeling of engagement and partnerships between health workers and relevant private sectors, such as between midwives and private sector suppliers. MNH Campaign IR 5 This can be integrated with IR 2, specifically Activity 2.4.7.2 to support strengthening provincial-level working groups, committees and MNH groups with the potential to participate in AMP and conduct capacity and training needs assessments. It can also contribute to Activity 2.2.2.5 in supporting the MOH to use dashboards to influence behavior in priority districts through sharing comparative data and rankings. MNH in the Community/Mentoring IR 1 There is an opportunity for this local solution to support AMP implementation in selected districts, under IR 2 Activity 2.4.7. The activity can be an entry point for strengthening AMP with district public and private hospitals, because it involved building a network between stakeholders who should be involved in the AMP. Engaging Indonesian Doctors Association (Ikatan Dokter Indonesia/IDI), IDAI, POGI, Hospitals, and Doctors Enabling Environment This local solution can be used to draw lessons learned for IR 2 Activity 2.4.7.3 to support AMP implementation in selected districts. AMMDes Ambulance Feeder IR 5 This can be integrated across local solutions in IR 5, such as village funding and other activities aimed at raising investment to increase funding for emergency transportation in USAID Jalin’s provinces. Decreased Mortality due to PE/E and Zoom Preeklamsi IR 2 This local solution can be integrated with IR 2 Activity 2.4.7.3 to support AMP implementation in selected districts. PE/E, ZOOM Preeklamsi, and AMP intend to reinforce good clinical governance. Both PE/E and Zoom Preeklamsi are efforts to identify factors associated with pregnancy risks and maternal deaths and strong collaboration between professional organizations and health workers from various level of health facilities under this local solution can support implementing and analyzing the AMP. Improving Health Worker Compliance with ANC and PNC SOPs, and Quality Improvements for Integrated ANC and PNC IR 2 This can be integrated with IR 1 activities, including Activity 1.1.2, to inform the implementation of mentoring at provincial and district hospitals and health centers. Table 7: Opportunities to Better Integrate Local Solutions with the new Results Framework 19 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV Birth Preparedness and Complication Readiness Program (Program Perencanaan Persalinan dan Pencegahan Komplikasi/P4K) Inovasi IR 5 Most activities under P4K Inovasi can contribute to IR 2.4.8. especially for 1) Awareness Raising for pregnant women, midwives and cadres that nutritional status influences fetal growth and maternal anemia; 2) improving the capacity of health workers including cadres. P4K Inovasi can also be integrated with IR 2 Activity 2.4.7.3 to support AMP implementation in selected districts. P4K Inovasi has successfully engaged stakeholders and involved partners from different sectors, ranging from district government units, health facilities, health workers, cadres, universities and students. This can serve as a strong base to support the implementation of AMP and ensure sustainable commitment to support MNH improvement. Gerai KIA IR 5 Through its ongoing collaboration with MOH Directorate of Health Promotion (Direktorat Promosi Kesehatan/PROMKES), this can support the MOH to reduce stunting, particularly MNH-related issues such as anemia in women of reproductive age and low birth weight, as included under IR 2 activities, specifically Activity 2.4.8 in the Year 3 Work Plan. It aligns with 1) Awareness Raising in which the nutritional status will directly impacts fetal growth and maternal anemia; and 2) Gender Equality that educated women will empower their health care seeking. PROMKES and Tangerang DHO has initiated this with a discussion on potentially expanding Gerai KIA to address the issue of stunting. National Programming: As noted in EQ 1, regional teams have not begun to implement the new national programs in the Year 3 Work Plan as of this evaluation. However, the MOH expects the Project to begin immediately supporting national GOI priorities at the regional level. Despite a lack of progress thus far, USAID Jalin’s regional teams and local partners expressed a willingness to implement national GOI programs from the Year 3 Work Plan. One stakeholder said, “All these activities were already implemented previously in the district… through the Expanding Maternal and Newborn Survival Project (EMAS) program… the DHO can support these activities when Jalin launches its program, because the DHO has sustained EMAS.” However, local stakeholders hope that the new national programs would work within the existing system. One stakeholder KII said, “The DHO expects that Jalin will not propose and implement new systems. It is better for USAID to support and strengthen the existing systems." TRANSITIONS AND SUSTAINABILITY USAID Jalin began handing over, or transitioning, local solutions to local partners and stakeholders in 2020. In February, the Project developed transition plans for approximately 21 activities at the district and provincial level and steps at the national level (see Annex 8 for transition and scaling plans). Most plans at the provincial and district levels involve handing over activities to local partners, documenting the program’s components, and then promoting documentation to stakeholders. The Project intends for several local solutions to inform national programming and for others to be handed over to national professional associations. However, in KIIs and regional workshops, regional staff and local partners and stakeholders expressed that this transition is occurring earlier than anticipated. Regional teams and their partners prefer transitioning local solutions gradually with the following steps: 1. Facilitate ongoing collaboration between partners. Stakeholders indicated that USAID’s standing represents a powerful motivator for actors to stay involved in partnerships until they become more formalized with time. 2. Lead M&E of local solutions for a sustained period. Stakeholders felt they lacked the time, resources, and skills to routinely conduct thorough data collection and analysis on new activities. Most local solutions have been implemented for less than a year and have not accumulated sufficient data to trace any associated effects on MNH in their target areas. 20 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV 3. Either: a. Adapt local solutions to GOI national programs. Jalin respondents believed that local solutions could be adapted to fit several IRs in the results framework. They stressed that their activities were new and malleable, and partners were committed to expanding and adapting them to address national MNH priorities, especially if the MOH recognized their efforts. Or, b. Advocate lessons learned from local solutions at the national level. Project staff and stakeholders believed that promoting the uptake of knowledge from regional activities would take more than the one-off workshops or conferences currently planned by Jalin (see table below). Local partners requested that USAID Jalin provide training and advice on how they could become advocates for their solutions and to help secure long-term funding from the private and public sectors. The concerns of regional teams and partners notwithstanding, it is understandable in terms of program management that the IP is preparing to handover local solutions in 2020 with the Project’s end scheduled for March 2021. Sustainability: The regional teams developed sustainability plans when initially designing their activities. Indeed, regional teams describe some of their key results in terms of sustainability according to regional workshops (see the table below). Regional teams incorporated most of their activities into local government systems and planning to some extent. For example, allocations of village funding for MNH and preparations to incorporate those into village plans have been made in Central Java, South Sulawesi, Banten, North Sumatera, and West Java. The South Sulawesi team has assisted the PHO with identifying funding for the floating ambulance local solution, including from private sector donations. That said, the North Sumatera, South Sulawesi, and West Java teams and their partners also expressed concerns about sustainability and incomplete transition plans in the regional workshops. SCALE AND REPLICATION USAID Jalin has developed partial plans to scale some local solutions (see Annex 8), which are unlikely to achieve its SOW’s goal of “sustainable MNH impact at scale.” The Project also created relatively more complete plans to replicate some local solutions in the provinces where they were piloted and in new provinces. The schedules for most replication exceed USAID Jalin’s anticipated end date of March 2021. Scaling Challenges: According to the desk review, USAID Jalin’s plans for leveraging change at scale largely consist of “documenting and sharing approaches… to scale through conferences, guidelines, training, mentoring, and their own networks” (see first table below). These plans include “development of best practice modules/toolkit. Promotion of best practice guide [and] Roundtables” with the GOI. The IP’s Jakarta HQ tasked regional teams with developing scaling plans. In KIIs, regional managers reported that it is difficult for them to develop specific plans for where to replicate outside their regions and prefer HQ take the lead in this. As of this evaluation, the Banten, Central Java, South Sulawesi and West Java teams had responded with plans to scale ten local solutions that are between two and five pages in length. However, these plans mostly refer to replicating the local solutions in new districts (see Annex 8) rather than leveraging change at scale by providing evidence from the implementation and testing of activities with the intend of systematically improving the delivery of life-saving approaches. HQ did not elect to use its central position and connection with the MOH to lead the development of scaling plans or coordinate across regional teams to promote replication (see EQ 3). Scaling Success: Despite these challenges, USAID Jalin has a successfully example of linking a local solution with national level MOH programming for greater MNH impact. In 2019, the Banten regional team secured a commitment from the PROMKES to promote its Gerai KIA local solution to increase ANC coverage. Previously, the activity had been led by the PHO, DHO, and PT. Sumber Alfamaria Trijaya, FOPKIA, and puskesmas. PROMKES funded a workshop showcasing Gerai KIA and is currently developing an application to promote and monitor the activity nationally. Through this evaluation, the 21 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV DE found that USAID Jalin could link others local solutions with similar national programs to scale their MNH impact (see table below). USAID Jalin currently anticipates transitioning most local solutions by September 2020. Replication: USAID Jalin has plans to replicate ten local solutions in new districts in provinces where they were piloted and four in new provinces (see Annex 8). Several, including the AMMDes Ambulance Feeder, MNH in the Workplace, and Floating Ambulance, had begun replicating to new districts and provinces by the time of this evaluation. The replication plans for cross-border referrals, Gerai KIA, and village funding have timeframes that exceed USAID Jalin’s anticipated end date of March 2021. Table 8: Links Between Local Solutions National Programs and Regulations LOCAL SOLUTION AND LOCATION POTENTIAL FOR REPLICATION AND SCALE POTENTIAL LINK TO NATIONAL PROGRAMS/STAKEHOLDERS Shuttle Service for Pregnant Women with Complication Program (Program Jemput Antar Ibu Hamil Bermasalah/Jamilah) and AMMDes Ambulance Feeder – Lebak, Banten This activity has the potential to be replicated within USAID Jalin’s six provinces and beyond in areas with similar characteristics and conditions. Alert village (Desa Siaga) is a program to reorient health services to become more participatory and bottom up. Based on the MOH Decree Number 564 / MENKES / SK / VI II / 2006, concerning Guidelines for the Implementation of the Development of the Alert Village. The alert village is a village whose population has the readiness of resources and ability and willingness to prevent and overcome health problems, disasters, and other health emergencies. This program is led by the Directorate of Health Promotion and Community Empowerment, Directorate General Public Health Cross Border Referral – Bogor Raya & Depok, West Java PHO West Java plans to replicate the model in all districts within the province especially in areas that border other provinces. USAID Jalin should identify and simplify the model in order to replicate and continue M&E to search for evidence of impact. This local solution supports government regulations, including the GOI Local Government Act No 32/2004 is a regional regulation on district / city cross-border reference areas for the health of individual referrals between districts / cities and provinces based on the ability and level of structured health service facilities and joint decisions between districts / cities to facilitate community access to health services in accordance with their health problems. In West Java, the cross-border referral system is regulated under the Governor Regulation No 64/2013. At central level, Directorate of Referral Health Services, DG Health Services, which has responsibility to formulate policies, norms, and standards, and provide technical guidance on referral health services. MNH in the Workplace – Garut, West Java Boyolali and Semarang, Central Java This activity has the potential to be replicated within USAID Jalin’s provinces and beyond in areas where factories exist. To gain lucrative contracts, factories must pay attention to workers, and these programs fit this need. The rights of women workers to obtain reproductive health benefits are regulated by MOH Directorate of Occupational Health and Sport (Direktorat Kesehatan Kerja dan Olahraga/KESJAOR). GP2SP of MOH/KESJAOR. MOH created GP2SP in 1996 and revitalized it in 2012 with the Ministry of Home Affairs, Manpower, and Women's Empowerment to improve female workers' health and nutrition to increase productivity. Its components include encouraging health and knowledge through advocacy and regulatory strengthening, improving accessibility and quality of workplace bases care, promoting reproductive health, improving nutritional status and encouraging breastfeeding during workhours. However, GP2SP lacks MNH components. 370 companies participated in GP2SP and are available in selected areas in Indonesia including all Jalin areas. Utilizing Village Fund – Sukabumi and Garut, West Java Langkat, North Sumatera Bone, South Sulawesi Brebes, Central Java Lebak, Banten All SUAID Jalin provinces, except East Java, have the potential to utilize village funds. Under current regulations, the proportion for health depends on each village’s priority. The proportion could increase as village health cadres voice needs during village consultation meetings. The Village Fund Program of the Ministry of Villages, Development of Disadvantaged Areas, and Transmigration and MOH – Community Strengthening. This can support MNH development programs as a DG of Public Health strategy. 22 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV Results Challenges Banten • Support for the allocation of village funds for MNH programs under Jamilah and the integration of sector and village planning for embedded village plans – a precursor to disbursing village funds Village Fund). • Health services available and improved access for referral services through Gerai KIA. • Securing local government budgets for replication of Jamilah in five additional puskesmas. • Likely sustainable relationship among local and provincial governments, private sector, health professionals, and the Ministry of Industry. • Lack of HQ guidance on how to contribute to national activities in Year 3 Work Plan. • Lack of technical guidance from HQ slowed the implementation of activities, including Potential Solutions with Sehati and Halodoc-IBI Partnership, and provided unclear prioritization directions. Central Java • Activities have been incorporated into government systems and planning through successful collaboration with multiple government units. • Succeeded in cultivating interest from new districts to replicate current local solutions in Boyolali and Brebes and MNH in the Workplace. • Designed activities according to regional needs and local perceptions of how to reduce maternal and neonatal mortality. • The local government, including the PHO, is process heavy and can be slow to respond. • In 2018 and 2019, the regional team prioritized engaging many actors from multiple sectors in cocreation rather than developing discrete activities with the potential for rapid implementation. East Java • Achieved buy-in from the provincial government. The PHO plans to make ANC-PNC SOPs applicable to all districts in East Java, like national SOPs. • Strong communication and coordination with the PHO, and other key partners in MNH, Forum for the Reduction of Maternal and Newborn Mortality (Forum Penurunan Kematian Ibu dan Bayi/PENAKIB) and POGI. • Secured local budget allocations in Jember for the of utilization of SOP-ANC. • Development of a Bupati Decree that support the improvement of health services and referral management. • Lack of clear guidance and technical assistance (TA) from HQ on how to identify priority districts and measure progress. • Delays in receiving MOH endorsement letter for work in East Java. • Inconsistent technical support due to staff turnover at HQ. • Local government is not committed to the Year 3 Work Plan activities. West Java • Complimented PHO and DHO programs by designing local solutions to fill gaps. • Developed local solutions with strong ownership from stakeholders, such as TOPDeAMOR and MNH in the workplace • Provided technical support for the allocation of village funds for MNH in Sukabumi. • Unclear sustainability guidance for some local solutions. • Additional time needed for local solutions to produce health outcomes. • Management challenges within the regional team. • Lack of HQ support for private sector engagement. • Unsure how to align local solutions with the new result framework in the Year 3 Work Plan. • Lack of personnel and funding to execute activities at the district level. North Sumatera • Ensured MNH funding support from the community via village funds and Health Operational Cost (Biaya Operasional Kesehatan/BOK). • Issued District Regulations (PERBUP) to support the MMR and NMR reduction programs. Under the village strategy plan similar programs exist. • Strong collaboration with universities, communities, health centers, the media, and religious leaders and organizations in addition to DHO and PHO. • The private sector is not used to multi-sectoral partnerships. • Unclear and constantly changing direction from HQ due to staff turnover. • HQ ignored information reported by the regional team and did not incorporate and share regional best practices. • Local solution, mini-MELA indicators cannot report into MELA plan indicators because they are too general, and HQ and the regional team differs on their definitions of results and success. • Stakeholders believe USAID Jalin will transition local solutions without adequate support. South Sulawesi • Ensured Bupati Decree on floating ambulance (currently in drafting process) for sustainability. • Utilized solutions to align with both PHO priorities, like the floating ambulance, and national policies, like the blood transfusion service activities. • Successfully built linkages between the PHO and Bappeda and the CSR Forum of Tonasa Company, Yusuf Kalla, and other companies. • Broad government support for floating ambulance, including funding from the Tonasa cement company. • Maternity waiting house built in Pangkep supported by the CSR Forum. • Project COP and Deputy Chief of Party (DCOP) visited to share the Year 3 Work Plan, but stakeholders need more information. • Require additional time to implement sustainability plans for local solutions. • Challenge due to the change in the head of PHO. • The Year 1 Work Plan did not incorporate regional teams’ input, and the Year 2 Work Plan focused on process rather than rapid implementation and monitoring for results. • Varying levels of commitment from district leadership. Some districts were eager to partner and others less so. • No formal mechanism for addressing regional team feedback. HQ’s regional manager meetings are not sufficient. Table 9: Key Results and Challenges Reported by Regional Teams in Workshops 23 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV EQ 2: STAKEHOLDER PRIORITIES USAID Jalin has significantly improved its engagement and collaboration with the MOH since Year 1; however, the MOH has unfulfilled expectations for regional support for national programs (see EQ 1). The Project has adequately addressed the needs and priorities of counterparts and stakeholders at the provincial and district levels. Since its inception, USAID Jalin’s MAB has been underutilized as a source of advice, networking, and advocacy. MINISTRY OF HEALTH USAID Jalin’s primary host-country counterpart is the MOH. Its primary focal point is KESGA and it also works closely with MOH Occupational Health and Sport Directorate (for MNH in Workplace), Health Promotion and Community Empowerment Directorate (related to module of Gerai KIA, and empower the community/local civil societies), Health Services Quality and Accreditation Directorate (related to Mentoring), Referral Health Services Directorate (related to cross border referral system), Health Services Facilities Directorate (related to technical guidance for AMMDes Feeder Ambulance), PUSDATIN (related to ICT/IR4), and PPJK (related to IR3). USAID Jalin has gradually adapted its programming and technical assistance to meet the MOH’s needs. KESGA did not approve the Project’s Year 1 Work Plan in 2017. Past observations indicate that the IP did not successfully explain how Jalin’s cocreation approach supported GOI priorities and strategies. In addition, KIIs suggested that MOH felt disconnected from USAID Jalin’s regional activities. One respondent referred to a workshop in Makassar, South Sulawesi, in October 2018, where MOH interpreted that “its invitation had been an afterthought, and they were right.” Improvements: USAID Jalin improved its relationship with the MOH over time by better explaining how its approach supported GOI priorities and strategies. USAID Jalin's current senior leadership team, which were hired in 2019, has improved the Project’s relationships with MOH, including the Public Health DG and Medical Services DG. According to Jalin KIIs, and as mentioned in EQ 1, “by establishing the Jalin Secretariat” the Project was able to “encourage collaboration between the two DGs.” Regional expectations: As noted in EQ 1, the MOH believes USAID Jalin’s regional teams can support GOI national programming but does not feel that the Project has delivered this support yet for IRs 1-4. A MOH KII respondent, declared that “in the past, [name withheld] updated the MOH on regional activities. But now who is following up on activities at the provincial level?” Personnel Changes in 2020: As noted above, when this evaluation was being revised in 2020, the DE observed that USAID Jalin did not to renew multiple staff contracts, including those who had improved its relationship with MOH. PROFESSIONAL ASSOCIATIONS USAID Jalin has involved professional associations, such as IBI, POGI, IDAI, and IDI, in implementing programs at the subnational and national levels. Past observations indicated that provincial branches of these associations were actively involved in related local solutions by regional teams. KIIs showed that the Project’s HQ intensively collaborated with POGI and IDAI to develop mentoring guidelines and with IBI to work with private midwives in 2020. This successful collaboration has left some professional associations, such as IBI, interested in additional support. PROVINCIAL HEALTH OFFICES USAID Jalin has successfully met most needs and priorities of PHOs in Banten, Central Java, East Java, North Sumatera, South Sulawesi, and West Java. In 2018, when USAID Jalin recruited its regional managers and staff, they rapidly met with PHOs to learn their priorities. Past observations confirmed that USAID Jalin’s regional teams integrated 18 different directives that they received in these meetings into their local solutions. This early coordination ensured regional activities met key PHO needs. All PHO representatives from the Project’s provinces appreciated its local solutions for addressing their specific needs and local contexts. One PHO KII respondent said, “Jalin’s local solutions activities have been carried out in accordance with local needs. This is considered by the PHO to be good 24 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV because it suits the specific needs of the region.” A Jalin respondent indicated that engaging the PHO in planning has been key for “sustainability as well as scalability through replication in other areas.” DISTRICT GOVERNMENTS At the district level, regional teams have successfully engaged subnational actors from different government units. Bappeda is a key counterpart for regional teams because it coordinates these units, and KIIs with Bappeda indicated that they appreciated the Project’s collaborative approach in 2018 and 2019 designing and implementing local solutions. One respondent confirmed that “Jalin's strategy in encouraging innovation to improve MNH is aligned with Bappeda’s strategy for development.” KIIs and regional workshops found that through Bappeda regional teams were increasing the technical capacity of government units including DHOs, hospitals, puskesmas, sub-district governments, and heads of villages. A Bappeda respondent claimed, “Jalin helps Bappeda educate government units at lower levels on the institutional and human resource capacity [necessary to support MNH]. In return, Bappeda facilitates mapping institution to be involved in local solutions.” Bappeda is a core agency for regional development programs. It has capacity to connect different government institutions and has appreciated USAID Jalin’s input when deciding which institutions to connect in partnerships to improve MNH programs. Regional teams have collaborated with the District Village and Community Empowerment Office (Dinas Pemberdayaan Masyarakat dan Desa/DPMD) to mobilize village funds to support MNH. A DPMD KII said, “Jalin's local solutions are aligned with the priorities of DPMD in translating the President's priorities over village fund allocation to basic education and health improvement.” OTHER LOCAL PARTNERS AND STAKEHOLDERS Regional teams have partnered with a variety of stakeholders in local solutions. KIIs and regional workshops suggested that USAID Jalin has succeeded in connecting multi-sectoral stakeholders with each other to support MNH. For example, a KII with a private sector actor stated that “Jalin’s contribution is very huge in identifying solutions to real problems and connecting the private sectors with local stakeholders.” A DHO respondent said, “Jalin played a significant role helping us access the industrial sector. Previously, we had difficulty building communication and relation with industry.” The result has been an increase in these actors’ awareness of MNH issues. In 2020, some partners and stakeholders became concerned that USAID Jalin intended to handover their activities too rapidly. One KII respondent pleaded, “Do not abandon the program. Yes [name withheld] will implement the program with or without Jalin… However, we had an agreement on a joint program” (see EQ 3 for more information on sustainability and transitions). MULTI-STAKEHOLDER ADVISORY BODY USAID Jalin has not fully utilized the MAB to advise, network, and advocated as stated in the Body’s Terms of Reference (TORs). Past observations suggest that the MAB has delivered advice to the Project in some cases but has not been given the chance to significantly network and advocate on its behalf. One respondent explained, “Jalin should give the MAB a clear role if they still need us to exist. Right now, we are only responding to what Jalin is presenting in its [quarterly] meeting.” A third posited that “access to information about Jalin is limited. Information is obtained through their weekly reports… this is not really enough for us to get involved in activities in the districts.” 25 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV RECOMMENDATIONS USAID Jalin should implement the steps outlined below. These recommendations are prioritized under each IR into those for immediate action and those for the Project to consider given its remaining time and resources. 1. On IR 1, USAID Jalin should support the MOH’s priority of using mentoring for sustaining improvements in clinical teams treating obstetric and newborn complications in key priority districts. a. USAID Jalin should immediately: i. Rapidly accelerate finalizing and gaining endorsement of the mentoring guidelines as a national document. This is a potential major Project legacy and a necessary precursor for initiating mentoring in vertical hospitals in October 2020 and in provincial and district hospitals in December2020. ii. In line with MOH expectations, ensure that the mentoring program is designed to be carried out by local governments in the future. Use the West Java regional team to advise on this approach, as the PHO there has sustained the EMAS mentoring approach. If applicable, use the Central Java model of mentoring as lessons learned on mobilizing stakeholder participation to improve the capacity of health workers in public and private hospitals. b. USAID Jalin should also consider: i. Facilitate increased coordination on mentoring between the Directorate of Health Services for Quality and Accreditation and KESGA, as these are the key counterparts for sustaining improvements in clinical hospital teams treating obstetric and newborn complications. Also, continue to engage other relevant actors for the national mentoring program, including JNPK, PPSDM￾MOH, POGI, and other professional organizations, to understand their needs and priorities for mentoring. ii. At the subnational level, develop mechanisms for supervision and monitoring to ensure improvements in clinical and managerial governance reflect best practices and SOPs. Develop practical guidance for DHOs, directors of hospitals, and local professional organizations explaining the mentoring guidelines and what steps should be carried out by mentors and mentees hospitals. Regional team should lead the development of this guidance with HQ coordinating their involvement. 2. Under IR 2, support the MOH objective of strengthening accountability and action focused on the main causes of maternal and neonatal death by providing regional support for MPDN and AMP. a. USAID Jalin should immediately: i. USAID Jalin’s regional teams should assist local stakeholders to report data on maternal and neonatal deaths in real time using the newly developed MPDN application b. USAID Jalin should also consider: i. HQ should provide updates to regional teams describing KESGA’s expectations for regional support of AMP, technical support on the MPDN application, and technical guidance on best practices for AMP to reinforce at the subnational level. 26 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV ii. Regional teams should work with the Bupati and DHO in MOH priority districts to ensure that an AMP team is functional within their areas of jurisdiction and capable of ensuring accurate and timely audits of maternal and newborn deaths within 72 hours of notification. 3. For IR 3, support the MOH’s goal of sustained improvement in quality of care by private midwives. a. USAID Jalin should immediately: i. Work with IBI to increase the availability of practical guidance for midwives’ competence, professionalism, and legal aspects and facilitate the monitoring of certifications, recommendations, and networking for puskesmas or clinics to meet BPJS requirements, and develop a real-time monitoring system on the registration and license status of PMBs-Bidan Delima. b. USAID Jalin should also consider: i. Foster engagement with government institutions, financing agencies, BPJS, and professional organizations identified by IB to gain their support for improvements in the quality of care and services of private midwives. ii. Advocate with IBI to strengthening coordination and the reporting system of PMBs-Bidan Delima with the puskesmas. This will enable the puskesmas to enhance and monitor the quality of care provided by PMBs-Bidan Delima for their community. It will also support documenting factors associated with maternal and neonatal deaths for the AMP. 4. Under IR 4, support MOH plans for an integrated and interoperable ICT system and accelerate integrating and scaling the referral platform. a. USAID Jalin should immediately: i. Complete a rapid strategic assessment on the existing ICT platforms and applications by March 2021. Review ICT readiness and competence at the district level and analyze the comprehensive factors, including those not related to ICT, that result in suboptimal utilization and ineffective function of current systems. ii. Assess the issues related to the communication between hospitals and lower level facilities, and develop draft SOPs for pre-referral stabilization, to be agreed and complied with by the hospitals and lower level health facilities. b. USAID Jalin should also consider: i. Design an advocacy strategy for ensuring hospital commitment for investing in human resources for managing the information and supporting infrastructure in order to provide real-time information for the referral system, including communicating with lower-level facilities. This advocacy can be integrated across IR1 mentoring. 5. For IR 5 – which primarily deals with local solutions – provide those activities with substantive support for scaling, replication, M&E, and transitioning to local partners. a. USAID Jalin should immediately: i. Readjust work plan timelines for implementing and transitioning local solutions and provide these to USAID, the MOH, and other stakeholders. COVID-19 and other issues have delayed the schedule, and partners have not received an up-to-date timetable. 27 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV ii. Design and implement responsible transition guidelines for each local solution. At a minimum, these guides should provide step-by-step instructions at the provincial level on who should be involved in activity management, what technical components and areas should be included, and financial aspects including budget allocations and unit cost for implementing the activities. iii. Develop and implement plans to scale the most advanced and promising local solutions, including the following activities that closely align with key MOH priorities and can be completed within six months: 1. Scale the Jamilah-AMMDes activities, including their village funding component, combining it with the Alert Village program (Desa Siaga) to reorient health services to become more participatory. This supports MOH Decree Number 564 / MENKES / SK / VI II / 2006, concerning Guidelines for the Implementation of the Development of the Alert Village. The Banten and North Sumatera regional teams should provide strategic advice to HQ to engage with the MOH Directorate Health Promotion and Community Empowerment and DMPD to begin national-level integration. 2. Scale the Cross-Border Referral local solution, which supports MOH policy on the Regionalization of Referral Systems (GOI Local Government Act No 32/2004). Draw on the solution’s MOU component to create a national template with minimum requirements for ICT support systems and that specifies key institutions that should be involved, including BPJS. The West Java regional team and potentially the West Java PHO should assist HQ with engaging the Directorate of Referral Health Services to promote the model. This local solution is also appropriate to replicate in several USAID Jalin provinces, most notably in Central Java between Brebes and Pemalang districts. 3. Scale MNH in Workplace by making it the standard MNH component of the MOH KESJAOR GP2SP program nationally. MOH revitalized GP2SP in 2012 with the Ministries of Home Affairs, Manpower, and Women's Empowerment to improve female workers' health and nutrition to increase productivity; however, it lacks a dedicated MNH component, which MNH in the Workplace can provide. The Central Java regional team should advise HQ on engaging KESJAOR, developing documentation on the concept and sets of sub-activities, and how best to gain uptake from the 370 companies participating in GP2SP. 4. Scale Village Funding utilization activities, which support the Village Fund Program of the Ministry of Villages, Development of Disadvantaged Areas, and Transmigration and the MOH Community Strengthening policies. Engage with the DG of Public Health Strategy, and utilize the experience of the Banten, North Sumatera, South Sulawesi, and West Java teams to develop models of funding and guidelines for advocacy by local communities on programs like health cadres and NGOs for health, as well as on planning and budget allocation at annual village and district assemblies. iv. Continue to collect M&E data on local solutions and implement as many planned assessments as possible, potentially through local national STTA to overcome a lack of regional M&E officers, to generate evidence to support replication and scaling. 28 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV b. USAID Jalin should also consider: i. Deliver a final report on each local solution being implemented (as opposed to those that the Project discarded or never developed beyond ideas) as part of transitioning these activities. Final reports should identify the problems that each activity addresses, describe how the solution was cocreation, and provide M&E data with sources. ii. Prepare a systematic review of each USAID Jalin province that captures lessons learned from implementing regional activities over the last three years. Document knowledge garnered from the successful engagement of large numbers of subnational actors in cocreation. Include detailed information on stakeholders’ priorities and agendas and how USAID can engage them in the future. A mother and baby in Lebak, Banten, who benefitted from the Project’ AMMDes Ambulance Feeder activity (Photo: DE). 29 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV ANNEX 1: JALIN PROJECT COMPONENTS Core Operating Principles Management Components Technical Focus Areas ● Support co-creation platforms that facilitate the design, financing and implementation, of technically sound and politically feasible local solutions, in order to break through barriers to progress in reducing maternal and newborn mortality. ● Adopt a whole of market approach that engages a variety of stakeholders including local and national governments and agencies, private (for profit and not for profit) sector actors, CSOs, media, academia, and other global partners. ● Influence systems operating at scale to improve population access to and utilization of essential, evidence-based, life-saving MNH services. ● Align all activities and efforts along a critical path to achieve sustainable results, focusing on a limited number of efforts critical to achieve desired MNH outcomes. ● Provide catalytic support for system improvements by providing critical information and evidence to inform problem definition, programmatic and policy decisions at all levels. ● Target and evaluate all support for the extent to which it benefits the poorest and most vulnerable. 1. Management, Measurement and Communication Expected Results: Partnership process managed to generate, implement, evaluate and spread successful local solutions and critical evidence, resulting in sustainable MNH impact at scale. 2. Technical Assistance Expected Results: Technical assistance efficiently deployed and managed to generate, implement, evaluate, and spread successful local solutions and critical evidence, resulting in sustainable MNH impact at scale. 3. Catalytical Financing Expected Results: Catalytic financing efficiently disbursed to support the incubation and implementation of local solutions. 1. Quality of Care Expected Result: Improved quality of care in public and private sector facilities. 2. Referral System Expected Result: Improved efficiency and effectiveness of the referral system at community and facility levels. 3. Local Governance Expected Result: Improved effectiveness, accountability, and responsiveness of local governance. 4. Financial Protection Expected Results: Improved financial protection for the poorest and most vulnerable. 5. Defining and Reaching the Poorest and Most Vulnerable Expected Results: Increased utilization of MNH services by the poorest and most vulnerable. 6. Evidence Utilization Expected Result: Improved utilization of evidence for decision-making and advocacy purposes. 30 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV ANNEX 2: PRIOR DE ACTIVITIES FOR PAST OBSERVATIONS ● EQ 1 (September – October 2018): Analyzed on Jalin project on designing and implementing of MNH Program based involvement of their staff and lessons learned harvested from this process for effectivity implementation of program ● EQ 2 (September – October 2018): Assessed roles and functions of MAB for Jalin and recommended improvement in structure, engagement and monitoring for MAB to maximized utilization of asset of knowledge from MAB in advising, networking, and advocating. ● EQ 3 (November 2018 – March 2019): Identified factors influencing in the stakeholders on formulated their decision and recommended effective engagement with the stakeholders. ● EQ 4 (April 2019 – present 2020): Identify and evaluate Jalin in applying core operating principles in conducting their local solutions and recommend improvements for performance of the local solutions, including in-depth assessments of: o The Halodoc-IBI partnership o Gerai KIA o MNH in the Workplace (multiple local solutions) o P4K Inovasi o Floating Ambulance o AMMDes Ambulance Feeder o Village Funding o Cross-Border Referral Ssytems ● EQ 5 (July – September 2019): Evaluated Jalin SOW responding MOH’s need and anticipate the USAID’s Journey to Self-Reliance (J2SR) and recommended changing of Jalin’s workplan supporting MOH’s request and ensuring the J2SR including in the workplan. ● EQ 6 (July – November 2019): Comprehensively reviewed activities of Jalin Regional Team on implementation of cocreation approach including in applying core operating principles; their relationship with the local stakeholder; and local solutions contributing to MNH at local context and potentially being national solutions. ● In addition to DE activities under the EQ 1 to the EQ, DE also conducted several activities responding Emergent Tasks (August 2018 – present) which associate with Jalin’s core operating principles; project management and relationship with the stakeholders; monitoring in the context for MNH in Indonesia; and currently in the context on impact of COVID-19 in MNH. The activities are: o Workshop Facilitation (August 2018 – present) for improvement of results framework and work planning o Support for Complexity Aware Monitoring (November 2018 – February 2019) for introducing monitoring for complex systems o Jalin Management Assessment (December 2018 – May 2019) for recommendations on management changes o Review of MELA plan (January – February 2019) for recommendation on improvements o Geographic Information System (GIS) mapping for MOH (February - July 2019) delivered GIS mapping for health facilities across Indonesia 31 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV o Jalin in graphics report (March – May 2019) assessed Jalin responsiveness to USAID requests o Review of Jalin’s new proposal, results framework, and USAID-DAI communication (June – August 2019) reported performance and management of Jalin; and o Studies for portfolio reviews (September – November 2019) in relation to MNH in Indonesia assisted to Mission portfolio review, EMAS Replication and Sustainability Study. 32 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV ANNEX 3: DOCUMENTS REVIEWED ● 4 documents on Jalin's Work Plan, including: - USAID Jalin Project Year 1 Work Plan - USAID Jalin Project Year 2 Work Plan - USAID Jalin Project Year 2 (Modified) & Year 3 Workplan September 2019 –December 2020 - USAID Jalin Project Addendum to Year 3 Work Plan: Covid-19 Activities April – September 2020 ● 9 documents on USAID Jalin Project Reports, including: - USAID Jalin Project Year 1 Annual Report - USAID Jalin Project Year 2 Annual Report - USAID Jalin Project Quarterly Report, Q1 FY2018 - USAID Jalin Project Quarterly Report, Q2 FY2018 - USAID Jalin Project Quarterly Report, Q3 FY2018 - USAID Jalin Project Quarterly Report, Q1 Project Year 2019 - USAID Jalin Project Quarterly Report, Q2 Project Year 2019 - USAID Jalin Project Quarterly Report, Q3 Project Year 2019 - USAID Jalin Project Quarterly Report, Q1 Project Year 2020 ● 3 documents on USAID Jalin’s Monitoring, Evaluation, Learning, and Adaptation (MELA) Plan, including: - USAID Jalin Project Year 1 MERLA Plan - USAID Jalin Project Year 2 MELA Plan - USAID Jalin Project Year 3 MELA Plan ● 51 documents on Jalin's Weekly Update, consists of: - 43 Jalin’s Weekly Update from February – December 2019 - 8 Jalin’s Weekly Update from January – March 2020 ● 250 documents on USAID Jalin’s local solutions, including: - 22 documents of local solutions concept notes. - 21 documents on local solutions work plan and budget - 15 documents on local solutions briefs. - 22 documents of mini-MELA of local solutions. - 160 documents on local solutions monthly report from September 2019 to March 2020. - 10 documents of local solutions scale up plan. ● 24 documents of assessment reports and related documents, including: - Consultant Trip Report – Richard Brough, National Implementation Specialist. - Methodology for Rapid Assessment of Bidan Delima Program. - Mapping Insurance Payment Relationships to Maternal and Neonatal Quality of Care. - Technical Assistance Activity Report JALIN/DAI to PPJK Ministry of Health. - USAID Jalin Maternal and Newborn Health Systems Value Chains Study. - Consultant Report: Access to Finance Expert for MNH Sub Sector (Private Midwives Practitioner and Clinics). - And other supporting documents. ● 5 documents on USAID Jalin’s scope of work and MOUs with partners, including: - The USAID Jalin Project Scope of Work with POGI. - Scope of Work to Support IBI. - Memorandum of Understanding Between Puls Trading Far East Limited and USAID Jalin Regarding MNH in the Workplace: WEALTH Program. - Statement of Work on DAI support on the development of a workplace health program for female factory workers, focused on MNH in approximately nine VFC supply chain factories. 33 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV ● 8 documents on USAID Jalin’s mentoring program, including: - 4 Invitations Letter from the MOH on the Establishment of Mentoring Task Force and Developing the Mentoring Guideline. - Draft of Mentoring Guidelines. - Draft of Mentoring Guidelines Executive Summary. - Draft of Mentoring Monitoring and Evaluation Strategy. - Draft of USAID Jalin Hospital Assessment Plan. ● 22 documents on USAID Jalin’s MAB activities from March 2019 to February 2020. ● Arah Kebijakan dan Rencana Aksi Program Kesehatan Masyarakat Tahun 2020-2024 (Policy Direction and Action Plan of Public Health Program Year 2020-2024), a presentation by Dr. Kirana Pritasari, MQIH., General Director of Public Health Directorate, Ministry of Health, in National Health Work Planning Meeting (Rapat Kerja Kesehatan Nasional/RAKERKESNAS) 2020. ● Keputusan Meteri Kesehatan Republik Indonesia No. HK 01.07/MENKES/150/2020 tentang Kelompok Kerja Percepatan Penurunan Angka Kematian Ibu dan Bayi Baru Lahir (Minister of Health Decree No. HK 01.07/MENKES/150/2020 on Task Force for the Acceleration of Maternal and Neonatal Mortality Reduction). ● Peraturan Menteri Kesehatan Republik Indonesia No. 28/2017 tentang Izin Penyelenggaraan Praktik Bidan (Minister of Health Regulation No. 28 of 2017 on License for Midwifery Practicing). ● Satuan Tugas Percepatan Penanganan COVID-19, Protokol Petunjuk Teknis Layanan Kesehatan Ibu dan Bayi Baru Lahir Selama Pandemi COVID-19 No. B-4 (05 April 2020). Task Force for COVID￾19 Countermeasures Acceleration: Technical Guideline Protocol regarding Maternal and Neonatal Care during COVID-19 Pandemic No. B-4 (April 5, 2020). ● Undang Undang Republik Indonesia No. 4 Tahun 2019 tentang Kebidanan. (Act No. 4 of 2019 on Midwifery). 34 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV ANNEX 4: USAID JALIN PROGRESS TOWARD YEAR 3 WORK PLAN MILESTONES USAID Jalin’s Quarter 2 2020 Report describes its progress against Year 3 Work Plan milestones as follows: Intermediate Result 1 Milestones Current Status (1 July) Expected Completion Date March 2020 milestones: Strengthened guidelines based on evidence adopted by MOH Partially achieved: Clinical/ managerial guidelines not yet finalized. September 2020 Clinical/ managerial guidelines approved by MOH MOH-led mentoring initiated in 7 vertical hospitals Not started: Mentoring not yet initiated. September 2020 Mentoring hospitals identified and mentoring initiated Strategy for MNH mentoring for private hospitals Not started: Strategy not yet developed. February 2021 Strategy developed September 2020 milestones: MOH-led mentoring implemented in all vertical hospitals and in hospitals in 65 Jalin priority districts Not started: Mentoring not yet initiated. February 2021 Mentoring complete in 65 districts Draft M&E system for mentoring (key indicators incorporated in SIMRS for public hospitals) Partially achieved: M&E strategy for mentoring developed and shared with the MOH. February 2021 System developed and operating Intermediate Result 2 Milestones Current Status (1 July) Expected Completion Date March 2020 milestones: MPDN available to be accessed by Puskesmas, PHO, DHO and hospitals MPDN is fully operational and undergoing piloting in several provinces. February 2021 MPDN in use in districts in all Jalin provinces. Accountability: Advocacy for inclusion of key MNH indicators in the five-year national development plan (especially quality, MPDSR, mentoring) Partially achieved: Policy brief submitted (with accountability as the theme and including draft indicators at various geographical levels), awaiting feedback fromMOH February 2021 Largely achieved: Depends on MOH feedback. Development of dashboard supported by TA MPDN is fully operational and undergoing piloting in several provinces. February 2021 MPDN in use in over 20% of districts in all Jalin provinces. MNH TWG: Established and functioning; and some strategic priorities identified Not started: Despite a Ministerial Decree being issued in late Feb 2020, there appears to be little MOH activity on operationalizing the MNH WG at present. Jalin engaging with M&E Sub-working group. February 2021 Largely achieved: Some strategic priorities should have been identified by this time. Pre-Eclampsia/ Eclampsia screening model in C. Java developed Partially achieved: PE/E screening model in Central Java now being piloted. How much training has been completed is unclear. Responsibility for this sub-activity was shifted to IR1 in February 2020. February 2021 Local Solution fully transitioned to POGI. 35 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV Minimum two manuscripts are ready to submit Largely achieved: USAID Jalin is working with PUSKA-UI to submit three manuscripts by August. September 2020 Three manuscripts will have been submitted in August. September 2020 milestones: Option to incentivize recording, reporting, review and response supported (option TBD) Not started. February 2021 Largely achieved: System for formal recognition of top performing hospitals, cities/districts, and provinces. Functioning system for exchange of information between hospital and district Partially achieved: Field testing of MPDN being negatively impacted by COVID-19 restrictions and management attention focused on COVID-19. February 2021 Functioning system being rolled out. Functioning system for flow of information from national to provincial and district levels Not started: Implementation delayed. February 2021 Largely achieved: Some flow of information is likely to have been achieved by February 2021. Dashboard in use in real-time; and published Limited progress with MPDN dashboard due to COVID-19 February 2021 Fully achieved. Dashboard published and in use in some Jalin provinces. MOH MNH WG producing recommendations with Jalin support Not started: Despite a Ministerial Decree being issued in late Feb 2020, there appears to be little MOH activity on this issue at present. Jalin engaging with M&E Sub-working group. February 2021 Largely achieved: Some recommendations may have been generated at this point, but this depends on the continued severity of the COVID-19 pandemic and the degree to which MOH can focus on the MNH WG. PE/E model disseminated nationally if appropriate Partially achieved: How much training has been completed unclear. Responsibility for this sub-activity was shifted to IR1 in February 2020. February 2021 Local Solution fully transitioned to POGI who will be disseminating nationally. Intermediate Result 3 Milestones Current Status (1 July) Expected Completion Date March 2020 milestones: Bidan Delima (BD) review conducted Fully achieved: BD review conducted in December 2019 Fully achieved Enhanced BD business model available Partially achieved: Discussion and meetings took place in preparation of the planned online BD co-creation workshop where the business model will be developed as part of this. September 2020 Business model available Co-creation workshop planned for beginning of July. Workplan development planned for end of July. Access to credit: North Sumatra financing model developed Partially achieved: Work by the Jalin team in North Sumatra in relation to access to finance for midwives is ongoing. February 2021 Advocacy activities conducted at the national level September 2020 milestones: 36 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV IBI supported to implement enhanced BD business model in Jalin provinces Partially achieved: Delayed as co-creation workshop needed to be postponed (due to COVID-19). February 2021 Largely achieved: Pilot Implementation of enhanced BD program finalized end of February and initial analysis / evaluation started in March 2021. Partnerships with private sector explored for resource generation (re BD) Not started: Will rely on completion of enhanced BD program. Partially achieved: After finalizing the pilot program sharing of lessons with partners (including private sector). Pilot implementation completed in two Jalin provinces Activity dropped: As Jalin is no longer engaged in strategic health purchasing Activity dropped Cost effectiveness model developed Activity dropped: As Jalin is no longer engaged in strategic health purchasing Activity dropped National model finalized (based on North Sumatra and other best practices) Activity delayed: local solution model not applicable for national scale up. Further evaluation required. February 2021 Decision on model for national strategy Intermediate Result 4 Milestones Current Status (1 July) Expected Completion Date March 2020 Milestones: ICT platform reviewed with recommendations shared with MOH Not started Commissioning of a rapid assessment of existing ICT-based referral platforms scheduled to start July 2020 September 2020 Rapid assessment of ICT platforms completed with costed implementation plan in place for piloting preferred option. Interoperable system designed Not started The interoperable system design will form part of the ICT referral review (milestone above). Rapid assessment of ICT platforms completed with costed implementation plan in place for piloting preferred option. SiCantik piloted and lessons learned documented Largely achieved SiCantik is now being piloted and lessons being documented. The LS will inform the rapid review (above). September 2020 Local Solution documented and transitioned September 2020 Milestones: Integrated system piloted in Jalin provinces and evaluated Not started Pilot not yet designed or initiated (see above) February 2021 System designed, piloted and evaluated Intermediate Result 5 Milestones Current Status (1 July) Expected Completion Date March 2020 Milestones Strengthening emergency referral networks: Mapping of MNH capacity (with availability) of public and private referral facilities in Jalin provinces using SIMRS conducted Not started Activity not yet initiated. February 2021 Strategy on how to proceed completed and rolled out in all six provinces 37 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV Cross-border referral system in Bogor City, Bogor District and Depok City (West Java) developed and tested Partially achieved Mapping of health facilities is complete. However, MOU drafting between health facilities – to test cross-border referral system in three districts – is delayed due to COVID￾19 pandemic. September 2020 Local Solution documented and transitioned Improving transport: Two transport pilots in remote areas completed with recommendations (South Sulawesi and Banten) Largely achieved The floating ambulance and the ambulance feeder local solutions have been successfully piloted are in the process of being documented. September 2020 Local Solutions documented and transitioned September 2020 Milestones Formal commitments to provide MNH referral services (including cross-border) based on competencies; documented through provincial policies (and signed MOUs); and implemented in select Jalin provinces Not started Activity not yet initiated. February 2021 Strategy on how to proceed completed and rolled out in all six provinces Lessons learned from cross-border model (West Java) shared nationally Partially achieved. September 2020 LSs documented and transitioned Sustainable options for implementation and funding of transport for emergency referrals identified and piloted in select Jalin provinces (with recommendations) Local solutions in process of being documented and transitioned to local partners September 2020 Local solutions documented and transitioned to local partners Lessons learned from South Sulawesi and Banten pilots shared nationally Not started Activity not yet initiated. September 2020 Local Solutions documented and transitioned 38 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV ANNEX 5: KEY ACHIEVEMENTS IN YEAR 1 ANNUAL REPORT USAID Jalin’s Year 1 Annual Report Executive Summary the following overall progress and successes: OVERALL PROGRESS During Jalin’s first year, in addition to completing operational set up in Jakarta and six regions, significant investment was made in developing key relationships. The Project took an opportunistic approach, in which stakeholder engagement and solution development took place concurrently, thus positioning Jalin as relationship broker and facilitator, rather than traditional project implementer. Jalin also established a Multistakeholder Advisory Body (MAB) that has provided and will continue to provide important guidance and input. Opportunistic implementation during Year 1 produced lessons learned, enabled Jalin to test and refine systems and processes, and built a foundation for Year 2 implementation. Understanding that institutional partnerships are driven by interpersonal relationships, Jalin focused much effort identifying and meeting with key individuals at national and subnational levels. Through this approach, the Project established partnerships with the MOH, including PHOs and DHOs, and other key stakeholders. Through these partnerships, Jalin developed problem statements, and finalized and communicated prioritization criteria that will inform ongoing design of local MNH solutions. These relationships also informed management and operations-focused adjustments, including updating the Project’s staffing structure and clarifying national and regional roles to ensure more responsive implementation. SUCCESSES Critical to success is building relationships and structuring mechanisms that allow productive partnerships to flourish. Year 1 success is demonstrated by the plans for Year 2 that were co￾determined and will be supported by Indonesian stakeholders. For example, District Chiefs in North Sumatra and South Sulawesi will work with Jalin to draft decrees to ensure village funds are allocated to community-level MNH activities. Partnerships. Partnerships at the national level facilitate effective programming at the subnational level. MOH’s endorsement of the Project opens doors with PHOs, and therefore DHOs, with whom Jalin will work most closely to design and implement local solutions. Successful relationship building at both levels was demonstrated in December when Jalin Regional Managers together with their PHO counterparts presented the Year 2 Work Plan to the MOH in Jakarta. Additionally, the Ministry of Villages, Development of Disadvantaged Regions, and Transmigration (KEMENDESA) have committed to assisting Jalin. KEMENDESA will identify village-level interventions that could be supported by the Village Fund. Jalin also established the MAB that informs Project programming approaches and brings a wider network of traditional and non-traditional actors to advocate for MNH in Indonesia. The MAB comprises 5 Task Forces that focus on quality of care and referrals, local governance, social protection, communication and evidence utilization, and private sector engagement. Jalin also began working with provincial level advisory bodies in North Sumatra and East Java that will support the Project moving forward. Communication. Maintaining and promoting wider interest about MNH issues, writ large, is germane to advocacy and awareness strategies that generate increased demand for adequate resources and quality services to address MNH challenges. Jalin began working with journalists, and other media outlets and professionals through targeted meetings and a series of seven roundtables— in Jakarta and all six target provinces—to gather insights on how to effectively engage media and disseminate MNH information. These and other events attracted significant local media coverage. In the Jakarta Post, in August and September, Jalin published an op-ed and a series of articles about the danger of malnutrition for pregnant women. Jalin also conducted a survey to better understand public awareness and perceptions about MNH, which will inform advocacy activities in Year 2. 39 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV Technical Assistance. Early in Year 1, Jalin provided expertise to synthesize evidence from previous USAID-funded studies and use it to build initial partnerships and inform problem definition activities. Jalin also worked with MOH to prioritize technical assistance packages that Jalin could support, contracted consultants to support Maternal and Perinatal Death Surveillance and Response (MPDSR) and MSS, and help PHOs and DHOs monitor and evaluate MSS. The project also conducted a “deep dive” review of issues related to quality of care, referrals, the JKN, and potential MNH data management frameworks. Management. Jalin completed operational set-up and all major start-up tasks in Jakarta and the 6 geographic target areas. Almost fully staffed, 70 Jalin members are in place and working in 7 functioning offices. Core policies and procedures are in place to help staff implement activities and manage partnerships that address the Jalin Theory of Change (TOC), Results Framework, and Work Plan activities. The project also laid the foundation for effective implementation and management through staff training and development. This included conducting Monitoring, Evaluation, Research, Learning, and Adaptation training for Jalin regional office staff; and conducting procurement training for relevant Jalin staff. 40 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV ANNEX 6: KEY ACHIEVEMENTS IN YEAR 2 ANNUAL REPORT USAID Jalin’s Year 2 Annual Report detailed the following key achievements: Year 2 Highlights • Strengthened GOI engagement: USAID Jalin effectively collaborated with the MOH to co-create, secure buy-in and endorsement for the new RF and strengthened engagement at all levels. The Project’s regional teams established constructive relationships with Provincial Health Offices (PHO) in all Jalin provinces. These extend to the district level in priority districts, and to provincial governors in Banten, Central Java, East Java, North Sumatra, South Sulawesi, and West Java. • Supported MOH priorities through Jalin Secretariat: USAID Jalin focused on embedding its activities in the MNH National Action Plan and Provincial Development Plans – to ensure implementation of the J2SR. Establishment of the Jalin Secretariat strengthened these important linkages with the MOH and ensured alignment with government priorities at all levels. The Project plans to expand the Secretariat model to all Jalin provinces in Year 3. By working hand in hand with the MOH, project activities are positioned to be sustained and taken over by the Ministry. • Strengthened relationship between USAID and MOH: Through the Jalin Secretariat, the Project facilitated the first meeting of the US Ambassador and the new Minister of Health, reinforcing USAID’s role in supporting the GOI agenda to reduce maternal and neonatal mortality. By supporting numerous meetings between USAID and the Directorate General of Public Health and other key MOH staff, the Secretariat has positioned USAID to be at the forefront of work with the MOH and provincial government to address the President’s call for action. • Generated and disseminated MNH evidence: USAID Jalin completed and disseminated the Every Mother and Newborn Counts (EMNC) study, using the findings to further the Maternal Perinatal Death Surveillance and Response (MPDSR) system and facilitate local government advocacy. The Project also finalized the study on Mapping of Insurance Payments against Quality of Care for MNH to analyze claim data. Both study results generated data that provides the MOH with evidence on where to focus interventions to reduce mortality (i.e. mentoring in hospitals is based on evidence generated, which the Project has contributed to). USAID Jalin also shared select data from these studies during the Indonesia Health Economics Association (InaHEA) conference in November 2019. • Expanded scalability of project activities: During Year 2, the geographic focus of the project broadened; shifting from individual districts to provincial level engagement. The Project is now supporting activities in 120 GOI priority districts for MNH at the national level and has a presence in 65 of these districts (increasing from 23 last year). In line with the Year 2 (modified) and Year 3 Workplan approved in December, USAID Jalin’s regional activities are more strategically linked with national level implementation, and are complemented by increased provincial level engagement, which supports future scale-up of project interventions. • Elevated clinical/managerial mentoring as a GOI priority intervention: Through USAID Jalin’s involvement, the MOH has made mentoring a high-level priority that improves quality of care and addresses mortality. During Year 2, USAID Jalin supported establishment of the MOH Task Force on Mentoring and drafted the mentoring guidelines. • Influenced establishment of the MOH MNH Working Group: At the request of the MOH, and in line with the Ministerial Decree, USAID Jalin played a key role in establishing the MNH Working Group and is actively supporting the operationalization of the group with Terms of Reference, analytical and technical support as well as serving as the Secretariat. • Reinvigorated the Multi-stakeholder Advisory Body (MAB): USAID Jalin helped rebuild the MAB, with new effective leadership, new membership, and reorganized structures for working groups 41 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV that are linked to project activities. The MAB continues to demonstrate a renewed commitment and vision, which presents a fresh opportunity for USAID Jalin. • Leveraged private sector engagement to support project activities: Throughout Year 2, USAID Jalin effectively collaborated with private sector partners to secure co-financing, facilitate implementation and replication of activities, and promote sustainability of the Project’s activities. Through these efforts, the VF Corporation (VFC) approved funding of approximately USD 175,000 for Jalin to administer an MNH in the Workplace program in nine VFC-affiliate factories throughout Indonesia which will begin in Year 3. • Established collaborative partnerships with professional associations: In Year 2, USAID Jalin focused on building close relationships with Perkumpulan Obstetri dan Ginekologi Indonesia (Indonesian Obstetrics and Gynecology Association – POGI) and Ikatan Bidan Indonesia (Indonesian Midwives Association – IBI) to ensure that the Project’s interventions are based on clinical evidence, best practices, and aligned with national efforts. The Project is formalizing these partnerships through grant agreements which will be issued in Year 3. • Standardized partnership approach: USAID Jalin developed standard partnership agreements across regions and partners. Memorandums of Understanding (MOU) were developed for PHOs and USAID Jalin to outline the goals and expectations of the collaboration. Cooperation Agreements (Perjanjian Kerjasama - PKS) were developed for partners of USAID Jalin activities to outline the roles, responsibilities and expectations of the parties. • Documented local solutions: The Project standardized documentation of local solutions, including “mini” Monitoring, Evaluation, Learning and Adaptation (MELA) plans, workplans, budgets and technical briefs, and developed a dashboard platform to display project progress and activities. • Linked local solutions to the new RF: Together with USAID, the Project identified 30 solutions that are aligned with the RF and progressed implementation of these activities. To clarify linkages between regional and national activities, the team defined how local solutions contribute to national activities to guide implementation going forward. • Progressed the implementation, sustainability, and scalability of local solutions in the following areas: o Mobilization of resources: Secured funding from other stakeholders to continue activities in 2020 through varied mechanisms, for example the use of village funds for ongoing Gerai Kesehatan Ibu dan Anak (Maternal and Child Health – KIA) activities in Banten to increase women’s MNH knowledge and facilitate improved access to quality antenatal care (ANC) in health facilities. o Legislation issued: Conducted successful advocacy to promote government regulations to support project implementation, such as issuance of the Regent Decree in Central Java, which provides the mandate for implementation of the PE/E program. o Scaling and sustainability: Supported plans to replicate successful pilot activities, strengthened local ownership, and embedded activities in existing systems to ensure sustainability, exemplified by the District Health Office’s (DHO) request to help replicate the Tukang Ojek Puskesmas dengan Elektronik Ambulans Motor program (Puskesmas Electronic Motorcycle Ambulance – TOPDeAMOR) in four other districts of West Java with similar terrain. o Systems strengthened: Engaged with public and private sector partners to improve referral systems, such as through efforts to develop an integrated cross-border referral network in West Java and strengthen the SiCantik referral application and utilization by health providers in East Java. o Delivery strengthened: Improved access to health services and contributed to accelerated referrals for mothers and newborns, particularly through activities focused 42 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV on supporting transport solutions in hard to reach areas through the ambulance feeder in Banten and the floating ambulance in South Sulawesi. 43 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV ANNEX 7: LOCAL SOLUTION DATABASE Province Solution Name District DE Status (EQ 6) Jalin FY 2019 Status Jalin Q1 FY 2020 Report Status Jalin Action in Q 1 FY 2020 Jalin's "Way Forward for Local Solutions" Document (Feb 2020) Jalin Q2 FY 2020 Status Jalin Action in Q 2 FY 2020 Banten Gerai KIA - addresses inadequate MNH knowledge among mothers and families leading to incomplete ANC and PNC visits. Unequal quality of midwives and volunteers lead to unstandardized counseling. Tangerang District Implementation Implementation Implementation Continued from FY 2019 Implementation Implementation Continued from Q1 FY 2020 Facilitating online counselling, via Gerai KIA Mobile. Banten Sehati Banten Discarded Oct￾2019 Discarded Implementation Continued from FY 2019 Discarded Discarded Discarded Banten Awal Bros Banten Discarded Oct￾2019 Discarded Discarded Jalin transition this after FY 2019. Not mentioned Discarded Discarded Banten Lasizmu Banten Discarded Oct￾2019 Discarded Discarded Jalin transition this after FY 2019. Not mentioned Discarded Discarded Banten IBI-Halodoc Partnership - improve access to medicines and supplies for private midwives in Banten via a mobile application. Tangerang District Discarded Jan￾2019 Discarded Implementation Continued from FY 2019 Discarded Discarded Discarded Banten Jamilah Feeder Ambulance - strengthen the Jamilah Program to improve access to health facilities for patients in remote areas and strengthen the cadre program. Lebak Implementation Implementation Implementation Continued from FY 2019 Implementation Implementation Continued from Q1 FY 2020 Utilizing the ambulance feeder for COVID-19 prevention and awareness activities Banten One Nurse, One Village - to improve the quality of neonatal care. Lebak In Design Design Discarded Jalin transition this after FY 2019. Not mentioned Discarded Discarded Banten Integrated Microfinance Institution for MNH Lebak Discarded Jun￾2019 Discarded Discarded Jalin transition this after FY 2019. Not mentioned Discarded Discarded 44 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV Central Java Financing for Bidan Delima Boyolali Boyolali Boyolali: Discarded Sep￾2018 Discarded Implementation Continued from FY 2019 Discarded Discarded Discarded Central Java Financing for Bidan Delima Semarang Semarang Semarang: Discarded Oct￾2018 Discarded Implementation Continued from FY 2019 Not mentioned Discarded Discarded Central Java SijariEMAS Innovation Technology Boyolali Discarded Sep￾2018 Discarded Implementation Continued from FY 2019 Discarded Discarded Discarded Central Java Hospital Blood Bank Boyolali Boyolali Boyolali, Semarang, & Brebes: Discarded Oct￾2018 Discarded Discarded Jalin transition this after FY 2019. Not mentioned Discarded Discarded Central Java Hospital Blood Bank Semarang Semarang Boyolali, Semarang, & Brebes: Discarded Oct￾2018 Discarded Discarded Jalin transition this after FY 2019. Not mentioned Discarded Discarded Central Java Hospital Blood Bank Brebes Brebes Boyolali, Semarang, & Brebes: Discarded Oct￾2018 Discarded Discarded Jalin transition this after FY 2019. Not mentioned Discarded Discarded Central Java Maternity waiting homes for pregnant women and family Semarang Discarded Oct￾2018 Discarded Discarded Jalin transition this after FY 2019. Not mentioned Discarded Discarded Central Java Communication, Information, and Education on MNH Central Java Discarded Jan￾2019 Discarded Discarded Jalin transition this after FY 2019. Not mentioned Discarded Discarded Central Java MNH in Workplace (PASTI) Boyolali - Improve access and quality of services for female factory workers Boyolali Boyolali: Implementation Implementation in Semarang & Boyolali Implementation Continued from FY 2019 Implementation Implementation Continued from Q1 FY 2020 Supported PT Pan Brothers Tbk. and PT SAMA to adapt the program during the COVID-19 pandemic situation Central Java MNH in Workplace (PASTI) Brebes - Improve access and quality of services for female factory workers Brebes Brebes: Discarded Oct￾2018 Discarded Discarded Discarded Not mentioned Discarded Discarded 45 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV Central Java MNH in Workplace (PASTI) Semarang - Improve access and quality of services for female factory workers Semarang Semarang; Implementation Implementation in Semarang & Boyolali Implementation Continued from FY 2019 Not mentioned Implementation Continued from Q1 FY 2020 Three companies in Semarang city will replicate the program, starting in April 2020. Central Java MNH in Community / Mentoring - Strengthen clinical and public accountability in referral hospitals Brebes Implementation Implementation Implementation Continued from FY 2019 Implementation Implementation Continued from Q1 FY 2020 Assisted 3 hospitals and 8 Puskesmas for mentoring program. Central Java Pre/eclampsia Eradication – Improve early detection of high￾risk pregnancies and readiness of health facilities Boyolali Implementation Implementation Implementation Continued from FY 2019 Implementation Implementation Continued from Q1 FY 2020. Assisted DHO to mentor four Puskesmas. The four Puskesmas are being prepared to mentor 21 other Puskesmas. Central Java TBD – Referrals (Becoming MNH in Community/Mentoring Program) Grobogan In Design Design Implementation Continued from FY 2019 Not mentioned Implementation Continued from Q1 FY 2020 Prepared 6 Puskesmas in Grobogan to mentor 24 other Puskesmas in Grobogan district. East Java Optimizing SiCantik Sidoarjo – digital application monitoring mechanism where doctors and midwives can monitor the data of pregnant women who are regarded as “high-risk”. Sidoarjo Sidoarjo: Implementation Implementation in Sidoarjo Implementation Continued from FY 2019 Implementation Implementation Continued from Q1 FY 2020 Advocated policies, hospitals for referral networks, and increased utilization of SiCantik. Integrating SICantik with SiManies application for referral. East Java Optimizing SiCantik Malang – digital application monitoring mechanism where doctors and midwives can monitor the data of pregnant women who are regarded as “high-risk”. Malang Malang: In design Design in Malang Discarded Discarded Not mentioned Discarded Discarded 46 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV East Java Improving Health Worker Compliance with ANC and PNC SOPs Malang Postponed Jan￾2019 Design Implementation Continued from FY 2019 Implementation Implementation Continued from Q1 FY 2020 Developed 8 SOPs for ANC and PNC and tested instruments for M&E. East Java Improving public awareness of MNH via the Media Jember Postponed Jan￾2019 Design Discarded Jalin transition this after FY 2019. Not mentioned Discarded Discarded East Java Quality Improvements for Integrated ANC and PNC - strengthening services for mothers and newborns by aligning ANC and PNC SOPs with national guidelines and training health workers on these. Jember Implementation Implementation Implementation Continued from FY 2019 Implementation Implementation Continued from Q1 FY 2020 Successfully advocated for three policies. Conducting internal audit to evaluate compliance of health workers in maternal care based on standards. North Sumatera P4K Inovasi - seeks to strengthen P4K, a Ministry of Health program launched in 2007 to improve MNH by increasing community involvement in ANC. Langkat Implementation Implementation Implementation Continued from FY 2019 Implementation Implementation Continued from Q1 FY 2020 Conducted a workshop on village fund allocation for MNH North Sumatera Access to Finance for Private Midwives and Clinic (A2F) - to finance them to adhere to standardize medical equipment, and medicine for providing service; and training. Deli Serdang Implementation Implementation Implementation Continued from FY 2019 Implementation Implementation Continued from Q1 FY 2020 Completed a field assessment in April 2020 on the supply and demand sides of access to finance for private midwives in East Java and Central Java. North Sumatera MNH Campaign – to address lack of awareness on MNH issues (self-rights, services, JKN, self￾assessments, etc.) 15 districts in North Sumatera Implementation Implementation Implementation Continued from FY 2019 Implementation Implementation Continued from Q1 FY 2020 Engaged seven content creators/influencers to maximize the MNH Campaign by promoting it through social media platforms 47 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV North Sumatera Online Transportation for Pregnant Women - provide discount for pregnant women who use services for ANC. Districts/Cities with available online transportation In design Design Discarded Jalin transition this after FY 2019. Not mentioned Discarded Discarded North Sumatera Saving Product for JKN - saving product to help low income groups pay premium by installments. Deli Serdang In design Design Discarded Jalin transition this after FY 2019. Not mentioned Discarded Discarded North Sumatera Bumil Trace – Android-based application to provide information on the pregnant women’s health profile and ANC Pilot in Deli Serdang as part of P4K In design Design Discarded Jalin transition this after FY 2019. Not mentioned Discarded Discarded North Sumatera District MNH Fund - to create a forum to manage funding for MNH, such as for subsidizing the JKN or BPJS premium, and to be established based on a Bupati Decree. Deli Serdang In design Design Implementation Continued from FY 2019 Discarded Discarded Discarded North Sumatera Sociological Research in Nias – identify relevant socio-cultural, economic, and behavioral factors related to the high maternal and newborn mortality rate among persons in Nias. 5 Districts in Nias In design Design Discarded Jalin transition this after FY 2019. Not mentioned Discarded Discarded South Sulawesi Android Application for Pregnant Women Tracking and Monitoring Bone Discarded (Pending) Dec￾2018 Discarded Discarded Jalin transition this after FY 2019. Not mentioned Discarded Discarded South Sulawesi Floating Ambulance – multiple phases: 1) Procurement of floating ambulances for islands and coastal areas 2) SOP for floating ambulance (medical and operational) 3) Multi-sectoral cooperation to support the utilization of floating ambulance to improve health services and referrals 4) Advocating for Governor Decree on the Utilization of Floating Ambulance 1-3. Pangkep 4. Sulawesi Selatan Province Implementation Implementation Implementation Continued from FY 2019 Implementation Implementation Continued from Q1 FY 2020 Facilitating MOU, finalizing SOP, referral pathway, and supporting socialization. South Sulawesi MNH Campaign – Tribun Timur publishes MNH-related news Provincial Discarded Feb￾2018 Discarded Discarded Jalin transition this after FY 2019. Not mentioned Discarded Discarded 48 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV South Sulawesi Bola Asseddingeng – capacity building for midwives on counseling skills to strengthen mother classes and improving access to transportation for referral Bone Implementation Discarded Jalin transition this after FY 2019. Not mentioned Transitioned Transitioned South Sulawesi Village fund allocation for MNH Bone Discarded (Pending) Apr￾2019 Discarded Implementation Continued from FY 2019 Implementation Implementation Continued from Q1 FY 2020 Training for midwives and cadres were conducted South Sulawesi Blood Donation Emergency Response Bulukumba Implementation Implementation Implementation Continued from FY 2019 Implementation Implementation Continued from Q1 FY 2020 Developed an android/web application for managing blood inventory and donor database. South Sulawesi PKK for Pregnant Women and 1,000 Days of Life Bulukumba Discarded Dec￾2018 Discarded Discarded Jalin transition this after FY 2019. Not mentioned Discarded Discarded South Sulawesi Encourage local public figures to socialize MNH Bulukumba Discarded Dec￾2018 Discarded Discarded Jalin transition this after FY 2019. Not mentioned Discarded Discarded South Sulawesi Pro-active home visits to pregnant women reluctant to receive ANC at a health facility Bulukumba Discarded Dec￾2018 Discarded Discarded Jalin transition this after FY 2019. Not mentioned Discarded Discarded South Sulawesi Improve Quality and Quantity of Family Planning Program (Keluarga Berencana, KB) Bulukumba Discarded Dec￾2018 Discarded Discarded Jalin transition this after FY 2019. Not mentioned Discarded Discarded South Sulawesi Health Insurance for Pregnant Women into the Highest Referring Health Facility Bulukumba Discarded Dec￾2018 Discarded Discarded Jalin transition this after FY 2019. Not mentioned Discarded Discarded South Sulawesi Puskesmas Internal Audit based on the MOH Regulation No. 44/201 Jeneponto Discarded Nov￾2018 during work planning Discarded Discarded Jalin transition this after FY 2019. Not mentioned Discarded Discarded South Sulawesi "Ibu-Ibu Syantik" MNH Education Forum - private sector helps to improve early detection of high-risk pregnancies Jeneponto Discarded Nov￾2018 during work planning Discarded Discarded Jalin transition this after FY 2019. Not mentioned Discarded Discarded 49 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV South Sulawesi Capacity improvement (on job training) for health workers Jeneponto Discarded Nov￾2018 during work planning Discarded Discarded Jalin transition this after FY 2019. Not mentioned Discarded Discarded South Sulawesi Advocating for Village Regulations for MNH Sinjai Discarded Oct￾2018 at workshop Discarded Implementation Continued from FY 2019 Implementation Implementation Continued from Q1 FY 2020 Still in early stages of the program. South Sulawesi Dasa Wisma Reactivation and Improvement Sinjai Discarded Oct￾2018 at workshop Discarded Discarded Jalin transition this after FY 2019. Not mentioned Discarded Discarded South Sulawesi Engaging IDI, IDAI, POGI, hospitals, and doctors to enhance coordination – improve relationships and willingness of doctors to accept others’ patients, revoking practice licenses, conducting clinic accreditation, and establishing a Public Safety Center (PSC) to support the referral system. Sinjai Implemented by Jalin in December 2018 Implementation Implementation Continued from FY 2019 Transitioned Transitioned Transitioned South Sulawesi Establish MNH Forum & Optimize social media for MNH Gowa Discarded Nov￾2018 Discarded Discarded Jalin transition this after FY 2019. Not mentioned Discarded Discarded South Sulawesi Monitoring, Detection, and Readiness for high-risk pregnancies Gowa Discarded Nov￾2018 Discarded Discarded Jalin transition this after FY 2019. Not mentioned Discarded Discarded South Sulawesi Improving Maternity Waiting Homes & Ambulances for Villages Gowa Discarded Nov￾2018 Discarded Discarded Jalin transition this after FY 2019. Not mentioned Discarded Discarded South Sulawesi MOU with CSR Forum to Improve MNH Sulawesi Selatan Province In design Design Implementation Continued from FY 2019 Discarded Discarded Discarded West Java Safe Mother and Child Software Jawa Barat Now cross border referrals Design Discarded Jalin transition this after FY 2019. Not mentioned Discarded Discarded West Java Saving Next Generation and Baby Memo Software – monitors the health status of pregnant women and nutrition status of children. Garut Discarded Apr￾2019 Discarded Discarded Jalin transition this after FY 2019. Not mentioned Discarded Discarded 50 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV West Java Tidung Suci - Puskesmas will accompany and monitor women’s health during premarital period and after marriage. Sukabumi Discarded Oct￾2018 Discarded Discarded Jalin transition this after FY 2019. Not mentioned Discarded Discarded West Java Supporting Evidence-Based MNH Policy Making - MNH research using a health status assessment tools developed by USAID. Sukabumi Discarded Oct￾2018 Discarded Discarded Jalin transition this after FY 2019. Not mentioned Discarded Discarded West Java Replicating and Strengthening Pregnant Women Screening Program - collaborate with OBGYNs from private hospitals to conduct screening at Puskesmas Karawang Discarded Jan￾2019 Discarded Discarded Jalin transition this after FY 2019. Not mentioned Discarded Discarded West Java Optimizing Private Sector Support for Improving Blood Supply in West Java Jawa Barat Discarded Jan￾2019 Discarded Discarded Jalin transition this after FY 2019. Not mentioned Discarded Discarded West Java TOPDeAMOR – utilizes ojek drivers near Puskesmas to serve as motorcycle ambulances transporting pregnant women for ANC, delivery, medicine, and midwife visits. Garut In Design Design Implementation Continued from FY 2019 Implementation Implementation Continued from Q1 FY 2020 Facilitated MOUs in eight villages and finalized two Village Regulation drafts West Java MNH in the Workplace (GP2SP) – 1) Pregnant mother class 2) Post-partum family planning program 3) Healthy Premarital Certification (Calon Pengantin Sehat/CATIN) for Women Workers at PT. Changsin. 4) Anemia and chronic malnutrition detection 5) Providing cold box for breast milk supply 6) Provide Mentoring for PT. Changsin Factory Clinic Garut Implementation Implementation Implementation Continued from FY 2019 Implementation Implementation Continued from Q1 FY 2020 Several modules finalized West Java Family Planning Contraception Services Mobile Unit - MUYAN is a program of BKKBN to use minibus to provide family planning services for populations in remote areas. Garut In Design Discarded Implementation Continued from FY 2019 Merged to GP2SP (Implementation) Merged to GP2SP (Implementation) Continued from Q1 FY 2020 Under the MNH in the Workplace (GP2SP) Local Solution 51 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV West Java Utilizing Village Fund - solve geographical Challenges and traditional birth attendants Garut In Design Implementation Continued from FY 2019 Implementation Implementation Continued from Q1 FY 2020 Advocating for regulation West Java Advocating for a regulation that converts TBA from midwives' partners into pregnant women’s companions – limit TBA’s role in delivery process Sukabumi Discarded Jul￾2019 Discarded Discarded Jalin transition this after FY 2019. Not mentioned Discarded Discarded West Java Utilizing Feeder Ambulances - utilizing AMMDeS multi-function vehicles to transport pregnant women to health facilities Sukabumi & Garut In Design Design Implementation Continued from FY 2019 Discarded Discarded Discarded West Java Zoom Preeklamsi - initial screening, risk identification, and referral treatment for preeclampsia and eclampsia (similar to activities in Central Java). Jawa Barat In Design Design Implementation Continued from FY 2019 Implementation Implementation Continued from Q1 FY 2020 Still in early stages of the program. West Java Cross-Border Referrals – 1) PONEK hospital standards mapping, analyzing 24-hour availability of OBGYNs, the existence of a hospital blood bank, and surgery room 2) MOU on cross-border information sharing, referral systems, budgeting, and maternal and perinatal audits 3) Strengthening the Implementation of Information and Communication System for Referral (SISRUTE) 4) Cross-border regulations Bogor District, Bogor City, Depok City In Design Design Implementation Continued from FY 2019 Implementation Implementation Continued from Q1 FY 2020 Developing integrated referral dashboard Central Java Strengthening MNH care in private and public health facility, as well as improving networks among health facilities Central Java Being designed New Year 3 activity Discarded Discarded Discarded 52 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV ANNEX 8: USAID JALIN PLANS FOR TRANSITIONING AND SCALING LOCAL SOLUTIONS LOCAL SOLUTION TRANSITION PLAN NATIONAL SCALING PLAN REPLICATION PLAN WITHIN PILOT PROVINCE REPLICATION PLAN OUTSIDE PILOT PROVINCE MNH in the community (Central Java) Document and share approach with the Mentoring Task Force. Documentation informs national mentoring approach Strengthening MNH Care (Central Java) Not to progress. Decreased mortality due to PE/E (also “Pre-Eclampsia / Eclampsia Eradication”) (Central Java) Transitioned to POGI and ed by POGI JKT with collaboration from POGI Central Java. Pull documentation together and transfer to POGI. POGI to prepare documentation on models. Support POGI to scale through conferences, guidelines, training, mentoring, and their own networks. Scale to Brebes, Grobogan, Banyumas, and Batang in Central Java by September 2020. Get lesson learn from implementation in Boyolali. Discuss in each new district the concept of Pre-Eclampsia (PE) Eradication. Determine targets and develop workplan in each district. Zero Mother Mortality (Zoom) Pre-Eclampsia (West Java) Scale to Sukabumi, Garut, and Indramayu in W. Java by December 2020. Provide evidence from implementation in Karawang. Involve POGI and PHO West Java as implementers. Facilitate a work visit by DHO to public hospital and selected Puskesmas to Karawang District. Strengthen the implementation of giving calcium and Acetosal to pregnant women at Puskesmas Parung Kuda Sukabumi, which began two years ago. Improving Health Worker Compliance with ANC + Postnatal (PNC) Standard Operating Procedures (SOPs) (East Java) Continue through March 2020. Convert paper￾based SOPs to electronic for use by midwives and doctors to replace “pocketbook.” Pull documentation together and transfer to POGI. Support POGI to strengthen and scale through conferences, guidelines, training, and mentoring. Quality improvements for Integrated ANC + PNC (East Java) Access to Finance (North Sumatra) Not to progress. Transferred from region to HQ. Financing for Bidan Delima (Central Java) Not to progress. Optimizing SiCantik (East Java) Continue through September 2020. Discuss with PHO for possible scaling. Develop national strategy for scaling the Information and Communication System for Maternal and Neonatal Referral Network (Sistem Informasi dan Komunikasi Jejaring Rujukan Maternal & Neonatal/SijariEMAS with PT. SIjarimas Teknologi Inovasi (STI). Scale to Pandeglang in Banten by November 2020. Initial workshop at district level of Pandeglang on development using Sicantik to improve referral and MNH data management. Work visit for those involved to East Java. Map stakeholders to support. 53 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV LOCAL SOLUTION TRANSITION PLAN NATIONAL SCALING PLAN REPLICATION PLAN WITHIN PILOT PROVINCE REPLICATION PLAN OUTSIDE PILOT PROVINCE SijariEMAS (Central Java) Not to progress. Women’s Health and Wellbeing (WEALTH) Project Administered by HQ Review to inform VFC proposal. Development of best practice modules/toolkit. Promotion of best practice guide. Roundtable with Indonesian Employers Association (Asosiasi Pengusaha Indonesia/APINDO) and KESGA. Scale to Tangerang in Banten by December 2020. Initial workshop in district level of Tangerang. Work visit of district government to Central Java. Map stakeholders to support the pilot. GP2SP includes Family Planning Contraception Services Mobile Unit (Mobil Unit Kendaraan Keluarga Berencana /MUYAN) All MNH solutions inform a review scheduled for Dec – March 2020. Continue GP2SP through September 2020. Productive, Active and Fit Workers Program (Pekerja Aktif Sehat Bugar & Produktif/PASTI) (Central Java) Continue PASTI through Apr 2020. In Banyumas, Sragen, and Kendal by December 2020. Work with POGI to get lesson learned from Boyolali and Semarang. Discuss with districts MNH at the working place based on the manual developed. Determine targets and workplan. Ambulance Feeder (Banten) Continue through September 2020. Develop best practice modules/ toolkit. Promotion of best practice guide. Promote through bi-annual presentation to MOH (includes 65 districts). Scale to Lebak, Pandeglang, and Serang of Banten in 2020- 2021. To replicate in 10 Puskesmas with potential source of budget allocations from district and provincial governments, CSR, and social events. Scale to South Sulawesi by August 2020. Workshop at provincial level. Work visit of provincial government to PT. KMWI as provider of AMMDes Ambulance feeder. Map stakeholders to provide support. Utilizing Feeder Ambulance (West Java) Promote manual/ toolkit. Not to progress. Floating Ambulance (South Sulawesi) Continue through June 2020. Scale to Makassar, Sinaji, Selayar, and Luwu Timur in South Sulawei. Ambulance is already available and waiting for the Governor to select districts. Adopt SOPs and guideline for floating ambulance crew. Maternal and neonatal emergency training for medicals and paramedics crew. Develop financial plan. Scale to Serang District in Banten by October 2020. Initial workshop at district level of Serang. Work visit of district government of Serang to South Sulawesi. Map potential stakeholders to support. Puskesmas Ojek Driver with Motorcycle Ambulance (Tukang Ojek Puskesmas dengan Elektronik Ambulans/TOPDeAMOR) (West Java) Continue through November 2020. Scale to Sukabumi of W. Java by December 2020. Provide evidence from TOPDeAMOR in Garut, including winning the innovation competition. Provide alternative transportation options for each region. Involve Regional General Hospital (Rumah Sakit Umum Daerah/RSUD) Jampang Kulon as a referral center for health centers in remote areas. Provide data and information to Sukabumi DHO, and facilitate visit of Sukabumi DHO and selected Puskesmas to Puskesmas Cihurip Garut. 54 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV LOCAL SOLUTION TRANSITION PLAN NATIONAL SCALING PLAN REPLICATION PLAN WITHIN PILOT PROVINCE REPLICATION PLAN OUTSIDE PILOT PROVINCE District MNH Fund (North Sumatra) Document whatever exists and share with STTA. Feeds into a strategy for use of MNH funds at provincial, district and village level (STTA). Utilizing Village Funds (West Java) Scale to Indramayu, Karawang, Bogor, and Cirebon in West Java in 2021-2022. Involve POGI, Indonesian Pediatricians Society (Ikatan Dokter Anak Indonesia/IDAI), and District Village and Community Empowerment Office (Dinas Pemberdayaan Masyarakat dan Desa/ DPMD) in workshop on new evidence on MNH in the Village using village funds. Facilitate regulation development, module development for DPMD village consultation center, and support technical supervision. Village Fund Allocation (South Sulawesi) Advocating Village Regulation for MNH (South Sulawesi) Cross Border Referrals (West Java) Continue through September 2020. Document and prepare guidelines to promote scaling. Scale to all districts in West Java by 2022 in phases. Involved PHO, DHO, the Hospitals, Professional Organization (POGI, IDAI, Association of Anesthesiologist and Intensive Therapy (Perhimpunan Dokter Spesialis Anestesiologi dan Terapi Intensif/PERDATIN), Hospital Association of Indonesia (Perhimpunan Rumah Sakit Seluruh Indonesia/PERSI), Indonesia Private Hospitals Association (Asosiasi Rumah Sakit Swasta Indonesia/ARSSI)). Provincial Cooperation Bureau and Districts/Municipalities Cooperation Coordination Team (Tim Koordinasi Kerjasama Daerah/TKKSD). Facilitate to establish integrated cross-border MN Emergency referral committee. Support the Comprehensive Emergency Obstetric and Newborn Care (Pelayanan Obstetrik Neonatal Emergensi Komprehensif/PONEK) Facilities mapping in all hospitals. Facilitate MoU Integrated maternal and neonatal emergency referral network between DHOs, All Hospitals, PERSI, ARSSI, and facilitate integrating local referral application. MoU with CSR Forum (South Sulawesi) (Enabler) Lead documentation, tools etc. for sharing with other Regional Offices. Not to progress. Document, promote, and establish across all provinces. IDI, IDAI, POGI Hospitals + Doctors to improve relationships – South Sulawesi MNH Campaign (North Sumatra) Continue through March 2020. 55 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV LOCAL SOLUTION TRANSITION PLAN NATIONAL SCALING PLAN REPLICATION PLAN WITHIN PILOT PROVINCE REPLICATION PLAN OUTSIDE PILOT PROVINCE Potential Solutions with Sehati (Banten) Not to progress. Halodoc-IBI Partnership (Banten) Not to progress. Available for IBI if they wish. Blood Donation Emergency Response (South Sulawesi) Continue June 2020 Scale to Bulukumba and Bone districts in South Sulawesi by December 2020. In five new villages in Bulukumba to be led by DHO Bone and initiated with USAID Jalin budget; and in 3 Puskesmas to discuss with DHO and Puskesmas in Bone. Gerai KIA (Banten) Continue through March 2020. Scale to Serang, Pandeglang, and Lebak in Banten in 2020- 2021. In each district, arrange five new locations of Gerai KIA-Stunting in Tangerang district will open. Ensure village funds and other local budgets will support the activities of the current Gerai KIA and in the new locations. P4K Inovasi (North Sumatra) Continue through July 2020. 56 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV ANNEX 9: USAID JALIN FEEDBACK ON THE MIDTERM FINAL EVALUATION REPORT November 6, 2020 HEADLINE OBSERVATIONS Issues Examples METHODOLOGICAL CONCERNS (AND DEVIATIONS FROM THE STATEMENT OF WORK [SOW) • Preliminary findings were not shared with the USAID Jalin. • Not all evaluation questions were answered, as per the SOW. OMISSIONS (INCLUDING CONTEXT) • The report does not sufficiently recognize the contribution of USAID and the Ministry of Health’s (MOH) in co-creating the Year 2 & 3 workplan and Results Framework (RF). • There is limited mention of USAID or MOH role in project successes and challenges. • The Midterm Evaluation (MTE) report does not recognize the leadership role of the MOH in the implementation of the Year 2 & 3 workplan. • The full list of respondents, and the final SOW is missing from the report. BALANCE AND TONE OF THE REPORT • There is repeated negative messaging: e.g. related to staff changes, the early transition of local solutions (LS), national and regional office divisions. • The Executive Summary omits important contextual information: the ‘redesign’ of the project in 2019 and its impact on delivery; COVID-19 and its impact on a) access to, and availability of the MOH b) the implications for the Jalin team. INACCURACIES • There are several inaccuracies in the document e.g. the statement that local solutions were not integrated within the Year 3 Work Plan and new RF. DETAILED OBSERVATIONS 57 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV Ref. Quotation Reference USAID Jalin’s comment Issue USAID Jalin’s request 1 The SOW states: “The DE will conduct the evaluation in close consultation with USAID, the implementing partner (IP), and the Government of Indonesia (GOI) Ministry of Health (MOH), which are the evaluation’s primary users.” “Preliminary findings presentation to discuss the summary of findings, conclusions, and recommendations with USAID, Jalin, and/or MOH, and provide feedback before the DE develops draft and final evaluation reports.” The report will address each EQ and other relevant issues. The DE will review comments on the initial draft to develop the final evaluation report. Final SOW USAID JALIN understands the evaluation followed this sequence of events: • April ‘20: USAID issues SOW • May ’20: DE collects data and reviews documentation • June ’20: DE presents (to USAID) preliminary findings • August ’20: DE revised MTE report • September 11: USAID informs Jalin the project would not be extended beyond base period (March 7th). • October 19: USAID shares MTE report with Jalin with the expectation to use findings and recommendations especially to improve Jalin's activities in the final phase of project implementation. Methodological concerns (and deviations from the SOW) Lessons can be learned from this MTE that will make future evaluations more collaborative, transparent, and also useful for the Implementing Partner. 58 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV Ref. Quotation Reference USAID Jalin’s comment Issue USAID Jalin’s request Preliminary findings were not shared with USAID Jalin. The project was also unaware DE was revising the report in August. Had MTE recommendations been shared with the project earlier (before October 19), project management would have acted upon recommendations sooner. 2 The SOW indicates: “For Jalin’s midterm evaluation, the DE will study Jalin’s overall period of performance to date to answer the following evaluation questions (EQs)”: ….5 EQs are listed SOW Only 3 out of the 5 EQs are addressed. Methodological concerns (and deviations from the SOW) This is an important deviation from the SOW; we would like to see the report explain why 2 EQs were not addressed. 3 The SOW indicates: The midterm’s purpose is to identify how USAID, Jalin, and its counterparts, stakeholders, and partners can improve Jalin in the future and to document learning and achievements taking place among the project and its partners. SOW The learning from the first 2.5 years of implementation could be more explicit in the report. For example, are there lessons related to working with the MOH that are useful for other IPs (such as establishing an embedded Secretariat)? Methodological concerns (and deviations from the SOW) It would be useful to the reader if the report is more explicit in terms of lessons learned which can be categorized as lessons for each stakeholder: USAID, IP and MOH. 59 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV Ref. Quotation Reference USAID Jalin’s comment Issue USAID Jalin’s request USAID JALIN devoted a section of the revised workplan to lessons learned which were created in the second half of 2019. 4 The SOW states the DE will answer the following EQ (EQ3): In order to improve Jalin’s administration going forward, what role has USAID’s and the IP’s project management played in Jalin’s successes and challenges to date? SOW This EQ was not addressed in the MTE report. Given that Jalin was co-created with USAID and MOH it is important to include relevant background on their respective roles in project successes and challenges. Methodological concerns (and deviations from the SOW) Omissions (including context) We recommend that the role of USAID and MOH in project successes and challenges should is addressed in the report. In addition, the report should explain why this EQ was not addressed. 5 The SOW states the DE will answer the following EQ (EQ4): How are Jalin and its partners capturing achievements, lessons learned, and performance data for local solutions and other project activities at the regional and national levels? SOW This EQ has not been addressed Methodological concerns (and deviations from the SOW) Omissions (including context) We recommend that the report explain why this EQ was not addressed. 6 SOW and ‘Final’ MTE report There is no alignment between the numbering and wording of the EQs in the SOW with the EQs in the report. Methodological concerns (and deviations from the SOW) If the report is redrafted in light of USAID Jalin's comments, then this issue could be addressed. 60 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV Ref. Quotation Reference USAID Jalin’s comment Issue USAID Jalin’s request 7 The SOW states: Qualitative data from key informant interviews (KII) and workshops will be coded and tallied whenever possible to determine themes, and key quotations will be extracted to emphasize findings. Qualitative analysis will include data reviews, development of a coding framework, identifying patterns and connections, content analysis, and potentially discourse analysis. The DE will seek to validate any quantitative data used before conducting descriptive analysis to identify trends, distributions, frequencies, and ratios. If divergent findings emerge, the DE will note this in its report and seek to generate nuanced explanations. SOW There are some quotations from KIIs and reference to desk review materials, but there is limited evidence of coding and/ or thematic presentation of qualitative findings. The USAID Jalin team is unaware of any validation of results with the implementing team. Methodological concerns (and deviations from the SOW) This presents an opportunity to strengthen the presentation of qualitative data. 8 Status of Regional Activities: USAID and the IP communicated extensively about regional programs. According to past observations, the IP proposed eliminating regional teams and activities in the redesign. A Jalin KII said, “We thought we were going only national. Body of the report This quotation is lacking context. The workplan was developed in response to USAID’s request; it was co-created with USAID and approved by MOH. Methodological concerns (and deviations from the SOW) We recommend that this observation is deleted from the report. 61 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV Ref. Quotation Reference USAID Jalin’s comment Issue USAID Jalin’s request Then local solutions came back due to the DE.” USAID Jalin’s Year 3 Work Plan shifted “from a bottom approach to top down,” according to a USAID KII. 9 As this evaluation was being revised in August 2020, the DE observed that USAID Jalin did not renew multiple staff contracts, including those working at the Secretariat. Body of the report This is outside the evaluation timeframe. Methodological concerns (and deviations from the SOW) Observations related to staff turnover should be deleted from the report. 10 Staff turnover: As noted in the Jalin Secretariat Section above, the DE observed that USAID Jalin did not renew multiple staff contracts in June￾August 2020 and hired new positions. This turnover has the potential to further delay Year 3 Work Plan implementation, because those whose contracts were not renewed include the Deputy Chief of Party for technical activities, all staff at the Jalin Secretariat, and some personnel responsible for Intermediate Result (IR) 1-3. Body of the report To achieve the Y2 & 3 workplan, USAID Jalin introduced, and managed necessary staffing changes based on a full understanding of staff skills and competencies. These staff changes fall outside the evaluation timeframe. USAID Jalin would also like to clarify that the Deputy Chief of Party Programs is with the project in his key role. The USAID Jalin team accommodated his request to work part time and focus on his Methodological concerns (and deviations from the SOW) The evaluation is projecting the results of these changes. We request that observations related to staff turnover should be deleted from the report. 62 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV Ref. Quotation Reference USAID Jalin’s comment Issue USAID Jalin’s request personal matters, this change was discussed and agreed with USAID. 11 As this evaluation was being revised in August 2020, the DE observed that USAID Jalin did not renew multiple staff contracts, including those working at the Secretariat. Body of the report USAID Jalin was not aware that in August the DE team revised the midterm evaluation as it is presented in this report. Methodological concerns (and deviations from the SOW) USAID Jalin was not informed of changes to the evaluation report in August. See point 1. This would have been a great opportunity for the project to provide feedback and strengthen the report. 12 N.A. N.A. The final report omits important information: • There is no list of people interviewed either through KII or during workshops. • The final SOW is not included as an annex. Omissions (including context) It will be useful to the reader to have the list of respondents and the final SOW included as annexes. 13 Mentoring guidelines, a deliverable for the Work Plan’s Intermediate Result (IR) 1, were behind schedule as of this evaluation. Regional teams have not begun to implement new national programs from this Work Plan due to a lack of clear and Exec Summary An assertion is made without explaining the background. Omissions (including context) The finding needs to be revised to include an explanation of why delays occurred. 63 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV Ref. Quotation Reference USAID Jalin’s comment Issue USAID Jalin’s request actionable guidance from the IP’s Jakarta headquarters (HQ). 14 N.A. N.A. The success of the Secretariat has not been adequately described. Omissions (including context) The excellent relations with MOH, at both the strategic and operational levels, with members of the Jalin team embedded within the MOH itself should be emphasized as a major success of the project under year 3. 15 USAID Jalin is not comprehensively supporting the 65 MOH priority districts. The Project has local solutions in 15 of these districts, which do not support all elements of MOH’s strategy. Body of the report The Y2 and 3 workplan includes activities that aim to reach all priority districts. USAID Jalin’s RF focuses on supporting improvements in MNH quality of care and referral systems at scale. Omissions (including context) It will be beneficial to the reader (given the re-focus of USAID Jalin in 2019) to understand which activities are local versus national in nature, and what the intended scale was. 16 N.A. N.A. The evaluation omits the leadership role of the MOH in the implementation of the Year 3 workplan that was heavily designed to directly support the MOH and the President’s priority to accelerate the MMR and NMR. Omissions (including context) USAID re-designed the project to not only be responsive to the priorities of the MOH but also to be fully led by the MOH. It will be useful to the reader to understand the role of MOH in the 64 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV Ref. Quotation Reference USAID Jalin’s comment Issue USAID Jalin’s request implementation of the Year 3 workplan. 17 Designing an interoperable information and communication technology (ICT) referral system under IR4 was behind schedule as of this evaluation. Furthermore, it is unclear that developing an interoperable platform alone will significantly improve the referral system. Exec Summary The statement that ‘it is unclear that developing an interoperable platform alone will significantly improve the referral system’ is at odds with the abstract which states ‘Findings do not suggest that USAID Jalin’s challenges are inherent to its design.’ The report does not sufficiently recognize the role of USAID, as an active partner in the original design and modified design (Year 2 & 3) of the project. The MOH’s approval of the Year 2 & 3 workplan is also not recognized. Omissions (including context) Balance and tone of the report The language in the abstract is misleading: it states challenges do not relate to design issues, yet the body of the report states otherwise. The report should recognize that USAID was an active partner in the original design and modified design (Year 2 & 3) of the project. In addition, the report should recognize that the Year 2 & 3 workplan and RF were co-created with USAID as well as the MOH. The MOH’s approval of the Year 2 & 3 workplan also needs to be recognized. 18 Lengthy Redesign: The process of redesigning these results and technical approach lasted approximately eight Body of the report The co-creation process was requested by USAID. The revised technical approach and the Balance and tone of the report The negative tone of the co-creation process, requested and led by 65 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV Ref. Quotation Reference USAID Jalin’s comment Issue USAID Jalin’s request months, beginning with the Mission’s Portfolio Review in May 2019 and concluding with the approval of the Year 3 Work Plan by USAID in November and MOH in December 2019. During this time, KIIs and workshops revealed that regional teams continued implementing local solutions, while senior HQ staff and short-term technical assistance (STTA) spent a significant amount of time responding to a Request for Proposal (RFP) issued by USAID and designing the new work plan. workplan was developed under the leadership of USAID and fully supporting the MOH priorities. Regional teams were managed by the HQ team in parallel with the workplan co-creation process. USAID agreed this had been a successful and well-managed process with much learning from the initial 18 months of the project and which resulted in a re￾designed project. USAID, should be addressed. The role of USAID in the redesign should be included in the report. 19 USAID Jalin has made limited progress toward its objective of contributing to Indonesia’s national goals of accelerated maternal and newborn mortality reduction in its three years of implementation. Abstract Many readers will only read the Abstract (and Executive Summary). Whilst these need to be concise, it is important they provide a balanced picture. The tone of the Abstract is negative and immediately indicates Jalin has significant problems. At the time of the review the project had a clear strategy, and Balance and tone of the report A more balanced view should be presented. The report should recognize the achievements of the project. 66 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV Ref. Quotation Reference USAID Jalin’s comment Issue USAID Jalin’s request relationships with the MOH, PHOs and USAID were strong: the ingredients for successful implementation were in place. 20 Executive Summary The Executive Summary provides an incomplete picture of key issues. Similar to the above point, the Executive Summary omits important contextual information: • The ‘redesign’ of the project in 2019 and its impact on delivery; • COVID-19 and its impact on a) access to, and availability of the MOH b) the implications for the Jalin team Balance and tone of the report Omissions (including context) A more balanced view should be presented. 21 As with IR 1 and 2, most work on IR 3 to date has been conducted by HQ rather than regional teams. Body of the report The full context needs to be provided. Balance and tone of the report Omissions (including context) We suggest that the report includes an explanation of why these activities are led by Jalin’s Jakarta office. 22 N.A. N.A. The report frequently refers to Jalin’s Jakarta office simply as ‘HQ.’ Balance and tone of the report To avoid ambiguity, it would be helpful if the 67 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV Ref. Quotation Reference USAID Jalin’s comment Issue USAID Jalin’s request This could be confused with DAI’s Global Health office based in Bethesda. The project has been managed in an integrated manner across all offices. The Regional Teams have been managed wholly by the Senior Leadership Team, lead by the Chief of Party (COP) to successfully address problems in management and staffing across all offices. project team is consistently referred to the Jakarta-based team as the ‘Jakarta office.’ We recommend that the report is adjusted to attribute success or failure of the project to USAID Jalin and not separately 'HQ’ versus Regional Offices. 23 N.A. N.A. The evaluation is not midterm. it focuses primarily on Year 3 workplan, which was responsive to the demands of USAID and was created in close consultation with the MOH. Balance and tone of the report Methodological concerns We recommend this is addressed in the report. 24 USAID Jalin did not comprehensively integrate local solutions into the Year 3 Work Plan and new results framework Exec Summary and body of the report and This sentence contradicts USAID’s approval of the workplan and numerous meetings held between USAID and the project during September – November 2019 where all LSs were discussed in detail. Inaccuracy The statement should be corrected. 68 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV Ref. Quotation Reference USAID Jalin’s comment Issue USAID Jalin’s request This is not accurate, the LSs were fully integrated. The USAID Jalin team was very responsive to USAID’s guidance and DE’s recommendations and fully incorporated the Local Solutions in the RF and in the workplan. 25 IR1: USAID Jalin has proposed an approach combining mentoring and accreditation to improve the quality of maternal and newborn services. Body of the report The accreditation is not part of the workplan and mentoring is an MOH-led activity. Inaccuracy The statement should be corrected. 26 The Year 3 Work Plan included an access to finance activity for private midwives and clinics in North Sumatera. However, HQ took over responsibility for this activity from the regional team. Body of the report This is incorrect. There was no effort to run this activity from Jakarta office. Inaccuracy The statement should be corrected. 27 In addition, USAID Jalin contributed a draft to Presidential Staff Office (Kantor Staf President/KSP) for the Protocol Petunjuk Praktis Layanan Kesehatan Ibu dan Bayi Baru Lahir selama Pandemi Covid 19 No: B-4 (05 April 2020) (Protocol for Practical Guidelines for Maternal and Newborn Health Services during the Covid-19 Pandemic). Body of the report This is incomplete. The national protocol (close to the draft written by the Jalin team) was finalized and published (May, 2020) by the President's Office, and USAID Jalin should receive full credit for this major contribution to the COVID-19 response. Inaccuracy Balance and tone of the report The statement should be corrected. 69 | MIDTERM EVALUATION OF THE USAID JALIN PROJECT USAID.GOV Ref. Quotation Reference USAID Jalin’s comment Issue USAID Jalin’s request 28 Please note that estimates of the local solutions have varied. The DE considers USAID Jalin to have developed 71 local solutions. USAID Jalin’s estimates have varied slight depending on the time period and the report. Please see Annex 7 for a Database of Local Solutions which tracks the local solutions since September 2019 and what the Project has done with them since according to Jalin’s reports (see Table 5 below for a summary). Body of the report Many of the ‘71 local solutions’ were ‘ideas’ and were not in implementation (as also recognized by DE on page 28 of the report.) During a meeting on 5 Nov 2019 between USAID, DE and Jalin it was agreed that Jalin implement 22 local solutions. ‘Ideas’ were not progressed (and referred to as ‘discarded’ in the report) for various reasons which are not explained in the report. Inaccuracy The report should explain that many of the ’71 local solutions’ were ideas and provide an explanation as to why they were not progressed. 29 Indicators to achieve or exceed targets for Year 3 (Sept 2020) by June 2020 (Result/Target) Body of the report There is an inconsistency. The box on p.10 indicates Jalin achieved 6 indicators, however Table 4 indicates Jalin achieved 5 indicators. The correct number is 7. Inaccuracy The report should be corrected.