TRANSFORM: MONITORING, EVALUATION, LEARNING AND ADAPTING (TRANSFORM: MELA) ACTIVITY TRANSFORM PROGRAM HIGH-LEVEL MONITORING (HLM) FINAL REPORT March 14, 2019 Submission Date: March 15, 2019 Contract Number: OAA-I-15-00028 Task Order No.: AID-663-TO-17-00001 Submitted to: USAID/Ethiopia Gebeyehu Abelti, COR Submitted by: Dr. Geoffrey Olupot, Chief of Party, Transform: MELA Activity, Addis, Ababa, Ethiopia Email: golupot@ethiopiatmela.com Phone: +251-986-35-6914 Principal contacts: Jenkins Cooper, Vice President, TMG Email: jenkinsc@the-mitchellgroup.com Phone: +001-202-567-1097 (Mobile) 202-350-0025 (Direct) This document was produced for review by the United States Agency for International Development Ethiopia (USAID/Ethiopia). A courtesy copy will be provided by the implementing partner to the Ethiopian Federal Ministry of Health and other relevant Government of Ethiopia offices. USAID/Transform Program High-level Monitoring Report, February 2019 Page 2 of 34 ACTIVITY INFORMATION Activity Title TRANSFORM: MONITORING, EVALUATION, LEARNING AND ADAPTING (Transform: MELA) Contract Number Contract No. OAA-I-15-00028 Task Order No.: AID-663-TO-17-00001 Name of Prime Implementing Partner The Mitchell Group, Inc. 1816 11th Street, NW Washington, DC 20001 Tel: 202-745-1919 Name(s) of Subcontractor(s)/Sub￾awardee(s) SUDCA, PRIN, & AAU/SPH Activity Start Date March 7, 2017 Activity End Date March 6, 2022 High-level Monitoring (HLM) Implementation Period October 2018 - February 2019 DISCLAIMER “The contents of this report are the sole responsibility of The Mitchell Group, Inc. and do not necessarily reflect the views of USAID or the United States Government.” Page 3 of 34 TABLE OF CONTENTS 1. BACKGROUND ...............................................................................................................................................8 1.1. Objectives of High-level Monitoring (HLM)...................................................................................................8 1.2. High-Level Monitoring (HLM) Indicators.......................................................................................................9 1.3. DESIGN & METHODOLOGY..........................................................................................................................11 1.4. Analysis Methods ........................................................................................................................................11 1.5. Limitations to the HLM and Data Quality Assurance Measures..................................................................12 2. HLM FINDINGS.............................................................................................................................................13 2.1. KEY RESULT AREA #1: INCREASED NUMBERS OF HEALTHY MOTHERS-SUCCESSFUL BIRTH OUTCOMES....13 2.1.1. Skilled Birth Attendance (SBA)..........................................................................................................13 2.1.2. Early Postnatal Care of Mothers (within two days of childbirth)......................................................14 2.2. Result Area #2: EXPANDED Access and Uptake of Family Planning ............................................................16 2.2.1. Modern Contraceptive Acceptance Rate (MCAR) ............................................................................16 2.2.2. Post Abortion Family Planning Use...................................................................................................17 2.3. Result Area#3: Increased numbers of healthy newborns ...........................................................................18 2.4. Result Area #4: Improved/Sustained Gains IN under Five Mortality...........................................................18 2.4.1. Pentavalent Third Dose Immunization Coverage .............................................................................18 2.4.2. Measles (MCV1) Immunization Coverage.........................................................................................19 2.4.3. Immunization Dropout Rate from Pentavalent First Dose to MCV1 ..............................................19 2.4.4. Proportion of Children Under Five Years of Age who had Presumed Pneumonia Treated with Antibiotics 21 2.4.5. Proportion of Children Under 5 years with Diarrhea who Received ORT & Zinc..............................21 2.4.6. Proportion of Children Under Five Years Screened for Acute Malnutrition.....................................22 2.4.7. Severe Acute Malnutrition................................................................................................................23 2.5. KEY RESULT AREA 5: Cross-Cutting..............................................................................................................25 2.5.1. Health Facilities with Essential Drug Available ................................................................................25 2.5.2. Health Facilities that Provide Post-Gender Based Violence Services ...............................................26 3. POSSIBLE FACTORS CONTRIBUTING TO THE DECLINE IN OBSERVED PERFORMANCE ................................26 4. CONCLUSION ...............................................................................................................................................28 5. LESSONS LEARNED.......................................................................................................................................28 6. RECOMMENDATIONS ..................................................................................................................................29 6.1. Recommendations to USAID .......................................................................................................................29 6.2. Recommendations to the Transform Program’s IPs....................................................................................29 6.3. Recommendations to the Government of Ethiopia/Federal Ministry of Health (GOE/FMOH)...................30 ANNEX 1: MATERNAL HEALTH INDICATORS BY INTERVENTION AND NON-INTERVENTION AREAS...................................................................................................................................................................31 Annex 2: Modern Contraceptive Acceptance Rate by Intervention and Non-intervention Areas..........................31 Annex 3: Performance on Immunization Indicators by Intervention and Non-intervention Woredas...................32 Annex 4: Performance on Diarrhea Treatment with ORS and Zinc, Screening for Malnutrition and Sever Acute Malnutrition indicators by Intervention and Non-intervention Woredas ...............................................................32 USAID/Transform Program High-level Monitoring Report, February 2019 Page 4 of 34 ACRONYMS AYH ANC Adolescent and Youth Health Antenatal Care BEmONC Basic Emergency Obstetric and Newborn Care CBNC CAR Community-based Newborn Care Contraceptive Acceptance rate CPR Contraceptive Prevalence Rate DHIS-2 District Health Information System-2 DHS Demographic and Health Survey EDHS EPHI EPI Ethiopia Demographic and Health Survey Ethiopian Public Health Institute Expanded Program of Immunization FGD Focus Group Discussion FMOH Federal Ministry of Health FP GAM Family Planning Global Acute Malnutrition GBV Gender Based Violence GOE Government of Ethiopia HC Health Center HDA HDR HEWs HEP Health Development Army Health in Developing Regions Health Extension Workers Health Extension Program HF Health Facility HH Household HMIS Health Management Information System HP HSTP Health Post The Health Sector Transformation Plan ICCM IDPs Integrated Community Case Management Internally Displaced Peoples IFHP Integrated Family Health Program IPs Implementing Partners KII KR Key Informant Interview Key Result LAFP MCAR MCV1 Long Acting Family Planning Modern Contraceptive Acceptance Rate Measles-containing-vaccine first-dose MELA ORT ORS Monitoring, Evaluation, Learning and Adapting Oral Rehydration Therapy Oral Rehydration Solution PAC PMA 2020 Post Abortion Care Performance Monitoring and Accountability (2020_) PHC Primary Health Care PNC Postnatal Care PPFP RHBs Post-partum Family Planning Regional Health Bureaus RH/FP Reproductive Health / Family Planning RMNCH SBA Reproductive, Maternal, Newborn, and Child Health Skilled Birth Attendance SAM Severe Acute Malnutrition SNNP SPSS Southern Nations, Nationalities and Peoples State Statistical Package for the Social Sciences TMG The Mitchell Group, Inc. USAID United States Agency for International Development USG United States Government WASH WHO YFS Water, Sanitation, and Hygiene World Health Organization Youth friendly Service Page 5 of 34 EXECUTIVE SUMMARY USAID gives due emphasis for measuring and documenting project and program achievements and shortcomings so that the Agency’s multiple stakeholders can gain an understanding of the returns on the investment in development activities. USAID recognizes that monitoring and evaluation is one of the key means through which it can obtain systematic and meaningful feedback on both the successes and shortcomings of the initiatives it supports. To measure progress towards the overall objective of the Transform activities, TMG was tasked by USAID/Ethiopia under the Monitoring, Evaluation, Learning and Adapting (Transform: MELA) Activity to conduct High-Level Monitoring (HLM during Q1 and Q2 of FY 2019. Purpose of the HLM The purpose of the HLM is to assess progress towards the achievement of results of the Transform program, and generate evidence for learning and adapting. This will also allow the Transform program’s Implementing Partners (IPs) to continuously assess progress towards the achievement of intended activity results and to provide a better understanding of the factors and context that may be contributing to observed results Methods: The analytical methods used in this HLM exercise include a “mixed methods” approach, relying on qualitative and quantitative data from both primary and secondary sources. The HLM exercise was designed and conducted in two phases: • Phase I: included collation and analysis of secondary data from the Federal Ministry of Health’s (FMOH) Health Management Information System (HMIS) and District Health Information System (DHIS2); a desk review of the Transform Program’s implementing mechanisms annual performance reports; random follow-up reports; FMOH annual reports: sectoral and other stakeholder documents; the Transform Program’s baseline survey; and meta-data analysis reports. • Phase II: was designed to address the "WHY" questions observed from the regions with declining trends for most of the HLM indicators. The primary data were collected from the FMOH and the regions of Amhara, Oromia, Afar, and Somali. Key Informant Interviews (KIIs) and Focus Group Discussions (FGDs) were conducted with participants from the FMOH, Regional Health Bureaus (RHBs), the Transform Program’s IPs and Zonal Health Officers. Site visits to selected districts (Woredas) were also performed. Analysis: Analysis of the data was mainly descriptive with a primary objective of conducting trend analysis of the value of the indicators between 2017 and 2018 in the Transform Program’s intervention areas. Distributions (visual presentation), trend analysis of the indicator’s values, comparative analysis of the indicator’s values (performance vs. targets, intervention vs. non-intervention, mean value performance) were conducted to assess the progress of Transform Programs’ interventions using a select set of indicators at the Programs level, disaggregated by regions USAID/Transform Program High-level Monitoring Report, February 2019 Page 6 of 34 where appropriate. Qualitative data was transcribed in Microsoft Word. The qualitative data were analyzed using thematic and content analysis. Some direct quotes have been included in the report. Key High-level Monitoring (HLM) Findings: 1. Skilled Birth Attendance (SBA): Nearly five in ten births were attended by skilled health personnel in the Transform Program’s intervention areas in 2018. There is a slight decline in the performance of SBA in 2018 as compared to 2017 in all regions except Tigray and Afar. A similar trend was observed in the non-intervention areas (54% to 50%). 2. Early Postnatal Care of Mothers: Three out of every four mothers received Postnatal Care (PNC) in the Transform Program’s intervention areas. A relatively low level of PNC is observed in the following developing regions: Afar, Gambella and Somali. There is a slight declining trend in 2018 as compared to 2017, except in the Southern Nations, Nationalities and Peoples State (SNNP) and Tigray regions. A similar declining trend is observed in non-intervention areas (65% to 63%). 3. Modern Contraceptive Acceptance Rate (MCAR): There is a slight improvement in MCAR in the Amhara, SNNP, Tigray and B/Gumuz regions. A substantial lower level of MCAR was found in the Afar, Gambella and Somali regions. Similar performance was observed in non-intervention areas 66% and 65% in 2017 and 2018, respectively. 4. Post Abortion Family Planning Use: One of the challenges for adolescent girls in Ethiopia is teenage pregnancy. At Youth Friendly Service (YFS) facilities, of 19,492 pregnancy tests, 6,349 (32%) were positive. In 2018, 80% of Post Abortion Care (PAC) clients received family planning methods at Transform: Public Health Care (PHC) YFS facilities. Comparison with the previous year (2017) was not possible due to unavailability of data. 5. Penta 3 Immunization Coverage: Overall, in the Transform program’s areas, Penta 3 immunization coverage was at 88% in 2018. Penta 3 immunization coverage significantly decreased in Afar and Gambella. The 2018 performance is slightly lower than the 2017 performance in most regions except the Tigray region. Similar slight declining trends were observed in non-intervention areas (86% and 85% in 2017 and 2018 respectively). 6. Measles Immunization Coverage: In the Transform program’s areas, about 8 in 10 children received measles immunization in 2018. Like Penta 3, the level of measles coverage showed a significant decrease in Afar, Gambella and Somali. Similar performance was observed in non-intervention areas (84% and 83% between 2017 and 2018, respectively). 7. Measles Dropout Rate from Penta-I: There was a one in ten defaulter rates of measles from Penta 1 at the Transform program’s level. An increasing trend of measles dropout rate was observed in all regions except B/Gumuz and Afar. Somali had the highest measles dropout rate in 2017 and 2018. Non-intervention areas had similar increasing trend, 12% and 13% in 2017 and 2018 respectively. 8. Under 5 Pneumonia Treatment with Antibiotics: Only one in four children under five received proper pneumonia treatment with antibiotics in the Transform program’s areas. Page 7 of 34 Overall, the 2018 performance is slightly higher than in 2017. Non-intervention areas also demonstrated a similar trend with 22% and 27% in 2017 and 2018 respectively. 9. Under 5 Screened for Acute Malnutrition: The under-five screening for acute malnutrition was generally low in all regions; the highest being in Tigray and the lowest in Afar. The level of under 5 screening for acute malnutrition was substantially lower in developing regions. The non-intervention areas showed a similar decline in performance from 2017 to 2018; 26% to 20%. 10. Severe Acute Malnutrition (SAM): A similar proportion of SAM among children under 5 was observed in 2017 and 2018. Relatively higher prevalence was observed in Afar and Somali; 8.9% and 19.6% in 2018 respectively. There was also increasing SAM cases seen in 2018 in Afar and Somali regions as compared to 2017. 11. Under 5 Children with Diarrhea Received Oral Rehydration Therapy (ORT) & Zinc: There was a low level of diarrhea treatment with Zinc and Oral Rehydration Solution (ORS) recorded among under 5 children (11 % at the Transform program’s level) in 2018. There is an observed differential in the level among the regions ranging from 14% in SNNP to 2% in Gambella. 12. Essential Drugs Availability (Agrarian Regions1 ): There was a 5% increase in the availability of essential drugs in the four agrarian regions between 2017 and 2018. While the Oromia region performed the least (57%), Tigray region has demonstrated the highest achievement (90%), which is also higher than the national performance (80%) in 2018. 13. Health Facilities (HFs) that Provide Post Gender Based Violence (GBV) Services (Agrarian Regions2 ): The proportion of HFs that provide post GBV services ranged from 84% in Tigray to 49% in SNNP. There is an improvement in the provision of post￾GBV services in 2018 as compared to 2017. The post-GBV services include treatment, counseling, provision of emergency contraceptives, and referrals, etc. 14. From qualitative data analysis the following are some of the identified possible factors for the observed performance include; intensified civil unrest and political instability in some parts of the country which in turn aggravated staff turnover at the health facilities and woredas’ health offices; reshuffling of regional health sector budget for humanitarian crisis management and other emergencies; weakened health sector management and linkages among different levels of the health system; continued reshuffling of leadership resulting in lesser commitment at different level of the health system, etc. 1 No data available for developing regions 2 No data available for developing regions USAID/Transform Program High-level Monitoring Report, February 2019 Page 8 of 34 1. BACKGROUND 1. Transform: MELA is undertaking High-Level Monitoring (HLM) to assess progress towards the achievement of the Transform program’s results. This is a periodic exercise that will continue throughout the life of the Transform program. HLM is different from Activity/Project Monitoring (A/PM) which deals with implementation monitoring, in the sense that it mainly focuses on the analysis of multiple datasets, where available, and generates a more meaningful performance trend on how interventions contribute towards the overall objectives of the Transform program. HLM encompasses performance and outcome level monitoring to create a knowledge base that will inform Transform program adaptive management. It also contributes to identifying learning/research questions for meta￾analysis that have the potential to yield substantive learning for future programming and to enhance impact. USAID/Ethiopia identified 19 HLM indicators (originally 18 and one additional indicator included at a later stage) to measure progress of the Transform program in achieving its overall results. These indicators are aligned with the Transform program’s four result areas, the Transform program’s IPs' performance monitoring indicators, and Ethiopia's Federal Ministry of Health’s (FMOH) needs. These indicators have Performance Indicator Reference Sheets (PIRS) which provide detailed descriptions of indicator definition, data collection and analysis methods, and data quality management procedures. They are mainly measured using secondary data sources from the FMOH HMIS/DHIS 2, the Transform program’s IPs and other stakeholders, and primary data collected through KIIs with FMOH, RHBs and IPs by Transform: MELA. 1.1.OBJECTIVES OF HLM 1 . 1 The purpose of the HLM is to assess progress towards the achievement of results of the Transform program and generate evidence for learning and adapting. This will also allow the Transform program’s IPs to continuously assess progress towards the achievement of intended activity results and to provide a better understanding of the factors and context that may be contributing to observed results. HLM is also to assess progress towards the achievement of results of the Transform program, generate evidence for learning and adapting. This will also allow the Transform program’s IPs to continuously assess progress towards the achievement of intended activity results and to provide a better understanding of the factors and context that may be contributing to observed results HLM will document the Transform program’s effectiveness, challenges, and progress against the stated objectives to date and helps to identify an adaptive approach to respond to changes. This effort will, therefore, provide independent accountability and reporting by ensuring timely results monitoring, as well as assessing the effectiveness of Reproductive, Maternal, Newborn and Child Health (RMNCH) interventions. Page 9 of 34 Specifically, the HLM analysis, will: 1. Assess the Transform program’s performance towards the achievement of its set objectives, providing both performance and contextual data for measuring and assessing changes within the USAID/Ethiopia Transform program; 2. Provide evidence-based data to inform decisions and effectively adapt programming; 3. Generate and document learning best practices; identify challenges during implementation; and provide context-specific recommendations to inform current designs within the implementation of RMNCH activities; and 4. Identify and monitor game changers or broad conditions/context that are beyond the Mission's control but could impede or improve implementation. In summary, with the understanding that the HLM exercise is focused mainly on monitoring performance at the outcome/program level, but not at an activity (IP) level, USAID/Ethiopia and the Transform program’s stakeholders will benefit from the availability of information and evidence, in the following ways: • Using 2017 & 2018 data analyzed performance trends of the select set of high-level indicators to assess the performance results of the Transform program; • Documented challenges to program implementation; • Documented possible reasons for observed patterns in the RMNCH performance and identify lessons learned; and • HLM findings that could inform and serve as inputs to the planned Transform program’s mid-term performance evaluations and mid-line and end-line impact evaluations. 1.2.HLM INDICATORS As mentioned above USAID identified 19 HLM indicators to measure the progress of the Transform program in achieving its overall results. These indicators are aligned with USAID/Ethiopia’s Transform “Result Areas”; the Transform IPs performance monitoring indicators; and the overall M&E needs of the FMOH. The indicators, which are selected for HLM, are listed below (Table 1): Table 1. HLM Indicators Transform Result Areas Indicators Note Remark KR1 Increased numbers of healthy mothers￾successful birth outcomes 1. Skilled birth attendance Analyzed 2. Early Postnatal Care (PNC) coverage for mother (within 2 days) Analyzed 3. The proportion of women giving birth who received uterotonics in the third stage of labor (or immediately after birth) Not￾Analyzed Analysis was not conducted as the indicator was not reportable by HMIS/DHIS2 and other data sources. IP random follow-up report for 2018 has not yet been produced. KR 2. Expanded access and uptake of 4. Modern Contraceptive Prevalence Rate (MCPR) Analyzed (with proxy indicator) MCPR requires a population-based survey. Due to this, Transform MELA analyzed Contraceptive Acceptance Rate. USAID/Transform Program High-level Monitoring Report, February 2019 Page 10 of 34 These indicators are sufficiently detailed and reliable to permit outcome/results monitoring, including spatial analysis and refigured comparison over time. Therefore, this HLM report is based on the twelve indicators shown as “analyzed” in Table 1 above. family planning 5. Post-Partum Family Planning (PPFP): Immediate post-partum contraceptive acceptance rate Not Analyzed Analysis was not conducted as the indicator was not reportable by HMIS, and other data sources. IP random follow-up report for 2018 has not yet been produced. 6. Post-abortion FP Analyzed Specific to Health Center’s (HC’s) providing youth￾friendly services in agrarian regions only; and data was not available for developing regions R3: Increased numbers of healthy newborns 7. The proportion of newborns who received post-natal care at least once within two days after birth Not Analyzed No data were available. KR 4: Sustained gains in and improved <5 mortality 8. Pentavalent 3 Immunization coverage (%) Analyzed 9. Measles (MCV1) immunization coverage (%) Analyzed 10. Immunization dropout rate from Penta 1 to MCV1 Analyzed 11. The proportion of under-five children with diarrhea who received ORT & Zinc Analyzed 12. Percent of children under five years of age who had presumed pneumonia treated with antibiotics Analyzed 13. The proportion of children under five with fever who are tested for malaria Not Analyzed No data available - requires population-based survey data 14. The proportion of children under five years screened for acute malnutrition Analyzed 15. Proportion of under 5 children with Severe Acute Malnutrition Analyzed Initially not proposed for HLM Cross-cutting 16. Percent of health facilities that provide post-GBV services Analyzed 17. Percentage of essential drugs available at health facilities Analyzed 18. The proportion of households that have access to basic sanitation facility Not Analyzed No complete data were available for all the regions. Reference is made to the National ONE WASH Phase I review, but it comprises only 3 regions (Tigray, SNNP, and Gambella) 19. Percent of households in the target areas practicing correct use of recommended household water treatment technologies Not Analyzed No data available - requires population-based survey data Page 11 of 34 1.3.DESIGN & METHODOLOGY The HLM exercise was designed and conducted in two phases. It utilized a mixed-methods approach, relying on qualitative and quantitative data from both primary and secondary sources. Phase I: This phase included collation and analysis of secondary data from the FMOH HMIS/DHIS2. The HMIS/DHIS2 dataset from the eight regions (329 intervention woredas (excluding scale up) and 396 nonintervention woredas) was utilized. Phase I also included a desk review of the Transform program’s implementing mechanisms, annual performance reports, random follow-up reports, FMOH annual report, sectoral and other stakeholder documents, the Transform program’s baseline survey, and meta-data analysis reports. The information obtained from Phase I served two purposes: a) constituted part of the Transform program’s HLM analysis data used to achieve the HLM exercise objectives; and b) provided information gaps that informed the design of the primary data collection for Phase II of the HLM exercise. Phase II: This phase was designed to address the "WHY" questions on the observed trends resulting from the interpretation of the secondary data analysis conducted during Phase 1 of the study. The primary data collection tools were developed and informed by observed results and conclusions of Phase I. Therefore, regions with declining trends for most of the HLM indicators were purposively selected for qualitative data collection. The primary data collection tools were developed collaboratively with USAID/Ethiopia. The primary data were collected from the FMOH in the regions of Amhara, Oromia, Afar, and Somali. Key Informant Interviews (KIIs) and Focus Group Discussions (FGDs) were conducted with participants from the FMOH, RHB, the Transform program’s IPs and Zonal Health Officers. Site visits to selected woredas were also carried out. Phase II of this study was also used to validate the quality of the data in addition to collecting data from the FGD and KII participants. Summary: Data from Phases I and II of the HLM exercise allowed triangulation of the data from different sources and, therefore, enhanced the quality and reliability of the data to generate evidence to address the HLM exercise objectives. 1.4.ANALYSIS METHODS All secondary HMIS/DHIS2 data were cleaned, checked, and validated to identify and address immediately any issues with possible outliers. The quantitative data for the identified HMIS/DHIS2 indicators were first organized as an Excel file before it was exported to Social Science program Statistical software (SPSS) for data management and analysis. Analysis of the data is mainly descriptive with the primary objective of conducting trend analysis of the value of the indicators between 2017 and 2018 in the Transform program’s intervention areas. Distributions, trend analysis of the indicators’ values, comparative analysis of the indicators’ values (performance vs. targets, intervention vs. non￾intervention, mean value performance) were conducted to describe the characteristics of a select USAID/Transform Program High-level Monitoring Report, February 2019 Page 12 of 34 set of the Transform program’s indicators at the Transform program’s level, disaggregated by regions where appropriate. Transform: MELA has undertaken a systematic analysis of primary and secondary data from different sources to establish trends of the selected HLM indicators. A regional comparison was also made, between and within the intervention and non-intervention areas of the Transform program at a point in time, as well as between 2017 and 2018. Additionally, a comparison of regional performance with national targets was conducted for each of the indicators for the year 2018. Furthermore, the values of the indicators in the Transform program’s intervention woredas were compared with regional targets for 2018. The qualitative analysis identified factors contributing to the decrease or increase of the indicators’ performance results using data collected from the FMOH, regional health bureaus, Zonal health offices and the Transform program’s IPs. Data from the FGDs and KIIs were transcribed and typed out in Microsoft Word. The FGDs and KIIs data were analyzed using thematic and content analysis. Some direct quotes have been included in the report. 1.5.LIMITATIONS TO THE HLM AND DATA QUALITY ASSURANCE MEASURES Although the methodology employed for the HLM study had appropriate statistical rigor, nevertheless, it was subject to two potential challenges: 1. Data Quality Issue with HMIS/DHIS2 and measures Taken: Transform: MELA utilized different approaches to ensure the credibility and reliability of the HMIS/DHIS2 dataset, including the following: • Reviewed the Ethiopian Public Health Institute (EPHI) Data Quality Review Reports for 2016 and 2018. From that review/assessment, more than 90% of the woredas reported data that is within the acceptable precision level (±10%) for sampled indicators (SBA, Penta3, and CAR); • Conducted data cleaning in collaboration with the FMOH team; • Performed quick analysis and identified outliers; • Validated outliers with the regional health bureaus and the observations were shared with the FMOH for further cleaning. This data cleaning process necessitated continuous consultation with the FMOH PPMED and Health Information Departments; and • Triangulation of data from various sources, such as the desk review of the Transform program’s implementing mechanisms; annual performance reports; random follow-up reports; the FMOH annual report; sectoral and other stakeholder documents; Transform program baseline survey; and meta-data analysis reports. 2. Unavailability of Data for Select HLM Indicators: Data was not available for seven HLM indicators, one of which was addressed using a proxy. Therefore, the HLM indicators, which are population-based, will be addressed during the planned mid-term and midline evaluations. Page 13 of 34 2. HLM FINDINGS The outcomes explored for the HLM analysis were analyzed in terms of intervention and non￾intervention areas and disaggregated by regions. The results of the analysis are linked to the results to which transform activities are contributing in order to measure the progress toward these results. 2.1.KEY RESULT AREA #1: INCREASED NUMBERS OF HEALTHY MOTHERS￾SUCCESSFUL BIRTH OUTCOMES 2.1.1. SKILLED BIRTH ATTENDANCE (SBA) All expecting mothers should have access to skilled care during pregnancy and childbirth to ensure prevention, early detection, and management of complications. Assistance by adequately trained health personnel with appropriate equipment is key to reducing maternal deaths. Figure 1 below shows the proportion of deliveries that were attended by skilled birth attendants in 2017 and 2018 in the Transform program’s intervention areas. Approximately more than five out of every ten births were attended by skilled health personnel in the Transform program’s intervention areas in 2018. However, the proportion of skilled birth attendance declined between 2017 and 2018 by 7%. As shown in Figure 1, except for the Tigray and Afar regions, a decrease in the proportion of deliveries by a skilled healthcare provider is observed in all regions. The highest percentage decline was observed in the Oromia region (11%) during the study period. Despite varying degrees of performance in 2018 among the regions, skilled birth attendance in the Transform program’s intervention areas is lower than the national performance (66%, HSTP, 2018 Woreda Based Core Plan) by 14 percentage points. A similar declining trend is observed in non-Transform program intervention areas (54% to 50% between 2017 and 2018 respectively). The distribution of SBA performance against woreda targets at the woreda level shows that in 2018, 116 woredas (40%) of the Transform program’s intervention woredas achieved less than 50% of their set target for the year 2018 (See the red color woredas, Map 1 below). Also, 20% of the intervention woredas achieved at least 80% of their set target in the same year. The woredas that achieved less than 50% of their set target (woredas in red color) need more support to increase the overall Transform program’s performance on skilled birth attendance. Transform implementing activities random follow ups may provide further exploration as to why these woredas are performing low. 59 20 60 36 18 60 69 31 52 53 20 54 34 17 49 66 25 57 79 Program level Afar Amhara B/G Gambella Oromia SNNP Somali Tigray Figure 1: Proportion of Births Attended by Skilled Health Personnel, by Region 2017 2018 National Target USAID/Transform Program High-level Monitoring Report, February 2019 Page 14 of 34 Map 1: Skilled Birth Attendance Performance by Woreda, 2018 2.1.2. EARLY POSTNATAL CARE OF MOTHERS (WITHIN TWO DAYS OF CHILDBIRTH) The World Health Organization (WHO) recommends that the first postnatal contact be done as early as possible, i.e., within 24 hours after birth. All post-partum mothers and newborns should receive at least three additional follow-up postnatal visits: a visit during the first 48–72 hours; another visit between days 7–14 days after birth; and a third visit six weeks after birth. (WHO’s recommendation on Post-Natal Care (PNC) of the mother and newborn, 2013). Figure 2 presents the proportion of mothers who received postnatal care within two days after childbirth in the Transform program’s intervention areas in 2017 and 2018. As shown in Figure 2, seven in ten mothers received postnatal care in the first two days (48 hours) after delivery in 2018. Figure 2 also shows that the proportion of mothers who received early postnatal care has declined by 4% between 2017 and 2018 in all Transform program supported regions, except for the Tigray and SNNP regions. The highest decline was observed in Oromia (10%), followed by Benishangul-Gumuz (6%), between 2017 and 2018. Despite a varying degree of performance in 2018, mothers who received postnatal care within two days after giving birth in the Transform program’s intervention areas is lower than the national performance average (77%, HSTP, 2018 Woreda Based Core Plan). A similar declining trend was observed in non-intervention areas between 2017 and 2018 (65% and 63% respectively), while their performance is relatively lower compared to the intervention areas 73% in 2017 and 69% in 2018. Page 15 of 34 Possible Factors for the Observed Performance Document review, FGDs, and KII findings suggest the following possible factors contributing to the observed slight decline: • Intensified civil unrest and political instability in some parts of the country which in turn aggravated staff turnover at the health facilities and woredas’ health offices • Heightened low level of community mobilization efforts by Health Extension Workers (HEWs) and Health Development Army (HDA) in some parts of the country. • Migration of HMIS data to DHIS-2 may have resulted in the variation of completeness and accuracy of reported data. The new system (DHIS-2) minimizes data manipulation which has been in the past years at different levels. One FGD participant in the Somali region had this to say: "In this year we have seen the worst public unrest in our community and especially in this region and this has, in turn, aggravated staff turnover at the health facilities and has contributed to the low performance of the skilled birth delivery. It left many facilities with no staff attending to patients, and it also affected the entire health system". One respondent from the Amhara regional health bureau explained the deterioration of the HEP as an essential factor for the decline in performance, stating: "The inefficient performance of the health extension program associated with the political turmoil resulted in the low motivation of HEWs and malfunctioning of the health development army." One respondent from the Somali regional health bureau, stated: “The general challenges of the health system in the developing regions coupled with aggravated political instabilities contributed for the delays in the implementation of the Transformational activities which in-turn affected the health service delivery in the regions including early postnatal care of mothers”. 73 37 73 78 22 77 76 49 74 69 32 71 72 17 67 78 47 80 92 Total Afar Amhara B/G Gambella Oromia SNNP Somali Tigray Figure 2. Proportion of mothers who received postnatal care within two days after childbirth 2017 2018 National Target USAID/Transform Program High-level Monitoring Report, February 2019 Page 16 of 34 Some of the factors which contributed to the low performance of the RMNCH services in the Somali region which KIIs discussants mentioned include: the recurrent drought in the region which forced the government to prioritize and allocate resources for humanitarian emergency and deprioritize other sectors; the internal conflict, which intensified in 2018 among different ethnic groups in the region, forced the health system functioning to deteriorate in the region; a lot of health facilities were closed and health workers left the area due to security concerns. One KII participant in Somali provided the following comment: "We had limited financial resources in 2018 as a significant portion of the regional budget was allocated for an emergency to take care of internally displaced people of the region and this created a shortage of stocks and equipment to provide the RMNCH services". Incompleteness of the DHIS2 is also mentioned as a major factor for the lower level indicator values. In the words of one of the FMOH participants: “Migration of HMIS data to DHIS2 may result in the variation of completeness and accuracy of reported data. The new system (DHIS2) minimizes data manipulation which has been in the past years at different levels. We believe we have got a better quality of data this year than the previous years." During the discussion with Oromia Regional Health Bureau officials, it was mentioned that there was high political commitment and a campaign to improve SBA in 2017 that contributed to better performance in SBA. However, the campaign did not extend into 2018, which was mentioned as one of the possible factors for the decline in SBA performance in 2018. In addition, the regional health bureau integrated a performance-based incentive mechanism to motivate health care providers for better performance in SBA, which, unfortunately, might have led to inflated reporting in 2017. One KII participant had this to say; “We placed a strong monitoring mechanism to improve the HMIS data reporting quality in 2018 with the motto of ‘No tolerance for false reporting!'. I think this might also have contributed to getting more precise data which in turn make the figures less than previous years because the figures are no longer inflated like before." A KII participant from FMOH had this to say; “Transform implementing partners should pay more attention to strengthen the capacity of regions through onsite mentoring and coaching with integrated supportive supervision” In general, the new internal displacement due to the political instability in the country and its consequence on the socioeconomic conditions of the people in different regions could have contributed to the low performance of health service delivery in 2018 with regards to the numbers of healthy mothers-successful birth outcomes. 2.2.RESULT AREA #2: EXPANDED ACCESS AND UPTAKE OF FAMILY PLANNING 2.2.1. MODERN CONTRACEPTIVE ACCEPTANCE RATE (MCAR) Figure 3 shows the proportion of women of reproductive age who accepted modern contraceptive methods, both new and repeating acceptors in the Transform program’s intervention areas in 2017 and 2018. As shown in Figure 3, overall there is no change in MCAR in 2017 and 2018 (67%) in Page 17 of 34 the Transform program’s intervention areas. However, modern contraceptive methods acceptance rate varies among the regions; with the lowest acceptance rate of 7% in Somali and the highest in Amhara (87%) during the year 2018. There is an increase in MCAR in Amhara, Benishangul￾Gumuz (B/G), SNNP and Tigray regions with the highest increase (10%) observed in the Benishangul-Gumuz region between 2017 and 2018. In 2018, the Amhara region exceeded the national target and the national performance by 8% and 17%, respectively. Similar performance is also observed in non-intervention areas, 66% and 65% between 2017 and 2018 respectively. Note: MCAR has been used in this analysis as a proxy indicator for Modern Contraceptive Prevalence Rate (MCPR). However, a document review showed that there was a 2.6% increase in MCPR among married women between 2017 (35.2%) and 2018 (37.8%) (Performance Monitoring and Accountability -PMA 2020, 2018). The Health Sector Transformation Plan (HSTP) has set a target to increase CPR to 55% by the end of the 2020. 2.2.2. POST ABORTION FAMILY PLANNING USE Post-abortion care is an important strategy to reduce maternal mortality by treating complications related to unsafe abortion and miscarriage, and by providing post abortion family planning counseling and services to prevent repeating unplanned pregnancies and abortions. According to the National Adolescent and Youth Health (AYH) Strategy, teenage pregnancy (13%) is one of the challenges for adolescent girls in Ethiopia. A review of Transform: Primary Health Care annual performance report shows that out of 922 PAC adolescent and youth clients, 809 (88%) received modern contraception at 244 YFS facilities in the four agrarian regions (See Figure 4)) Note: data was not available for developing regions. 67 26 86 51 24 68 66 12 62 67 21 87 61 19 66 70 7 65 79 Program Level Afar Amhara B/G Gambella Oromia SNNP Somali Tigray Figure 3. Modern Contraceptive Acceptance Rate, by region 2017 2018 National Target USAID/Transform Program High-level Monitoring Report, February 2019 Page 18 of 34 Figure 4: Post Abortion Clients that Received Family Planning Services 2.3.RESULT AREA#3: INCREASED NUMBERS OF HEALTHY NEWBORNS Analysis was not done due to unavailability of data. 2.4.RESULT AREA #4: IMPROVED/SUSTAINED GAINS IN UNDER FIVE MORTALITY 2.4.1. PENTAVALENT THIRD DOSE IMMUNIZATION COVERAGE Figure 5 presents the proportion of children under one year of age who received the third dose of pentavalent vaccine in the Transform program’s intervention areas in 2017 and 2018. As shown in the Figure, 88% of the children received pentavalent third dose immunization in 2018 in the Transform program’s intervention areas with a slight decrease (2%) from 2017 performance. Regionally, the performance in Oromia and Somali remained the same at 90% and 70% respectively between 2017 and 2018, and the Tigray region showed slight improvement, 3%. However, the Gambella and Afar regions have demonstrated the highest decline in pentavalent third dose immunization (by 22% and 20%, respectively). The 2018 performance in the Transform program’s intervention areas was below the national performance by 8%. Although the performance for children who received pentavalent third dose immunization in non-intervention areas was lower than the performance in the Transform program’s intervention areas in the two years, there were similar declining trends in intervention and non-intervention areas of 2% and 1% respectively between 2017 and 2018. 90 88 91 93 89 90 95 70 88 88 68 90 90 67 90 91 70 9198 Program-level Afar Amhara B/G Gambella Oromia SNNP Somali Tigray Figure 5. Penta 3 Immunization Coverage (%), by Region 2017 2018 2018 National target Page 19 of 34 2.4.2. MEASLES (MCV1) IMMUNIZATION COVERAGE In the Transform Program’s intervention areas, about eight in ten children under one year of age received measles immunization in 2018, with the highest percentage in SNNP (89%) and the lowest in Gambella (61%). As shown in Figure 6, there was a 4% decline in measles immunization coverage in the overall Transform program’s intervention areas between 2017 and 2018. Within this area, there was a decline in all regions except Benishangul-Gumuz, where it remained the same over the last two years at 88% with Afar and Gambella registering the highest drop, 21%. The 2018 performance in the Transform program’s intervention areas was below the national performance by 6%. Although the performance in measles immunization coverage in non-intervention areas was lower than in the Transform program intervention areas between 2017 and 2018, there was a similar declining trend in intervention and non-intervention areas of 4% and 1% respectively. The national performance in measles immunization coverage was higher than the program level performance by six percentage points. 2.4.3. IMMUNIZATION DROPOUT RATE FROM PENTAVALENT FIRST DOSE TO MCV1 Pentavalent first dose to measles dropout rate in the Transform program’s intervention areas for 2017 and 2018 is presented in Figure 7. The data showed that the measles immunization dropout rate among children who received the first dose of pentavalent immunization was about 11% in 2018. There was an overall increase in the dropout rate of 2%, from 9% in 2017 to 11% in 2018 in the Transform program’s intervention areas, thereby showing a performance decline. Regionally, Somali registered the highest dropout rate of 26% in 2018 showing a performance decline of 3% from 2017; Benishangul-Gumuz registered the steepest decline of pentavalent first dose to measles dropout, by 4%. The immunization dropout rate from pentavalent first dose to measles in non-intervention areas was slightly higher than in the Transform program’s intervention areas for both 2017 and 2018 (12% and 13%, respectively). There was a 2% increase in the dropout rate of pentavalent first dose to measles immunization in the intervention woredas as compared to a 1% increase in non￾88 85 89 88 82 87 94 69 87 84 64 87 88 61 85 89 62 86 96 Program Level Afar Amhara B/G Gambella Oromia SNNP Somali Tigray Figure 6. Measles Immunization Coverage (%), by Region 2017 2018 National Target USAID/Transform Program High-level Monitoring Report, February 2019 Page 20 of 34 intervention woredas between 2017 and 2018. There was also a 2% increase in immunization dropout from pentavalent first dose to measles at the national level as compared to the Transform program’s intervention areas. Map 2 below shows the distribution of woredas by measles drop-out rate. As the color changes to darker red, the drop-out rate increases and as the color changes to greener, the drop-out rate decreases. As seen from the map, more than half of the intervention woredas (52%) registered less than 10% measles drop-out rate. Forty-four (44%) of woredas registered a measles drop-out rate greater of more than 10%, showing the need for relatively concentrated support to bring down drop-out rates. 9 17 5 11 16 12 5 23 6 11 14 7 7 15 14 6 26 9 Program-level Afar Amhara B/G Gambella Oromia SNNP Somali Tigray Figure 7. Measles Drop-out Rate from Pentavalent first dose, by region 2017 2018 Page 21 of 34 2.4.4. PROPORTION OF CHILDREN UNDER FIVE YEARS OF AGE WHO HAD PRESUMED PNEUMONIA TREATED WITH ANTIBIOTICS Figure 8 below shows the percentage of children under five years in the Transform program’s intervention areas who had presumed pneumonia treated with antibiotics in 2017 and 2018. Overall, 25% or one in four children who had presumed pneumonia were treated with antibiotics in 2018. This shows an improvement in pneumonia treatment with antibiotics of 6% between 2017 and 2018. The regional disaggregation shows that Benishangul-Gumuz had the highest proportion (32%) of children under five years who had presumed pneumonia treated with antibiotics and Afar had the lowest (14%) in 2018. Although the performance in treating children under five years who had presumed pneumonia with antibiotics is higher in non-intervention areas as compared to the Transform program’s intervention areas between 2017 and 2018, there was a similar increasing trend in intervention and non-intervention areas of 6% and 5% respectively. However, no region has reached the target set under the HSTP to increase the proportion of children under five years who had presumed pneumonia treated with antibiotics to 48.2% in 2018. 2.4.5. PROPORTION OF CHILDREN UNDER 5 YEARS WITH DIARRHEA WHO RECEIVED ORT & ZINC Figure 9 presents the proportion of children with diarrhea under five years in the Transform program’s intervention areas who received ORT and Zinc treatment in 2018. In the Transform program’s intervention areas, only 11% received treatment with ORT and Zinc in 2018. Regionally, the lowest performance was observed in Gambella (2%) and the highest in SNNPR (14%), which is still very low. Note: Data on diarrhea treatment with ORT and Zinc was not available for the Tigray region because this is a new indicator included in DHIS2, and the region did not start implementing the DHIS 2 system where the indicator was supposed to be tracked. This could also account for the incompleteness of data for this specific indicator, because it is being newly tracked through the FMOH DHIS-2. 19 19 22 24 15 14 22 24 25 27 14 19 32 22 23 31 15 26 Program level Afar Amhara B/G Gambella Oromia SNNP Somali Tigray Figure 8. Proportion of children under five who had pneumonia treated with antibiotics, by region 2017 2018 USAID/Transform Program High-level Monitoring Report, February 2019 Page 22 of 34 2.4.6. PROPORTION OF CHILDREN UNDER FIVE YEARS SCREENED FOR ACUTE MALNUTRITION Figure 10 shows the proportion of children under 5 years screened for acute malnutrition in 2017 and 2018. The proportion of children screened for acute malnutrition was generally low across the Transform program’s intervention areas. Within the Transform program’s intervention areas, there was a decline of 8% in children under five years screened for acute malnutrition between 2017 and 2018. The proportion of screened children for acute malnutrition has declined in all regions between 2017 and 2018; the decline in performance is more pronounced in the SNNP region (13%) followed by the Oromia region (12%). The Tigray region registered the highest performance (60%) in screened children under five years for acute malnutrition, while Afar registered the lowest performance of 3% in 2018. Although the performance in screening children under five years for acute malnutrition in non￾intervention areas was lower than in the Transform program’s intervention areas between 2017 and 2018, there was a similar declining trend in intervention and non-intervention areas of 8% and 6% respectively. 35 6 44 24 13 33 33 7 64 27 3 43 20 8 21 20 6 60 Program level Afar Amhara B/G Gambella Oromia SNNP Somali Tigray Figure 10. Proportion of children under five screened for acute malnutrition, by region 2017 2018 Page 23 of 34 2.4.7. SEVERE ACUTE MALNUTRITION Figure 11 shows the proportion of under 5 children with Severe Acute Malnutrition (SAM) in 2017 and 2018. Within the Transform program’s intervention areas, the proportion of SAM cases were similar for both years. All regions, except Afar and Somali, maintained the low proportion in both years or slightly reduced between 2017 and 2018. Somali region exhibited the highest proportion with significant increase of 3% over the two-year period. The proportion in Somali region exceeded the threshold level of 15% of Global Acute Malnutrition (GAM) set by WHO. The other highest proportion was observed in the Afar region, with an increase of 0.5% over the two-year period. It was also noticed that in the regions where high proportions of children were screened for malnutrition (Figure 10), the proportion of SAM cases was found to be significantly lower. This could be associated with early identification of malnourished children which leads to early treatment/intervention, and, therefore, improved outcome. Non-intervention areas showed a higher proportion of SAM cases compared with the Transform program’s intervention areas in 2017 and 2018 (3.0% and 2.9% in 2017 and 2018 respectively). The spatial distribution of Severe Acute Malnutrition (SAM) cases at the woreda level showed more than 71% of the Transform program’s intervention woredas had below 1% SAM cases. Most of the woredas in the Somali and Afar regions showed a higher proportion of SAM cases (greater than 5%) relative to other regions. In addition, few woredas in the Afar (2), Somali (2) and SNNP (1) regions have more than 15% SAM cases which require urgent intervention. 1.6 8.4 0.6 0.5 1.8 0.8 1.1 16.6 1.7 0.6 8.9 0.5 0.5 1.8 0.7 1.0 19.6 0.4 Program Level Afar Amhara B/Gumuz Gambella Oromiya SNNPR Somali Tigray 2017 2018 Figure 11. Proportion of Children Under Five with Severe Acute Malnutrition, by Region USAID/Transform Program High-level Monitoring Report, February 2019 Page 24 of 34 MAP 3: Transform Intervention Woredas SAM Prevalence (%), 2018 Possible factors for the observed performance The available evidence attributes the decline in performance of child health outcomes to the effects of the aggravated conflict on the quality of services and availability at the public health facilities, erratic cold chain, inadequate linkage of the services with the community, and poor Expanded Program of Immunization (EPI) data management and utilization. Varying degree of implementation of under 5 children health services was also cited as a factor for the variation on the performance of the regions. A key informant at the FMOH noted the following: “Tigray region has fully implemented cost-free under 5 health services at health facilities that improve the health-service utilization of the community. Most of the other regions have implemented it only partially. This may have contributed to the better performance of child health indicators in the Tigray region." The key informant added that: “The performance of developing regions in implementing RMNCH services is still very low. One of the factors for the low performance of the developing regions particularly for child health services is the fact that some programs are not yet implemented in these regions for various reasons.” KII and FGD participants in the Amhara region separately had this to say: “The low performance of child immunization in recent years, and especially this year (2018) has become highly politicized. We have some activists posting negative propaganda on immunization as if the government is intentionally doing it to reduce the population size of the Amhara region”. Page 25 of 34 The unusual and prolonged draught appears to contribute to the low performance of child outcomes in the pastoral regions, mainly in Afar and Somali. Due to the nomadic lifestyle of the population, despite the willingness of most mothers to have their children immunized and screened for acute malnutrition, they might not be within or near to health facilities during the time of immunization. A KII participant from the Somali region health bureau stated: "Internal Conflict among different ethnic groups in the region forced the health system to collapse, and a lot of health facilities were closed, and health workers left the area due to security concerns." Another KII participant from the Somali regional health bureau added this: "Internal displacement of people exacerbated the outbreak of Acute Watery Diarrhea (AWD) which affected more than 32,000 people in the region in 2018, and there was Measles outbreak around Moyale area. This has affected the health system to shift the focus to treating emergencies than engaging in routine services" A Poor information management system was also mentioned as the main cause for the observed decline in the performance of child health indicators. One KII participant from the Afar region had this to say: "The issue of getting complete child health service data is a challenge in our region. There is underreporting as there is poor information recording and reporting at facility level, no electronic information management system is implemented even at the hospital level." 2.5.KEY RESULT AREA 5: CROSS-CUTTING 2.5.1. HEALTH FACILITIES WITH ESSENTIAL DRUG AVAILABLE Figure 12 presents essential drugs available in health facilities (health centers and primary hospitals) in the agrarian regions of the Transform program’s intervention areas in 2017 and 2018. Overall, essential drugs availability has shown a 5% increase in the four agrarian regions between 2017 and 2018, as an aggregate and considering regions independently. While the Oromia region performed the least, the Tigray region has demonstrated the highest achievement (90%), which is also higher than the national performance (80%) in 2018. Note: data for developing regions was not available. USAID/Transform Program High-level Monitoring Report, February 2019 Page 26 of 34 2.5.2. HEALTH FACILITIES THAT PROVIDE POST-GENDER BASED VIOLENCE SERVICES Figure 13 shows the percentage of health facilities in the Transform program’s intervention areas of the agrarian regions that provided post-GBV services in 2017 and 2018. In general, the percentage of health facilities providing post-GBV services has increased by 5% between the periods. In 2018, about 6 in 10 health facilities in the agrarian intervention areas were providing post-GBV services; Tigray being the region with the highest percentage of health facilities providing post-GBV service (84%) and SNNP the least (49%). However, in pastoral intervention areas, Dubti 3 hospital is the only health facility providing post-GBV services in 2018. Post-GBV services include treatment, counseling, provision of emergency contraceptives, and referrals, etc. 3. POSSIBLE FACTORS CONTRIBUTING TO THE DECLINE IN OBSERVED PERFORMANCE 3 USAID Transform: Health in Developing Regions, Annual Progress Report, FY 2018 54 54 51 44 80 59 59 56 49 84 Agrarian regions Amhara Oromia SNNP Tigray Figure 12. Percentage of Health Facilities (HFs) providing Post-GBV services in agrarian regions 2017 2018 54 54 51 44 80 59 59 56 49 84 Agrarian regions Amhara Oromia SNNP Tigray Figure 13. Percentage of Health Facilities (HFs) providing Post-GBV services in agrarian regions 2017 2018 Page 27 of 34 Based on the qualitative analysis, the following are some of the operational factors affecting sector performance; o Migration from HMIS to DHIS2 - a variation of completeness and accuracy of reported data among the regions. A respondent from FMOH had this to say: “Migration of HMIS data to DHIS2 may result in the variation of completeness and accuracy of reported data. The new system (DHIS2) minimizes data manipulation which has been in the past years at different levels. We believe we have got a better quality of data this year than the previous years." o Weak data management system at the health facility level, i.e, the problem in recording and reporting service data at health facilities. o Increased staff turnover at all levels due to unrest in some regions. o The regional health bureaus of Somali, Amhara and Oromia respondents had this to say: • Continued civil unrest has caused reshuffling of regional health sector budget for humanitarian crisis management and other emergencies (drought, Acute watery diarrhea, measles outbreak, internally displaced peoples); and increased influx of Internally Displaced People (IDPs) in some of the regions constrained the existing health service utilization. • Inefficient performance of the health extension program associated with low motivations of HEWs. • Weakened health sector management and linkages among different levels of the health system. • Continued reshuffling of leadership resulting in lesser commitment at different level of the health system. USAID/Transform Program High-level Monitoring Report, February 2019 Page 28 of 34 4. CONCLUSION • There is a general slight decline in most of the HLM indicators between 2017 and 2018, and similar trends are observed between the Transform program’s intervention areas and non-intervention areas. • There is comparable performance between the national and the Transform program’s intervention areas. • Aggravated civil unrest and political instability in some parts of the country in 2018 could have resulted in; increased staff turnover; reshuffling of health sector budget for humanitarian emergency crisis management; loose referral linkages; etc. could have contributed to the observed performance of key indicators. • The fact that some of the regions, such as Tigray, have fully implemented cost-free under￾five health services may have positively affected the service seeking behavior of the community, especially for RMNCH services. • The health system is not resilient enough to bear unexpected shocks and crises such as political instability, drought, and outbreaks that are happening in the country. • Although there was a slight decline in the performance in many of the HLM indicators, most of which are outcome level indicators, therefore, a noticeable change is not expected over a one-year period. • The HLM findings provide a good basis for the planned mid-term evaluations where the “why” questions will be examined more deeply. 5. LESSONS LEARNED • There is a direct relationship between ownership of data and its quality. Data quality can be improved with a strong sense of ownership at all levels. This requires increased utility and accountability at all levels of the health system. • Utility of routine health related data, e.g., HMIS/DHIS2, can be enhanced with several data quality assurance exercises supplemented with additional primary qualitative data. • M&E systems should take into consideration periodic context monitoring – context seems to play a significant role in the success of interventions. • The health system is still not resilient enough to bear unforeseen shocks such as political instability, drought, and outbreaks that could affect performance. • Good collaboration among the Transform program’s stakeholders and the GoE is critical, since it enabled Transform: MELA to access relevant data and information to the HLM exercise and enhanced the culture of data sharing. Page 29 of 34 6. RECOMMENDATIONS 6.1.RECOMMENDATIONS TO USAID ▪ Support to identify and integrate resiliency elements into the health system to enhance its ability to withstand shocks. ▪ Consider conducting a systems analysis of the health system/sector to better understand key factors impacting performance. ▪ Continued support to the revitalization and implementation of an effective and context-specific Health Extension Program (HEP), more specifically to pastoralist regions. ▪ It appears that some the Transform program’s IPs are spreading resources too thinly and, therefore, it is difficult to demonstrate results. It is recommended they strategically focus their limited resources in manageable areas. The Afar and Somali regional health bureau and regional IP staff had this to say: "Transform HDR, and the Regional Health Bureau team are closely working together to conduct joint supportive supervision and annual planning exercise. However, for better coordination, a plan alignment exercise would need to be conducted based on regional priorities and some degree of flexibility in shifting resources of activities already planned and covered by the RHB instead of Transform HDR investing in the same in the targeted woredas and facilities staff.” The Transform IP staff in Afar and Somali had this to say: “Better to scale down and focus on specific areas of support targeting manageable number of health facilities for better impact than trying to reach out to too many health facilities." ▪ Transform: MELA, with the support of USAID/Ethiopia, conduct a Data Quality Assessment (DQA) with the objective of support the FMOH to improve identified data quality issues. ▪ Continued support to FMOH to strengthen the HMIS, including DHIS2 roll-out in all the health facilities. ▪ Continue to provide essential support to the FMOH for an effective and context￾specific health extension program. 6.2.RECOMMENDATIONS TO THE TRANSFORM PROGRAM’S IPS ▪ Continue strengthening the coordination with the RHB: Conduct joint workplan alignment exercise with the RHB to align strategies and activities with regional priorities. ▪ There seems to be high expectations from the regions about what the IPs could support. There is a need for joint planning and requires frequent engagement with RHB heads for developing relationships, rapport and clarity on the scope of work. in order to minimize unplanned demands ▪ Planned activity implementation is slowed by the lack of strong institutional activity implementation and frequent turnover among regional and woreda leadership, especially in Somali. Therefore, continue to strengthen effective partnership arrangements that put FMOH and RHB at the center. This may encourage ownership of activities and improve the pace of activity implementation. USAID/Transform Program High-level Monitoring Report, February 2019 Page 30 of 34 ▪ Transform IPs should align their reporting to their set results targets and use achieved outputs and activities to support/demonstrate their contribution to the achievement of the said results. ▪ In consultation with USAID/Ethiopia and regional governments, consider focused implementation of key activities with the greatest potential to improve outcomes. ▪ Capacity strengthening efforts should pay more attention to onsite mentoring and coaching with integrated supportive supervision. ▪ Strengthen implementing interventions that enhance ownership and utilization of service data by the health facilities, and, hence improving the quality. ▪ Consider more support for regions with recent marked indicator performance declines and for those regions chronically underperforming. 6.3.RECOMMENDATIONS TO THE GOVERNMENT OF ETHIOPIA/FEDERAL MINISTRY OF HEALTH (GOE/FMOH) ▪ Strengthen backstopping and periodic supportive supervision to the RHBs. ▪ Support regions to strengthen accountability mechanism at all levels of the health system to improve data quality, reporting, and use. ▪ Improve the supplies management and availability of essential drugs at primary health care units. ▪ Revitalize and implement an effective and context-specific Health Extension Program (HEP) and devise mechanisms to motivate health care providers. ▪ Follow-up and support the full implementation of cost-free under-five health services. Page 31 of 34 ANNEX 1: MATERNAL HEALTH INDICATORS BY INTERVENTION AND NON￾INTERVENTION AREAS Region The proportion of births attended by a skilled healthcare provider The proportion of mothers who received postnatal care within two days of birth 2017 2018 2017 2018 Intervention (%) Non￾intervention (%) Intervention (%) Non￾intervention (%) Intervention (%) Non￾intervention (%) Intervention (%) Non￾intervention (%) Afar 20 44 20 44 37 58 32 59 Amhara 60 55 54 52 73 65 71 64 B/Gumuz 36 53 34 50 78 81 72 80 Gambella 18 20 17 19 22 23 17 20 Oromia 60 61 49 51 77 75 67 64 SNNP 69 71 66 68 76 77 78 79 Somali 31 28 25 32 49 39 47 41 Tigray 53 35 57 39 74 45 80 53 Total 59 54 52 50 73 65 69% 63 ANNEX 2: MODERN CONTRACEPTIVE ACCEPTANCE RATE BY INTERVENTION AND NON￾INTERVENTION AREAS Regions 2017 2018 Intervention (%) Non￾intervention (%) Intervention (%) Non￾intervention (%) Afar 26 49 21 44 Amhara 86 89 87 92 B/Gumuz 51 53 61 57 Gambella 24 19 19 17 Oromiya 68 74 66 70 SNNPR 66 77 70 79 Somali 12 13 7 15 Tigray 62 54 65 55 Grand Total 67 66 67 65 USAID/Transform Program High-level Monitoring Report, February 2019 Page 32 of 34 ANNEX 3: PERFORMANCE ON IMMUNIZATION INDICATORS BY INTERVENTION AND NON-INTERVENTION WOREDAS Proportion of children who received Penta-III immunization Proportion of children who received Measles Immunization Children dropout rate from Penta I Dose to MCV1 Region Intervention (%) Non￾intervention (%) Intervention (%) Non￾intervention (%) Intervention (%) Non￾intervention (%) 2017 2018 2017 2018 2017 2018 2017 2018 2017 2018 2017 2018 Program-level 90 88 86 85 88 84 83 81 9 11 12 13 Afar 88 68 89 85 85 64 84 78 17 14 15 15 Amhara 91 90 85 86 89 87 83 84 5 7 6 7 Benishangul￾Gumuz 93 90 96 95 88 88 93 95 11 7 14 9 Gambella 89 67 72 77 82 61 64 69 16 15 29 19 Oromia 90 90 89 87 87 85 86 81 12 14 13 15 SNNP 95 91 93 89 94 89 92 87 5 6 6 7 Somali 70 70 69 69 69 62 62 63 23 26 28 24 Tigray 88 91 70 75 87 86 70 71 6 9 7 12 ANNEX 4: PERFORMANCE ON DIARRHEA TREATMENT WITH ORS AND ZINC, SCREENING FOR MALNUTRITION AND SEVER ACUTE MALNUTRITION INDICATORS BY INTERVENTION AND NON-INTERVENTION WOREDAS The proportion of under-five children screened for malnutrition The proportion of Under 5 children with diarrhea received ORT & Zinc Proportion of under-five children with Severe Acute Malnutrition (SAM) Region Intervention (%) Non￾intervention (%) Interventio n (%) Non￾interventio n (%) Intervention (%) Non￾Intervention (%) 2017 2018 2017 2018 2018 2018 2017 2018 2017 2018 Program￾level 35 27 26 20 11 14 1.6 1.7 3.0 2.9 Afar 6 3 11 7 5 10 8.4 8.9 17.5 16.2 Amhara 44 43 31 32 7 10 0.6 0.5 0.5 0.4 B/G 24 20 20 22 8 14 0.5 0.5 0.5 0.3 Gambella 13 8 11 8 2 3 1.8 1.8 4.7 7.3 Oromia 33 21 29 20 12 15 0.8 0.7 0.9 0.8 SNNP 33 20 25 15 14 22 1.1 1.0 1.2 0.8 Somali 7 6 7 7 5 7 16.6 19.6 14.0 13.3 Tigray 64 61 42 41 - - 0.6 0.4 0.3 0.3 Page 33 of 34 LIST OF RELEVANT DOCUMENTS AND DATASETS USED. 1. FMOH HMIS/DHIS-2 datasets, 2017 & 2018 datasets from the 8 Transform program supported regions. 2. FMoH, Annual Health Sector Performance Report; EFY 2010(2017/18) 3. FMoH HSTP, Woreda Based Health Sector Annual Core Plan; EFY 2011 (2018/19) 4. FP 2020, Family Planning Brief; June-July 2018 survey 5. Transform: Primary Health Care, Year 2 Annual Progress Report 6. Transform: Primary Health Care Random Follow-ups Reports 7. Transform: Health in Developing Regions, Year 2 Annual Progress Report 8. Transform program Baseline Survey Report 2018 9. Transform: MELA Metadata Analysis Reports (unpublished 2018). USAID/Transform Program High-level Monitoring Report, February 2019 Page 34 of 34 Amref Africa 2017. Transform/ Health in Developing Regions Monitoring, Evaluation and U.S. Agency for International Development USAID/Ghana US Mission Compound 24 4th Circular Road Cantonments P.O. Box 1630 Accra, Ghana BP 49 U.S. Agency for International Development USAID/Ethiopia Entoto Road Addis Ababa, Ethiopia