in TRANSFORM: MONITORING, EVALUATION, LEARNING AND ADAPTING (Transform: MELA) Activity MID-TERM PERFORMANCE EVALUATION OF USAID TRANSFORM HEALTH IN DEVELOPING REGIONS (HDR) ACTIVITY FINAL REPORT July 2020 Submission Date: July 28, 2020 USAID/Ethiopia Gebeyehu Abelti, COR Submitted by: The Mitchell Group, Inc. (TMG) Contract Number: OAA-I-15-00028 Task Order No.: AID-663-TO-17-00001 Transform: MELA Activity, Addis Ababa, Ethiopia Dr. Geoffrey Olupot, Chief of Party Email: golupot@ethiopiatmela.com Phone: +251-986 356 914 Principal Contacts: Jenkins Cooper, Project Coordinator / Vice President, TMG Email: jenkinsc@the-mitchellgroup.com Phone: +001-202-567-1097 (Mobile) +001-202-350-0025 (Direct) This document was produced for review by the United States Agency for International Development Ethiopia (USAID/Ethiopia). It was prepared by The Mitchell Group, Inc. (TMG) for USAID/Ethiopia for the “Transform: Monitoring, Evaluation, Learning, and Adapting (Transform: MELA) Activity." Mid-term Performance Evaluation Report USAID Transform: HDR Activity i TABLE OF CONTENTS ACRONYMS .......................................................................................................................... iii EXECUTIVE SUMMARY........................................................................................................... 1 For the GOE/MOH & RHB .................................................................................................... 11 SECTION 1: INTRODUCTION................................................................................................. 14 1.1 Background of USAID Transform Health in Developing Regions (HDR) Activity.............. 14 1.1.1 Activity Description ......................................................................................................... 14 1.1.2 Transform: HDR Development Hypothesis................................................................... 15 1.2 Purpose and Use of the Mid-term Performance Evaluation of USAID Transform Health In Developing Regions (HDR) Activity....................................................................................... 17 1.3 Evaluation Questions................................................................................................... 17 SECTION 2: EVALUATION DESIGN & METHODOLOGY ........................................................... 18 2.1 EVALUATION DESIGN................................................................................................... 18 2.2 SAMPLING................................................................................................................... 18 2.3 SURVEY INSTRUMENTS................................................................................................ 22 2.4 DATA COLLECTION....................................................................................................... 22 2.5 DATA ANALYSIS ........................................................................................................... 23 2.6 ETHICAL PROCEDURES ................................................................................................. 25 2.7 LIMITATIONS OF THE EVALUATION .............................................................................. 25 SECTION 3: KEY FINDINGS.................................................................................................... 27 3.1 RELEVANCE.................................................................................................................. 27 3.2 EFFICIENCY .................................................................................................................. 31 3.3 EFFECTIVENESS............................................................................................................ 39 3.3.1FAMILY PLANNING (FP)................................................................................................ 42 3.3.2MATERNAL HEALTH..................................................................................................... 51 3.3.3NEWBORN HEALTH...................................................................................................... 61 Mid-term Performance Evaluation Report USAID Transform: HDR Activity ii 3.3.4CHILD HEALTH ............................................................................................................. 64 3.3.5Social Behavior Change and Communication (SBCC)..................................................... 70 3.3.6CROSS-CUTTING - WOMEN DECISION-MAKING AND MALE INVOLVEMENT................... 73 3.3.7TRANSFORM: HDR ACTIVITY PERFORMANCE ON set targets at midterm ...................... 76 3.3.8ANALYSIS OF HEALTH FACILITIES READINESS AND PRACTICE IN PREVENTION AND MANAGEMENT OF POST-PARTUM HEMORRHAGE (PPH)...................................................... 78 4. SUSTAINABILITY ELEMENTS ......................................................................................... 81 5. LESSONS LEARNED........................................................................................................... 82 6. MAJOR OPERATIONAL CHALLENGES............................................................................. 83 7. CONCLUSIONS ............................................................................................................. 84 8. RECOMMENDATIONS FOR KEY STAKEHOLDERS............................................................ 85 Annexes.............................................................................................................................. 88 Mid-term Performance Evaluation Report USAID Transform: HDR Activity iii ACRONYMS ANC Antenatal Care ARI Acute Respiratory Tract Infection CBNC Community-based Neonatal Care CBO CI CoE CPR Community-Based Organization Confidence Interval Centers of Excellence Contraceptive Prevalence Rate DHIS2 DHS DID District Health Information System Demographic and Health Survey Differences in Difference EA Enumeration Areas EDHS Ethiopia Demographic and Health Survey ENC FGD Essential newborn care Focus Group Discussion FMOH Federal Ministry of Health FP Family Planning GBV Gender-Based Violence GOE Government of Ethiopia HC Health Center HDR Health in Developing Regions HEW HF Health Extension Workers Health Facility HH Household HP Health Post HSTP Health Sector Transformation Plan ICCM Integrated Community Case Management IFA IP Iron and Folic Acid Implementing Partner ITN Insecticide Treated Net KII Key Informant Interview KMC KPI LAFP Kangaroo Mother Care Key Performance Indicators Long-Acting Family Planning LOP MDHS MELA Life of Project Mini Demographic and Health Survey Monitoring, Evaluation, Learning, and Adapting MCPR MCV MNCH OR PHC PHEM PNC Modern Contraceptive Prevalence Rate Measles Vaccine Maternal, Newborn, and Child Health Odds Ratio Primary Health Care Public Health Emergency Management Postnatal Care PPFP Post-partum Family Planning RH/FP Reproductive Health / Family Planning Mid-term Performance Evaluation Report USAID Transform: HDR Activity iv RHB RMNCH Regional Health Bureau Reproductive, Maternal, Newborn, and Child Health SBA TMG TOT USAID Skilled Birth Attendance The Mitchell Group, Inc. Training of Trainers United States Agency for International Development USG WASH United States Government Water Sanitation and Hygiene WoHO Woreda Health Office Mid-term Performance Evaluation Report USAID Transform: HDR Activity 1 EXECUTIVE SUMMARY This report covers the findings, conclusions, and recommendations of the mid-term performance evaluation of the USAID/Ethiopia-funded Transform Health in Developing Regions of Ethiopia (Transform: HDR) Activity. The evaluation was conducted by The Mitchell Group, Inc. (TMG) under the USAID/Ethiopia-funded Transform Monitoring, Evaluation and Learning (Transform: MELA) Activity between November 2019 and February 2020. Transform: HDR is a five-year (USD 34,949,637) activity implemented in collaboration with the Government of Ethiopia (GOE)/Federal Ministry of Health (FMOH). The activity’s strategic objective was to help Transform: HDR increase utilization of quality high-impact Maternal, Newborn, and Child Health and Family Planning (MNCH/FP) services in 1) Afar, 2) Benishangul Gumuz, 3) Gambella, and 4) Somali. Launched in 2017, Transform HDR implemented by a consortium led by AMREF (the prime awardee) with Project HOPE, IntraHealth, and General Electric as sub- awardees. The purpose of the Transform: HDR mid-term performance evaluation was to assess the continued relevance of the activity, and its efficiency, effectiveness, and sustainability of results. The evaluation flagged midcourse corrections needed to improve the effectiveness of the activity for the remaining duration of the activity. It will inform USAID/Ethiopia and the GOE/FMOH concerning future design/programming in the areas of Family Planning, Maternal, Newborn, and Child Health. Evaluation Design/Statistical Methods The Transform: HDR Activity mid-term performance evaluation household survey utilized a similar design to that of the overall Transform Program baseline survey conducted in 2017, to allow a comparison of results. The evaluation included a household/population survey of 2,635 women aged 15-49, supplemented by health facility data collection from 97 facilities, and qualitative data collection comprised of 23 key informant interviews (KIIs). The key stakeholders interviewed included the FMOH, Regional Health Bureau (RHB), woreda health officials, and other stakeholders. Seven (7) Focus Group Discussions (FGDs) conducted with communities targeting different age groups. The evaluation analysis includes descriptive statistics (univariate, bivariate, and multivariate/regression analysis), and difference-in-difference (DID) calculations. The results are disaggregated by region, and by Transform: HDR Activity intervention and non-Transform intervention areas. The evaluation also explored determinants for the key outcomes (Family Planning, Maternal Health, Newborn, and Child Health). The analytical approach adopted for this mid-term evaluation to assess effectiveness involves a comparison of changes across key indicators between the baseline survey (2017) and mid-term (2019) in Transform: HDR intervention areas versus non-Transform areas. Mid-term Performance Evaluation Report USAID Transform: HDR Activity 2 The analysis also included documenting changes in key outcomes of interest within Transform: HDR intervention groups and comparing results at the midpoint to those of the baseline. Household-level surveys and KIIs, focus group panels and a review of documents from USAID/ Ethiopia and Transform: HDR, including the Health Sector Transformation Plan (HSTP), Mini Demographic and Health Survey (MDHS) 2019, provided both qualitative and additional quantitative evidence that shaped findings and the team’s recommendations. Limitations of the Evaluation The methodology employed for the mid-term performance evaluation, while appropriate and statistically rigorous, nevertheless had some limitations: ▪ The absence of the Enumeration Area (EA) list from the Ethiopian Central Statistical Agency (CSA), which is in charge of the national census and establishment of EAs, necessitated the use of the list of kebeles for each intervention area as a sampling frame for the first stage of sampling. Transform: MELA utilized kebele guides to clarify and identify boundaries of the selected kebeles and to ensure the complete listing of households in selected gotts. Sampling with replacement was employed, especially where security concerns and inaccessibility impeded data collection in some enumeration areas. ▪ Another limitation was isolating confounding factors while using DID analysis in the context of development programs taking place in both USAID/Ethiopia Transform: HDR Activity areas and the surrounding areas outside the boundaries of the evaluated regions. During the survey, partner mapping information was gathered from both Transform: HDR intervention sites and non-Transform sites. The evaluation team found that a myriad of other partner-funded programs exists in both Transform: HDR intervention sites and non￾Transform sites. Thus, it is difficult to definitively discern the results and impact of other woreda-level, MOH, or donor financed programs on achieving MNCH/FP targets. The evaluation team relied on the DID and causality tracking strategies analysis to overcome these challenges. Hence the DID results should be interpreted, bearing in mind these considerations. ▪ An additional limitation was the evaluation team’s difficulty in obtaining budget data on MNCH/FP programs operating in non-USAID Transform: HDR supported sites, hampering the team’s ability to carry out a comparative assessment of efficiency based on financial investments. Summary of Key Evaluation Findings There have been significant performance improvements between the baseline (2017) and mid￾term evaluations. The Transform: HDR Activity is on track to achieve the intended results. The evaluation concluded that the Transform: HDR Activity is well aligned with the GOE’s health Mid-term Performance Evaluation Report USAID Transform: HDR Activity 3 sector transformation agenda and is making promising progress towards achieving the intended results. Transform: HDR has been particularly successful in contributing to the achievement of most of the Family Planning, and Maternal, Newborn, and Child Health outcomes and is on track to achieve these planned results. A comparative statistical analysis between the baseline (2017) and the mid-term (2019)shows that the groundwork has been laid for advancing towards the full achievement of all of the Activity’s targets, with some targets already achieved by the midpoint. Transform: HDR supported areas registered a significant performance improvement compared to non-Transform: HDR areas. Relevance1 : Transform: HDR’s strategies and approaches are well aligned with the priorities of the regional health bureaus in the targeted regions. The initial joint planning exercise with these regions allowed the regional health teams to shape Transform: HDR and adapt it to the contextual challenges faced within their communities and their special health service delivery requirements. Some of the most relevant and responsive strategies and approaches embraced by Transform: HDR has been the supportive supervision, outreach, and mobile health services. Quality improvement and maternal health technologies are aligned with RHB and woreda needs and support the woreda transformation agenda. There was consensus among the RHBs that these approaches are going beyond the fixed health facility, and are well-adapted and responsive to the regional needs (observed during the measles outbreak campaigns and pre-service training). Transform: HDR interventions provided strong on-site post-training follow-up coaching and mentoring of health workers. The continued close coordination of activities and tailored technical assistance provided to woredas have also ensured that programs are relevant to meeting the requirements of the communities. Efficiency2 : The initial Activity design called for 58 woredas to be covered across the four regions. Given the complexity of reaching these regions, the staffing configuration and other challenges led to a decision made by the Transform: HDR team in consultations with the GOE/MOH to concentrate technical assistance in only 20 learning woredas and provide intensive support to 24 Centers of Excellence (CoE) and the associated health facilities. Some, but not all, activities were carried out in the remaining 38 woredas. Transform: HDR has several noteworthy examples of services and technologies (i.e., mobile outreach and supply of ultrasound) that have been rolled out, and this support is attracting more clients to the health facilities, thereby enhancing the utilization of services. Notwithstanding this positive accomplishment, the activity struggled to recruit and deploy qualified staff at the regional and sub-regional levels to implement its interventions. Presently, there are 54 technical staff and 60 support staff deployed. However, several stakeholders expressed concern that Transform: HDR was too thinly staffed to be able to successfully implement its interventions and build meaningful and strategic local partnerships. Key Informants (KIs) also noted that the team was too small to cover all aspects of the ambitious MNCH/FP agenda. A short-term strategy that could be employed by Transform: HDR is to assign work in the regions, districts, and specific geographies to their central staff, while its field team 1 Relevance: The extent to which the objectives of a development intervention are consistent with beneficiaries’ requirements, country needs, global priorities and partners’ and donors’ policies 2 Efficiency: A measure of how economically resources/inputs (funds, expertise, time, etc.) are converted to results. Mid-term Performance Evaluation Report USAID Transform: HDR Activity 4 focuses their support on the regions, to ensure a balanced workload between central teams and field-level technical support. Most of the implementation management systems reviewed, including work plans, budgets, management information systems, and logistics, while adequate to initiate program activities, may need to be strengthened and refined as Transform: HDR expands to more woredas and new hard-to-reach populations within the existing woredas. There is also a need to rethink the Transform: HDR technical staffing structure. An ideal staffing structure should be one that is able to effectively support regional and local health systems to deliver health services and to strengthen strategic partnerships with other key actors in recognition of their niches and comparative advantages. Finally, while Transform: HDR is supporting the District Health Information Software System (DHIS2) rollout, greater technical assistance is required to strengthen data utilization, especially for learning and decision-making. Of particular importance, data concerning the health-seeking patterns and outcomes of pastoral communities should be reviewed and used to refine service delivery approaches for this hard-to-reach, underserved population. Transform: HDR allocated 60% of funds to maternal health; 24% for FP; and 16% for child and newborn health. Maternal health in all regions showed significant improvements in most indicators. This is explained by the activity resource intensity in this thematic area. Effectiveness3 : The evaluation documents achievements in most of the MNCH/FP interventions in the Transform: HDR areas. Evidence also shows that Transform: HDR has performed well and achieved its targets, exceeding expectations in some areas. This achievement warrants recognition. However, the performance varies from region to region. Another major finding is that, while Transform: HDR indicators are improving, non-Transform: HDR areas have shown a decline in performance in key outcomes (e.g., MCPR, Skilled Birth Attendance, ANC4+, etc.) which should be a cause for concern for the GOE (see Table A for details). The performance was compared between baseline and mid-term and non-Transform: HDR intervention sites, including set targets’ in the life of the project (LOP). The difference in performance could be attributed to the level of Transform: HDR support in relation to the external and local financing for non-Transform: HDR sites. The findings from the evaluations have shown the factors responsible for the observed performance in Transform: HDR intervention sites and non-Transform: HDR sites in family planning, maternal health, and child health seem to be similar, with limited varying factors specific to pastoral regions. Also, due to similar population characteristics and spillover effect within woredas in the same regions, the evaluation found that determinants tend to be the same between Transform: HDR supported sites and non-Transform: HDR sites. Thus, the observed differences in trends could be explained by partners and their associated investment intensities in their interventions/programs as the plausible determinants. However, there are some differences between the pastoral regions and agrarian regions. A summary of the performance for key thematic areas is as follows: 3 Effectiveness: The extent to which the development intervention’s objectives were achieved, or are expected to be achieved, taking into account their relative importance. Mid-term Performance Evaluation Report USAID Transform: HDR Activity 5 ✔Family Planning: The use of modern family planning (FP) methods in Transform: HDR sites has significantly improved. Modern Contraceptive Prevalence Rate (MCPR) among married women increased by 2.9 percentage points (18.9% at baseline and 21.8% at mid-term), and the use of Long-acting Family Planning Methods(LAFP) increased by 2.1 percentage points (3.0% at baseline and 5.1% at mid-term). Unmet need for family planning among married women declined by 1.8 percentage points (from 25.1% to 23.3%). Demand for modern contraceptive methods among married women increased over time, rising from 44% at baseline to 45.1% at midterm. The use of family planning methods in the extended postpartum period also showed a significant increase from the baseline (12.2% at baseline and 16.2% at mid-term). Of note, FP does not appear to be improving in non-intervention areas. For example, the MCPR among married women has declined from 26% to 23.45%, and LAFP and post-partum family planning (PPFP) also showed a slight decline. Unmet need for family planning among married women also declined by 6.2 percentage points. Demand for modern contraceptive methods among married women declined by 8.7 percentage points, from 56.1% to 47.4%. ✔Maternal Health: The majority of key maternal health (MH) indicators have improved. Skilled birth attendance (SBA) significantly increased from 28.1% at baseline to 34.3% at mid-term (by 6.2 percentage points). Early postnatal care for the mother within two days also increased significantly from 22.9% at baseline to 40.1% by mid-term (by 17.2 percentage points). Women who received MNCH services through a mobile health team increased significantly from 6.4% at baseline to 24.8% at mid-term (by 18.4 percentage points). The above notwithstanding, the proportion of women who received four or more antenatal care (ANC) visits did not change, and Iron and Folic Acid supplementation for at least 90 days dropped significantly by 8.3 percentage points in intervention areas. Similar to the results in family planning, the majority of maternal health indicators in non-Transform: HDR sites declined. A discussion with KIIs and a review of the technical health literature on this subject found that there are several technical explanations for these findings: The efforts to boost performance need to introduce innovative and effective strategies to overcome barriers such as fear of contraceptive’s side effects, lack of women’s autonomy in health care decisions, poor health service provision, women low agency, and cultural and traditional/religious restrictions, including value attached to children. In addition, context specific actions need to be considered in order to close the gaps in regional inequalities. Coupled with increase in availability of free and long-acting contraceptive methods, Transform: HDR has started working through community awareness forums using local CBOs, religious or clan leaders, and this has significantly contributed to minimize some of the myths and increase service utilization. In non-Transform: HDR intervention sites, there are partners, and the few need to adapt these approaches. The frequent stock-outs and poor stock management of essential products could increase the gaps coupled with the health extension program which is still weak in developing regions to contribute to the better achievements in contraceptive prevalence rates. 5,6 Mid-term Performance Evaluation Report USAID Transform: HDR Activity 6 • There is also a need to actively engage and motivate health extension workers (HEWs), expand new health service facilities, and increase emphasis on maternal health. All of these factors are believed to have contributed to improving the coverage of maternal care in woredas and regions4 . These findings should be viewed within the context of the dynamic and changing political and social context faced by some of these regions. • The other common reasons hindering pregnant mothers from attending ANC include women's lack of awareness of its importance, distance to health facilities, and limited availability of affordable transportation. On the other hand, important determinants that are facilitating ANC attendance include higher education level, a woman's ability to make healthcare decisions, partner involvement, and MCH messaging.56 • Regional KIIs from the pastoralist regions of Afar and Somali all echoed the factors responsible for observed maternal services utilization to include socio-cultural factors, the mobile lifestyle of the populations; distance to health facility; and limited availability of waiting areas at health facilities. Other factors mentioned affecting the utilization of skilled delivery services include low education on maternal health, limited availability of information; lack of skilled delivery experience; and quality of antenatal care utilization78 . The poor-quality services that are not mother-friendly as well as poorly equipped health facilities available may not convince mothers to deliver in health facilities9 . It is important to address individual, family, cultural, and structural factors to substantially increase deliveries attended by skilled attendants. • Other factors raised that affect maternal service utilization include the readiness of the health facilities in providing quality care in terms of having regular, uninterrupted electricity, improved water sources in the facility, and long travel distance to the health facility16. These utilities are very important in providing skilled delivery services1011 . • Performance in post-natal care (PNC) is attributed to the cultural practice that encourages confinement, lack of mothers’ awareness, distance and scattered nature of villages and homes throughout rural areas, increased workload of HEW, not enough HEW in rural areas especially in developing regions, are among the barriers to achieving postnatal care targets and results. The PNC utilization is particularly low in the pastoralist regions. The very low PNC is related to the low use of SBA. Other barriers are related to physical 4 Interviews in FMOH Key Informant. 5 23 Fekadu Y, Mesfin A, Haile D, and Stoecker B. Factors associated with nutritional status of infants and young children in Somali Region, Ethiopia: a cross- sectional study, BMC Public Health (2015) 15:846 6 Fekadu H, Tesfahun A and Mequanent M. Late initiation of antenatal care and associated factors among pregnant women in Addis Zemen primary hospital, South Gondar, Ethiopia. Reproductive Health (2019) 16:73. https://doi.org/10.1186/s12978-019-0745-2 7 RHB KIIs from Afar and Somali 8 Tesfaye B, Atique S, Azim T, and Kebede M. Predicting skilled delivery service use in Ethiopia: dual application of logistic regression and machine learning algorithms. BMC Medical Informatics and Decision Making (2019) 19:209 9 WHO. Policy dialogue. Improving Skilled Birth Attendance in Ethiopia. Accessed on 25 December 2019. https://www.who.int/evidence/sure/pdimprovingskilledbirthattendanceethiopia.pdf?ua=1 10 Woreda KIIs and FGD from KII in Afar Mille Woreda. 11 Ethiopian Public Health Institute (EPHI)/Federal Ministry of Health and ICF International. Ethiopia Service Provision Assessment Plus-Census. Addis Ababa, Ethiopia. 2014. https://www.ephi.gov.et/images/pictures/FINAL%20draft%20SPA+%20%20REPORT%20survey%20tools%20adjusted%20- %20Dec%2024%20%202015.pdf accessed 25 November 2019. Mid-term Performance Evaluation Report USAID Transform: HDR Activity 7 distance, low maternal education, rural residence, and lack of awareness. Thus, it is critical to strengthen the implementation of strategies that help mothers seek postnatal care. ✔Newborn Health: The majority of the newborn health indicators showed considerable improvement at the mid-term, compared with the baseline values. For example, early postnatal care for the newborn (within two days of birth) significantly increased from 23.4% at baseline to 37.8% at mid-term (an increase by 14.4 percentage points). Also, early initiation of breastfeeding (within 1 hour of birth) significantly increased from 53.5% at baseline to 81.9% at mid-term (an increase of 28.4 percentage points). However, the proportion of newborns who received essential newborn care (ENC) remains the same between the two periods, 8.6% at baseline versus 8.2% at the mid-term. A similar performance trend is observed in non-Transform: HDR sites for essential newborn care and early initiation of breastfeeding. While postnatal care within two days increased significantly from baseline in the intervention areas, it was stagnant in non￾Transform: HDR areas. ✔Child Health: Many of the key child health indicators have improved in Transform: HDR sites compared with the baseline values. Almost all immunization indicators improved. Measles coverage significantly increased by 23.5 percentage points (from 28.1% at baseline to 51.6 % at mid-term). Similarly, Penta 3 coverage significantly increased by 8.7 percentage points (from 31.3% at baseline to 40% at mid-term). Full immunization also increased by 5.2 percentage points (from 27.6% at baseline to 32.8% at mid-term). Acute respiratory tract infection (ARI) treatment with antibiotics increased significantly by 57.6 percentage points (from 16.8% at baseline to 74.4% at mid-term). Children 24-59 months who were given deworming medication improved from 27.4% at baseline to 33.7% at mid-term (by 6.3 percentage points), and children under five years who slept under Insecticide Treated Net (ITN) during the previous night of the survey significantly increased from 53.9% at baseline to 68.7% at mid-term (a 14.8 percentage points increase). The one area where there was no substantial progress was on the treatment of children for diarrhea with ORS and Zinc. The evaluators consistently found that there were stock￾outs of Zinc tablets at health facilities. Factors affecting child health include parental lack of, or levels of education, poor awareness of immunization, the distance between their home and health facilities, lack of antenatal care, and home delivery. Religious beliefs and fear of the side effects of vaccinations result in low vaccination coverage or untimely vaccinations, which result in low immunization coverage1213. . . . . . Other factors include the availability of trained health workers to deliver child health services, sustainable funding, vaccine stock-outs and logistics problems, data issues, and weak laboratory infrastructure. Sustainability: The Transform: HDR Activity supports government priorities for woreda transformation. Transform: HDR implementation modalities are an integral part of the GOE’s health sector strategy, and the activity has no parallel implementation structure. This promotes self-reliance and the sustainability of MNCH/FP interventions and enhances woreda transformation. Other initiatives to enhance the sustainability of Activity results include political 12 Woreda KIIs and FGD from KII in Afar Mille Woreda. 13 WHO. Policy dialogue. Improving Skilled Birth Attendance in Ethiopia. Accessed on 25 December 2019. https://www.who.int/evidence/sure/pdimprovingskilledbirthattendanceethiopia.pdf?ua=1 Mid-term Performance Evaluation Report USAID Transform: HDR Activity 8 will development, co-ownership/implementation of activities such as mentoring and supportive supervision to enhance skills within the facilities; other partnerships with stakeholders beyond government; program and service adaptions; and organizational capacity building14 . Sustainability, however, remains a challenge in these particular developing regions of Ethiopia. These regions have relatively weaker health systems, outdated equipment, and staffing shortages for all categories of medical personnel. The costs of delivering services through mobile outreach areas are also higher than costs associated with reliance on fixed facilities. Transform: HDR’s Strengths: ▪ Highly responsive team and systems in place. ▪ Good collaboration within the consortium that has extensive field implementation programs in the regions. ▪ High marks and praise for the Transform: HDR Technical Assistance (TA) team in support of centers of excellence. ▪ USAID/Ethiopia is regarded as responsive by the RHBs and non-health authorities. Transform: HDR has succeeded in elevating a positive image for USAID/Ethiopia. ▪ Transform: HDR is proving to be a useful mechanism for local governments to achieve important health reform measures, including woreda transformation in the challenging developing region settings. Conclusions The Transform: HDR Activity has achieved impressive health results and has successfully built a management and assistance platform that is responsive to the GOE/MOH’s health sector transformation agenda. Thus, Transform: HDR is on track to meet most of its intended targets by the end date of the activity. Transform: HDR is positively regarded across the intervention regions by RHBs and woredas and is providing tangible and meaningful assistance to the GOE/MOH to achieve its desired goal to end preventable maternal, newborn, and child deaths. Most importantly, beneficiaries note that Transform: HDR has led to improved access and quality of care. The evaluation found that some targets could have been even more ambitious. Target setting may need to consider factors such as past experiences in achieving similar targets in similar settings, expert opinions, and available resources. As this evaluation was designed as a mid-term performance evaluation, it focused on assessing “progress towards” or the “likelihood of achieving intended results” rather than “measuring the full achievement of intended results and attribution.” The analysis also assessed Transform: HDR’s contribution to different thematic area results. With this premise in mind, the evaluation concluded that the activity is on track to achieve all of its intended results. 14 Transform HDR technical team KIs Mid-term Performance Evaluation Report USAID Transform: HDR Activity 9 Key Recommendations for Stakeholders: Recommendation/Action Actor to Address Timing Priority # USAID/Ethiopia 1 Given the good results and the immense unmet needs in the four regions, if funding is available, USAID/Ethiopia might consider expanding the activity and increasing its investment in this successful set of interventions. Consideration should be given to ways to mitigate the deteriorating health situation in the non￾Transform: HDR sites by expanding collaboration with other external partners. USAID/Ethiopia Immediate 2 Promote/support an expanded and accelerated financial investment by the GOE to ensure equity and accelerated outreach to the hard-to-reach communities. USAID/Ethiopia Immediate 3 On the issue of Transform: HDR’s targets, since some targets have already been achieved, there is now a need to assess the feasibility of revising the targets upwards vs. expanding the interventions to new woredas for the remaining duration of the activity. USAID/Ethiopia Immediate 4 Despite the initial success in reaching some pastoral communities, site visits by evaluators revealed there is a need to identify sustainable approaches to scale-up MNCH/FP services for pastoral populations. The current strategies employed in pastoral regions are at an early stage, and there are multiple options with varied costs and benefits. USAID/Ethiopia Immediate 5 Consider building into future programming links to water and sanitation, female education programs, and potential income generation and livestock management for pastoralist areas (Afar and Somali). For the predominantly pastoralist regions and communities (Afar and Somali), any future MNCH programming should examine ways to link and/or factor health interventions that affect animals. USAID/Ethiopia Future Design/ Programming (Beyond the life of Transform: HDR) 6 Of particular concern, Transform: HDR is operating in pastoralist and semi-pastoral regions; the evidence shows that the performance in Afar and Somali is different from those of Benishangul Gumuz and Gambella. Therefore, USAID/Ethiopia should consider designing/supporting varying context-specific interventions tailored to different regional contexts (Afar and USAID/Ethiopia Future Design/ Programming (Beyond the life of Transform: HDR) Mid-term Performance Evaluation Report USAID Transform: HDR Activity 10 Recommendation/Action Actor to Address Timing Priority Somali) and Benishangul Gumuz and Gambella. This could also include a relatively higher investment for Afar and Somali compared to Benishangul Gumuz and Gambella to realize equity. Transform: HDR 7 Strengthen regional technical capacity by integrating MNCH/FP material from the activity into the existing pre-service curriculum for health science colleges. Transform: HDR Immediate/ Beyond the life of Transform: HDR 8 Give more technical assistance and support to the thematic areas such as child health to include the treatment and prevention of diarrheal disease where the team observed relatively weak￾performance. This should include the supply of drugs and equipment to treat diarrheal diseases, besides the technical assistance, given the widespread stock-out of ORS and Zinc. In order to achieve these targets, more expert technical assistance should also be devoted to rethinking how to improve the ANC-4 visit target, including the feasibility of spouses accompanying their partners to these visits, the obstacles to pregnant women using iron folate, and women’s decision-making related to FP, to be more effective in achieving targets. Transform: HDR Immediate 9 Enhance/adopt SBCC strategies that focus on influencing and changing the behaviors of the communities and contributing to the use of MCH services. The SBCC approach should consider the establishment/strengthening of locally acceptable high impact community awareness forums using local CBOs, religious or clan leaders to address some of the myths inhibiting health service utilization. Best practices from across Ethiopia should be considered. Transform: HDR Immediate 10 Consider adopting a more in-depth participatory planning process – Theory of Change (TOC) – to better capture regional and woredas needs, adjust strategies and approaches that work well in specific contexts, and refine planning assumptions. Although the current participatory planning process applied supports continued alignment of strategies, the TOC presents a more robust planning approach suited for complex contexts, like developing regions. Transform: HDR Immediate Mid-term Performance Evaluation Report USAID Transform: HDR Activity 11 Recommendation/Action Actor to Address Timing Priority 11 To narrow the gender gap, Transform HDR should ensure that all programming is designed to address the unique regional needs and mitigate barriers faced by women and men, boys, and girls to access service. Transform: HDR Immediate 12 Support the MOH/RHB and woredas to improve the use of performance information for decision-making and planning. Transform: HDR Immediate 13 Transform: HDR should work with USAID/Ethiopia to build/strengthen the Capacity for Public Health Emergency Management (PHEM) at the woreda level. The developing regions are prone to man-made and natural disasters, like floods, disease outbreaks, tribal conflicts, etc. Therefore, strengthening the PHEM capacity may be critical to rapidly respond to emergency situations while sustaining momentum on MNCH/FP programs. This, in the long run, will help build resilience at the community and woreda level, which is critical to achieving health and development outcomes. Transform: HDR/USAID Immediate 14 Realign the existing staff and, where feasible and within operating budgets, expand the number of technical staff to meet requirements. In the short-run, Transform: HDR may assign regions, districts, and specific geographies to their central staff so that they will be focused on their support to the regions and ensure that a good balance between central teams and field-level technical support is addressed in the short run. Transform: HDR Immediate FOR THE GOE/MOH & RHB 15 Support the scale-up of woreda-to-woreda and facility-to-facility (peer-to-peer) experience sharing to facilitate learning, improve service delivery, and enhance performance. GOE/FMOH, RHBs Immediate/ Beyond the life of Transform: HDR 16 Explore innovative ways to significantly strengthen coordination amongst key health and non-health development actors operating in developing regions for greater synergy, to avoid duplication, and to achieve better results. GOE/FMOH, RHBs Immediate/ Beyond the life of Transform: HDR 17 Considering the evaluation findings related to a possible deterioration in MNCH/FP performance in non-Transform sites, identify ways to leverage additional domestic resources and GOE/FMOH, RHBs Immediate/ Beyond the life of Transform: Mid-term Performance Evaluation Report USAID Transform: HDR Activity 12 Recommendation/Action Actor to Address Timing Priority other donor and private sector investments to accelerate the expansion and enhance the quality of MNCH/FP services in these areas. HDR 18 Assist the developing regions in addressing/managing the Human Resources for Health (HRH) staffing issues in a more structured and systematic manner that addresses attraction, recruitment, deployment, and retention. This may include recognition of high performing staff and health teams; recruitment and retention bonuses for those who agree to serve in the developing regions; and continuing professional development opportunities when they faithfully work in their placement for certain years; career growth. GOE/FMOH, RHBs Immediate/ Beyond the life of Transform: HDR 19 Strengthen the health infrastructure (electricity and water in particular) and supply chain that is affecting access and utilization of services, including transportation/ambulance services. GOE/FMOH, RHBs Immediate/ Beyond the life of Transform: HDR Mid-term Performance Evaluation Report USAID Transform: HDR Activity 13 Table A: USAID Transform: HDR Activity: Summary of Selected Performance Results Area at Mid-term (2019) Compared with Baseline (2017) in Transform: HDR Intervention and Non￾intervention Areas Result Area Indicator Transform: HDR Intervention Sites Transform: HDR Non￾Intervention Sites (overall) Baseline (%) Mid-term (%) Baseline (%) Mid-term (%) Family Planning MCPR among currently married women 18.9 21.8 26 23.5 LAFP methods among currently married women 3.6 6.1 6.8 4.5 Unmet Need for Family Planning 21.4 20.3 25.5 21.3 Use of modern contraception after birth (PPFP) 12.2 16.2 21.2 20 Maternal Health Women who took IFA supplement for >=90 days 23.3 15 24.1 12.1 Women with four or more ANC for their last birth 26 26.4 38.4 33.3 Skilled Birth Attendance 28.1 34.3 43.8 25.3 PNC for the mother within two days of birth 22.9 40.1 35.6 42 Women who received MNCH services through mobile health team 6.4 24.8 12.5 23.3 Newborn Health Early PNC for the newborn within two days of birth 23.4 37.8 37 38.7 Newborns who received Essential Newborn Care 8.6 8.2 7.8 7.8 Early initiation of breastfeeding 53.5 81.9 55.9 90.5 Child Health (<5) Full immunization 27.6 32.8 27.6 32.4 Measles 28.1 51.6 28.1 55.1 Penta 3 coverage 31.3 40 31.3 47.2 Children with Symptoms of ARI treated with antibiotics 16.8 74.4 19 71 Diarrhea treatment with ORS and Zinc 28.2 26.2 33.9 33.7 Children under 5 who slept under ITN 53.9 68.7 49 62.2 Children received Deworming in the last six months 27.4 33.7 24.1 38.8 SBCC Percentage of audience who recall hearing or seeing FP/RH message 48.1 37.8 52.3 33.2 Percentage of audience who recall hearing or seeing MNCH message 19.7 34.8 24.0 33.9 Cross￾cutting Women's participation in decisions regarding their health 82.3 79.3 80.1 78 Women accompanied by their spouse during ANC 44.4 40.8 52.3 46.7 Women accompanied by their spouse during birth at a health facility 78.2 79.5 72.9 79.7 Households enrolled in CBHI scheme 1.6 6.5 0.7 1.3 Mid-term Performance Evaluation Report USAID Transform: HDR Activity 14 SECTION 1: INTRODUCTION 1.1 BACKGROUND OF USAID TRANSFORM HEALTH IN DEVELOPING REGIONS (HDR) ACTIVITY 1.1.1 Activity Description Prime Partner Amref Health Africa Activity Start Date May 4, 2017 Activity End Date May 3, 2022 Life of activity budget $34,949,637 Name(s) of Subcontractors/Sub￾awardees Project HOPE, IntraHealth, and General Electric Transform: HDR seeks to significantly reduce morbidity and mortality among mothers and children under five years of age in the Developing Regional States (DRS) of Ethiopia. It is one of USAID/Ethiopia’s Health Office investments aimed at Preventing Child and Maternal Deaths. The Transform awards’ overall goal is to support the GOE’s attainment of its five-year Health Sector Transformation Plan (HSTP), primarily, the reduction of the Maternal Mortality Rate (MMR), Total Fertility Rate (TFR), Infant Mortality Rate (IMR), Neonatal Mortality Rate, and Under-Five Mortality. This activity implements interventions through a woman-, child- and girl-centered three-pronged strategic approach focusing on: 1. Increased access to integrated quality high impact MNCH/FP services; 2. Improved health-seeking behavior enhanced by reduced gender inequalities; and 3. Improved evidence-based decision-making and program learning, all with the overall goal of strengthening DRS health systems. The expected health results of Transform: HDR include: ▪ Increased numbers of healthy mothers with successful birth outcomes; ▪ Expanded access and uptake of family planning; ▪ Increased numbers of healthy newborns; and ▪ Sustained reductions in maternal and child mortality. Transform: HDR’s strategic pillars include: a) provision of comprehensive and integrated clinical interventions in MNCH/FP; b) implementation of context and culturally appropriate communication strategies for MNCH/FP service uptake; c) capacity development in the management and use of data for decision-making and action; d) partnerships for health systems strengthening; and e) gender equity programming. Mid-term Performance Evaluation Report USAID Transform: HDR Activity 15 Through these interventions, the activity supports the ambitious HSTP targets for MNCH/FP. Its aim is to contribute at least 50% of the HSTP MNCH impact and outcome indicators by the end of its five-year implementation period. Transform: HDR is being implemented in Afar, Somali, Benishangul Gumuz, and Gambella in 58 woredas and supports 15 hospitals, 158 health centers, and 847 health posts in these regions. 1.1.2 Transform: HDR Development Hypothesis IF demand and utilization of high-impact MNCH/FP services increases, THEN reducing preventable child and maternal deaths will be realized. To increase demand and utilization of high-impact MNCH/FP services in DRS regions, Transform: HDR, in collaboration with the FMOH and the RHB, planned to: ▪ Strengthen the provision of comprehensive and integrated clinical interventions in MNCH/FP; ▪ Implement context and culturally appropriate communication strategies for MNCH/FP service uptake; ▪ Strengthen capacity development in the management and use of data for decision￾making and action; ▪ Strengthen partnerships for health systems strengthening; and ▪ Promote and support gender equity programming. Mid-term Performance Evaluation Report USAID Transform: HDR Activity 16 CDCS GOAL: Ethiopia’s Transformation to a Prosperous and Resilient Country Accelerated Mission’s DO2: Increased Utilization of Quality Health Services Activity Strategic Objective: Increased utilization of quality high-impact MNCH/FP services in Afar, Benishangul-Gumuz, Gambella and Somali regions by 2021 Indicator 1: MCPR, Indicator 2: % of children received Penta 3 by 12 Mths, Indicator 3: % children fully immunized, Indicator 4: % births attended by SBAs, Indicator 5: % pregnant women receiving ANC 4+ IR1: Increased access to integrated, quality high-impact MNCH/FP services at health facility and community levels Indicators 1.1. Contraceptive acceptance rate; 1.2. Skilled Birth attendance rate; 1.3. ANC 4+, 1.4. Penta 3 covera 1.5. Full immunization coverage, 1.6. % PW treated for malaria, 1.7 # of children treated for malaria, 1.8. % of newborns with asphyxia resuscitated & survived, 1.9. % of sick young infants treated for sepsis, IR2: Strengthened health systems to provide quality MNCH/FP services Indicator 2.1: % of HFs who received technical support or supervision in the past 6 months by district health care managers IR3: Increased demand for high￾impact MNCH/FP services Indicator 3.1. Number of community members reached awareness raising activities IR4: Improved strategic information for evidence based decision-making and program learning Sub IR 1.1. Increased availability CEmONC services at selected primary hospitals and health centers Indicator 1.1.1. % of health facilities providing CEmONC services Sub IR 1.2. Increased availability of culturally appropriate and acceptable MNCH/FP service packages including BEmONC at health centers Indicators 1.2.1 % of health facilities providing BEmONC services 1.2.2 % of USG-assisted service delivery sites providing FP counseling & services Sub IR 2.1. Strengthened capacity of human resources for health focusing on MNCH/FP services provision Indicator 2.1.1. # of Health Care Providers trained on MNCH/FP program areas Indicator 2.1.1. # of people Sub IR 2.2. Leadership and coordination capacity of regional, district and facility level MNCH/FP structures strengthened Indicator 2.2.1. # of people trained with project support on LMG Sub IR 3.1. Increased community’s knowledge and awareness on MNCH/FP services Indicator 3.1.1. % of audience who recall hearing or seeing a specific USG-supported FP/RH message Indicator 3.1.2. % of audience who recall hearing or seeing a specific USG-supported MNCH-N Sub IR 3.2. Improved SBCC services to address gender norms and women empowerment Indicator 3.2.1. % of men who accompany their spouse or partner to at least one ANC visit Indicator 3.2.2. % of men who were at the health facility during birth Sub IR 4.1. Strengthened HMIS system to capture MNCH/FP data Indicator 4.1.1: % of health facilities (Primary Hospitals, HCs) submitting complete HMIS report on time for the recent reporting period Sub IR 4.2. Strengthened community health information system to capture MNCH/FP data Indicator 4.2.1. % of HPs submitting complete HMIS report on time Sub IR1.3: Improved community-based outreach MNCH/FP services through Mobile Health Team. Indicators 1.3.1. # of women served by a mobile health team. 1.3.2. # of newborns and children U5 served by a mobile health team Sub IR1.5: Strengthened Bi-direction referral system between health facility & community. Indicator 1.5.1. # of referral review meetings conducted Sub IR2.3: QA/QI approaches implemented to ensure high￾impact MNCH/FP interventions Indicator 2.3.1. % of health facilities that have annual quality improvement plans in place Sub IR2.4: Strengthened supply chain management capacity and resource mobilization Indicator 2.4.1. % of USG supported service delivery points experiencing stock out of tracer drugs Sub IR1.4: Increased deployment of women MNCH/FP service providers Indicator 1.4.1. % of female midwives deployed in the activity implementation woredas Sub IR3.3: Women and girls empowered to obtain MNCH/FP services Indicator 3.3.1. % of women who participate in decisions regarding their own healthcare Sub IR4.3: Improved operational studies and evaluations to track activity performance and inform decision-making Indicator 4.3.1. Number of operational research conducted for MNCH/FP for health system improvements Sub IR4.4: DRS focused MNCH/FP knowledge management and information sharing platform established Indicator 4.4.1. Number of districts documenting and sharing best practices and lessons learned Transform: HDR Results Framework Mid-term Performance Evaluation Report USAID Transform: HDR Activity 17 1.2 PURPOSE AND USE OF THE MID-TERM PERFORMANCE EVALUATION OF USAID TRANSFORM HEALTH IN DEVELOPING REGIONS (HDR) ACTIVITY The purpose of the mid-term performance evaluation is to assess the continued relevance and validity of the design and planning assumptions made in the original activity design for the remaining two years, identify any potential midcourse corrections to improve program effectiveness, and document progress to date and challenges. Evaluation results and findings are expected to inform USAID/Ethiopia, MOH, and Transform: HDR’s long-term planning, decision￾making, and specific direction for future programming in the developing regions of Ethiopia. The evaluation will also provide an opportunity to share lessons learned with USAID/Ethiopia, MOH, and other key Transform stakeholders. The primary intended audience for the mid-term performance evaluation report is USAID/Ethiopia and the Government of Ethiopia who will use the findings and recommendations of the evaluation to review USAID/Ethiopia’s and the GOE’s collaborative investment in the health sector for four developing regions in Ethiopia (Afar, Benishangul Gumuz, Gambella, and Somali). 1.3 EVALUATION QUESTIONS The mid-term performance evaluation report will assist USAID/Ethiopia, MOH, and Transform: HDR in answering the following key evaluation questions: 1. Have the USAID Transform: HDR strategies and implementation mechanisms continued to be relevant and appropriate to the targeted regions? 2. How efficient are Transform: HDR’s approaches and strategies for achieving intended results? 3. What has been the effectiveness (“likelihood of achieving results”) of Transform: HDR’s approaches in contributing to improving MNCH/FP outcomes? 4. How has Transform: HDR facilitated local ownership and sustainability? 5. How has Transform: HDR considered gender dynamics in the activity design and implementation? 6. What are the lessons learned and recommendations for USAID/Ethiopia related to Transform: HDR? Mid-term Performance Evaluation Report USAID Transform: HDR Activity 18 SECTION 2: EVALUATION DESIGN & METHODOLOGY15 2.1 EVALUATION DESIGN The mid-term performance evaluation of Transform: HDR utilized a mixed-method approach, including both extensive quantitative and qualitative methods. Data was also gathered and synthesized from primary and secondary sources such as activity background and resource documents relevant to Transform: HDR. The evaluation team triangulated data from these sources to generate evidence to address the evaluation questions. Activity effectiveness was considered by comparing data between 2017 and the end of 2019, where 2017 is the base period (X1) compared with the mid-level period, 2019 (X2). Both quantitative and qualitative data collection instruments were used, such as structured and semi-structured questionnaires (Annex 2). The methodological approaches utilized in this evaluation support “contribution” measurement and NOT “attribution” of high-level results to Transform: HDR’s activity. The difference-in-difference analysis was used to account for changes associated with prior activities, comparing USAID/Ethiopia Transform: HDR intervention areas and non-Transform: HDR areas. A series of meetings were held with USAID/Ethiopia and Transform: HDR to agree on the evaluation design, methodology, evaluation questions, and data collection tools. This resulted in a consensus among key stakeholders about evaluation tools and the evaluation timeline. 2.2 SAMPLING A. Household-level Survey Sample The household survey followed a two-stage sampling technique. The mid-term evaluation design provided a representative estimate for Transform: HDR intervention regions: Afar, Benishangul Gumuz, Gambella, and Somali. Since neither the census frame nor a master sample is available at this time because of the upcoming census, an alternative frame was considered, which is a lower administrative unit (kebele) and from which villages (gotts) will be selected randomly. First Stage–selecting sample kebeles/health posts from the four regions. Kebeles were selected using a simple random sampling technique and assumed 30 households (HHs) per kebele as an optimal representative sample size. Second Stage–a selection of households from the selected kebeles. First, gotts were selected randomly. A fresh list of households was prepared for each selected gott that serves as a frame 15 Detailed Evaluation Methodology and Sample Size Calculation can be found in Annex 5 Mid-term Performance Evaluation Report USAID Transform: HDR Activity 19 from which the households with eligible respondents were systematically selected. Interviews were conducted with the assistance of kebele guides, and with clearly identified boundaries of the study sites. The eligibility criteria for selecting the households were women aged 15-49 regardless of marital status. The study populations for the mid-term performance evaluation are the main beneficiaries/recipients of the activity: ▪ women of reproductive age 15-49; ▪ children under age 5; ▪ primary health care facilities; and ▪ FMOH, RHB, and woreda health sector management team was overseeing the health service delivery. The sample size of the household survey was determined considering the precision and capability of measuring changes on key outcome indicators of Transform: HDR. Key performance indicators (KPIs) were considered in the sample size calculation, where the indicator that gave the largest sample size was used to determine the sample size. Thus, the sampling requirements of all other indicators were satisfied. The sample size was determined using statistical software considering the detecting power of measuring changes over time on the key outcome indicators. The formula used for estimating the sample size is described below. n = (Zα/2+Zβ) 2 * (p1(1-p1) + p2(1-p2)) * d * nr (p1-p2) 2 Where: α: The probability of type I error (significance level) is the probability of rejecting the true null hypothesis. β: The probability of type II error (1 – the power of the test) is the probability of not rejecting the false null hypothesis. d: is the design effect nr: is non-response rate P1 and P2 are the proportions of the outcome indicators of interest at baseline and end-line. The evaluation team also considered the following key indicators in calculating the sample size for the household survey: ▪ Skilled Birth Attendance (SBA); ▪ Modern Contraceptive Prevalence Rate (MCPR); and ▪ Full Immunization Coverage. Mid-term Performance Evaluation Report USAID Transform: HDR Activity 20 All three indicators gave different sample sizes, and the indicator that gave the largest sample size was considered for sample size determination, which was MCPR (Annex 1). This evaluation utilized two independent population proportion sample size estimation techniques to calculate the sample size for the household survey. The team used STATA software: power and sample size calculation utility. In the calculation, we considered a 95% confidence level (α), and 80% statistical power (β), design effect of 1.5 and 2% non-response rate, based on experience conducting the Transform Program Baseline Survey (2017). The estimated proportion of the indicator (P1) for each region is taken from the Baseline Survey report. A total of 2,635 households with women aged 15-49 (2,102 from Transform: HDR intervention area, and 533 from non- Transform: HDR areas) were randomly selected and interviewed (Table 1). Table 1: Number of Households Surveyed16 Region Transform: HDR Sites Non- Transform: HDR Sites Total # of HH # of Surveyed women 15-49 # of <5 children # of women 15-49 # of <5 children Afar 480 594 114 173 594 B/ Gumuz 600 500 150 185 750 Gambella 482 331 119 115 601 Somali 540 677 150 237 690 Total 2102 2102 533 710 2635 B. Health Facility Sampling (HFS) The purpose of the health facility assessment was to assess the linkage and effectiveness of the primary health care delivery system. For this evaluation, the primary health care unit was defined as a health post, a health center, and a primary hospital. A sampling of health facilities is linked to the sample for the household survey. The designated health post that serves the selected kebele (for the household survey) was automatically selected for the HF assessment. Health centers are selected based on the linkages they have with the selected health posts. In the same way, primary hospitals (if any) that are linked to the health centers are selected. In some cases, there may not be primary hospitals available to serve the population. Therefore, the total number of primary hospitals depends on the availability of primary hospitals in the selected health center areas. 16 Sample size in B/G is larger than in Afar and Somali because P1 is much less is Afar and Somali compared to B/G (5 & 6 versus 42). The higher Proportion the higher sample size required to detect change. Mid-term Performance Evaluation Report USAID Transform: HDR Activity 21 Accordingly, a total of 97 health facilities from Transform: HDR intervention sites and 30 health facilities from non-Transform HDR sites were assessed using a standardized questionnaire (Table 2). The triangulation of facility assessment and household level data was analyzed together with population data to assess the effectiveness of USAID Transform: HDR Activity performance, including target outcomes. Table 2: # of Health Facilities Assessed from Transform: HDR Intervention and Non-intervention Areas Region Intervention Non-intervention Primary Hospital Health Center Health Post Total Primary Hospital Health Center Health Post Total Afar 1 12 13 26 0 4 3 7 Benishangul Gumuz 2 17 19 38 0 3 5 8 Gambella 1 6 5 12 1 2 6 9 Somali 1 16 4 21 0 5 1 6 Total 4 51 41 97 1 14 15 30 *Note: some health posts were closed/not functional during the survey especially in Gambella, Afar, and Somali regions C. Key Informant Interviews (KIIs) The evaluation team conducted one-on￾one interviews with a variety of stakeholders involved with and/or familiar with Transform: HDR’s implementation, including MOH, Regional Health Bureaus, Transform HDR staff (central office and regional team), and woreda health offices (USAID Transform: HDR intervention and non-USAID Transform: HDR woredas). A total of 23 KIIs were conducted, and all interviews were audio-taped, translated, and transcribed. D. Focus Group Discussions (FGDs) Seven Focus Group Discussions (FGDs), led by qualitative evaluation experts fluent in the local language, were comprised of 6-12 participants residing in the catchment areas of the program. The FGDs were done to capture community knowledge, attitudes, and practices (KAPs) related Summary of the Key Informants Interviewed • Federal Ministry of Health: Health Special Support Directorate (1 KII) • Transform: HDR Activity management and technical at the central level (2 KIIs) • Transform: HDR Activity regional team (4 KIIS) • Regional Health Bureaus (Afar, B/Gumuz, Gambella and Somali regions) ( 4 KIIs) Mid-term Performance Evaluation Report USAID Transform: HDR Activity 22 to the quality and accessibility of MNCH/FP services and client satisfaction. The FGDs were comprised of: ▪ Women aged 15-24 ▪ Women aged 25-49 ▪ Male aged 15-24 ▪ Adult and sexually active men aged 25-50 The FGDs’ responses were audio-taped, translated, transcribed, coded, and analyzed by thematic areas. 2.3 SURVEY INSTRUMENTS For the household surveys and Health Facility Assessment, extensive structured and semi￾structured questionnaires were developed. For comparability, the survey instruments included standard questions from well-known sources such as the Demographic and Health Survey (DHS), the MOH health assessment instruments, and other health-related nationally administered questionnaires. The tools and instruments greatly benefit from consultations with all key stakeholders (Annex 2 and Annex 3). The individual household questionnaire template was prepared in English and later translated into local languages, Amharic and Somali. Transform: MELA back-translated instruments to ensure accuracy. The final questionnaires were then uploaded to smart mobile phones/tablets using the Open Data Kit (ODK) software, and interviews were conducted using a tablet-based approach to ensure seamless data collection. For the Public Health Facility (HF) Assessment, the data collection tools were developed so that service providers and administrators could explain the status of service provision at health facilities in the study area. Structured questionnaires were developed and administered to HF service providers to gather information on their opinion regarding their assessment of the effectiveness of health service delivery in the study sites, any improvements in service delivery due to the assistance received from Transform: HDR, etc. 2.4 DATA COLLECTION A three-day training was held for the ten supervisors and 40 data collectors, four qualitative experts, and one senior evaluation specialist, in Addis Ababa and at the regional level. The enumerator and data collection training covered topics such as research ethics in the field, rights of human subjects during the survey, survey methodology and protocol, sampling procedures, informed consent, qualitative and quantitative data collection tools, interviewing techniques, data management, security and quality, and gender considerations during data collection. The survey tools were pre-tested during this training before launching data collection, and survey Mid-term Performance Evaluation Report USAID Transform: HDR Activity 23 instruments were revised accordingly. Data collection took place between November 11 - December 30, 2019. 2.5 DATA ANALYSIS All household and facility data were cleaned, checked, and validated to immediately identify and address issues during data collection, a process that took place directly with supervisors and data collectors. Distributions and bivariate analyses were conducted to describe the characteristics of a select set of Transform: HDR intervention indicators. Pearson’s chi-square tests were performed to assess the statistical significance of differences between the USAID Transform: HDR intervention and non-Transform: HDR sites at the midpoint for the same set of indicators. Logistic regression was used to conduct more detailed assessments regarding the predictors of key result areas, including modern family planning utilization, antenatal care utilization, delivery by a skilled birth attendant, and child vaccination. The analysis of household data focused on: ▪ Frequency and percentage calculations, cross-tabulations, and multivariate regression analysis of key performance indicators. ▪ Comparison of mid-term vs. baseline key performance indicator values. ▪ Difference-in-Differences (USAID Transform: HDR intervention sites and non-USAID Transform sites), including analysis of actors operating in non-USAID Transform sites. Household and health facility data analysis were conducted using SPSS and STATA software, and qualitative data of key Informant and FGDs were analyzed using Atlas ti software. Data analysis to identify determinants of outcome indicators: A binary logistic regression was fitted to identify the determinant factors (adjusted effect) for FP uptake and MCH service utilization based on the mid-term evaluation survey. Women’s region of residence, age, educational level, paid work status, religion, a number of children ever born, the ideal number of children a woman wants to have, exposure to FP/RH messages, exposure to MCH messages, spouses accompany women during ANC, spouses accompany women during delivery, access to basic sanitation, access to improved water, use of water treatment, availability of handwashing station with water and soap, women’s participation in their own health care decisions, and membership of community-based health insurance are included as covariates. First, a bivariate analysis was conducted between the outcome variables and the covariates. Then those covariates found to be significant at 10% are included in the multivariate logistic regression analysis. Collinearity between the covariates was checked using a variance inflation factor (VIF) and found to be non-substantial; the goodness-of-fit of the models was checked using the Hosmer and Lemeshow test and found to be a good fit. Finally, the Odds Ratios (OR) and their Mid-term Performance Evaluation Report USAID Transform: HDR Activity 24 95% confidence intervals (CI) are reported for each of the variables in the final model. The statistical significance of the variables is decided based on the non-inclusion of zero in the 95% CI. The Difference-In-Differences (DID) Analysis was conducted to quantify the causal effects of Transform: HDR’s interventions on outcomes of interest. The Transform Baseline Survey (2017) and Transform Mid-term Evaluation (2019) provided data for intervention and non-intervention woredas and were used to evaluate the contribution of the Transform Program interventions in improving family planning methods uptake, and maternal, newborn and child health service utilization. The hypothetical DID below is used to show the estimation of the effects/contribution of an intervention by comparing the changes in outcomes over time between a population that is enrolled in an intervention “Transform: HDR” (the intervention group) and a population that is not “non-Transform: HDR sites,” also called the comparison group. This means the DID estimated the contribution of Transform: HDR by calculating the difference between the change in the key indicator values in Transform: HDR intervention areas (between the baseline and mid-term), and the change in non-Transform: HDR intervention areas (between the baseline and mid-term). The diagram below thus illustrates how the DID has been used in this estimation. Outcome Indicators The main outcome indicators considered in this analysis are Family Planning Indicators [contraceptive prevalence rate (CPR), long-acting (LA) family planning methods use, postpartum family planning use and unmet need for FP]; Maternal Health Indicators [antenatal care (ANC), skilled birth attendance (SBA), postnatal care (PNC), and iron-folic acid supplementation (IFS)], and Cross-cutting Issues (basic sanitation, spouse accompany to ANC, spouse accompany to delivery), and child health [measles vaccine (MCV), exclusive breastfeeding]. Mid-term Performance Evaluation Report USAID Transform: HDR Activity 25 2.6 ETHICAL PROCEDURES All the survey team (evaluation specialists, qualitative experts, household data collectors, and supervisors) were trained in ethical research procedures, including informed consent, the privacy of participants, and confidentiality. The survey followed standard Ethiopian ethical procedures. Permission was obtained from the Head of Household and consent granted by the participant before the interview commenced. For women under the age of 18, additional parental permission and participant consent was obtained before data collection. 2.7 LIMITATIONS OF THE EVALUATION ▪ The absence of the Enumeration Area (EA) list from the Ethiopian Central Statistical Agency (CSA), which is in charge of the national census and establishment of EAs, necessitated the use of the list of kebeles for each intervention area as a sampling frame for the first stage of sampling. Transform: MELA utilized kebele guides to clarify and identify boundaries of the selected kebeles and to ensure the complete listing of households in selected gotts. Sampling with replacement was employed, especially where security concerns and inaccessibility impeded data collection in some enumeration areas. ▪ Another limitation was isolating confounding factors while using Difference-in-Difference analysis in the context of development programs taking place in both USAID/Ethiopia Transform: HDR Activity areas and the surrounding areas outside the boundaries of the evaluated regions. During the survey, partner mapping information was gathered from both Transform: HDR intervention sites and non-USAID-Transform sites. The evaluation team found that a myriad of other partner-funded programs exists in both Transform: HDR intervention sites and non-USAID-Transform sites. Thus, it is difficult to definitively discern the results and impact of other woreda-level, MOH, or donor financed programs on achieving MNCH/FP targets. The evaluation team relied on the DID and causality tracking strategies analysis to overcome these challenges. Hence, the DID results should be interpreted, bearing in mind these considerations. ▪ An additional limitation was the evaluation team’s difficulty in obtaining other development actors' health-related investment data on MNCH/FP programs operating in non-USAID Transform: HDR sites, which hampered the team’s ability to carry out a comparative assessment of efficiency based on the investments. ▪ This being a population survey, recall bias on self-reporting due to time lag (like in the last one year, two years, etc.) could have had an influence on the extent to which respondents remembered what they received and their practices. Mid-term Performance Evaluation Report USAID Transform: HDR Activity 26 ▪ Given that this evaluation was designed as a mid-term performance evaluation, it assessed progress towards as opposed to measuring the full achievement of intended results in addition to establishing what is working and possible factors. It also measured the contributions made by Transform: HDR to development results under the effectiveness question. It is important to take note of this when interpreting results from this evaluation. Mid-term Performance Evaluation Report USAID Transform: HDR Activity 27 SECTION 3: KEY FINDINGS Introduction This section presents the qualitative and quantitative results and assesses whether the Transform: HDR mechanism has continued relevance and is appropriate to the context and needs of the targeted regions. This section also analyzes the efficiency of Transform: HDR’s strategies and approaches and “likelihood of achieving results,” and effectiveness in contributing to improving MNCH/FP outcomes at mid-course. This section also presents the findings related to local ownership and sustainability and documents the activity’s work to address gender dynamics. 3.1 RELEVANCE The relevance of Transform: HDR was evaluated based on the following domains: a) the extent to which there are alignment and complementarity with national priorities; b) the extent to which the stakeholders have a clear understanding and ownership of the activity; c) the extent to which the strategies, approaches, and interventions are culturally appropriate to the regional context; and d) the extent to which the Transform: HDR implementation activities are strategically designed and positioned to achieve its objectives and reinforce the capacity of the MOH and regional and woreda health systems to deliver quality health services. A) Alignment/Contribution to National Priorities & Clear Understanding and Ownership by Stakeholders of the Transform: HDR Activity: The evidence from the evaluation revealed that Transform: HDR is relevant and aligned to USAID/Ethiopia and the Government of Ethiopia (GOE) RMNCH Strategic Priorities17 . Transform: HDR’s strategy and approaches are derived from the GOE/MOH’s national priorities and are thus relevant and aligned to the FMOH’s HSTP18 and the needs of the country. The design of the Transform: HDR Activity started with a highly participatory consultation process based on data generated from the national and regional level that allowed for the GOE to align the design with its priorities, and hence ownership of Transform: HDR. The evaluation team found that the design of Transform: HDR set a good course for the activity as it started with a consultation with national and regional stakeholders. A second factor that has reinforced ownership is the participatory annual joint planning exercise where RHBs and other sector offices jointly plan work assignments and adjust interventions and their roll-out according 17 A review of documents such as Activity description, Annual work plans 2017, 2018 & 2019; and quarterly and annual performance reports, HSTP, all demonstrated alignment to national priorities. 18 Document review of HSTP document. Mid-term Performance Evaluation Report USAID Transform: HDR Activity 28 to regional health sector priorities. The annual planning exercise between Transform: HDR and regions and woredas ensures the continued relevance of the activity’s interventions, strategies, and approaches within regional contexts. B) Stakeholder Clear Understanding and Ownership of the Stakeholders’ Relationship to the Transform: HDR Activity: A key factor that has led to local ownership is the institutional capacity support and continuous technical assistance (TA). Many stakeholders interviewed commented that the IP team was responsive to the region’s requirements and was able to respond at important moments, such as during periods of emergencies. Another positive finding was that the majority of programming and implementation approaches and tools introduced by the IP team were formally adopted by the MOH program. The evaluators observed that the RHBs were using and adapting the technical assistance (TA) team’s tools. For example, outreach mobile health services have proved vital in delivering health services to harder to reach communities. Regional health bureaus expressed appreciation for the capacity building training for health workers, strong post-training follow-up, coaching, and mentoring. The funding for medical equipment and supplies was also highly appreciated and was viewed as strengthening the region’s health service delivery system. Beyond health workers' training support, the four19 RHBs also commented on the valued support provided for medical equipment and supplies to strengthen the health facilities and properly equip the mobile outreach teams. • Stakeholders positively perceived the IP team’s TA and engagement as being highly supportive of the regions and woreda’s MNCH/FP objectives and work plans. • Stakeholders confirmed that Transform: HDR interventions are being implemented in line with the strategies and priorities of the regional health bureaus and local contexts. • Stakeholders positively noted their engagement or complementarity and acknowledged the role and importance of Transform: HDR in supporting regions in areas of MNCH/FP. 19 Afar, Benishangul Gumuz, Gambella, and Somali Mid-term Performance Evaluation Report USAID Transform: HDR Activity 29 C) Strategies, Approaches, and Interventions are Culturally Appropriate for the Regional Context The evaluation team observed that Transform: HDR continues to be relevant in supporting government priorities, and the mechanism’s flexible response to emergencies is adapted to the cultural context. For example, during a measles outbreak, the team set up a mobile vaccination campaign to reach migrating pastoralists. Regional and woreda stakeholders interviewed perceive that Transform: HDR interventions are being implemented in line with the strategies and priorities of the regional health bureaus, and local contexts. D) Transform: HDR implementation Activities are Strategically Designed and Positioned to Achieve Its Objectives and Reinforce the Capacity of the MOH and Regional and Woreda Health Systems to Deliver Quality Health Services. A major finding is that two of the most relevant and responsive approaches embraced/adapted by Transform: HDR has been the supportive supervision and the improvement of outreach and mobile health services. There was consensus among the four20 RHBs supported by the activity, including woredas sampled, that these approaches (that are going beyond the fixed health facility) were well-adapted and responsive to regional needs during the measles outbreak campaigns and pre-service training. Another point made by the RHBs was that the strong on-site post-training follow-up coaching and mentoring of health workers at fixed facilities was also highly regarded. 20 Afar, Benishangul Gumuz, Gambella, and Somali “Transform: HDR is our project and is relevant in its support to outreach and mobile health services – we encourage this activity to be continued and scaled up. The supply of materials for EPI, maternal health activities are also essential for the woreda health offices. Transform: HDR also adjusted its activities based on the situation of the woreda like participating in outbreak management; a fund should be set-up to take care of outbreaks.” -- KII in Afar Mille Woreda. Mid-term Performance Evaluation Report USAID Transform: HDR Activity 30 It appears, however, that there is some misunderstanding in some regional offices and woredas about the expected transformative outcomes the activity can support. This may be in part due to the use of the term “Transform” as the title of the activity. Some regional stakeholders expected and were disappointed that the activity did not fully transform the woredas’ health system. Evaluators reported that comments were made expressing some dissatisfaction with the activity because it did not live up to its ambitious expectations for rapid and accelerated “transformation” of the entire health system. There is a need for Transform: HDR to reinforce during each visit to stakeholders, the objectives of the activity, the results expected, and clearly articulate the breadth and limitations of the scope of the activity. Some RHB expects Transform: HDR to be responsible for the entire woreda’s to health system transformation while Transform: HDR is only supported specific areas of the transformation agenda. This has resulted in some dissatisfaction with the scope of the activity and not enough recognition of the impact of results achieved thus far. The IP team needs to translate in lay terms the importance of the results of the activity and possibly advise the RHBs on how they can use and adapt the tools and approaches to achieve other woreda health objectives. A respondent from B/Gumuz RHB summed up his views and what other KIIs21 also expressed: In Summary: ▪ There is a strong alignment between Transform: HDR strategies approaches and results with the FMOH HSTP and USAID/Ethiopia’s strategic priorities for MNCH/FP. This was created as a result of consultative meetings during the design of the activity, and enhancement through joint planning meetings with regions does exist, which contributes to the realization of the HSTP priorities. ▪ Approaches embraced/adapted by Transform: HDR, such as supportive supervision, outreach, 21 KIIs from woredas and RHBs shared this view “As the name indicates “Transform: HDR” aimed to transform health in developing regions and addresses equity issues by supporting all transformation agendas. However, Transform: HDR intervention packages are very specific and focus only on RMNCH services. The support for Transform: HDR is very limited in accelerating woreda transformation in our regional context. Transform: HDR intervention strategies are more rigid on the scope of the project and unable to respond to the changing context of the regions.” “Transform: HDR Activity gives the impression that it will help the regional government transform the health sector. But its support vs. RHB expectation with project performance at the operational level is not adequate. The support is spread too thin to yield meaningful transformational results. In addition, the thinly dispersed support can’t help us draw useful lessons.” – A Key Informant at B/Gumuz RHB Mid-term Performance Evaluation Report USAID Transform: HDR Activity 31 and mobile health services, are responsive and culturally sensitive and adapted to the needs of the communities. ▪ The supply of medical equipment and supplies to strengthen the health facilities and properly outfit the mobile outreach teams were highly appreciated by FMOH, regional, and woreda stakeholders. ▪ Joint planning exercises with regional and woreda stakeholders have ensured Transform: HDR interventions continue to be relevant and contextually and culturally appropriate to the activity’s contexts under which it is being implemented. ▪ Transform: HDR is appreciated and owned by communities and key stakeholders. “Transform HDR is our project and is relevant in its support to outreach and mobile health services – we encourage this activity to be continued and scaled up22.” Moving Forward, Transform: HDR may consider: ▪ Revisiting its communication strategies to key stakeholders to ensure that there is regular communication to remind stakeholders of the scope of the Transform: HDR Activity and the importance of the work that is being achieved, and ways that tools already developed under Transform: HDR can be adapted to advance other woreda health priorities. 3.2 EFFICIENCY Transform: HDR’s efficiency was assessed based on its management structure, administrative and logistics arrangements, networking, and partnership strategy, and efficiency in resource allocation and expenditure (including its ability to respond to changing contexts and resource allocation). Based on this review, the evaluation team determined the activity has efficient systems that advance the attainment of intended results, as discussed below. Strategies & Approaches: The initial design called for 58 woredas to be covered. This proved to be unattainable with the current budget and staffing configuration and the overwhelming unmet need and high expectations from the regions. This led to a joint decision by USAID/Ethiopia and the MOH/RHBs to revise the number of woredas to receive a full-service package of health systems strengthening and governance interventions. The number of full-service woredas is now 20, with the remaining 38 woredas receiving some elements of this package. These 20 full-service 22 KII in Afar Mille Woreda summed up other KIIs views. Mid-term Performance Evaluation Report USAID Transform: HDR Activity 32 woredas constitute Centers of Excellence23 (CoE), a term used to refer to the selected health facilities in the 20 learning woredas in 20 zones, of the 58 by Transform: HDR operational woredas, is intended to invest more and create health facilities that provide quality comprehensive MNCH/FP services and used as a point for catchment mentorship to surrounding health centers and health posts. In hindsight, it is clear this was a good decision as the original plan was overly ambitious, given the logistics and security obstacles. Despite this change in plan, evaluators noted some excellent examples of services and technologies at both the full service and modified service plan woredas, such as the well-outfitted mobile outreach teams and the supply of medical equipment, including the ultrasound, that served to attract more clients to the health facilities. This, in turn, was observed to have improved the utilization of health services. Staffing: Transform: HDR operates with a relatively small technical team for such a large geographic catchment area. The staffing pattern and staff alignment are presented in Table 4 below. Table 4: Transform: HDR Activity Staffing LOCATION TECHNICAL STAFF SUPPORT STAFF TOTAL CENTRAL OFFICE 18 10 28 CLUSTER 9 15 24 REGIONAL 27 35 62 TOTAL 54 60 114 KIIs noted that the geographic spread between the regional center and intervention woredas remains a challenge. This is especially difficult given the small technical team spread across so many specialty areas coupled with the difficulties of identifying and recruiting qualified staff to work at the regional and sub-regional level. It is also complicated by weak health systems. For example, the project team has to travel more than 400 km from the Benishangul Gumuz regional center to Wombera Woreda. While this provides a good regional context, it is important to appreciate the limited human resources and competencies within local 23 Centre of Excellence (CoE) is a term used to refer to selected health facilities in the 20 learning woredas in 20 zones, of the 58 by Transform: HDR operational woredas, is intended to invest more resources and create health facilities that provide quality comprehensive MNCH/FP services and used as a point for catchment mentorship to surrounding health canters and health posts. These canters, (20 health canters and 4 primary hospitals) received medical equipment – for new-born resuscitation to portable V-scan and high definition ultrasounds to improve diagnostic and treatment capacity. To improve leadership and governance as well as service quality, these canters have been included in different Quality Improvement and LMG trainings and projects. “Transform: HDR Activity is efficient in human resource utilization, but its cluster coordinators are not able to reach all woredas on time. They need to invest in recruiting adequate qualified personnel to be able to cover their catchment areas.” – Key Informant at Afar RHB Mid-term Performance Evaluation Report USAID Transform: HDR Activity 33 government health structures. Consequently, regional and woreda level partners’ expectations for Transform: HDR’s technical support are vivid. The thin staff and their deployment in clusters may need to be revisited for the remaining years of the activity. The evaluation team learned that the achievement of meaningful results might necessitate detailing more experts at the woreda level as opposed to the regions. This will require adjustments to be done to have more technical staff at the cluster level closer to the point of interventions than at the regional level. This will allow Transform: HDR to get grounded at the point of intervention and to provide timely support/mentorship to the local health system. The AOR for Transform: HDR acknowledged difficulties the activity team has faced in identifying and recruiting qualified staff who are ready to work in the developing regions. During the KII, he had this to say: “Transform: HDR has had challenges identifying and recruiting qualified staff for regional and sub￾regional levels. This, coupled with the pressure to try to ensure qualified staff from diverse clans and ethnic groups. Further, USAID is trying to prioritize gender balance, and it is even more immensely challenging to recruit women to work in developing regions.” KII feedback from Somali and Afar regional and woreda health sector officials noted that the Transform: HDR targeted regions are facing high and persistent staff shortages. Health facility staff are limited due to resistance by qualified health personnel to be assigned and to serve in developing regions. There is high staff turnover aggravated by harsh weather conditions and limited staff incentives to work in the targeted regions. A KII from Afar summed up the views echoed by his counterparts from the Somali region and other woredas KIIs: “Tailored capacity building support on Leadership, Management, and Governance (LMG) development and execution of tailored Leadership, Management and Governance (LMG) trainings suitable for officers/leaders at all leadership levels in developing regions health structure will definitely contribute to improved planning, executing, and evaluating health priorities of the regions with improved accountability. This will also help strengthen the PHCU that are expected to contribute the UHC in the developing regions Woredas”. The evaluation team observed that there is a pressing need for comprehensive human resources for health (HRH) strategies in the developing regions to aggressively advance the recruitment, deployment, utilization, professional development of existing staff, and retention. This may require government incentives to serve in these regions and ample recognition of high performing staff. More attention should be given by the RHBs to continued professional development opportunities for the health providers and public health teams. 24 24 FMOH Special Support Directorate staff Mid-term Performance Evaluation Report USAID Transform: HDR Activity 34 Transform: HDR is helping to address the staffing gap by building a pool of master trainers specialized in MNCH/FP. Transform: HDR has supported a Training of Trainers (TOT) pool at the regional level by closely working with the RHBs’ Regional Universities, Health Science Colleges, and Reference Labs. Going forward, more can be done by Transform: HDR to specifically equip the master trainers to address the full spectrum of high impact MNCH/FP interventions. The master trainers have an important role to play in improving the quality and quantity of the health workforce by incorporating in-service training on topics associated with how to treat: a) complications during delivery, and b) improved case management of common illnesses for children in addition to BEmONC, ICMNCI. The evaluation team has suggested these topics be included in the pre-service curriculum and that this might be a cost-effective sustainability measure. 25 Transform: HDR capacity building is appreciated and regarded as efficient by regions and woreda stakeholders. “Among the Transform: HDR interventions capacity building is perceived to be highly efficient. The project provided technical support to the woreda and health professionals through capacity building. Transform: HDR train only cluster supervisors; each cluster supervisor, in turn, trains all the health extension workers under their supervision. This approach enabled the project to train many professionals with a limited budget and eventually strengthen the system. For instance, in our woreda, there are five clusters and about 62 health extension workers under each cluster. When Transform: HDR organized ICCM and IMNCI training for health extension workers in Assosa, only 5 cluster supervisors got this training. Those supervisors also provide onsite training to all health extension workers under their cluster without incurring any additional Transform: HDR Activity costs. We believe the Transform: HDR training model is both efficient in terms of system strengthening and is cost-effective (Benishangul Gumuz, Woreda Health Office staff).” However, most regional respondents felt that Transform: HDR should be selective in giving training and focus more on on-site training, and had this to say: “Transform: HDR and other partners are providing too many training courses. Some RHBs have decided to reduce the amount of training in their region, except for new initiative familiarization and the introduction of new advances and those that we deem high an impact. The majority of our problems are lack of inputs (supplies and equipment) at the facility level, so instead of providing training frequently, it is good to shift the budget to address those shortages and equip the facilities well and focus on cheaper training options like on-job/on-site training and mentorship. In order to transform the woreda, the project should focus on supplying more medical equipment” -- (Benishangul, regional health bureau) Another factor that was viewed as inefficient at the initial stage of implementation of Transform: HDR was the lack of a crisis response fund. The four developing regions are widely known to be “emergency-prone” due to natural disasters and other political and security issues. Yet, the activity design did not originally include ways to address and adapt to these challenges. The 25 KII with Transform: HDR COP and Technical Director. Mid-term Performance Evaluation Report USAID Transform: HDR Activity 35 recent approval by USAID/Ethiopia for the allocation of a “Crisis Modifier Fund” was well received by the regions. Building the capacity of developing regions to rapidly address public health emergency management related to women and children should be one of the interventions at the top of Transform: HDR’s priority list of things to do to build capacity. The developing regions are prone to man-made and natural calamities, like floods, disease outbreaks, tribal conflicts, etc. Hence, strengthening the public health emergency management capacity is critical to respond to emergency situations and maintain the momentum on MNCH/FP services. In the long run, this will help build resilience at both the community and woreda levels, which are critical to achieving health and development outcomes26 . Transform: HDR efficiency has also been enhanced by mobility and settlement pattern studies, which led to the design of a “Comprehensive Mobile Health Outreach Services” strategy. The pilot Comprehensive Mobile Health Outreach Service appears promising and is presently being documented and discussed for future scale-up. The introduction of new technologies to facilitate service delivery, such as the introduction of mobile ultrasound, the simplified gestation estimation tool, and task shifting for middle-level health workers, were positively received by stakeholders. These innovations are motivating women to attend ANC services despite the costly long-distance travel to hospitals and improving the effective utilization and interpretation of Essential ANC lab results and the introduction of technology to improve the quality of ANC services. For example, the introduction of EQA for ANC lab services for syphilis and hepatitis testing and outreach ANC diagnostic services using the mobile Vscan, have led to increased service uptake. Support to Data Management: Transform: HDR assisted the GOE/DHIS2 to roll-out its capacity strengthening support platform. Key assessments conducted by Transform: HDR to date are informing management performance decision-making and resource allocation for improved outcomes. For example, the review of the Transform: HDR Geographic Information System (GIS) assisted mobility assessment for both Afar and Somali pastoralist communities revealed that cost-effective and comprehensive mobile outreach health services for the pastoralist population demand reconfiguration27. Mapping pastoral communities’ mobility routes and destinations, together with their existing facilities, help enhance the availability and accessibility of health and other basic services in pastoral regions. This mapping is particularly relevant for the implementation of mobile health services during the seasonal movement of pastoral communities through the identification of routes and destinations. The results from the study 26 FGD with FMOH Special Support Directorate staff 27 Transform: HDR GIS assisted mobility assessment for both Afar and Somali pastoralist communities, Assessment Report, 2018 Mid-term Performance Evaluation Report USAID Transform: HDR Activity 36 showed, water points, grazing lands, animal health, human health points, schools, market places, and grazing lands, all of which are vital information for the non-health sector offices. These mobility patterns and pastoral ways of life will definitely demand the reconfiguration of a cost-effective, comprehensive mobile outreach health service for pastoralists with the necessary supplies. The mapping could help the IP team, and the woredas to increase immunization coverage, ANC services, and improve malaria case management. The mobile teams can also integrate the use of mobile ultrasound for pregnant mothers. Regional and woreda stakeholders recognized and appreciated the training and mentorship approach and opportunities to Transform: HDR. “…. In addition, we have learned how to measure the outcome through various methods like how to strengthen the quality improvement team; training was given how to analyze incoming information and reports and how to use it as an input for decision. The project helped us how to implement a community scorecard to use it as an input for the action plan preparation. It also helped us to train health extension workers to monitor community-based neonatal care” -- (Benishangul, woreda health office) While Transform: HDR has led to some improved use of activity data, there is little evidence that the routine GOE/DHIS2 data is analyzed, disseminated, and used by stakeholders to make adjustments in the programming of resources or fine-tuning strategies. Much more can be done to use the data to reveal where the greatest unmet need is and support resource planning at the various health service delivery levels. For example, data on FP was not observed to be driving decisions about method mix or contributing to decisions on the organization or timing of ANC sessions to better engage men to accompany their wives. A Comparison of Transform: HDR and Non-Transform: HDR Supported Projects This refers to any comparative strengths and weaknesses of Transform: HDR Activity in relation to non-Transform project implementation. The comparison was drawn based on the design, implementation, management or monitoring, capacity, flexibility, etc. of the project. Transform: HDR, when compared with other non-Transform: HDR projects, was complimented by stakeholders for the participatory manner in which selection of implementation woredas is made. It has high flexibility. Its interventions are driven by government priority areas, working closely with the government. There is high government ownership of Transform: HDR interventions, efficient and effective post-training follow-ups, coaching and mentorship, and supervision is integrated and done with the health office staff. A KII respondent from Afar had this to say; “…. Yes, Transform: HDR is different from other non-USAID supported projects because they are closely working with the government. They want the government to take ownership of all Mid-term Performance Evaluation Report USAID Transform: HDR Activity 37 activities. Transform: HDR activities are government-driven and are realistic. HDR is a flexible project within the seated thematic area. What I know from other projects is that they come with one activity plan for five years and implement the same thing throughout the course of action. They may change their targets, but they don’t change activities whereas Transform: HDR generates activities with stakeholders every year based on need……” -- (RHB, Afar) Some of the possible factors why the Transform: HDR intervention woredas are performed better than the non-Transform intervention areas in some of the indicators may be due to the following reasons identified during the survey. First, there is likely to have been some “spillover effect” within woredas in the same regions, which could also make non-Transform intervention areas have similar performance as Transform: HDR intervention. The population characteristics are similar, and people likely move between facilities, and due to the lack of proper records, it is difficult to detect this movement without a much more detailed examination of the health registers. Evaluators noted that the determinants tended to be the same between Transform: HDR supported sites and non-Transform: HDR sites. However, there are some differences between the pastoral regions and agrarian regions. Selection of woredas at start-up of the Transform: HDR Activity could account for the similar observed proportions while the DID has accounted for differences within cohorts. Transform: HDR, together with the regional health bureaus, selected very remote regions at the initiation of the activity. These locations tended to be in woredas, where there were no partners operating and Transform: HDR started at the most basic level of the new introduction of interventions. When Transform: HDR started, there were also many external development partners operating in most of the non-intervention areas. These partners continued their support, and this has led to good performance on some health indicators. Spillover effect: At the start-up, Transform: HDR provided TOT for health professionals indiscriminately from the intervention and non-intervention woredas. This might have led to a spillover effect and positively influenced the performance of the non-intervention areas. Some of the partners involved in non-Transform: HDR woredas in developing regions include: Gambella: Doctors with Africa – MNCH support including renovation of health facilities, Maternity Waiting Home construction, financial support for maintenance of ambulances, and support for basic infrastructure such as water tanks to health facilitates; Core Group/Mekane Yesus- support immunization campaigns with fuel and transportation, training HEWs, EPI focal and volunteers; UNFPA – support in the area of MNCH such as capacity building training, supportive supervision and financial support; John Snow Inc – supports Extended Program for Immunization through technical support and capacity building; and the Catholic Mission Clinic – provides maternal and newborn health services. Mid-term Performance Evaluation Report USAID Transform: HDR Activity 38 Somali: Lowland WASH supporting WASH activities; Save the Children/ANCHOR providing support in health advocacy and FGM, and Mercy Corps supporting health system strengthening. Afar: Emory University is supporting maternal and child health by encouraging health facility delivery for mothers, ANC attendance and services, referral linkage, nutrition, etc. Plan International, UNICEF, and AMREF provide support on areas of Nutrition, including distribution of therapeutic foods (F-100), train health professionals, provide vaccination campaigns, and support outreach services. Benishangul Gumuz: Emory University is supporting Community-Based Maternal and Neonatal Health such as ANC, Delivery, PNC, and Neonatal up to 2 months as their major focus area. The support includes in-service training, continuous supportive supervision, and review meetings. Emory also supports children’s feeding practice by preparing different demonstrations to trainees at the health center and health post level to the selected mothers. IRC is working on immunization by preparing different awareness creation programs and defaulter tracing mechanisms in addition to technical support through in-service training, continuous supportive supervision, and review meetings. One WASH program is supporting hygiene and sanitation. KII from Benishangul Gumuz, Mengae Woreda health office head, a non-Transform HDR woreda had this to say: “To ensure community ownership and sustainability, EMORY works with selected HDAs at the community level, and the selected HDAs diffuse the information using their 1 to 5 networks as a result in awareness towards health-seeking behavior, child nutrition including child feeding practice, all of which have significantly improved. Similarly, IRC also works on community surveillance and EPI by providing training to the HDAs to expand EPI coverage and to strengthening the surveillance system. So, we can say that this combination added a good value on our health systems because the woreda EPI coverage is increasing and malnutrition burden is minimizing”. There are other development partners working in both Transform intervention sites and non￾Transform supported sites. The findings from the evaluations have also shown the factors responsible for the observed performance in Transform: HDR intervention sites and non￾Transform: HDR sites in family planning, maternal health, and child health, which seem to be similar, with limited varying factors specific to pastoral regions. Also, due to similar population characteristics and the spillover effect within woredas in the same regions, the evaluation found determinants tending to be the same between Transform: HDR supported sites and non￾Transform: HDR sites. Thus, the observed differences in trends could be explained by partners and the level of investment in their interventions/programs as plausible determinants. However, there are some differences between the pastoral regions and agrarian regions. Mid-term Performance Evaluation Report USAID Transform: HDR Activity 39 Summary: What makes Transform: HDR efficient? • The flexibility of the senior management in understanding and responding to the regional team’s needs; • Improved logistics management in supporting technical assistance to the intervention areas. • Developing and building a pool of master trainers/experts from regions to facilitate and cascade training. • Identifying and responding to regions’ critical needs, e.g., medical supplies and equipment. • Improved GOE leadership commitment and willingness to work with Transform: HDR. Bottlenecks for Efficiency • The geographic spread vs. limited technical staff at the regional and local levels. • Adapts and responding to natural and man-made disasters in “emergency prone” regions. • Challenges associated with recruiting and deploying qualified staff to regional and sub￾regional levels. • Shortage of trained public health practitioners and willingness to work in the regions. Moving Forward to Improve Efficiency May Require: • Revisit/reassigning technical staffing to provide meaningful and timely support to the targeted woredas. • Revisit resource allocation by reallocating more resources to thematic areas and regions with low performance. • There is a tradeoff that needs to be determined by USAID/Ethiopia as to the number of geographic areas to be covered versus achieving the spectrum of both health systems strengthening objectives and health results, and quality of care. • Promote innovative ways to support regional and woreda stakeholders to improve the use of data for decision-making and planning. 3.3 EFFECTIVENESS A) Background and Overall Findings To evaluate the effectiveness of the Transform: HDR Activity, data was collected from households on key performance indicators. The household survey followed methods similar to the Transform Activity baseline survey to allow a comparison of results. A total of 2,635 women of reproductive age were interviewed from Transform: HDR targeted regions (Afar, Somali, Gambella, and Benishangul Gumuz) covering USAID Transform: HDR intervention sites and non-Transform: HDR sites. The study assessed and analyzed data on five result areas: Family Planning, Maternal Mid-term Performance Evaluation Report USAID Transform: HDR Activity 40 Health, Newborn Health, Child Health, and Cross-cutting (Gender). Difference-in-Differences (DID) analysis was employed to measure the change in Transform: HDR intervention sites between baseline and mid-term evaluation compared with the change achieved in non￾Transform sites. The evaluation team also collected and analyzed data from health facilities to assess the interaction between the supply of health services and utilization. B) Respondent Demographic Characteristics All respondents were women of reproductive age (32.4% were below age 25, and 84.4% were below age 40) in Transform: HDR intervention sites. The mean age of the respondents was 29 years in both intervention and non-intervention sites. More than half of the respondents (54.4% in the intervention and 51.5% in non-intervention areas) had no formal education. The majority of the respondents (63.9%) are Muslims, followed by Protestants (19%) in the intervention sites. Four in five women (81%) in intervention areas are married or cohabitating. Table 5 below provides additional details of the surveyed women. Matching by background characteristics was considered to control its effect while conducting the DID analysis to increase confidence in the sample in the intervention and non-intervention areas, in addition to checking whether there was a difference between intervention and non-intervention areas, which showed similarities in demographic characteristics. Table 5: Respondent Demographics Characteristics at Mid-term, 2019 Characteristics Transform: HDR Intervention Sites (N= 2102) Non-Transform HDR Sites (N=623) Age Group 15-19 16.3% 13.5% 20-24 16.1% 18.1% 25-29 19.8% 22.5% 30-34 16.6% 18.1% 35-39 15.6% 16.2% 40-44 9.6% 6.9% 45-49 6.0% 4.7% N 2102 623 Educational Status No formal education 54.42% 51.52% Primary education (1 - 8) 35.16% 37.08% Secondary education (9 - 12) 7.61% 8.67% Above secondary education 2.81% 2.73% N 2102 623 Religion Muslim 63.89% 63.24% Protestant 18.98% 18.78% Catholic 0.38% 0.32% Orthodox 13.75% 17.34% Other 3.00% 0.32% N 2102 623 Marital status Single/Never married 10.09% 7.22% Married or cohabitating 81.35% 86.04% Divorced 3.19% 3.85% Widow 4.76% 2.41% Separated 0.62% 0.48% N 2102 623 * There is a significant difference between the two groups. Mid-term Performance Evaluation Report USAID Transform: HDR Activity 41 C) Analysis of Key MNCH/FP Indicators Four approaches have been used to analyze the performance of key MNCH/FP indicators across the five result areas: Family Planning, Maternal Health, Newborn Health, Child Health, and Cross￾cutting (Gender). The four analytic approaches are: 1) Change in proportion of key indicators in the Transform: HDR intervention sites from the baseline (2017) compared with the change observed in non-Transform: HDR sites; 2) Region-specific performance trends on key indicators at mid-term by comparing with baseline; (3) Difference-in-Differences (DID) analysis; and 4) Comparing Transform: HDR’s target at midpoint against mid-term indicator values. Mid-term Performance Evaluation Report USAID Transform: HDR Activity 42 3.3.1 FAMILY PLANNING (FP) a) Change in proportion of key FP indicators in the Transform: HDR intervention sites (Baseline vs. Mid-term) compared to the non-Transform: HDR areas Improvement in the use of LAFP is one of the successful achievements of the activity. Family planning, including the use of modern contraceptive methods (MCPR), in Transform: HDR intervention areas, has shown a significant increase from baseline (Figure 1). The MCPR among married women increased by 2.9 percentage points (18.9% at baseline and 21.8% at mid-term, p=0.015), and the use of LAFP increased by 2.1 percentage points (3.0% at baseline and 5.1% at mid-term, p<0.001). At mid-term, the most prevalent LAFP method used was Implanon, which is used by 4.5% of women, while there was a negligible proportion of women using the IUCD (0.1%) and Jadelle (0.5%). The use of family planning methods in the extended postpartum period (PPFP) also showed a significant increase from the baseline (12.2% at baseline and 16.2% at mid-term, p=0.029); injectable (7.9%), lactational amenorrhea method (3.9%), and Implanon (3.3%) were the most used methods whereas no woman used the IUCD method during the extended postpartum period at the mid-term. Unmet need for family planning showed a decline from the baseline, but the change is not statistically significant (21.4% at baseline and 20.3% at mid-term, p=0.182). Unmet need for family planning methods among married women also declined from 25.1% at baseline to 23.3% (by 1.8 percentage points). Demand for family planning methods among married women increased from 44% at baseline to 45.1% at mid-term (by 1.1 percentage points). This positive result was achieved despite many socio-cultural obstacles that past family planning programs in these regions have encountered. It is important to invest and build on these important and positive achievements. Figure 1: Performance on Family Planning Indicators between Baseline & Mid-term Compared by Intervention and Non-intervention Sites, 2019 18.9 26.0 3.0 5.9 12.2 21.2 21.4 25.5 21.8 23.5 5.1 3.9 16.2 20.0 20.3 21.3 Transform HDRNon-Transform HDR Transform HDRNon-Transform HDR Transform HDRNon-Transform HDR Transform HDRNon-Transform HDR MCPR among Married LAFP PPFP Unmet need for FP Baseline Midterm Mid-term Performance Evaluation Report USAID Transform: HDR Activity 43 In Transform: HDR intervention areas, further disaggregation of MCPR shows that 4.9% of married women used modern contraceptive methods for limiting children, and 16.9% have used contraception for spacing children (at the mid-term). Comparatively, the unmet need for limiting children among married women was 3.1%, and for spacing children, it was 20.2%. The overall demand for modern contraceptive methods among married women was 45.1%, where 8% was the demand for limiting children, and 37.1% was for spacing children. This shows that 55% of married women did not have the need for modern contraceptive methods (Figure 2(A)). The demand for modern contraceptive methods among married women is a bit higher in non￾Transform: HDR areas compared to Transform: HDR sites (Figure 2(B)). Figure 2: Demand for Modern Contraceptive Methods among married women for Transform: HDR Intervention (A) and Non-intervention (B) Areas, March 2019 The findings showed that the majority of the family planning indicators are increasing in Transform: HDR intervention areas but regrettably are declining in non-Transform intervention areas. MCPR among married women declined from 26% to 23.45% in non-intervention areas. Similarly, LAFP and PPFP showed a decline in proportion in non-intervention areas, though the decline is not statistically significant. The absence of an external partner providing a concerted effort in non-Transform: HDR intervention sites could have contributed to the observed differential gap. “……. According to Afar Culture, married people don’t use contraceptives. From the beginning, if a person doesn’t give birth, she will not get married. In general, there are no such things as a contraceptive after marriage in Afar culture. We always want to give birth to more children. Currently, there are some people who are utilizing it, but previously it was not known. According to our culture, one man can marry four wives. He wants each of his wives to give birth to at least ten children. Women also want to give birth like their husbands” …. (Afar, male, 25-50 old FGD participants) MCPR used for Limiting 5% MCPR used for Spacing 17% Unmet need for Limiting 3% Unmet need for Spacing 20% No need for Modern Methods 55% (A) MCPR used for Limiting 2% MCPR used for Spacing 21% Unmet need for Limiting 5% Unmet need for Spacing 19% No need for Modern Methods 53% (B) Mid-term Performance Evaluation Report USAID Transform: HDR Activity 44 Women in the Benishangul FGD had this to say: “…. The problem is our culture and husbands. Based on our religious point of view, family planning is forbidden. It is a sinful act. It’s our husbands who don’t want us to use family planning. Most of the women don’t use family planning because of their husbands’ influence. But it is wrong. Family planning is important than these cultures…... (Benishangul, FGD, women, 25-49). The survey results also show that 5.3% of women in Transform: HDR intervention sites reported they have been turned away or did not receive services when they wanted FP services from health facilities. Some of the reasons include a) the providers were not there (19.8%), b) the facility was closed (24%), and c) commodity stock-out (51%). The health facility assessment in the Transform: HDR intervention sites shows that 25% of health centers had stock-outs of Implanon NXT28 in the last six months before the survey. In non-Transform HDR intervention sites, 4.0% of women reported that they were turned away or did not receive services when they wanted FP services. The main reasons for not receiving the FP service were commodity stock-out, reported by 59.1% of the women, and the facility was closed, reported by 31.8% of the women. The health facility assessment also shows that 86% of the health facilities provide at least one modern method of family planning services,the remaining 14% (5 health centers and nine health posts) do not provide any FP method services. Injectables (78%), pills (80%), and implants (68%) are offered in most of the health facilities. All primary hospitals offer emergency contraceptive methods, IUCD insertion services, and implants (Implanon and/or Jadelle). More than 62% of health centers offer implants (Implanon and/or Jadelle) and emergency contraceptive methods. Only half of the health centers (50%) offer IUCD insertion services (see Table 6 for details). Of the health centers and health posts assessed, 84% provide at least two temporary methods29 , and 55% provide at least four temporary methods. In non-Transform: HDR sites, 90% of the health facilities provide at least one modern family planning method, where pills are provided in 87% of health facilities, injectables in 77%, and implants in 30% of the health facilities. Expanding access to the different choices of family planning will give women greater choices in selecting a method that meets their needs for delaying, spacing, or limiting pregnancy. In addition, having a wide choice of methods will meet the needs of some discontinuers if they have the option of switching; the addition of one new method to the available method mix can increase contraceptive prevalence by as much as eight percentage points30. Ensuring availability 28 Implanon is the only method included in the tracer drugs list 29 Temporary modern methods include: Pill, injectables, implants, condom and emergency contraception Note: Considerable number of health posts in Gambella and Somali were not functional at the time of survey; either HEW not available or infrastructure does not exist. 30 Ross, J. and J. Stover. 2013. “Use of modern contraception increases when more methods become available: Analysis of evidence from 1982-2009,” Global Health Science and Practice, 1(2): 203–212 Mid-term Performance Evaluation Report USAID Transform: HDR Activity 45 and steady and consistent supply of FP commodities at health posts and health centers will contribute to increased utilization of FP services. Table 6: Methods of Family Planning Offered at Primary Health Care Facilities (%) FP Methods Provided Primary Hospital Health Center Health Post Pill 100 87 78 Injectable 75 85 78 Implanon 100 75 0 Jadelle 50 50 0 IUD 75 50 0 Emergency contraception 100 62 0 Condom 50 79 41 At least two temporary modern methods 100 85 85 At least four temporary modern methods 50 73 23 # of health facilities offering any modern contraceptive method 4 52 37 # of health facilities not providing any FP method 0 5 9 Note: The Use of SDM was included as a modern FP method in the household survey analysis (MCPR) even though the proportion is very small (0.1%). In addition, none of the Health facilities reported that they provide SDM. b) Region Level Performance on Key Family Planning Indicators at Mid-term compared with Baseline values Regional variation in FP usage was apparent. Somali and Gambella showed declining performance in the majority of FP indicators from baseline, whereas Benishangul Gumuz showed improvement in all FP indicators. Compared with other regions, Somali has the lowest performance in all FP indicators, both at mid-term and baseline (Figure 3). Unmet need for family planning in Somali is 6.7%, while CPR is 0.2%, showing that the need for family planning is not yet recognized in the society. Reasons for not using FP methods for women in Somali region include: around half of the women (47%) responded that FP use is against their religion, 27% said they want to get pregnant, 12% of the women said they do not have a partner, 6.3% of the women said it is because they do not know where to obtain FP methods, 2.8% said their partner/spouse refuses to use FP methods, and 3.3% of women said supply is not always available. Transform: HDR’s baseline survey and other studies showed that religion is a determinant factor for the utilization of FP methods. Benishangul Gumuz performed better on most FP indicators and were better than the agrarian regions. For example, MCPR in Benishangul Gumuz is 54.5%, exhibiting the highest increase by 12.8 percentage points from baseline (41.7% at baseline and 54.5% at mid-term) compared with other regions. Mid-term Performance Evaluation Report USAID Transform: HDR Activity 46 Figure 3(a): Region level Performance on Key Maternal Health Indicators at Mid-term, 2019 Figure 3(b): Region level Performance on Key Maternal Health Indicators at Mid-term, 2019 c) Difference-in-Differences (DID) Analysis of Key FP Indicators The contribution of Transform: HDR interventions in improving FP uptake is estimated through matching the baseline characteristics (women’s age, education level, paid work, and participation in decision-making) in the Transform: HDR intervention and non-intervention groups (see Table 7 for details). The Transform: HDR intervention contributed to the 3-percentage point increase in MCPR, although the increase is not statistically significant. The activity significantly contributed to increasing the uptake of LAFP by 4.1% (95% CI: 1.7% – 6.4%; p<0.001) though part of this increment was due to the decline in non-Transform HDR areas. Moreover, Transform: HDR intervention contributed to reducing the unmet need for contraception by three percentage points, but the decline was not statistically significant. 4.3 4.5 17.9 36.4 0.2 0.6 0.8 10.7 6.9 0.2 16.2 45.5 0.8 0 2.3 15.3 Afar Somali Gambella Benishangul Gumuz Afar Somali Gambella Benishangul Gumuz MCPR among Married LAFP Baseline Midterm 6.4 3.1 9.6 30.5 16.6 8.8 26.1 35.6 6.7 0.8 7.5 53.1 15.4 6.7 25.3 32.3 Afar Somali Gambella Benishangul Gumuz Afar Somali Gambella Benishangul Gumuz PPFP Unmet need for FP Baseline Midterm Mid-term Performance Evaluation Report USAID Transform: HDR Activity 47 Table 7: DID of Family Planning (FP) Indicators, Disaggregated by Transform: HDR Intervention Sites & Non-Transform HDR Intervention Sites, Between Baseline (2017) & Mid-term (2019) Indicators Baseline Survey (Dec 2017) Mid-term Evaluation Survey (Dec 2019) The Difference in Difference (DID) Interventi on Sites (%) Non- Non￾Interventi on Sites (%) Diff (95% CI) Interve ntion Sites (%) Non￾Interven tion Sites (%) Diff (95% CI) Family Planning MCPR 18.9 25.2 -6.3 (-9.4; - 3.1)*** 18.3 21.3 -3.0 (-6.1; 0.1)* 3.2 (-1.3; 7.7) Long-Acting (LAFP) 3.6 6.4 -2.8 (-4.6; - 1.0)*** 5.1 3.8 1.3 (-0.5; 3.1) 4.1 (1.7; 6.4)*** Unmet need for FP 24.2 30.1 -5.9 (-9.2; - 2.6)*** 20.3 23.2 -2.9 (-6.2; 0.4)* -3.0 (-1.7; 7.7) Note: The proportions for the DID analysis are calculated after matching and may slightly differ from the indicator observed proportions. *** p<0.01; ** p<0.05; * p<0.1; Diff – Difference in proportions; DID – Difference-in-Differences; CI – Confidence Interval Determinants of Family Planning Services Utilization Apart from regional variation, the use of modern contraception is affected by age, religion, the number of children, hearing FP/RH messages, and a woman working for pay (see Table 8 for details). The findings show that women aged 25-29 are 2.23 times more likely to use a family planning method than women aged 15-19 (AOR=1.96; CI 1.01 -3.79). A woman from the Orthodox religion is 2.57 times more likely to use contraceptive methods compared to a woman from the Muslim religion. Similarly, a woman who has one or two children is three times more likely to use FP methods compared with a woman who has no children. Hearing an FP/RH message is also a determinant factor for FP use, i.e., women who heard FP/RH messages in the last few months are 1.84 times more likely to use modern contraception compared to women who have not received FP messages. On the other hand, education level and women’s participation in decision-making seem to have little effect on FP method utilization. A woman who attended primary education is 1.1 times more likely to use FP methods compared with a woman with no formal education. Women who participate in household decisions are 1.18 times more likely to use FP methods than their counterparts. Mid-term Performance Evaluation Report USAID Transform: HDR Activity 48 Table 8: Factors Affecting the Use of Modern Contraception Among Married Women in Transform: HDR Sites, 2019. Indicators CPR (n=2102; event=385) LA methods (n=2102; event=107) Unmet need (n=2102; event=426) OR 95% CI OR 95% CI OR 95% CI Region Afar (Ref) 1.0 1.0 1.0 B/Gumuz 5.55 3.39−9.11 14.98 4.97−45.14 2.47 1.62−3.76 Gambella 2.23 1.16−4.29 1.90 0.49−7.39 1.82 1.01−3.28 Somali 0.04 0.01−0.29 NA NA 0.54 0.33−0.89 Age 15 - 19 (Ref) 1.0 1.0 1.0 20 – 24 1.43 0.69−2.94 0.56 0.23−1.35 0.77 0.44−1.34 25 – 29 2.23 1.06−4.69 0.48 0.19−1.18 0.50 0.28−0.9 30 – 34 1.98 0.88−4.46 0.29 0.1−0.84 0.43 0.23−0.82 35 – 39 1.32 0.56−3.11 0.10 0.03−0.34 0.26 0.13−0.53 40+ 0.60 0.23−1.53 0.06 0.02−0.25 0.1 0.05−0.24 Education No education (Ref) 1.0 1.0 1.0 Primary 1.07 0.74−1.54 1.56 0.88−2.75 1.62 1.18−2.22 Secondary + 1.06 0.6−1.88 1.70 0.75−3.88 1.63 1.00−2.66 Paid work No (Ref) 1.0 1.0 Yes 0.75 0.53−1.08 0.61 0.37−1.02 Religion Muslim (Ref) 1.0 1.0 1.0 Orthodox 2.57 1.65−4.01 1.52 0.84−2.75 0.77 0.50−1.18 Protestant and others 0.90 0.55−1.47 1.17 0.58−2.34 1.27 0.79−2.05 Child ever born 1 - 2 (Ref) 1.0 1.0 1.0 3 – 4 0.82 0.52−1.29 1.35 0.71−2.59 1.54 1.04−2.28 5 – 6 0.73 0.43−1.27 1.00 0.43−2.34 1.89 1.17−3.05 7+ 1.36 0.69−2.68 1.89 0.61−5.92 3.20 1.79−5.73 The ideal number of children 0 - 3 (Ref) 1.0 1.0 1.0 4 – 5 0.88 0.52−1.5 0.89 0.44−1.81 0.89 0.53−1.49 6 – 9 0.59 0.33−1.05 1.34 0.6−3.01 1.12 0.65−1.95 10+ 0.17 0.09−0.34 0.45 0.14−1.49 0.44 0.24−0.8 RH/FP message No (Ref) 1.0 1.0 1.0 Yes 1.84 1.34−2.52 1.46 0.92−2.32 1.39 1.07−1.82 Women's participation in health care decisions No (Ref) 1.0 1.0 Yes 1.18 0.81−1.72 0.78 0.57−1.07 Mid-term Performance Evaluation Report USAID Transform: HDR Activity 49 Non-inclusion of zero in the 95% CI indicates statistical significance. Shaded cells indicate non-significant variables in the bivariate analysis. The pastoralist communities are mobile in search of water and pasture, and there is no well￾developed model to provide services while they are moving. Additionally, pastoralists may have problems getting refills as they move from site to site. The low result observed in CPR could also be associated with socio-cultural factors in the developing regions of Ethiopia. A respondent from the Somali Regional Health Bureau (RHB) had this to say: “Here, the value given for children is unquenchable, and it affects the utilization of family planning services despite that demand creation efforts are made.” Local values supported by religious traditions towards family planning were documented as a stumbling block to using family planning. Local values support communities and couples having more children; husbands have low interest in using family planning methods, and more value is attached to more children, and hence a big family size is another evident factor hindering FP promotion and use. This is particularly the case in the Somali, Afar and Gambella regions. A 70-year-old male pastoralist respondent in Afar had this to say: “If any person comes and talks about family planning in front of me or here in Afar at all, we will kick him. There is no demand for the utilization of family planning methods here; we don’t need it. Here, couples need to give births to more children. More children support us, and boys provide protection to the family. Husbands are not interested in family planning, and there is no awareness creation on that.” An FGD participant in Gambella Region had this to say: “If women do not give birth to their next child in five years, men suspect that they are using FP. So, the men will send the women back to their parents and claim for the dowry (cattle) to be returned back to the men's family. Because of this, women will not try to use a family planning method in our clan (or this community).” A 34-year-old man, FGD participant in Etang Woreda. “Women sometimes use FP methods without the knowledge of their husbands, but when husbands come to know it, they will force the women to remove it. They say they want more children … So, we will be forced to remove it right away though we do not want to have pregnancy… Usually, the husbands’ parents influence to have more children because they pay about 30,000 Birr as dowry. So, they say, ‘We didn’t pay this high amount of dowry just for two children, you should have as many children as possible’”. FGD participants of the women group, Itang Woreda. Mid-term Performance Evaluation Report USAID Transform: HDR Activity 50 The efforts to boost FP performance need to introduce innovative and effective strategies to overcome barriers such as fear of contraceptives’ side effects, lack of women’s autonomy in health care decisions, and poor health service provision, women's low agency, and cultural and traditional/religious restrictions. In addition, context-specific actions need to be considered in order to close the gaps in regional inequalities. Coupled with increases in the availability of free and long-acting contraceptive methods, Transform: HDR has started working through community awareness forums using local CBOs, religious or clan leaders, and this has significantly contributed to minimizing some of the myths and increasing service utilization. In non￾Transform: HDR intervention sites, there are few partners, and the few need to adapt these approaches. There is also a high turnover of trained staff from different hard-to-reach health facilities due to work environment, lack of drinking water, road accessibility and more hard-to￾reach communities, which remain a challenge, The frequent stock-outs and poor stock management of essential products could increase the gaps coupled with the health extension program which is still weak in developing regions to contribute to the achievement of contraceptive prevalence rates3132 Summary: Family Planning The Transform: HDR interventions made a significantly positive contribution to the increasing use of LAFP contraception (by 4.1%). The low result observed in MCPR (as shown in the DID analysis) could be mainly associated with socio-cultural factors in the developing regions. Moving forward, to increase the utilization of family planning, Transform: HDR may consider: ▪ Expanding RH/FP messaging (women who heard FP messages are 1.8 times more likely to use FP). Even if the regression results show this from the quote, it seems that a lack of awareness is not an issue as such. ▪ Strengthen SBCC messaging and other interventions targeting men and associated cultural beliefs. ▪ Support expanding access to contraceptive methods, including availing different choices of FP methods at primary health care centers and ensuring the supply of the most sought after methods, such as Implanon. ▪ Expand work with communities and religious leaders to change social norms that might be inhibiting the uptake of contraceptives (women from Orthodox religion are 2.57 times more likely to use FP compared with women from Muslim religion). ▪ Intensify community interventions in Afar and Somali regions to increase demand for FP. ▪ Intervene to satisfy the contraceptive demand in B/G and Gambella regions. ▪ Target younger groups to reduce the unmet need (unmet need is higher among age 15-19 years age group). ▪ Offer expanded access to permanent contraception for women with more than three children who wish to end their fertility as unmet need increase as the number of children increases. 31 Taye E, Mekonen D, Debele T. Prevalence of post -partum modern family planning utilization and associated factors among postpartum mothers in Debre Tabor town, North West Ethiopia, 2018. BMC Res Notes (2019) 12:430 https://doi.org/10.1186/s13104-019-4464- 0SEARCH 32 Somali and Afar RHB KIIs concurred and echoed these factors. Mid-term Performance Evaluation Report USAID Transform: HDR Activity 51 3.3.2MATERNAL HEALTH a) Change in proportion of key Maternal Health indicators between Baseline vs. Mid-term in the Transform: HDR intervention areas and non-Transform: HDR areas The majority of key maternal health indicators have shown improvement in the Transform: HDR intervention areas compared with their baseline values (Figure 4). Skilled birth attendance significantly increased from 28.1% at baseline to 34.3%, by 6.2 percentage points, (p-value=0.01); early postnatal care for the mother within two days increased significantly, from 22.9% to 40.1% by 17.2 percentage points (p-value<0.01); and women received MNCH services through mobile health team significantly increased from 6.4% to 24.8%, by 18.4 percentage points (p￾value<0.01). Transform: HDR has been supporting the provision of MNCH services through the mobile health team in Afar and Somali regions. The proportion of women that received MNCH service through the mobile health team in these regions was 31.6% at mid-term, where a majority (82.9%) reported they received vaccination service, followed by child health service (38.8%) and health education (32.9%). This shows that mobile health service might have positively contributed to the observed improvement in child immunization services, as described in Section 3.3.4. On the contrary, the data also show that maternal health services were provided less frequently through mobile health service, where only 17.1% of women reported receiving FP service, 16.8% ANC service, and 10.2% PNC service. In addition, early initiation of ANC within the first trimester significantly increased from 24.7% at baseline to 45.0% at mid-term, by 20.3% (p-value <0.01). Key informant interviews with Woreda Health Office (WoHOs) also revealed that mobile ultrasound had attracted more pregnant women to come to health facilities for ANC, which might have contributed to increased early initiation of ANC in Transform HDR intervention sites. KIIs from woredas in Afar and Gambella Region had this to say: “…Previously we do not have any ultrasound, but now Transform: HDR provided us one ultrasound, and it plays a major role in improving maternal health service during ANC and delivery… the number of pregnant mothers in ANC increased dramatically due to the ultrasound. When we tell the pregnant mothers in the community that they will be able to see the fetus with ultrasound, they become motivated to come to the facility.” -- WoHO head, Afar Another KII, Etang WoHO MNCH coordinator, Gambella had this to say; “The utilization of ANC service increased four times than it was before immediately after the introduction of the Ultrasound…It encourages mothers to come for ANC follow up. Mothers from other neighboring woredas were even visiting the health center due to the ultrasound.” Women received essential components of ANC (among those who had at least one ANC) also Mid-term Performance Evaluation Report USAID Transform: HDR Activity 52 increased significantly by 7.8 percentage points, from 27.4% to 35.2% (p-value = 0.01). Of the mothers who have attended at least one ANC visit, 89% reported that their BP was monitored, 72% reported they received urine tests, and 78% provided a blood sample. Counseling on nutrition and pregnancy danger signs was also provided for 65% and 50%, respectively, during the ANC visits. However, the proportion of women who received four or more ANC visits did not change over the two periods. Even though early initiation of ANC has significantly increased in the last two years, ANC 4+ visits did not show any progress. The data shows that of women who initiated early ANC visits, close to half of them (48%) received four or more ANC visits. But from those who had four or more visits, 81% of them had initiated early ANC visits. This might be related to the quality of ANC services provided and/or lack of awareness with the importance of having at least four visits, which again can be related to counseling services provided at first contact with these mothers. Iron and Folic Acid supplementation for at least 90 days significantly dropped by 8.3 percentage points in intervention areas (p<0.05). Similar to the FP results, there has been a deterioration in performance in maternal health indicators in non-Transform: HDR sites from their baseline values: skilled birth attendance significantly declined by 18.5 percentage points (p-value<0.01); women who received four or more ANC visits also declined 5.1 percentage points (p-value = 0.36); and Iron and Folic Acid supplementation for at least 90 days declined by 12.0 percentage points. Whereas women received MNCH services through the mobile health team significantly increased. Early postnatal care also showed an increase of 6.4 percentage points, though it is not statistically significant. A comparison between intervention and non-intervention sites at the mid-term shows that performance on the majority of indicators is much higher in intervention areas. Skilled birth attendance was significantly higher in intervention areas (34.3%) than non-intervention areas (25.3%) %) (p-value: 0.02). Iron and Folic Acid supplementation are also higher in intervention areas compared with non-intervention areas (15.0% versus 12.1%), though the difference is not statistically significant (p-value>0.1); the same is true with women who received MNCH services through the mobile health team (24.8% versus 23.3%). The proportion of women receiving four or more ANC visits is lower in intervention areas (26.4%) than non-intervention areas (33.3%), though it is not statistically significant (p-value> 0.05), meaning that there is not enough evidence to conclude that the performance in non-intervention sites is different from intervention areas with respect to four or more ANC coverage. It is important to note that Transform: HDR intervention areas had less performance in all maternal health indicators at baseline compared with non-Transform: HDR areas. In other words, Transform: HDR sites started from a lower baseline compared to non-Transform intervention areas. Mid-term Performance Evaluation Report USAID Transform: HDR Activity 53 Regional KIIs from the pastoralist regions of Afar and Somali all echoed the factors responsible for observed maternal services utilization to include socio-cultural factors, the mobile lifestyle of the populations, distance to the health facility, and limited availability of waiting areas at health facilities. Other factors mentioned affecting the utilization of skilled delivery services include low education on maternal health, limited availability of information, lack of skilled delivery experience, and quality of antenatal care utilization3334. The poor-quality services that are not mother-friendly and poorly equipped health facilities may not convince mothers to deliver in health facilities35. It is important to address individual, family, cultural, and structural factors to substantially increase deliveries attended by skilled attendants. Figure 4 (a): Performance on Maternal Health Indicators between Baseline & Mid-term Compared by Intervention & Non-intervention Areas, 2019 Figure 4 (b): Performance on Maternal Health Indicators between Baseline & Mid-term Compared by Intervention & Non-intervention Areas, 2019 33 RHB KIs from Afar and Somali 34 Tesfaye B, Atique S, Azim T, and Kebede M. Predicting skilled delivery service use in Ethiopia: dual application of logistic regression and machine learning algorithms. BMC Medical Informatics and Decision Making (2019) 19:209 35 WHO. Policy dialogue. Improving Skilled Birth Attendance in Ethiopia. Accessed on 25 December 2019. https://www.who.int/evidence/sure/pdimprovingskilledbirthattendanceethiopia.pdf?ua=1 24.7 30.1 26.0 38.4 27.4 41.7 23.3 24.1 45.0 53.9 26.4 33.3 35.2 36.5 15.0 12.1 Transform HDRNon-Transform HDR Transform HDRNon-Transform HDR Transform HDRNon-Transform HDR Transform HDRNon-Transform HDR Early initiation of ANC ANC 4+ Essential components of ANC IFA Baseline Midterm 28.1 43.8 22.9 35.6 6.4 12.5 34.3 25.3 40.1 42.0 24.8 23.3 Transform HDR Non-Transform HDR Transform HDR Non-Transform HDR Transform HDR Non-Transform HDR SBA Early PNC for mothers MNCH services through mobile health Baseline Midterm Mid-term Performance Evaluation Report USAID Transform: HDR Activity 54 Moreover, the health facility assessment findings show that women in intervention areas have relatively better access to Basic Emergency Obstetric Newborn Care (BEmONC) services. Half of the health centers in intervention areas (51%) are providing BEmONC signal functions, whereas only 28.6% of the health facilities are providing the service in non-intervention areas. Of the five primary hospitals assessed in intervention areas, two of them provide all Comprehensive Emergency Obstetric Newborn Care (CEmONC) signal functions, but no primary hospital included in the study provides CEmONC signal function in non-intervention areas. Besides, all hospitals in intervention and non-intervention areas provide delivery service for 24 hours a day and seven days a week, whereas a higher proportion of health centers in intervention areas (88.2%) are providing delivery service for 24 hours a day and seven days a week, compared with non￾intervention areas (85.7%). These might be among the contributing factors for the higher-skilled delivery rate in intervention areas. Majority of the health centers in Transform: HDR intervention areas had Magnesium Sulphate injection (68.6%) and Oxytocin injections (90.2%), but a lesser proportion of health centers in non-intervention areas had these drugs at the time of the assessment. The drugs were available in all primary hospitalsin both areas. However, the findings showed lower access to comprehensive laboratory services (VDRL, Hgb, HIV Blood group, urinalysis, and pregnancy test) in health centers in intervention areas compared with non￾intervention areas (Table 9). Table 9. Availability of Maternal Health Service in Health Centers Indicator Transform HDR Intervention Areas Non-intervention % N % N Health facilities providing BEmONC signal functions 49.0 51 28.6 14 Health facilities providing delivery service for 24 hours a day seven days a week 88.2 51 85.7 14 Health Facilities providing comprehensive laboratory service for ANC 31.4 51 57.1 14 Magnesium Sulphate (MgSO4) 50pecent in 20ml injection available 68.6 51 42.9 14 Ferrous sulphate + folic acid 92.2 51 92.9 14 Oxytocin 10 units/ml in 1 ml ampoule injection available 92.2 51 85.7 14 Other factors raised that affect maternal service utilization include the readiness of the health facilities in providing quality care in terms of having regular, uninterrupted electricity, improved water sources in the facility, and long travel distance to the health facility. These utilities are very important in providing skilled delivery services3637 . 36 Woreda KIIs and FGD from KII in Afar Mille Woreda. 37 Ethiopian Public Health Institute (EPHI)/Federal Ministry of Health and ICF International. Ethiopia Service Provision Assessment Plus-Census. Addis Ababa, Ethiopia. 2014. https://www.ephi.gov.et/images/pictures/FINAL%20draft%20SPA+%20%20REPORT%20survey%20tools%20adjusted%20- %20Dec%2024%20%202015.pdf accessed 25 November 2019. Mid-term Performance Evaluation Report USAID Transform: HDR Activity 55 b) Region Level Performance on Key Maternal Health Indicators at Mid-term There is a significant variation among the four regions on key maternal health indicators where Transform: HDR is being implemented (Figure 5). Benishangul Gumuz exhibited significant improvement in all maternal health indicators except for Iron and Folic Acid supplementation: four or more ANC visits increased by 16.1 percentage points, skilled birth attendance increased by 22.2 percentage points, and early PNC within two days increased by 29.3 percentage points, but Iron and Folic Acid supplementation declined by 14.8 percentage points. Compared with other regions, Benishangul Gumuz exhibits the highest performance in the majority of the indicators during the mid-term. The performances in Benishangul Gumuz on the majority of maternal health indicators are even higher than the MDHS findings of the region: the proportion of pregnant women who received at least four ANC visits (65.8% at mid-term versus 55.9% on 2019 MDHS); Iron and Folic Acid supplementation (21.7% versus 12.0%); and early postnatal care for the mother within two days (66.4% versus 45.3%). Overall, Somali exhibits the least performance on the majority of the maternal health indicators and is the region that is relying the most heavily on mobile health team care for MNCH services. All regions showed performance improvement in early PNC for mothers compared with the baseline values. Figure 5(A): Performance on Maternal Health Indicators in Transform: HDR Supported Regions 2019 15.8 24.6 34.5 30.2 16.5 16.9 23.7 49.7 36.1 27.2 55.1 66.3 12.1 9.6 22.6 65.8 Afar Somali Gambella Benishangul Gumuz Afar Somali Gambella Benishangul Gumuz Early initiation of ANC ANC 4+ Baseline Midterm Mid-term Performance Evaluation Report USAID Transform: HDR Activity 56 Figure 5(B): Performance on Maternal Health Indicators in Transform: HDR Supported Regions 2019 Figure 5(C): Performance on Maternal Health Indicators in Transform: HDR Supported Regions 2019 The proportion of women who received MNCH services through the mobile health team showed a significant increase in Somali and Afar (Figure 5(c)). Ensuring that there are positive health outcomes from this care is crucial, as noted above, for the Somali region. Further examination of the MNCH data showed that maternal health services, child health services, and family planning services are integral to mobile health services. However, it seems that mobile health service places more emphasis on vaccination services than maternal health services. Another reason is that vaccination acceptance may not require a complex change in the population’s views on the intervention, while ANC service is more complex. The findings showed that the vaccination really improved when mobile health services were provided (Figure 6). 40.7 16.9 21.7 22.7 17.7 5.9 11.8 36.5 48.4 28.4 27 34 8.3 6.3 20 21.7 Afar Somali Gambella Benishangul Gumuz Afar Somali Gambella Benishangul Gumuz Essential components of ANC IFA Baseline Midterm 17.7 21.4 50.9 31.8 10.5 17.6 34.2 37.1 10 5.1 31.1 11.8 44.3 54 29.6 22 47.2 66.4 27.9 34.8 Afar Somali Gambella Benishangul Gumuz Afar Somali Gambella Benishangul Gumuz Afar Somali SBA Early PNC for mothers MNCH services through mobile health Baseline Midterm Mid-term Performance Evaluation Report USAID Transform: HDR Activity 57 However, the provision of maternal health services (family planning, ANC, PNC) and other services through the mobile health team is still low and warrants further discussion. The evaluation team acknowledged the improvements in most of the maternal health indicators in Afar and Somali in the report, though the coverage is still low compared with other regions. Thus, improving the service provided through mobile health teams may help in boosting the coverage, as in the case of vaccination. Figure 6: Mobile Health Service Provided by Region & Type, 2019 c) Difference-in-Differences Analysis of Key Maternal Health Indicators The DID analysis shows that Transform: HDR significantly contributed to the improvement in SBA in intervention sites (see Table 10 for details). The proportion of skilled birth attendance increased significantly over the intervention period by 21.7 percentage points (95% CI: 11.7% – 31.7%; p<0.001), but part of this increment is due to the decline in the SBA in the non￾intervention areas (from 40.1% to 26.3%). Thus, apart from its contribution to boosting SBA, the intervention helped in mitigating the potential decline in SBA in the intervention woredas. The DID also shows that Transform: HDR significantly contributed to the increase in early postnatal care in intervention areas by 10.8 percentage points (p<0.05). Mid-term Performance Evaluation Report USAID Transform: HDR Activity 58 Table 10: Difference-in-Differences Analysis of Maternal Health Indicators, 2019 Indicators Baseline Mid-term DID Transfor m HDR Intervent ion Sites (%) Non￾interven tion Sites (%) Diff (95% CI) Transfor m HDR Interven tion Sites (%) Non￾interventi on Sites (%) Diff (95% CI) Maternal health ANC 4+ 26.4 32.9 -6.5 (-13.1; 0.2)* 36.7 41.8 -5.1 (-14.1; 3.9) 1.4 (-9.8; 12.6) SBA 26.6 40.1 -13.5 (-20.2; - 6.8)*** 34.4 26.3 8.2 (0.7; 15.6)** 21.7 (11.7; 31.7)*** Early PNC 22.4 32.8 -10.4 (-17.1; - 3.7)*** 40.2 39.9 0.3 (-7.1; 7.7) 10.8 (0.8; 20.8)** IFA 3 months 22.9 22.2 0.7 (-9.1; 10.5) 15.0 11.1 3.8 (-4.6; 12.2) 3.1 (-9.8; 16.3) Child health Early PNC newborn 23.2 35.5 -12.3 (-18.9; - 5.6)*** 38.0 36.9 1.1 (-6.3; 8.5) 13.4 (3.4; 23.4)*** Note: The proportions for the DID analysis are calculated after matching and may slightly differ from the indicator observed proportions. *** p<0.01; ** p<0.05; * p<0.1; Diff – Difference in proportions; DID – Difference-in-Differences; CI – Confidence Interval Determinants of Maternal Health Services Utilization: The following covariates (identified from a conceptual framework for maternal health service utilization) are included in the logistic regression analysis to identify the determinants of maternal health services utilization: women’s region of residence, age, educational level, paid work status, religion, the number of children ever born, the ideal number of children a woman wants to have, exposure to MNCH messages. Others include spouses accompanying women during ANC, and during delivery, women’s participation in purchasing hygiene and sanitation products; IFA intake and women’s participation in decisions regarding their health care are included as covariates (see Table 11 for details). Early ANC visits are higher in the Benishangul-Gumuz (OR=3.77; 95% CI: 1.68–8.44) and Gambella (OR=4.33; 95% CI: 1.20 – 15.66) regions compared to the Afar region. Male involvement positively contributes to early ANC initiation; women who are accompanied by their spouses for ANC visits are 5.31 times more likely to visit ANC early. Similarly, women who heard MNCH messages are 2.7 times more likely to initiate early ANC. Women in the Benishangul-Gumuz and Gambella regions are more likely to receive four or more ANC compared to women in Afar region. Women with primary education are 82% (95% CI: 1 – 127%) more likely to receive ANC 4+ visits compared to those with no education. Compared to Muslim religious followers, Protestants are 66% less Mid-term Performance Evaluation Report USAID Transform: HDR Activity 59 likely to have ANC 4+ visits. Women accompanied by their spouses are 2.15 times more likely to visit ANC 4+ than women who are not accompanied by their spouses. SBA is higher among Benishangul-Gumuz (2.09 times) and Gambella (4.33 times) region women compared to women in Afar region. Women who have 3 or 4 children are 47% less likely to have SBA than women with 1 or 2 children. Women who are accompanied by their spouses for ANC visits are more likely to have SBA than those who are not accompanied by their husbands (OR=2.53; 95% CI: 1.63 – 3.93). Table 11: Adjusted Odds Ratios (OR) & 95% CI from the Logistic Regression Model for the Determinants of Maternal Health Services Utilization in Transform: HDR Sites (2019) Indicators Early ANC (n=458; event=206) ANC 4+ (n=341; event=125) SBA (n=482; event=166) OR 95% CI OR 95% CI OR 95% CI Region Afar (Ref) 1.0 1.0 1.0 Benishangul￾Gumuz 3.77 (1.68−8.44) 7.96 (3.49−18.19) 2.09 (1.03−4.23) Gambella 4.33 (1.2−15.65) 4.94 (1.4−17.44) 4.33 (1.29−14.61) Somali 1.21 (0.65−2.27) 1.48 (0.61−3.57) 0.42 (0.21−0.84) Education No education (Ref) 1.0 1.0 1.0 Primary 1.04 0.64−1.7 1.82 1.01−3.27 1.26 0.77−2.06 Secondary + 2.15 0.91−5.09 1.34 0.51−3.56 1.56 0.69−3.54 Religion Muslim (Ref) 1.0 1.0 1.0 Orthodox 0.46 0.17−1.24 2.20 0.77−6.32 1.56 0.65−3.74 Protestant and others 0.44 0.14−1.41 0.34 0.12−0.98 0.33 0.11−1 Child ever born 1 - 2 (Ref) 1.0 3 – 4 0.53 0.31−0.9 5 – 6 0.76 0.4−1.42 7+ 0.62 0.29−1.34 The ideal number of children 0 - 3 (Ref) 1.0 1.0 1.0 4 – 5 0.53 0.16−1.83 1.27 0.4−4.04 2.82 1.00−7.86 6 – 9 0.47 0.14−1.61 1.24 0.39−3.99 2.63 0.92−7.51 10+ 0.40 0.12−1.39 0.83 0.26−2.73 2.19 0.75−6.41 Spouse accompany at ANC No (Ref) 1.0 1.0 1.0 Yes 5.31 3.34−8.45 2.15 1.22−3.79 2.53 1.63−3.93 MCH message No (Ref) 1.0 1.0 Yes 2.71 1.71−4.29 1.38 0.89−2.15 CBHI No (Ref) 1.0 1.0 1.0 Yes 1.12 0.42−3 2.2 0.77−6.35 1.21 0.5−2.92 Non-inclusion of zero in the 95% CI indicates statistical significance. Shaded cells indicate non-significant variables in the bivariate analysis. Mid-term Performance Evaluation Report USAID Transform: HDR Activity 60 Summary: Maternal Health In summary, Transform: HDR interventions positively contributed to the increase of skilled birth attendance and early postnatal care. However, more effort needs to be done to improve all maternal indicators, by giving special emphasis to four or more ANC coverages and Iron and Folic Acid supplementation for pregnant women. Moving forward, to boost maternal health service utilization, including use of mobile teams, Transform: HDR could consider interventions with more emphasis on: ▪ Promoting male involvement: covariate analysis shows that women accompanied by their spouse for ANC visits are 5.3 times more likely to initiate early ANC, 2.2 times more likely to receive four or more ANC visits, and 2.5 times more likely to receive skilled birth attendance. ▪ MNCH Messaging: Women who heard MNCH messages are 2.7 times more likely to initiate early ANC and 1.4 times more likely to receive skilled birth attendance. ▪ Providing more support for less performing regions: Women in the Gambella and Benishangul Gumuz regions are more likely to receive four or more ANC visits compared to women in Afar. Hence, intensifying interventions in the Somali and Afar regions may help to improve maternal health. ▪ Support readiness of the health facilities in providing quality care in terms of regular, uninterrupted electricity, improved water source in the facility, and long travel distance to the health facility. These utilities are very important to provide skilled delivery services. ▪ Address individual, family, cultural and structural factors to substantially increase deliveries attended by skilled attendants. Mid-term Performance Evaluation Report USAID Transform: HDR Activity 61 3.3.3 NEWBORN HEALTH a) Change in proportion of key newborn indicators in the Transform: HDR intervention sites (Baseline vs. Mid-term) compared to the change in non-Transform: HDR sites The majority of the newborn health indicators show that Transform: HDR intervention sites have shown a considerable improvement at the midpoint, compared with the baseline values: early postnatal care for the newborn (within two days of birth) significantly increased from 23.4% at baseline to 37.8% at mid-term, an increase by 14.4 percentage points (p-value<0.01). Also, early initiation of breastfeeding (within 1 hour of birth) significantly increased from 53.5% to 81.9%, an increase of 28.4 percentage points (p-value <0.01). However, the proportion of newborns who received essential newborn care (ENC) remains the same between the two periods, 8.6% at baseline versus 8.2% at the midpoint (Figure 7). Disaggregation by type of essential newborn care service shows that 50% of newborns received vitamin K, 55% TTC eye ointment, 79% skin to skin contact, and 33% received umbilical cord care with ointment applied. A similar performance trend is observed in non-Transform: HDR sites for essential newborn care and early initiation of breastfeeding. While postnatal care within two days increased significantly from baseline in intervention areas, it doesn’t show any change in non-Transform: HDR areas but did not decline since the baseline. Performance in Post-natal Care (PNC) is attributed to the cultural practice that encourages confinement, lack of mothers’ awareness, distance and scattered nature of villages and homes throughout rural areas, increased workload of HEW, not enough HEW in rural areas especially in developing regions are among the barriers to achieving postnatal care targets and results. The PNC utilization is particularly low in the pastoralist regions. The very low PNC is related to the low use of skilled birth attendance. Other barriers are related to physical distance, low maternal education, rural residence, and lack of awareness. Thus, strengthening the implementation of strategies that help mothers seek postnatal care is critical3839 . A comparison between intervention and non-intervention areas at mid-term revealed that there is no performance difference between intervention and non-intervention areas in the majority of the newborn indicators. Early postnatal care for the newborn within two days of birth is slightly lower in intervention areas (37.8%) than in non-Transform: HDR areas (38.7%), but the difference is not statistically significant (p-value>0.1). The proportion of newborns who received ENC in intervention areas (8.2%) is almost the same as non-intervention areas (7.8%). But this does not mean that there is no significant change between baseline and mid-term for intervention woredas relative to the non-intervention woredas, which will be explained later in the DID 38 RHB KIIs from Afar and Somali. 39 27 Berhe A, Bayray A, Berhe Y, Teklu A, Desta A, Araya T, et al. (2019) Determinants of postnatal care utilization in Tigray, Northern Ethiopia: A community based cross-sectional study. PLoS ONE 14(8): e0221161. https://doi.org/ 10.1371/journal.pone.0221161 Mid-term Performance Evaluation Report USAID Transform: HDR Activity 62 analysis section. However, early initiation of breastfeeding in intervention areas is significantly lower (p-value < 0.01) compared to non-intervention areas. Figure 7: Performance on Newborn Health Indicators at Baseline & Midterm Compared by Intervention and Non￾intervention Sites, 2019 While there is no performance difference between intervention and non-intervention areas in newborn health indicators, the findings of the health facility assessment show that, in the intervention areas, there are relatively better newborn health care services in the health facilities. The proportion of health posts providing community-based newborn care (CBNC) is higher (60%) in intervention areas compared with non-intervention areas (33.3%). Also, the proportion of health centers with functional newborn corner is much higher (49.0%) in intervention areas, compared with non-intervention areas (14.3%). And 68.6% of health centers in intervention areas were practicing Kangaroo Mother Care (KMC) for preterm or low birth weight babies, whereas only 50% of health centers were practicing KMC in non-intervention areas. Three out of the five primary hospitals in intervention areas provide KMC service. In addition, the use of a safe childbirth checklist was found to be better practiced in health facilities in intervention areas: 86% of the health centers in intervention areas use the safe childbirth checklist compared to 71% in non-intervention areas. b) Regional Level Performance on Key Newborn Health Indicators The regional-level performance at mid-term on key newborn health indicators shows that early postnatal care is highest in Benishangul and least in Somali. Essential newborn care is generally low in all regions, even though Somali exhibited the highest performance (19%) and Afar the least performance (2.2%) at mid-term. Afar and Benishangul Gumuz showed performance improvement in essential newborn care and early postnatal care, compared with baseline values, 23.4 37.0 8.6 7.8 53.5 55.9 37.8 38.7 8.2 7.8 81.9 90.5 Transform HDR Non-Transform HDR Transform HDR Non-Transform HDR Transform HDR Non-Transform HDR Early PNC for newborns ENC Early initiation of breastfeeding Baseline Midterm Mid-term Performance Evaluation Report USAID Transform: HDR Activity 63 but exhibited declining performance in both indicators. The rates of early initiation of breastfeeding did not appear to vary by region. (Figure 8). Figure 8: Performance on Newborn Health Indicators by Region at Mid-term, 2019 c) Difference-in-Differences (DID) Analysis on Key Newborn Health Indicator The findings of the DID analysis show that Transform: HDR has significantly contributed to improving the utilization of early postnatal care service for newborns within two days by 13.4 percentage points (Table 12). Table 12: Difference-in-Difference Analysis of Newborn Health Indicators, 2019 Indicators Baseline Mid-term DID Transfor m HDR Interventi on Sites (%) Non￾Transfor m HDR Sites (%) Diff (95% CI) Transfor m HDR Interventi on Sites (%) Non￾Transfor m HDR Sites (%) Diff (95% CI) Child health Early PNC newborn 23.2 35.5 -12.3 (-18.9; - 5.6)*** 38.0 36.9 1.1 (-6.3; 8.5) 13.4 (3.4; 23.4)*** Note: The proportions for the DID analysis are calculated after matching and may slightly differ from the indicator observed proportions. *** p<0.01; ** p<0.05; * p<0.1; Diff – Difference in proportions; DID – Difference-in-Differences; CI – Confidence Interval 12.3 22.1 36.8 30.5 0 25 13.3 0 52.3 35.4 82.6 48.8 29.6 20.5 38.7 66.4 2.2 19 8.5 8.6 87.9 77.7 83.8 77.7 Afar Somali Gambella Benishangul Gumuz Afar Somali Gambella Benishangul Gumuz Afar Somali Gambella Benishangul Gumuz Early PNC for newborns ENC Early initiation of breastfeeding Baseline Midterm Mid-term Performance Evaluation Report USAID Transform: HDR Activity 64 3.3.4 CHILD HEALTH a) Change in proportion of key child indicators in the Transform: HDR intervention sites (Baseline vs. Mid-term) compared to the change in non-Transform: HDR sites The mid-term evaluation findings demonstrated that many of the key child health indicators had shown improvements in Transform: HDR intervention areas compared with the baseline values. Almost all immunization indicators increased in performance between baseline and mid-term (Figure 9). Measles coverage shows a significant increase by 23.5 percentage points (from 28.1% at baseline to 51.6% at mid-term) (p-value<0.01). Similarly, Penta 3 coverage significantly increased by 8.7 percentage points (from 31.3% at baseline to 40% at mid-term) (p-value <0.01). Full immunization also increased by 5.2 percentage points (from 27.6% at baseline to 32.8% at mid-term), though it is not statistically significant (p-value>0.05). Figure 9: Performance on Child Immunization Indicators at Baseline & Mid-term Compared by Intervention and Non-intervention Sites, 2019 Other child health indicators also showed significant improvement from the baseline: ARI treatment with antibiotics increased significantly by 57.6 percentage points (from 16.8% at baseline to 74.4% at mid-term, p-value < 0.01); the proportion of children 24-59 months who were given drugs for intestinal worms in the last six months of the survey shows a significant increase from 27.4% at baseline to 33.7% at mid-term (by 6.3 percentage points, p-value < 0.01); and percentage of children under five years who slept under ITN in the previous night of the survey significantly increased from 53.9% at baseline to 68.7% at mid-term (a 14.8 percentage points increase, p-value<0.01). However, the proportion of children who had diarrhea and were treated with ORS and Zinc did not change from the baseline (28.2% at baseline and 26.2% at mid￾term) even though diarrhea incidence had shown a significant increase by 4.6 percentage points 27.6 27.6 28.1 28.1 31.3 31.3 32.8 32.4 51.6 55.1 40.0 47.2 Transform HDR Non-Transform HDR Transform HDR Non-Transform HDR Transform HDR Non-Transform HDR Full Immunization Measles Penta 3 Baseline Midterm Mid-term Performance Evaluation Report USAID Transform: HDR Activity 65 (p < 0.01) (Figure 10). Findings from the health facilities assessment show that there were frequent stock-outs of ORS and Zinc at health facilities in Transform: HDR intervention sites, which might contribute to the low performance in threatening children with diarrhea. Only 13.5% of health facilities in Transform: HDR intervention sites had both ORS and Zinc at the time of the health facility survey. A similar performance trend was observed in non-Transform: HDR sites in immunization and other child health indicators between the baseline and mid-term where there was a significant increase in ARI treatment with antibiotics, ITN use by under-five children, and children who are given a drug for intestinal worms. Figure 10(a): Performance on Child Health Indicators at Baseline & Mid-term Compared by Intervention & Non-intervention Sites, 2019 Figure 10(b): Performance on Child Health Indicators at Baseline & Mid-term Compared by Intervention & Non-intervention Sites, 2019 6.3 5.8 16.8 19.0 28.2 33.9 5.8 4.4 74.4 71.0 26.2 33.7 Transform HDR Non-Transform HDR Transform HDR Non-Transform HDR Transform HDR Non-Transform HDR Symptoms of ARI ARI treated with antibiotics Treatment of diarrhea with ORS and Zinc Baseline Midterm 53.9 49.0 27.4 24.1 68.7 62.2 33.7 38.8 Transform HDR Non-Transform HDR Transform HDR Non-Transform HDR ITN use among <5 children Deworming Baseline Midterm Mid-term Performance Evaluation Report USAID Transform: HDR Activity 66 b) Regional Level Performance on Immunization The regional-level performance at the mid-term on key child health indicators shows that there is considerable variation among the four regions in terms of performance on child health indicators. Immunization indicators for the Benishangul-Gumuz region were higher for Penta 3 coverage (67.9%) and full immunization (56%) while the Gambella region had the highest measles coverage levels (65.5%). Performances in the Afar and Somali regions are relatively lower for all three immunization indicators (Figure 11). Figure 11: Immunization Performances by Region, Mid-term (2019) A comparison of child health indicators between intervention and non-intervention areas at the mid-term shows that Transform: HDR intervention areas performed better only in a few indicators (Figure 12). Children under five years who slept under ITN is significantly higher in intervention areas (68.7%) compared with non-intervention areas (62.2%) (p-value<0.01). The proportion of under-5 children with symptoms of ARI and treated with antibiotics is slightly higher in intervention areas (74.4%) compared with non-intervention areas (71.0%), although it is not statistically significant (p-value = 0.7). Full immunization coverage is almost the same in the intervention and non-intervention areas (32.8% and 32.4%, respectively). Performance on all other child health indicators is, however, lower in Transform: HDR intervention sites compared with non-intervention sites. RHB KIIs from Afar, Somali, Gambella, and Benishangul-Gumuz all attributed the observed performance to include: ….. Parental lack of or low levels of education, poor awareness of immunization, the distance between their home and health facilities, lack of antenatal care, and home delivery. Religious 9.9 20.4 30.4 61.4 15.4 29.3 34.0 41.6 8.7 27.1 39.9 65.0 17.3 26.1 36.2 56 42.1 51.1 65.5 56.1 22.6 32.6 41.4 67.9 Afar Somali Gambella Benishangul Gumuz Afar Somali Gambella Benishangul Gumuz Afar Somali Gambella Benishangul Gumuz Full Immunization Measles Penta 3 Baseline Midterm Mid-term Performance Evaluation Report USAID Transform: HDR Activity 67 beliefs and fear of the side effects of vaccinations result in low vaccination coverage or untimely vaccinations, which result in low immunization coverage40 ……. Other factors include sustainable funding, vaccine stock-outs, and logistics problems, data issues, and weak laboratory infrastructure. 41 ” c) Regional Level Performance on Key Child Health Indicators Gambella had the highest rates of coverage for children under five who slept under ITN the previous night (75.6%) at mid-term but had the lowest level of children with symptoms of ARI (14.2%) and diarrhea incidence (29.0%). The change in performance between baseline and mid￾term in the proportion of children with symptoms of ARI and diarrhea incidence is highest in Gambella (by 6.4 and 11.1 percentage points, respectively). Children with symptoms of ARI treated using antibiotics at mid-term are highest in Afar (92.3 %) and relatively high in the use of ORS and Zinc to treat diarrhea (35.1%) next to Benishangul-Gumuz (35.6%). The performance change for these indicators between baseline and mid-term is also highest in Afar - treatment of ARI with antibiotics (by 75.6 percentage points) and diarrhea treatment with ORS and Zinc (by 12.3 percentage points). The highest level of use for treating children with a deworming medication for intestinal worms at mid-term (77.6%) as well as the greatest change in performance between baseline and mid-term (by 19.5 percentage points) for this indicator is observed in Benishangul-Gumuz (Figure 12). 40 RHB KIIs from Afar, Somali, Gambella and Benishangul-Gumuz 41 Richard Mihigo et al. Challenges of immunization in the African region. The Pan African Medical Journal. 2017;27 (Supp 3):12. doi:10.11604/pamj.supp.2017.27.3.12127 Mid-term Performance Evaluation Report USAID Transform: HDR Activity 68 Figure 12: Child Health Performances Indicators of Intervention Areas between Baseline and Mid-term, Transform: HDR Regions, 2019 d) Difference-in-Differences Analysis of Key Child Health Indicators The DID analysis shows that, although there are positive changes in the performance of the majority of the child health indicators, Transform: HDR’s contribution to the change is not statistically significant (Table 13). 3.7 4.77.89.916.7 0.0 17.2 25.6 10.6 5.1 17.9 16.2 22.8 33.3 32.9 26.5 2.2 3.7 14.2 7.4 92.3 64.0 69.6 81.1 16.2 9.7 29.0 18.1 35.1 15.4 16.0 35.6 Afar Somali Gambella Benishangul-Gumuz Afar Somali Gambella Benishangul-Gumuz Afar Somali Gambella Benishangul-Gumuz Afar Somali Gambella Benishangul-Gumuz Symptoms of ARI ARI treated with antibiotics Diarrhea incidence Treatment of diarrhea with ORS and Zinc Baseline Midterm 28.8 35.3 77.3 84.6 20.7 3.5 31.1 58.1 74.1 27.7 77.4 75.7 56.9 72.8 75.6 72.7 24.4 15.7 25.9 77.6 41.3 28.8 74.4 80.4 Afar Somali Gambella Benishangul-Gumuz Afar Somali Gambella Benishangul-Gumuz Afar Somali Gambella Benishangul-Gumuz Children under 5 years who slept under ITN Deworming (2-5yrs) Exclusive breastfeeding Mid-term Performance Evaluation Report USAID Transform: HDR Activity 69 Table 13: Difference-in-Differences Analysis of Child Health Indicators, 2019 Indicators Baseline Midterm DID Interventi on Sites (%) Non￾interventi on Sites (%) Diff (95% CI) Interventi on Sites (%) Non￾interventi on Sites (%) Diff (95% CI) Child health ARI symptom 7.8 6.9 0.8 (-1.6; 3.1) 5.8 4.2 1.6 (-0.2; 3.3)* 0.7 (- 2.2; 3.6) ARI treatment 17.7 22.1 -4.4 (-21.4; 12.6) 74.4 67.4 7.0 (-7.3; 21.3) 11.3 (- 10.8; 33.2) Diarrhea incidence 14.7 18.8 -4.0 (-7.5; - 0.5)** 16.5 14.1 2.5 (-0.2; 5.2)* 6.5 (2.0; 11.0)** * Diarrhea treatment 28.1 27 1.1 (-10.3; 12.4) 26.2 34 -7.8 (-16.4; 0.8)* -8.9 (- 23.2; 5.4) Full immunization 26.7 29 -2.3 (-10.9; 6.3) 32.8 30.7 2.1 (-6.9; 11.1) 4.4 (- 7.9; 16.7) ITN use among < 5 children 56.2 49.6 6.6 (1.9; 11.3)*** 68.7 60.3 8.4 (4.7; 12.1)*** 1.7 (- 4.2; 7.6) Note: The proportions for the DID analysis are calculated after matching and may slightly differ from the indicator observed proportions. *** p<0.01; ** p<0.05; * p<0.1; Diff – Difference in proportions; DID – Difference-in-Differences; CI – Confidence Interval Mid-term Performance Evaluation Report USAID Transform: HDR Activity 70 3.3.5 SOCIAL BEHAVIOR CHANGE AND COMMUNICATION (SBCC) a) Change in proportion of key SBCC indicators in the Transform: HDR intervention sites (Baseline vs. Mid-term) compared to non-Transform: HDR areas Two indicators were identified to assess the performance on MNCH and FP/RH messaging for Transform: HDR Activity: percentage of audience who recall hearing or seeing FP/RH message, and percentage of the audience who recall hearing or seeing MNCH/FP message. The findings showed that the performance on MNCH messaging significantly increased from 19.7% at baseline to 34.8% at mid-term (by 15.1 percentage points, p-value<0.01) in Transform: HDR intervention Summary of Child Health While Transform: HDR has clearly supported some improvements on child health indicators with a marked improvement in some regions, the improvements thus far are not consistent across all regions. The performance improvements in most of the child health indicators are basically the same in Transform: HDR intervention and non-Transform: HDR areas. To boost performance, Transform: HDR could consider interventions with more emphasis on: ▪ Intensifying support targeting reducing incidence of diarrhea and case management. ▪ Child Health Messaging: women who heard MCH messages are 1.7 and 2.4 times more likely to use ORS and Zinc for diarrhea treatment and use antibiotics for ARI treatment, respectively. ▪ Improving availability of ORS and Zinc drugs: Support improving availability of ORS and Zinc at HPs and HCs is important to improve diarrhea treatment. ▪ Mobile health service: It may require assessing the effectiveness of Transform: HDR’s support to mobile health services in order to enhance the overall child health service outcomes. ▪ SBBC messaging should be strengthened to enhance parental education, poor awareness of immunization, improve attendance of antenatal care, and reduce home delivery. Religious beliefs and fear of the side effects of vaccinations result in low vaccination coverage or untimely vaccinations, which result in low immunization coverage. ▪ Other structural factors that need to be addressed, include: sustainable funding, vaccine stock-outs and logistics problems, data issues and weak laboratory infrastructure Mid-term Performance Evaluation Report USAID Transform: HDR Activity 71 areas, whereas FP/RH messaging declined significantly by 10.3 percentage points between the two periods (from 48.1% at baseline to 37.8% at mid-term, p-value<0.01). A similar performance trend is observed in non-Transform: HDR areas in both indicators (Figure 13). Figure 13. Performance on key SBCC indicators between Baseline and Midterm in Transform: HDR intervention areas and Non-Transform: HDR areas. In some regions, it seems there are strongly religious and cultural barriers against conveying family planning messages in the community, which might have contributed to the declining performance in FP/RH messaging. “People come and ask whether health workers have been providing us messages on family planning methods and birth spacing, just like you are doing now; otherwise, no health care provider in this area came and talked to us about family planning… Even if they try to teach us, no one will accept them; rather, we will beat them. We do not like it” …. (Afar, male, 35 years old FGD participants) Looking into the sources of information, the findings showed that health workers were the main source of information for MNCH and FP/RH messaging in Transform: HDR intervention areas at mid-term (Figure 14). Majority of the women (84.5%) reported that they had heard the message from health workers, 34.7% heard during a community event, and 18.9% heard from radio episodes. 48.1 52.3 19.7 24.0 37.8 33.2 34.8 33.9 Transform HDR Non-Transform HDR Transform HDR Non-Transform HDR FP/RH Messaging MNCH Messaging Baseline Midterm Mid-term Performance Evaluation Report USAID Transform: HDR Activity 72 Figure 14. MNCH/FP messaging by the source of information in Transform: HDR intervention areas in Mid-term. b) Region Level Performance on Key SBCC Indicators at Mid-term The regional level disaggregation for these indicators showed that there is a noticeable variation in MNCH and FP/RH messaging in Transform: HDR intervention regions. All regions showed improvement in MNCH and FP/RH messaging, where the greatest improvement was shown in Somali (by 22.3 percentage points), followed by Afar (by 20.5 percentage points). On the contrary, the performance on FP/RH messaging declined in most of the regions: Somali, Gambella, and Benishangul Gumuz showed a declining performance on FP/RH messaging from the baseline, whereas the performance in Afar showed a slight improvement. The highest decline was observed in Benishangul Gumuz. Figure 15: Region level Performance on Key SBCC Indicators at Mid-term, 2019 18.9 6.2 1.5 .7 39.5 85.1 .7 .7 Radio Television Newspaper Pamphlet Community Event Health Workers School Other sources 41.4 42.9 43.2 66.1 18.9 16.6 19.1 24.4 44.6 33.1 29.3 43.5 39.4 38.9 27.8 33.0 Afar Somali Gambella Benishangul Gumuz Afar Somali Gambella Benishangul Gumuz FP/RH Messaging MNCH Messaging Baseline Midterm Mid-term Performance Evaluation Report USAID Transform: HDR Activity 73 3.3.6CROSS-CUTTING - WOMEN DECISION-MAKING AND MALE INVOLVEMENT a) Change in Proportion of Key Gender-related Indicators in the Transform: HDR Intervention Sites (Baseline vs. Mid-term) Compared to Non-Transform: HDR Sites The performance of Transform: HDR on gender-related indicators is illustrated in Figure 16. The data show that gender indicators in Transform: HDR intervention areas have not changed much compared with baseline values. Women accompanied by their spouses during delivery for their last birth shows a slight increase, 78.2% at baseline to 79.5% at mid-term. An area that has not improved is women's participation in decisions regarding their health. The levels, in fact, appear to have significantly declined, from 82.3% at baseline to 79.3% at mid-term (by 3.0 percentage points, p-value= 0.010). The proportion of women accompanied by their spouse during ANC visits for their last birth also declined slightly from 44.4% at baseline to 40.8% at mid-term (by 3.6 percentage points), though it is not statistically significant (p-value= 0.135). Therefore, it may require exploring new approaches that enhance male involvement in the regions. Figure 16: Women's Decision-Making & Male Involvement Indicators’ Performances between Baseline (2017) vs. Mid-term (2019 and between Intervention and Non-intervention areas). Moreover, it does not appear that there is any difference in women’s decision-making for health care between intervention and non-intervention sites at mid-term. (Figure 16). Women's participation in decisions regarding their health in the intervention areas is only slightly higher than the performance in non-intervention areas (79.3% versus 78.0%, p-value= 0.523). Women who were accompanied by their spouses during delivery for their last birth tend to be similar in both the intervention and non-intervention areas (79.5% versus 79.7%, p=0.973). The proportion of women accompanied by their spouse during ANC visits for their last birth is slightly lower in the intervention areas as compared to the non-intervention areas (40.8% versus 46.7%). 82.3 44.4 79.3 78.2 40.8 80.1 79.5 52.3 72.978.0 46.7 79.7 Women's participation in decisions regarding their own health Women accompanied by their spouses during ANC Women accompanied by their spouses during delivery Intervention sites (Baseline) Intervention sites (Midterm) Non Intervention sites (Baseline) Non Intervention sites (Midterm) Mid-term Performance Evaluation Report USAID Transform: HDR Activity 74 b) Regional Comparison of Gender-related Indicators Performance in Transform: HDR Intervention Sites at Mid-term Regional comparisons of gender-related indicators for women’s participation in decisions regarding their health is highest in the Somali region (89.1%). Men in the Benishangul-Gumuz region achieved the highest rates for accompanying their spouses during one or more of their ANC visits (62.8%) and delivery for their last birth (86.4%) (see Figure 17 for details). Figure 18: Gender-related Indicators’ Performances in Transform: HDR Intervention Sites, Mid-term (2019) Transform: HDR accomplished some gender-related activities. Efforts were made in conducting gender analysis that informed the development of gender strategy to implement associated recommendations. For example, Transform: HDR initiated a one-stop Post-Gender-Based Violence (GBV) center at Dubti Hospital in the Afar region. The initiative has generated useful insight that may help women to get holistic service/support but offered an opportunity for the different sectors to work together to improve women’s roles in developing programs and benefit from available services. Previously, gender-based violence in Afar region was underreported. For instance, women who ever experienced sexual violence in Afar region were only 3.1% in the 2016 EDHS. After the establishment of the one-stop GBV center at Dubti Hospital in Afar, GBV service-seeking behavior has improved. About 40 people visited the hospital after the hospital resumed GBV services. Except for one, the others who received the services were women. The magnitude of the problem in the region can be observed clearly once more people are aware of the services provided by Dubti Hospital and when other GBV service providers are established closer to the community. The package of services in the one-stop center should consider local contexts of gender dynamics. What works for Afar may not necessarily work for Somali. In the Somali region, an effort to adapt Afar’s experience did not contextualize realities on the ground. Mid-term Performance Evaluation Report USAID Transform: HDR Activity 75 c) The Difference-in-Differences Analysis of Gender Indicators, 2019 The contribution of the Transform: HDR interventions in increasing women’s decisions on healthcare are not significant (see Table 14). Table 14: Difference in Proportions & DID of Gender Indicator Between Baseline & Mid-term Indicators Baseline (%) Mid-term (%) DID Interventio n Sites (%) Non￾Intervention Diff (95% CI) Interventio n Sites Non￾interventio n Diff (95% CI) Women’s Decisions on Healthcare 82.1 79.3 2.8 (-0.3; 5.9)* 79.2 78.6 0.6 (-2.7; 3.9) -2.2 (-6.9; 2.5) Note: The proportions for the DID analysis are calculated after matching and may slightly differ from the indicator observed proportions. *** p<0.01; ** p<0.05; * p<0.1; Diff – Difference in proportions; DID – Difference-in-Differences; CI – Confidence Interval Gender and Socio-cultural Barrier Assessment42: The Transform: HDR Gender and Socio-cultural barrier assessment revealed that there are some cultural norms embedded in the developing regions such as female genital mutilation (FGM), gender-based violence against women (GBV) and lack of willingness to accept family planning interventions. This calls for an intervention that focuses on influencing and changing the behaviors of the community and contributing to the use of MCH services. Considering the establishment/strengthening of locally acceptable high impact community awareness forums using local CBOs, religious or clan leaders may significantly contribute to minimizing some of the myths and increasing service utilization. Gender Transformative Intervention: This is a long-term intervention which should aim at decreasing or eliminating the gender imbalance and inequality that is deeply rooted in the developing regions’ communities at a different level in each region. While this activity was found to be committed to gender equality and strives for equal outcomes for women and girls, and men and boys, to narrow the gender gap, Transform: HDR should ensure that all programming is designed to address the unique needs and mitigate barriers faced by women and men, boys and girls to access service. 42 Transform: HDR Gender and Socio-cultural barrier assessment report, 2018 Mid-term Performance Evaluation Report USAID Transform: HDR Activity 76 3.3.7 TRANSFORM: HDR ACTIVITY PERFORMANCE ON SET TARGETS AT MIDTERM The evaluation analyzed Transform: HDR’s performance progress against key MNCH/FP outcome indicators at the mid-term to determine the likelihood of achieving its life of project (LOP) targets. Per USAID’s standard and recommendations + or – 10% of the target as an acceptable margin. Anything beyond 110%, requires a revision of target or halting of implementation as per other criteria and anything below 90% can further be divided into two - more effort or target revision Overall, Transform: HDR is on track to reach the majority of the indicators. There are, however, a few indicators (see Table 15 below) that need special emphasis to meet the life of project targets. Summary of Gender In Transform: HDR intervention areas, there is a positive development in men accompanying their spouses during deliveries. However, a deeper exploration needs to be done on why more women are not able to make a decision to seek health care and what is impeding them from doing so. - Engage the non-health community leaders more effectively so they understand the benefits of allowing women to make more health decisions and explore regional innovations to catalyze this initiative. - Once discussions take place with USAID/Ethiopia, there may be a need to revise targets for gender indicators: Transform: HDR may need to revise upward its gender targets, especially for the target for women decision￾making for their own health and male accompaniment for ANC. - Mid-term Performance Evaluation Report USAID Transform: HDR Activity 77 Table 15: Transform: HDR Activity Performance on Set Targets at Mid-term Thematic Area KPI Activity Target at Mid-term (%) A Performanc e at Mid￾term (%) B % Achieved (B-A)/A LOP Targets Likelihood of Achieving LOP Targets Family Planning MCPR 21.5 21.8 101% 23.3 On track- It is within the acceptable performance margin. More likely to achieve the LOP Target PPFP 13.1 16.2 124% 14.1 Achieved above 110%. Requires revision of target (upward) Maternal and Newborn Health SBA 31.0 34.3 111% 40.3 Achieved above 110%. Requires revision of target (upward) ANC 4+ 31.8 26.4 83% 49.6 Achieved below the acceptable margin- More effort required or target revision Early PNC for mothers 24.0 40.1 167% 40.3 Achieved above 110%. Requires revision of target (upward) Iron and Folic Acid (IFA) 39.9 15.0 38% 51.1 Achieved far below the acceptable margin- More effort required or target revision MNCH services through mobile health 6.4 24.8 287% 6.4 Achieved above 110%. Requires revision of target (upward) Early PNC for newborn 25.8 37.8 146% 33.1 Achieved above 110%. Requires revision of target (upward) Child Health Full Immunization 31 32.8 106% 40.4 On track- It is within the acceptable performance margin. More likely to achieve the LOP Target Measles 28.9 51.6 178% 32.0 Achieved above 110%. Requires revision of target (upward) Penta 3 35.2 40.0 113% 46.7 Achieved above 110%. Requires revision of target (upward) Symptoms of ARI 5.2 5.8 112% 8 Achieved above 110%. Requires revision of target (upward) ARI treated with antibiotics 21.5 74.4 246% 32.7 Achieved above 110%. Requires revision of target (upward) Treatment of diarrhea with ORS and Zinc 30.0 26.2 87% 36.8 Achieved below the acceptable margin- More effort required or target revision Cross-cutting: Gender Women's decisions regarding their health 86.4 79.3 92% 90.1 On track- It is within the acceptable performance margin. More likely to achieve the LOP Target Women accompanied by the spouse during ANC visits 48.8 40.8 84% 55.1 Achieved below the acceptable margin- More effort required or target revision Women accompanied by a spouse during delivery 65.3 79.5 122% 71.4 Achieved above 110%. Requires revision of target (upward) Mid-term Performance Evaluation Report USAID Transform: HDR Activity 78 3.3.8 ANALYSIS OF HEALTH FACILITIES READINESS AND PRACTICE IN PREVENTION AND MANAGEMENT OF POST-PARTUM HEMORRHAGE (PPH) Post-partum Hemorrhage (PPH) is commonly defined as a blood loss of 500 ml or more within 24 hours after birth. PPH is the leading cause of maternal mortality in low-income countries and the primary cause of nearly one-quarter of all maternal deaths globally43 . Based on the request from USAID/Ethiopia, the evaluation assessed health facilities readiness, practice in prevention and management of PPH cases including the use of a childbirth checklist, providing Basic Emergency Obstetric and Newborn Care (BEmONC) signal functions, presence of Maternal Death Review Committees, Availability of Oxytocin, availability of ferrous sulfate and folic acid, and women’s stay in HFs for at least 24 hours after delivery. In assessing facilities readiness in the prevention and management of PPH cases, the findings are presented below and in Table 16: • A total of 23 out of 56 (41%) of the health facilities reported that they had at least one PPH case in the last three months, including referrals. A total of 50 PPH cases (both treated and referred) were reported from these facilities in the last three months preceding conducting the assessment. • A safe childbirth checklist is used by 86% of health facilities surveyed, both Health Centers and Primary Hospitals. • 49% of Health Centers reported that they are providing all BEmONC signal functions. The components of the BEmONC signal functions provided in the health centers were: administration of parenteral uterotonics (in 72% of the HCs), administration of Anticonvulsant drug (75% of HCs), manual removal of placenta (in 77% of the HCs), removal of retained product (86% of the HCs), assisted vaginal delivery (90% of the HCs), and newborn resuscitation (in 88% of the HCs). • The availability of Oxytocin in health centers and hospitals was 93%, where 12% of them had stock-outs in the last six months for at least one week (7 days). In the absence of Oxytocin, health facilities reported that they use misoprostol, ergometrine, or refer the mother to a higher level of care. • Half of the women (51%), who had a live birth at health facilities in the last year before the survey, stayed in health facilities for at least 24 hours after delivery; 44% and 64% of women delivered at health centers and hospitals, respectively. • Iron and folic acid supplements were available in 83% of health facilities44. However, the proportion of women who took iron tablets for at least three months during their last pregnancy was only 15%. • Maternal Death Review Committees were present in 54% of health facilities, and most (87%) of them regularly meet and conduct death audits. 43 WHO recommendations for the prevention and treatment of postpartum hemorrhage 44 Primary hospitals, health centers and health posts Mid-term Performance Evaluation Report USAID Transform: HDR Activity 79 Table 16: Health Facilities (HF) Readiness, Practice in Prevention & Management of PPH Cases Indicators % N 1 The proportion of health facilities which are using safe childbirth checklist 85.7 56 2 The proportion of health centers providing BEmONC signal functions 49 52 3 Proportion facilities with the presence of maternal death review committees 54 52 4 The proportion of health facilities where Oxytocin45 is available 91 56 5 Availability of Ferrous sulfate and folic acid at Health centers and Primary Hospitals 83 97 6 The proportion of women who took iron tablets for at least three months during their last pregnancy 14 247 7 The proportion of women who stayed in HFs for at least 24 hours after delivery 51 168 8 The proportion of HFs reporting at least one PPH case in the last three months (including referrals). 41 56 Primary hospitals and health centers in Transform: HDR regions reported challenges related to the management of PPH, including lack of skilled health care providers, stock-out of uterotonic drugs, lack of equipment for monitoring PPH, road accessibility coupled with security problems, lack of ambulance services and transportation, shortage of blood in hospitals, and shortage of infrastructures such as water supply and electricity. Community awareness of danger signs and complications of the post-partum period and a higher proportion of home deliveries are also mentioned as critical challenges in the management and prevention of PPH. Thus, to improve health facilities’ readiness and ability to manage PPH cases, interventions should focus on increasing the supply of skilled health service providers, devising context-specific interventions and messaging that motivates mothers to deliver at health facilities, take iron tablets and improve referral linkages. Summary of Delivery Methods & Programming Approaches Appreciated by Stakeholders This evaluation also assessed the effectiveness of delivery methods and programming approaches to the Transform: HDR Activity. Some of the approaches appreciated by stakeholders include: ▪ Transform: HDR has prioritized supporting a comprehensive service package in fewer woredas (20) and hence establishing learning centers of excellence for all supported regions. ▪ The medical equipment purchased through Transform: HDR, such as the portable V-scan ultrasound and BEMoC supplies, enabled the supported woredas to increase demand and 45 Uterotonic drug is a life-saving drug which needs to be available all times. 91% availability shows there is stockouts, health care providers mentioned this as a challenge. Mid-term Performance Evaluation Report USAID Transform: HDR Activity 80 utilization of ANC1 to ANC4 follow-up. The provision of medical technologies improved case detection and increased client flow. This support by USAID/Ethiopia was highly appreciated and could be scaled up. ▪ GPS mapping is helping beyond health: Transform: HDR conducted GIS assisted mobility assessments for Afar and Somali pastoralist communities. Mapping pastoral communities’ mobility routes and destinations together with their existing facilities, has helped enhance the availability and accessibility of health services and other basic services in pastoral regions. This mapping is particularly relevant for the implementation of mobile health services during the seasonal movement of pastoral communities through the identification of routes and destinations. The result of the study, however, showed water points, grazing lands, animal health, human health points, schools, market-place, etc., are vital information for the sector offices other than health. This strategy should be scaled up and conducted routinely. ▪ The Outreach Model, a one-stop service/center for SGVB, and the reinforced ANC labs have made important and scalable contributions to improving the quality of care interventions implemented by Transform: HDR. ▪ The support to increase the use of community scorecards to enhance health governance and accountability was also found to hold providers accountable for improving health care quality. Moving forward, more emphasis should be placed on supporting the government to step up community health governance and link it to more resource mobilization. Transform: HDR has a strong post-training follow-up mechanism for health workers. Capacity building training followed by post-training follow-up and on-site mentorship and coaching improved service quality (RMNCH) with EPS, ESOG, EMWA, which has been one of the approaches appreciated to be successful and should be scaled-up. “Transform: HDR provides supplies and medical equipment to some of the public health facilities in the operation woredas including drugs. Transform: HDR is efficient in post-training to follow up to ensure the outcomes of the training, unlike other partners. On top of that, we have an effective training tracking mechanism to identify trainees and to address equity. Transform: HDR project is also pioneering in introducing safe childbirth checklist in the region which contributes a lot in achieving maternal and neonatal goals” --Key informant at Benishangul Gumuz. “Transform: HDR unlike other partners operating in the region is found to be unique in its approaches as it is: (1) providing grass root level supervision and coaching, (2) provision of relevant training to fill perceived knowledge gaps of health care workers, (3) being flexible based on the demand of the woreda & (4) perform multiple activities. The capacity building activities were said to be an effective intervention followed by supply of equipment to selected health facilities. -- Key informant at Afar RHB Mid-term Performance Evaluation Report USAID Transform: HDR Activity 81 • Integrated and comprehensive outreach health services have proved beneficial in reaching and delivering the required health services to the harder to reach areas. • Random and routine follow-ups were adopted from Transform: PHC to identify and address gaps and are being used to make adjustments in intervention, providing targeted support and mentoring. At the end of each random follow-up visit, the Transform: HDR team holds a brief meeting with the regional and woreda stakeholders to reflect on key findings, identify areas that need to be followed by HP, HC, WoHO, ZHD and RHB, and develop action plans. • The Quality Improvement Initiative at 31 health facilities resulted in the RHB, adding a quality assurance focal person to lead health performance teams who review compliance with service standards and quality of patient care. • ANC lab quality/EQA malaria/QI initiatives – Transform: HDR implemented Quality Improvement projects. The results were shared nationally and internationally, and the gap in laboratory services for ANC, in particular, and whole service, in general, remains a challenge. There is marked improvement in services uptake in the 24 CoE health facilities. This is an initiative that should be scaled up. 4. SUSTAINABILITY ELEMENTS Transform: HDR has integrated key elements of sustainability into its programming, including: ▪ Woreda Transformation: Transform: HDR supports government priorities for Woreda Transformation. Transform: HDR implementation modalities are integral and part of the government health sector (no parallel implementation structure). This promotes self￾reliance and the sustainability of MNCH/FP interventions and enhances Woreda Transformation. ▪ Technical Capacity: providing key RMNCH/FP training packages and ensuring consistent implementation of the packages through regular mentoring and supportive supervision by skilled mentors. ▪ Institutional Capacity support: Transform HDR supportsthe RHB and its structuresto enhance their operational and leadership capacity through Leadership, Governance, and Management (LMG) training. ▪ Coaching and Mentorship support using the public system structures – Woreda level interventions, either at woreda health offices or at primary hospitals, are important to sustaining gains on service delivery and program management. Catchment based clinical mentorship, leadership development coaches, public finance management mentorship, connected woreda mentorship, and quality improvement mentorship are all about establishing and building sustainable capacity at the RHBs and woreda level. ▪ Monitoring and Evaluation: the support to roll out the community scorecards and engagement to monitor/provide oversight for the quality of care and customer satisfaction is positive, yet much more needs to be done. Transform: HDR support to the DHIS2 has the Mid-term Performance Evaluation Report USAID Transform: HDR Activity 82 potential to strengthen the use of performance information management decision-making for performance improvements and resource allocation and hence supporting woreda transformation. RHB and woreda KIIs had this to say with regards to sustainability: “To us, sustainability included strategies, tools, or approaches which are being implemented or designed to ensure the sustainability of Transform: HDR interventions/initiatives. An example included integration with woreda transformation, community participation, system strengthening, capacity building, working with other stakeholders, etc.” KII from Benishangul, RHB summed up what other KIIs views are by saying: “….. In general, Transform: HDR project design has been developed by focusing on the government priorities and has worked on the existing government systems and structures, with the government sectors. The objectives of the project are definitely in line with government priorities. The project has worked on creating quality MNCH/FM services through strengthening the existing systems. Owing to these reasons, the project results will lead to Woreda Transformation, which is sustainability”. Overall, sustainability remains a challenge in developing regions as the health system is not strong, and there is limited budgetary and resource mobilization capacity. USAID/Ethiopia and the Transform: HDR should discuss and outline the goals/strategies for sustainability in the current budget and resource-scarce climate. 5. LESSONS LEARNED Key lessons learned and good practices gleaned during the implementation of the Transform: HDR Activity includes: ▪ Comprehensive and integrated mobile and outreach (MNCH/FP +ANC lab) is good to fill gaps in service provision and attract new users. ▪ The capacity building pays off if supported with post-training follow-up, mentoring, and coaching. ▪ The weak health system and health infrastructure in the developing regions require extra effort and resources for improvement. ▪ Ownership and engagement of government structures at all levels(region, zone, woreda, and health facility) are critical to the effective delivery of results. ▪ Men could play a significant role in MNCH/FP service uptake, but thus far, the activity may not have fully employed strategies to attract men. ▪ Engaging local universities in operations research (OR) and other assessments may ensure the acceptance and validity of research findings. ▪ GPS mapping is helping beyond health: Transform: HDR conducted GIS assisted mobility assessments for Afar and Somali pastoralist communities. Mapping pastoral communities’ Mid-term Performance Evaluation Report USAID Transform: HDR Activity 83 mobility routes and destinations together with their existing facilities, has helped enhance the availability and accessibility of health services and other basic services in pastoral regions. This mapping is particularly relevant for the implementation of mobile health services during the seasonal movement of pastoral communities through the identification of routes and destinations. ▪ The Crisis Modifier Fund was a positive midcourse correction that was directly responsive to regional requests to support emergencies. ▪ The random follow-up approach, as a monitoring tool, adapted from Transform: PHC offers opportunities to provide demand-driven technical support and coaching- mentoring health care providers at the health facility level. ▪ The decision to reduce the number of woredas from 58 to 20 to receive the full scale and scope of health service, systems strengthening, and health governance packages was an excellent decision. The number of full-service woredas is now 20, with the remaining 38 woredas receiving some elements of this package, which was a good midcourse correction. Implementation is now strategically focused on 20 Learning Woredas (LWs) having 24 CoE Health Facilities. This is to increase depth and coverage that gradually leads to improved outcomes at the LWs in particular and in the regions in general. The learning from learning woredas will then be replicated in other non-learning woredas in the regions. 6. MAJOR OPERATIONAL CHALLENGES Major challenges faced during the implementation of the Transform: HDR Activity: • The decision of Transform: HDR to limit the number of technical staff at the implementation level could pose a challenge to the timely and effective delivery of services in a vast and challenging environment. • The absence of prior comprehensive MNCH/FP interventions in the developing regions meant that Transform: HDR had no opportunity to learn from them and had to start from the ground up to identify and address gaps. This factor also may have led to a slow start-up and eventually slowed the pace of activity implementation. • The mismatch between stakeholders’ expectations (addressing equity issues by supporting all aspects of the woredas’ transformation) reflected in the activity’s name “Transform: HDR” and the current scope of Transform: HDR is causing disappointment among RHBs. • Security has remained a challenge and threat to service delivery in some parts of the developing regions. • The Transform: HDR Activity faces huge logistics challenges due to the remote nature of these regions, poor roads, and limited transportation and communications barriers. • Weak health system and infrastructure, including lack of electricity, water, etc. • Mobility of the community may increase with anticipated changes in weather. • Deep-rooted cultural norms and traditions are hampering the effort to change behavior-FGM, GBV. Mid-term Performance Evaluation Report USAID Transform: HDR Activity 84 • Increased resistance to FP due to the implication of large family size, the large size of a tribe; population size. • Supply-side readiness may hamper the ability to respond to the MNCH/FP demand generated by Transform: HDR. 7. CONCLUSIONS The FMOH, with support from USAID/Ethiopia through the Transform: HDR Activity, has launched a groundbreaking initiative to improve MNCH/FP services in the developing regions of Ethiopia. There have been significant performance improvements between the baseline (2017) and the mid-term (2019). The groundwork has been laid for advancing towards program targets, with some targets already achieved. There are several important and positive FP, child health, and newborn health results that point to the effectiveness of Transform: HDR. The Transform: HDR supported sites registered better improvements compared to non-Transform sites. Relevance: The design of the Transform: HDR Activity’s strategies and approaches fit with health sector priorities in the developing regions. The joint planning exercise with regions has supported Transform: HDR in adjusting its approaches. As a result, this has ensured its continued relevance to the region’s health sector priorities and changing context. Also, with the connection between pastoral regions and animals, health systems service delivery should focus its attention on animals. Efficiency: Most of the implementation management systems reviewed, such as the work plans and budgets, were found to be efficiently supporting the achievement of programmatic results. However, there is a mismatch between Transform: HDR’s technical and geographic scope and focus on its human and financial resources. There is a need to rethink the technical staffing structure of Transform: HDR to effectively support regional and local health systems to deliver health services and strengthen strategic partnerships with other key actors in recognizing their niches and comparative advantages. Transform: HDR is supporting the DHIS2 rollout. However, there is a need to strengthen data utilization, especially for learning and decision-making. Data concerning the health-seeking patterns and outcomes of pastoral communities should be reviewed and used to refine service delivery approaches for this hard-to-reach, underserved population. Effectiveness: Evidence shows that there are performance improvements in most of the outcome level thematic areas (FP, maternal health, newborn health, and child health) between baseline and mid-term. There are variables/determinants associated with these thematic areas, which include the level of literacy, access to health information, paid work, etc. Therefore, it may require strengthening linkages between technical areas of interventions (e.g., Health, with Education, Livelihoods, Resilience, WASH, etc.). Mid-term Performance Evaluation Report USAID Transform: HDR Activity 85 In conclusion, evidence from this evaluation shows that Transform: HDR has performed well and achieved almost all of its targets at mid-term, exceeding expectations in some areas. However, the performance varies from region to region. Another major finding is that while Transform: HDR indicators are improving non-Transform: HDR areas have shown a decline in performance in key outcomes (e.g., MCPR, SBA, ANC4+, etc.), which should be a cause for concern by the GOE/MOH. The performance was compared between baseline and mid-term, and non-Transform HDR intervention sites include project set targets (LOP). The difference in performance could be attributed to the level of USAID/Transform: HDR support in relation to the external and local financing for non-Transform: HDR sites. The findings from the evaluations show that the factors responsible for the observed performance in Transform: HDR intervention sites and non￾Transform: HDR sites in family planning, maternal health, and child health seem to be similar, with limited varying factors specific to pastoral regions. Also, due to similar population characteristics and spillover effect within woredas in the same regions, the evaluation found determinants tending to be the same between Transform: HDR supported sites and non￾Transform: HDR sites. Thus, the observed differences in trends could be explained by partners and the intensities in their interventions/programs as the plausible determinants. However, there are some differences between the pastoral regions and agrarian regions. Sustainability: Sustainability remains a challenge in developing regions as the health system is not strong, and there is limited budgetary and resource mobilization capacity. Transform: HDR must specify what parameters of sustainability are feasible in these remote regions. Given that this evaluation was designed as a mid-term performance evaluation, it focused on assessing “progress towards” or the “likelihood of achieving intended results” with regards to the effectiveness question, as opposed to “measuring the full achievement of intended results and attribution.” The analysis also assessed Transform: HDR contribution to different thematic area results. Based on this premise, the evaluation concluded that the activity is on track to achieve the intended results. It is important to note this when interpreting results from this mid-term performance evaluation. 8. RECOMMENDATIONS FOR KEY STAKEHOLDERS Recommendation/Action Actor to Address Timing Priority # USAID/Ethiopia 1 Given the good results and the immense unmet needs in the four regions, if funding is available, USAID/Ethiopia might consider expanding the activity and increasing its investment in this successful set of interventions. USAID/Ethiopia Immediate Mid-term Performance Evaluation Report USAID Transform: HDR Activity 86 Recommendation/Action Actor to Address Timing Priority Consideration should be given to ways to mitigate the deteriorating health situation in the non-Transform: HDR sites by expanding collaboration with other external partners. 2 Promote/support an expanded and accelerated financial investment by the GOE to ensure equity and accelerated outreach to the hard-to-reach communities. USAID/Ethiopia Immediate 3 On the issue of Transform: HDR’s targets, since some targets have already been achieved, there is now a need to assess the feasibility of revising the targets upwards vs. expanding the interventions to new woredas for the remaining duration of the activity. USAID/Ethiopia Immediate 4 Despite the initial success in reaching some pastoral communities, site visits by evaluators revealed there is a need to identify sustainable approaches to scale-up MNCH/FP services for pastoral populations. The current strategies employed in pastoral regions are at an early stage, and there are multiple options with varied costs and benefits. USAID/Ethiopia Immediate 5 Consider building into future programming links to water and sanitation, female education programs, and potential income generation and livestock management for pastoralist areas (Afar and Somali). For the predominantly pastoralist regions and communities (Afar and Somali), any future MNCH programming should examine ways to link and/or factor health interventions that affect animals. USAID/Ethiopia Future Design/ Programming (Beyond the life of Transform: HDR) 6 Of particular concern, Transform: HDR is operating in pastoralist and semi-pastoral regions; the evidence shows that the performance in Afar and Somali is different from those of Benishangul Gumuz and Gambella. Therefore, USAID/Ethiopia should consider designing/supporting varying context-specific interventions tailored to different regional contexts (Afar and Somali) and Benishangul Gumuz and Gambella. This could also include a relatively higher investment for Afar and Somali compared to Benishangul Gumuz and Gambella to realize equity. USAID/Ethiopia Future Design/ Programming (Beyond the life of Transform: HDR) Mid-term Performance Evaluation Report USAID Transform: HDR Activity 87 Recommendation/Action Actor to Address Timing Priority Transform: HDR 7 Strengthen regional technical capacity by integrating MNCH/FP material from the activity into the existing pre￾service curriculum for health science colleges. Transform: HDR Immediate/ Beyond the life of Transform: HDR 8 Give more technical assistance and support to the thematic areas such as child health to include the treatment and prevention of diarrheal disease where the team observed relatively weak-performance. This should include the supply of drugs and equipment to treat diarrheal diseases, besides the technical assistance, given the widespread stock-out of ORS and Zinc. In order to achieve these targets, more expert technical assistance should also be devoted to rethinking how to improve the ANC-4 visit target, including the feasibility of spouses accompanying their partners to these visits, the obstacles to pregnant women using iron folate, and women’s decision-making related to FP, to be more effective in achieving targets. Transform: HDR Immediate 9 Enhance/adopt SBCC strategies that focus on influencing and changing the behaviors of the communities and contributing to the use of MCH services. The SBCC approach should consider the establishment/strengthening of locally acceptable high impact community awareness forums using local CBOs, religious or clan leaders to address some of the myths inhibiting health service utilization. Best practices from across Ethiopia should be considered. Transform: HDR Immediate 10 Consider adopting a more in-depth participatory planning process – Theory of Change (TOC) – to better capture regional and woredas needs, adjust strategies and approaches that work well in specific contexts, and refine planning assumptions. Although the current participatory planning process applied supports continued alignment of strategies, the TOC presents a more robust planning approach suited for complex contexts, like developing regions. Transform: HDR Immediate 11 To narrow the gender gap, Transform HDR should ensure that all programming is designed to address the unique Transform: HDR Immediate Mid-term Performance Evaluation Report USAID Transform: HDR Activity 88 Recommendation/Action Actor to Address Timing Priority regional needs and mitigate barriers faced by women and men, boys, and girls to access service. 12 Support the MOH/RHB and woredas to improve the use of performance information for decision-making and planning. Transform: HDR Immediate 13 Transform: HDR should work with USAID/Ethiopia to build/strengthen the Capacity for Public Health Emergency Management (PHEM) at the woreda level. The developing regions are prone to man-made and natural disasters, like floods, disease outbreaks, tribal conflicts, etc. Therefore, strengthening the PHEM capacity may be critical to rapidly respond to emergency situations while sustaining momentum on MNCH/FP programs. This, in the long run, will help build resilience at the community and woreda level, which is critical to achieving health and development outcomes. Transform: HDR/USAID Immediate 14 Realign the existing staff and, where feasible and within operating budgets, expand the number of technical staff to meet requirements. In the short-run, Transform: HDR may assign regions, districts, and specific geographies to their central staff so that they will be focused on their support to the regions and ensure that a good balance between central teams and field-level technical support is addressed in the short run. Transform: HDR Immediate FOR THE GOE/MOH & RHB 15 Support the scale-up of woreda-to-woreda and facility-to￾facility (peer-to-peer) experience sharing to facilitate learning, improve service delivery, and enhance performance. GOE/FMOH, RHBs Immediate/ Beyond the life of Transform: HDR 16 Explore innovative ways to significantly strengthen coordination amongst key health and non-health development actors operating in developing regions for greater synergy, to avoid duplication, and to achieve better results. GOE/FMOH, RHBs Immediate/ Beyond the life of Transform: HDR 17 Considering the evaluation findings related to a possible deterioration in MNCH/FP performance in non-Transform GOE/FMOH, Immediate/ Mid-term Performance Evaluation Report USAID Transform: HDR Activity 89 Recommendation/Action Actor to Address Timing Priority sites, identify ways to leverage additional domestic resources and other donor and private sector investments to accelerate the expansion and enhance the quality of MNCH/FP services in these areas. RHBs Beyond the life of Transform: HDR 18 Assist the developing regions in addressing/managing the Human Resources for Health (HRH) staffing issues in a more structured and systematic manner that addresses attraction, recruitment, deployment, and retention. This may include recognition of high performing staff and health teams; recruitment and retention bonuses for those who agree to serve in the developing regions; and continuing professional development opportunities when they faithfully work in their placement for certain years; career growth. GOE/FMOH, RHBs Immediate/ Beyond the life of Transform: HDR 19 Strengthen the health infrastructure (electricity and water in particular) and supply chain that is affecting access and utilization of services, including transportation/ambulance services. GOE/FMOH, RHBs Immediate/ Beyond the life of Transform: HDR Mid-term Performance Evaluation Report - USAID Transform: HDR Activity 88 ANNEXES Annex A. USAID Transform: HDR Activity: Performance by result area at Mid-term (2019) compared with the baseline (2017) in Transform: HDR intervention and non-intervention areas Key Performance Indicators Transform: HDR Intervention areas (%) Non-Transform: HDR areas (%) Afar Somali Gambella Benishangul Gumuz Total N HDR Non￾interventi on overall N Expanded access and uptake of family planning 1 Modern Contraceptive Prevalence Rate (CPR) among all women Baseline 4.3 4.5 17.9 36.4 19.1 2440 21.8 564 Midterm 6.9 0.2 16.2 45.5 18.3 2102 20.5 623 2 Modern Contraceptive Prevalence Rate (CPR) among currently married women Baseline 5.1 6.2 21.6 41.7 18.9 1928 26.0 458 Midterm 7.9 0.2 22.1 54.5 21.8 1710 23.5 536 3 CPR for LAFP methods among all women Baseline 0.2 0.6 0.8 10.7 3.0 2440 5.9 564 Midterm 0.8 0.0 2.3 15.3 5.1 2102 3.9 623 4 CPR for LAFP methods among currently married women Baseline 0.2 0.9 0.8 12.2 3.6 1928 6.8 458 Midterm 1.0 0.0 3.2 17.6 6.1 1710 4.5 536 5 Unmet Need for Family Planning Baseline 16.6 8.8 26.1 35.6 21.4 2440 25.5 564 Midterm 15.4 6.7 25.3 32.3 20.3 2102 21.3 623 6 Use of modern contraception after birth (PPFP) Baseline 6.4 3.1 9.6 30.5 12.2 616 21.2 146 Midterm 6.7 0.8 7.5 53.1 16.2 481 20.0 150 7 Family planning counseling after birth (PPFP counseling) Baseline 11.8 8.4 11.7 25.8 14.5 613 23.4 145 Midterm 25.2 0.8 3.8 44.2 18.5 481 11.3 150 Increased numbers of healthy mothers-successful birth outcomes 8 The proportion of women who had their first ANC within the first three months of pregnancy Baseline 15.8 24.6 34.5 30.2 24.7 607 30.1 143 Midterm 36.1 27.2 55.1 66.3 45.0 458 53.9 141 9 Women with birth in the last year who took Iron and Folic Acid supplement for at least 90 days Baseline 17.7 5.9 11.8 36.5 23.3 159 24.1 58 Midterm 8.3 6.3 20.0 21.7 15.0 227 12.1 91 10 Pregnant women who slept under ITN the previous night Baseline 41.7 66.7 87.0 98.0 74.6 228 60.5 43 Midterm 67.1 59.5 76.7 64.3 65.1 235 57.6 66 11 The proportion of women who had four or more ANC for their last birth in the last year Baseline 16.5 16.9 23.7 49.7 26.0 611 38.4 146 Midterm 12.1 9.6 22.6 65.8 26.4 474 33.3 147 12 Baseline 40.7 16.9 21.7 22.7 27.4 420 41.7 108 Mid-term Performance Evaluation Report - USAID Transform: HDR Activity 89 Women who received essential components of ANC (among whom had at least one ANC) Midterm 48.4 28.4 27.0 34.0 35.2 349 36.5 115 13 Skilled Birth Attendance Baseline 17.7 21.4 50.9 31.8 28.1 616 43.8 146 Midterm 31.1 11.8 44.3 54.0 34.3 481 25.3 150 14 Early Postnatal Care for the mother within two days Baseline 10.5 17.6 34.2 37.1 22.9 616 35.6 146 Midterm 29.6 22.0 47.2 66.4 40.1 481 42.0 150 15 Women who received MNCH services through mobile health team Baseline 10.0 5.1 3.8 6.4 6.4 2363 12.5 550 Midterm 27.9 34.8 19.9 17.2 24.8 2102 23.3 623 Increased numbers of healthy newborns (birth to 28 days) 16 Early Postnatal Care for the newborn within two days of birth Baseline 12.3 22.1 36.8 30.5 23.4 616 37.0 146 Midterm 29.6 20.5 38.7 66.4 37.8 481 38.7 150 17 Postnatal Care for the newborn within seven days Baseline 13.6 26.7 36.8 32.5 25.3 616 39.0 146 Midterm 34.1 23.6 42.5 69.0 41.4 481 39.3 150 18 Proportion of Newborns who received Essential Newborn Care (Vit K, TTC eye ointment, cord care with ointment - among births that occur in health facility) Baseline 0.0 25.0 13.3 0.0 8.6 175 7.8 64 Midterm 2.2 19.0 8.5 8.6 8.2 195 7.8 64 19 Early initiation of breastfeeding (within 1 hour of birth) Baseline 52.3 35.4 82.6 48.8 53.5 325 55.9 93 Midterm 87.9 77.7 83.8 77.7 81.9 470 90.5 147 20 The proportion of infants 0–5 months of age who are fed exclusively with breast milk Baseline 74.1 27.7 77.4 75.7 66.1 233 71.7 46 Midterm 41.3 28.8 74.4 80.4 55.1 187 75.9 54 21 % of women who delivered in a health facility and stayed at a health facility for at least 24 hrs Baseline Midterm 73.8 46.7 40.4 43.8 50.6 168 39.5 38 Child health (<5) 22 Children 12-23 months who are fully immunized Baseline 9.9 20.4 30.4 61.4 27.6 381 27.6 381 Midterm 17.3 26.1 36.2 56 32.8 332 32.4 108 23 Measles vaccination Baseline 15.4 29.3 34.0 41.6 28.1 381 28.1 381 Midterm 42.1 51.1 65.5 56.1 51.6 332 55.1 108 24 Children 12-23 months who received Penta3 vaccination within first year of birth Baseline 8.7 27.1 39.9 65.0 31.3 381 31.3 381 Midterm 22.6 32.6 41.4 67.9 40.0 332 47.2 108 25 % of infants ages 6 to 8 months who receive complementary foods in addition to a milk source (i.e., breast milk, formula, or other milk) Baseline Midterm 47.1 33.3 41.7 75.0 48.2 112 50.0 42 26 Children under five who had symptoms of ARI in the past two weeks Baseline 3.7 4.7 7.8 9.9 6.3 1698 5.8 365 Midterm 2.2 3.7 14.2 7.4 5.8 2,073 4.4 707 27 Baseline 16.7 0.0 17.2 25.6 16.8 107 19.0 21 Mid-term Performance Evaluation Report - USAID Transform: HDR Activity 90 Proportion of under 5 children with symptoms of ARI (Pneumonia) treated with antibiotics Midterm 92.3 64.0 69.6 81.1 74.4 121 71.0 31 28 Diarrhea incidence Baseline 10.6 5.1 17.9 16.2 12.0 1828 14.6 403 Midterm 16.2 9.7 29.0 18.1 16.6 2,073 14.3 707 29 Diarrhea treatment with ORS+Zinc Baseline 22.8 33.3 32.9 26.5 28.2 220 33.9 59 Midterm 35.1 15.4 16.0 35.6 26.2 343 33.7 101 30 Children under 5 who slept under ITN the previous night Baseline 28.8 35.3 77.3 84.6 53.9 1842 49.0 406 Midterm 56.9 72.8 75.6 72.7 68.7 2,073 62.2 707 31 Children 2-5 years who were given drug for intestinal worms in the last six months Baseline 20.7 3.5 31.1 58.1 27.4 532 24.1 384 Midterm 24.4 15.7 25.9 77.6 33.7 1,330 38.8 464 SBCC 32 Percentage of audience who recall hearing or seeing a FP/RH message Baseline 41.4 42.9 43.2 66.1 48.1 2440 52.3 295 Midterm 44.6 33.1 29.3 43.5 37.8 2,102 33.2 623 33 Percentage of audience who recall hearing or seeing a specific USG-supported MNCH message Baseline 18.9 16.6 19.1 24.4 19.7 2362 24.0 132 Midterm 39.4 38.9 27.8 33.0 34.8 2,102 33.9 623 Cross-cutting 34 Households that have access to a basic sanitation facility Baseline 16.3 31.5 2.3 5.9 14.4 2439 17.7 564 Midterm 7.5 17.4 4.2 19.0 12.6 2,100 8.0 623 35 Households that have hand washing facility with soap/ash and water Baseline 0.2 1.5 0.2 1.7 0.9 2440 2.3 564 Midterm 3.3 0.0 1.7 2.8 2.0 2,100 1.6 623 36 Households using water treatment technology Baseline 16.1 22.3 6.5 5.5 12.9 2440 14.4 564 Midterm 11.5 18.9 4.0 4.3 9.6 2,100 12.5 623 37 Women's participation in decisions regarding their own health Baseline 85.1 93.4 71.0 79.0 82.3 1895 80.1 447 Midterm 74.6 89.1 67.6 82.4 79.3 1,708 78.0 536 38 Women accompanied by their spouse during ANC visits for their last birth ((from the last one-year birth who had at least one ANC visit) Baseline 53.3 12.3 42.4 52.8 44.4 419 52.3 107 Midterm 43.0 15.0 45.3 62.8 40.8 481 46.7 150 39 Women accompanied by their spouse during the birth of their last child (from the last one-year birth who delivered at HF) Baseline 87.8 61.1 66.7 90.3 78.2 220 72.9 85 Midterm 80.4 71.4 70.2 86.4 79.5 195 79.7 64 40 Percentage of households that are enrolled in CBHI schemes Baseline 0.0 0.5 5.1 0.9 1.6 2439 0.7 563 Midterm 0.0 0.0 0.8 22.0 6.5 2,100 1.3 623 Mid-term Performance Evaluation Report - USAID Transform: HDR Activity 88 Annex 1. STATEMENT OF WORK (SOW) & EVALUATIONS PROTOCOL FOR MID￾TERM PERFORMANCE EVALUATIONS OF TRANSFORM IMPLEMENTING MECHANISMS MELA-Midterm Evaluations Design _Revised SOW-31_05_19.docx Annex 2. Midterm Performance Evaluation – Survey tools Household Survey Questionnaire_Final.docx Health Facility Assessment tool for Health Centers and Primary Hospitals_10182019.docx Health Facility Assessment tool for Health Posts_Final.docx Annex 3. Qualitative data collection tools – KII and FGD guides KII guide for FMoH _Transform HDR_19_11_19.docx KII guide for RHB and WoHO_Transform HDR_19_11_19.docx KII guide for Transform HDR Implementer_19_11_19.docx FGD guide_ Final.docx Annex 4. List of relevant documents reviewed List of Documents Reviewed_HDR.xlsx Annex 5. List of Key informants List of interviewed Key Informants.xlsx Midterm Performance Evaluation Report _ USAID Transform HDR Activity 89 89 U.S. Agency for International Development USAID/Ethiopia Entoto Road Addis Ababa, Ethiopia U.S. Agency for International Development USAID/Ethiopia Entoto Road Addis Ababa, Ethiopia U.S. Agency for International Development