Original Submission Date: August 2020 Revised Submission Date: January 2021 Submitted To: Gebeyehu Abelti, COR USAID/Ethiopia Submitted By: The Mitchell Group, Inc. (TMG) 1816 11th St., NW Washington, DC 20001 (202) 745-1919 Contract Number: OAA-I-15-00028 Task Order No.: AID-663-TO-17-00001 Contract Name: Transform: Monitoring, Evaluation, Learning and Adapting (Transform: MELA) Activity Principal Contacts: Dr. Geoffrey Olupot, Chief of Party Email: golupot@ethiopiatmela.com Phone: +251-986 356 914 Jenkins Cooper, Project Coordinator / Vice President, TMG Email: jenkinsc@the-mitchellgroup.com Phone: +001-202-567-1097 (Mobile) +001-202-350-0025 (Direct) This document was produced for review by the United States Agency for International Development Ethiopia (USAID/Ethiopia). It was prepared by The Mitchell Group, Inc. (TMG) for USAID/Ethiopia for the “Transform: Monitoring, Evaluation, Learning, and Adapting (Transform: MELA) Activity." TRANSFORM: MONITORING, EVALUATION, LEARNING AND ADAPTING (TRANSFORM: MELA) ACTIVITY USAID /ETHIOPIA TRANSFORM PROGRAM MID-LINE EVALUATION FINAL REPORT Transform Program Midline Evaluation Report Page i ACTIVITY INFORMATION Activity Title TRANSFORM: MONITORING, EVALUATION, LEARNING AND ADAPTING (Transform: MELA) Contract Number Contract No.: OAA-I-15-00028; Task Order No.: AID-663-TO-17-00001 Name of Prime Implementing Partner The Mitchell Group, Inc. 1816 11th Street, NW Washington, DC 20001 Tel: 202-745-1919 Name(s) of Subcontractor(s)/Sub￾awardee(s) Local M&E Partners approved by USAID/Ethiopia: PRIN, SuDCA, Panafrica and Addis Ababa University/School of Public Health (AAU/SPH) Activity Start Date March 7, 2017 Activity End Date March 6, 2022 Midline Evaluation of Transform Program November 2019 – July 2020 Transform Program Midline Evaluation Report Page ii Table of Contents ACRONYMS iv EXECUTIVE SUMMARY 1 1. INTRODUCTION 11 1.1. Background of the Transform Program 11 1.2. Scope of the Midline Evaluation 13 1.3. Evaluation Questions 14 1.4. Evaluation Design and Methodology 15 1.4.1. Study Settings 15 1.4.2. Design 16 1.4.3. Methods 17 1.4.3.1. Sampling Design 17 1.5. Data Collection Tools 19 1.5.1. Data Collection 19 1.6. Data Quality Assurance and Management 20 1.7. Data Analysis 21 1.8. Ethical Procedures 22 1.9. Limitations to the Study 22 2. EVALUATION FINDINGS 23 2.1. RELEVANCE 23 2.1.1. Alignment with Strategic Priorities 23 2.1.2. Contextual and Cultural Appropriateness 24 2.1.3. Current Geographic Targeting 25 2.1.4. Stakeholder Perceptions of the Transform Program 26 2.1.5. Strengths, Lessons Learned, and Missed Opportunities on Program Relevance 27 2.1.6. Recommendations and Implications for Program Relevance 29 2.2. EFFICIENCY 30 2.2.1. Contributions to Programmatic Results 30 2.2.2. Management Structure 31 2.2.3. Partnerships 33 2.2.4. Innovative Approaches to Attain Better Results 34 2.2.5. Recommendations and Implications for Programmatic Efficiency 35 2.3. EFFECTIVENESS 36 2.3.1. Progress Toward Transform Program Outcomes 37 2.3.1.1. Respondent Demographic Characteristics 37 2.3.1.2. Analyses of Key MNCH/FP Performance Indicators – Comparisons and DID 38 A. FAMILY PLANNING 38 B. MATERNAL HEALTH 46 C. NEWBORN HEALTH 56 D. CHILD HEALTH 60 E. Water, Hygiene, and Sanitation (WASH) 66 F. CROSS CUTTING: Gender and Community-Based Health Insurance (CBHI) 67 2.3.2. Effectiveness in Delivery Methods and Programming Approaches 69 2.3.3. Stakeholder Satisfaction with the Transform Program 71 Transform Program Midline Evaluation Report Page iii 2.3.4. Enabling and Hindering Factors Regarding Effective Implementation 72 2.3.5. Recommendations for Achieving Intended Results 73 2.4. TRANSFORM PROGRAM SUSTAINABILITY 74 2.4.1. Sustainability Elements, Strategies and Approaches Put in Place 74 2.4.2. Mobilizing Key Stakeholders 76 2.4.3. Enhancing Ownership at Different Levels 76 2.4.4. Key Obstacles to Sustainability 77 2.4.5. Recommended Changes to Ensure Sustainability 78 2.4.6. Best Practices to Scale-up in Remaining Years 79 2.5. ADDRESSING GENDER 80 2.5.1. Program Contributions to Gender Issues 80 2.5.2. Best Practices in Gender Programming 80 2.5.3. Obstacles to Addressing Gender Equity 81 2.5.4. Recommendations to Improve Gender Programming 81 2.6. CHALLENGES AND LESSONS LEARNED 82 2.6.1. Challenges Faced and Strategies for Mitigating Those Challenges 82 2.6.2. Innovations to Scale and Apply Elsewhere 83 2.6.3. Key Lessons Learned 84 3. CONCLUSION 84 3.1. Transform Program’s Likelihood of Achieving Intended Results and Targets 84 3.2. Implications for USAID/Ethiopia Future Programming 85 3.3. Implications for GOE/FMOH Future Programming 86 4. RECOMMENDATIONS 88 Transform Program Midline Evaluation Report Page iv ACRONYMS ACT Arteminisin-based Combination Therapy ANC Antenatal Care ARI Acute Respiratory Infection BEmONC Basic Emergency Obstetric and Newborn Care CBHI Community-based Health Insurance CBNC Community-based Neonatal Care CEmONC Comprehensive Emergency Obstetric and Newborn Care CoE Centers of Excellence CPR Contraceptive Prevalence Rate CSA Central Statistical Agency DHIS2 District Health Information Systems 2 DHS Demographic and Health Survey DID Difference-in-Differences DRS Developing Regional States EA Enumeration Area EMWA Ethiopian Midwives Association ENC Essential Newborn Care EPSA Ethiopian Pharmaceuticals Supply Agency EQA External Quality Assurance EPSA Ethiopian Pharmaceutical Supply Agency ESOG Ethiopian Society of Obstetricians and Gynecologists FGD Focus Group Discussions FMOH Federal Ministry of Health FP Family Planning GBV Gender-based Violence GIS Geographic Information System GOE Government of Ethiopia HC Health Center HDA Health Development Army HDR Health in Developing Regions HEW Health Extension Workers HH Household HP Health Post HRH Human Resources for Health HSTP Health Sector Transformation Plan ICCM Integrated Community Case Management IFHP Integrated Family Health Program IP Implementing Partner ITN Insecticide Treated Net IUD Intrauterine Device KI Key Informant Transform Program Midline Evaluation Report Page v KII Key Informant Interview KPI Key Performance Indicator LAFP Long-Acting Family Planning LAM Lactational Amhenoria LOP Life of Project MCPR Modern Contraceptive Prevalence Rate MELA Monitoring, Evaluation, Learning, and Adapting MFI Microfinance Institutions MNCH/FP Maternal, Newborn, and Child Health and Family Planning NICU Neonatal Intensive Care Unit NOFO Notice of Funding Opportunity ODK Open Data Kit ORS Oral Rehydration Solution OWNP One WASH National Program PHC Primary Health Care PHCU Primary Health Care Unit PMP Performance Management Program PMCD Preventable Maternal and Child Death PNC Postnatal Care PPFP Post-partum Family Planning PTI Primary Health Care Transformation Initiative RH Reproductive Health RHB Regional Health Bureau SBA Skilled Birth Attendant SDM Standard Days Method SGBV Sexual and Gender-Based Violence SNNP Southern Nations, Nationalities and Peoples State TA Technical Assistance TMG The Mitchell Group, Inc. ToC Theory of Change ToT Training of Trainers TPHC Transform Primary Health Care UHC Universal Health Care USAID United States Agency for International Development USG United States Government VSLA Village Savings and Loan Association WASH Water, Sanitation, and Hygiene WASHCO WASH Committee WDA Women’s Development Army WHO World Health Organization WTT WASH Technical Team WWT Woreda WASH Team USAID/ Ethiopia Transform Program Midline Evaluation Report Page 1 EXECUTIVE SUMMARY Background and Evaluation Purpose USAID/Ethiopia contracted The Mitchell Group, Inc. (TMG) as the Prime Contractor to lead and implement the Transform: Monitoring, Evaluation, Learning, and Adapting (Transform: MELA) Activity. Transform: MELA is the evaluation component of the broader Transform program, which aims to reduce maternal and child deaths in Ethiopia. The Activity is comprised of three core Activities: 1) Transform: Primary Health Care (Transform: PHC) operating in Amhara, Oromia, the Southern Nations, Nationalities, and Peoples’ (SNNP) region, and Tigray; 2) Transform: Health in Developing Regions (Transform: HDR), operating in Somali, Afar, Gambella, and Benishangul-Gumuz; and 3) Transform: Water, Sanitation, and Hygiene (Transform: WASH), operating across all eight regions. In addition, the Transform program includes cross-cutting initiatives such as gender equity, access to community-based health insurance, and sanitation. The fourth Activity, Transform: MELA, provides performance measurement and management functions to the Transform program as a whole. The Transform program (Transform suite of activities) is a five-year program running from 2017 to 2022. The purpose of the Transform program Midline Program Evaluation is to assess the continued relevance of the program, its efficiency, its effectiveness, and the sustainability of its results. Its emphasis lies in the results from a survey of 6,350 households, which assesses progress toward key indicators of program effectiveness in the Transform intervention areas. In addition, the evaluation draws from key informant interviews (KIIs), Activity-level reports, health facility visits, and other sources of information to identify areas of success and continued challenges related to the core activities that comprise the Transform program. Data collection for the evaluation took place in early 2020. The results will inform both USAID/Ethiopia and the Federal Ministry of Health (FMOH) regarding the remainder of the Transform program as well as future Preventable -GUMU Maternal and Child Death (PMCD) programming. Evaluation Design/Statistical Methods The midline evaluation is a cross-sectional study that followed a two-stage sampling methodology. Prior to sampling, lists of Transform program targeted woredas in the eight regions were obtained and study participants were selected using a two-stage sampling technique. The first stage of random sampling identified health posts/kebeles in the woredas. The second stage sampling entailed the random selection of households from the sampled kebeles. The household survey included 6,350 women of reproductive age (15-49 years) across the eight regions, assessing outcomes in five categories: 1) family planning, 2) maternal health, 3) newborn health, 4) child health, and 5) cross-cutting outcomes that include sanitation, hygiene, community-based health insurance, and gender. Among the respondents, 5,097 were from intervention areas, and 1,253 were from non-intervention areas. USAID/ Ethiopia Transform Program Midline Evaluation Report Page 2 To determine progress since the baseline survey, the analysis relied on comparing proportions, and difference-in-differences (DID) design was used to determine the contribution of the Transform program to MNCH/FP outcomes of interest. The Transform baseline survey (2017- 2018) and Transform midline survey (2019-2020) both generated data on key indicators from both intervention and non-intervention areas. Using the two waves of surveys, the analysis team compared the change over time from baseline to midline in the intervention areas to the change over the same period in the non-intervention areas. The results indicate the extent to which outcomes have changed during the period of the Activity using data from the intervention areas. The non-program related factors that may have had some impact on outcomes, even in the absence of the program using data from the non-intervention areas, were taken into account using DID and causality tracking strategies analysis. The Midline Evaluation utilized quantitative and qualitative primary and secondary data. The qualitative data was purposively collected from stakeholder respondents selected as KIIs and FGDs. A total of 89 KIIs were conducted at the local, regional, and national levels. Summary of Evaluation Findings by Evaluation Questions I. Relevance: Across the three core activities of PHC, HDR, and WASH, the Transform program remains relevant and appropriate. The activities adhere to the strategic objectives of the Government of Ethiopia (GOE)/Federal Ministry of Health (FMOH) as well as to the goals of the regional health bureaus (RHBs) and health sector priorities. In particular, the co-creation process during Transform design has ensured that there would be clear understanding, buy￾in, and GOE and local ownership from the design phase of the program. Furthermore, collaboration with national and regional stakeholders has been strong, ensuring that stakeholders perceive the program as appropriate. The wide geographic spread across the developing regional states poses some operational challenges to the relevance of the HDR and WASH activities, and the ambitious introduction of new business models under the WASH Activity has not always been congruent with the needs of communities or the financial capabilities of local entrepreneurs. The Transform: HDR Centers of Excellence (CoEs) could be a good model in providing vital learning for scaling up. Nevertheless, stakeholders view the Transform program as generally appropriate for addressing program objectives, and the collaboration with GOE/FMOH counterparts reinforces its consistency with GOE goals. II. Efficiency: Capacity building, mentoring, and the continuous adaptation of the program’s interventions and management structure based on theories of change (ToC) has been a strong component of the program. Recruitment and deployment of an adequate number of skilled staff, particularly women, remain a challenge, especially in the hard-to-reach areas associated with the HDR and WASH activities. The deployment of technical assistance by the Activities and the regional governments also introduces some difficulties in providing on-site technical assistance and consultations where they are needed. USAID/ Ethiopia Transform Program Midline Evaluation Report Page 3 The co-location of Activity offices within government health offices has proven to be an important source of efficiency in the Transform program. The evaluation team found that co￾location-built teamwork between the Transform program and the regional health bureaus zones, and woredas. The co-location of Transform Activity offices within RHBs has also improved the working conditions of Transform Activities’ field staff including facilitating joint supervision, joint planning, smooth communication, etc. III. Effectiveness: The effectiveness of the Transform program in making progress toward its programming goals was assessed primarily through the household survey, which provided measures for a series of key Maternal, Newborn, and Child Health and Family Planning (MNCH/FP) outcomes. The proportions and DID analyses outline changes across MNCH/FP outcomes since the baseline, accounting for the comparative changes in Transform program intervention and non-intervention areas over the same period. It is important to keep in mind that the results are midline indicators and observed values of progress. These results do not constitute a final assessment of program effectiveness, either in positive or negative terms. In order to assess whether the Transform program is on track to achieve the intended results requires the availability of Transform program level targets for each key performance indicator (KPI). Given that targets were not set for Transform’s key performance indicators, the evaluation team relied on aggregating different Activity-level KPIs in addition to drawing inference from individual Transform program implementing mechanisms’ performance. a) Family Planning o Across the Transform program targeted area, the contraceptive prevalence rate increased by 3 percentage points. o The overall use of long-acting contraception increased by 2.3 percentage points. Contributing factors for observed improvements in family planning are associated with the active engagement of health extension workers, increased knowledge and awareness of FP methods, and increased availability of long-acting family planning (LAFP) methods at health facilities. However, cultural and traditional restrictions, fear of side effects, and inadequate health service provision are some of the factors that hinder the regions’ modern contraceptive prevalence rate (MCPR) performance. Women registered in community-based health insurance (CBHI) programs were found to be more likely to access FP and maternal health services than those who had yet to register with CBHI programs.1 b) Maternal Health o Essential Antenatal Care increased by 8.4 percentage points across the program intervention areas. o Birth attendance by a skilled professional increased by 16.2 percentage points overall. 1 https://www.researchgate.net/publication/332999123_Effects_of_community_based_health_insurance_on_modern_family_planning_utiliz ation_in_Ethiopia USAID/ Ethiopia Transform Program Midline Evaluation Report Page 4 o Insecticide Treated Net (ITN) usage improved in relative terms, but the difference resulted from a decrease in usage in non-intervention areas, rather than an increase in the Transform program target areas. o Antenatal Care (ANC) visits and 4+ ANC visits have not shown improvement. The program is on the right track in terms of improving maternal care. According to one informant, “Better maternal health results can be achieved by actively engaging health extension workers (HEWs), expanding new health service facilities, and increasing emphasis on maternal health. All of these factors [are] believed to have contributed to improving the coverage of maternal care in woredas and regions.”2 Emphasis should be given on how to increase efficiency of existing resources to achieve program targets. Other factors cited that affect the utilization of skilled delivery services include lack of information, low education on maternal health, lack of skilled delivery experience, and frequency and quality of antenatal care utilization. The poor quality of services that are not mother friendly and poorly equipped health facilities may not convince mothers to deliver in health facilities. It is important to address individual, family, cultural, and structural factors to substantially increase deliveries attended by skilled attendants. The other common reason cited that hinders pregnant mothers from attending ANC services includes women's lack of awareness of its importance, distance to health facilities, and limited availability of affordable transportation. On the other hand, essential determinants that facilitate ANC attendance include higher education levels, women's ability to make healthcare decisions, partner involvement, and MCH messaging. The utilization of maternal services in Afar and Somali is significantly lower compared to all other regions, both at baseline and mid-term. According to KIIs in Afar and Somali, other factors that limit the uptake of maternal health services in their regions include sociocultural factors, the mobile lifestyle of pastoralist populations, distance to health facilities , and the absence of waiting areas at health facilities. c) Newborn Health o Early initiation of breastfeeding in Transform program intervention areas declined by 8.8 percentage points compared to the performance in non-Transform program areas. The factors cited for the decline in breastfeeding include a lack of information, low education among mothers, cultural beliefs, etc. o Other indicators of newborn health did not show relative improvement at the midline stage, owing to comparable improvements in non-intervention areas. d) Child Health o The share of children suffering from acute respiratory illness declined by 2 2 A key informant from the Oromia RHB USAID/ Ethiopia Transform Program Midline Evaluation Report Page 5 percentage points. Further investigation to ascertain whether there is an actual decrease in children suffering from acute respiratory illness or decline in the ability to identify cases will be conducted. o The incidence of diarrhea increased by 3.6 percentage points. Further investigation will be utilized to establish if the increase in incidence of diarrhea is a result of improved detection or actual increase in incidence of diarrhea cases. o The rate of exclusive breastfeeding actually declined by 5.7 percentage points. o The rates of full immunization and child ITN use improved, but not to a statistically significant degree. e) WASH o Access to basic sanitation facilities increased by 7.3 percentage points. o The share of households using appropriate water treatment declined by 3.3 percentage points. f) Cross-cutting: Gender and CBHI o Women accompanied by their spouses during the birth of their last child increased slightly from 81.2% to 83.2%. o Enrollment in CBHI improved by 8.2 percentage points. IV. Sustainability: Numerous program initiatives provide valuable contributions to sustainability and have received high marks from FMOH and regional local stakeholders. Those initiatives include: o The twinning activities for low- and high-performing woreda health facilities. o Coaching and mentorship activities are successfully helping to build the capacity of local stakeholders in ways that improve both program effectiveness and sustainability. o The technical assistance (TA) initiatives and the co-location of cluster offices with government health offices. o While the Activity does not yet have tangible results at this point in the program lifecycle, the “Her Space” initiative is popular, practical, and anecdotally effective in building the knowledge and confidence of young women around reproductive issues. o Woreda-to-woreda and facility-to-facility (peer-to-peer) experience sharing has proven to be an effective means of facilitating learning and improving service delivery. o GPS mapping has proven to be a helpful means to understand the needs of, and ultimately serve, populations in the most difficult areas. o Capacity building trainings, followed by post-training follow-up and on-site mentorship and coaching, have improved service quality. These efforts receive strong support from local stakeholders in KIIs. USAID/ Ethiopia Transform Program Midline Evaluation Report Page 6 There remain some barriers to sustainability. These include high staff turnover, the difficulty of serving hard-to-reach areas, limitations of resource and technical materials, cultural factors that undermine programming efforts around family planning and gender equity, and difficulty in creating access to credit for local entrepreneurs working in health and sanitation services. V. Gender Equity: In addition to the “Her Space” initiative, the Transform program work on sexual and gender-based violence (SGBV) efforts to provide counseling and community-level awareness on gender-related issues is critical and should continue. The gender assessment activities also provided a helpful understanding of the gender dynamics within the context of Transform programming. However, the cultural and traditional norms, alongside a shortage of trained female staff members, undermine potential progress on gender issues. Limitations to the Study Three challenges limited the evaluation methodology: 1) Non-Transform programs operate in both the intervention and non-intervention areas, and it was difficult to isolate confounding factors across intervention and non￾intervention areas. The DID analysis was used to isolate compounding factors under these circumstances. Thus, it is difficult to definitively discern the results and impact of other woreda-level, FMOH, or donor-financed programs on achieving MNCH/FP targets in the Transform geographies in the context of development programming and effectiveness. The evaluation team relied on the DID and causality tracking strategies analyses to overcome these challenges. The DID results should be interpreted bearing in mind these considerations. 2) The absence of an Enumeration Area (EA) list from the Ethiopian Central Statistical Agency (CSA) imposed constraints on the sampling frame. Transform: MELA utilized kebele guides to clarify and identify boundaries of the selected kebeles and to ensure the complete listing of households in selected Gotts.3 3) Security concerns and inaccessibility impeded data collection in some enumeration areas. Sampling with replacement was employed to address these challenges. Conclusions Inferentially, from the three Transform program Activities’ performance, the Transform program is on track and making a significant contribution to the GOE Health Sector Transformation Plan (HSTP) agenda and on meeting most of its intended targets by the end of the program period. 3 In Ethiopia, a Gott is a small, informal administrative unit, equivalent to a neighborhood or block. Several Gotts comprise a Kebele. USAID/ Ethiopia Transform Program Midline Evaluation Report Page 7 Key Recommendations for Stakeholders The evaluation revealed numerous lessons learned and implications for future programming. These are summarized in the following set of recommendations that are broken down by responsible agent and timing priority for USAID/Ethiopia and the GOE/FMOH. Recommendation/Action Responsible Agent Timing Priority # USAID/Ethiopia 1 USAID/Ethiopia should work with implementing partners to identify ways to increase efficiency of existing resources to achieve Activity and program targets. USAID/Ethiopia Immediate 2 USAID should work with the FMOH/RHBs to build a resilient health system for the MNCH/FP for the remaining Transform program period. USAID may wish to provide leadership in defining health system resiliency elements for the MNCH/FP theme. USAID/Ethiopia Immediate 3 USAID/Ethiopia should set Transform program-level targets to measure the effectiveness of the program as a whole. These targets should also have some indicators that allow aggregation/consolidation of performance across the Transform portfolio implementing mechanisms. This will allow the Transform activities to be the drivers by which the Transform program targets are achieved, and hence contributors to Transform program targets. USAID/Ethiopia Immediate 4 USAID/Ethiopia should promote the twinning of low￾performing and high-performing woreda health facilities (already being supported by some Transform Activities) to take advantage of emerging best practices and to reinforce collaboration. USAID/Ethiopia Immediate 5 USAID/Ethiopia should consider expanding TA in thematic areas with little discernible performance improvements to date. USAID/Ethiopia Immediate USAID/ Ethiopia Transform Program Midline Evaluation Report Page 8 Recommendation/Action Responsible Agent Timing Priority 6 USAID/Ethiopia should forge additional and stronger relationships with key local stakeholders and partners, including financial institutions who provide credit to entrepreneurs. USAID/ Transform: WASH Immediate 7 USAID/Ethiopia should consider competitively selecting woredas that have multi-sectoral investments that support the achievement of MNCH/FP objectives. Future activities should be designed in the same regions, ensuring that the woredas have a robust transformation agenda to complement and reinforce USAID/Ethiopia’s health interventions. The need for multi-sectoral response systems should be emphasized as some of the interventions are difficult to be successfully implemented and sustained by the health sector alone. USAID/Ethiopia Future Programming/ Beyond the life of Transform Program 8 USAID/Ethiopia should continue using co-creation in designing new projects. USAID/Ethiopia Beyond the life of Transform Program 9 USAID/Ethiopia should explore innovative ways to strengthen coordination amongst key health and non￾health development actors operating in the developing regions to improve synergy, to avoid duplication, and to achieve better results. USAID/Ethiopia Immediate/ Beyond the life of Transform Program 10 USAID/Ethiopia should consider some of the deteriorating outcomes in MNCH/FP performance in non-Transform sites and identify ways to leverage additional domestic resources and other donor and private sector investments to accelerate the expansion and to enhance the quality of MNCH/FP services in these areas. USAID/Ethiopia Immediate/ Beyond the life of Transform Program 11 To narrow the gender gap, USAID/Ethiopia should consider targeting younger women and their partners in order to begin changing cultural norms over time. USAID/Ethiopia Immediate/ Beyond the life of Transform Program 12 USAID/Ethiopia should reinforce the already impressive coaching and mentoring initiatives to build local capacity USAID/Ethiopia Immediate USAID/ Ethiopia Transform Program Midline Evaluation Report Page 9 Recommendation/Action Responsible Agent Timing Priority and to provide a sustainable foundation for MNCH/FP progress. 13 USAID/Ethiopia should promote an expanded and accelerated financial investment by the GOE to ensure equity and accelerated outreach to the hard-to-reach communities. Transform: HDR Immediate/ Beyond the life of Transform Program 14 USAID/Ethiopia should set aside funding that can be used for unforeseen but critical needs. For example, for innovative ways for businesses to reach consumers with new health and sanitation products, for targeting areas where the midline results indicate a need for greater attention, and for emergency support related to the COVID￾19 pandemic. Transform: WASH Immediate/ Beyond the life of Transform Program # Government of Ethiopia (GOE)/Federal Ministry of Health (GOE/FMOH) 15 The GOE/FMOH should strengthen coordination mechanisms with key stakeholders to tailor actions towards improving synergistic efforts related to MNCH/FP. GOE/FMOH, RHBs Immediate/ Beyond the life of Transform Program 16 The GOE/FMOH should improve resource mobilization and information use to support enhanced evidence-based decision-making and to ensure leadership stability. GOE/FMOH, RHBs Immediate/ Beyond the life of Transform Program 17 The GOE/FMOH should consider using twinning as a viable approach for technical assistance across regions. GOE/FMOH, RHBs Immediate/ Beyond the life of Transform Program 18 The GOE/FMOH should support the scale-up of woreda-to￾woreda and facility-to-facility (peer-to-peer) experience sharing to facilitate learning, to improve service delivery, and to enhance performance. GOE/FMOH, RHBs Immediate/ Beyond the life of Transform Program 19 The GOE/FMOH should explore innovative ways to strengthen coordination amongst key health and non￾health development actors to maximize synergies. GOE/FMOH, RHBs Immediate/ Beyond the life of Transform USAID/ Ethiopia Transform Program Midline Evaluation Report Page 10 Recommendation/Action Responsible Agent Timing Priority Program 20 The GOE/FMOH should expand the dissemination of Transform program-wide best practices to build sustainable programs that build on the successes of Transform: PHC, Transform: HDR, and Transform: WASH. GOE/FMOH, RHBs Immediate 21 The GOE/FMOH/RHB/regional government should assist the developing regions in managing their human resources for health (HRH) staffing issues in a more structured and systematic manner that addresses attraction, recruitment, deployment, and retention. This may include recognition of high performing staff and health teams, recruitment and retention bonuses for those who agree to serve in the developing regions, and continuing professional development opportunities when health staff faithfully work in their placements for a certain period of time. In the short run, the GOE/FMOH may focus on supporting training and demonstration centers at the regional level in collaboration with local universities. GOE/FMOH, RHBs Immediate/ Beyond the life of Transform Program USAID/ Ethiopia Transform Program Midline Evaluation Report Page 11 1. INTRODUCTION 1.1. Background of the Transform Program USAID recognizes the critical importance of MNCH/FP for sustainable development. With this in mind, the Agency launched the Transform Program, a five-year (2017–2021) program to reduce preventable maternal and child mortality and morbidity in Ethiopia. Transform operates across eight regions in Ethiopia and comprises three principal Activities: PHC, HDR, and WASH. The Transform: PHC Activity aims to reduce maternal, newborn, and child mortality and morbidity through support to both the national-level and woreda-level of the GOE/FMOH’s HSTP. Transform: PHC’s focus is on advancing health outcomes related to maternal, newborn, and child health (MNCH); family planning (FP); and reproductive health (RH). It operates in four major regions – Amhara, Oromia, the SNNP region, and Tigray – and targets a total of 412 woredas over the 5-year life of the program. The Activity’s phased approach specified that “360 woredas are reached at the end of the second year, and 52 more by year three.” So far, Transform: PHC has supported 405 woredas. The USAID/Ethiopia Transform Program, together with the GOE/FMOH and regional health bureaus, is implementing myriad interventions to improve MNCH/FP outcomes. Some of the initiatives and strategies under the Transform: PHC Activity include: • Twinning high-and low-performing woredas and facilities. • Expanded quality improvement collaboration beyond MNCH to include FP, adolescent youth health and development, and child health and development. • Scale-up of the “Her Space” initiative, which empowers girls, in an integrated component beyond reproductive health. • Innovative clinical skill labs held at health centers to strengthen peer-to-peer education and promote practical hands-on experience for practitioners and interns. • Random follow-ups that provide real-time data to inform technical support staff as well as useful internal monitoring systems. • Annual ToC exercises to identify regional priorities for the following year, including common challenges across woredas. • Woreda grants that provide additional resources to finance woreda health sector priorities and needs. The FMOH has since replicated many aspects of the woreda subgrant management in other non-Transform: PHC woredas using government financing. • Coaching and mentorship support mechanisms at woreda health offices and primary hospitals. • The provision of solar suitcases to solve power problems at health centers and to increase the delivery of services. The Transform: HDR Activity seeks to significantly reduce morbidity and mortality among mothers and children under five years of age in the developing regional states (DRS) of Ethiopia. USAID/ Ethiopia Transform Program Midline Evaluation Report Page 12 It is one of USAID/Ethiopia’s Health Office investments aimed at preventing child and maternal deaths. This Activity implements interventions through a woman-, child- and girl-centered three￾pronged strategic approach focusing on: 1) access to integrated quality, high-impact MNCH/FP services, 2) improved health-seeking behavior enhanced by reduced gender inequalities, and 3) improved evidence-based decision-making and program learning. These three strategies have the overall goal of strengthening DRS health systems. Transform: HDR operates in the regions of Somali, Afar, Gambella, and Benishangul-Gumuz. Some of the initiatives and strategies under Transform: HDR include: • Refocusing the effort to 20 selected woredas from the 59 woredas and identifying health facilities to build for COE status with the provision of full packages of support. The aim of this strategy is to support the government in transforming some of the woredas, documenting the learning and capitalizing on the learning for scale-up to other woredas as well as continuing minimal support for the rest of the woredas. • Medical equipment provisions including portable V-scan ultrasound and basic emergency obstetric and newborn care (BEmONC supplies) as well as medical technologies to improve case detection and increased client flow. • GPS mapping of pastoral communities’ mobility routes and destinations in the pastoralist regions of Afar and Somali. • A one-stop service center for SGVB, along with reinforced antenatal care (ANC) labs to improve the quality-of-care interventions. • An expansion of the use of community scorecards to enhance health governance and accountability. • The development of a training of trainers (ToT) pool at the regional level that builds on relationships with the RHBs’ regional universities, health science colleges, and reference labs. • A “Comprehensive Mobile Health Outreach Services” component to ensure regular MNCH/FP service provision and access with underserved areas. • Collaboration with the GOE and the FMOH’s District Health Information Systems 2 (DHIS2) initiative on the roll-out of its capacity strengthening support platform. The Transform: WASH Activity develops and tests market-based models to increase the demand and supply of quality and affordable WASH products and services through scalable and replicable business models. This first-of-its-kind Activity is intended to support and increase the capacity of the GOE’s One WASH National Program (OWNP) to market, promote, and improve the utilization of WASH products and services. The Activity goal is to reduce preventable death and illness in Ethiopia due to diarrheal disease, particularly among children under five. Transform: WASH operates across all eight regions in which either PHC or HDR interventions are taking place and the Dire Dawa City Administration. In terms of concrete initiatives and strategies, some of the Transform: WASH interventions include the following: USAID/ Ethiopia Transform Program Midline Evaluation Report Page 13 • The introduction of four core business model activities for local vendors: door-to-door simple toilet upgrades (product installation service), basic manufacturing and installation, advanced toilet manufacturing and construction, and plastic slab sales and support. • The use of market research and consumer surveys to develop and test sanitation products and marketing strategies prior to launch. • Trained female sales agents who engage with the population, help market products, and ensure on-time delivery. • Capacity building for woreda WASH teams (WWT) to strengthen their skills and their efficiency in local budgeting, planning, procurement, and financial management for the OWNP implementation at a sub-national level. • Designing, testing, and demonstrating different sanitation products in different contexts and regions. • The introduction of different financing options and seed funds such as village savings and loan associations (VSLAs) and microfinance institutions (MFIs) to improve financial capacity for households and business enterprises. In addition to the Transform: PHC, Transform: HDR, and Transform: WASH Activities, the Transform program emphasizes cross-cutting objectives related to improving gender equity and advancing gender-related outcomes in Ethiopia. The Transform program also prioritizes strategies and approaches that enhance the sustainability of program objectives. Transform: MELA is a component of the broader Transform program responsible for evaluating the effectiveness of the programmatic activities in achieving their objectives. In 2018, Transform: MELA completed an extensive baseline survey that included information from over 6,500 households, 119 health facilities, focus groups, and KIIs across the eight regions, with a 4:1 ratio of responses from intervention and non-intervention areas. The purpose of the baseline survey was to provide measurements according to key program indicators and to establish a foundation against which progress could be measured at subsequent points in the life of the program. This report addresses those changes in program outcomes, using a comparison of results from the baseline to a subsequent midline evaluation survey. 1.2. Scope of the Midline Evaluation This evaluation covers USAID/Ethiopia’s core evaluation questions – notably centering on relevance, efficiency, and effectiveness – using data and information from the three principal program Activities with special emphasis on the quantitative results that measure effectiveness. To evaluate the effectiveness of Transform program interventions, Transform: MELA collected extensive household-level data on key performance indicators, using the same methodological USAID/ Ethiopia Transform Program Midline Evaluation Report Page 14 approach as the Transform: MELA baseline survey to ensure comparability in the findings. The household survey included 6,350 women of reproductive age (15-49 years) in the eight regions of the USAID/Ethiopia Transform program intervention areas including Amhara, Oromia, SNNP, Tigray, Afar, Somali, Gambella, and Benishangul-Gumuz. The study assessed outcomes in five categories: 1) family planning, 2) maternal health, 3) newborn health, 4) child health, and 5) cross-cutting outcomes that include sanitation and hygiene, CBHI, and gender. Among the 6,350 respondents to the household survey, 5,097 were from intervention areas and 1,253 were from non-intervention areas. Proportions analysis was employed to measure the change in indicators from Transform program intervention areas between the baseline and midline surveys, relative to the change over that same period in non￾Transform program areas. DID analysis was used to measure contribution of the Transform program interventions to MNCH/FP outcomes. The evaluation team also collected and analyzed data from health facilities to assess the interaction between the supply of health services and their utilization. Household data provided utilization data, while health facility data provided information on the supply of health services. USAID/Ethiopia recognizes that monitoring and evaluation are key means through which to obtain systematic and meaningful feedback on both the successes and shortcomings of program activities. The USAID Evaluation Policy, as well as recent revisions made to the Automated Directives Systems 200 series, demonstrate the Agency’s commitment to generating strong monitoring and evaluation data and applying evidence-based learning. To adhere to USAID’s Evaluation Policy, Transform: MELA devoted the first half of 2020 to conducting a survey that closely replicated the baseline survey. The purpose of the midline evaluation is to provide high-quality monitoring and evaluation data for USAID/Ethiopia, and to employ rigorous statistical methods to compare changes in key indicators since the baseline period. The results will guide the U.S. Government (USG) and GOE in adaptive learning and management of the RMNCH portfolio that forms the basis of the Transform program. By identifying areas of progress related to program interventions, as well as maternal and child health outcomes that have not improved since the baseline, the midline evaluation will inform stakeholders – including USAID, USG, GOE and, in particular, the FMOH – about the programming needs and opportunities that lie ahead for the remainder of the program to better address critical needs and shortcomings. 1.3. Evaluation Questions The Transform program midline evaluation aims to assist USAID/Ethiopia and the GOE/FMOH in answering the following key evaluation questions: 1. Have the USAID/Ethiopia Transform program strategies and implementation mechanisms continued to be relevant and appropriate to the targeted regions? USAID/ Ethiopia Transform Program Midline Evaluation Report Page 15 2. How efficient are the Transform program approaches and strategies in achieving intended results? 3. How effective have the Transform program activities been in contributing to improved MNCH/FP outcomes, in terms of achieving results? 4. How have the Transform program activities facilitated local ownership and sustainability? 5. How have the Transform program activities considered gender dynamics in the program design and implementation? 6. What are the lessons learned and recommendations for USAID/Ethiopia related to the Transform program and future design/programming? 1.4. Evaluation Design and Methodology 1.4.1. Study Settings The baseline survey covers the following eight regional states of Ethiopia: Tigray, Amhara, Oromia, SNNPR, Gambella, Afar, Somali, and Benishangul-Gumuz. The Transform: MELA team, with support from the Transform implementing partners (IPs), USAID/Ethiopia, and the GOE/FMOH, determined the scope of data collection in the intervention and non-intervention areas within each state in order to track progress on key measures related to MNCH/FP. Figures 1a and 1b show the locations from which the surveyed household population was drawn in the eight regions of Ethiopia. Figure 1a. Study Areas Covered by the Midline Evaluation, March 2019 USAID/ Ethiopia Transform Program Midline Evaluation Report Page 16 Figure 1b: Map of Surveyed Locations 1.4.2. Design The midline evaluation of Transform program effectiveness is based on the analysis of proportions between the Transform baseline survey (2017-2018) and Transform midline survey (2019-2020). The DID design allows for stronger causal inferences (controlled for confounding variables) than a straightforward analysis of differences over time. The Transform baseline survey and Transform midline survey both generated data on key indicators from intervention and non￾Transform areas. Using the two waves of survey data, the analysis team was able to compare the change over time, from baseline to midline, in the intervention areas to the change over the same period in the non-intervention areas. The results indicate the extent to which outcomes have changed during the period of Activity (using data from the intervention areas), taking into account the secular or non-program related factors that may have had some impact on outcomes even in the absence of the program (using data from the non-intervention areas). The Transform program ensured that intervention and non-intervention areas were comparable at the baseline study. Figure 2 below provides an example that illustrates how DID is estimated. USAID/ Ethiopia Transform Program Midline Evaluation Report Page 17 Figure 2: Difference-in-Differences (DID) Illustration In addition to the proportions and DID analyses to evaluate effectiveness, the evaluation relies on the KIIs, focus group discussions (FGDs), and health facility analyses to explain observed findings on each of the core program activities. 1.4.3. Methods 1.4.3.1. Sampling Design The midline survey is a cross-sectional study that followed a two-stage sampling methodology. First, prior to sampling, lists of Transform program targeted woredas in the eight regions were obtained from the USAID/Ethiopia and Transform IPs. These included 359 woredas in the Transform: PHC intervention regions, 58 woredas in the Transform: HDR intervention regions, and 40 woredas (within the eight Transform intervention regions, except Dire Dawa) receiving support from the Transform: WASH Activity at the time of the survey. The first stage of random sampling identified health posts/kebeles in the intervention woredas as primary sampling units, drawing from the list of health posts in those regions. Health posts were selected using a simple random sampling technique. An optimal size of 30 households (HHs) per kebele was used to determine the number of kebeles needed for each region. The second stage of the sampling methodology entailed the random selection of households from the sampled kebeles. Two to three Gotts were randomly selected from each sampled kebele and a fresh list of households for each selected Gott was prepared, which served as a sampling frame to select 30 households per kebele. The key eligibility criterion for household selection was a woman age 15-49, regardless of marital status, having residential status there. If more than one such eligible respondent was a member of a selected household, the enumeration team used a Kish Grid method to randomly select one respondent among the eligible women. USAID/ Ethiopia Transform Program Midline Evaluation Report Page 18 The survey design included sampling from comparison woredas that did not receive the Transform interventions. The comparison woredas were randomly selected from the non￾intervention woredas in each region, to ensure that comparison woredas were selected from each zone of a region. Woredas adjacent to the selected intervention woredas were intentionally excluded to minimize potential spillover effects, and the number of comparison woredas was determined with the goal of achieving the desired intervention-to-comparison ratio. The two￾stage random sampling procedure for selecting kebeles, and then households within kebeles, proceeded in the same manner in the comparison woredas as in the intervention woredas. The sample size for intervention areas in each region was determined using power calculations with a 95 percent confidence interval and 80 percent statistical power, anticipating a potential 10-percent difference in outcomes between baseline and midline. Sample size for the comparison sites was determined assuming a four-to-one ratio (4:1) between intervention and comparison groups. Hence, the calculation yielded a total of 6,118 households with women age 15-49 (4,890 from intervention and 1,228 from non-intervention sites). As the assessment also considered studying the effects of the overlapping of two Transform activities on the outcome of interest, oversampling of overlapping woredas was considered during sample size determination to ensure representativeness of sampled woredas. More specifically, Transform: WASH overlapping woredas with Transform: PHC or Transform: HDR were oversampled to ensure representativeness of sample for Transform: WASH intervention sites. Hence, the total sample size became 6,360 households, to be collected from 212 kebeles (170 from intervention and 42 from non-intervention sites). In addition to the household survey, the evaluation team also collected data from primary health care facilities through structured interviews with health service providers, including HEWs, Maternal and Child Health Care Wards for each selected Health Center (HC), and at hospitals in order to see the linkage between supply (health care service availability) and demand (health service utilization) in the sampled areas and assess the effectiveness of health service delivery in the study areas. Hence all the primary health care facilities – a health post, a health center, and a primary hospital – designated to serve the study population were identified and included in the study. The health post serving the population in the sampled kebele was automatically included in the study, the catchment health center linked with the selected health post was included, and the primary hospital attached with the selected health center was also included in the study. As a result, a total of 309 health facilities from the intervention woredas (144 health centers, 141 health posts, and 24 primary hospitals) and 74 health facilities from the comparison woredas (36 health posts, 34 HCs, 4 primary hospitals) were covered by the assessment. For the qualitative study, woredas in intervention areas were purposively selected, for KIIs and FGDs, from those that received and/or did not receive performance improvement grants based on their performance. A total of 89 KIIs were conducted, which included a range of stakeholders USAID/ Ethiopia Transform Program Midline Evaluation Report Page 19 at the local, regional, and national levels. Table 1 below provides an overview of the number and categories of KIs across the three Transform Activities. Table 1. Summary of Key Informant Interviews Category Transform: PHC Transform: HDR Transform: WASH Federal Ministry of Health 2KIIs: Health Extension Program and Primary Health Care, MCH Directorates 1KII - Health Special Support Directorate 1KII - Hygiene and Environmental Health Directorate IP management and technical team at central level 3 KIIs 2 KIIs 2KIIs IP’s Regional team 4 KIIs – in four regions 4 KIIs – in four regions 4KIIs Regional government bureaus 4 KIIs – RHBs 4 KIIs - the four RHBs 4KIIs - Regional Water Bureaus/WASH Coordinators Woreda health office 18 KIIs – Heads of Woreda Health Offices 12 KIIs – Heads of WoHOs 12KIIs Sales agents, retailers 12KIIs and case stories 1.5. Data Collection Tools Transform: MELA, in collaboration with the evaluation IPs (Transform: PHC, Transform: HDR, and Transform: WASH) and local consultants, conducted field level data collection covering the eight Transform targeted regions in late 2019. The data were collected using smartphones for the household survey; paper-based questionnaires for the health facility surveys; and discussion guide and audio recorders for KIIs and FGDs. 1.5.1. Data Collection Each data collection team included an average of five interviewers (enumerators) and one supervisor. Prior to the data collection, the survey team, with the assistance of kebele guides, observed and demarcated the boundaries of the study kebeles, identified the available number of Gotts in the kebeles, and listed them on the Gott registration form, from which two to three Gotts were randomly selected for inclusion in the survey. The survey team registered all households within the selected Gotts using a household listing form in order to establish a sampling frame to select 30 households per kebele. The household listing was done either by USAID/ Ethiopia Transform Program Midline Evaluation Report Page 20 going from house-to-house and enumerating members of the households or by reviewing pre￾registered household listingsfor each Gott whenever available. Guides from the local community were recruited to assist interviewers in ensuring that all households were covered, and field supervisors performed random checks to ensure accuracy and coverage of data. Once households were listed, the selection of respondents was done using a random number function. If the selected household had more than one eligible respondent, the survey team used a Kish Grid methodology to randomly select only one respondent to be interviewed. The evaluation team conducted an extensive document review that provided initial evidence to the evaluation in addition to validating the evaluation findings. Some of the documents reviewed include: the Transform program documents, Transform IPs’ award agreements, project design documents, workplans, performance management plans, quarterly reports, annual performance reports, monitoring and evaluation (M&E data), Mini- DHS - 2019, OWNP program documents, and other related documents including Transform Activity assessments such as the random follow-up reports. The evaluations team also conducted one-on-one interviews with a variety of stakeholders involved and/or familiar with Transform program implementation from the FMOH, USAID/Ethiopia, RHBs, Transform IP staff (head office and field team), Ministry of Water and Resources, Regional Bureau of Water, woreda health offices, health facilities, and woreda water desk offices. 1.6. Data Quality Assurance and Management To assure high-quality data, a four-day training was conducted in Addis Ababa for 26 supervisors and field coordinators and then cascaded to 101 data collectors in Addis Ababa and in four regional capitals of Ethiopia namely: Dire Dawa, Bahir Dar, Mekelle, and Hawassa. Data collectors had at least a BA/BSc degree, were fluent in both Amharic and English, had prior experience collecting data, and were fluent in other regional languages as necessary including Afarigna, Afan Oromo and Tigrigna. Supervisors had an MA/MSc in a health or social science field and previous experience supervising studies. Most of the enumerators and supervisors were part of the baseline survey, which increased familiarity with the questionnaire and process. The training covered topics such as research ethics in the field, rights of human subjects during research, research methodology and protocol, sampling procedures, informed consent, data collection tools, interviewing techniques, data management, security and quality, and gender considerations during data collection. The structure of the training included presentations, role￾playing, review sessions and review of survey instruments. Once the data collectors were trained on the paper-based data collection tool, they received training on how to collect data using the electronic data collection template designed through an Open Data Kit (ODK) and upload the collected data to the server. USAID/ Ethiopia Transform Program Midline Evaluation Report Page 21 The survey tools were reviewed and pre-tested by data collectors in the field in all languages during the training before the commencement of data collection in order to ensure consistency with the baseline survey. Once the data collection began, survey coordinators worked across teams to facilitate sensitization of regional and local leadership, field logistics, and to provide the first level of data quality assurance. The Transform: MELA team provided the second layer of quality assurance in the field by supervising the data collection in all eight Transform program regions. 1.7. Data Analysis Field level data checking and verification was done to immediately identify and address any issues during data collection, a process that involved both supervisors and data collectors. After completion of data collection, the analysis team did further data cleaning and validation, addressed refusals and non-responses, and developed an organized dataset suitable for analysis. The data analysis was conducted using SPSS and STATA software. Once the data were prepared for analysis, the team evaluated distributions and conducted bivariate analyses to determine patterns regarding program indicators. The evaluation team additionally performed Pearson’s chi-square tests to assess the statistical significance of differences between the intervention and comparison areas in the midline survey data. Next, analysts merged the midline data with the baseline data in order to conduct the proportions and DID analyses described above. The analysis of proportions compared changes in proportions between baseline and midline for both Transform program intervention sites and non￾intervention sites. The DID analyses determine, with documented statistical precision, the extent to which change occurred in the intervention areas from the baseline period to the midline period, accounting for broader changes that also affected the non-intervention areas. In addition, the analyses included controls for a number of basic demographic features including age, marital status, level of education, and paid work status. The analysis also considered women’s empowerment indicators (women’s participation in their own health care decisions) and CBHI as covariates. The analyses focused on key indicators, such as modern family planning utilization, antenatal care utilization, delivery by a skilled birth attendant (SBA), child vaccination, women’s decision making, and availability of improved sanitation facilities. The qualitative data analysis for KIIs and FGDs involved transcribing the audio recorded interviews and translating to English. Then, a codebook was developed based on the evaluation questions and the transcriptions were coded and analyzed using Atlas.ti software. Finally, the evaluation team relied on observational data and document review to conduct a strength and gaps analysis, cataloguing areas where the program is on track and areas where USAID/ Ethiopia Transform Program Midline Evaluation Report Page 22 barriers and challenges persist. A geospatial analysis was conducted to assess the distributions and supply of MNCH/FP services. 1.8. Ethical Procedures Approval to conduct the survey was obtained from the Ethiopian Public Health Institute Scientific and Ethical Review Board. Data collectors and supervisors were trained in ethical research procedures including informed consent, privacy of participants, and confidentiality. The study followed standard ethical procedures. At all surveyed households, permission was obtained from the head of household and consent granted by the participant prior to the interview taking place. For women under the age of 18, additional parental permission and participant assent were obtained prior to data collection. After the completion of data collection, data were stored with no personally identifiable participant information. 1.9. Limitations to the Study Three challenges limited the evaluation methodology: 1) Non-Transform programs operate in both the intervention and non-intervention areas, and it was difficult to isolate confounding factors across intervention and non￾intervention areas. The DID analysis was used to isolate confounding factors under these circumstances. Thus, it is difficult to definitively discern the results and impact of other woreda-level, GOE/FMOH, or donor-financed programs on achieving MNCH/FP targets in the Transform geographies in the context of development programming and effectiveness. The evaluation team relied on the DID and causality tracking strategies analyses to overcome these challenges. The DID results should be interpreted bearing in mind these considerations. 2) The absence of an enumeration area (EA) list from the Ethiopian Central Statistical Agency (CSA) imposed constraints on the sampling frame. Transform: MELA utilized kebele guides to clarify and identify boundaries of the selected kebeles and to ensure the complete listing of households in selected Gotts.4 3) Security concerns and inaccessibility impeded data collection in some enumeration areas. Sampling with replacement was employed to address these challenges. Being midline, this evaluation focused on assessing progress towards intended results of the Transform program as opposed to measuring the full achievement of the expected results. It is essential to take note of this when interpreting results from this evaluation. 4 In Ethiopia, a Gott is a small, informal administrative unit, equivalent to a neighborhood or block. Several Gotts comprise a Kebele. USAID/ Ethiopia Transform Program Midline Evaluation Report Page 23 2. EVALUATION FINDINGS 2.1. RELEVANCE Consistent with USAID’s approach to evaluations, one key component of the midline evaluation is the relevance of the Transform program intervention. Relevance is gauged by the appropriateness of the program in four different ways, as the findings below are organized. First, is the program in alignment with the objectives and targets of the GOE, as well as USAID’s strategic priorities regarding MNCH/FP and WASH? Second, is the program contextually and culturally appropriate to the targeted regions? Third, is the geographic targeting of the program appropriate and suited to local needs? Fourth, do stakeholders perceive the program as useful, consistent with needs, and a complement to the GOE/FMOH and other local efforts regarding MNCH/FP and WASH? In addition to addressing these considerations related to relevance, the report outlines strengths, lessons learned, and missed opportunities up to the midline and proposes recommendations for the subsequent period of the Activity and for future programming. The following subsections bring together information from each of the core program Activities – Transform: PHC, Transform: HDR, and Transform: WASH – in evaluating program relevance as vehicles by which the Transform program achieves its intended results. The findings for relevance are based primarily on document review and key informant interviews. 2.1.1. Alignment with Strategic Priorities Transform: PHC Activity: A desk review of the Transform: PHC literature5 and related studies, policies, and guidelines confirmed that the Activity closely adheres to the strategic objectives of the FMOH as well as to the goals of RHBs. In particular, the co-creation process ensured from the design phase of the Activity that there would be clear understanding, buy-in by the GOE, and local ownership of this Activity. The evaluation team also reviewed the GOE/FMOH Health Sector Transformation Plan (HSTP 1) and found that the Activity has a strong alignment with national priorities. The annual ToC reviews were an additional factor that has reinforced alignment and ownership through participatory, annual joint planning exercises in which RHBs and woredas jointly review needs, review the work plan, adjust interventions, and roll out these adjustments according to regional and woreda health sector priorities. Transform: HDR Activity: HDR also began with several protocols designed to ensure its alignment with USAID/Ethiopia and GOE/FMOH MNCH/FP strategic priorities.6 In fact, Transform: HDR’s strategy is derived from the GOE/FMOH’s national priorities in order to explicitly address the 5 Annual workplans for 2017, 2018 & 2019, and quarterly and annual performance reports of the past three years, including other strategy documents. 6 A review of documents such as Activity description, Annual work plans 2017, 2018 & 2019; and quarterly and annual performance reports, HSTP, all demonstrated alignment to national priorities. USAID/ Ethiopia Transform Program Midline Evaluation Report Page 24 GOE/FMOH’s HSTP7 and the needs of the country. The design of the Transform: HDR Activity started with a highly participatory consultation process based on data generated from the national and regional level that allowed the GOE/FMOH to align the design with its priorities and to bolster host-country ownership of Transform: HDR. The evaluation team found that the design of Transform: HDR set a good course for the Activity as it started with consultation with national and regional stakeholders. An additional factor that has reinforced alignment and ownership is the participatory annual planning exercise in which sector offices jointly plan work assignments and adjust interventions and their roll-out according to regional health sector priorities. Transform: WASH Activity: The Transform: WASH theory of change/results framework, implementation science, and associated performance measurements are closely aligned with the USAID WASH Activity design document.8 In addition, the evaluation team saw evidence that the WASH Activity is consistent with, and making a positive contribution toward, the GOE/FMOH’s national and regional WASH priorities. The intended results may have been set too high from the outset, however, as adaptation of new behaviors and technologies is proving to be a slower than anticipated process even as it advances in a positive direction. Future programming designs and the associated performance measures may need to be set within the Activity’s level of contribution and influence to allow for the ease of measuring the cause and effect of its interventions. 2.1.2. Contextual and Cultural Appropriateness Transform: PHC: The PHC strategies, approaches, and interventions are contextually and culturally appropriate to the project contexts in which they have been implemented. In particular, Transform: PHC has instituted annual ToC reviews based on evidence generated through consultations with key stakeholders including FMOH, RHBs, woreda health offices, universities, and other private actors. Transform: HDR: The evaluation team observed that Transform: HDR continues to be relevant in supporting government priorities and the mechanism’s flexible response to emergencies have been adapted to the cultural context. For example, during a measles outbreak, the Transform 7 Document review of HSTP document. 8 USAID NFO no. RFA-663-16-000007 Transform/Water, Sanitation, Hygiene (WASH) Page. 17-19 “The Transform: HDR project provided technical support to the woreda and health professionals through capacity building in a way that we all understood. Transform: HDR’s supportive supervision, outreach, and mobile health services respond well to the culturally sensitive needs of the communities and is efficient, and we appreciate that. Transform: HDR trains only cluster supervisors; each cluster supervisor, in turn, trains all the health extension workers under their supervision. This approach enabled the project to train many professionals with a limited budget and eventually strengthen the system.” Somali RHB Respondent USAID/ Ethiopia Transform Program Midline Evaluation Report Page 25 team set up a mobile vaccination campaign to reach migrating pastoralists. Regional and woreda stakeholders interviewed perceived that Transform: HDR interventions are being implemented in line with the strategies and priorities of the regional health bureaus and local contexts. In addition, HDR has embraced a role in supportive supervision, outreach, and mobile health services that respond well to the culturally sensitive needs of the communities. One participant noted: Transform: WASH: Regarding WASH, a key informant who serves as a staff member in the FMOH best summarized the attention of the WASH Activity to contextual and cultural sensitivities in the following quote: “Several steps have been taken by Transform: WASH to respond to the local context, and to make the interventions a good contextual as well as cultural fit into the local context of target intervention areas. Prototyping of new and existing sanitation products, design summits, and design competitions for sanitation products and services were some of the activities carried out as part of the Activity’s actions in this regard.” 2.1.3. Current Geographic Targeting Transform: PHC: The Activity focuses on four regions of Ethiopia: 1) Amhara, 2) Oromia, 3) SNNP, and 4) Tigray. The evaluation team found numerous development programs taking place in both USAID/Ethiopia Transform: PHC Activity areas and the surrounding areas outside the boundaries of the evaluated regions. With other partner-funded programs operating in both Transform: PHC intervention sites and non-USAID-Transform sites, it was difficult to definitively discern the results and impact of Transform program activities. However, the evaluation team relied on the DID and causality tracking strategies analysis to assess the contribution to development results. Transform: HDR: The Activity focuses on the four developing regional states in Ethiopia: 1) Afar, 2) Benishangul-Gumuz, 3) Gambella, and 4) Somali. The rural, low population density contexts created geographic spread and dilution of program resources between the regional center and intervention woredas that remain a challenge. This was especially difficult given the limited technical team spread across so many specialty areas coupled with the difficulties of identifying and recruiting qualified staff to work at the regional and sub-regional levels. For example, the project team has to travel more than 400 km from the Benishangul-Gumuz regional center to Wombera woreda. It is important to appreciate the link between limited human resources and competencies within local government health structures and the geographic targeting of the Activity. Thus, whereas the regions identified for Transform: HDR Activity interventions are indeed highly relevant, the geographic context complicated Activity implementation in ways described later in the report. Transform: HDR adapted the Activity to this challenge in terms of supporting selected districts with full packages and minimal support to the rest. This included refocusing the effort to 20 selected woredas from the 59 woredas and identifying health facilities to build for COE status with the provision of full packages of the support. USAID/ Ethiopia Transform Program Midline Evaluation Report Page 26 Transform: WASH: The Activity is also subject to concerns regarding the wide spread of its geographic targeting. At the regional level, a SNNP WASH focal person noted that the Activity should have focused on a few geographic areas to ensure the testing and roll out of cost-effective and appropriate WASH activities designed for scaling- up. From the central perspective, however, the geographic targeting serves the GOE/FMOH’s objectives. A key FMOH Sanitation and Hygiene Director said: “The GOE/FMOH believes that Transform: WASH had a good pilot roll-out plan precisely because from the outset the program set out to prove that sanitation products could be marketed in diverse geographic locations of the country and adapted to both agrarian and pastoralist economies. From the central perspective, the learning emerging from the implementation of sanitation marketing activities in those areas has been and will continue to be instrumental in shaping policy directions.” 2.1.4. Stakeholder Perceptions of the Transform Program Transform: PHC: Stakeholders, including communities, generally agreed that Transform: PHC work is relevant to their needs and the woreda Transformation agenda. In particular, the co￾creation process helped many stakeholders to see Transform: PHC as an important strategic partner (see text box below). There is a strong sense of ownership of the Transform: PHC Activity from the government stakeholders at the GOE/FMOH, regional, and woreda levels. It appears, however, that there is some misunderstanding in some woredas about the expected transformative outcomes the Activity can support, which shapes views of program relevance when stakeholders see only minimal program Activity in their local areas. Some regional stakeholders were disappointed to learn that the Activity did not cover the entire GOE/FMOH HSTP. A respondent from Oromia RHB summed up what other KIIs also expressed: “The project gives the impression that it will help the government transform the health sector. But its support vs. RHB expectations regarding project performance at the operational level is not adequate. The support is spread too thin to yield meaningful, transformational results. Also, the thinly-dispersed support can’t help us draw useful lessons.” Based upon this and other KII inputs, there is a need to create a common understanding on the scope of the Transform program and activities and to support FMOH/RHB partners in developing a suitable long-term strategic gain. “…. As the government is working on reducing maternal and child mortality today more than ever, Transform PHC is also focusing on this point with special emphasis on newborn care, family planning, etc. in our region and woredas. This is a fundamental point! Reproductive maternal newborn and child health is the main concern of the ministry of health today. These are the basics that maternal and child mortality reduction can be achieved. This is what relevance means to us. And, we can realize that the Transform: PHC project is implemented in an integrated manner with our program. Therefore, the relevance of Transform: PHC is not debatable…” --- Gorchie Woreda health office head, SNNP echoing the views of most KIs. USAID/ Ethiopia Transform Program Midline Evaluation Report Page 27 Transform: HDR: Regarding the Transform: HDR Activity, the evaluation team found that the design set a good course, as it started with consultation with national and regional stakeholders. A second factor that has reinforced ownership is the participatory annual joint planning exercise where RHBs and woreda and other sector stakeholders jointly plan work assignments and adjust interventions and their roll-out according to regional health sector priorities. The annual planning exercise between Transform: HDR and regions and woredas ensures continued relevance of the Activity’s interventions, strategies, and approaches within regional contexts. In addition, a key factor that has led to local ownership is the institutional capacity support and continuous TA. Many stakeholders interviewed commented that the IP team was responsive to the region’s requirements and was able to respond at important moments, such as during periods of emergencies. One representative example is the following statement from a stakeholder in Afar Mille Woreda: Transform: WASH: The Activity has involved a wide range of stakeholders from national to sub￾national levels at the design, planning, and implementation stages. Moreover, to disseminate and share learning at the sub-national level and to drive the scale-up of sanitations innovations, regional working alliance platforms in all intervention regions were supported. Some KIIs felt that the presence of “Water” in the WASH terminology creates a misrepresentation of the Transform: WASH scope, as the activities do not focus explicitly on water. Overall, a common perspective among stakeholders interviewed for the evaluation was that they commended the Activity for being responsive to market needs and for the involvement of the sanitation enterprise. However, the program’s core business model activities – door-to-door simple toilet upgrades (product installation service), basic manufacturing and installation, advanced toilet manufacturing and construction, and plastic slab sales and support – were not always congruent with the needs of the communities. 2.1.5. Strengths, Lessons Learned, and Missed Opportunities Regarding Program Relevance Transform: PHC offers three important strengths and lessons: ▪ There is a strong alignment between Transform: PHC strategies, approaches and results with the FMOH and USAID/Ethiopia’s strategic priorities for MNCH/FP, hence, contributing to the realization of the HSTP priorities. “Transform: HDR is our project and is relevant in its support to outreach and mobile health services including emergencies – we encourage this activity to be continued and scaled up.” USAID/ Ethiopia Transform Program Midline Evaluation Report Page 28 ▪ A co-creation approach during the design of the Transform: PHC allowed for strategic alignment of the Activity with FMOH and RHB priorities and ensured key stakeholder ownership of the process and Activity. ▪ The use of approaches and strategies like the ToCs, has ensured that Transform: PHC interventions continue to be relevant and contextually and culturally appropriate to the Activity’s context. Transform: HDR also offers three useful insights and lessons learned: ▪ The interventions have provided strong on-site, post-training follow-up coaching and mentoring of health workers. The continued close coordination of activities and tailored technical assistance provided to woredas have also ensured that programs are relevant to meeting the requirements of the communities. ▪ TA provision has been a key source of local ownership. ▪ Intervention areas that are too geographically vast can undermine the perceived relevance of the Activity from the perspective of regional and sub-regional leaders in the sense that those leaders do not see a strong program presence in their local areas. Transform: WASH also has an ambitious agenda that has yielded both advantages and missed opportunities in terms of relevance including the following: ▪ The Activity’s four business models – door-to-door simple toilet upgrades (product installation service), basic manufacturing and installation, advanced toilet manufacturing and construction, and plastic slab sales and support – are scalable from the local to the national level, but they were not always congruent with the needs of the community, as KIs indicated. Each of these models aimed to account for available sanitation products and services as well as the market segmentation of consumers, but with varying results in the intervention regions. ▪ The Activity has a high degree of local ownership. KIs and business leaders noted that Transform: WASH provided vital, concrete, and relevant marketing, pricing, and supply chain skills for suppliers. When KIs were asked about these elements of the program, the most common response was that they felt they had the tools and skills to take ownership. ▪ The wide geographic spread of Transform: WASH activities poses a challenge. ▪ Key Informants note that steps have been taken by Transform: WASH to respond to the local context and to make the interventions culturally appropriate. Prototyping of new and existing sanitation products, design summits, and design competitions for sanitation products and services have been effective tools in this regard. USAID/ Ethiopia Transform Program Midline Evaluation Report Page 29 2.1.6. Recommendations and Implications for Program Relevance Evaluating the collective results from the Transform: PHC, HDR, and WASH activities brings attention to several recommendations and implications regarding the relevance of Transform and related programs going forward including recommendations to: ▪ Strengthen collaboration with other external partners involved in public health initiatives in the same regions to mitigate programmatic overlap and to ensure that stakeholders have appropriate expectations of each program Activity. Interventions are outside the scope of the Transform program that are addressing priority health needs in targeted regions and woredas. For example, some woredas mentioned that their priority health problems such as TB, malaria, nutrition for under-five children, should be addressed by Transform: PHC. Collaboration could help to address these concerns in ways that fortify the mission to address MNHC/FP and WASH needs, while also helping to mitigate program overlap. ▪ Continue to invest in strong relationships with FMOH and local stakeholders from the design phase of activities, both to ensure their contextual and cultural appropriateness and to reinforce local ownership. ▪ Consider limiting the intervention areas to smaller geographic/targeted areas of intervention, particularly if the intervention areas include vast, rural zones. ▪ Invest in outreach and mobile support components of the Transform program. Doing so will help to ensure that intervention activities reflect updates to the annual ToCs and that local stakeholders remain engaged in program adaptations. ▪ There have been good results despite the immense unmet needs in the four developing regions served by the HDR Activity, but perceived program relevance suffers because the initiatives are diluted over the vast area being covered. If funding is available, USAID/Ethiopia might consider expanding the activities to ensure greater perceived relevance. In addition, program relevance could be enhanced by taking the interventions to some non-Transform: HDR sites, where the health situation has continued to deteriorate as indicated by some of the low outcomes observed including expanding collaboration with additional non-Transform partners. ▪ There is a need to create a common understanding of the scope of the Transform program and hence each Activity’s and support FMOH/RHB partners to focus on an appropriate and relevant long-term strategic gain. USAID/ Ethiopia Transform Program Midline Evaluation Report Page 30 2.2. EFFICIENCY Transform program efficiency is determined by the management structure, administrative and logistics arrangements, networking and partnership strategies, resource allocations, and expenditures. If those aspects of program implementation are executed well, the program is better able to attain its intended results within the period of performance and the allotted budget. To evaluate the efficiency of the Transform program at its mid-point, the evaluation team considered the program’s contributions to programmatic results, the effectiveness of the Transform management structure, the nature of its partnerships, and the extent to which it employs innovative approaches to attain better results. The report draws from each of the three Transform activities to develop the findings, and it offers recommendations to conclude this section. The information comes primarily from Key Informant Interviews and a review of progress on program indicators. 2.2.1. Contributions to Programmatic Results Transform: PHC Activity: The evaluation team analyzed secondary data from the Transform: PHC Activity – including quarterly and annual performance reports, terms of reference reports, random follow-up reports, evaluations, and assessment reports – to determine the extent of programmatic impact at the woreda and regional levels. The results indicate that: ● At the startup of the intervention (Year 1, 2017), only 12% of the woredas were considered to be “high-performing”, but by Year 3 (2019), that figure had risen to 29%9 . ● The proportion of woredas labeled as medium-performing rose from 29% to 40%. ● In the Amhara region, more than half (52%) of the medium-performing woredas shifted to high-performing status, though about 22% of the medium-performing woredas declined to low-performing woredas. Furthermore, use of ToC exercises have contributed positively to Activity operations, and the strength and gap analysis conducted by the evaluation team suggests that the Transform: PHC Activity has integrated a periodic outreach tool for hard-to-reach communities into its portfolio, which helps to reinforce programmatic results in an efficient manner. Transform: HDR: The HDR Activity’s support of a ToT program and its outreach to regional universities and labs creates new stakeholders who can magnify the efforts of the implementing partner and increase program efficiency. The Mobile Health Outreach program similarly allows the program to do more with less by extending benefits to underserved areas without the need for program bases. Transform: HDR assistance to the GOE and its DHIS2 initiative helps to strengthen government capacity, which again improves program efficiency by increasing the number of capable stakeholders who can advance program objectives. Key interventions 9 Transform: PHC used 14 evaluation criteria to assess the performance of woredas which were agreed to with FMOH and USAID. USAID/ Ethiopia Transform Program Midline Evaluation Report Page 31 conducted by Transform: HDR to date include a management performance decision-making and resource allocation initiative for improved outcomes. Better decision-making improves the use of resources as well as the timeframes for the implementation of activities, thus enhancing efficiency. On the other hand, the four developing regions served by Transform: HDR (Somali, Afar, Gambella, and Benishangul-Gumuz) are widely known to be emergency-prone due to natural disasters and other political and security issues, and the lack of crisis response funds early in the project constituted a missed opportunity to efficiently contribute to programming objectives. Some funds came later, and an emergency fund has now been established, much to the relief of numerous KIs who contributed their perspective. Transform: WASH: Transform: WASH has successfully engaged with the major private sanitation enterprises and sales agents. This has been a key step in getting activities up and running and ensuring the efficient implementation of planned initiatives. Nevertheless, Transform: WASH continues to treat the design of sanitation enterprises as an iterative process given that the enterprises remain less profitable for entrepreneurs. Failure to quickly land on the most effective enterprise design undermines the efficiency of the program to some degree. In the meantime, 33 woredas have developed WASH procurement plans for infrastructure and have effectively managed timely bidding processes to procure contractors for these infrastructure projects. As a result of capacity-building support from the Activity, the woreda WASH teams were able to articulate clear sanitation planning budgets. Transform: WASH utilizes the existing GOE health delivery structure that relies on health extension workers (HEWs) and local radio to create community awareness for sanitation and demand creation for WASH products and services. While in principle this should have been an efficient way to reach health consumers who seek health services at health posts, over half of the locally based KIs, along with many household respondents, said that because of the HEWs’ heavy workload and substantial clinical care and primary prevention duties, they rarely have the time to engage in lengthy discussions with clients about the benefits of these new and improved sanitation products. 2.2.2. Management Structure Transform: PHC: Most of the Transform: PHC management systems include staffing, on-site GOE￾led staff training and mentoring, joint deployment, co-location of expert long- and short-term TA alongside government health teams, and logistics. Each of these components contributes to efficiency and supports the achievement of program results. The ToC exercise has helped in coordinating the participation of management stakeholders for improved results. Another important approach introduced is the use of evidence from “random follow-ups” by technical USAID/ Ethiopia Transform Program Midline Evaluation Report Page 32 teams to support Activity program decision-making and support services provided. These visits reinforced in-service learning on all of the interventions and provided an invaluable opportunity for staff and supervisors to gauge health facility requirements. Activity focal points at government structures were found to be working, with the GOE helping to facilitate communication, coordination, and Activity implementation. The evaluation found that, overall, Transform: PHC has fostered strong collaboration with the GOE at all levels. Transform: HDR: Most of the implementation management systems reviewed, including work plans, budgets, management information systems and logistics, while adequate to initiate program activities, may need to be strengthened and refined for the remaining period of Transform: HDR Activity. There is also a need to rethink the Transform: HDR technical staffing structure. Based on the evaluation team’s observations and discussion with various actors, the challenge of getting qualified and competent expertise with strong local context knowledge stands out in pastoral regions. The current Transform: HDR staffing structure may need to be revisited to effectively support regional and local health systems to deliver health services and to strengthen strategic partnerships with other key actors in recognition of their niches and comparative advantages. Finally, while Transform: HDR is supporting the DHIS2 rollout, greater technical assistance is required to strengthen data utilization, especially for learning and decision-making. Of particular importance, data concerning the health-seeking patterns and outcomes of pastoral communities should be reviewed and used to refine service delivery approaches for this hard-to-reach, underserved population. Capacity building has been an effective component of Transform: HDR’s management structure. As one KII from Afar noted: Recruitment and deployment of adequate staff remains a pressing need. Transform: WASH: Transform: WASH has its main coordination offices (hubs) in most of the regions except Afar, Gambella, and Benishangul-Gumuz, where the hub is based in Amhara, Oromia, SNNP, Tigray, and Somali. Technical assistance for the implementation of interventions at the woreda level is mainly deployed from the regional center. This approach was found to have several drawbacks, despite it being more cost-effective than having full operating teams in each region. The main negative feedback voiced by multiple retailers/suppliers and sales agents was that, given the competing demands on the Transform: WASH team, it was often difficult to get on-site consultations. “Tailored capacity building support on Leadership, Management, and Governance development and execution of tailored trainings suitable for officers/leaders at all leadership levels in developing regions health structure will definitely contribute to improved planning, executing, and evaluating health priorities of the regions with improved accountability. This may also include establishing/strengthening regional training centers in collaboration with local universities. This will also help strengthen the primary health care units (PHCUs) that are expected to contribute to the goal of universal health care (UHC) in the developing regions’ woredas.” USAID/ Ethiopia Transform Program Midline Evaluation Report Page 33 On the positive side, the central level Transform policy work has led to efficient “fast-tracking” of program-approved sanitation products by the management structure within the FMOH. For example, the Transform: WASH Activity is working with government counterparts to revise some key taxation regulations that have limited products from entering the marketplace. 2.2.3. Partnerships Transform: PHC: The PHC Activity has cluster offices that are typically co-located with government health offices, and they jointly implement activities with government counterparts, thereby reducing costs and improving efficiency. The evaluation team found that co-location was effective as it built teamwork between the Transform: PHC and the RHBs, woredas, and zones. The Activity’s field offices -- co-located within government health offices -- the closeness and the team harmony, including reduced field office rental expenses, make more funding available for implementation. The closeness and the team harmony have had a positive effect in ensuring efficiency for needs identification, planning, joint supportive supervision, and execution. Another good example of partnership and sustainability is that Transform: PHC is jointly carrying out Integrated Supportive Supervision Visits with the GOE teams. These visits are conducted by a small team of experienced staff and, the evaluators noted, with a sense of shared responsibility. This ensures skill transfer and efficient utilization of resources. The evaluation team observed two supervisory outreach visits and noted the collegial nature of supervisory visits. Transform: HDR: The Transform: HDR Activity has been too thinly staffed to be able to successfully build meaningful and strategic local partnerships. KIs at five separate woreda locations noted that the team was too small to cover all aspects of the ambitious MNCH/FP agenda. A short-term strategy that could be employed by Transform: HDR is to assign work in the regions, districts, and specific geographies to their central staff, while the field team focuses its support on the regions to ensure a balanced workload between central teams and field level technical support. Transform: HDR partnership with the central level has been relatively promising. For example, Transform: HDR assisted the GOE/DHIS2 to roll out its capacity strengthening support platform. Transform: HDR is also working with GOE to enhance management performance decision-making and resource allocation for improved outcomes. Transform: WASH: As a result of Transform: WASH’s advocacy support to the GOE/FMOH, hygiene and sanitation products have received tax-exempt status, thereby reducing the time it takes to import and deliver a product to the consumer. Transform: WASH also supported the GOE/FMOH in its work to revise the National Sanitation Marketing Guideline that was last drafted and issued in 2013. The revised guidance will fast-track new product entry and new product development. Transform: WASH has also created solid partnerships with most stakeholders at the national level (with the FMOH) and at the regional and woreda levels to facilitate the implementation of interventions. This structuring of the relationship with government staff has accelerated the planning and execution of OWNP at the woreda level. USAID/ Ethiopia Transform Program Midline Evaluation Report Page 34 Though Transform: WASH is focusing on sanitation marketing, it has not yet developed strategic partnerships with other stakeholders involved in water supply and hygiene-related interventions to provide a complete package of water and sanitation services to improve WASH outcomes. In addition, Transform: WASH has not yet fortified its relationship with financial institutions, which could be helpful to ensure that loans are accessible for entrepreneurs wishing to conduct business in the sanitation market. 2.2.4. Innovative Approaches to Attain Better Results Transform: PHC: In addition to participating in a number of networks and working groups that improve efficiency—including USAID’s Gender Champions Network, the national FP technical working group, Child Health and Development working group, and adolescent and youth health development and technical committees organized by the FMOH—Transform: PHC has created additional opportunities for efficiency. For example, Transform: PHC has led joint planning exercises and established new guidelines, such as the mentorship guidelines adopted by the GOE. These networking platforms permit Transform: PHC to more broadly share its materials and ideas with stakeholders in an efficient manner. KIIs also indicate that the Transform: PHC Activity has successfully implemented on-site training, adopted fast-reporting mechanisms to communicate results, and developed financial accountability mechanisms for the use of Activity funds. Transform: HDR: The introduction of new technologies to facilitate service delivery, such as the introduction of mobile ultrasound, the simplified gestation estimation tool, and task shifting for middle level health workers was positively received by stakeholders. These innovations are motivating women to attend ANC services despite the costly long-distance travel to health facilities, and they are helping to improve the use and interpretation of essential ANC lab results and the quality of ANC services. For example, the introduction of External Quality Assessments (EQAs) for ANC lab services for syphilis, as well as hepatitis testing, outreach ANC, and diagnostic services using the mobile V-scan have led to increased service uptake. Transform: HDR’s contribution to the mapping of pastoral communities’ mobility routes and destinations has also been an important innovation. This mapping is particularly relevant for the implementation of mobile health services during the seasonal movement of pastoral communities. KIIs indicated that such mapping can help pastoralists and stakeholders pinpoint the locations of water points, grazing lands, animal health points, human health points, schools, marketplaces, and grazing lands, all of which are vital complements to household wellbeing and public health. Assessing the effect of mapping pastoral communities’ mobility routes and destinations on service utilization by comparing with non-intervention areas would be interesting to measure the effectiveness of this approach. Combining this mapping with their existing facilities will help to enhance availability and accessibility of health and other basic services in pastoral regions. This is an innovation that could be usefully expanded. USAID/ Ethiopia Transform Program Midline Evaluation Report Page 35 Transform: WASH: Discussions with KIIs in each region indicate that the sanitation marketing system seems to be active and promising in bridging demand, supply, and delivery models in some regions. In addition, the Transform: WASH Activity has been supporting innovations related to the construction of toilets using local materials such as bamboo. However, there are still supply-related challenges, as the sanitation products are mainly imported from abroad. Transform: WASH established relationships and supported microfinance institutions (MFIs) with seed money to support sanitation enterprises in the intervention areas. Despite the availability of seed money, there were several impediments encountered by Transform: WASH in enabling enterprises to qualify for these loans. A key consideration is whether these enterprises are creditworthy and have the means to pay back the loans. Other obstacles are that interest rates are still high given the uncertainty of borrowers’ credit worthiness and that outlets for credit remain too few in number to provide a critical mass of loans to generate a thriving supply of sanitation goods. One example is the VSLA, organized at the community level to provide loans to households to purchase sanitation products and services. However, VSLA was fully operational in only 45% of the study sites. 2.2.5. Recommendations and Implications for Programmatic Efficiency The Transform program has been largely efficient in the implementation of its activities and its pursuit of intended results. A principal barrier to greater efficiency is the dispersed nature of the program, and related understaffing issues, that plague the Transform: HDR Activity and the Transform: WASH Activity in Transform: HDR-overlapping regions. Other recommendations for refining the Transform program and improving its efficiency build on the core tenets of efficiency cited at the beginning of the section: ▪ Management Structure: o Continue to strengthen government ownership of Transform program interventions, with post-training follow-ups, coaching and mentorship, and supervision that is integrated and done with health office staffs. o Work with central and regional governments to restructure staffing to build stronger partnerships with local stakeholders and health systems. The support should include working with regions to implement full operating teams in each region rather than having coordination take place at higher levels. ▪ Expenditures and Resource Allocation: Expand crisis response funds for emergency-prone areas, which can help to offset the risks that emerge with unexpected events such as natural disasters. ▪ Networking and Partnerships: o Prioritize tailored sanitation awareness messaging efforts among different community segments, and devise ways to improve HEWs’ working conditions in order to meet demand for their services that detract from sanitation awareness efforts. USAID/ Ethiopia Transform Program Midline Evaluation Report Page 36 o Continue to build strong relationships with GOE partners and expand those relationships to financial institutions and other potential service providers that complement program objectives. ▪ Administration: Support the enrollment of the graduation plan for high-performing woredas and an improvement plan for low-performing ones. The support should also include a plan for maintaining model households that graduated NOT to degenerate. ▪ Logistics: o Expand start-up capital for entrepreneurs associated with the Transform program WASH sanitation marketing and ensure that suppliers and entrepreneurs are exposed to the importance of corporate social responsibility. o Expand the mapping of pastoralist services to ensure that rural communities have access to information that can improve the efficient uptake of health services. 2.3. EFFECTIVENESS Effectiveness accounts for the progress that the Transform program has made in achieving desired results. To evaluate the effectiveness of Transform program interventions, the evaluation team drew from the three Transform program activities’ (Transform: PHC, HDR, and WASH) household-level data on key performance indicators collected from 6,350 women of reproductive age across the eight regions of Ethiopia. In order to assess whether the Transform program is on track to achieve the intended results, the evaluation team considered Transform program individual Activity level targets for KPIs to deduce program level effectiveness. The results presented here cover the progress toward outcomes in five areas: 1) family planning, 2) maternal health, 3) newborn health, 4) child health, and 5) cross-cutting (including sanitation and hygiene, community-based health insurance and gender). The evaluation team used analysis of proportions comparing changes in proportions between baseline and midline for both Transform program intervention sites and non-intervention sites. DID analysis was also used to measure the change in Transform program intervention areas between baseline and midline compared with the change achieved in non-Transform program areas. This section also uses information from the three Activity reports to evaluate effectiveness in terms of delivery methods and frequency, stakeholder satisfaction, and other enabling or hindering factors. It closes with recommendations for future programming regarding Transform program effectiveness. USAID/ Ethiopia Transform Program Midline Evaluation Report Page 37 2.3.1. Progress Toward Transform Program Outcomes 2.3.1.1. Respondent Demographic Characteristics Table 2 below presents the distribution of midline survey respondents by background characteristics. The midline survey collected data from 6,350 respondents, all of whom were women of reproductive age (15-49 years), with 5,097 drawn from intervention areas and 1,253 from non-intervention areas. The response rate was 99.8%. The mean age was 30 years old, and the vast majority (82.8% in intervention and 84.3% in non-intervention areas) were below the age of 40. Nearly half of respondents (50.3% in intervention and 50.8% in non-intervention areas) had no formal education, and only 2.9% in intervention and 2.4% in non-intervention areas attended above a secondary level education. In the intervention areas, only 29.3% of respondents had paid work and the remaining 70.7% did not have a job. In the non-intervention areas, 33.4% of the respondents had paid work. Four out of five (80%) respondents were married. In general, there are no statistically significant differences between Transform program intervention area and non-Transform program area respondents with respect to the demographic characteristics. To reduce selection bias and to minimize potential pre-treatment differences between respondents in the intervention and non-intervention areas, the evaluation team employed propensity score matching for the DID analyses that follow. Table 2. Respondent Demographics Characteristics Characteristics Transform Program Intervention Areas Non-intervention Areas (%) N (%) N P-value Age of the Woman 15 – 19 12.6 641 11.2 140 0.173 20 – 24 15.3 778 16.8 211 25 – 29 19.7 1006 21.0 263 30 – 34 17.2 875 18.9 237 35 – 39 18.0 919 16.4 206 40 – 44 9.9 504 8.6 108 45 – 49 7.3 374 7.0 88 Level of Education No formal education 50.3 2565 50.8 637 0.585 Primary (1 - 8) 36.5 1859 37.4 468 Secondary (9 - 12) 10.3 526 9.4 118 Above secondary education 2.9 147 2.4 30 Marital Status Single/Never married 10.6 539 9.0 113 0.055 Married or cohabitating 79.7 4062 82.4 1033 Divorced 5.2 263 5.5 69 Widow 3.5 179 2.2 28 Separated 1.1 54 .8 10 Paid Work Status Paid work 29.3 1491 33.4 419 0.004 No work 70.7 3606 66.6 834 USAID/ Ethiopia Transform Program Midline Evaluation Report Page 38 Note: A significant difference in paid work status exists between Transform program intervention and non-Transform program areas. 2.3.1.2. Analyses of Key MNCH/FP Performance Indicators – Comparisons and DID To assess the progress achieved in Transform program intervention areas, the evaluation team compared outcomes on key MNCH/FP indicators between the baseline (2017) and midline (2019), which were then compared to the changes observed in non-Transform intervention areas using DID analysis. The evaluation team also conducted regression analysis to identify determinants of the MNCH/FP service utilization. However, the comparative progress in health service availability in primary health care facilities in Transform program intervention and non￾Transform program areas does not figure in the analyses due to the limited number of facilities covered during the baseline. The results of the analyses imply the combined influence of the Transform: PHC, HDR, and WASH activities. A. FAMILY PLANNING Family planning represents efforts to limit or space childbirths through the use of contraceptive methods, either modern or traditional. Modern contraceptive methods include female and male sterilization, oral contraceptive pills, intrauterine contraceptive devices (IUD), implants, injectables, female and male condoms, emergency contraception, standard days method (SDM), and lactational amenorrhea (LAM). The rhythm method and withdrawal, as well as folk methods, are grouped as traditional. Changes in Contraceptive Use Indicators As Figure 3.a below shows, most family planning practices improved from baseline to midline in Transform program intervention areas. Modern Contraceptive Prevalence Rate (MCPR) among married women increased significantly from 32.3% at baseline to 35.3% at midline (p￾value<0.01). Use of long-acting family planning method (LAFP) among married women also increased significantly from 8.3% at baseline to 11.1% at midline (p<0.01). The unmet need for family planning, which captures the share of women who wish to benefit from a particular service but who have not had access to that service (i.e., access to contraception), showed slight improvement between the two periods: dropping by 0.1 percentage point (p=0.45). The unmet needs are still high which needs to be addressed in order to improve the reproductive health of women. As Figure 3.b below shows, the use of postpartum family planning (PPFP) methods during the postpartum period also increased, from 22.9% at baseline to 28.7% at midline (p<0.01). Provision of family planning counseling during the postpartum period also increased from 22.9% at baseline to 24.5%, though it is not statistically significant (p = 0.19). Conversely, the family planning indicators generally exhibited declines in non-Transform program areas. USAID/ Ethiopia Transform Program Midline Evaluation Report Page 39 Figure 3(a): MCPR and Unmet Need for FP between Baseline & Midline for Transform Program Intervention Areas and Non-Transform Areas, March 2019 Figure 3(b): Changes in PPFP between Baseline & Midline for Transform Program Intervention Areas and Non-Transform Areas, March 2019 Regarding the types of family planning methods used by women in the study, injectables are the most frequently used method among married women, used by approximately two-thirds of women generally and about 14% post-partum, followed by Implanon and LAFP methods in both Transform program intervention areas and non-Transform areas. IUDs and male condom are the least utilized modern contraceptive methods. Even though improvements were observed in utilization of family planning services in general, the practice was not uniform across regions and there are some catchment areas still facing challenges to bring sound improvement in family planning service utilization. Figures 4.a, 4.b, and 4.c below depict the performance change of MCPR, LAFP, and PPFP use between baseline and mid-term by region. The analysis result shows that Afar, Somali, and Gambella had the lowest 32.3 34.2 8.3 9.6 26.7 27.4 35.3 32.3 11.1 7.9 26.6 28.8 Transform Program Intervention Non-Intervention Transform Program Intervention Non-Intervention Transform Program Intervention Non-Intervention MCPR among Married LAFP among Married Unmet need for FP Baseline Midline 22.9 26.6 22.9 26.7 28.7 28.9 24.5 20.0 Transform Program Intervention Non-Intervention Transform Program Intervention Non-Intervention PPFP PPFP Counselling Baseline Midline USAID/ Ethiopia Transform Program Midline Evaluation Report Page 40 performance at baseline and mid-term compared with all other regions. Benishangul-Gumuz had comparable performance with the four big (agrarian) regions though it is part of the developing regions. The highest performance increase in MCPR was observed in Benishangul-Gumuz (12.8 percentage points) followed by Amhara (4.5 percentage points), whereas MCPR declined in Somali by 6 percentage points and in Tigray by 1.3 percentage points. On the other hand, the highest performance increase in LAFP among married women was seen in SNNP with an increase by 6.3 percentage points, followed by Tigray with an increase by 5.7 percentage points. In contrast, the performance in Amhara declined by 4 percentage points. The highest performance increase in use of contraception after birth was seen in Benishangul-Gumuz which increased by 22.6 percentage points, followed by Amhara, an increase by 11.6 percentage points. However, the highest performance decline was seen in Tigray, by 4.2 percentage points. Figure 4(a): Performance Change on MCPR between Baseline and Midline in Transform Program Intervention Areas Disaggregated by Region, March 2019 Figure 4(b): Performance Change on LAFP Use between Baseline and Midline in Transform Program Intervention Areas Disaggregated by Region, March 2019 42 48.2 45.2 41.4 4.3 4.5 17.9 36.4 42.5 52.7 47.1 40.1 6.9 0.2 16.2 45.5 O r o m i a A m h a r a S N N P T i g r a y A f a r S o m a l i G a m b e l l a B . G u m u z M C P R a m o n g M a r r i e d Baseline Midterm 10.5 13.1 11.9 14.8 0.2 0.6 0.8 10.7 14.4 9.1 18.2 20.5 0.8 0 2.3 15.3 O r o m i a A m h a r a S N N P T i g r a y A f a r S o m a l i G a m b e l l a B . G u m u z L A F P a m o n g M a r r i e d Baseline Midterm USAID/ Ethiopia Transform Program Midline Evaluation Report Page 41 Figure 4(c): Performance Change on PPFP Method Use between Baseline and Midline in Transform Program Intervention Areas Disaggregated by Region, March 2019 The KIs at the FMOH and regions had this to say about family planning challenges: A key informant at the FMOH said that while the use of family planning methods had increased, there was still a need to address the root causes of non-use of contraception. The informant noted: Additional KIIs held with RHBs and their downstream structures as well as FGDs conducted with community members also confirmed the deep rooted traditional and cultural barriers hindering family planning service utilization in some places of the country. For example, a respondent from the Somali RHB had this to say: 28.233 41.7 34.4 6.4 3.1 9.6 30.5 35.6 44.6 48.8 30.2 6.7 0.8 7.5 53.1 O r o m i a A m h a r a S N N P T i g r a y A f a r S o m a l i G a m b e l l a B . G u m u z P P F P Baseline Midterm “Factors affecting the utilization of family planning include cultural and traditional restrictions, fear of contraceptive’s side effects, lack of women’s autonomy in health care decisions, and poor health service provision. Thus, family planning programming should be designed around effective and innovative strategies to overcome these barriers to include traditional/religious restrictions, fear of side effects and women low agency”. “I would associate the improvements in family planning use to the existence of the health extension workers and an increase in the availability of free long-acting contraceptive methods. Improving the quality of services and addressing the structural issues related to the utilization of family planning methods is critical in closing the inequality gaps. However, to further improve performance on family planning, there is a need to introduce innovative and effective approaches that effectively address barriers such as fear of side effects and traditional/religious restrictions.” “Here, the value given for children is unquenchable, and it affects the utilization of family planning services despite that demand creation efforts are made.” USAID/ Ethiopia Transform Program Midline Evaluation Report Page 42 In addition, FGD participants from Afar and Gambella regions made the following two illustrations. Hence, the efforts to boost FP performance need to introduce innovative and effective strategies to overcome barriers such as fear of contraceptives’ side effects, lack of women’s autonomy in health care decisions, poor health service provision, women's low agency, and cultural and traditional/religious restrictions. Coupled with increases in the availability of free and long-acting contraceptive methods, the Transform program has started working through community awareness forums using community-based organizations, religious or clan leaders. Research has shown that these efforts have significantly contributed to minimizing some of the myths and increasing service utilization. 10 Availability of Family Planning Services at Primary Health Care Facilities In Transform program intervention areas, different FP methods were available (Table 3). All public health facilities in the study provide at least one type of modern method of family planning services. Injectables (93.1%), pills (92.8%), and implants (79.5%) are offered in most of the health facilities. Emergency contraception is available in 79.2% of primary hospitals and 81.4% of the health centers. None of the health posts were offering emergency contraception. As the table indicates, the provision of numerous FP services was better in Transform areas than non￾Transform areas for health clinics and health posts, with less obvious differences at hospitals. Overall availability of FP methods is nearly universal. Some of the variation in the specific method availability is related to method preferences due to religious and cultural reasons. 10 Taye E, Mekonen D, Debele T. Prevalence of post -partum modern family planning utilization and associated factors among postpartum mothers in Debre Tabor town, North West Ethiopia, 2018. BMC Res Notes (2019) 12:430 https://doi.org/10.1186/s13104-019-4464-0SEARCH “If any person comes and talks about family planning in front of me or here in Afar at all, we will kick him. There is no demand for the utilization of family planning methods here; we do not need it. Here, couples need to give births to more children. More children support us, and boys provide protection to the family. Husbands are not interested in family planning, and there is no awareness creation on that.” - A 70-year-old male respondent in Afar “If women do not give birth to their next child in five years, men suspect that they are using FP. So, the men will send the women back to their parents and claim for the dowry (cattle) to be returned back to the men's family. Because of this, women will not try to use a family planning method in our clan (or this community).” - A 34-year￾old man, FGD participant in Etang Woreda in Gambella Region USAID/ Ethiopia Transform Program Midline Evaluation Report Page 43 Table 3: Proportion of Health Facilities Providing Family Planning Service by Type of Health Facility, March 2019 Type of Service Transform Program Intervention Areas Non-intervention Areas Primary Hospital Health Center Health Post Primary Hospital Health Center Health Post Pill 100.0 95.2 89.1 100.0 97.1 86.1 Injectable 95.8 93.8 92.0 100.0 94.1 80.6 Condom provision 83.3 86.9 29.9 100.0 91.2 16.7 IUCD insertion 87.5 68.3 9.5 66.7 58.8 13.9 IUCD removal 87.5 66.9 .7 66.7 55.9 2.8 Implanon insertion 100.0 89.7 66.4 100.0 82.4 41.7 Implanon removal 100.0 88.3 23.0 100.0 79.4 38.1 Jadelle insertion 83.3 69.0 19.0 66.7 50.0 9.5 Jadelle removal 83.3 66.9 13.0 100.0 47.1 9.5 Vasectomy 41.7 3.4 0.0 66.7 0.0 0.0 Tubal ligation 58.3 2.8 0.0 66.7 0.0 0.0 Emergency contraceptive 79.2 81.4 0.0 100.0 67.6 0.0 Health facilities providing at least one modern FP method 100.0 96.6 93.6 100.0 97.1 91.7 In examining the relationship between demand and supply of family planning services, 8.5% of women in Transform program intervention areas reported they have been turned away or did not receive services when they wanted FP services from health facilities (Table 4). The main reasons for not receiving FP services were commodity stockout (49.4%), the facility was closed on the day of their arrival (25.1%), and providers not available (17.8%). Table 4: Proportion of Women Who Have Ever Been Turned Away from Getting FP Services When They Wanted in Transform Program Intervention and Non-intervention Areas by Reason for Not Receiving the Services Indicator Transform program Intervention Areas Non-intervention Areas % N % N Women turned away when wanted to use FP 8.5 4037 7.0 1026 Reason for not receiving FP service ✓ Facility was closed 25.1 342 31.9 72 ✓ Provider not available 17.8 342 18.1 72 ✓ Commodities out of stock 49.4 342 45.8 72 ✓ Provider denied service 6.7 342 2.8 72 ✓ Health problem .9 342 1.4 72 USAID/ Ethiopia Transform Program Midline Evaluation Report Page 44 Difference-in-Differences Analysis of Family Planning The effectiveness of the Transform program in improving family planning service uptake is estimated by matching the women’s characteristics at baseline (women’s age, education level, paid work, participation in decision-making and CBHI) in the Transform program intervention and non-intervention areas, and then comparing changes in outcomes over time. The results show a statistically significant increase of 4.2 percentage points (95% CI: 0.5% – 7.9%; p<0.05) in the MCPR following the Transform program implementation. Similarly, the proportion of long-acting family planning method users increased by 4.6 percentage points (95% CI: 2.2% – 7.0%; p<0.001) over the intervention period. Transform program implementation also contributed to an increase in postpartum family planning use by 3.6 percentage points and a decline in unmet need for family planning by 2.4 percentage points, although the declines in the latter two were not statistically significant (Table 5). Table 5: DID Analyses for Family Planning Outcomes Indicators Baseline Mid-term DID Interventi on (%) Non￾interventi on (%) Diff (95% CI) Interventi on (%) Non￾interventi on (%) Diff (95% CI) Family planning CPR 32.8 33.9 -1.0 (-3.5; 1.5) 35.3 32.1 3.2 (0.6; 5.7)* 4.2 (0.5; 7.9)* LA users 8.5 9.8 -1.3 (-2.9; 0.3) 11.1 7.9 3.2 (1.6; 4.8)* 4.6 (2.2; 7.0)* Postpartum FP use 23.7 28.3 -4.5 (-9.2; 0.2) 28.7 29.6 -0.9 (- 5.8; 4.0) 3.6 (-3.2; 10.5) Unmet need for FP 29.8 33.0 -3.2 (5.7; - 0.6)* 26.6 32.2 -5.6 (- 7.9; - 3.2)* -2.4 (-5.9; 1.1) * p<0.05; Diff – Difference in proportions; DID – Difference-in-Differences; CI – Confidence Interval Determinants of FP Use Bivariate and multivariate regression analyses were conducted to identify the factors associated with FP use. Apart from regional variation, the use of modern contraception is affected by age, education, religion, ideal number of children, and exposure to FP/RH messages. The findings show that women aged 20-24 are 1.46 times more likely to use a family planning method than women aged 15-19, though the likelihood of using modern contraceptive methods declines with age. A woman who attended secondary education is 1.31 times more likely to use FP methods compared with a woman with no formal education. Hearing an FP/RH message is also a determinant factor for FP use, i.e., women who heard FP/RH messages in the last few months are 1.37 times more likely to use FP method compared to women who have not received FP messages. Similarly, a woman who wants to have more children is less likely to use FP methods, i.e., women who want to have 4 -5 children are 28% less likely to use FP methods compared with a woman who wants to have 0 - 3 children, and the likelihood of using FP methods declines as USAID/ Ethiopia Transform Program Midline Evaluation Report Page 45 the ideal number of children increases. However, paid work status, number of children ever born, and women’s participation in decision-making seem to have little effect on FP method utilization. On the other hand, long-acting contraceptive use is significantly associated with age and having ever born a child. Unmet need for contraception is also associated with age, religion, and having ever born a child (Table 6). Table 6: Adjusted Odds Ratios (OR) and 95% CI from the Logistic Regression Model for the Determinants of FP Use among Currently Married Women, Transform Midline Survey (2019) Indicators CPR (n=6350; event=1903) LA methods (n=6350; event=577) Unmet need (n=6350; event=1716) OR 95% CI OR 95% CI OR 95% CI Age 15 - 19 (Ref) 1.0 1.0 1.0 20 – 24 1.46 0.96─2.22 0.71 0.43─1.17 0.60 0.42─0.85 25 – 29 1.28 0.84─1.96 0.51 0.31─0.84 0.44 0.31─0.62 30 – 34 1.22 0.78─1.9 0.39 0.23─0.67 0.32 0.22─0.47 35 – 39 1.02 0.65─1.62 0.25 0.14─0.45 0.19 0.13─0.29 40+ 0.65 0.4─1.04 0.24 0.13─0.43 0.1 0.04─0.08 Education No education (Ref) 1.0 1.0 1.0 Primary 1.22 1.04─1.43 1.15 0.92─1.43 1.03 0.89─1.2 Secondary + 1.31 1.02─1.67 1.00 0.72─1.4 1.13 0.9─1.42 Paid work No (Ref) 1.0 1.0 1.0 Yes 1.05 0.9─1.23 1.11 0.9─1.36 Child ever born 1 - 2 (Ref) 1.0 1.0 1.0 3 – 4 1.05 0.86─1.29 1.28 0.97─1.68 1.67 1.38─2.02 5 – 6 1.16 0.9─1.49 1.54 1.1─2.17 2.17 1.71─2.74 7+ 1.13 0.83─1.54 1.16 0.74─1.81 3.38 2.52─4.52 Ideal number of children 0 - 3 (Ref) 1.0 1.0 4 – 5 0.72 0.56─0.92 1.05 0.77─1.42 1.14 0.91─1.42 6 – 9 0.49 0.38─0.64 1.09 0.77─1.54 1.26 0.98─1.61 10+ 0.23 0.16─0.33 0.74 0.45─1.22 0.60 0.44─0.81 RH/FP message No (Ref) 1.0 1.0 Yes 1.37 1.19─1.57 1.04 0.86─1.26 1.08 0.95─1.23 Women's participation in health care decisions No (Ref) 1.0 Yes 1.10 0.92─1.31 Non-inclusion of zero in the 95% CI indicates statistical significance. Shaded cells indicate non-significant variables in the bivariate analysis. USAID/ Ethiopia Transform Program Midline Evaluation Report Page 46 Geospatial Analysis: Family Planning Service Availability Versus Utilization The map in Figure 5 below shows that utilization of modern contraceptive methods seems to be above the national average of 35% in some woredas in the Benishangul-Gumuz, Amhara, SNNP, and Oromia regions. On the other hand, woredas in Somali and a considerable number of woredas in Oromia and Afar have a low level of MCPR usage. Availability of the family planning method of choice in the health facilities (at least two methods) seems universal except in Somali and a few woredas around the regional boarders of SNNP, Gambella, and Afar. The family planning method availability, as well as utilization, seems to be lower in a majority of the woredas in the Somali region. Figure 5: Married Women Using Modern Contraceptive Methods and Health Facilities Offering at Least Two FP Methods of Choices B. MATERNAL HEALTH Maternal health interventions aim to reduce preventable maternal and child death in Ethiopia. Women who adopt sound practices before, during, and after childbirth are much more likely to remain healthy and give birth to healthy children. To measure progress achieved in maternal health since the baseline, the analysis compares the change in Transform program intervention areas with non-Transform intervention areas among women who had a live birth in the 12 months preceding the survey. USAID/ Ethiopia Transform Program Midline Evaluation Report Page 47 Changes in Key Maternal Health Indicators The findings of the midline evaluation revealed improvement in a majority of maternal health indicators in the Transform program intervention areas compared to the changes seen in non￾Transform areas (Figures 6.a, 6.b, 6.c). Figures 6(a), 6(b), 6(c): Performance on Maternal Health Indicators between Baseline and Midline by Transform Program Intervention & Non-intervention Areas, March 2019 27.6 33.8 41.0 47.3 34.1 44.9 50.1 54.0 36.4 41.4 36.0 39.2 Transform Program Intervention Non-Intervention Transform Program Intervention Non-Intervention Transform Program Intervention Non-Intervention Early initiation of ANC ANC 4+ Coverage Essential components of ANC 6.a Baseline Midterm 21.7 26.0 46.5 55.0 35.5 39.9 22.1 20.3 51.2 40.0 47.1 43.0 Transform Program Intervention Non-Intervention Transform Program Intervention Non-Intervention Transform Program Intervention Non-Intervention IFA for 90 days or more Skilled Birth Attendance Early Postnatal Care for the mother 6.b Baseline Midterm 57.4 54.4 6.6 8.4 22.6 10.3 52.1 35.2 21.9 18.3 9.8 13.5 Transform Program Intervention Non-Intervention Transform Program Intervention Non-Intervention Transform Program Intervention Non-Intervention Pregnant women slept under ITN MNCH services through mobile health team Malaria treatment services through mobile health team 6.c Baseline Midterm USAID/ Ethiopia Transform Program Midline Evaluation Report Page 48 In Transform program intervention areas, early initiation of antenatal care (before 16 weeks) increased significantly from 27.6% at baseline to 50.1% (p <0.01); skilled birth attendance increased significantly from 46.5% to 51.2% (p = 0.01); and early postnatal care for the mother within two days following birth increased significantly from 35.5% to 47.1% (p <0.01). In non￾Transform program areas, the majority of the MNCH indicators showed declining performance from baseline. However, early initiation of ANC showed significant performance improvement from 33.8% at baseline to 54.0% at midline (p <0.01). This increase is true for intervention and non-intervention areas and could be associated with the pregnancy conferences implemented by GOE. In addition, the share of women who received MNCH/FP services through a mobile health team significantly increased from 6.6% to 21.9% (p <0.01) (Figure 7). However, the use of malaria treatment actually declined more significantly in Transform intervention areas. The midline data further show that the provision of vaccination (66.7%) and child health services (35.1%) rose relative to non-Transform areas, though health education and malaria treatment declined (Figure 6). Use of ANC services was nearly identical, both in aggregated terms and regarding specific components of ANC. In non-Transform program areas, the proportion of women who received MNCH services through the mobile health team significantly improved from 8.4% to 18.3% (p <0.01). All other indicators showed declining performance or did not change over the period. Figure 7: Type of Mobile Health Service Provided in Transform Program Intervention Areas and Non-intervention Areas, March 2019 Looking into the regional level performance, the analysis results revealed that performance trends on the majority of maternal health indicators varies from region to region. Overall, Afar and Somali had the lowest performance at baseline and mid-term compared with all other regions, but Benishangul-Gumuz had relatively better performance which is comparable to agrarian regions (Figures 8.a, 8.b, 8.c, 8.d). 1.2 4.7 9.8 12.1 14.2 17.5 32.1 35.1 66.7 0.0 5.7 13.5 10.9 13.1 19.7 36.2 29.3 61.6 NTD ONCHOCERCIASIS … MALARIA PNC ANC FP HEALTH EDUCATION CHILD HEALTH SERVICE VACCINE Non-Transform Program Areas Transform Program Intervention USAID/ Ethiopia Transform Program Midline Evaluation Report Page 49 Figures 8(a), 8(b), 8(c), 8(d): Performance Change on Maternal Health Indicators between Baseline and Midline in Transform Program Intervention Areas Disaggregated by Region, March 2019 31.3 35.8 21.9 35.7 15.8 24.6 34.5 30.2 57.6 59.2 34.5 73.9 36.1 27.2 55.1 66.3 O r o m i a A m h a r a S N N P T i g r a y A f a r S o m a l i G a m b e l l a B e n . G u m u z E a r l y A N C 8.a E a r l y i n i t i a t i o n o f A N C Baseline Midterm 53.4 5559.5 60.4 16.5 16.923.7 49.7 42.1 46.5 51.5 70.8 12.1 9.6 22.6 65.8 O r o m i a A m h a r a S N N P T i g r a y A f a r S o m a l i G a m b e l l a B e n . G u m u z A N C 4 + C o v e r a g e 8 . b A N C 4 + c o v e r a g e Baseline Midterm 50.35963.2 85.7 17.7 21.4 50.9 31.8 65.9 71.5 52 88.5 31.1 11.8 44.3 54 O r o m i a A m h a r a S N N P T i g r a y A f a r S o m a l i G a m b e l l a B e n . G u m u z SBA 8 . c S k i l l e d B i r t h A t t e n d a n c e ( s b a ) Baseline Midterm USAID/ Ethiopia Transform Program Midline Evaluation Report Page 50 The observed increase in SBA coverage may be due to the expansion of health services, active engagement of a HEW or the Health Development Army (HDA), and an increased emphasis on maternal and newborn care in recent years.11,12 Literature states that most maternal deaths occur due to complications that arise during labor, delivery, and the immediate postpartum period, with obstetric hemorrhage being the main medical cause of maternal death. Thus, the availability and accessibility of skilled care at the time of childbirth are critical factors for avoiding maternal deaths occurring from preventable obstetric complications.13 The pastoralist regions of Afar and Somali demonstrate low utilization of maternal health services, which can be attributed to the mobile lifestyle of these populations, distance to health facilities, absence of waiting areas at health facilities, and sociocultural factors.14 National and regional KIs (FMOH, RHBs in Afar, SNNP, Amhara, Oromia, and Gambella) all expressed some variant of the following: Numerous KIs stressed the importance of programming that addresses individual, family, cultural and structural factors to substantially increase deliveries attended by skilled attendants. The 11 WHO. Policy dialogue. Improving Skilled Birth Attendance in Ethiopia. Accessed on 25 December 2019. https://www.who.int/evidence/sure/pdimprovingskilledbirthattendanceethiopia.pdf?ua= 12 MOH. Independent Review Team. Mid-Term Review Comprehensive Final Report. Ethiopia’s Health Sector Transformation Plan (HSTP) (2015/16-2019/20 GC or 2008 – 2012 EFY). August 2018 13 Family Care International: Safe Motherhood A Review; 2007. http://www.familycareintl.org/UserFiles/File/SM_A Review_Exec_Sum_Final_1.pdf 14 11 Ahmed M, Demissie M, Abraha A, Worku A and Berhane Y. Reasons for low level of skilled birth attendance IN AFAR Pastoralist communty North East Ethiopa: Qualtiative exploration. Pan African medical Jorunal,30: 51, 2018 28.2 59.6 46.864.3 10.5 17.6 34.2 37.1 45.2 57.7 52.8 66.7 29.6 22 47.2 66.4 O r o m i a A m h a r a S N N P T i g r a y A f a r S o m a l i G a m b e l l a B e n . G u m u z E a r l y P N C f o r m o t h e r s 8 . d E a r l y P N C f o r m o t h e r s Baseline Midterm “Factors affecting the utilization of skilled delivery services include lack of skilled delivery experience, lack of information, frequency and quality of antenatal care utilization and low education on maternal health. The poor￾quality services that are not mother friendly and poorly equipped health facilities may not convince mothers to deliver in health facilities”. “…Previously we do not have any ultrasound, but now Transform: HDR provided us one ultrasound, and it plays a major role in improving maternal health services during ANC and delivery…the number of pregnant mothers in ANC increased dramatically due to the ultrasound. When we tell the pregnant mothers in the community that they will be able to see the fetus with ultrasound, they become motivated to the facility.” – WoHO head, Afar USAID/ Ethiopia Transform Program Midline Evaluation Report Page 51 introduction of mobile ultrasound services by the Transform program might have contributed to the significant increase in early initiation of ANC. Key informant interviews with woreda health offices (WoHOs) revealed that mobile ultrasound had attracted more pregnant women to come to health facilities for ANC. KIIs from woredas in Afar and Gambella Region had this to say: Another KII, Etang WoHO MNCH coordinator, Gambella had this to say: Availability of Maternal Health Services at Primary Health Care Facilities: The health facility assessment showed that availability of maternal health services is to some extent better in Transform program intervention areas compared to non-Transform program areas. A majority (63.9 %) of health centers in Transform program intervention areas provide Basic Emergency Obstetric Newborn Care (BEmONC) signal functions, compared to 52.9% in non-Transform areas. Further, 66.7% of the primary hospitals provide Comprehensive Emergency Obstetric Newborn Care (CEmONC) signal functions, which is equal to the rate in non-Transform areas. In addition, all the primary hospitals and 95.8% of the health centers in Transform program intervention areas provide delivery service 24 hours a day and seven days a week (compared to 100% and 94.1%, respectively, in non-Transform areas). Seventy-two percent (72%) of the health centers and 67% of the primary hospitals have maternal waiting rooms for pregnant mothers who have reached their expected delivery date, compared to 72% and 25% (one of the four primary hospitals), respectively, in non-Transform areas. In Transform program areas, comprehensive laboratory services (venereal disease, hemoglobin, HIV blood group, urinalysis, and pregnancy test) were available in 52.1% of health centers and 95.8% of the primary hospitals at the time of the assessment. These rates represent upticks compared to the non-intervention areas (47% and 75% [three out the four primary hospitals], respectively). Difference-in-Differences Analysis of Maternal Health The DID analysis indicates that the Transform program interventions significantly improved essential ANC coverage by 8.4 percentage points (95%CI: 0.2% – 16.6%; p<0.05). Transform program interventions also significantly contributed to the increase in skilled birth attendance, by 16.3 percentage points (95% CI: 8.7% – 23.9%; p<0.01). However, the observed increment was not due to an increase in skilled birth attendance in the intervention woredas, but rather to a reduction in skilled birth attendance in the non-Transform areas (from 52.3% to 42%). Although it is not statistically significant, the Transform program interventions contributed to an increase in the proportion of women who took iron and folic acid supplements for three months during pregnancy and received early PNC within two days of delivery (Table 7). However, four or more ANC visits and early ANC visits did not show improvement through to midline. The observed decline in the proportion of women who received four or more ANC visits could also be associated with a decrease in a male spouse or partner accompanying women during at “The utilization of ANC service increased four times than it was before immediately after the introduction of the ultrasound…It encourages mothers to come for ANC follow up. Mothers from other neighboring woredas were even visiting the health center due to the ultrasound.” USAID/ Ethiopia Transform Program Midline Evaluation Report Page 52 least one ANC visit. According to recent research, women accompanied by their spouses to the ANC clinic are 1.5 times more likely to have four or more ANC visits compared to women who have not been accompanied by their spouses, including registration in CBHI15. This study concluded that women registered in CBHI were more likely to access maternal health and had better proportions of uptake than those with no registration. Table 7: The Difference in Proportions and DID of Maternal Health Indicators Disaggregated by Transform Program Intervention and Non-intervention Areas (Baseline 2017 and Midline 2019), March 2019 Indicators Baseline Midline DID Interven tion (%) Non￾intervention (%) Diff (95% CI) Interventi on (%) Non￾interventi on (%) Diff (95% CI) Maternal health ANC 4+ 41.0 44.6 -3.7 (-9.0; 1.6) 45.1 50.0 -4.8 (-10.8; 1.3) -1.1 (-9.3; 7.1) Early ANC 26.2 32.7 -6.5 (-11.6; 1.4) 50.1 54.9 -4.9 (-10.2; 0.4) 1.6 (-5.8; 9.0) Essential ANC 35.3 48.2 -13.0 (-18.9; - 7.1)* 36.0 40.6 -4.6 (-10.5; 1.3) 8.4 (0.2; 16.6)* SBA 45.1 52.3 -7.1 (-12.4; - 1.8)* 51.2 42.0 9.1 (3.6; 14.6)* 16.3 (8.7; 23.9)* Early PNC 35.4 38.0 -2.7 (-8.0; 2.6) 47.1 43.8 3.3 (-2.2; 8.8) 6.0 (-1.6; 13.6) IFA for 3month 22.2 27.2 -5.1 (-12.3; 2.2) 22.1 23.7 -1.6 (-7.7; 4.5) 3.5 (-5.9; 12.9) ITN for women 58.4 52.5 5.9 (-2.9; 14.7) 52.1 35.3 16.8 (8.7; 24.8)* 10.9 (-1.1; 22.8) * p<0.05; Diff – Difference in proportions; DID – Difference-in-Differences; CI – Confidence Interval Other common factors hindering pregnant mothers from attending ANC include women's lack of awareness of its importance, distance to the health facility, and unavailability of transportation. According to RHB KIIs in Tigray, Oromia, SNNP, and Amhara, other essential determinants that facilitate ANC attendance include higher education level, woman's ability to make healthcare decisions, partner involvement, and MCH messaging.16,17 Regional KIIs from the agrarian regions of SNNP, Amhara, and pastoralist regions of Afar and Somali all echoed the factors responsible for observed maternal services utilization to include socio-cultural factors, distance to the health facility, and limited availability of waiting areas at health facilities. Regional KIIs from the pastoralist regions of Afar and Somali also mentioned factors such as the mobile lifestyle of the populations and distance to the health facility. Other factors mentioned affecting the utilization of skilled delivery services include low education on maternal health, limited availability of information, lack of skilled delivery experience, and quality of antenatal care utilization. The 15https://www.researchgate.net/publication/332999123_Effects_of_community_based_health_insurance_on_modern_familyplanning_utiliz ation_in_Ethiopia 16 RHB KIs from Tigray, Oromia, SNNP and Amhara 17 Fekadu H, Tesfahun A and Mequanent M. Late initiation of antenatal care and associated factors among pregnant women in Addis Zemen primary hospital, South Gondar, Ethiopia. Reproductive Health (2019) 16:73. https://doi.org/10.1186/s12978-019-0745-2. USAID/ Ethiopia Transform Program Midline Evaluation Report Page 53 poor-quality services that are not mother-friendly and poorly equipped health facilities may not convince mothers to deliver in health facilities.18 It is important to address individual, family, cultural, and structural factors to substantially increase deliveries attended by skilled attendants. Determinants of Maternal Health Services Utilization To identify the determinants of maternal health service utilization, the following covariates were included in the logistic regression analysis: women’s region of residence, age, educational level, paid work status, religion, number of children ever born, ideal number of children a woman wants to have, exposure to MNCH messages, whether the spouse accompanied the woman during ANC, whether the spouse accompanied the woman during delivery, whether the woman participates in her own health care decisions, and CBHI. Region of residence, age, education, MNCH/FP message, and spouses accompanying women to ANC are significantly associated with early initiation of ANC visits (Table 8). Women in age groups of 20–24, 25–29, and 35–39 years are 1.29, 1.26, and 1.35 times, respectively, more likely to initiate ANC visits in the first trimester compared with pregnant women below 19 years old. Women with a secondary education are 2.27 times more likely to initiate ANC visits early compared with women with no education. Women who heard MNCH messages are 1.9 times more likely to initiate early ANC visits, compared with women who did not hear MNCH messages. The recommended four or more ANC visits are significantly associated with region, education, MNCH messaging, and spousal accompaniment of women to ANC visits. Women with a primary education and secondary education are 1.33 and 2.11 times more likely to receive four or more ANC visits, respectively, compared to those with no education. Women who heard MNCH messages are 1.62 times more likely to receive four or more ANC visits, compared with women who did not hear MNCH messages. Women accompanied by their spouses during ANC visits are 1.73 times more likely to visit ANC services at least four times than women who are not accompanied by their spouses. SBA services are also significantly associated with region, age, education, paid work status, MNCH message and spouses accompanying women to ANC. Women in age group 20–24 are 1.28 times more likely to utilize SBA services than women below the age of 19, and the likelihood of receiving SBA increases with age. Women with primary and secondary education are 1.41 and 2.14 times more likely to receive SBA services, respectively, compared to those with no education. Women who have 3 or 4 children are 28% less likely to have SBA than women with 1 or 2 children. Women who are accompanied by their spouses for ANC visits are 1.76 times more likely to have SBA than those who are not accompanied by their husbands. 18 WHO. Policy dialogue. Improving Skilled Birth Attendance in Ethiopia. Accessed on 25 December 2019. https://www.who.int/evidence/sure/pdimprovingskilledbirthattendanceethiopia.pdf?ua=1 USAID/ Ethiopia Transform Program Midline Evaluation Report Page 54 Table 8: Adjusted Odds Ratios (OR) and 95% CI from the Logistic Regression Model for the Determinants of Maternal Health Services Utilization, Transform Midline Survey (2019) Indicators Early ANC (n=1235; event=628) ANC 4+ (n=1029; events=474) SBA (n=1298; unmet=634) OR 95% CI OR 95% CI OR 95% CI Region Afar 0.32 0.15─0.69 0.07 0.03─0.16 0.08 0.04─0.17 Amhara 0.52 0.29─0.94 0.38 0.22─0.64 0.37 0.22─0.63 Benishangul-Gumuz 1.01 0.5─2.02 1.41 0.74─2.68 1.42 0.74─2.72 Gambella 1.09 0.49─2.43 0.60 0.27─1.3 0.61 0.28─1.35 Oromia 0.51 0.28─0.95 0.32 0.18─0.57 0.33 0.19─0.59 SNNP 0.36 0.18─0.72 1.21 0.62─2.38 1.17 0.59─2.31 Somalia 0.23 0.11─0.52 0.09 0.04─0.21 0.09 0.04─0.22 Tigray (Ref) 1.0 1.0 1.0 Age 15 - 19 (Ref) 1.0 1.0 20 - 24 1.29 0.73─2.28 1.28 0.7─2.36 25 - 29 1.26 0.69─2.29 1.48 0.78─2.81 30 - 34 1.09 0.55─2.15 1.75 0.84─3.67 35 - 39 1.35 0.65─2.82 2.38 1.07─5.3 40+ 1.09 0.45─2.63 1.49 0.55─4.02 Education No education (Ref) 1.0 1.0 1.0 Primary 1.16 0.85─1.59 1.33 0.97─1.82 1.41 1─1.99 Secondary + 2.27 1.39─3.72 2.11 1.35─3.29 2.14 1.33─3.45 Paid work No (Ref) 1.0 1.0 1.0 Yes 1.22 0.86─1.74 1.28 0.91─1.8 1.28 0.91─1.81 Child ever born 1 - 2 (Ref) 1.0 1.0 3 – 4 0.87 0.59─1.28 0.72 0.48─1.08 5 – 6 0.76 0.47─1.25 0.78 0.45─1.35 7+ 1.07 0.58─1.97 0.68 0.34─1.36 Ideal number of children 0 - 3 (Ref) 1.0 1.0 1.0 4 – 5 1.39 0.77─2.49 1.36 0.82─2.27 1.36 0.81─2.27 6 – 9 1.00 0.54─1.84 1.22 0.71─2.08 1.20 0.68─2.13 10+ 0.78 0.4─1.53 1.30 0.69─2.44 1.29 0.66─2.52 MNCH message No (Ref) 1.0 1.0 1.0 Yes 1.89 1.45─2.47 1.62 1.22─2.16 1.60 1.2─2.13 Spouse accompany ANC No (Ref) 1.0 1.0 1.0 Yes 2.87 2.19─3.77 1.73 1.3─2.31 1.76 1.31─2.35 Spouse accompany delivery No (Ref) USAID/ Ethiopia Transform Program Midline Evaluation Report Page 55 Indicators Early ANC (n=1235; event=628) ANC 4+ (n=1029; events=474) SBA (n=1298; unmet=634) OR 95% CI OR 95% CI OR 95% CI Yes Women's participation in health care decisions No (Ref) 1.0 Yes 0.98 0.71─1.37 CBHI No (Ref) 1.0 1.0 1.0 Yes 1.17 0.82─1.67 1.06 0.75─1.5 1.06 0.75─1.51 Geospatial Analysis: Family Planning Service Availability Versus Utilization The map in Figure 9 below shows that most woredas had a low proportion of women who had at least four ANC visits for the births they had in the last 12 months prior to the assessment. Only a few woredas in Benishangul-Gumuz, Amhara, Gambella, and Tigray had a high proportion of women who had four or more ANC visits. Woredas in SNNP, Oromia, Gambella, and Somali had particularly low proportions of women who had four or more ANC visits. It should be noted, however, that complete laboratory services for pregnant mothers were not widely available, specifically in Somali, Afar, and SNNP. Figure 9. Women Received 4+ ANC Visits and Availability of Laboratory Service, Midline, March 2019 SBA services were found to be high (above 80%) in a few woredas in Oromia, Amhara, and Tigray as shown in Figure 10 below. A considerable number of woredas in Afar, Benishangul-Gumuz, Oromia, SNNP, and Somali region had SBA services below the national average of 51.2%. In USAID/ Ethiopia Transform Program Midline Evaluation Report Page 56 addition, a majority of health centers in Somali and Afar and some woredas in Amhara, Oromia, Benishangul-Gumuz, and Gambella did not provide BEmONC signal functions. In general, the pastoralist regions of Afar and Somali fare more poorly than other areas, having less access to health facilities and services and, as a result, less uptake of the activities aimed at bolstering child and maternal health. Figure 10: Women Received Skilled Birth Attendance and Health Facility Providing BEmONC Signal Functions, Midline, March 2019 C. NEWBORN HEALTH The midline evaluation assessed key newborn health indicators to measure progress in Transform program intervention areas. Findings were again compared to non-intervention areas to quantify the contribution of the Transform program in improving newborn outcomes in the Transform program intervention areas. Changes in Newborn Health Indicators Performance on all newborn indicators showed improvement from the baseline in Transform program intervention areas (Figure 11). The proportion of newborns receiving postnatal care (PNC) within 48 hours after birth increased by 10.8 percentage points, from 31.8% at baseline to 42.6% at midline (p <0.01). Initiation of breastfeeding within one hour of birth also showed significant improvement, from 66.3% at baseline to 85.4% at midline (p <0.01); comparing mid- USAID/ Ethiopia Transform Program Midline Evaluation Report Page 57 term with baseline value only, excluding non-intervention woredas. The proportion of newborns who were delivered at health facilities and received essential newborn care (ENC) services19 improved from 7% at baseline to 12.7% at midline (p <0.01). A similar performance trend was observed in non-Transform program areas for all newborn indicators, except that the improvement in early postnatal care was not statistically significant (P-value=0.19). Figure 11: Performance on Newborn Health Indicators at Baseline and Midline Compared by Intervention and Non-intervention Areas, March 2019 The improvements in newborn health indicators are particularly notable for a few reasons. Cultural practices in the region typically encourage confinement. Limited awareness of public health guidelines among new mothers can also be a detriment to newborn care. Finally, HEWs are typically overworked and unable to dedicate sufficient time to new mothers. Under these conditions, the improvements documented from baseline to midline suggest real, positive change in newborn care. Given other potential barriers, such as physical distance and rural residency, it remains important to strengthen the implementation strategies that help mothers seek postnatal care.20,21 The survey also showed better outcomes in Transform areas compared to non-Transform areas across the board in terms of critical tasks for newborn care just after birth including skin contact, eye ointment, vitamin K, and umbilical cord care (Figure 12). However, 28% of the mothers do not know if any of the essential newborn care services was given to their babies or not. 19 Essential newborn care includes Vitamin K, TTC eye ointment, cord care with ointment, and skin-to-skin contact immediately after delivery. 20 RHB KIIs from Afar and Somali. 21 27 Berhe A, Bayray A, Berhe Y, Teklu A, Desta A, Araya T, et al. (2019) Determinants of postnatal care utilization in Tigray, Northern Ethiopia: A community based cross-sectional study. PLoS ONE 14(8): e0221161. https://doi.org/ 10.1371/journal.pone.0221161 31.8 34.2 7.0 4.5 66.3 59.9 42.6 37.8 12.7 10.4 85.4 87.2 Transform Program Intervention Non-Intervention Transform Program Intervention Non-Intervention Transform Program Intervention Non-Intervention Early PNC for newborn ENC Early initiation of breastfeeding Baseline Midterm USAID/ Ethiopia Transform Program Midline Evaluation Report Page 58 Figure 12: Type of Essential Newborn Services Provided for Health Facility Deliveries by Transform Program Intervention and Non-intervention Areas, March 2019 Comparison of the performance change in newborn health indicators at the regional level indicates that there was a considerable variation among regions. The highest increase in early postnatal care was observed in Benishangul-Gumuz, an increase of 35.9 percentage points, followed by Afar, an increase by 17.3 percentage points. The performance in Somali declined by 1.6 percentage points (Figures 13.a, 13.b, and 13.c) Essential newborn care remained to be least in Oromia and Afar. The highest increase was seen in SNNP, by 17.9 percentage points, but the performance declined in Somali and Gambella. No big variation was observed in early initiation of breastfeeding. Figure 13(a), 13(b), and 13(c): Performance Change on Newborn Health Services Uptake between Baseline and Midline in Transform Program Intervention Areas Disaggregated by Region, March 2019 13.a Early PNC for Newborn 28.1% 33.0% 47.3% 47.9% 79.3% 25.4% 24.6% 38.8% 45.5% 73.1% Dont know Umblical cord care with ointment Vitamin K TTC eye ointment Had skin to skin contact Non-Transform Program Areas Transform Program Intervention 25.5 46.8 42.3 50.0 12.3 22.1 36.8 30.5 41.5 46.9 52.8 52.1 29.6 20.5 38.7 66.4 O r o m i a A m h a r a S N N P T i g r a y A f a r S o m a l i G a m b e l l a B e n . G u m u z E a r l y P N C f o r n e w b o r n s Baseline Midterm USAID/ Ethiopia Transform Program Midline Evaluation Report Page 59 13.b ENC 13.c Early Initiation of Breastfeeding Availability of Newborn Health Services at Primary Health Care Facilities The health facility assessment showed that 78.6% (110 out of 140) of the health posts in Transform program intervention areas provide community-based newborn care (CBNC) service. About 62% (88 out of 141) of the health centers have a newborn corner with heater/radiant warmer and resuscitation equipment. The data also shows that 83% (120 out of 144) of the health centers and 96% (23 out of 24) of the primary hospitals practice kangaroo mother care. Furthermore, 91.7% (22 out of 24) of the primary hospitals have a Neonatal Intensive Care Unit (NICU). In non-Transform program areas, those rates are almost all lower: 56% (20 out of 36) of the health posts provide CBNC service; 38% (13 out of 34) of health centers have a newborn corner with heater/radiant warmer and resuscitation equipment; and kangaroo care for preterm or low birth weight babies is practiced in 71% (24 out of 34) of the health centers and in all (four) primary hospitals. However, unlike in Transform intervention areas, NICUs were available in all primary hospitals in non-Transform areas. 0.03.8 20.7 1.5 0 25 13.3 0 2.2 13.9 38.6 12.8 2.2 19 8.5 8.6 O r o m i a A m h a r a S N N P T i g r a y A f a r S o m a l i G a m b e l l a B e n . G u m u z E N C Baseline Midterm 79.2 78.7 69.2 80.1 52.3 35.4 82.6 48.8 93.3 83.7 89.3 85.3 87.9 77.7 83.8 77.7 O r o m i a A m h a r a S N N P T i g r a y A f a r S o m a l i G a m b e l l a B e n . G u m u z E a r l y i n i t i a t i o n o f b r e a s t f e e d i n g Baseline Midterm USAID/ Ethiopia Transform Program Midline Evaluation Report Page 60 Difference-in-Differences Analysis of Newborn Health Although there was significant performance improvement from the baseline in Transform program intervention areas, the DID analysis did not show a statistically significant improvement in newborn health indicators relative to non-Transform program areas (Table 9). As the table shows, the lack of statistically significant improvement is generally a function of comparable increases in the non-intervention areas, which may be associated with support from other partners providing support to regions and woredas. Interestingly, early initiation of breastfeeding declined significantly by 8.8 percentage points (95% CI: 1.5% – 16.1%; p<0.05) in intervention areas over the intervention periods compared with the performance in non-intervention areas. Essential newborn care also declined by 1.4 percentage points. Table 9: The Difference in Proportions and DID of Newborn Health Indicators Disaggregated by Transform Program Intervention and non-Transform Program Areas (Baseline 2017 and Midline 2019), March 2019 Indicators Baseline Mid-term DID Interventi on (%) Non￾Interven tion. (%) Diff (95% CI) Interve ntion (%) Non￾Interve ntion (%) Diff (95% CI) Newborn Health Early PNC newborn 31.7 34.1 -2.4 (-7.5; 2.7) 42.6 39.3 3.3 (-2.0; 8.6) 5.8 (-1.6; 13.2) ENC 7.3 3.3 4.1 (-0.4; 8.6) 12.7 10.0 2.7 (-1.4; 6.8) -1.4 (-7.5; 4.7) Early BF 64.7 58.5 6.2 (0.5;11.9) ** 84.0 86.6 -2.6 (-7.1; 1.9) -8.8 (-16.1; - 1.5)* * p<0.05; Diff – Difference in proportions; DID – Difference-in-Differences; CI – Confidence Interval D. CHILD HEALTH Child health interventions mainly focus on preventing deaths from childhood illness and improving the health status of children under five. The midline evaluation assessed key child health indicators to measure progress achieved in the Transform program intervention areas, again compared to the change observed in non-intervention areas. This section summarizes the effects of both preventive and treatment child health indicators including both child immunization and the diagnosis and treatment of important childhood illnesses. Changes in Proportions of Child Health Indicators Most child health indicators improved from the baseline values in Transform program Intervention areas. All three immunization indicators increased in performance (Figure 14). Measles coverage showed the highest increase at 27.1 percentage points (p <0.01), followed by Penta 3 coverage, which increased from 39.4% at baseline to 50.9% at midline (p <0.01). Full immunization also increased significantly from 34.3% to 39.1% between baseline and midline (p USAID/ Ethiopia Transform Program Midline Evaluation Report Page 61 = 0.027). In non-Transform areas, conversely, immunization rates increased, but by smaller margins that were not statistically significant. Figure 14: Performance on Immunization Indicators in USAID Transform Program Intervention and Non-Transform Program Areas, Baseline (2017) vs. Midline (2019), March 2019 Figure 17 presents performance trends regarding the treatment and prevention of common childhood illnesses. Three out of the six indicators showed significant improvement in Transform program intervention areas. In particular, the share of children 2-5 years of age who were given drugs for intestinal worms in the last six months increased from 32.1% at baseline to 38.9% at midline (p <0.01). Exclusive breastfeeding also increased from 62.8% to 67.8% between the two periods, though the change is not statistically significant (p=0.058). The proportion of under-five children with acute respiratory infection (ARI) symptoms who were treated with antibiotics increased by 38.9 percentage points from the baseline value of 27.8% to 66.7% at midline (p <0.01). Children with ARI symptoms also decreased significantly, from 7.1% at baseline to 5.2% at midline (p<0.01). However, diarrhea incidence among children under five increased significantly, from 10.4% at baseline to 14.2% at midline (p <0.01), while diarrhea treatment with oral rehydration salts (ORS) and Zinc remained statistically similar over the period of Activity (28.3% at baseline versus 28.7% at midline). KIs at FMOH, RHBs of Afar, SNNP, Amhara, Oromia, and Gambella had this to say on Penta 3 coverage: In non-intervention areas, ARI treatment with antibiotics and deworming showed a significant improvement between baseline and midline (p < 0.01) (Figure 15). However, diarrhea treatment 39.4 49.0 32.7 38.1 34.3 47.1 50.9 50.1 59.8 63.0 39.1 38.6 Intervention Non-Intervention Intervention Non-Intervention Intervention Non-Intervention Penta 3 Coverage Measles Coverage Full Immunization Baseline Midline “Low Penta 3 coverage has been due to lack of antenatal care, distance between their home and health facility, parental low levels of education, poor awareness of immunization, and home delivery. Religious beliefs and fear of the side effects of vaccinations result in low vaccination coverage, which result in low immunization coverage. Data issues and weak laboratory infrastructure have also impacted immunization coverage in the country.” USAID/ Ethiopia Transform Program Midline Evaluation Report Page 62 with ORS and Zinc and exclusive breastfeeding declined between the two periods, though not statistically significant at 95% confidence level. Figure 15: Performance Trend on Treatment and Prevention of Common Childhood Illness in USAID Transform Program Intervention and Non-intervention Areas, March 2019 Low observed child health outcome could be associated with: parental lack of or low levels of education, poor awareness of immunization, distance between their home and health facilities, lack of antenatal care, and home delivery. Religious beliefs and fear of the side effects of vaccinations result in low vaccination coverage or untimely vaccinations, which result in low immunization coverage.22 Other factors include: sustainable funding, vaccine stock-outs and logistics problems, data issues and weak laboratory infrastructure.23 These factors were cited by KIs and are consistent with the literature on challenges related to immunization. The evaluation also looked into regional performance in selected child health indicators. Figures 16 and 17 below depict the performance change in full immunization coverage and diarrhea treatment with Zinc and ORS, respectively. All regions, except Amhara and Benishangul-Gumuz, showed performance improvement on full immunization coverage, the highest increment being in Oromia region with an increase of 19.7 percentage points. On the other hand, diarrhea treatment with Zinc and ORS declined in all regions except in Amhara, Afar, and Benishangul￾22 RHB KIIs from Afar, SNNP, Somali, Amhara, Gambella and Benishangul-Gumuz 23 Richard Mihigo et al. Challenges of immunization in the African region. The Pan African Medical Journal. 2017;27 (Supp 3):12. doi:10.11604/pamj.supp.2017.27.3.12127 7.1 6.2 27.8 26.1 10.4 13.2 28.3 29.4 32.1 31.3 62.8 73.6 5.2 5.1 66.7 60.7 14.2 13.7 28.7 28.8 38.9 41.4 67.8 70.2 Intervention Non-Intervention Intervention Non-Intervention Intervention Non-Intervention Intervention Non-Intervention Intervention Non-Intervention Intervention Non-Intervention symptoms of ARI in the past two weeks Symptoms of ARI treated with antibiotics Diarrhea incidence Diarrhea treatment with ORS+Zinc Deworming Exclusive breast feeding Baseline Midline USAID/ Ethiopia Transform Program Midline Evaluation Report Page 63 Gumuz. The highest decline was in the Somali and Gambella regions, a decline of 18 and 17 percentage points, respectively. Figure 16: Performance Change on Full Immunization Coverage between Baseline and Midterm in Transform Program Intervention Areas Disaggregated by Region, March 2019 Figure 17: Performance Change on Diarrhea treatment with Zinc and ORS between Baseline and Midline in Transform Program Intervention Areas Disaggregated by Region, March 2019 The evaluation also assessed malaria-related indicators for under-five children, noting some improvement between baseline and midline (see Figure 18). In Transform intervention areas, children under five who slept under an ITN the previous night increased significantly from 42.8% at baseline to 51.8% at midline (p <0.01). The share of under-five children who received first￾line antimalaria drugs (Quartem) and Artemisinin Combination (ACT) tablet and ACT rectal artemisinin-based combination therapy (ACT) treatment for children with a fever in the last two weeks who had received antimalaria drugs increased significantly from 17.4% at baseline to 60.5% at midline (p<0.01). However, at midline, only 11.3% of the under-five children who had a fever in the last two weeks were treated with any type of antimalaria drugs. Those who sought treatment within the first 24 hours of fever onset dropped by 10 percentage points, from 46.9% 24.1 67.4 29.744.2 9.920.430.4 61.4 43.8 48.8 34.551.2 17.326.136.2 56 O r o m i a A m h a r a S N N P T i g r a y A f a r S o m a l i G a m b e l l a B e n . G u m u z F u l l I m m u n i z a t i o n Baseline Midterm 27.5 18.4 38.0 27.3 22.8 33.3 32.9 26.5 28.1 41.8 29.9 20.6 35.1 15.4 16.0 35.6 O r o m i a A m h a r a S N N P T i g r a y A f a r S o m a l i G a m b e l l a B e n . G u m u z D i a r r h e a t r e a t m e n t w i t h O R S + Z i n c Baseline Midterm USAID/ Ethiopia Transform Program Midline Evaluation Report Page 64 at baseline to 36.9% at midline (p <0.01). Further, children under five with a fever who were tested for malaria decreased significantly, from 25.6% at baseline to 21.1% at midline (p = 0.036). Similar trends persisted in the non-Transform areas. While the proportion of under-five children who slept under an ITN the previous night and the share of children receiving first-line ACT treatment increased significantly between baseline and midline, other indicators show significant declines over the period of Activity. Figure 18: Malaria Indicators Performances in USAID Transform Program Intervention and Non-intervention Areas, Baseline (2017) Vs Midline (2019) Availability of Child Health Services at Primary Health Care Facilities Integrated community case management (ICCM) represents one of the major community level health interventions targeting prevention and treatment of childhood illnesses in under-five children. The health facility assessment showed that 88.6% of the health posts in Transform program intervention areas provide ICCM services. Half of the health posts (51%) had functional cold storage systems (refrigerators), which are used primarily to store vaccines for under-five children. And 95% of health centers had functional refrigerators. Zinc and ORS for diarrhea treatment were available in most health facilities at the time of the assessment, i.e., in 60% of the health posts, 63% of the health centers, and 75% of the primary hospitals. In addition, Amoxicillin 250mg was available in 83% of the health posts, 90% of the health centers, and 96% of the primary hospitals. However, in non-Transform program areas, the rates were comparable or even higher. For example, 86% of the health posts provide ICCM service, 56% of health posts had functional refrigerators, 44% had Zinc and ORS, and 75% had Amoxicillin 250mg at the time of the 42.8 39.6 46.9 51.7 25.6 29.6 17.4 30.4 51.8 46.9 36.9 38 21.1 21.1 60.5 75 Intervention Non-Intervention Intervention Non-Intervention Intervention Non-Intervention Intervention Non-Intervention Children under 5 who slept under ITN the previous night Under 5 children who seek treatment in the first 24 hours of onset of fever Under 5 children with fever who are tested for malaria Under 5 children receiving first-line ACT/treatment Baseline Midline USAID/ Ethiopia Transform Program Midline Evaluation Report Page 65 assessment. And 85% of health centers had functional refrigerators, 53% had Zinc and ORS, and 94% had Amoxicillin 250mg at the time of the assessment. All three primary hospitals had functional refrigerators, Zinc and ORS, and Amoxicillin 250mg. Difference-in-Differences Analysis of Child Health DID analysis was done for selected key child health indicators, with findings presented in Table 10. The difference in the incidence of diarrhea increased significantly, by 3.6 percentage points (95% CI: 0.5% – 6.7%), over the intervention period. Additionally, the difference in early initiation of breastfeeding declined by 8.8 percentage points (95% C.I: 1.5 – 16.5). Table 10. Difference in Proportions and DID of Child Health Indicators, Disaggregated by Transform Program Intervention and Non-intervention Areas, (Baseline 2017 & Midline 2019 Indicators Baseline Mid-term DID Intervention. (%) Non￾Interventi on (%) Diff (95% CI) Intervention . (%) Non￾Intervent ion (%) Diff (95% CI) Child health, immunization and nutrition Early PNC newborn 31.7 34.1 -2.4 (-7.5; 2.7) 42.6 39.3 3.3 (-2.0; 8.6) 5.8 (-1.6; 13.2) ENC 7.3 3.3 4.1 (-0.4; 8.6) 12.7 10.0 2.7 (-1.4; 6.8) -1.4 (-7.5; 4.7) Fever treatment 45.9 53.7 -7.8 (-16.8; 1.2) 36.9 35.6 1.3 (-5.0; 7.6) 9.1 (-1.9; 20.1) ARI symptom 8.5 6.5 2.0 (0.2; 3.7)* 5.1 5.1 0.1 (-1.0; 1.3) -1.9 (-4.1; 0.2) ARI treatment 29.0 32.4 -3.3 (-15.8; 9.2) 66.7 64.1 2.5 (-8.7; 13.6) 5.9 (-10.9; 22.7) Diarrhea incidence 13.0 16.2 -3.2 (-5.5; - 0.8)* 14.2 13.8 0.4 (-1.6; 2.4) 3.6 (0.5; 6.7)* Diarrhea treatment 28.5 30.5 -2.0 (-10.8; 6.8) 28.7 31.5 -2.8 (-9.3; 3.7) -0.8 (-11.8; 10.2) Full immunization 33.8 38.3 -4.5 (-11.1; 2.2) 39.1 37.9 1.2 (-5.1; 7.4) 5.7 (-3.3; 14.7) ITN child 45.8 42.3 3.5 (0; 7.0)* 51.8 45.7 6.1 (3.4; 8.8)* 2.6 (-1.7; 6.9) Early BF 64.7 58.5 6.2 (0.5; 11.9)* 84.0 86.6 -2.6 (-7.1; 1.9) -8.8 (-16.1; -1.5)* Exclusive BF 62.6 70.9 -8.3 (-16.5; 0)* 67.8 70.5 -2.7 (-10.5; 5.1) -5.7 (-5.7; 17.1) * p<0.05;; Diff – Difference in proportions; DID – Difference-in-Differences; CI – Confidence Interval Awareness creation-related activities were implemented among the HEWs regarding the treatment of children at the health posts level. However, given that there are gaps in this “If we can treat these cases at the health posts level, patient load at the health centers can be reduced. On the other hand, what we consider as one of the problems is improving the community’s awareness related to service uptake. Concerning service utilization, awareness creation is needed. Uptake of services, particularly child health services, maternal health services during pregnancy, antenatal care, and others, is good. However, we believe that community awareness has to increase and is slowly improving concerning institutional delivery.” USAID/ Ethiopia Transform Program Midline Evaluation Report Page 66 regard, cases of pneumonia have increased at the health center level. According to one KI in SNNP region: E. Water, Hygiene, and Sanitation (WASH) This section considers Transform program performance regarding WASH, CBHI enrollments, and gender. WASH interventions are paramount in protecting children and mothers against communicable diseases, such as diarrhea, and creating more enabling health environments. WASH-related performance was assessed using a series of indicators including access to basic sanitation facilities, availability of handwashing stations with soap and water, and use of water treatment technology at the household level. Performance on gender was measured by women’s participation in decisions regarding their own health care, the share of women accompanied by their spouses during ANC visits, and the share of women accompanied by their spouses during delivery. Performance on CBHI was assessed using the share of households that are enrolled in CBHI schemes. Changes in Proportions of WASH Indicators Figure 19 below depicts the performance change in sanitation and hygiene indicators between baseline and midline in Transform program intervention areas and non-Transform program areas. In Transform program intervention areas, handwashing facilities with soap/ash and water was the only indicator that showed significant improvement from baseline, whereas the other two indicators did not change from baseline. Households that have handwashing facilities with soap/ash and water increased from 1.1% at baseline to 3.1% at midline (p< 0.01). In the non￾Transform program areas, households using water treatment technology increased significantly from 10.4% at baseline to 13.3% at midline (p = 0.012), whereas access to basic sanitation facilities significantly declined over time. USAID/ Ethiopia Transform Program Midline Evaluation Report Page 67 Figure 19: Sanitation and Hygiene Indicators Performances in USAID Transform Program Intervention and Non-intervention Areas, Baseline (2017) Vs Midline (2019) F. CROSS CUTTING: Gender and Community-Based Health Insurance (CBHI) Gender-related interventions aim to enhance women’s decision-making opportunities in the health context. CBHI enrollment is also believed to empower women by creating demand for health services and leading to increased service utilization for themselves and their children. Cognizant of these potential benefits, the Transform Program invested in CBHI to improve households’ enrollment and renewal alongside a separate USAID health financing activity that promotes CBHI. Changes in Proportions of Cross-Cutting Indicators Community-Based Health Insurance: CBHI enrollment demonstrated improvement in Transform program intervention areas as the share of households enrolled in CBHI schemes increased significantly from 16.3% at baseline to 31.8% at midline (p <0.01) (Figure 20). Similar performance improvement was observed in non-Transform program areas, though to a lesser degree (8.7 percentage points, p < 0.01). At midline, of the households enrolled in CBHI schemes, a somewhat lower proportion of households in the Transform program intervention areas renewed their membership recently compared with non-Transform program areas (74.2% versus 76.1%, respectively). Figure 20: CBHI Enrollment in Transform Program Intervention Areas and Non-intervention Areas (Baseline 2017 Versus Midline 2019), March 2019 9.8 15.0 1.2 2.1 11.5 10.0 10.4 6.5 3.1 2.0 10.8 13.3 Intervention Non-Intervention Intervention Non-Intervention Intervention Non-Intervention Households that have access to basic sanitation facility Households that have handwashing facility with soap/ash and water Households using water treatment technology Baseline Midline 16.3 17.2 31.8 25.9 Transform Program Intervention Non-Transform Program areas CBHI Enrollement Baseline Midline USAID/ Ethiopia Transform Program Midline Evaluation Report Page 68 Gender–Women’s Decision-Making and Male Involvement: Regarding gender-related indicators, no promising change was observed in Transform program intervention areas (Figure 21). Women accompanied by their spouses during the birth of their last child increased slightly from 81.2% to 83.2%, though the change was not statistically significant (p =0.18). Conversely, women's participation in decisions regarding their own health declined statistically significant from 83.3% at baseline to 81% (p <0.01). Women accompanied by their spouses during at least one ANC visit for their last birth also declined significantly, from 54.2% at baseline to 44.2% at midline (p <0.01). A similar performance trend was observed in non-Transform program areas, as well, though the magnitude of change differs. Figure 21: Performance on Gender Indicators in Transform Intervention Areas and Non￾Transform Program Areas, Baseline (2017) Versus Midline (2019) Availability of Gender-Related Health Services at Primary Health Care Facilities The health facility assessment examined gender-related capacity building activities and services provided at the facility level. In Transform program intervention areas, 69% of health centers and 87.5% of primary hospitals have staff trained on topics related to gender. Specifically, about half of the health centers (48.6%) and 54% of primary hospitals have staff trained on male engagement, gender-based violence, or gender mainstreaming. In addition, 83% of the health centers and 78% of the primary hospitals reported making efforts to integrate gender in reproductive, maternal, newborn and child health services. Some of the activities that health facilities report undertaking during the period of Activity include: a) providing counseling services on gender-based violence (GBV), genital mutilation, gender equity, and female rights to education; b) promoting male engagement in RMNCH; c) organizing pregnant woman conferences and community level awareness creation events on gender including in schools; d) advocating for government to prioritize gender issues; and e) providing treatment to those exposed to violence and linking them to legal bodies. In non-Transform program areas, 50% of health centers and two out of the three primary hospitals have staff trained on gender-related topics. Specifically, only 26.5% of the health centers and three out of the four primary hospitals have staff trained on male engagement, GBV, 83.3 82.3 54.2 56.0 81.2 77.2 81.0 80.9 44.2 46.3 83.2 84.3 Intervention Non-Intervention Intervention Non-Intervention Intervention Non-Intervention Women's participation in decisions regarding their own health Women accompanied by their spouses during ANC visits for their last birth Women accompanied by their spouses during the birth of their last child Baseline Midline USAID/ Ethiopia Transform Program Midline Evaluation Report Page 69 or gender mainstreaming. Furthermore, 60% of the health centers and three out of the four primary hospitals reported having exerted efforts to integrate gender in reproductive and MNCH/FP services. The Difference-in-Differences Analysis on WASH, Gender and CBHI Outcomes: The DID analysis assessed the contribution of the Transform program to WASH, gender, and CBHI outcomes (Table 11). The findings indicate that the difference in the proportion of basic sanitation facilities availability increased by 7.3 percentage points (95% CI: 4.9% – 9.6%; p<0.001) and enrollment in CBHI by 8.2 percentage points (95% CI: 5.1% – 11.3%; p<0.001). However, the proportion of households using appropriate water treatment methods declined over the intervention period by 3.3 percentage points (95% CI: 0.9% – 5.6%; p<0.001). Table 11: Difference in Proportions and DID of Cross-Cutting Indicators, Disaggregated by Transform Program Intervention and Non-intervention Areas, Baseline 2017 and Midline 2019, March 2019 Indicators Baseline Mid-term DID Interv. (%) Non￾interv. (%) Diff (95% CI) Interv. (%) Non￾interv. (%) Diff (95% CI) Cross-cutting Basic sanitation 9.7 14.2 -4.5 (-6.3; -2.7)* 9.9 7.1 2.8 (1.2; 4.4)* 7.3 (4.9; 9.6)* Hand washing station with water and soap 0.8 1.4 -0.6 (-1.4; 0.2) 2.3 2.2 0 (-0.5; 0.5) 0.6 (-0.4; 1.6) Appropriate water treatment 10.9 10.4 0.5 (-1.3; 2.3) 10.8 13.6 -2.8 (-4.4; - 1.2)* -3.3 (-5.6; - 0.9)* Spouse Accompany ANC 51.9 56.2 -4.3 (-8.4; -0.2)* 44.3 46.9 -2.7 (-8.2; 2.8) 1.6 (-5.3; 8.5) Spouse Accompany Delivery 73.6 76.8 -3.2 (-6.3; 0) 83.0 87.2 -4.1 (-10.1; 2.0) -1.0 (-7.9; 5.9) CBHI 16.2 15.9 0.3 (-2.0; 2.6) 31.8 23.3 8.5 (6.3; 10.6)* 8.2 (5.1; 11.3)* * p<0.05;; Diff – Difference in proportions; DID – Difference-in-Differences; CI – Confidence Interval 2.3.2. Effectiveness in Delivery Methods and Programming Approaches To evaluate broader effectiveness in the delivery of interventions, the evaluation team relied on reports from KIIs associated with each of the three principal activities: PHC, HDR, and WASH. Transform PHC: The Transform: PHC Activity designed and implemented coaching and mentorship support using the public health system structures – woreda-level interventions, either at woreda health offices or at health facilities – to ensure effective service delivery and program management. Routine and random follow-ups identified and addressed gaps in service provision. USAID/ Ethiopia Transform Program Midline Evaluation Report Page 70 Catchment based clinical mentorship, leadership development coaches, public finance management mentorship, connected woreda mentorship, and quality improvement mentorship are Transform components that build sustainable capacity at the RHB and at the woreda levels, based on the evaluation team’s observation and conversations with stakeholders. The GOE has adopted some of the innovative approaches introduced by Transform: PHC for effective service delivery, including mentorship guidelines, sub-grant management to woredas, and twinning partnerships. On the other hand, review of Transform: PHC performance reports revealed that, although the context is being monitored, there was no systematic reporting and no section dedicated to context monitoring, mitigation, and reporting in the Activity performance reports. In the Ethiopian context, there is a need to deliberately monitor, report, and document context monitoring and mitigation with regard to their effects on programming. Transform HDR: Despite operating in a challenging context, the Transform: HDR Activity has successfully undertaken a number of measures to ensure the effective delivery of services. For example, the Activity has prioritized the following: ● The purchase of portable V-scan ultrasound, BeMOC supplies, etc. enabled the supported woredas to increase demand and utilization of ANC1 to ANC4 follow-up. Provided the equipment remains in good condition, the benefits to local women are tangible. ● GPS mapping to assist mobility assessments for Afar and Somali pastoralist communities. ● An outreach model, one-stop service/center for SGBV, and reinforced ANC labs to improve the quality-of-care interventions. ● Support for community scorecards to hold service providers accountable. ● Routine and random follow-ups to identify and address gaps in service provision. As RHB KI from Afar noted: Transform WASH: The Transform: WASH Activity has prioritized a number of innovations that have contributed positively to service delivery methods and that have been received well in woredas. For example, the Activity has: ● Allowed for a range of price points within its four business models, commensurate with business upgrades. ● Provided tailored business development support to business partners, including technical skills training to ensure the quality of sanitation product production. “Transform: HDR unlike other partners operating in the region is unique in its approaches as it is: (1) providing grass roots level supervision and coaching, (2) providing relevant training to fill perceived knowledge gaps of health care workers, (3) being flexible based on the demands of the woreda, and (4) performing multiple activities.” USAID/ Ethiopia Transform Program Midline Evaluation Report Page 71 ● Involved sales agents in the sanitation marketing business at the community level, which case study evidence suggests has increased the supply, demand, and use of products. The Transform WASH Activity’s door-to-door approach to sales and installation of improved sanitation products has not been universally well received, as local masons do not view these products as a profitable business. Some think it takes time away from their existing business in some intervention areas and yet it is not all that well-paying. Evidence from KIIs and case studies points to potentially dropping this marketing approach and focusing more on effective marketing that translates into higher sales and utilization rates. 2.3.3. Stakeholder Satisfaction with the Transform Program Transform PHC: Evidence from the evaluation suggests that the strategies and results of the Transform PHC Activity are well aligned with the shared FMOH and USAID/Ethiopia strategic priorities for the health sector and are an important element in the GOE’s HSTP priorities. Co￾creation from the outset of the design ensured a clear understanding, buy-in, and ownership among stakeholders. The relevance of the Activity was echoed by numerous stakeholders, including FMOH, and regional and woreda stakeholders. All FGDs, including KIs at regional and woreda levels, said that Transform PHC is supporting activities relevant to their needs and priorities. A KI Director at the FMOH summed it up in the following statement: Transform HDR: Many stakeholders interviewed commented that the IP team was responsive to the region’s requirements and was able to respond at important moments, such as during periods of emergencies. Furthermore, KIIs indicated the following general trends: ● They view the IP team’s TA and engagement as being highly supportive of the regions’ and woredas’ MNCH/FP objectives and work plans. ● They view Transform HDR interventions as consistent with the strategies and priorities of the regional health bureaus and local contexts. ● They have been pleased with their engagement and the complementarity of efforts, and they acknowledge the role and importance of Transform HDR in supporting regions in areas of MNCH/FP. Transform WASH: Several stakeholders interviewed commended the team for being responsive to market needs and the involvement of sanitation enterprises. Another notable accomplishment “Transform Primary Health Care (PHC) is our project, we’re involved in the co-creation exercise that involved us at FMoH, RHB, USAID and this process aligned the project results to the HSTP strategic priorities. Therefore, Transform PHC is supporting government priorities. The public sector is doing everything while Transform: PHC is playing coordination and facilitation roles.” USAID/ Ethiopia Transform Program Midline Evaluation Report Page 72 for the program has been strengthening the business skills and knowledge of regional/zonal stakeholders, which those stakeholders have recognized as an important benefit. These influential local leaders at nearly every site surveyed were articulate and knowledgeable about the One WASH activities. There are still some stakeholders who feel that the WASH Activity should also incorporate water-related programming in an explicit manner, particularly regarding strategic partnerships that could address water supply and hygiene issues in a more robust manner. Generally, however, the KIIs suggest that stakeholders are notably pleased with the WASH interventions as of the midline evaluation. 2.3.4. Enabling and Hindering Factors Regarding Effective Implementation A review of the Transform: PHC Activity indicates that cultural and traditional restrictions, fear of side effects, and poor health service provision beyond the Activity interventions impose potential limitations on effectiveness regarding MCPR outcomes. Other factors that hinder Activity success, particularly in terms of the utilization of health services, include women's lack of awareness of the importance, distance to health facilities, and limited availability of affordable transportation. On the other hand, important determinants that facilitate the use of health services include higher education levels, women's ability to make healthcare decisions, partner involvement, and MCH messaging. KIIs also noted the social context that some of these regions face, which can create opportunities, but which can also cause stakeholders to move cautiously in implementing changes due to uncertainty over the pending landscape. For example, a KII with an RHB in Tigray provided some potential reasons for the low MCPR rate as follows: Data on Transform: HDR notes one key hindrance to program effectiveness – the vast area to cover with a limited technical staff. Relatedly, Transform HDR faces huge logistical challenges due to the remote nature of these regions, poor roads, limited transportation, and communications barriers. Security/mobility remains a challenge in some parts of the developing regions. Furthermore, a particularly weak health system and infrastructure in the developing regions, including lack of electricity, water, etc., limit the effectiveness of Transform HDR. Finally, KIIs revealed some deep-rooted cultural norms and traditions – such as patriarchal dominance and large family status – that may hamper efforts to change behavior, particularly regarding outcomes such as female genital mutilation (FGM), GBV, and childbirth spacing. This issue came up routinely in conversations with stakeholders. “Cultural and traditional restrictions, fear of contraceptive’s side effects, lack of women’s autonomy in health care decisions, and poor health service provision are some of the factors that might be responsible for hindering the region’s MCPR performance.” USAID/ Ethiopia Transform Program Midline Evaluation Report Page 73 A review of the Transform: WASH Activity also indicates some challenges due to local gender dynamics and other cultural norms. For example, a KII in Oromia indicated that the female sales agents were not always warmly embraced by suppliers, despite their positive effects. However, the evaluation team has found that the WASH Activity is well-suited to adapt to contextual factors. As one KI from the FMOH stated: Beyond the environmental and cultural factors, a limitation to the WASH activities is the credit worthiness of potential enterprises, which may not have had an opportunity to establish an adequate credit history or collateral prior to the Transform WASH interventions. Interest rates remain high for many potential stakeholders, which hinders their engagement in Transform WASH. Finally, the urgency to fully implement activities and to seek profits under the WASH initiatives has meant that piloting and testing of implementation strategies has perhaps not received the full attention it deserves. Piloting various strategies for marketing, design, and sales could streamline the activities and ensure greater profits after an initial investment of time. 2.3.5. Recommendations for Achieving Intended Results Some concrete recommendations that follow from the evaluation team’s review of Transform program effectiveness include the following: ● To combat some of the cultural challenges, the Transform program might collaborate more closely with the education sector to promote female school retention and enrollment as women in secondary education are 1.4 times more likely to use contraceptive methods. ● To ensure behavior change in some of the areas in which little progress was observed, the Transform program might tailor interventions to younger women as a means of reaching the target audience before childbirths mount and unmet need increases. ● The Transform program can expand its work with communities and religious leaders to change social norms that might be inhibiting uptake of contraceptives and other program initiatives. ● To ensure adequate outreach to the more challenging geographic areas covered by the program, the Transform program might consider adding more technical assistants and making those TAs available to woredas through more and smaller teams. ● The Transform program might consider moving WASH activities away from door-to￾door sales and focusing instead on community-wide marketing while working to build stronger relationships with potential lenders so that more entrepreneurs have access “Several steps have been taken by the Transform: WASH to respond to the local context, and to make sure the interventions represent a good contextual as well as cultural fit into the local context of intervention target areas.” USAID/ Ethiopia Transform Program Midline Evaluation Report Page 74 to necessary credit. ● The Transform program may also promote efforts among HEWs to explicitly target intervention themes in which improvement did not occur over the period of Activity, including pre-natal ANC visits and exclusive breastfeeding. Indicators that did not improve can also be stressed in gender outreach work with male partners. 2.4. TRANSFORM PROGRAM SUSTAINABILITY The sustainability of the Transform program refers to its capacity to ensure a continuation of program activities and benefits beyond the lifecycle of the grant and the period of performance. To evaluate the sustainability of the Transform program, the evaluation team considered the strategies and approaches put in place to achieve sustainability, the program’s role in mobilizing local stakeholders, best practices that can be taken to scale in subsequent years, key challenges to program sustainability, and recommendations regarding what can be done to ensure and enhance sustainability. 2.4.1. Sustainability Elements, Strategies and Approaches Put in Place Each of the three program activities put in place mechanisms to enhance overall program sustainability. The following outlines notable steps taken in this regard. Transform PHC: PHC has integrated the following strategies/approaches and elements into its interventions to help ensure the sustainability of its intended results after the project ends: ● Transform PHC implementation modalities are treated as an integral part of the government health sector, so there is no parallel implementation structure. ● Some local governments (e.g., the Oromia special zone of the Amhara region) have adopted a twinning approach of allocating resources for peer-to-peer learning among high- and low-performing woredas within their zones. Twinning of high-performing woredas to low-performing woredas has allowed the low-performing woredas to learn from their high-performing counterparts. ● Transform PHC has also incorporated coaching and mentorship support at the woreda level, including clinical mentorship, leadership development coaches, public finance management mentorship, connected woreda mentorship, and quality improvement mentorship. So far, the expansion of these initiatives through public sector resources as well as other partners is going well, which signals that the strategy is effective and a candidate for scale-up. Transform HDR has also instituted a number of mechanisms to enhance sustainability. Site visits by evaluators revealed there remains a need to identify sustainable approaches to scale-up MNCH/FP services for pastoral populations in the Transform: HDR intervention areas. Some of the concrete approaches to sustainability include: USAID/ Ethiopia Transform Program Midline Evaluation Report Page 75 ● A partnership with Emory University that offers in-service training, continuous supportive supervision and review meetings, all of which help to build the capacity of local stakeholders so they can sustain program progress. ● Woreda Transformation: Transform HDR supports government efforts to promote woreda self-reliance and the sustainability of MNCH/FP interventions. ● Improving Technical Capacity: Transform HDR provides key RMNCH/FP training packages and ensures consistent implementation of the packages through regular mentoring and supportive supervision by skilled mentors. ● Institutional Capacity: Transform: HDR supports the RHBs and their structures to enhance their operational and leadership capacity through Leadership, Governance, and Management training. ● Monitoring and Evaluation: Transform: HDR promotes the use of community scorecards and engagement to provide oversight for the quality of care and customer satisfaction, which ensures that best practices are rewarded and quality care becomes a lasting and sustainable norm. The results are positive, yet more needs to be done. Transform WASH has at its core the goal of sustainability: The Activity aims to create and support profitable private sector enterprises that will endure and succeed beyond the lifecycle of WASH. Some of the approaches WASH has taken to achieve sustainability include: ● The introduction of four business models for household construction and fabrication of sanitation products. ● Capacity building trainings at the woreda level so that stakeholders can plan, procure, and manage OWNP activities. ● Implicated HDAs and Women’s Development Armies (WDAs) directly in sanitation marketing and business ownership that businesses can expand and remain generating new revenues after the Activity ends.24 The evidence shows that the program has implemented a series of feasible initiatives that promote the likelihood of sustainability in the long run. However, the health system is largely depending on external funding and self-reliance is unrealistic in the short run. The Transform program does include Crisis Modifier Funds, which provide extra assistance in the event of emergencies that help to keep community health provisions from backsliding. But building 24 HDAs and WDAs are groups of volunteer community health workers. For more information, see Svea Closser, Harriet Napier, Kenneth Maes, Roza Abesha, Hana Gebremariam, Grace Backe, Sarah Fossett, Yihenew Tesfaye, Does volunteer community health work empower women? Evidence from Ethiopia’s Women’s Development Army, Health Policy and Planning, Volume 34, Issue 4, May 2019, Pages 298–306, https://doi.org/10.1093/heapol/czz025. USAID/ Ethiopia Transform Program Midline Evaluation Report Page 76 resilience and reaching self-sufficiency require a larger, longer term commitment of government resources. In addition, the human resource situation is dire, both in quantity and quality, and leadership instability remains high. 2.4.2. Mobilizing Key Stakeholders The twinning initiative that is part of Transform: PHC has served as a concrete way to mobilize local stakeholders, allowing those in lower-performing woredas to draw lessons from their counterparts in higher-performing woredas. The coaching and supervision aspects of PHC have also helped to generate buy-in from stakeholders. One KI from the Huleteju Woreda Health Office had this to say about the mobilizing effects of the coaching and supervision initiatives: Transform: HDR also receives high marks from the evaluation team in terms of its mobilization of local stakeholders. In particular, HDR has worked closely with existing government structures, which has helped to reinforce buy-in at both the central GOE and the local levels. In addition, the Leadership, Governance, and Management training have been instrumental in enhancing leadership capacity, as have the coaching and mentorship aspects of the Activity. The decision to ultimately focus the full slate of activities on a reduced number of woredas further helped to reinforce the support that local stakeholders feel in the intervention locations. Transform: WASH devoted significant effort to establishing and supporting WASH teams at different levels, including Woreda WASH Teams (WWT), WASH Technical Teams (WTT), Kebele WASH Teams, and WASH Committees (WASHCOs) that can take over the activities after the completion of Transform WASH. There remains a challenge in terms of mobilizing local stakeholders to market sanitation products, as it is not viewed as profitable and poses a threat to the profitability of new sanitation enterprises. Continued marketing initiatives may help to keep those entrepreneurs engaged. 2.4.3. Enhancing Ownership at Different Levels Grants to woredas have provided an important mechanism for Transform: PHC to enhance ownership at different levels, in particular by filling budget gaps with local input. Furthermore, KII respondents noted that they appreciate the experience of grants management, including the oversight of these grants, which has provided valuable expertise for woredas to implement future performance-based financing that may be provided by the GOE. The FMOH has replicated many aspects of the woreda subgrant management in other non-Transform: PHC woredas using government financing. This process engages both the woredas and the FMOH, enhancing engagement and ownership across both levels. “If you conduct onsite coaching and onsite supervision, the institutions will sustain. Furthermore, if we identify training gaps in a certain health institution, then the training will be given there and the gaps become filled; in this way the institution becomes capable of providing all services properly. Therefore, Transform PHC is playing [a] vital role in sustainability. If you take previous partners, they came with their own agenda, whether it was long-term or not, and endorsed funds to run that agenda only.” USAID/ Ethiopia Transform Program Midline Evaluation Report Page 77 The design for Transform HDR started with a highly participatory consultation process based on data generated from the national and regional levels that helped the GOE to align the design with its priorities. A second factor that has reinforced ownership is the participatory annual joint planning exercise where RHBs and other sector offices jointly plan work assignments and adjust interventions and their roll-out according to regional health sector priorities. The annual planning exercise between Transform: HDR and regions and woredas ensures continued relevance of the Activity’s interventions, strategies, and approaches within regional contexts. Finally, the technical assistance initiative at the woreda level has helped to ensure local ownership over implementation. Transform: WASH’s design specifically addresses sanitation marketing, which builds on a key GOE/FMOH WASH strategy. Further, Transform WASH plans are approved by, and emanate from, the GOE’s OWNS. Finally, a basic tenet of the GOE’s sanitation marketing strategy, and one embraced by Transform: WASH, is that all programs must nurture and advance household and woreda level ownership and sustainability. WASH has aimed to do this by empowering local entrepreneurs to advance private sector opportunities for sanitation improvement. Overall, the FMOH and woreda level stakeholders’ express perceptions of real ownership regarding Transform program activities. There may be opportunities to further implicate regional level stakeholders in the oversight process, but ownership in the critical areas of planning (at the FMOH level) and implementation (at the woreda level) appears to be thriving at this point in the program lifecycle. 2.4.4. Key Obstacles to Sustainability While the Transform program seems to have already made a positive impact on some key indicators related to child and maternal health, there remain obstacles to the program’s sustainability. Some obstacles are resource-related, while others are highly dependent on contextual and environmental factors. The evaluation team noted the following potential obstacles to sustainability from KIIs, discussions, and observations of health facilities. ● Health sector leadership and technical staff turnover remain a challenge and may hinder sustainability. KIs reported that there continues to be high health worker staff turnover and very high vacancy rates for key clinical positions across all regions. If those problems persist, it will be more difficult to ensure sustained continuation of programming. ● Harder-to-reach areas were found to require more intensive day-to-day, hands-on technical assistance. While reaching such areas has been a priority in the Transform program, sustaining outreach to those areas may become more challenging later. ● Resource and technical materials provided to woredas pose a dilemma. It is important to incentivize health workers and to provide the equipment that allows clinics to serve their target populations, but it may undermine sustainability if equipment is in need of repairs that are not forthcoming later. USAID/ Ethiopia Transform Program Midline Evaluation Report Page 78 ● A related and pressing concern is that regions have limited budgetary and resource mobilization capacities that limit their ability to become self-reliant. ● Vaccine stock-outs and logistics problems remain high and may become more pronounced following Transform programming. ● Cultural factors and low education levels are difficult to overcome within the program’s short time horizons, so progress made during the course of the program could be difficult to sustain in the absence of regular programming. ● The ability of entrepreneurs to obtain credit and foreign currency to import sanitation products poses a longer-term threat to the sustainability of the WASH Activity in particular. ● Affordability of products may also limit their widespread adoption and the continued success of a market-based approach to sanitation. 2.4.5. Recommended Changes to Ensure Sustainability Steps can be taken to enhance the likelihood of program success beyond the period of performance. Among the feasible action items that might be considered are the following: ● Provide a comprehensive package for systems strengthening support to a selected number of woredas. This could help to improve efficiency and performance in those locations while also establishing a model for scaling and sustainability. ● Consider integrating community interventions in a few woredas, targeting women to improve empowerment and the rolling-out of best practices to scale up to other areas. ● Integrating the HEW fully into the health system could be an effective and efficient means of ensuring program gains in the future. ● Strengthen regional technical capacity by integrating MNCH/FP material from the Activity into the existing pre-service curriculum for health science colleges. ● Assist the developing regions to address HRH staffing issues in a more structured and systematic manner that addresses attraction, recruitment, deployment, and retention. This will establish a pool of health workers for the future. ● A future design should include a private sector innovation fund to pilot and test new commercially viable WASH interventions to reach those who are unable to pay for services. This may create market opportunities that persist beyond the program. USAID/ Ethiopia Transform Program Midline Evaluation Report Page 79 2.4.6. Best Practices to Scale-up in Remaining Years Several practices, some of which overlap across the three core Transform Activities and some of which are unique to a particular Activity, have shown success and merit scaling up: ● The TA initiatives and the co-location of Transform PHC cluster offices with government health offices has reduced costs and improved efficiency. The evaluation team found that co-location was effective as it built teamwork between the Transform: PHC and the RHBs, woredas, and zones. ● The twinning activities for low- and high-performing woredas has been very successful. This also extends to woreda-to-woreda and facility-to-facility (peer-to￾peer) experience sharing which has proven to be an effective means of facilitating learning and improving service delivery. ● Woreda grants have proven to be effective in filling local budget gaps while also promoting ownership. ● Coaching and mentorship activities are successfully helping to build the capacity of local stakeholders in ways that improve both program effectiveness and sustainability. ● While the Activity does not yet have tangible results at this point in the program lifecycle, the “Her Space” initiative is popular, practical, and anecdotally effective in building the knowledge and confidence of young women around reproductive issues. ● The Transform: HDR mobility and settlement pattern studies have led to the design of a Comprehensive Mobile Health Outreach Services strategy, which has enhanced Activity effectiveness. The pilot Comprehensive Mobile Health Outreach Services appear promising and suitable for future scale-up. ● The provision of medical technologies at the woreda level is a concrete way to bolster desired outcomes, as women are more likely to take advantage of the services that reduce child and maternal death when appropriate technologies are available. New medical technologies have helped increase service utilization/coverage. The effectiveness in improving maternal/newborn health outcome will be measured at end-line evaluation. ● GPS mapping has proven to be a helpful means of understanding the needs of, and ultimately serving, populations in, the most difficult areas. ● Capacity building trainings, followed by post-training follow-up and on-site mentorship and coaching, have improved service quality (RMNCH) via regular contributions from the Ethiopian Pharmaceutical Supply Agency (EPSA), representatives of the Ethiopian Society of Obstetricians and Gynecologists (ESOG), and members of the Ethiopian Midwives Association (EMWA). These efforts receive strong support from local stakeholders. ● Service uptake in the Centers of Excellence has shown a marked improvement. More such centers could be added. USAID/ Ethiopia Transform Program Midline Evaluation Report Page 80 ● Business financing solutions, either through seed money, VSLA, or supplier subsidies, are critical to expanding marketing-based sanitation solutions. 2.5. ADDRESSING GENDER The Transform program includes a cross-cutting dimension focused on strategies to address gender equity in the course of improving maternal and child health in Ethiopia. This component has focused in particular on women’s decision-making in health-related matters and on male involvement in the responsibilities around childbirth and care that conventionally fall most heavily on women. In addition, the program has prioritized the availability of gender-related services at health facilities. The evaluation considered the program’s contributions to gender issues, the best practices that emerged in Transform’s gender programming, the obstacles the program has faced in addressing gender equity, and the recommendations that could improve programming around gender issues. 2.5.1. Program Contributions to Gender Issues The Transform program initiated a number of activities to fortify gender equity and women’s involvement in decision-making. Among them are the following: ● Family counseling to promote the presence of spouses at pre-natal ANC visits and delivery; ● Family counseling to promote women’s decision-making around health-related issues; ● Counseling services on GBV, FGM, gender equity, and female rights to education; ● Community-level awareness events on gender, including in schools. Advocating for the government to prioritize gender issues; and ● Providing treatment to those exposed to violence and linking them to legal bodies. 2.5.2. Best Practices in Gender Programming The gender programming component of Transform has met with mixed success. In both the analyses reported above on Effectiveness and in the Transform Activities reports developed separately, indicators for gender equity have, generally speaking, not shown improvement over time or relative to the non-intervention areas, which may indicate the need for a coordinated, multi-sectoral response. The Transform: HDR Activity’s one-stop GBV center at Dubti Hospital in the Afar region achieved anecdotal success. According to KIIs, the initiative has generated useful insight that may help women to get holistic service/support while also offering an opportunity for the different sectors to work together to improve women’s roles in developing programs and benefit from available services. The Transform: WASH Activity’s gender assessment at the outset of the program proved USAID/ Ethiopia Transform Program Midline Evaluation Report Page 81 to be a useful tool for understanding gender dynamics within the context of the WASH marketplace and for identifying barriers to women’s participation in WASH businesses as well as the use of WASH products and services. 2.5.3. Obstacles to Addressing Gender Equity Across the three activities that comprise the Transform program, three important obstacles stand out as barriers to progress on gender equity. First, cultural and traditional norms in Ethiopia implicitly condone the practices of FGM and GBV, creating for the program activities the task of overturning longstanding beliefs in a relatively short period of time. Second, the program faced difficulties in hiring an adequate number of female staff at the regional and sub-regional level. As one KI stressed, “it is immensely challenging to recruit women to work in the developing regions”, and this lack of female staff can undermine the comfort and potential collaboration with women in those localities. Third, the monitoring tools employed at the Activity level often lacked clear and robust means of accounting for progress on gender equity. For example, the Institutional Strength Index used by Transform WASH for monitoring its performance does not include gender-related questions and parameters. Although the Activity has customized UNICEF’s gender monitoring tool used for WASH activities, the tool does not capture gender-related outcome results, which could be added to this critical index. 2.5.4. Recommendations to Improve Gender Programming The cultural barriers to changing norms around gender equity are profound. This calls for an intervention which focuses on influencing and changing the behaviors of the community and contributing to use of MNCH/FP services. The establishment and strengthening of locally acceptable, high-impact community awareness forums using local community-based organizations and religious or clan leaders may significantly contribute to minimizing some of the myths and increasing gender equity. The Transform program may also consider prioritizing activities such as the “Her Space” activity and the provision of gender mainstreaming manuals in health facilities. It is too early to assess the impact of these promising activities, but the evaluation team observed a consistent and concerted effort to roll them out in RHBs. It is important that the program continue to seek ways to generate breakthroughs on gender issues. In addition, the Transform program should support government entities to mobilize and coordinate a multi-sectoral response to ensure that all programming is designed to address the unique needs and mitigate barriers faced by women, men, boys, and girls to access service. USAID/ Ethiopia Transform Program Midline Evaluation Report Page 82 2.6. CHALLENGES AND LESSONS LEARNED Despite the various achievements observed and documented during the first half of the Transform program, as with any activity, there are always areas for improvement. The evaluation team identified some challenges that affect all three of the program activities to varying degrees. In outlining those challenges below, the report also underscores the efforts undertaken so far to mitigate them. In addition, this section notes the innovations that have taken place within the program that have potential to be taken to scale or applied by other partners working on MNCH/FP, as well as the key lessons learned to this point in the program’s lifecycle. 2.6.1. Challenges Faced and Strategies for Mitigating Those Challenges ● Continuous health sector leadership and technical staff turnover remain problematic. KIs reported that there continues to be high health worker staff turnover and very high vacancy rates for key clinical positions. In addition, the program has frequently confronted a shortage of skilled service providers. To remedy this challenge, the Transform program can continue to emphasize its coaching and mentoring activities as well as its TA activity. This will ensure that new health workers have the skills and confidence to succeed in their jobs, thereby reinforcing retention. ● Security has remained an implementation challenge in some parts of the regions, for example, West Oromia and the south of SNNP. This affected the regular provision of technical support and monitoring activities. Relatedly, service provision was more difficult in the hard-to-reach areas. Expanding the model of intensive TA that is used in other regions could be an effective means of at least partially mitigating these challenges. In the absence of additional funding, TA could be shifted from fully staffed areas to harder to reach areas, but additional funding for TA would represent an optimal approach. The security problems would still be difficult to circumvent. ● Limits to the number of technical staff at the implementation level pose a challenge to the timely and effective delivery of services in a vast and challenging environment. This challenge will likely become particularly pronounced when the responsibility falls to the GOE to sustain the program. A potential mitigation strategy is to expand training programs in the near term. ● Deep-seated cultural norms and traditions remain a barrier to Transform program efforts to change behavior, especially related to GBV and FGM, as well as male engagement in childbirth and childcare and resistance to family planning. To address the issue, the Transform program must continue its efforts to raise awareness, especially among younger women and their partners. Younger couples may be more willing to entertain alternatives to a patriarchal relationship, which has been a longstanding cultural norm in Ethiopia and which has led to the acceptance of SGBV and male dominance even among many Ethiopian women. Change may begin to emerge organically as younger couples share more experiences around childbirth, childcare, and family planning. USAID/ Ethiopia Transform Program Midline Evaluation Report Page 83 2.6.2. Innovations to Scale and Apply Elsewhere Despite the entrenched challenges the Transform program has faced, it has also made notable progress on some of the indicators outlined in the Effectiveness section above, in addition to establishing effective programming regarding training, mentoring, and collaboration. The evaluation team sees the following innovations as appropriate for scaling up and for sharing with other partners working on similar issues: ● The program’s networking and partnership initiatives have proved to be important and successful. In addition to participating in numerous different networking groups to improve the allocation of resources, the Transform program has led joint planning activities with the FMOH and has established new guidelines for mentorship training. Staff also take part in a national FP technical working group, a Child Health and Development working group, adolescent and youth health development programs, and technical committees organized by the FMOH. ● Tools, like the mentorship guidelines that Transform: PHC is sharing through different networks and technical working groups, have the potential to improve efficiencies and results, and the TA initiatives help both with effectiveness and sustainability. During the remaining life of the program, broader sharing of the full range of these tools and templates should be a priority. The introduction of new, innovative technologies to facilitate service delivery, such as the introduction of mobile ultrasound, the simplified gestation estimation tool and task shifting for middle level health workers that are part of Transform: HDR, were positively received by stakeholders. These innovations are motivating women to attend ANC services despite the costly long-distance travel to hospitals, and improving the effective utilization and interpretation of essential ANC lab results. This is also something that could be scaled if resources permit. ● Transform has created new mentorship guidelines that are expanding the range of stakeholders who contribute expertise to the initiatives at the national, regional, and sub￾regional levels. This has paid dividends in terms of design, planning, and implementation of activities. Even in cases where results are not yet evidence from key health indicators, the benefits of prioritizing local ownership are likely to improve program effectiveness over time while also supporting the sustainability of Transform. ● As noted above, the innovative twinning activity that pairs low-performing woredas with high-performing ones helps both in terms of effectiveness and sustainability. The initiative could be expanded to include a wider number of woredas or applied to other programs. USAID/ Ethiopia Transform Program Midline Evaluation Report Page 84 2.6.3. Key Lessons Learned The successes and challenges confronted during the course of the Transform program have generated several key lessons. Among them are the following: ● Sub-granting to woredas has proved to be an important mechanism to advance woredas’ MNCH priority activities. Ensuring that the receipt of a small grant does not displace or replace GOE funding should be considered as a criterion for Transform support. ● Decentralized mentorship by teaming and twinning high-performing health centers with low-performing health centers is effective in sharing good practices and improving service delivery. Networking through GOE/FMOH coordination platforms has also proved to be important to sharing tools and experiences. ● The allocation of the Crisis Modifier Fund ensures that there are funds available for crisis￾prone regions when an unforeseen crisis emerges. ● Local stakeholder ownership and engagement at all levels (FMOH, region, zone, woreda, and health facility) are critical to the effective delivery of results and the sustainability of the program. ● Men can play a significant role in MNCH/FP service uptake, but thus far the program has not been able to bridge the cultural divide that limits receptivity to male engagement. ● The random follow-up approach offers opportunities to provide demand-driven technical support and mentoring to health care providers at the health facility level. ● The decision to reduce the number of woredas that receive the full scale and scope of health services from 58 to 20 was an excellent decision, given the challenges imposed by hard-to-reach locations. Regarding Transform WASH in particular, but also as a general rule, pre-testing and piloting new products is vital before making substantial investments in their use. All new product launches need to focus on consumer needs. 3. CONCLUSION 3.1.Transform Program’s Likelihood of Achieving Intended Results and Targets Transform: PHC is on track to meet most of its intended targets by the end date of the Activity. Transform: PHC is positively regarded across the intervention regions by RHBs and woredas and is providing tangible and meaningful assistance to the GOE/FMOH to achieve its desired goal to reduce preventable maternal, newborn, and child deaths. Most importantly, beneficiaries note that Transform: PHC has led to improved access and quality of care. The evaluation found that USAID/ Ethiopia Transform Program Midline Evaluation Report Page 85 some targets could have been set too high. Target setting may need to consider factors like past experiences in achieving similar targets, exert opinions, resources available, contextual factors, etc. Several indicators have demonstrated performance improvement between the baseline (2017) and midline (2019). Although some of the improvements reached statistical significance, especially those related to newborn care, some did not. Some indicators showed declines in performance. Child health indicators such as exclusive breastfeeding, use of ITNs, and prevention of diarrhea all failed to show improvement. Given that these indicators are at the outcome and a few at impact level, it is not surprising that the increments in these indicators are not statistically significant given that results at this level of the results chain often yield significant changes over a longer period. However, the overall performance of Transform: PHC on key performance indicators registered marked improvements, and these improvements are also supported by qualitative data. The Transform: HDR Activity has achieved impressive health results and has successfully built a management and assistance platform that is responsive to the GOE/FMOH’s health sector transformation agenda. The twinning initiative proved to be very effective, and Transform’s mentorship and coaching activities have aided in terms of both effectiveness and sustainability. As a result, Transform HDR is on track to meet most of its intended targets by the end date of the Activity. Program performance varies from region to region, and the hard-to-reach areas will take more work. The Transform WASH design is still relevant and solid. Transform WASH, while slow in diversifying the product line, is advancing towards its intermediate objectives of self-sustaining, profitable enterprises and the steadily rising demand for products while still remaining consistent with the performance management framework outlined in USAID’s Transform WASH original Request for Application. Some targets remain ambitious (e.g., Indicator: Number of people gaining access to a basic sanitation service as a result of USG assistance, Number of households that successfully access financial support to facilitate the purchase of basic sanitation products or services), in particular the scope of anticipated demand and profitability, but as demand for new sanitation products begins to improve, those goals will become more realistic. Based upon the data, the three Transform program activities’ performance is on track to contributing to the FMOH Health Sector Transformational Agenda and to meet most of their intended targets by the end date of the Activity. 3.2.Implications for USAID/Ethiopia Future Programming The evaluation has led to the following conclusions and implications regarding USAID programming efforts around the Transform program to this point: USAID/ Ethiopia Transform Program Midline Evaluation Report Page 86 ● The competitive selection of woredas for multisectoral investments that support the achievement of MNCH/FP objectives can strengthen outcomes. Future activities should be designed in the same regions, ensuring that the woredas have a robust transformation agenda that will complement and reinforce USAID/Ethiopia’s health interventions. ● Co-creation in designing new projects has proven to be an effective tool. This approach should also include bottom-up co-creation with the regions and woredas. ● Benefits could come from building links to water and sanitation, female education programs, and potential income generation and livestock management for pastoralist areas (Afar and Somali) into future programming. ● Context-specific interventions tailored to different regional contexts could help to improve upon program gains. This could also include a relatively higher investment for Afar and Somali compared to Benishangul-Gumuz and Gambella to realize equity. ● Demand for new sanitation products can be strengthened through expanded marketing efforts at more sites in the country. Sanitation marketing efforts might also be intensified as part of programming in existing intervention sites in order to strengthen demand for products and services. ● A pressing need for financing remains for WASH-related enhanced water supply, sanitation, and hygiene marketing. More robust outreach from USAID/Ethiopia to financial institutions that provide credit to entrepreneurs is also needed. 3.3.Implications for GOE/FMOH Future Programming An important conclusion of the evaluation for the GOE/FMOH is that coordination mechanisms with key stakeholders to guide and tailor actions towards improving synergistic efforts related to MNCH are critical. In addition, sustainability goals that can be integrated into ongoing programming will ensure that results are sustained beyond the life of the Transform program. This could include a deliberate effort made now to identify and define resiliency elements to be integrated into program interventions and strengthened in crisis-prone regions. Third, resource mobilization will be critical if the FMOH hopes to replicate program efforts to provide technical resources to health facilities in need. Resource mobilization will also be critical to support enhanced evidence-based decision-making and ensure leadership stability. Fourth, based on the findings of this evaluation, the GOE/FMOH has reasons to support the scale￾up of woreda-to-woreda and facility-to-facility (peer-to-peer) experience sharing to facilitate learning, improve service delivery, and enhance performance. Fifth, it may be important for the GOE/FMOH to assist the developing regions in managing their HRH staffing issues in a more structured and systematic manner that addresses attraction, recruitment, deployment, and retention. This may include recognition of high-performing staff and health teams, recruitment and retention bonuses for those who agree to serve in the USAID/ Ethiopia Transform Program Midline Evaluation Report Page 87 developing regions, and continuing professional development opportunities when they faithfully work in their placement for a certain period of time. In the short run, the government may focus on supporting training and demonstration centers at the regional level in collaboration with local universities. USAID/ Ethiopia Transform Program Midline Evaluation Report Page 88 4. RECOMMENDATIONS From KIIs, FGDs, survey results, interviews at health facilities, and the review of Transform activity reports, the evaluation team noted numerous successes and some challenges, all of which can be taken into account to improve the Transform program as well as future programming. Below are some of the key recommendations that could positively influence Transform at the program level. Recommendation/Action Responsible Agent Timing Priority # USAID/Ethiopia 1 USAID/Ethiopia should work with implementing partners to identify ways to increase efficiency of existing resources to achieve Activity and program targets. USAID/Ethiopia Immediate 2 USAID should work with the FMOH/RHBs to build a resilient health system for the MNCH/FP for the remaining Transform program period. USAID may wish to provide leadership in defining health system resiliency elements for the MNCH/FP theme. USAID/Ethiopia Immediate 3 USAID/Ethiopia should set Transform program-level targets to measure the effectiveness of the program as a whole. These targets should also have some indicators that allow aggregation/consolidation of performance across the Transform portfolio implementing mechanisms. This will allow the Transform activities to be the drivers by which the Transform program targets are achieved, and hence contributors to Transform program targets. USAID/Ethiopia Immediate 4 USAID/Ethiopia should promote the twinning of low￾performing and high-performing woreda health facilities (already being supported by some Transform Activities) to take advantage of emerging best practices and to reinforce collaboration. USAID/Ethiopia Immediate 5 USAID/Ethiopia should consider expanding TA in thematic areas with little discernible performance improvements to date. USAID/Ethiopia Immediate USAID/ Ethiopia Transform Program Midline Evaluation Report Page 89 Recommendation/Action Responsible Agent Timing Priority 6 USAID/Ethiopia should forge additional and stronger relationships with key local stakeholders and partners, including financial institutions who provide credit to entrepreneurs. USAID/ Transform: WASH Immediate 7 USAID/Ethiopia should consider competitively selecting woredas that have multi-sectoral investments that support the achievement of MNCH/FP objectives. Future activities should be designed in the same regions, ensuring that the woredas have a robust transformation agenda to complement and reinforce USAID/Ethiopia’s health interventions. The need for multi-sectoral response systems should be emphasized as some of the interventions are difficult to be successfully implemented and sustained by the health sector alone. USAID/Ethiopia Future Programming / Beyond the life of Transform Program 8 USAID/Ethiopia should continue using co-creation in designing new projects. USAID/Ethiopia Beyond the life of Transform Program 9 USAID/Ethiopia should explore innovative ways to strengthen coordination amongst key health and non￾health development actors operating in the developing regions to improve synergy, to avoid duplication, and to achieve better results. USAID/Ethiopia Immediate/ Beyond the life of Transform Program 10 USAID/Ethiopia should consider some of the deteriorating outcomes in MNCH/FP performance in non￾Transform sites and identify ways to leverage additional domestic resources and other donor and private sector investments to accelerate the expansion and to enhance the quality of MNCH/FP services in these areas. USAID/Ethiopia Immediate/ Beyond the life of Transform Program 11 To narrow the gender gap, USAID/Ethiopia should consider targeting younger women and their partners in order to begin changing cultural norms over time. USAID/Ethiopia Immediate/ Beyond the life of Transform Program USAID/ Ethiopia Transform Program Midline Evaluation Report Page 90 Recommendation/Action Responsible Agent Timing Priority 12 USAID/Ethiopia should reinforce the already impressive coaching and mentoring initiatives to build local capacity and to provide a sustainable foundation for MNCH/FP progress. USAID/Ethiopia Immediate 13 USAID/Ethiopia should promote an expanded and accelerated financial investment by the GOE to ensure equity and accelerated outreach to the hard-to-reach communities. Transform: HDR Immediate/ Beyond the life of Transform Program 14 USAID/Ethiopia should set aside funding that can be used for unforeseen but critical needs. For example, for innovative ways for businesses to reach consumers with new health and sanitation products, for targeting areas where the midline results indicate a need for greater attention, and for emergency support related to the COVID-19 pandemic. Transform: WASH Immediate/ Beyond the life of Transform Program # Government of Ethiopia (GOE)/Federal Ministry of Health (GOE/FMOH) 15 The GOE/FMOH should strengthen coordination mechanisms with key stakeholders to tailor actions towards improving synergistic efforts related to MNCH/FP. GOE/FMOH, RHBs Immediate/ Beyond the life of Transform Program 16 The GOE/FMOH should improve resource mobilization and information use to support enhanced evidence￾based decision-making and to ensure leadership stability. GOE/FMOH, RHBs Immediate/ Beyond the life of Transform Program 17 The GOE/FMOH should consider using twinning as a viable approach for technical assistance across regions. GOE/FMOH, RHBs Immediate/ Beyond the life of Transform Program USAID/ Ethiopia Transform Program Midline Evaluation Report Page 91 Recommendation/Action Responsible Agent Timing Priority 18 The GOE/FMOH should support the scale-up of woreda￾to-woreda and facility-to-facility (peer-to-peer) experience sharing to facilitate learning, to improve service delivery, and to enhance performance. GOE/FMOH, RHBs Immediate/ Beyond the life of Transform Program 19 The GOE/FMOH should explore innovative ways to strengthen coordination amongst key health and non￾health development actors to maximize synergies. GOE/FMOH, RHBs Immediate/ Beyond the life of Transform Program 20 The GOE/FMOH should expand the dissemination of Transform program-wide best practices to build sustainable programs that build on the successes of Transform: PHC, Transform HDR, and Transform: WASH. GOE/FMOH, RHBs Immediate 21 The GOE/FMOH should assist the developing regions in managing their human resources for health (HRH) staffing issues in a more structured and systematic manner that addresses attraction, recruitment, deployment, and retention. This may include recognition of high performing staff and health teams, recruitment and retention bonuses for those who agree to serve in the developing regions, and continuing professional development opportunities when health staff faithfully work in their placements for a certain period of time. In the short run, the GOE/FMOH may focus on supporting training and demonstration centers at the regional level in collaboration with local universities. GOE/FMOH, RHBs Immediate/ Beyond the life of Transform Program USAID/ Ethiopia Transform Program Midline Evaluation Report Page 92 U.S. Agency for International Development USAID/Ethiopia Entoto Road Addis Ababa, Ethiopia