Mid-term Performance Evaluation Report – Transform: PHC Activity Page 1 TRANSFORM: MONITORING, EVALUATION, LEARNING AND ADAPTING (TRANSFORM: MELA) ACTIVITY MID-TERM PERFORMANCE EVALUATION OF USAID TRANSFORM PRIMARY HEALTH CARE (PHC) ACTIVITY FINAL REPORT July 2020 Submission date: July 28, 2020 USAID/Ethiopia Gebeyehu Abelti, COR Submitted by: The Mitchell Group, Inc. (TMG) Contract Number: OAA-I-15-00028 Task Order No.: AID-663-TO-17-00001 Transform: MELA Activity, Addis Ababa, Ethiopia Dr. Geoffrey Olupot, Chief of Party Email: golupot@ethiopiatmela.com Phone: +251-986 356 914 Principal Contacts: Jenkins Cooper, Project Coordinator / Vice President, TMG Email: jenkinsc@the-mitchellgroup.com Phone: +001-202-567-1097 (Mobile) +001-202-350-0025 (Direct) This document was produced for review by the United States Agency for International Development Ethiopia (USAID/Ethiopia). It was prepared by The Mitchell Group, Inc. (TMG) for USAID/Ethiopia for the “Transform: Monitoring, Evaluation, Learning, and Adapting (Transform: MELA) Activity." Mid-term Performance Evaluation Report – Transform: PHC Activity Page i Table of Contents Table of Contents............................................................................................................................. i List of Tables .................................................................................................................................iii List of Figures................................................................................................................................ iv Acronyms........................................................................................................................................ v EXECUTIVE SUMMARY ........................................................................................................ 1 1 INTRODUCTION ............................................................................................................. 17 1.1 BACKGROUND OF USAID TRANSFORM PRIMARY HEALTH CARE (PHC) ACTIVITY................................................................................................................................ 17 1.2 PURPOSE AND USE OF THE MID-TERM PERFORMANCE EVALUATION ...... 19 1.3 EVALUATION QUESTIONS....................................................................................... 19 2 EVALUATION DESIGN AND METHODOLOGY ........................................................ 20 2.1 EVALUATION DESIGN .............................................................................................. 20 2.2 EVALUATION METHODOLOGY.............................................................................. 20 2.3 TRAINING OF SURVEY TEAM................................................................................. 24 2.4 DATA COLLECTION................................................................................................... 25 2.5 DATA ANALYSIS........................................................................................................ 25 2.6 ETHICAL CONSIDERATIONS ................................................................................... 27 2.7 LIMITATIONS OF THE EVALUATION .................................................................... 27 3 KEY FINDINGS................................................................................................................ 29 3.1 RELEVANCE................................................................................................................ 29 3.2 EFFICIENCY................................................................................................................. 32 3.3 EFFECTIVENESS AND PROGRESS TO ACHIEVING MNCH/FP HEALTH OUTCOMES............................................................................................................................. 48 3.3.1 FAMILY PLANNING (FP).................................................................................... 50 3.3.2 MATERNAL HEALTH ......................................................................................... 57 3.3.3 NEWBORN HEALTH ........................................................................................... 64 3.3.4 CHILD HEALTH ................................................................................................... 67 3.3.5 SOCIAL BEHAVIOR CHANGE AND COMMUNICATION (SBCC) ............... 72 3.3.6 CROSS-CUTTING................................................................................................. 74 3.3.7 COMPARISON OF MID-TERM TRANSFORM: PHC ACTIVITY TARGETS VS. MID-TERM PERFORMANCE..................................................................................... 79 3.3.8 COMPARISON OF USAID/ETHIOPIA TRANSFORM: PHC-ONLY AND TRANSFORM: PHC-WASH OVERLAP AREAS AT MID-TERM.................................. 83 Mid-term Performance Evaluation Report – Transform: PHC Activity Page ii 3.3.9 SUSTAINABILITY ELEMENTS.......................................................................... 83 3.3.10 TRANSFORM: PHC’S PROMISING APPROACHES AND STRATEGIES NOTED BY STAKEHOLDERS .......................................................................................... 85 3.3.11 ANALYSIS OF THE COMBINED EFFECTS BETWEEN USAID TRANSFORM: PHC AND GROWTH THROUGH NUTRITION (GTN) ACTIVITY..... 87 3.3.12 ANALYSIS OF HEALTH FACILITIES READINESS AND PRACTICE IN PREVENTION AND MANAGEMENT OF POST-PARTUM HEMORRHAGE (PPH)... 88 4 KEY IMPLEMENTATION CHALLENGES ................................................................... 90 5 KEY LESSONS LEARNED DURING THE IMPLEMENTATION OF THE TRANSFORM: PHC ACTIVITY ............................................................................................ 90 6 CONCLUSIONS................................................................................................................ 92 7 KEY RECOMMENDATIONS FOR STAKEHOLDERS: ............................................... 94 ANNEXES.................................................................................................................................... 94 Annex 1: USAID Transform: PHC Activity: Performance by Result Area at Mid-term (2019) Compared with the Baseline (2017) in Transform: PHC Intervention and Non-intervention Areas......................................................................................................................................... 97 Annex 2: Transform: PHC Three-Year (3) Resource Allocation/Expenditure (USD)........... 101 Annex 3: Adjusted Odds Ratios and 95% CI from the Logistic Regression Model for the Determinants of FP Use Among Currently Married Women, Transform Mid-term Survey (2019)...................................................................................................................................... 102 Annex 4: Adjusted Odds Ratios and 95% CI from the Logistic Regression Model for the Determinants of Maternal Health Services Utilization, Transform Mid-term Survey (2019) 103 Annex 5: Adjusted Odds Ratios (OR) and 95% CI from the Logistic Regression Model for the Determinants of Child Health Services Utilization, Transform Mid-term Survey (2019), PHC Intervention Woredas.............................................................................................................. 105 Annex 6: Institutional Deliveries Before and After the Installation of a Solar Suitcase (Source Transform: PHC Annual Report)............................................................................................ 106 Annex 7: Statement of Work (SOW) and Evaluations Protocol for Mid-Term Performance Evaluations of Transform Implementing Mechanisms........................................................... 107 Annex 8: Misterm Evaluation tools........................................................................................ 107 Annex 9: List of Relevant Documents Reviewed................................................................... 107 Annex 10: List of Key Informants.......................................................................................... 107 Mid-term Performance Evaluation Report – Transform: PHC Activity Page iii List of Tables Table 1. Key Recommendations for USAID, Transform: PHC and GOE/FMOH, RHBs........... 11 Table 2: USAID/Ethiopia Transform: PHC Activity: Summary of Selected Performance Results Areas at Mid-term (2019) compared with baseline (2017) in Transform: PHC Intervention and Non-intervention Areas................................................................................................................. 15 Table 3. Number of Households Surveyed ................................................................................... 22 Table 4: Health Facilities Assessed in Intervention and Non-Intervention Areas........................ 23 Table 5: Distribution of Transform: PHC Team........................................................................... 33 Table 7: Woredas’ Performance Status Shift in the Transform: PHC Targeted Regions (2017 vs. 2019............................................................................................................................................... 36 Table 8. Respondent Demographics Characteristics .................................................................... 49 Table 9: Family Planning Methods Offered by Type and Health Facilities (%).......................... 52 Table 10: Proportion of Women Who Could not Get Family Planning Service When They Want to Use, by Region and Reason for not Receiving the Service ...................................................... 52 Table 11. The Difference in Proportions and DID of Family Planning Indicators, Disaggregated by Transform: PHC Intervention and Non-intervention Areas (Baseline, 2017 and Mid-term 2019), Ethiopia......................................................................................................................................... 55 Table 12. The Difference in Proportions and DID of Maternal Health Indicators, Disaggregated by Transform: PHC Intervention and non-Transform Areas (Baseline, 2017 and Mid-term 2019), Ethiopia......................................................................................................................................... 62 Table 13. Difference in Proportions and DID of Newborn Health Indicators, Disaggregated by Transform: PHC Intervention & Non-Transform: PHC Areas (Baseline, 2017 & Mid-term 2019). ....................................................................................................................................................... 67 Table 14. The Difference in Proportions & DID of Child Health Indicators, Disaggregated by PHC Intervention and Non-Transform Areas (Baseline, 2017 and Mid-term, 2019), Ethiopia............ 71 Table 15. Transform: PHC Activity Performance on Set Targets at Mid-term............................ 81 Table 16: Combined Effects between USAID Transform: PHC Activity & Growth through Nutrition (GtN) ............................................................................................................................. 88 Table 17: Health Facilities Readiness, Practice in Prevention & Management of PPH Cases .... 89 Mid-term Performance Evaluation Report – Transform: PHC Activity Page iv List of Figures Figure 1:Transform: PHC’s Development Hypothesis & Results Framework (RF).Error! Bookmark not defined. Figure 2: Difference in Difference Illustration .............................................................................. 26 Figure 3: Overview of Woredas Performance Status and Shifts in Transform PHC Intervention Areas (between Yr. 1, 2017 & Yr. 3, 2019).................................................................................... 34 Figure 4: Family Planning Indicators Performance in Transform: PHC Intervention and Non￾intervention Areas (Baseline vs. Mid-term).................................................................................. 51 Figure 5: Key Indicators Performance on Family Planning by Region .......................................... 54 Figure 6: Key Indicators Performance on Family Planning by Region .......................................... 54 Figure 7: The Proportion of Performance of Key Maternal Health Indicators (Baseline vs. Mid￾term) and Comparison between Intervention and Non-intervention Areas ............................... 59 Figure 8: Proportion of Performance of Key Maternal Health Indicators (Baseline vs. Mid-term) and Comparison between Intervention and Non-intervention Areas ......................................... 59 Figure 9: Regional Performance of Key Maternal Health Indicators at Mid-term ....................... 61 Figure 10: Regional Performance of Key Maternal Health Indicators at Mid-term ..................... 61 Figure 11: Performance of Newborn Health Indicators in Transform PHC Intervention Areas (between Baseline, 2017 and Mid-term, 2019)............................................................................ 65 Figure 12: Regional Performance of Key Newborn Health Indicators at Mid-term..................... 66 Figure 13: Child Health Indicator Performance of Transform: PHC Intervention and Non￾intervention Areas (Baseline, 2017 vs. Mid-term, 2019............................................................... 68 Figure 14: Child Health Indicator Performance of Transform: PHC Intervention and Non￾intervention Areas (Baseline, 2017 vs. Mid-term, 2019).............................................................. 69 Figure 15: Regional Child Health Indicator Performance – Transform: PHC Intervention Areas (Midterm, 2019)............................................................................................................................ 70 Figure 16: Regional Child Health Indicator Performance – Transform: PHC Intervention Areas (Baseline vs. Mid-term)................................................................................................................. 70 Figure 17: Sanitation and Hygiene Indicator Performance in USAID Transform: PHC Intervention Area (Baseline vs. Mid-term) ........................................................................................................ 75 Figure 18: Regional Sanitation and Hygiene Indicator Performance in USAID Transform: PHC Intervention Areas (Baseline vs. Mid-term).................................................................................. 75 Figure 19: Household CBHI Enrollment in Transform: PHC Intervention Areas & Non￾Intervention Areas........................................................................................................................ 76 Figure 20; Household CBHI by Region (Baseline, 2017 & Mid-term, 2019) ................................. 76 Figure 21: Household CBHI Membership Renewal in Transform: PHC Intervention Areas and Non-Intervention Areas; and by Region (Mid-term, 2019) .......................................................... 77 Figure 22: Women’s Participation in Decision-Making and Male Involvement........................... 78 Figure 23: Regional Women’s Decision-Making and Male Involvement in USAID Transform: PHC Intervention Areas (Baseline vs. Mid-term).................................................................................. 79 Figure 24: Comparison of USAID Transform: PHC-only and Transform: PHC and WASH Overlap Areas at Mid-term......................................................................................................................... 83 Mid-term Performance Evaluation Report – Transform: PHC Activity Page v Acronyms ANC Antenatal Care ARI Acute Respiratory Infection BEmONC Basic Emergency Obstetric and Newborn Care CBHI Community-based Health Insurance CBNC CI Community-based Neonatal Care Confidence Interval CPR Contraceptive Prevalence Rate CRC Caring, Respectful and Compassionate CSA Central Statistical Agency CSPro Census and Survey Processing System DHS Demographic and Health Survey EA Enumeration Areas EDHS Ethiopia Demographic and Health Survey FGD Focus Group Discussions FMOH Federal Ministry of Health FP Family Planning GBV Gender-Based Violence GOE Government of Ethiopia GtN Growth through Nutrition HC Health Center HDR HEP Health in Developing Regions Health Extension Program HF Health Facility HH Household HP Health Post ICCM Integrated Community Case Management IFHP Integrated Family Health Program IMNCI Integrated Management of Newborn and Childhood Illness IP Implementing Partner ITN Insecticide Treated Net KII Key Informant Interview LAFP LOA LOP MCH/FP Long-Acting Family Planning Life of Activity Life of Project Maternal and Child Health and Family Planning MCPR Modern Contraceptive Prevalence Rate MELA Monitoring, Evaluation, Learning, and Adapting MNH Maternal and Newborn Health ORS Oral Rehydration Solution PHC Primary Health Care PMCD Preventable Maternal and Child Death PNC Postnatal Care PPFP Post-partum Family Planning PRIN PTI PRIN International Research and Training Consultancy PLC Primary Health Care Transformation Initiative RH/FP Reproductive Health / Family Planning Mid-term Performance Evaluation Report – Transform: PHC Activity Page vi SBA Skilled Birth Attendant SNNP Southern Nations, Nationalities and Peoples State TMG The Mitchell Group, Inc. UNFPA USAID United Nations Population Fund United States Agency for International Development USG United States Government WASH Water, Sanitation, and Hygiene WHO World Health Organization ZHD Zonal Health Departments Mid-term Performance Evaluation Report – Transform: PHC Activity Page 1 EXECUTIVE SUMMARY Background and Evaluation Purpose: USAID/Ethiopia contracted The Mitchell Group, Inc. (TMG), working under the umbrella of the USAID/Ethiopia-funded Transform Monitoring, Evaluation, Learning and Adapting (Transform: MELA) Activity to conduct a mid-term performance evaluation of the Transform Primary Health Care (PHC) Activity. USAID/Transform: PHC is a five-year, US$120 million Activity operating in four regions of Ethiopia: 1) Amhara, 2) Oromia, 3) SNNP, and 4) Tigray. The purpose of the Transform: Primary Health Care (Transform: PHC) mid-term performance evaluation was to assess the continued relevance of the Activity, and the efficiency, effectiveness, and sustainability of results. The evaluation flagged mid-course corrections needed to improve the effectiveness of the Activity for the remaining 18 months of the Activity. It will also inform USAID/Ethiopia and the Federal Ministry of Health (FMOH) concerning future Preventable Maternal and Child Death (PMCD) programming. The evaluation was conducted between November 2019 and February 2020. Evaluation Design/Statistical Methods: The Transform: PHC mid-term performance evaluation utilized a similar design to that of the baseline survey conducted in 2017 to allow for a comparison of results. The evaluation included a household/population survey of 3,625 women aged 15-49, supplemented by health facility data collection from 256 facilities, and qualitative data collection, which was comprised of 42 key informant interviews (KIIs). The key stakeholders interviewed included the FMOH, RHB, and woreda health officials. Twelve focus group discussions (FGDs) were conducted with communities and key stakeholders. The evaluation report uses descriptive statistics, univariate, bivariate, and multivariate analysis, and Difference-in-Differences (DID) calculations to present the results. The results are disaggregated by region, and by Transform: PHC Activity intervention and non-Transform intervention areas. This evaluation also explored determinants for the key outcomes (Family Planning, Maternal Health, and Child Health). The analytical approach adopted for this mid-term evaluation of the Transform: PHC Activity effectiveness involves a comparison of changes across key indicators between the Transform Program Baseline survey (2017) and mid-term (2019) in Transform: PHC intervention areas versus non-Transform areas. The analysis also included documenting changes in key outcomes of interest to USAID and stakeholders within Transform: PHC intervention groups and comparing results at the midpoint to those of the baseline. Household-level surveys, focus group panels (FGPs) and other interviews and a review of documents from USAID/Ethiopia, Transform: PHC, Mid-term Performance Evaluation Report – Transform: PHC Activity Page 2 the Government of Ethiopia and stakeholders (including the Health Sector Transformation Plan (HSTP), Mini-DHS) provided both qualitative and additional quantitative evidence that shaped findings and the team’s recommendations. Limitations of the Evaluation: • The methodology employed for the mid-term performance evaluation, while appropriate and statistically rigorous, nevertheless had some limitations. The absence of the Enumeration Area (EA) list from the Ethiopian Central Statistical Agency (CSA), which is in charge of the national census and establishment of EAs, necessitated the use of the list of kebeles for each intervention area as a sampling frame for the first stage of sampling. Transform: MELA utilized kebele guides to clarify and identify boundaries of the selected kebeles and to ensure the complete listing of households in selected gotts. Sampling with replacement was employed, especially where security concerns and inaccessibility impeded data collection in some enumeration areas. • Another limitation was isolating confounding factors while using Difference-in￾Differences (DID) analysis in the context of development programs taking place in both USAID/Ethiopia Transform: PHC Activity areas and the surrounding areas outside the boundaries of the evaluated regions. During the survey, partner mapping information was gathered from both Transform: PHC intervention sites and non-USAID Transform sites. The evaluation team found that a myriad of other partner-funded programs exists in both Transform: PHC intervention sites and non-USAID Transform sites. Thus, it is difficult to definitively discern the results and impact of other woreda-level, FMOH, or donor￾financed programs on achieving MNCH/FP targets. The evaluation team relied on the difference-in-differences and causality tracking strategies analysis to overcome these challenges. Thus, the DID results should be interpreted, bearing in mind these considerations. There will also be a spillover effect since TPHC is fully supporting the RHBs and Zonal Health Departments (ZHDs) on which these institutions are supporting all areas. • Given that this evaluation was designed as a mid-term performance evaluation, it was only possible to assess progress towards intended results as opposed to measuring the full achievement of those expected results. It is essential to take note of this when interpreting results from this evaluation. An additional limitation was the evaluation team’s difficulty in obtaining budget data on MNCH/FP programs operating in USAID Transform: PHC supported sites and non-intervention sites, hampering the team’s ability to carry out a comparative assessment of efficiency based on financial investments. The purpose of highlighting those limitations is to guide the readers. Mid-term Performance Evaluation Report – Transform: PHC Activity Page 3 Summary of Evaluation Findings: The conclusion from the evaluation findings is that overall, the USAID/Ethiopia Transform: PHC Activity is positively contributing to the achievement of most of the Activity result areas. Of note, the evaluators found: ● An increase in performance in thematic areas in Transform: PHC supported areas between the baseline and mid-term, with Transform: PHC significantly contributing to key health outcomes related to Family Planning (FP), Maternal Health, Newborn Health, Child Health, Hygiene and Sanitation and Community Based Health Insurance (CBHI). Table 2 below summarizes the major findings by key health indicators and shows that Transform: PHC-supported areas registered better improvements compared to non-Transform areas. ● Evidence from this evaluation also documents that FP, Maternal Health, and Child Health outcomes have common determinants. Other factors, including household income/wealth, education status, and living conditions (rural/urban), are among the common determinants for MNCH/FP1 . These include literacy levels, religious beliefs, access to health messages, and CBHI enrollment, which has an element of empowerment. Political commitment is the other factor2 responsible for the demand and utilization of MNCH/FP. Relevance3 : The design of the Transform: PHC Activity was found to be strong and valid. Government stakeholders highly value the Activity. As a result of Transform: PHC, USAID/Ethiopia is perceived as a key strategic partner in advancing the HSTP. A key strength of the program is that, from the outset, the design began with the co-creation process that allowed for up-front involvement of government and key stakeholders. The Activity is well-aligned with the government’s HSTP, and the interventions are consistent with the FMOH’s and USAID/Ethiopia’s strategic priorities for MNCH/FP. The use of approaches and strategies, such as the Theory of Change (ToC) workshops, ensures that Transform: PHC interventions continue to be contextually appropriate and flexibly refined on an annual basis as needs change. Over 80% of the KIIs at the regional and woreda level shared the view that Transform: PHC interventions support the achievement of the health sector transformation plan priorities, which was summed up by a KII respondent from the Huletejunese Woreda Health Office in Amhara region. “USAID/Ethiopia Transform: PHC generally focuses on assisting the implementation of the health sector transformation plan (HSTP). It is mainly concerned with the issues of quality and equity of service, particularly the five areas of quality, which include maternal and child health. Transform: PHC covers all the KPIs that are related to mothers and children. But it is not involved in areas such as TB and HIV. In more 1 Ethiopia Mini-DHS, 2019 2 Transform: PHC Senior Management KIs 3 Relevance: The extent to which the objectives of a development intervention are consistent with beneficiaries’ requirements, country needs, global priorities and partners’ and donors’ policies. Mid-term Performance Evaluation Report – Transform: PHC Activity Page 4 detail, Transform: PHC is involved in mothers’ and children-related issues such as nutrition, Antenatal care (ANC) skilled deliveries, Maternal health, family planning, and in monitoring and evaluation4 .” About 70% of Transform: PHC beneficiaries interviewed during the KIIs and focus groups praised the Activity and were more likely to seek services at Activity supported health facilities. Recipients also noted that they were highly satisfied with the customer service at the Transform: PHC supported sites. Women in Transform: PHC regions were more likely to seek care for their infants and newborns and use a skilled birth attendant for deliveries. Efficiency5 : Most of the Transform: PHC management systems, including staffing, on-site GOE-led staff training and mentoring, joint deployment, and co-location of expert long- and short-term technical assistance alongside government health teams, and logistics are efficient and support the achievement of Activity results. The Theory of Change (ToC) exercise has helped in coordinating the participation of stakeholders for improved results. Another vital approach introduced is the use of evidence from “random follow-ups” by technical teams to support Activity programming decision-making and support services provided. These visits reinforced in-service learning on all of the interventions and provided an invaluable opportunity for staff and supervisors to gauge health facility requirements. Activity focal points at government structures were found to be working, with the GOE helping to facilitate communication, coordination, and Activity implementation. The joint supervision the Activity’s staff members are conducting with the government counterparts contributed to skills and knowledge transfer to government staffs so that they can support other health facilities in the same way (one way the project extended its hands to the ground); this was another efficient approach supported by Transform: PHC. The evaluation found that, overall, Transform: PHC has fostered strong collaboration with the GOE at all levels. The woreda grants were found to be catalyzing woredas to prioritize health service improvement plans and needs. Areas for improvement, in terms of efficiency, include the need to better collaborate with other development actors who are financing complementary interventions in the same regions. Another area for improvement was that some found the grant-making process to be overly complex (as it requires procedural approval from the donor) and led to some delays in implementing woredas' service improvement priorities. 4 KII respondent from HuletejuHuletejuHuletejuHuletejuHuletejuHuletejuHuletejuneseHuleteju Woreda Health Office in Amhara region. 5 Efficiency: A measure of how economically resources/inputs (funds, expertise, time, etc.) are converted to results. Mid-term Performance Evaluation Report – Transform: PHC Activity Page 5 Effectiveness6 : Several MNCH/FP indicators demonstrated positive performance improvement between the baseline (2017) and the mid-term (2019). The performance was compared between baseline and mid-term, and non-Transform: PHC intervention sites include project set targets (LOP). The findings from the evaluation have shown the factors responsible for the observed performance in Transform: PHC intervention sites and non-Transform: PHC sites in Family Planning, Maternal Health, and Child Health seem to be similar, with limited varying factors. Also, with the spillover effect within woredas in the same regions, the evaluation found determinants tending to be the same between Transform: PHC supported sites and non-Transform: PHC sites. Thus, the observed differences in trends could be explained by partners and the intensities in their interventions/programs as the plausible determinants. See Table 2 for details. Also, the following is a summary of the performance for key thematic areas: ▪ Family Planning: There was an improvement in the performance of the Modern Contraceptive Prevalence Rate (MCPR) between the baseline and mid-term in Transform: PHC intervention areas (44.3% at baseline and 45.8% at mid-term). However, the change was not statistically significant. The prevalence of Long-Acting Family Planning (LAFP) and Post-partum Family Planning (PPFP) users increased, from 12.5% at baseline to 15.4% at mid-term and from 34.3% at baseline to 40.3% at mid-term, respectively. There is a growing acceptance of post-partum family planning, particularly in Southern Nations, Nationalities, and Peoples State (SNNP). However, Transform: PHC is expanding health facilities providing LAFP, PPFP and FP integrations with other service units such as HIV, Youth Friendly Service, Delivery, and CHD, which are promising practices to improve MCPR in the long term and as an improvement in the quality of care for FP services. Key informants from regional offices believe these observed improvements in family planning are associated with the active engagement of health extension workers, increased knowledge and awareness of FP methods, and increased availability of long￾acting family planning methods at health facilities7 . Cultural and traditional restrictions, fear of side effects, and inadequate health service provision are some of the factors responsible that might be hindering the region’s MCPR performance8 . Women registered 6 Effectiveness: The extent to which the development intervention’s objectives were achieved, or are expected to be achieved, taking into account their relative importance. 7 Key Informants from Amhara, Oromia and SNNP RHBs. 8 KII Tigray RHB Mid-term Performance Evaluation Report – Transform: PHC Activity Page 6 in CBHI were found to be more likely to access FP and maternal health services than those who had yet to register with CBHI9 . ▪ Maternal Health: There was a slight improvement in the use of Skilled Birth Attendants (SBAs) between the baseline and mid-term in Transform: PHC intervention areas by two percentage points (from 66.4% to 68.4%), but the increase is not statistically significant. Early Postnatal Care within two days showed a significant increase from the baseline by 5.7 percentage points. The proportion of women who took iron and folic acid for at least three months during their last pregnancy significantly increased from 20.9% to 26.6%. The proportion of pregnant women who slept under an insecticide-treated bed net (ITN) the previous night also increased from 37.9% at baseline to 39.2% at mid-term. Early initiation of antenatal care (less than 16 weeks) has significantly increased by 24.5% between baseline (30.8%) and mid-term (55.3%). However, the proportion of women who received essential components of ANC declined slightly, from 39.4% at baseline to 38.2% at mid-term, but the decline was not statistically significant. The proportion of women who received four or more ANC visits showed a significant decline, from 57.3% at baseline to 52.0% at mid-term (5.3 percentage points). The KIIs from RHBs and woredas in the four regions attributed the declines to include women's lack of awareness of the importance of attending at least 4 ANC visits, distance to health facilities, and limited availability of affordable transportation. Non-Transform: PHC intervention areas showed similar trends in maternal health indicators. The reason there appears to have been the minimal difference between the Transform: PHC and the non-Transform: PHC sites on ANC visits is that there are substantial ANC services available in the non-Transform: PHC sites such as those financed by the United Nations Population Fund (UNFPA) and Relief Society of Tigray, etc. The Transform: PHC interventions contributed to a significant increase in early initiation of ANC visits by 11.9% and an increase in the uptake of essential ANC services by 10.9%. The DID analysis shows that Transform: PHC interventions contributed to an increase in Skilled Birth Attendance by 15.3%. “We understand Transform: PHC sites could be performing better than non-Transform: PHC supported sites. In non-Transform: PHC sites, and there is a need to actively engage health extension workers (HEWs), expand new health service facilities, and increase emphasis on 9https://www.researchgate.net/publication/332999123_Effects_of_community_based_health_insurance_on_modern_family_ planning_utilization_in_Ethiopia Mid-term Performance Evaluation Report – Transform: PHC Activity Page 7 maternal health. All of these factors believed to have contributed to improving the coverage of maternal care in woredas and regions10.” These findings need to be viewed in the context of the dynamic and changing political and social context that some of these regions face. The other common reason hindering pregnant mothers from attending ANC includes women's lack of awareness of its importance, distance to health facilities, and limited availability of affordable transportation. On the other hand, essential determinants that are facilitating ANC attendance include higher education level, woman's ability to make health care decisions, partner involvement, and MCH messaging11 12 . ▪ Newborn Health: Early postnatal care for the newborn (within two days of birth) significantly increased from 40.8% at the baseline to 47.9% at the mid-term (an increase of 7.1 percentage points). Early initiation of breastfeeding (within 1 hour of birth) significantly increased from 77.6% to 88.1% (10.5 percentage points). The proportion of newborns who received essential newborn care has also significantly increased from 6.3% at the baseline to 15.4% at mid-term, with a 9.1 percentage points increase. ▪ Child Health: The prevalence of exclusive breastfeeding increased significantly between the baseline (59.4%) and mid-term (76.8%). The incidence of children suffering from acute respiratory infections (ARI) declined significantly by 7.9% between the baseline and the mid-term. Furthermore, the performance in the treatment of children for ARI with antibiotics significantly improved from 36.2 % at baseline to 52.3% at mid-term. The incidence of diarrhea significantly increased, while the proportion of children who received treatment for diarrhea with oral rehydration solution (ORS) and Zinc has shown some modest improvement, from 28.4% at baseline to 31.4% at mid-term. However, the increase was not statistically significant. ▪ Cross-cutting Issues: In the Transform: PHC interventions areas, access to basic sanitation facilities has shown some improvements. The percentage of households that have access to basic sanitation facilities has significantly increased in Transform: PHC intervention areas from 5.9% at baseline to 8.3% at mid-term (2.4 percentage points). Households that have handwashing facilities with soap and water increased significantly in Transform: PHC intervention areas from 0.8 % at baseline to 4.2% (by 3.4 percentage points). Households 10 A key informant from the Oromia RHB 11 Fekadu Y, Mesfin A, Haile D, and Stoecker B. Factors associated with nutritional status of infants and young children in Somali Region, Ethiopia: a cross- sectional study, BMC Public Health (2015) 15:846 12 Fekadu H, Tesfahun A and Mequanent M. Late initiation of antenatal care and associated factors among pregnant women in Addis Zemen primary hospital, South Gondar, Ethiopia. Reproductive Health (2019) 16:73. https://doi.org/10.1186/s12978-019- 0745-2 Mid-term Performance Evaluation Report – Transform: PHC Activity Page 8 using appropriate water treatment technology have increased in Transform: PHC intervention areas from 10.4% at baseline to 11.7% at mid-term, but the change is not statistically significant. The Tigray region reported the most considerable change and a related decrease in the incidence of diarrhea. Of note, households enrolled in CBHI increased significantly, and there is a marked difference in enrollment between Transform: PHC intervention areas (from 29.8% at baseline to 50.5% at mid-term) and non-Transform: PHC areas (from 30.2% at baseline to 47.6% at mid-term). However, since other actors are also supporting CBHI-related activities, there was no analysis conducted to attribute this increase to the Transform: PHC Activity. ▪ Gender: The proportion of women accompanied by spouses during the delivery of their last child shows a slight decrease, from 84.9% at baseline to 84.6% at the mid-term. But the team also found that there was a significant decline in the percentage of women whose partners accompanied them to at least one ANC visit in Transform: PHC intervention areas. Women's participation in decisions regarding their health care remains a key challenge as there was no apparent change between the two periods. Sustainability13: The Transform: PHC Activity has integrated sustainability elements into its programming, such as introducing innovative approaches to support strengthening the local capacity of the health system, some of which have been adopted by the government. For example: • Some local governments (e.g., the Oromia special zone of Amhara Region) have adapted twinning and allocating resources for peer-to-peer learning within their zones. High￾performing woredas have been twinned with low-performing ones to allow the low￾performing woreda to learn from the high-performing counterpart. • The government officials credited Transform: PHC Activity in positively supporting government priorities for health systems strengthening. • Transform: PHC implementation modalities have proved integral, and part of the government health sector (no parallel implementation structure) reported to have gone as far as strengthening and improving the functionality of government systems. • Coaching and mentorship support has also helped in establishing a cadre of mentorship experts who are working within government structures to improve the capacity of health facility and public health staff. 13 Sustainability The continuation of benefits from a development intervention after major development assistance has been completed. The probability of continued long-term benefits. The resilience to risk of the net benefit flows over time. Mid-term Performance Evaluation Report – Transform: PHC Activity Page 9 • Other interventions mentioned to enhance sustainability include LAFP back-up support to health posts, woreda-level planning exercises, on-site training arrangements, clinical skill labs, creation of local pull of trainers, and training capacity14 . Although Transform: PHC has incorporated sustainability elements into its programming, sustainability remains a challenge since GOE/FMOH and regions have limited budgetary and resource mobilization capacity to become completely financially self-reliant. Promising Approaches and Strategies Noted by Transform: PHC Stakeholders: ● Twinning among high- and low-performing woredas and facilities. ● Expanded Quality Improvement collaboration beyond MNH to include FP, Adolescent Youth Health and Development, and Child Health and Development. ● Scale-up of the “Her Space initiative”- empowering girls in an integrated manner beyond Reproductive Health. ● Innovative clinical skill labs held at the health center level to strengthen peer-to-peer education and promote practical hands-on experience for practitioners and interns. ● Random follow-ups provide real-time data to inform technical support and a useful internal monitoring system. ● Annual Theories of Change exercise based on evidence generated and consultations with key stakeholders including FMOH, RHB, woreda health offices, universities, and other private sector providers has enabled the Transform: PHC and GOE partners to identify regional priorities for the following year, including common challenges across woredas. ● Woreda grants have provided additional resources to finance woreda health sector priorities and needs. Stakeholders appreciated the mechanism at different levels since it fills woreda budget gaps. Furthermore, KII respondents also acknowledged the experience of grants management, including the oversight of these grants, which has provided valuable expertise for woredas to implement future performance-based financing that may be financed by the GOE. The FMOH has replicated many aspects of the woreda subgrant management in other non-Transform: PHC woredas using government financing. ● Coaching and mentorship support using the public system structures: Woreda level interventions, either at woreda health offices or at primary hospitals, are essential to sustaining gains on service delivery and program management. Catchment based clinical mentorship, leadership development coaches, public finance management mentorship, connected woreda mentorship, and quality improvement mentorship are all about establishing built-in capacity at the woreda level. So far, the expansion of these initiatives, through public sector resources as well as other partners, is going well, which signals that 14 Transform: PHC technical staff KI Mid-term Performance Evaluation Report – Transform: PHC Activity Page 10 the strategy is useful and a candidate for scale-up. ● The provision of solar suitcases by Transform: PHC Activity is solving power problems at health centers and increasing the delivery of services. Regional respondents said this technology was found to be effective. ● Leadership development (building data-savvy leaders): Leaders at primary healthcare facilities using data properly leads to appropriate decision-making and enhances the process of addressing problems. The connected woreda strategy, supported by the Activity, regularly measures the use of quality data at the primary level of healthcare and is led by the woreda and facility leadership. Conclusions The Transform: PHC Activity has achieved impressive health results and has successfully built a management and assistance platform that is responsive to the GOE/FMOH’s health sector transformation agenda. Thus, Transform: PHC is on track to meet most of its intended targets by the end date of the Activity. Transform: PHC is positively regarded across the intervention regions by RHBs and woredas and is providing tangible and meaningful assistance to the GOE/FMOH to achieve its desired goal to end preventable maternal, newborn, and child deaths. Most importantly, beneficiaries note that the Activity has led to improved access and quality of care. The evaluation found that some targets could have been set high. Target setting may need to consider factors like past experiences in achieving similar targets, expert opinions, resources available, contextual factors, etc. Given that this evaluation was designed as a mid-term performance evaluation, it focused on assessing “progress towards” or the “likelihood of achieving intended results” with regards to the effectiveness question, as opposed to “measuring the full achievement of intended results and attribution.” The analysis also assessed Transform: PHC contribution to different thematic area results. Based on this premise, the evaluation concluded that the Activity is on track to achieve the intended results. It is important to note this when interpreting results from this mid￾term performance evaluation. Mid-term Performance Evaluation Report – Transform: PHC Activity Page 11 Key Recommendations for Stakeholders Table 1. Key Recommendations for USAID, Transform: PHC and GOE/FMOH, RHBs Recommendation/Action Actor to Address Timing Priority # USAID/Ethiopia 1 USAID/Ethiopia and the GOE should revisit some of Transform: PHC’s targets to be achieved over the life of the Activity. Some targets appear to have been set high, and the target setting should consider factors like past experiences in achieving similar targets, exert opinions, resources available, contextual factors, etc. Reporting should then be linked to result areas and revised performance targets. USAID/ Ethiopia Immediate 2 USAID/Ethiopia should consider realigning the budget, if feasible, to provide additional resources for interventions that are under-performing (e.g., Maternal Health, ANC4+, SBA), and scaling up others that were found to be successful. USAID/ Ethiopia Immediate 3 USAID/Ethiopia should ensure that context monitoring and reporting is mandatory for Transform: PHC. The success of any initiative requires gaining a sound understanding of the two-way interaction between Activity and context, taking quick and effective actions to minimize negative impacts and maximize the positive effects of interventions. A review of Transform: PHC performance reports revealed that, although the context is being monitored, there was no systematic reporting or a section dedicated to context monitoring, mitigation, and reporting in the Activity performance reports. In the Ethiopian context, there is a need to deliberately monitor and document context monitoring and mitigation concerning its effect on programming and how programming is shaping it. USAID/ Ethiopia Immediate 4 For future programming, USAID/Ethiopia may want to consider competitively selecting woredas that have multisectoral investments that support the achievement of MNCH/FP objectives. Future PHC activities should be designed in the same regions, ensuring that the woredas have a robust transformation agenda that will complement and reinforce USAID/Ethiopia’s health interventions. Consideration might be given to competitively selecting the USAID/ Ethiopia Future Design/ Programming (Beyond the life of Transform: PHC) Mid-term Performance Evaluation Report – Transform: PHC Activity Page 12 Recommendation/Action Actor to Address Timing Priority next round of woredas for participation in such programming. 5 Continue using co-creation in designing new projects. This approach should also include bottom-up co-creation with the regions and woredas. USAID/ Ethiopia Future Design/ Programming (Beyond the life of Transform: PHC) For Transform: PHC Activity Implementer(s) 6 Consider the approach to technical assistance and the gaps in thematic areas where there are low or, as yet, no discernible performance improvements in some of the maternal health indicators such as 4+ANC, and women who received essential components of ANC. Transform: PHC should also intervene in identified determinants that affect performances. In some thematic areas, this may necessitate revisiting strategies and approaches to be more productive. The approaches considered should also be geared towards narrowing regional gaps in key outcomes. Transform: PHC Immediate 7 Provide a comprehensive package for systems strengthening support to a selected number of woredas for better efficiency and to improve the quality of services being provided, hence, scaling up the achievement of desired outcomes. Transform: PHC Immediate 8 Assess the effectiveness of the woreda grant towards improving MNCH/FP outcomes in the supported woredas. Transform: PHC Immediate 9 Consider integrating community interventions in a few woredas, targeting women to improve empowerment and roll out learnings to scale up to other areas. Transform: PHC Immediate 10 Work with the FMOH and RHBs, including woredas, to ensure that there is a roll-out of a graduation plan for high￾performing woredas and transition them to “model” woredas. On the other hand, as there are varying regional performances, mechanisms should be devised to improve the performance of low-performing woredas to medium and high. Transform: PHC Immediate 11 To increase utilization of family planning methods and maternal health services, Transform: PHC should Transform: PHC Immediate Mid-term Performance Evaluation Report – Transform: PHC Activity Page 13 Recommendation/Action Actor to Address Timing Priority increase/intensify its support to more women to register in CBHI and interventions modeled around working more with communities, community leaders, and religious leaders and FP/Reproductive Health messaging. 12 Continue and strengthen collaboration with Transform: MELA to enhance the effectiveness of the operations research studies conducted by the Activity. This will help improve the relevance and help to maximize the use of vital information and recommendations made in those studies. Transform: PHC Immediate 13 Strengthen/forge strategic partnerships with key actors, recognizing their niches and comparative advantages; provide support to government and mobilize additional resources around outcomes. This should include a joint gap analysis of regions/woredas, agreement on interventions with the highest potential to achieve given results, including who is responsible for what intervention. Sharing of planned activities and joint outcome monitoring should be central to these partnerships. Transform: PHC Immediate 14 RHBs and Transform: PHC should ensure that the receipt of a small grant does not displace or replace GOE funding and should be considered as a criterion for Transform: PHC support in all supported regions and woredas. Transform: PHC Immediate Recommendations for GOE/FMOH, RHBs 15 Strengthen the coordination mechanisms with key stakeholders to guide and tailor actions towards improving synergistic efforts related to MNCH. GOE/FMOH, RHBs Immediate 16 RHBs and Transform: PHC should clearly define sustainability elements that should be integrated into ongoing programming to ensure that results are sustained beyond the life of the Activity. This should include a deliberate effort made now to identify and define resiliency elements to be integrated into Activity interventions and strengthened in crisis-prone regions. GOE/FMOH, RHBs Immediate 17 Continue to strengthen the health extension program as the HEWs are vital to the primary care system. This may require revisiting the Health Extension Program (HEP) management GOE/FMOH, RHBs Immediate/ Long-term Mid-term Performance Evaluation Report – Transform: PHC Activity Page 14 Recommendation/Action Actor to Address Timing Priority and motivation strategies, and ensuring the continued acceptability of the program to the communities served. Besides, integrating the HEW fully into the health system without losing its unique identity could help reduce inequalities to access and increase the utilization of essential, high-impact health interventions. 18 Improve resource mobilization and information use to support enhanced evidence-based decision-making and ensure leadership stability. GOE/FMOH, RHBs Immediate/ Long-term 19 Going forward, GOE/FMOH may want to consider using twinning as a viable approach for technical assistance across regions. GOE/FMOH, RHBs Immediate/ Long-term 20 Consider scaling-up the number of health facilities that have access to a solar power supply. This system was found to be useful in supplementing the power supply at health facilities, thus improving staff morale and the quality of care. GOE/FMOH, RHBs Immediate/ Long-term Mid-term Performance Evaluation Report – Transform: PHC Activity Page 15 Table 2: USAID/Ethiopia Transform: PHC Activity: Summary of Selected Performance Results Areas at Mid-term (2019) compared with baseline (2017) in Transform: PHC Intervention and Non-intervention Areas Thematic Areas Key Performance Indicators Transform: PHC Intervention Areas Non-Transform: PHC Areas Baseline (%) Mid-term (%) Baseline (%) Mid-term (%) Family Planning MCPR among currently married women 44.3 45.8 42.1 41.0 LAFP methods among currently married women 12.5 15.4 12.5 10.9 Unmet Need for Family Planning 31.2 31.2 28.9 34.0 Use of modern contraception after birth (PPFP) 34.5 40.3 32.6 38.5 Maternal Health Early initiation of ANC 30.8 55.3 38.0 51.8 Women who took IFA supplement for >=90 days 20.9 26.6 27.4 29.1 Women with four or more ANC for their last birth 57.3 52.0 57.0 52.3 Women who received essential components of ANC 39.4 38.2 47.6 36.0 Skilled Birth Attendance 66.4 68.4 67.4 54.4 PNC for the mother within two days of birth 49.1 54.8 44.7 42.9 Newborn Health Early PNC for the newborn within two days of birth 40.8 47.9 31.1 36.8 Newborns who received Essential Newborn Care 6.3 15.4 2.2 11.9 Early initiation of breastfeeding 77.6 88.1 64.9 85.6 Child Health (<5) Full immunization 41.3 44.4 41.6 43.5 Measles coverage 37.5 67.4 41.5 65.3 Penta 3 coverage 47.7 58.6 55.8 57.5 Children with Symptoms of ARI treated with antibiotics 36.2 52.3 32.0 63.0 Diarrhea treatment with ORS and Zinc 28.4 31.4 24.0 25.6 Children under 5 who slept under ITN 32.2 35.0 30.6 31.7 Children received Deworming in the last six months 36.6 45.5 40.0 41.9 Cross￾cutting Households that have access to basic sanitation facility 5.9 8.3 12.9 5.6 Mid-term Performance Evaluation Report – Transform: PHC Activity Page 16 HHs with handwashing facility with soap & water 0.8 4.2 1.7 2.2 Households using appropriate water treatment technology 10.4 11.7 7.4 12.9 Women's participation in decisions regarding their health 84.2 83.4 84.2 80.2 Women accompanied by their spouse during ANC 62.2 48.6 59.2 42.9 Women accompanied by their spouse during birth at a health facility 82.9 84.6 81.1 89.1 Households enrolled in CBHI scheme 28.8 50.5 30.2 47.6 Mid-term Performance Evaluation Report – Transform: PHC Activity Page 17 1 INTRODUCTION 1.1 BACKGROUND OF USAID TRANSFORM PRIMARY HEALTH CARE (PHC) ACTIVITY Activity Description The Transform: PHC Activity is a five-year (January 2017-December 2021) cooperative agreement currently implemented by a consortium of international and local organizations that includes: Pathfinder International, JSI Research and Training Institute Inc. (JSI), Abt Associates, EnCompass, the Malaria Consortium, and the Ethiopian Midwives Association. This consortium works in close collaboration with local Government and Non-Governmental partners. Transform: PHC is contributing to reductions in Maternal, Newborn, and Child mortality and morbidity through support to both the national-level and woreda-level of the Health Sector Transformation Plan (HSTP) of the Government of Ethiopia (GOE). Transform: PHC’s focus is advancing health outcomes related to maternal, newborn, and child health (MNCH); family planning (FP); and reproductive health (RH) within Ethiopia’s four major regions of Amhara; Oromia; Southern Nations, Nationalities, and Peoples’ (SNNPR); and Tigray. The Activity targets a total of 412 woredas over the five years of the Activity with a phased-approach that specified that “360 woredas are reached at the end of the second year, and 52 more by year three.” So far, Transform: PHC has supported 405 woredas. The Transform: PHC Activity’s primary focus is on achieving the following objectives and results: ● Support the attainment of the following four HSTP transformational agenda objectives: (1) woreda transformation; (2) caring, respectful, and compassionate (CRC) providers; (3) quality and equity in health care; and (4) advancing the information revolution. Achieving these results is key to ensuring sustainable progress towards the ultimate HSTP goal of healthy, productive, and prosperous Ethiopians. Transform: PHC Activity has four intermediate objectives that appear in the results framework below: 1. Improved management and performance of health systems; 2. Increased sustainable quality of service delivery across the PHCU’s continuum of care; 3. Improved household and community health practices and health-seeking behaviors; and 4. Enhanced program learning to impact policy and programming related to PCMD. Mid-term Performance Evaluation Report – Transform: PHC Activity Page 18 Figure 7:Transform: PHC’s Development Hypothesis & Results Framework (RF) R1.1 Established and strengthened innovative processes to sustainably enhance health system management and performance R1.2 Enhanced functionality of the health system within the context of primary level care R1.3 Strengthened transformational leadership, governance, and management at the woreda and PHCU level R2.1 Strengthened skills for delivery of quality and integrated RMNCAH-N services R2.2 Improved provider behaviors and communication skills toward a compassionate, respectful, and caring health workforce R2.3 Improved management of health service delivery and oversight of service quality R2.4 Innovative service delivery interventions to impact EPCMD introduced and scaled up R3.1 Increased individual- and household-level care￾seeking behavior and uptake of healthy practices R3.2 Strengthened enabling environment for health-seeking behavior, including community engagement in health service oversight in health service oversight R4.1 Strengthened health system capacity to generate learning and evidence R4.2 Evidence of what works in PCMD informed by results from program learning and iterative adaptation R4.3 Evidence utilized to inform programming and policy with local and global stakeholders and global stakeholders Goal: Healthy, Productive, Prosperous Ethiopians Objective: Health Sector Transformational Plan's Agenda and PCMD Achieved R1: Improved management and performance of health systems R2: Increased sustainable quality of service delivery across the PHCU’s continuum of care R3: Improved household and community health practices and health-seeking behaviors R4: Enhanced program learning to impact policy and programming related preventable child and maternal deaths (PCMD) Figure 6:Transform: PHC’s Development Hypothesis & Results Framework (RF) R1.1 Established and strengthened innovative processes to sustainably enhance health system management and performance R1.2 Enhanced functionality of the health system within the context of primary level care R1.3 Strengthened transformational leadership, governance, and management at the woreda and PHCU level R2.1 Strengthened skills for delivery of quality and integrated RMNCAH-N services R2.2 Improved provider behaviors and communication skills toward a compassionate, respectful, and caring health workforce R2.3 Improved management of health service delivery and oversight of service quality R2.4 Innovative service delivery interventions to impact EPCMD introduced and scaled up R3.1 Increased individual- and household-level care￾seeking behavior and uptake of healthy practices R3.2 Strengthened enabling environment for health-seeking behavior, including community engagement in health service oversight in health service oversight R4.1 Strengthened health system capacity to generate learning and evidence R4.2 Evidence of what works in PCMD informed by results from program learning and iterative adaptation R4.3 Evidence utilized to inform programming and policy with local and global stakeholders and global stakeholders Goal: Healthy, Productive, Prosperous Ethiopians Objective: Health Sector Transformational Plan's Agenda and PCMD Achieved R1: Improved management and performance of health systems R2: Increased sustainable quality of service delivery across the PHCU’s continuum of care R3: Improved household and community health practices and health-seeking behaviors R4: Enhanced program learning to impact policy and programming related preventable child and maternal deaths (PCMD) Figure 5:Transform: PHC’s Development Hypothesis & Results Framework (RF) R1.1 Established and strengthened innovative processes to sustainably enhance health system management and performance R1.2 Enhanced functionality of the health system within the context of primary level care R1.3 Strengthened transformational leadership, governance, and management at the woreda and PHCU level R2.1 Strengthened skills for delivery of quality and integrated RMNCAH-N services R2.2 Improved provider behaviors and communication skills toward a compassionate, respectful, and caring health workforce R2.3 Improved management of health service delivery and oversight of service quality R2.4 Innovative service delivery interventions to impact EPCMD introduced and scaled up R3.1 Increased individual- and household-level care￾seeking behavior and uptake of healthy practices R3.2 Strengthened enabling environment for health-seeking behavior, including community engagement in health service oversight in health service oversight R4.1 Strengthened health system capacity to generate learning and evidence R4.2 Evidence of what works in PCMD informed by results from program learning and iterative adaptation R4.3 Evidence utilized to inform programming and policy with local and global stakeholders and global stakeholders Goal: Healthy, Productive, Prosperous Ethiopians Objective: Health Sector Transformational Plan's Agenda and PCMD Achieved R1: Improved management and performance of health systems R2: Increased sustainable quality of service delivery across the PHCU’s continuum of care R3: Improved household and community health practices and health-seeking behaviors R4: Enhanced program learning to impact policy and programming related preventable child and maternal deaths (PCMD) Figure 4:Transform: PHC’s Development Hypothesis & Results Framework (RF) R1.1 Established and strengthened innovative processes to sustainably enhance health system management and performance R1.2 Enhanced functionality of the health system within the context of primary level care R1.3 Strengthened transformational leadership, governance, and management at the woreda and PHCU level R2.1 Strengthened skills for delivery of quality and integrated RMNCAH-N services R2.2 Improved provider behaviors and communication skills toward a compassionate, respectful, and caring health workforce R2.3 Improved management of health service delivery and oversight of service quality R2.4 Innovative service delivery interventions to impact EPCMD introduced and scaled up R3.1 Increased individual- and household-level care￾seeking behavior and uptake of healthy practices R3.2 Strengthened enabling environment for health-seeking behavior, including community engagement in health service oversight in health service oversight R4.1 Strengthened health system capacity to generate learning and evidence R4.2 Evidence of what works in PCMD informed by results from program learning and iterative adaptation R4.3 Evidence utilized to inform programming and policy with local and global stakeholders and global stakeholders Goal: Healthy, Productive, Prosperous Ethiopians Objective: Health Sector Transformational Plan's Agenda and PCMD Achieved R1: Improved management and performance of health systems R2: Increased sustainable quality of service delivery across the PHCU’s continuum of care R3: Improved household and community health practices and health-seeking behaviors R4: Enhanced program learning to impact policy and programming related preventable child and maternal deaths (PCMD) Figure 3:Transform: PHC’s Development Hypothesis & Results Framework (RF) R1.1 Established and strengthened innovative processes to sustainably enhance health system management and performance R1.2 Enhanced functionality of the health system within the context of primary level care R1.3 Strengthened transformational leadership, governance, and management at the woreda and PHCU level R2.1 Strengthened skills for delivery of quality and integrated RMNCAH-N services R2.2 Improved provider behaviors and communication skills toward a compassionate, respectful, and caring health workforce R2.3 Improved management of health service delivery and oversight of service quality R2.4 Innovative service delivery interventions to impact EPCMD introduced and scaled up R3.1 Increased individual- and household-level care￾seeking behavior and uptake of healthy practices R3.2 Strengthened enabling environment for health-seeking behavior, including community engagement in health service oversight in health service oversight R4.1 Strengthened health system capacity to generate learning and evidence R4.2 Evidence of what works in PCMD informed by results from program learning and iterative adaptation R4.3 Evidence utilized to inform programming and policy with local and global stakeholders and global stakeholders Goal: Healthy, Productive, Prosperous Ethiopians Objective: Health Sector Transformational Plan's Agenda and PCMD Achieved R1: Improved management and performance of health systems R2: Increased sustainable quality of service delivery across the PHCU’s continuum of care R3: Improved household and community health practices and health-seeking behaviors R4: Enhanced program learning to impact policy and programming related preventable child and maternal deaths (PCMD) Figure 2:Transform: PHC’s Development Hypothesis & Results Framework (RF) R1.1 Established and strengthened innovative processes to sustainably enhance health system management and performance R1.2 Enhanced functionality of the health system within the context of primary level care R1.3 Strengthened transformational leadership, governance, and management at the woreda and PHCU level R2.1 Strengthened skills for delivery of quality and integrated RMNCAH-N services R2.2 Improved provider behaviors and communication skills toward a compassionate, respectful, and caring health workforce R2.3 Improved management of health service delivery and oversight of service quality R2.4 Innovative service delivery interventions to impact EPCMD introduced and scaled up R3.1 Increased individual- and household-level care￾seeking behavior and uptake of healthy practices R3.2 Strengthened enabling environment for health-seeking behavior, including community engagement in health service oversight in health service oversight R4.1 Strengthened health system capacity to generate learning and evidence R4.2 Evidence of what works in PCMD informed by results from program learning and iterative adaptation R4.3 Evidence utilized to inform programming and policy with local and global stakeholders and global stakeholders Goal: Healthy, Productive, Prosperous Ethiopians Objective: Health Sector Transformational Plan's Agenda and PCMD Achieved R1: Improved management and performance of health systems R2: Increased sustainable quality of service delivery across the PHCU’s continuum of care R3: Improved household and community health practices and health-seeking behaviors R4: Enhanced program learning to impact policy and programming related preventable child and maternal deaths (PCMD) Figure 1:Transform: PHC’s Development Hypothesis & Results Framework (RF) Mid-term Performance Evaluation Report – Transform: PHC Activity Page 19 1.2 PURPOSE AND USE OF THE MID-TERM PERFORMANCE EVALUATION The purpose of the Transform: PHC Activity mid-term performance evaluation is to assess the continued relevance of design and planning assumptions made in the original Activity design and to identify any potential mid-course corrections needed to improve the effectiveness of the Activity in achieving stated objectives. The evaluation also provides an opportunity to glean and share lessons-learned with USAID/Ethiopia, GOE/FMOH, and other key Transform stakeholders. The specific objectives of the mid-term performance evaluation of Transform: PHC are: • Evaluate the effectiveness, progress to-date and document challenges of the Transform: PHC Activity against stated objectives and IRs; • Assess overall approaches and the ability of Transform: PHC to respond to changes in targeted areas; • Examine the sustainability of Transform: PHC’s approaches concerning systems or institutional level support and continued relevance to the communities; and • Inform future programming of Transform: PHC, including a forward-looking strategy. The primary intended audience for the mid-term evaluation report is USAID/Ethiopia, GOE/FMOH, and Transform: PHC implementers who will use these findings and recommendations to plan mid-course revisions and inform future USAID investments in the health sector in Ethiopia. 1.3 EVALUATION QUESTIONS The mid-term performance covers six overarching questions posed by USAID/Ethiopia: 1. Have the USAID/Ethiopia Transform: PHC strategies and implementation mechanisms continued to be relevant and appropriate to the targeted regions of Amhara, Oromia, SNNP, and Tigray? 2. How efficient are Transform: PHC’s approaches and strategies for achieving intended results? 3. What has been the effectiveness (“likelihood of achieving results”) of Transform: PHC’s approaches in contributing to improving MNCH/FP outcomes? 4. How has Transform: PHC facilitated local ownership and sustainability? 5. How has Transform: PHC considered gender dynamics in the Activity design and implementation? 6. What are the lessons learned and recommendations for USAID/Ethiopia related to Transform: PHC and future design/programming? Mid-term Performance Evaluation Report – Transform: PHC Activity Page 20 2 EVALUATION DESIGN AND METHODOLOGY 2.1 EVALUATION DESIGN The mid-term performance evaluation of the Transform: PHC Activity utilized a mixed-method approach, including both extensive quantitative and qualitative methods. Data was also gathered and synthesized from primary and secondary sources, such as project documents. The evaluation team triangulated data from these sources to generate evidence to address the evaluation questions. Activity effectiveness was considered by comparing data between 2017 and the end of 2019, where 2017 is the base period (X1) compared with the mid-level period, 2019 (X2). Both quantitative and qualitative data collection instruments were used, such as structured and semi-structured questionnaires (Annex 6). The methodological approaches utilized in this evaluation support “contribution” measurement and NOT “attribution” of high-level results to the Transform: PHC Activity. The Difference-in￾Differences analysis was used to account for changes associated with prior activities, comparing Transform: PHC intervention areas and Non-Transform: PHC areas. A series of meetings were held with USAID/Ethiopia and Transform: PHC implementers to agree on the design, methodology, evaluation questions, and data collection tools. This resulted in a consensus among key stakeholders about evaluation tools and the evaluation timeline. 2.2 EVALUATION METHODOLOGY SAMPLING A. Household-Level Survey Sample [Detailed Evaluation Methodology and Sample Size Calculation; can be found in Annex 7] The household survey followed a two-stage sampling technique. The mid-term evaluation design provided a representative estimate for the Transform: PHC Activity intervention regions: Amhara, Tigray, Oromia, and SNNP. Since neither the census frame nor a master sample is available at this time because of the upcoming census, an alternative frame was considered, which is a lower administrative unit (kebele), and from which villages (gotts) will be selected randomly. First Stage–selecting sample Kebeles/Health Posts from the four regions. Kebeles were chosen using a simple random sampling technique and assumed 30 households (HHs) per kebele as an optimal representative sample size. A total of 121 kebeles/health posts from intervention and non-intervention areas were randomly selected. Second Stage–a selection of households from the selected kebeles. First, gotts were chosen randomly. A fresh list of households was prepared for each selected gott that serves as a frame from which the households with eligible respondents were systematically selected. Interviewers, with the assistance of kebele guides, clearly identified the boundaries of the study Mid-term Performance Evaluation Report – Transform: PHC Activity Page 21 sites. The eligibility criteria for selecting the households were women aged 15-49 regardless of marital status. The study populations for the mid-term performance evaluation are the primary beneficiaries/recipients of the Activity: ▪ women of reproductive age 15-49; ▪ children under age-5; ▪ primary health care facilities; and ▪ FMOH, RHB, and woreda health sector management team overseeing the health service delivery The household survey’s sample size was determined considering the precision and capability of measuring changes on key outcome indicators of Transform: PHC Activity. Key performance indicators (KPIs) were considered in the sample size calculation, where the indicator that gave the largest sample size was used to determine the sample size; thus, the sampling requirements of all other indicators were satisfied. The sample size was determined using statistical software considering the detecting power of measuring changes over time on the key outcome indicators. The formula used for estimating the sample size is described below. n = (Zα/2+Zβ) 2 * (p1(1-p1) + p2(1-p2)) * d * nr (p1-p2) 2 Where: α: The probability of type I error (significance level) is the probability of rejecting the true null hypothesis. β: The probability of type II error (1 – the power of the test) is the probability of not rejecting the false null hypothesis. d: is the design effect nr: is non-response rate P1 and P2 are the proportions of the outcome indicators of interest at baseline and end-line. The evaluation team also considered the following key indicators in calculating the sample size for the household survey: ● Skilled Birth Attendance (SBA); ● Modern Contraceptive Prevalence Rate (MCPR); and ● Full Immunization Coverage. This evaluation utilized two independent population proportions sample size estimation techniques to calculate the sample size for the household survey using STATA software: power and sample size calculation utility. In the calculation, we considered a 95% confidence level (α), Mid-term Performance Evaluation Report – Transform: PHC Activity Page 22 and 80% statistical power(β). On average, a 10% change anticipated for MCPR; 15% change for SBA; and 15% change for Full Immunization from baseline to end-line (average increment taken from the Transform: PHC Activity’s set targets and the Ethiopia Demographic and Health Survey (EDHS) increment from 2011 to 2016). MCPR gave the largest sample size because the larger sample size is required to detect minimum changes. The estimated proportion of the indicator (P1) for each region was taken from the Transform Program Baseline Survey Report, 2017. To control the loss of effectiveness by the use of a two-stage sampling technique as opposed to the simple random sampling, we used a design effect of 1.75. Therefore, the calculated sample sizes were multiplied by 1.75 design effects for the four regions (Amhara, Oromia, SNNP, and Tigray). A total of 3,625 households with women aged 15-49 (2,664 from Transform: PHC intervention areas, and 961 from non-Transform: PHC areas) were randomly selected and interviewed (Table 3). Table 3. Number of Households Surveyed Region Transform: PHC Intervention Areas Non-Transform: PHC Areas Total # of HH Surveyed # of women 15-49 # of <5 children # of women 15-49 # of <5 children Amhara 689 441 210 128 899 Oromia 657 536 301 287 958 SNNP 657 539 240 181 897 Tigray 661 498 210 167 871 Total 2664 2014 961 763 3625 B. Health Facilities (HF) Sample The purpose of the health facility assessment was to assess the linkage and effectiveness of the primary health care delivery system. For this evaluation, the Primary health care system was defined as a health post, a health center, and a primary hospital. A sampling of health facilities was linked to the sample for the household survey. The designated health post that serves the selected kebele (for the household survey) was automatically chosen for the HF assessment. Health centers are selected based on the linkages they have with the selected health posts. In the same way, primary hospitals (if any) that are linked to the health centers were chosen. In some cases, there may not be primary hospitals available to serve the population. Therefore, the total number of primary hospitals depends on the availability of primary hospitals in the selected health center areas. Mid-term Performance Evaluation Report – Transform: PHC Activity Page 23 A total of 256 health facilities (114 health centers, 121 health posts, and 21 primary hospitals were assessed using a standardized questionnaire (Table 4). (The detailed sampling technique is elaborated in the SOW and appears in Annex 7.) Health facility data were analyzed together with population data to assess the effectiveness of Transform: PHC’s performance, including target outcomes. Table 4: Health Facilities Assessed in Intervention and Non-Intervention Areas Region Intervention Non-intervention Primary Hospital Health Center Health Post Total Primary Hospital Health Center Health Post Total Amhara 2 25 25 52 0 5 5 10 Oromia 5 26 26 57 1 6 6 13 SNNPR 5 24 25 54 0 5 5 10 Tigray 7 19 24 50 1 4 5 10 Total 19 94 100 213 2 20 21 43 C. Key Informant Interviews (KIIs) The evaluations team conducted one-on-one interviews with a variety of stakeholders involved and familiar with the Transform: PHC Activity’s implementation, including FMOH/MCH directorate, Regional Health Bureaus, Transform: PHC staff (central office and regional team), and woreda health offices (USAID Transform: PHC intervention and non-USAID Transform: PHC woredas). A total of 31 KIIs were conducted, and all interviews were audiotaped, translated, and transcribed. D. Focus Group Discussions (FGDs) FGDs were conducted and included to add greater context to the survey responses regarding community members’ knowledge, attitudes, behavior, and practices. Four major categories of focus group participants identified were: • Women age 15-24 • Women age 25-49 • Males age 15-24 • Adult men age 25-50 FGDs consisted of 6-12 participants who live in the catchment area of the study. The focus group discussions were facilitated by senior qualitative experts and one note-taker who spoke the local Key Informants: • Federal Ministry of Health: HEP and Primary Health Care; MCH Directorates (2 KIIs) • Transform: PHC Activity management and technical team at central (Addis level) (3 KIIs) • Transform: PHC Activity regional team (4 KIIs) • Regional Health Bureaus (Amhara, Oromia, SNNP and Tigray regions) (4 KIIs) • Selected woreda health office heads from the four regions (18 KIIs) Mid-term Performance Evaluation Report – Transform: PHC Activity Page 24 language fluently. Twelve FGDs conducted in the study regions and FGDs were audio-taped, translated, and transcribed. E. Survey Instruments and Tools Structured and semi-structured questionnaires were developed for the household surveys and Health Facility Assessment. For comparability, the survey instruments included standard questions from credible sources such as the Demographic and Health Survey (DHS), the FMOH health assessment instruments and other nationally administered, health-related questionnaires. The tools and instruments greatly benefitted from a consultation with all key stakeholders (Annex 6). The individual household questionnaire was prepared in English and later translated into local languages (Amharic, Afaan Oromo, and Tigrigna). Transform: MELA back-translated instruments to ensure accuracy. The questionnaires were then uploaded to smart mobile phones/tablets using the Open Data Kit (ODK) software, and interviews were conducted using a tablet-based approach to ensure seamless data collection. For the Public Health Facility (HF) Assessment, the data collection tools were developed so that service providers and administrators could shed light on the status of service provision at health facilities in the study area. Structured questionnaires were developed and administered to HF service providers to gather information on their opinion regarding their assessment of the effectiveness of health service delivery in the study sites, any improvements in service delivery due to the assistance received from the Transform: PHC Activity, etc. The qualitative instruments and checklists for KIIs and FGDs were developed to understand and capture the experience of public health and woreda officials and their views on the Transform: PHC program health team norms and attitudes as well as beneficiaries’ opinions and satisfaction level regarding the quality of health services being provided. 2.3 TRAINING OF SURVEY TEAM A three-day training was conducted for the survey team (16 supervisors and 65 data collectors, five qualitative experts, and two senior evaluation specialists) in Addis Ababa and the four regional capitals of Ethiopia. The training covered topics such as research ethics in the field, rights of human subjects during the survey, survey methodology and protocol, sampling procedures, informed consent, qualitative and quantitative data collection tools, interviewing techniques, data management, security and quality, and gender considerations during data collection. The survey tools were pre-tested during the training. Mid-term Performance Evaluation Report – Transform: PHC Activity Page 25 2.4 DATA COLLECTION Transform: MELA, in collaboration with its local M&E partner (PRIN), conducted field-level data collection from households and health facilities covering the four Transform: PHC targeted regions (Oromia, Amhara, SNNP and Tigray). The field data collection was conducted from November 11, 2019, through the end of December 31, 2019. The data was collected using smartphones for the household survey, paper-based questionnaires for the health facility assessment, and audio recorder for qualitative data collection. Transform: MELA also recruited experienced qualitative data collectors and senior researchers to conduct FGDs and KIIs with key stakeholders in the four regions as well as Addis level with the aid of audio recorders (see Annex 6). 2.5 DATA ANALYSIS All household and facility data were cleaned, checked, and validated. The distributions and bivariate analyses were conducted to describe the characteristics of a select set of Transform: PHC indicators. Additionally, the evaluation team performed Pearson’s chi￾square tests to assess the statistical significance of differences between the USAID Transform: PHC intervention and non-USAID Transform: PHC areas at mid-term for the same set of indicators. Logistic regression was used to conduct more detailed assessments regarding the predictors of key outcomes, including modern family planning utilization, antenatal care, skilled birth attendance utilization, child vaccination, and seeking treatment for childhood illness. The analysis of household data focused on: • Frequency and percentage of calculations. • Comparison of mid-course vs. baseline key performance indicator values. • Cross-tabulations and multivariate regression analysis of key performance indicators. • Difference-in-Difference analysis (DID). • Assessment of progress and changes in MCH/FP health indicators in Transform: PHC sites with the comparative analysis done in Transform: WASH overlap sites. • Analysis of the performance of USAID Growth through Nutrition (GtN) sites with Transform: PHC in overlap sites vs. Transform: PHC stand-alone sites. For the qualitative data (KII and FGDs) before the start of the analysis, all of the interviews were translated and transcribed in an appropriate format. A codebook was developed based on the evaluation questions. The transcriptions were coded and analyzed using Atlas. Ti software. The Difference-In-Differences (DID) Analysis was conducted to quantify the causal effects of Transform: PHC interventions on outcomes of interest. The Transform baseline survey (2017) and Transform mid-term evaluation (2019) provided data for intervention and non-Transform: PHC areas and were used to estimate the contribution of Transform interventions in improving Mid-term Performance Evaluation Report – Transform: PHC Activity Page 26 key MNCH/FP outcomes. Using the two waves of surveys, this analysis compares the key FP and MCH indicators of the intervention site population against the non-intervention areas. DID is typically used to estimate the effect of an intervention by comparing the changes in outcomes over time between a population that was enrolled in an intervention or program (the intervention group) and a population that is not (the comparison group). The figure below illustrates how DID is estimated. Figure 2: Difference-in-Differences Illustration In conducting the DID analysis, covariates/variables that could potentially predict, for example, the use of FP and MCH services, were included in the analysis: women’s ages (15–19, 20–24, 25–29, 30–34, 35–39, 40–44, and 45–49), women’s educational level (no education, primary, secondary and above secondary), and paid work status (no paid work, paid work). Female participation in their own health care decisions and CBHI enrollment status were included as covariates. The inclusion of such covariates was done for all outcomes/thematic result areas of interest. The exploratory analysis first evaluated frequencies and percentages for each of the categorical variables. Specifically, variables were assessed for near-zero variation (presence of very few observations in any class). Then a difference-in-differences analysis was performed to estimate the contribution of Transform: PHC interventions. The DID design compares changes over time to Transform: PHC intervention (mid-term - baseline) and non-Transform areas outcomes (mid￾term - baseline). DID has been widely-used when panel data or repeated cross-sections are available for intervention impact assessments. The DID estimation offers an unbiased result while accounting for time-invariant unobserved heterogeneity. To reduce selection bias, the DID is estimated by restricting the analysis only to the matched sample (using propensity score matching) at the baseline. The control covariates, including women’s age, education level, paid work, participation in decision-making, and CBHI, are used to match the intervention and non-intervention groups at the baseline. Once matched subjects are identified, the difference in the proportion of each outcome is calculated between the intervention and non-intervention groups, and then DID of outcomes are calculated between Mid-term Performance Evaluation Report – Transform: PHC Activity Page 27 the baseline and the mid-term surveys. A p-value of less than 0.05 was declared as significant and used to interpret the results. Furthermore, the 95% confidence intervals are calculated by adding and subtracting 1.96 times the standard errors from the estimates. 2.6 ETHICAL CONSIDERATIONS All survey team members (evaluation specialists, qualitative experts, household data collectors, and supervisors) were trained in ethical research procedures, including informed consent, the participants’ privacy, and confidentiality. The survey followed standard ethical procedures. Permission was obtained from the head of the household for all participants with a separate consent form signed by the interviewee before the conduct of the actual interview. For women under the age of 18, additional parental permission and participant consent were obtained before administering the survey. 2.7 LIMITATIONS OF THE EVALUATION The methodology employed for the mid-term performance evaluation, while appropriate and statistically rigorous, nevertheless had some limitations. The absence of the Enumeration Area (EA) list from the Ethiopian Central Statistical Agency (CSA), which is in charge of the national census and establishment of EAs, necessitated the use of the list of kebeles for each intervention area as a sampling frame for the first stage of sampling. Transform: MELA utilized kebele guides to clarify and identify boundaries of the selected kebeles and to ensure the complete listing of households in selected gotts. Sampling with replacement was employed, especially where security concerns and inaccessibility impeded data collection in some enumeration areas. Another limitation was isolating confounding factors while using Difference-in-Difference (DID) analysis in the context of development programs taking place in both USAID/Ethiopia Transform: PHC Activity areas and the surrounding areas outside the boundaries of the evaluated regions. During the survey, partner mapping information was gathered from both Transform: PHC intervention sites and non-USAID-Transform sites. The evaluation team found that a myriad of other partner-funded programs exists in both Transform: PHC intervention sites and non-USAID Transform sites. Thus, it is difficult to definitively discern the results and impact of other woreda￾level, FMOH, or donor-financed programs on achieving MNCH/FP targets. The evaluation team relied on the difference-in-differences and causality tracking strategies analysis to overcome these challenges. Thus, the DID results should be interpreted, bearing in mind these considerations. There will also be a spillover effect since TPHC is fully supporting the RHBs and ZHDs on which these institutions are supporting all areas. Given that this evaluation was designed as a mid-term performance evaluation, it was only possible to assess progress towards intended results as opposed to measuring the full achievement of those expected results. It is essential to take note of this when interpreting results from this evaluation. Mid-term Performance Evaluation Report – Transform: PHC Activity Page 28 An additional limitation was the evaluation team’s difficulty in obtaining budget data on MNCH/FP programs operating in USAID Transform: PHC supported sites and non-intervention sites, hampering the team’s ability to carry out a comparative assessment of efficiency based on financial investments. The purpose of highlighting those limitations is to guide the readers. Mid-term Performance Evaluation Report – Transform: PHC Activity Page 29 3 KEY FINDINGS This section presents key findings on Transform: PHC progress and contribution towards achieving its Intermediate Results in the targeted regions. It also presents findings and lessons learned, conclusions, recommendations, and progress related to the evaluation questions of relevance, efficiency, effectiveness, (“likelihood of achieving results”), sustainability/local ownership, and gender dynamics. 3.1 RELEVANCE The relevance of Transform: PHC was evaluated based on the extent to which: a) There are alignment and complementarity with national priorities; b) The stakeholder has a clear understanding and ownership of the Activity; c) The strategies, approaches, and interventions are culturally appropriate for the regional context; and d) The Transform: PHC implementation activities are strategically designed and positioned to achieve its objectives and reinforce the capacity of the FMOH and regional and woreda health systems to deliver quality health services. Alignment/contribution to national priorities and the USAID/Ethiopia strategies: Transform: PHC’s strategy and approaches derived from the GOE/FMOH’s national priorities and, as such, are relevant to and aligned with the FMOH’s HSTP needs. The Transform: PHC Activity’s Theory of Change/Results Framework (TOC/RF), and associated Performance Measurements (PMs) aligned with the USAID/Ethiopia Transform Program description. The program as designed (the TOC/RF and PMs) contributes to the Ethiopia Health Sector’s Transformation strategies. Finally, the conceptual framework and design of the Transform: PHC Activity were based on the FMOH’s HSTP guidance and guidelines. The goal of Transform: PHC is “Healthy, Productive, Prosperous Ethiopians.” To achieve this goal, Transform: PHC supports four interrelated Intermediate Results (IRs) derived from, and therefore consistent with, the FMOH’s HSTP objectives, demonstrating national alignment priorities. A desk review of the Transform: PHC literature15 and related studies, policies, and guidelines confirmed that the Activity is closely coordinating with the FMOH and woreda and civil society organizations at the community level that includes beneficiaries. Stakeholders, including communities, generally agreed that Transform: PHC work is relevant to their needs and the woreda Transformation agenda. 15 Annual workplans for 2017, 2018 & 2019, and quarterly and annual performance reports of the past three years, including other strategy documents. Mid-term Performance Evaluation Report – Transform: PHC Activity Page 30 Evidence from this evaluation showed that the team believes that the strategies and results of the Transform: PHC Activity are well aligned with both the shared FMOH and USAID/Ethiopia’s strategic priorities for the health sector and are an essential element in the achievement by Ethiopia of its HSTP priorities. Co-creation from the outset of the design ensured a clear understanding, buy-in, and ownership of the scope of this Activity. The FMOH Health Sector Transformation Plan (HSTP 1) also reviewed to ensure the Activity’s alignment with national priorities. The relevance of the Activity was also echoed by other stakeholders, including FMOH, and regional and woreda stakeholders. All Focus Group Discussants, including Key Informants at regional and woreda levels, said that Transform: PHC is supporting activities relevant to their needs and priorities. Strategies, approaches, and interventions culturally appropriate to the context of the Activity contexts under which they are implementing: Transform: PHC is utilizing several strategies and techniques to assess and understand the internal and external program contexts, make adjustments in program schedules and collaborate with other stakeholders. The TOC tool to carry out a continuous strength and gap analysis, conducted annually, is helping each regional team to vary intervention packages for high, medium, and low performing woredas and across a variety of geographic and ethnic contexts. The gap analysis also led to the formulation of an integrated periodic outreach tool for hard-to-reach communities. Another example is the use of an Emergency and Resilience Guideline by Transform: PHC to address how to intervene in, and support, regional and local emergency response. Transform: PHC’s annual report (2018) notes that “the annual TOC has helped us understand how the project is progressing and gauge progress toward meeting expected results. The TOC is serving as a project compass, helping to find a way through the fog of complex systems, discovering a path as it goes along”16 . 16 Transform: PHC’s Annual Report. “In our woreda, Transform: PHC generally focuses on assisting the implementation of the HSTP. It is mainly concerned with the issues of quality and equity of service, particularly the five areas of quality which include maternal and child health. Transform: PHC covers all the KPIs that are related to mothers and children. But it is not involved in areas such as TB and HIV. In more detail, Transform: PHC is involved in mothers and children related issues such as, nutrition, Antenatal care (ANC) skilled deliveries, Maternal health, family planning, and in monitoring and evaluation.” -- A respondent from Huletejunese Woreda Health Office in Amhara “Transform Primary Health Care (PHC) is our project, we’re involved in the co-creation exercise that involved us at FMoH, RHB, USAID and this process aligned the project results to the HSTP strategic priorities. Therefore, Transform: PHC is supporting government priorities. The public sector is doing everything while Transform: PHC is playing coordination and facilitation roles.” – A Director at the FMoH Mid-term Performance Evaluation Report – Transform: PHC Activity Page 31 The Transform: PHC strategies, approaches, and interventions are contextually and culturally appropriate to the project contexts under which they are implementing. The Transform: PHC has instituted annual Theories of Change (TOCs) reviews based on evidence generated and consultations with key stakeholders, including FMOH, RHB, woreda health offices, universities, and other private actors. The TOC exercises have enabled the Activity to identify regional and national priorities for the following year, common challenges among woredas, (e.g., model kebele graduation criteria and documentation, Integrated Periodic Outreach Services (IPOS) to address equity gaps), interventions that require modification (e.g., public finance management) and additional interventions (e.g., public health emergency). They also helped Transform: PHC and GOE teams to identify issues for national-level advocacy. The criterion for the woreda management standard is an example highlighted during an Amhara TOC. Transform Program’s Activity strategic positioning to carry out the intervention/support to the FMOH and subsequent structures: The design of the Transform: PHC Activity allowed for government and critical stakeholder ownership due to the co-creation process. It also ensured the alignment of the intervention's design, strategies, and approaches from the outset. GOE stakeholders highly value and perceived Transform: PHC as an essential and strategic partner. There is a strong sense of ownership of Transform: PHC activities from the government stakeholders at the FMOH, regional, and woreda level. All KII respondents at the federal and regional levels believe that the Transform: PHC Activity has continued to be relevant. It appears, however, that there is some misunderstanding in some woredas about the expected transformative outcomes the Activity can support. This may be in part due to the use of the term “Transform” as the title of the Activity. Some regional stakeholders expected and were disappointed that the Activity did not cover the entire GOE/FMOH Health Sector Transformation Agenda. “…. As the government is working on reducing maternal and child mortality today more than ever, Transform PHC is also focusing on this point with special emphasis on new born care, family planning etc. in our woreda. This is a fundamental point! Reproductive maternal newborn and child health is the main concern of the ministry of health today. These are the basics that maternal and child mortality reduction can be achieved. This is what relevance means to us. And, we can realize that the Transform: PHC project is implemented in an integrated manner with our program. Hence the relevance of Transform: PHC is not debatable…” - -- Gorchie Woreda health office head, SNNP. Mid-term Performance Evaluation Report – Transform: PHC Activity Page 32 A respondent from Oromia RHB summed up his views and what other KIIs also expressed: “The project gives the impression that it will help the government transform the health sector. But its support vs. RHB expectation with project performance at the operational level is not adequate. The support spread too thin to yield meaningful, transformational results. Also, thinly-dispersed support can’t help us draw useful lessons.” 3.2 EFFICIENCY Transform: PHC efficiency was assessed based on the Activity management structure, administrative and logistics arrangements, networking, and partnership strategy, and efficiency in resource allocation and expenditure, including its ability to respond to changing contexts. Based on this review, the evaluation team determined that the Activity has efficient systems that advance the attainment of intended results, as discussed below. Management Structure and Arrangements: Transform: PHC has a strong technical and operational capacity and is well-staffed with a team of 388 technical and administrative staff at the various levels of the health system. The Technical Assistance (TA) team supports the GOE/FMOH at the central, regional, and cluster coordination levels. The TA at the woreda level included support for the woreda program officers, administrative and finance staff, and was highly appreciated. The distribution of the Transform: PHC team is as follows. In Summary: ▪ There is a strong alignment between Transform: PHC strategies, approaches and results with the FMOH and USAID/Ethiopia’s strategic priorities for MNCH/FP, thereby, contributing to the realization of the HSTP priorities. ▪ A co-creation approach during the design of the Transform: PHC allowed for strategic alignment of the Activity FMOH and RHB specific priorities and ensured key stakeholder ownership of the process and activity. ▪ The use of approaches and strategies like the Theory of Change (TOC) has ensured Transform: PHC interventions continue to be relevant and contextually and culturally appropriate to the Activity’s contexts under which it is being implemented. Moving Forward, Transform: PHC may consider: ▪ Strengthening collaboration with other external partners involved in interventions outside the scope of Transform: PHC that are addressing priority health needs of targeted regions and woredas. For example, some woredas mentioned that their priority health problems such as TB, malaria, and nutrition for under-five children, should be addressed by Transform: PHC. Mid-term Performance Evaluation Report – Transform: PHC Activity Page 33 Table 5: Distribution of Transform: PHC Team Offices Technical Staff Support Staff Total Staff Country 32 47 79 Regional 58 43 101 Cluster 110 98 208 Total 200 188 388 The staffing configuration has allowed this large-scale Activity to be based closer to the intervention areas and has made it possible for the TA team to monitor and mentor woredas more closely. Transform: PHC cluster offices, mainly co-located with government health offices, jointly implement activities with government counterparts, thereby reducing costs and improving efficiency. The evaluation team found that co-location was useful as it built teamwork between the Transform: PHC and the RHBs, woredas, and zones. Transform: PHC has placed coordinators at regional and zonal offices and program officers at zonal levels with competent program assistants to handle administration and finance issues. This strategy has increased implementation and follow-up capacities of Transform: PHC. The Activity’s field offices, co-located within government health offices, have reduced field office rental expenses, thereby making more funding available for program implementation. Another good example of partnership and sustainability is that Transform: PHC is jointly carrying out Integrated Supportive Supervision Visits with the GOE teams. These visits were conducted by a small group of experienced staff and, the evaluators noted, created a sense of shared responsibility. This ensures skills transfer and efficient utilization of resources. The evaluation team observed two supervisory outreach visits and recorded the collegial supervisory visits. Mid-term Performance Evaluation Report – Transform: PHC Activity Page 34 Woredas Performance Status in Transform: PHC Targeted Regions The evaluation team analyzed secondary data from the Transform: PHC Activity to examine changes in shifts in woredas’ performance between Year 1 (2017) and Year 3 (2019). The result (Figure 3) shows that: • At the startup of the intervention (Year 1, 2017), only 12% of the woredas were considered to be “high-performing.” But in Year 3 (2019), the proportion of woredas that are labeled as high-performing significantly increased to 29% of the total number of woredas (331). • There is an increase in the proportion of woredas labeled as medium-performing. It shows a rise of 11% (from 29% in Year 1, 2017 to 40% in Year 3, 2019). • Similarly, the proportion of low-performing woredas has declined from 58% (Year 1) to 31% (Year 3). This shows that a significant number of woredas’ performance status shifted to medium and high. • Thus, while there is a likelihood to achieve the target to graduate a higher number of woredas, more time would be required, and the current pace of rolling out and strengthening interventions accelerated to meet the graduation targets. With 18 months remaining in the Activity and possible program slow-down as a result of the COVID-19 outbreak, the issue of graduation and an exit strategy from the Transform: PHC sites needs to be revisited (see Table 7 for details on regional variations). Figure 3: Overview of Woredas Performance Status and Shifts in Transform PHC Intervention Areas (between Yr. 1, 2017 & Yr. 3, 2019) 12% 29% 58% 29% 40% 31% High Medium Low Woredas performance status, Yr1, 2017 Woredas performance status, Yr3, 2019 Mid-term Performance Evaluation Report – Transform: PHC Activity Page 35 Regional Variations in Woreda Performance: The analysis results in Table 6 show that: Amhara Region: There is a significant increase in the proportion of low-performing woredas (58%) shifting to medium- and high-performing, 30%, and 28%, respectively. More than half (52%) of the medium-performing woredas moved to high-performing status. However, about 22% of the medium-performing woredas declined and shifted to low-performing woredas. More than 40% of the woredas are still classified as low-performing. About 10% of the high￾performing woredas declined to low-performing status. Thus, it may require exploring further why some woredas at high- and medium-performing status fell into low-performing status. Transform: PHC may need to tailor its technical assistance plan accordingly. Mid-term Performance Evaluation Report – Transform: PHC Activity Page 36 Table 6: Woredas’ Performance Status Shift in the Transform: PHC Targeted Regions (2017 vs. 2019) Region # of Woredas with Performance Status (Year 1, 2017) # of Woredas with Performance Status (Year 3, 2019) The proportion of woredas with Performance Shift Status (Year 3, 2019) Status # of Woredas High Medium Low High Medium Low Amhara High 10 5 4 1 50% -40% -10% Medium 23 12 6 5 52% 26% -22% Low 43 12 13 18 28% 30% 42% Subtotal 76 29 23 24 Oromia High 16 11 3 2 69% -19% -13% Medium 31 11 11 9 35% 35% -29% Low 86 23 30 33 27% 35% 38% Subtotal 133 45 44 44 SNNP High 12 4 7 1 33% -58% -8% Medium 38 12 26 - 32% 68% 0% Low 53 1 26 26 2% 49% 49% Subtotal 103 17 59 27 Tigray High 3 3 - - 100% 0% 0% Medium 5 2 2 1 40% 40% -20% Mid-term Performance Evaluation Report – Transform: PHC Activity Page 37 Low 11 1 5 5 9% 45% 45% Subtotal 19 6 7 6 Overall Total High 41 23 14 4 56% -34% -10% Medium 97 37 45 15 38% 46% -15% Low 193 37 74 82 19% 38% 42% Total 331 97 133 101 29% 40% 31% Data source: Transform: PHC Activity Administrative Data, January 2020 Mid-term Performance Evaluation Report – Transform: PHC Activity Page 38 Grant Management: The woreda grants initiative was designed to flexibly address local requirements and support and build woreda budgeting, and proposal development and grants management and budgeting skills. The evaluation team noted that woreda grants had provided additional resources to finance woredas’ health sector priorities, and the recent addition of crisis funding to respond to new and emerging priorities was welcome. Grant writing skills training at the woreda level did transfer some concrete new skills to woreda officials to better compete for funding from external donors and GOE grants mechanisms. On balance, while woreda officials wished that the grants process entailed less paperwork, the rigor of the process was viewed as new and essential. The grants process was a vital element of the program that involved broader community engagement and governance, which are essential for the sustainability of health programs. The grants mechanism was also viewed as a means to fill woreda budget gaps. However, some expressed concerns with regards to the capacity to access “break-through grants,” and conveyed that they still do not have the in-house capability to identify opportunities and prepare proposals for this kind of grant. Eight regional and woreda KII respondents indicated that Transform: PHC is efficient and has excellent working relations with RHBs and woreda stakeholders. A respondent from Oromia, Woliso health official, noted that “The Woreda grant budget has helped to fund the maternity waiting rooms17. However, it lacks the flexibility to respond to emergencies like outbreaks and its reporting template of the grant fund and often faces several requests to revise the reports”. The Theory of Changes18 (TOC) as a Management Tool: The TOC exercises have generated evidence to support the findings of an enhanced and efficient Transform: PHC. Positive progress is observed in the operations of the Activity. Also, a strength and gap analysis is conducted annually across a variety of geographic and ethnic contexts to identify gaps as areas for improvement. The gap analysis has also led to the formulation of an efficient integrated periodic outreach tool for hard-to-reach communities. Networking and Partnership Arrangements: Transform: PHC participates in a high number of different platforms (networks/working groups). Participation in these groups has helped Transform: PHC know who is doing what, and this has promoted the efficient allocation of resources by avoiding duplication of efforts and funding. For example, Transform: PHC staff 17 Maternity waiting rooms are rooms where expecting mothers wait until they’re ready to give birth. 18 A theory of change is the articulation of the underlying beliefs and assumptions that guide a service delivery strategy and are believed to be critical for producing change and improvement. Theories of change represent beliefs about what is needed by the target population and what strategies will enable them to meet those needs. They establish a context for considering the connection between a system’s mission, strategies, and actual outcomes while creating links between who is being served, the strategies or activities that are being implemented, and the desired outcomes. Mid-term Performance Evaluation Report – Transform: PHC Activity Page 39 are members of USAID’s Gender Champions Network, the national FP technical working group, Child Health and Development working group, adolescent and youth health development, and technical committees organized by the FMOH. Transform: PHC has also supported regional and Zonal partner forums to work collaboratively on gap identification. These platforms have helped Transform: PHC to share and disseminate workable strategies and test approaches with a broader audience. The platforms have also led to joint planning and new guidelines, such as the mentorship guidelines adopted by the government. These networking platforms permit Transform: PHC to more broadly share its materials and ideas with stakeholders. Further, the tools and workable strategies,such as the mentorship guidelines that Transform: PHC is sharing through different networks and technical working groups, have the potential to improve efficiencies and, therefore, provide better results. During the remaining period of the Activity, broader sharing of the full range of Transform: PHC tools and templates should be a priority. Despite the success of its networking and partnership arrangements, there is still room for improvement. For example, a respondent from SNNP RHB said one issue is that the other partners are not learning from Transform: PHC: “There is a weak collaboration between Transform: PHC and other development partners that I am aware of. The other organizations operating in the region have a lot to learn from Transform: PHC, including the approaches that it follows to strengthen the existing health system is something that other organizations should adopt.” KII SNNP, RHB Transform: PHC may need to support and help strengthen the capacity of government structures to effectively collaborate and learn for the achievement of better results using existing platforms. Transform: PHC already can do so as it has worked with regional stakeholders to create a pool of local trainers. These local trainers are providing on-site training and are supporting Transform: PHC with regional and woreda capacity building, activities implementation, and monitoring and oversight. It has also increased the Activity’s efficiency, a view shared by all of its four regional offices during the interviews. “…. There are no organized efforts and capacities specifically built so far to access and use the ”break-through grants” and their requirements. This has resulted in RHB’s inability to organize woredas to assess and identify real gaps and design innovative ideas required to access break-through grants; otherwise, applying to accomplish to routine activities won’t be enough to get it. There are unorganized applications from universities and RHB itself, but a woreda grant is not for academic purpose and needs to look out of the box for innovative ideas for health programming.” -- The Tigray, RHB Key Informant Mid-term Performance Evaluation Report – Transform: PHC Activity Page 40 The on-site training has proved efficient as it is being provided by the health facility staff themselves within their catchment area while Transform: PHC plays a facilitatory role. In addition to saving time and resources, on￾the-job training ensures that more people were trained at the same facility, that user-friendly tools are employed for easy and quick application, and provides the availability of cases for gaining practical experience. It also allows for continuity of quality care when there is attrition or rotation of providers. It enhances team spirit by minimizing unhealthy competition and introduces a culture of peer￾to-peer support and helps develop multi-skilled providers. Key informants mentioned that the activities that contributed to efficiency included: 1) fast reporting mechanisms to communicate results from the zones to the woreda and the RHBs using email and telegram; and 2) financial accountability and transparency in the use of Activity funds. Major stakeholders such as political and religious leaders noted that these approaches served as models for improved governance and decision-making for the use of health resources. Transform: PHC is working closely with local and elected political leaders to allow a safe and flexible environment for the Activity to continue to work even during periods of insecurity in some regions. The Transform: PHC partners’ mapping was frequently mentioned as an efficiency tool as this approach has enabled Transform: PHC to avoid duplication in similar areas. Responding to Changing Contexts: Transform: PHC Engagement in Supporting Emergency Response Actions This section addresses the flexibility and ability of the Transform: PHC mechanism to respond to changing contextual uncertainties. For example, in Tigray, the Transform: PHC team provided short-term TA to the RHB to address the acute watery diarrhea (AWD) outbreak in 2019. As a result, the FMOH expressed an interest in Transform: PHC playing a more significant role in supporting crisis-prone corridors of the four regions beyond the budget level currently allocated. “In each zone, we have created a pool trainer who conducts onsite training and mentorship. These trainers have been capacitated in mentorships and coaching, so it is not only the Activity staffs but also the public sector staffs are our big resources. That is so, because we have capacitated them and have TOT, and they can train and support these woredas under their administration.” –KII Respondent in SNNP “Regarding the collaboration of Transform: PHC with other partners in the woreda, we prepared NGO mapping together with Transform: PHC that contained a list of NGOs working in the woredas, the area of interventions, project date of start and end period. Finally, we exchanged information concerning this with woreda finance and Transform: PHC on yearly basis. Besides, they trained us on how to identify and propose particular support we need from the respective partner. Previously, the trend was different. We did not have the capacity to preparing proposals by ourselves and asking partner(s) for assistance.” –Regional Health Official, SNNP) Mid-term Performance Evaluation Report – Transform: PHC Activity Page 41 In recognition of Transform: PHC’s crisis mitigation role, a director at the FMOH had this to say: “Transform PHC should institutionalize the allocation of flexible crisis modifier funds relevant to the changing context (e.g., during the measles outbreak, scabies, and civil unrest). This is important for risk mitigation and establishing a resilient health system, especially for regions/areas which are crisis-prone. At the onset, there was nothing like crisis funds to Transform: PHC programming. Lately, they avail of these funds with significant delays, and yet crisis needs immediate attention.” One area that may require further strengthening, the evaluation team noted, is related to the disbursement of funds to regions during an emergency. Although Transform: PHC has financial teams in the field, the final decisions are made in Addis, and its financial management system is heavily centralized. Transform: PHC also relies on USAID/Ethiopia’s financial disbursement processes, which also experience delays in the disbursement of funds. This implementation bottleneck should be factored into grant timelines and work plans. Three (3) KIIs reported that this delayed the timely response to several emergencies, such as the measles outbreaks in SNNPR. Given these issues, it is clear that more specific and frequent communication about financial disbursements needs to be reported to regional IP and woreda teams and the RHBs stakeholders, including the process for obtaining “crisis modifier” funds. USAID/Ethiopia and Transform: PHC should identify ways to accelerate the USAID approval process. Data Management and Use: A lot of data was generated at all levels of the health system, including data generated by the Transform: PHC to FMOH, RHB, and woredas. Visits by the evaluation team to health facilities revealed that less than 60% of the health officials were using data to inform and adjust their activities and operations. However, the facilities reported the data was being reported regularly. There is a need to move the needle beyond reporting “accountability for results” to embracing performance management and supporting management decision-making that shapes service delivery and resource allocation. For example, in lower-performing, regions the team did not observe data being used to do cohort analysis or problem-solving, for instance, around the issue of the number of ANC visits or the use of zinc for the treatment of diarrhea. Overall, the team did not see many examples of data driving health decisions. However, several government staff acknowledge Transform: PHC has helped them begin to utilize the data they collect; this appeared to be a relatively new development. A respondent from Huletejunese Woreda, a Health Office in Amhara, summed up what other key informants said, “Transform: PHC is collaborating with us in many ways, by filling the gaps and to be here and reach high performing PHC. Regarding delivery at health centers, there was an inflated data that showed about 90% in [2010 E.C.] and before, with such data, it was difficult to identify a problem. But, Mid-term Performance Evaluation Report – Transform: PHC Activity Page 42 after we conducted a baseline assessment at health centers, we found it was about 24% only. There is now skilled birth attendant assisted deliveries about 42% at health centers and about 18% at hospitals in our woreda. We are also targeting home delivery free, which is to ensure no mothers deliver at home. We still have problems with the quality of our data. We report regularly, and we need to begin using this data to improve service delivery in our woreda. Taking the geographical barriers and limited infrastructures in the woreda, we found it better, showing us we are performing well.” Comparison Between Transform: PHC with non-USAID Transform Projects Regional KII participants compared Transform: PHC-supported sites and non-Transform sites based on the following parameters: comparative strengths and weaknesses of the Transform: PHC Activity concerning non-USAID Transform projects. FGAE, UNICEF, L10K, Save the Children, PSI, Marie Stopes International, Engender Health, IPAS and CIFF were the development actors most often cited by respondents operating in non-Transform: PHC sites. Several key informants said they believe that Transform: PHC-supported woredas are performing better than non-Transform sites because Transform: PHC’s technical support is from the region to the grassroots level. The Transform: PHC Activity also empowers regions and woredas through planning and supervision tools. Transform: PHC was also found to reinforce a bottom-up approach and objective quality assurance. The technical assistance strategy is designed to mentor local teams through on-the-job training and other sustainable ways to introduce quality health delivery services and practices. KIs noted, with satisfaction, the participatory process adopted by Transform: PHC starting from initial planning. KIs also appreciated the evidence-based methods used by Transform: PHC and the approach to training on-site rather than costly off-site training they have observed with other donor￾funded programs that offer high per diems that are not sustainable. The on-site learning also was viewed as less disruptive to the health facilities that are facing critical staff shortages and cannot afford to have key staff sent out for training. A KII respondent from Huletejunese Woreda Health Office in Amhara region summed up what other KIIs said and had this to say about the relevance and uniqueness of the Transform: PHC local training, mentoring and technical assistance program: “Transform: PHC is different from other partners in points such as: it develops partner mapping with us, the other partners give us subgrant budgets while Transform: PHC showed us how to design a proposal and request for a budget. The budget is also fixed, which will be utilized based on the action plan. All of the activities PHC implements at the health center level are transformation agenda topics. There is no report format it developed on its own. It fully adopted the report formats of the health system, so simplified the exchange of reports with us. But, others such as FGAE come up with a different format and asked information not to validate in our context. Besides, the approaches they applied in the pilots were effective”. Mid-term Performance Evaluation Report – Transform: PHC Activity Page 43 The Transform PHC Activity deliberately selected a slice of a zone (50% in most areas) within its four intervention regions to maximize the influence of spill-over effects, and to help ZHDs focus on non￾Transform PHC woredas. Some non-intervention sites have shown improvements where there are no other palpable investments reported. Thus, this success can partly be attributed to this approach and Activity investments. It should also be noted that other non-Transform partners are operating in both Transform PHC intervention sites and non-Transform PHC sites. Besides, other USG and non￾USG investments have been implemented in Transform PHC sites and non-Transform sites. Given that this being a midterm performance evaluation and conducted at mid-way of the Activity, the focus was to assess progress towards achieving intended results and measuring the Activity’s contribution. The understanding that the non-Transform PHC sites are not true controls, hence the conduct of this evaluation took that premise into account and no emphasis on the attribution of results. Some of the Development Partners Working in non-Transform: PHC Sites Overall, it appears that many partners operating in non-Transform: PHC sites are implementing vertical activities. What follows is a sample list of some of the identified health sector partners working in non-Transform: PHC sites. (Note: the list below is not exhaustive.) It may require a separate comprehensive assessment to understand the detailed scope of the support provided by each non-Transform: PHC health sector partner, the resources (level of investment) they have allocated to each woreda, and its impact on the change in the MNCH/FP outcomes. Region Sampled woreda Partners supporting the woreda Tigray Seharti Samre ✓ Relief Society of Tigray (REST): Works in WASH, SBCC sessions on ANC, delivery, immunization, WASH, Exclusive Breastfeeding, Growth Monitoring Program and family planning. There are Productive SafetyNet Program arrangements for pregnant and lactating mothers. ✓ World Vision Ethiopia: Supports WASH, especially in organizing slab producing associations, constructed Maternity wait rooms at health centers where pregnant mothers stay and wait for their delivery. This partner also donated ambulance and drugs for health facilities. ✓ Concern Worldwide: Focuses on nutrition and support in adolescent/youth health. ✓ Sekota Declaration: Works mainly on maternal and child nutrition such as deworming, capacity building. Korem ✓ CO WASH and ONE WASH: government-led programs focus on sanitation infrastructure development and public awareness ✓ Institute for Healthcare Improvement (IHI), which is supporting quality of health service at facilities, improved MCH services quality at health institutions. SNNP Sodo Zuriya ✓ Engender Health, through its “Go Green Initiatives,” has been supporting/providing the Family planning service – Long-acting methods. Arbegona ✓ Growth through Nutrition supports Nutrition and WASH Mid-term Performance Evaluation Report – Transform: PHC Activity Page 44 Oromia Dawe ✓ The woredas appear to have no other development partners Debre Libanos working with them in areas of RMNCH. Amhara Debark Zuria ✓ John Snow Inc. (JSI) L10K: Supports Health Centers with furnishings (i.e., beds, chairs, mattresses, etc.). Mehal Sayient ✓ UNICEF: Nutrition for children under five; and One WASH program for WASH-related activities. A respondent at the Mehal Sayient Woreda Health Office, Amhara Region (non-Transform: PHC) stated: “Our woreda is one from all other Woredas which doesn’t get support from partners. Sometimes UNICEF works with this woreda in nutrition areas supporting under-five children. They deposit the money to the Woreda account. They assign for what purpose we could use the money; it could be for training, supervision, etc. and then we do accordingly. One WASH works in our woreda with building latrine in health posts and health centers.” MNCH focal person, Debre Libanos woreda; Oromia Region (non-Transform: PHC), also had this to say; “We do not have any support at all. Be it on family health or other areas, there are no partner organizations providing support to our woreda. We work with what the government has provided.” “We are also working on family planning at the health center level. In addition, identification of pregnant women, though it is up to the standard and there is a gap in early identification before 16 weeks of gestation, we are working on it. In addition, regarding delivery service, the delivery [institutional delivery] coverage is low since we have a gap in birth preparedness. Especially, the HEWs, after identification of pregnant women, be it before or after 16 weeks [of gestation], there is a gap in making women attend the four follow-ups [ANC] until she gives birth. Regarding immunization, whether a woman gives birth at a health facility or home, we have a form we use, the HEWs through outreach program though not strong, provide home to home follow-up visits.” ----MNCH focal person, Debre Libanos woreda (non-Transform: PHC) Oromia Region ----- We have a problem in supporting the HEWs, monitoring their performances, conducting evaluations, and taking different measures. Moreover, the problem related with service provision is that the HEWs fail to be at the health posts during the working hours and days and provide community health service, thus, the service provision at the community level is weak. There is a huge gap. Regarding the health services provided at the health center level, for the antenatal care, the problem is counselling a woman when to go to a health post, what is there is they provide some services. We sometimes look at their medical cards; there is a gap in providing counselling as per the package. The community, especially women, complain that health workers are not friendly to them. They say, ‘’they insult us, force as to wait for a long time.’’ --------MNCH focal person, Korem, Tigray (non-Transform: PHC) ----- Mid-term Performance Evaluation Report – Transform: PHC Activity Page 45 The MNCH focal person, Korem, Tigray (non-Transform: PHC) further had this to say; The national standard says that pregnant women should be identified before 16 weeks of gestation or within 4 months [of pregnancy]. If we identify within 4 months, until she attends the four follow-up services, her interest to give birth at a health facility and attended by a health worker increases. We, for example, complete service, most of the time, they attend 1st, 2nd and 3rd antenatal care, 4th antenatal care coverage is very low. I think that incomplete antenatal care follow-up contributes to this.” Not all women have awareness regarding the benefits of family planning, pregnancy, child vaccination, child nutrition. Mostly, there is what we call KAP, they could have a bit of awareness and there could be a good attitude, but there is still a problem in the practice. The challenges, as I tried to mention earlier, these challenges, challenges and problems are similar, especially, what I consider as a challenge in this Woreda, because of the failure of the HEWs to keep open the health posts five days a week and during the work hours our performance is below the expectation. Family health related activities are not just done for a month or a week. A respondent from Debark Zuria WHO; Amhara (non-Transform: PHC) had this to say; “….. Our first focus is to make mothers to give birth at the Health center. To do so we have to follow women through their first visit through to the fourth. It’s only those women followed up who ends up delivering at the health centers. To facilitate this, together with health extension workers, we prepare conferences and strengthen the link between health extension workers and midwifes. We have used an example of a woman who last year had placenta retained and her people took her back home so they can treat her traditionally, but she passed away. This case has helped us in discussing with the community, we often tell them about her as an example. Also, giving good health education about health, nutrition, children health by using our development army has helped us to easily achieve our desired results. We give trainings by including administrators, agriculture professionals and health extension workers. Agricultural professionals teach about types and uses of cereals, vegetables and the like and the Health extension worker teaches them how to cook etc. In addition, we have very fast ambulance services. Sometimes we face some problem due to financial bureaucracy procedures because most of the time health issues are not time bound, mostly its emergency, and we have to respond fast. Now day’s people attitudes are getting better towards health, they understand the benefit of using health services including maternal waiting homes. With regards to child health, the community now understand the usefulness of vaccination. Every family vaccinate their children, they make sure their children get Mid-term Performance Evaluation Report – Transform: PHC Activity Page 46 the vaccines. They also understood the consequences of missing vaccination. They fear epidemic might happen, so they communicate with the health extension workers if they missed their vaccination. Health extension workers work by integrating vaccination with maternal health, nutrition assessment, family planning. They also register mothers; give Iron Folic Acid, children birth date, vaccination status etc.” Despite these improvements in Tigray, some health care workers feel that they are on their own. A health office Curative and Rehabilitative Core Process Owner stated: “We don’t have any development partners at all. It is the government that supplies medical equipment/drugs in an interrupted way, and we borrow drugs from nearby health facilities when running short. There is a serious problem of a drug shortage, and private pharmacies inflate drug prices that WHO doesn’t have control.” In Amhara too, some workers felt relatively unsupported. A respondent at the Mehal Sayient Woreda Health Office stated: “Our woreda is one of other Woredas which doesn’t get support from partners. Sometimes UNICEF works with this woreda in nutrition areas supporting under-five children. They deposit the money to the Woreda account. They assign for what purpose we could use the money. It could be for training, supervision, etc. and then we do accordingly. One WASH works in our woreda with building latrine in health posts and health centers.” In Oromia, some of the non-intervention woredas’ health office respondents mentioned there are no partners currently working with them to support MNCH/FP. Respondents confirmed a lack of support from partner organizations, including this MNCH focal person, Debre Libanos woreda, who said: “We do not have any support at all. Be it on family health or other areas, and there are no partner organizations providing support to our woreda. We work with what the government has provided.” Mid-term Performance Evaluation Report – Transform: PHC Activity Page 47 In summary, Transform: PHC’s efficiency is characterized by: • A well-staffed technical team of about 200 experts; • Good participation of stakeholders enhanced by ToC exercise • Prioritized regional and woredas needs; • The random follow-up actions are being used to make adjustments in interventions, and are providing targeted technical assistance and support, training, mentoring and Operation Research (e.g., Weight for Height measurement tool is currently being used in 100 health facilities); • Focal points at government structures who help to facilitate communication, coordination and activity implementation; • Strong collaboration with the GOE at all levels; • Helping recipients to prioritize health-related work through some of the support like crisis modifier and woreda grants; • On-site or on-the-job trainings which save resources and time and have prevented the absence of staff from the HF for off-site training; • Stakeholder mapping; niches, areas of operation and the effective use of technical working group meetings around results, ensure efficiency in the use of scarce resources; and • A use of data for reporting purposes; however, there’s limited (but growing) use of data for management improvements. Mid-term Performance Evaluation Report – Transform: PHC Activity Page 48 Areas for Improvement Include: • Ensure timely flow of funds from the headquarters to the field during emergencies. • There are still gaps in establishing meaningful collaboration between Transform: PHC and other development partners operating in the regions; and • There are still gaps in the capacity of the regions and woredas to identify potential areas for breakthrough grants. Moving Forward: Transform: PHC should consider the following during the remaining period of the Activity: • Ensure that all templates, tools, and materials developed are widely shared with stakeholders and networks. • USAID/Ethiopia and Transform: PHC should develop a graduation strategy for Transform: PHC’s high-performing woredas and a plan for low performers. • Communicate to regional teams and woreda stakeholders the process for “woreda grant approvals,” and work with USAID/Ethiopia to identify ways to expedite USAID/Ethiopia’s approval processes. • Put rapid response procedures in place at Transform: PHC as USAID/Ethiopia uses the Activity as a vehicle to respond to health emergencies and unforeseen disease outbreaks. • Support strengthening the capacity of government structures to enhance effective collaboration and learning for the achievement of better results using existing platforms. • Strengthen regional and woreda capacities to access and use the breakthrough grant’s and their requirements. 3.3 EFFECTIVENESS AND PROGRESS TO ACHIEVING MNCH/FP HEALTH OUTCOMES Data were collected from households on key performance indicators to evaluate the effectiveness of the Transform: PHC Activity. The household survey followed similar methodological approaches as those employed during the Transform program baseline survey to allow for comparison of results. A total of 3,625 women of reproductive age were interviewed from Transform: PHC targeted intervention regions (Amhara, Oromia, SNNP, and Tigray) covering USAID/Ethiopia Transform: PHC intervention areas and non-Transform: PHC areas. The mid-term evaluation also collected and analyzed data from health facilities to assess the interaction between the supply of health services and utilization. Respondent Demographic Characteristics All respondents were women of reproductive age 15-49: 24.3% were below the age of 25 years, and 81% were below the age of 40 in Transform: PHC intervention areas. In non￾Transform: PHC areas, 25.4% were below the age of 25, and 81.9% were below the age of 40. Mid-term Performance Evaluation Report – Transform: PHC Activity Page 49 The mean age of the respondents was 31 years in both areas. Less than half of the respondents (48% in the intervention and 47% in non-intervention areas) had no formal education. In terms of religion, a plurality of respondents (58%) are Orthodox, followed by Muslims (22%) in the intervention areas. Married or cohabitating women constitute 78% in both intervention and non-intervention areas. Table 8 below provides additional details of the surveyed women. Table 7. Respondent Demographics Characteristics Characteristics Transform: PHC Intervention Areas (N=2664) Non-Transform: PHC Areas (N=961) Age Group 15-19 9.8% 9.8% 20-24 14.5% 15.6% 25-29 19.9% 18.8% 30-34 17.5% 19.3% 35-39 19.5% 18.4% 40-44 10.2% 10.0% 45-49 8.6% 8.1% Educational Status No Education 48.4% 46.5% Primary Level 37.5% 37.3% Secondary Level 11.2% 13.6% Grade 12+ 2.9% 2.6% Religion Muslim 21.9% 13.2% Protestant 19.3% 21.5% Catholic 0.3% 0.1% Orthodox 58.3% 64.7% Other 0.3% 0.4% Marital status Single/never married 10.9% 10.8% Married or Cohabitating 78.7% 78.4% Divorced 6.7% 6.6% Widow 2.5% 2.7% Separated 1.2% 1.6% Number of Living children woman has 0 14.5% 14.6% 1-2 27.3% 24.2% 3-4 24.7% 25.2% 5+ 33.6% 36.0% Analysis of Key MNCH/FP Performance Indicators Below is the analysis of the effectiveness of Transform: PHC interventions on improving MNCH/FP thematic areas. Four approaches were used to analyze health performance across the various interventions (FP, maternal health, newborn health, child health, and cross￾cutting programs, including sanitation and hygiene, community-based health insurance, and gender). • Change in the proportion of key indicators in the Transform: PHC intervention areas (Baseline, 2017 vs. Mid-term, 2019) and non-Transform: PHC areas; Mid-term Performance Evaluation Report – Transform: PHC Activity Page 50 • Region-specific performance trends on key indicators at mid-term by comparing with baseline; • Availability of services in the health facilities, • The Difference-in-Differences (DID), • Comparing Transform: PHC’s set performance targets against midpoint key indicator performance values, and; • Regression analysis to identify determinants. 3.3.1 FAMILY PLANNING (FP) Family planning represents efforts to limit or space the number of children through the use of contraceptive methods. Contraceptive methods are classified as modern and traditional methods. Modern methods include female and male sterilization, oral contraceptive pills, intrauterine contraceptive devices (IUD), implants, injectables, female and male condoms, emergency contraception, standard days method (SDM), and lactational amhenoria (LAM). Methods such as the rhythm method and withdrawal, as well as folk methods, are grouped as traditional. The evaluation assessed the performance improvement on key family planning indicators, which is further compared by intervention versus non-intervention area as well as across regions. A. Changes in the Performance of Key Maternal Health Indicators (Baseline vs. Mid-term) in Transform: PHC Intervention and non-Transform: PHC Area The evaluation shows that the use of family planning methods in the Transform: PHC intervention areas has improved slightly over time. The figure below shows changes in key FP indicators between the baseline and mid-term: ● Modern Contraceptive Prevalence Rate (MCPR) among married women increased by 1.5 percentage points, but the change was not statistically significant (44.3% at baseline and 45.8% at mid-term, p-value 0.163). ● Use of Long-Acting Family Planning (LAFP) methods significantly increased by 2.9 percentage points (12.5% at baseline and 15.4 at mid-term, p-value 0.003). ● The use of Family Planning methods in the extended post-partum period (PPFP) also showed a significant increase (5.8 percentage points) from the baseline (34.5%) to mid-term (40.3%), p-value 0.026. ● Key informants from regional offices believe these observed improvements in family planning are associated with the active engagement of health extension workers, the introduction of long-acting methods, and increased availability of FP commodities at health facilities19 . ● There was no change in the proportion of unmet need for FP in Transform: PHC intervention areas compared to the baseline, 31.2%. But there is a significant increase in unmet need in non-Transform: PHC intervention areas of 5.1 percentage points (p￾19 Key Informants from Amhara, Oromia and SNNP RHBs. Mid-term Performance Evaluation Report – Transform: PHC Activity Page 51 value 0.013). The observed values are for all women (married and unmarried). Other surveys show that the unmet need for FP is higher among women with no formal education, lowest quintile, and in rural areas than their counterparts20 where most of Transform: PHC interventions are implemented and could also explain the higher unmet need. Figure 4: Family Planning Indicators Performance in Transform: PHC Intervention and Non￾intervention Areas (Baseline vs. Mid-term) Availability of FP Services at Primary Health Care Facilities In Transform: PHC intervention areas, different FP methods were available. All public health facilities assessed provide at least one type of modern method of family planning services. Injectables (98%), Pills (97%), and Implants (95%) were offered in most of the health facilities. Emergency contraception is available in nine out of 10 primary hospitals and health centers. None of the health posts were offering this contraceptive method. Regarding long-acting family planning methods, 99% of surveyed health centers, and 90% of health posts offer implant (Implanon or Jadelle) methods. Five percent of the surveyed health posts provide Intra-uterine contraceptive device (IUCD) insertion service, but not removal, and this may be one reason why women are not getting IUCDs at health posts despite IUD insertion and removal training offered to level IV HEWs. IUCD service availability in the surveyed health facilities, other than health posts, was 95% and 78% in primary hospitals and health centers, respectively (Table 9). Among all health posts assessed, 96% provide at least two methods of family planning, and one-in-four (26%) provide at least four modern family planning methods. None of the health posts offer all choices of temporary21 methods. 20 EDHS and PMA2020 21 Temporary methods include: Pill, injectables, implants, IUCD, condom and emergency contraception 44.3 42.1 12.5 12.5 34.5 32.6 31.2 28.9 45.8 41.0 15.4 10.9 40.3 38.5 31.2 34.0 Transform PHC Non￾Transform PHC Transform PHC Non￾Transform PHC Transform PHC Non￾Transform PHC Transform PHC Non￾Transform PHC Baseline Midterm Mid-term Performance Evaluation Report – Transform: PHC Activity Page 52 Table 8: Family Planning Methods Offered by Type and Health Facilities (%) FP Methods Provided Primary Hospital Health Center Health Post Pill 100 100 93 Injectable 100 99 97 Implanon 100 98 90 Jadelle 90 80 19 IUD 95 78 5 Emergency contraception 79 92 0 Condom 90 92 26 At least two temporary modern methods 100 100 96 At least four temporary modern methods 95 97 26 Number of health facilities offering any modern contraceptive method 19 94 100 In examining the relationship between demand and supply of family planning services, 11% of women reported they have been turned away or did not receive services when they wanted FP services from health facilities (Table 10). The 11% who did not receive FP services gave the following reasons for not receiving FP services: a) commodity stockout (47%); b) the facility was closed on the day of their arrival (26%), or c) providers not available (17%). Among the 11% who were turned away, the highest proportion was in Amhara 17.4%, and the lowest was in Tigray 7.9%, with 9.1% in Oromia and 9.2 in SNNP. Table 9: Proportion of Women Who Could not Get Family Planning Service When They Want to Use, by Region and Reason for not Receiving the Service Indicator Transform: PHC Intervention Areas Oromia(%) Amhara(%) SNNP(%) Tigray(%) Overall (%) N Women 15 - 49 turned away or did not receive services when they wanted FP services from health facilities 9.1 17.4 9.2 7.9 11.0 2261 Reason for not receiving FP service • Facility was closed 38.5 15.5 37.0 25.0 26.5 249 • Provider not available 11.5 23.3 18.5 7.5 17.3 249 • Commodities out of stock 40.4 50.5 33.3 65.0 47.0 249 • Provider denied service 3.8 10.7 11.1 2.5 8.0 249 • Health of problem 5.8 0.0 0.0 0.0 1.2 249 The findings also showed that 17% of the health centers and 13% of health posts had stockouts of Implanon NXT in the last six months. Stockout could be attributed to an Mid-term Performance Evaluation Report – Transform: PHC Activity Page 53 inefficient supply chain system which resulted in overstock in some facilities and a weak inventory system and low order fill rate.22 In summary: The analysis of the data from this evaluation shows that Long-acting FP methods are available in the health facilities and could have also accounted for the observed improvements. Thus, ensuring the availability of FP commodities, functionality, and availability of providers at health posts and health centers contributesto increased utilization of FP services. Expanding access to the different choices of contraception for both men and women is essential to meet their needs for delaying, spacing, or limiting pregnancy23 . It was widely recognized in the literature that having a wide choice of contraceptives will meet the needs of some who may not be content with their current method and would like to switch. The addition of one new method to the available method mix can increase contraceptive use by as much as eight (8) percentage points24 . B. Regional Variations in Performance of Family Planning between Baseline and Mid-term The Amhara region showed the highest modern method contraception prevalence rate but the lowest rates of LAFP use. SNNP has the highest unmet need for LAFPs compared to the other regions. The highest CPR rate for LAFP is shown in Tigray (20.5%). A similar trend was observed in the baseline results. A KII with an RHB in Tigray gave some potential reasons for the low MCPR rate as follows: “Cultural and traditional restrictions, fear of contraceptive’s side effects, lack of women’s autonomy in health care decisions, and poor health service provision are some of the factors that might be responsible for hindering the region’s MCPR performance.” The highest increase in MCPR was observed in Amhara region, with a 4.5% increase (48.2% at baseline and 52.7% at mid-term). The highest rise in LAFP (6.3%) was observed in SNNP (11.9% at baseline and 18.2% at mid-term), followed by a 5.7% increase in Tigray region. Unmet needs remained unchanged in all regions between the baseline and mid-term surveys. Regarding other regional variations in method adoption, there were no significant regional differences in the availability of implants at health facilities. For example, among HC and hospitals assessed in Amhara, Oromia, and Tigray regions, 100% of them provide implants. But IUD insertion is provided in 94% of HC and hospitals in Tigray compared to only in 78% of HC and hospitals in Amhara region. 84% and 69% in Oromia and SNNP, respectively. 22 IPLS Survey report, April 2019 23 RHB and Woreda KIs from Oromia, SNNP and Amhara 24 Ross, J. and J. Stover. 2013. “Use of modern contraception increases when more methods become available: Analysis of evidence from 1982-2009,” Global Health Science and Practice, 1(2): 203–212 Mid-term Performance Evaluation Report – Transform: PHC Activity Page 54 Figure 5: Key Indicators Performance on Family Planning by Region Figure 6: Key Indicators Performance on Family Planning by Region C. The Difference-in-Differences Analysis Results: Family Planning The contribution of the Transform: PHC Activity in improving FP uptake was estimated through matching the women’s characteristics at baseline (women’s age, education level, paid work, participation in decision-making, and CBHI) in the Transform: PHC intervention and non-Transform: PHC areas. To reduce selection bias, the DID is estimated by restricting the analysis only to the matched sample (using propensity score matching) at baseline. The proportion of LA contraceptive users significantly increased by 4.6% (95% CI: 1.0% – 8.1%; p<0.05) over the intervention period. The intervention’s contribution is significant in reducing the unmet need for contraception; over the intervention period, the unmet need for contraception declined significantly by 6.6% (95% CI: 1.7% - 11.5%; p<0.001) as compared to non-Transform: PHC intervention areas. The result shows a 2.4% increase in the MCPR in the Transform: PHC intervention areas compared to the non-intervention sites, although the increment was not statistically significant. This may be due in part to the lack of availability of quality FP services in non-Transform regions. Therefore, Transform: PHC interventions significantly contributed to LA contraceptive users’ increase by 4.6% and reducing the unmet need for contraception by 6.6%. 4248.2 45.2 41.4 10.5 13.1 11.9 14.8 42.5 52.7 47.1 40.1 14.4 9.1 18.2 20.5 O R O M I A A M H A R A S N N P T I G R A Y O R O M I A A M H A R A S N N P T I G R A Y M C P R A M O N G M A R R I E D L A F P A M O N G M A R R I E D Baseline Midterm 28.2 33 41.7 34.4 33.2 26.2 34 31.7 35.6 44.6 48.8 30.2 33.3 26.9 34.6 30.1 O R O M I A A M H A R A S N N P T I G R A Y O R O M I A A M H A R A S N N P T I G R A Y P P F P U N M E T N E E D F O R F P Baseline Midterm Mid-term Performance Evaluation Report – Transform: PHC Activity Page 55 Table 10. The Difference in Proportions and DID of Family Planning Indicators, Disaggregated by Transform: PHC Intervention and Non-intervention Areas (Baseline, 2017 and Mid-term 2019), Ethiopia. FP Indicators Baseline Values (Dec 2017) Mid-term Evaluation Values (Dec 2019) Difference-in￾Differences (DID) (95% CI) Transform: PHC Intervention (%) Non￾Transform areas (%) Diff (95% CI) Transform: PHC Intervention (%) Non￾Transform areas (%) Diff (95% CI) MCPR (n=5403) 45.7 43.5 2.2 (-1.5; 5.9) 45.8 41.2 4.6 (0.8; 8.3)*** 2.4 (-2.9; 7.7) LAFP (n=5403) 12.8 12.9 -0.1 (-2.6; 2.4) 15.4 10.9 4.5 (1.9; 7.0)*** 4.6 (1.0; 8.1)** PPFP (n=1282) 35.5 34.9 0.6 (-6.8; 8.0) 40.3 38.3 2.0 (-5.6; 9.6) 1.4 (-9.2; 12.0) Unmet need for FP (n=5971) 34.8 36.4 -1.6 (-5.1; 1.9) 31.2 39.3 -8.2 (-11.5; -4.9)*** -6.6 (-11.5; - 1.7)*** Note: The proportions for the DID analysis are calculated after matching and may slightly differ from the indicator observed proportions. *** p<0.01; ** p<0.05; * p<0.1; Diff – Difference in proportions; DID – Difference-in-Differences; CI – Confidence Interval; D. Determinants of FP Use Bivariate and multivariate regression analyses were conducted to identify determinant factors associated with family planning use. The findings indicate that: • CPR is significantly associated with the age of women, literacy level, religious beliefs, children ever born, the desired number of children, and exposure to FP/RH messages. Accordingly, compared to young women (15–19 years of age), the odds of CPR are lower among older women (30 + years) by 51–73%. • Women with a primary level of education are 1.24 times more likely to use contraception than their non-educated counterparts. • Protestants are more likely to use FP by 42% (95% CI: 1.01 – 1.99) than Muslim women. • Those women who have 5 or 6 children are more likely to use FP (OR=1.41; 95% CI: 1.01–1.96) than those with 1 or 2 children. • Expectedly, women who would like to have many children are less likely to use contraception. Exposure to FP/RH messages increases the odds of using contraception by 41% (OR=1.41; 95% CI: 1.18– 1.68). • Long-acting FP (LAFP) use is significantly associated with the women’s region of residence, age, and whether a woman has ever given birth to a child. Compared to women in Amhara region, the odds of LA methods use is higher among women in SNNP region (1.91 times) and Tigray (2.05 times) region. Women in the age range 20– Mid-term Performance Evaluation Report – Transform: PHC Activity Page 56 39 are less likely to use LAFP methods than 15–19-year-old women. Those who gave birth to 5 or 6 children are 1.66 times more likely to use LAFP methods compared to those with fewer children (1 or 2 children). • Unmet need for contraception is associated with age, religion, and whether a woman has ever given birth to a child. Younger women have a higher unmet need compared to their older counterparts. For example, women in the age group 25–29 are 66% less likely to have unmet needs compared to women in the age range 15–19 years. Unmet need steadily increases as the number of children per woman has increased; for example, a woman with 3 to 4 children has 2.2 times more unmet need compared to women with 1 to 2 children. Similar results were seen in the mid-term performance evaluation, and are consistent with DHS and PMA surveys based on proportions for the 15-19 years’ MCPR, LA and Unmet need in lesser than age groups of age 20-24 or 25-29. However, in the regression analysis, when other variables like education, religion, FP message, etc. controlled, 15-19 age groups, seem to have higher proportions; after controlling for the above confounding factors. • Several FGD discussants in all the four regions had similar misconceptions and beliefs including supply-related issues, and these views are echoed by FGD discussants in the Gorchie woreda of SNNP, who had this to say: “Some women do not want to utilize the FP service due to various reasons: lack of awareness about side effects and the belief that FP methods can cause infertility. Implanon can be displaced from the original place of insertion and cause sickness. If not removed timely, it will expose to disease. Some think using the FP method is like killing a baby, which is a sinful act. And others still want to have more children than they have. Health Posts are not providing some of the preferred services, there are no skilled health workers, no electric light, no FP commodities, and no health workers teaching the community about FP in the community, the HEW assigned to the HP is living in the town and provides the service sometimes. So, people must travel on foot for long distances to access the service from the HC.” A key informant at the FMOH said that while the use of FP methods had increased, there was still a need to address the root causes of non-use of contraception. He said: “I would associate the improvements in family planning use to the existence of the health extension workers and an increase in the availability of free long-acting contraceptive methods. Improving the quality of services and addressing the structural issues related to the utilization of family planning methods is critical in closing the inequality gaps. However, to further improve performance family planning, there is a need to introduce innovative and effective approaches that effectively address barriers such as fear of side effects and traditional/religious restrictions.” Mid-term Performance Evaluation Report – Transform: PHC Activity Page 57 3.3.2 MATERNAL HEALTH Maternal health represents interventions aimed at reducing preventable maternal and child death in Ethiopia. Women who adopt sound practices before, during, and after childbirth are much more likely to remain healthy and give birth to healthy children. The indicators included under this section aimed at measuring progress achieved in maternal health over time which is compared by intervention versus non-intervention areas and across regions A. Changes in the Performance of Key Maternal Health Indicators between Baseline and Mid-term in Transform PHC intervention and non-Transform areas. Women who had a live birth in the 12 months preceding the survey were assessed for their experiences with maternal health-related services. The findings of the mid-term evaluation revealed that half (three out of six) of the key maternal health indicators had shown significant improvement in the Transform: PHC intervention areas compared with the baseline values. • The proportion of pregnant women who slept under ITN the previous night increased from 37.9% at baseline to 39.2% at mid-term (1.3 percentage points increase, p-value: 0.39). • Early initiation of antenatal care (less than 16 weeks) has significantly increased from 30.8% at baseline to 55.3% at mid-term (an increase of 24.5 percentage points, p￾value <0.001); • The proportion of women who took iron and folic acid for at least three months during their last pregnancy significantly increased from 20.9% to 26.6% (by 5.7 percentage points, p-value =0.048); Going Forward: To increase the utilization of FP methods, Transform: PHC might consider interventions modeled around the above variables, including: • RH/FP Messaging (women who heard FP messages are 1.4 times more likely to use FP). • Collaborate more effectively with education sector to promote female school retention and enrollment (women in secondary education are 1.4 times more likely to use contraceptive methods). • Tailor interventions for younger women to prevent early unplanned births while also continuing to reach women having more than three children -- unmet need increases as the number of children increases. • Support health posts to integrate and expand the family planning method mix, including IUCDs. • Expand working with communities and religious leaders to change social norms that might be inhibiting uptake of contraceptives. Mid-term Performance Evaluation Report – Transform: PHC Activity Page 58 • The proportion of women who received four or more ANC visits showed a significant decline from 57.3% at baseline to 52.0% at mid-term (by 5.3 percentage points, p￾value <0.001). There was consensus among the four25 Transform-supported regions that mothers’ lack of awareness about the importance of ANC visits coupled with the distance between mothers’ homes and health facilities, and lack of transport to health facilities are among the challenges concerning ANC4+ coverage. Another issue is the beneficiary’s perception of the quality of the services. • The proportion of women who received essential components of ANC26 showed a slight decline from 39.4% at baseline to 38.2% at mid-term. Still, the decline is not statistically significant (by 1.2 percentage points, p-value: 0.70). Of the mothers who have attended at least one ANC visit, 86.9% reported that their BP was monitored, 73.1% reported they received a urine test, and 80.9% provided a blood sample. Counseling on nutrition and pregnancy danger signs was provided to 83.4% and 57%, respectively, during the ANC visits. Thus, the Activity may give more attention to improve counseling on danger signs for pregnancy and support health facilities to do more urine tests for pregnant women as part of ANC services. • Skilled birth attendance showed a slight increase of 2 percentage points (from 66.4% to 68.4%), but the increase is not statistically significant. • Early Postnatal Care (within two days) showed a significant increase from baseline by 5.7 percentage points (p=0.033). 25 Four Transform supported regions: Amhara, Oromia, SNNP and Tigray. 26 Essential Components of ANC include: Blood Pressure measurement, blood and urine sampling, nutritional counseling and information of danger sign of pregnancy complications. “There is a need and more effort to be given to increasing 4 ANC visits. We need accurate and reliable data to assess regional targets. There is also a need to actively engage health extension workers, expand new health service facilities, and increase emphasis on maternal health, all these may increase ANC4+ visits. Mothers’ lack awareness of the importance of ANC visits coupled with the distance between mothers’ homes and health facilities, and lack of transport to health facilities are among the challenges concerning ANC4+ coverage, low quality health services. All these factors discourage pregnant women from returning for repeated ANC visits.” –KII from Oromia RHB Mid-term Performance Evaluation Report – Transform: PHC Activity Page 59 Figure 7: The Proportion of Performance of Key Maternal Health Indicators (Baseline vs. Mid-term) and Comparison between Intervention and Non-intervention Areas Figure 8: Proportion of Performance of Key Maternal Health Indicators (Baseline vs. Mid￾term) and Comparison between Intervention and Non-intervention Areas In Transform: PHC areas, essential ANC laboratory tests were widely available at health centers and hospitals. As a result, the majority (83.9%) of pregnant women who visited the health facilities for ANC service (from the household data) had received either urine analysis or blood tests; tests recommended during pregnancy. Among women who had at least one ANC visit at health facilities, only 57% of them reported that they had been counseled about danger signs of pregnancy complications, of which vaginal bleeding was the most communicated maternal danger sign (for 80.6% of the pregnant women counseled), followed by severe headache (51.5%) and vaginal gush or fluid (34.6%). The health facility data showed that in Transform: PHC intervention areas, 72% of the health centers provide all components of Basic Emergency Obstetric Newborn Care (BEmONC) 30.8 38.0 57.3 57.0 39.4 47.6 55.3 51.8 52.0 52.3 38.2 36.0 Transform PHC Non-Transform PHC Transform PHC Non-Transform PHC Transform PHC Non-Transform PHC Early initiation of ANC ANC 4+ Coverage Essential components of ANC Baseline Midterm 20.9 27.4 66.4 67.4 49.1 44.7 26.6 29.1 68.4 54.4 54.8 42.9 Transform PHC Non-Transform PHC Transform PHC Non-Transform PHC Transform PHC Non-Transform PHC IFA for 90 days or more Skilled Birth Attendance Early Postnatal Care for the mother Baseline Midterm Mid-term Performance Evaluation Report – Transform: PHC Activity Page 60 services and 73.7% of primary hospitals provided all components of Comprehensive Emergency Obstetric Newborn Care (CEmONC) signal functions. All primary hospitals and health centers included in the assessment provide delivery service for 24 hours a day and seven days a week. In addition, 63% of the health centers in transform: PHC intervention areas provide comprehensive laboratory service (VDRL, Hgb, HIV, Blood group, Urinalysis, and Pregnancy test); pregnancy test is the most widely available test (in 98% of the HCs), followed by HIV test (93%), urinalysis (90%), and VDRL (88%) whereas only 67% of the health centers provide Hgb test. Almost all (18 out of 19) primary hospitals provide comprehensive laboratory service in Transform: PHC intervention areas. B. Regional Variations in Performance of Maternal Health between Baseline and Mid-term Region-level performance at the mid-term on key maternal health indicators shows there is a considerable variation among the four regions. Tigray and Oromia made substantial progress on the majority of key maternal health indicators. All four regions showed significant performance improvement in early initiation of ANC (within 16 weeks of pregnancy). The performance on IFA supplementation for 90 days and early PNC coverage increased in all regions, except in Amhara. All regions showed performance improvement in skilled birth attendance, except SNNP, where the performance dropped by 11.2 percentage points (from 63.2% to 52%). However, the performance on four or more ANC coverage showed a decline in all regions, except in Tigray, where there was a 10.4 percentage points increase (from 60.4% to 70.8%). The proportion of pregnant women who received essential components of ANC also declined in all regions except in SNNP, where a performance improvement by 11.7 percentage points was observed. Compared to other regions, Tigray exhibits the highest performance in most of the maternal health-related indicators at mid-term. Early initiation of ANC and SBA is lowest in SNNP at the mid-term compared to other regions and may require more assistance. Even though it shows considerable progress from baseline, Oromia is the region that exhibited the lowest performance in most of the maternal health indicators at the mid-term compared with other regions. Mid-term Performance Evaluation Report – Transform: PHC Activity Page 61 Figure 9: Regional Performance of Key Maternal Health Indicators at Mid-term Figure 10: Regional Performance of Key Maternal Health Indicators at Mid-term C. Difference-in-Difference (DID) Analysis: Key Maternal Health Indicators The contribution of the Transform: PHC Activity in improving maternal health service utilization is estimated using a difference-in-difference analysis described in the methodology section. The DID analysis found that Transform: PHC significantly contributed to: ● An increase in early ANC visits during the first trimester by 11.9 percentage points (95%CI: 1.1% – 22.7%; p<0.05). ● An increase in the proportion of women who gave birth in health facilities assisted with SBA by 15.3 percentage points (95% CI: 4.7% – 25.9%; p<0.001). Part of this increase is due to the decline in performance in non-intervention areas. ● An increase in the uptake of essential ANC services by 10.9 percentage points (95% CI: 0 – 22.2%); and the change is significant at a 90% confidence level (Table 12). 31.3 35.8 21.9 35.7 53.4 55 59.5 60.4 23.9 50.5 28.1 57.6 55.9 59.2 34.5 73.9 42.1 46.5 51.5 70.8 21.5 46.7 39.8 46.9 Oromia Amhara SNNP Tigray Oromia Amhara SNNP Tigray Oromia Amhara SNNP Tigray Early ANC ANC 4+ Coverage Essential components of ANC Baseline Midterm 7 34.9 21.7 21.3 50.3 59 63.2 85.7 28.2 59.6 46.8 64.3 20.9 25.5 28.4 30.6 65.9 71.5 52 88.5 45.2 57.7 52.8 66.7 Oromia Amhara SNNP Tigray Oromia Amhara SNNP Tigray Oromia Amhara SNNP Tigray IFA for 90 days SBA Early PNC Baseline Midterm Mid-term Performance Evaluation Report – Transform: PHC Activity Page 62 Table 11. The Difference in Proportions and DID of Maternal Health Indicators, Disaggregated by Transform: PHC Intervention and non-Transform Areas (Baseline, 2017 and Mid-term 2019), Ethiopia. Indicators Baseline (Dec 2017) Mid-term (Dec 2019 DID Transform: PHC Interventio n Areas (%) Non￾Transform: PHC Areas (%) Diff (95% CI) Transform: PHC Intervention Areas (%) Non￾Transform: PHC Areas (%) Diff (95% CI) ANC 4+ (n=1162) 57.4 54.5 2.9 (-4.9; 10.7) 52.0 50.4 1.6 (-6.8; 10.0) -1.3 (-12.9; 10.3) Early ANC (n= 1216) 28.7 36.3 -7.6 (-15.2; 0)* 55.3 51.0 4.3 (-3.5; 12.1) 11.9 (1.1; 22.7)** Essential ANC (n= 1155) 41.1 47.5 -6.4 (-14.2; 1.4) 38.2 33.7 4.5 (-3.5; 12.5) 10.9 (0; 22.2)* SBA (n= 1251) 65.5 66.5 -1.0 (-8.4; 6.4) 68.4 54.1 14.3 (6.8; 21.7)*** 15.3 (4.7; 25.9)*** Early PNC (n= 1251) 49.4 42.9 6.5 (-1.3; 14.3)* 54.8 43.9 10.9 (3.1; 18.7)*** 4.3 (-6.7; 15.3) IFS 3-month (n= 751) 21.1 31.7 -10.6 (-20.2; -0.9)** 26.6 32.7 -6.2 (-14.8; 2.4) 4.4 (-8.5; 17.3) Note: The proportions for the DID analysis are calculated after matching and may slightly differ from the indicator observed proportions. *** p<0.01; ** p<0.05; * p<0.1; Diff – Difference in proportions; DID – Difference-in-Differences; CI – Confidence Interval D. Factors Associated with Maternal Health Services Utilization The following covariates are included in the logistic regression analysis to identify the determinants of maternal health services utilization: women’s region of residence, age, educational level, paid work status, religion, the number of children ever born, the ideal number of children a woman wants to have, exposure to MCH messages, spouses accompanying women during ANC, spouses accompanying women during delivery, women’s participation in their own health care decisions, and CBHI membership (see Annex 4 for details). The findings indicate that: • MCH messaging, the number of children ever born, and spouses accompanying women to ANC are significantly associated with early ANC visits. Women in Tigray are more than twice as likely to have early ANC visits (OR=2.42; 95% CI: 1.27 – 4.63) as women in the Amhara region. Women who heard MCH messages in the last few months are 1.4 times more likely to have early ANC visits than their counterparts. ● Childbearing experience negatively associated with early ANC; women who have 3 or 4 children and 5 or 6 children are 49% and 65% less likely, respectively, to have early ANC than women with 1 or 2 children. Further research on these findings could shed more light on possible determinants for the observed trend. ● The recommended four or more ANC visits are significantly associated with the region, education, and religion. Women in the SNNP and Tigray regions are 4.25 times and 3.17 times, respectively, more likely to have at least 4 ANC compared to women in the Mid-term Performance Evaluation Report – Transform: PHC Activity Page 63 Amhara region. The better-educated women are the more likely to have 4+ ANC; those who have more than a secondary level of education are twice as likely to have 4+ ANC than their non-educated counterparts (OR=2.22; 95% CI: 1.19–4.12). Orthodox and Protestant religious followers are 55% less likely (95% CI: 15 – 76%) and 88% (95% CI: 68 – 95%) respectively, less likely to attend the recommended 4+ ANC visits than their Muslim religious counterparts. ● The observed decline in the proportion of women who received four or more ANC visits could also be associated with a decrease in a male spouse or partner accompanying women during at least one ANC visit. Women accompanied by their spouses to the ANC clinic are 1.5 times more likely to have four or more ANC visits compared to women who have not been accompanied by their spouse (see Annex 4), including registration in CBHI27 . This study concludes that women registered in CBHI were more likely to access FP and maternal health and had better proportions of uptake than those with no registration. ● Exposure to MCH messages, the number of children ever born, and CBHI membership are significantly associated with skilled birth attendance. Women who heard MCH messages in the last few months are 1.8 times more likely to use SBA compared to women who haven’t. Evidence from this evaluation also showed that 46.4% of the women said they had not heard FP/RH messages in the last few months from any source. Again, around 40% of women interviewed said that they had not received messages related to MCH in the last few months. The presence of HEWs in the communities seems not to give confidence MCH messages are being provided to communities. More messaging needs to be done. Women who gave birth to 3 or 4 children and 5 or 6 children are 74% and 93%, respectively, less likely to have SBA than women who gave birth to 1 or 2 children. ● Being a member of CBHI has positively increased the odds of delivery through SBA more than six-fold (OR=6.66; 95% CI: 1.81 – 24.53). Even if maternal health services are delivered free of charge, CBHI enrollment has shown to empower women to seek out health care services and contributed to improved healthcare-seeking behavior among community members. The findings on the contribution of CBHI in this evaluation are consistent with other studies that women registered in CBHI were more likely to access FP and maternal health and had better proportions of uptake than those with no registration28 . ● The other common reasons hindering pregnant mothers from attending ANC include women's lack of awareness of its importance, distance to the health facility, and unavailability of transportation. Other essential determinants that facilitate ANC attendance include higher education level, woman's ability to make healthcare 27https://www.researchgate.net/publication/332999123_Effects_of_community_based_health_insurance_on_modern_family planning_utilization_in_Ethiopia 28https://www.researchgate.net/publication/332999123_Effects_of_community_based_health_insurance_on_modern_familyplanning_ut ilization_in_Ethiopia Mid-term Performance Evaluation Report – Transform: PHC Activity Page 64 decisions, partner involvement, and MCH messaging29 . 30 3.3.3 NEWBORN HEALTH Interventions aimed at newborn health constitute a critical intervention area in reducing newborn deaths in Ethiopia. To that end, the mid-term evaluation assessed key newborn health indicators to measure progress achieved in Transform: PHC intervention areas, which is further compared with non-intervention areas and across the implementation region. A. Changes in Proportions of Newborn Health Indicators between Baseline and Mid-term in Transform: PHC Intervention Areas and non-Transform Areas The mid-term evaluation finding on newborn health shows all newborn indicators showed improvement from the baseline in Transform: PHC intervention areas. The initiation of breastfeeding within one hour of birth saw the highest increase, by 10.5 percentage points (from 77.6% at baseline to 88% at mid-term, p-value<0.01). The proportion of newborns received postnatal care within 48 hours after birth also significantly increased from 40.8% at baseline to 47.9% at mid-term (by 7.1 percentage points, P-value =0.01). The proportion of newborns delivered at health facilities that received essential newborn care services31 showed significant improvement by 9.1 percentage points (from 6.3% at baseline to 15.4% at mid-term, p-value<0.01). Disaggregation by type of essential newborn care service revealed that skin-to-skin contact service was provided for the majority of the health facility births but umbilical cord care with ointment care was provided less frequently: 46.5% of newborns received vitamin K, 44.3% received TTC eye ointment, 80.7% of newborns had skin-to-skin contact, and 32.8% received umbilical cord care with ointment applied. The low coverage for Essential Newborn Care could be associated with mothers not being informed/told by healthcare providers about the care that was provided to their baby. The survey shows that 35% of the mothers do not know if either of the services was given to their babies or not. The health facility assessment showed that 86% of the health posts in Transform: PHC intervention areas provide community-based newborn care (CBNC) service and 70% of the health centers have a newborn corner with heater/radiant warmer and resuscitation equipment; whereas, in non-Transform: PHC areas, 74% of the health posts provide CBNC service and 55% of health centers have a newborn corner with heater/radiant warmer and resuscitation equipment. The data also shows that 94% of the health centers and primary 29 RHB KIs from Tigray, Oromia, SNNP and Amhara 30 Fekadu H, Tesfahun A and Mequanent M. Late initiation of antenatal care and associated factors among pregnant women in Addis Zemen primary hospital, South Gondar, Ethiopia. Reproductive Health (2019) 16:73. https://doi.org/10.1186/s12978-019-0745-2. 31 Essential newborn care includes Vitamin K, TTC eye ointment, cordcare with ointment, and skin-to-skin contact immediately after delivery. Mid-term Performance Evaluation Report – Transform: PHC Activity Page 65 hospitals practice kangaroo mother care, and 100% of the primary hospitals have a Neonatal Intensive Care Unit (NICU). A similar performance trend was observed in non-Transform: PHC intervention areas, as well, concerning newborn health indicators performance. However, the comparison between intervention and non-intervention areas at mid-term revealed that Transform: PHC intervention areas performed better in newborn health indicators. For example, postnatal care for newborns within two days is significantly higher (p-value <0.01) in intervention areas (68.4%) than non-intervention areas (54.4%). The performance on other indicators, such as the proportion of newborns who received essential newborn care, and early initiation of breastfeeding, is also higher in Transform: PHC intervention areas compared to non￾intervention areas, although the differences are not statistically significant at the 95% confidence level. It is important to note that Transform: PHC intervention areas had better performance at baseline compared with non-intervention areas. Figure 11: Performance of Newborn Health Indicators in Transform: PHC Intervention Areas (between Baseline, 2017 and Mid-term, 2019). The health facility assessment studied the availability of newborn health care services in Transform: PHC intervention areas. The findings showed that the majority of the health posts (86%) provide community-based newborn care (CBNC). The proportion of health centers with functional newborn corner (with heater/radiant warmer and resuscitation equipment) was 70% and all primary hospitals have a NICU. The majority (93%) of health centers in intervention areas were practicing Kangaroo Mother Care (KMC) for pre-term or low birth weight babies. All primary hospitals in intervention areas provide KMC service. In addition, 67% of the health centers and 79% of the primary hospitals use a safe childbirth checklist. 40.8 31.1 6.3 2.2 77.6 64.9 47.9 36.8 15.4 11.9 88.1 85.6 Transform PHC Non-Transform PHC Transform PHC Non-Transform PHC Transform PHC Non-Transform PHC Early PNC for newborn ENC Early initiation of breastfeeding Base line Midterm Mid-term Performance Evaluation Report – Transform: PHC Activity Page 66 B. Regional Variations in Performance of Newborn Health between Baseline and Mid-term The region-level performance on key newborn health indicators shows that all of the regions showed considerable improvements from baseline to mid-term (Figure12). The highest increase in early PNC (within two days) was observed in Oromia region, which increased by 16 percentage points. In contrast, the smallest increment is seen in Amhara region, an increase of 0.1 percentage points. Similarly, SNNP exhibited the highest progress in the performance of newborns received Essential Newborn Care (an increase by 17.9 percentage points), whereas the least performance improvement was observed in Oromia region. Besides, region level comparison at mid-term revealed that: ● SNNP exhibits the highest performance in almost all newborn indicators during the mid-term compared with other regions: early postnatal care for the newborn within two days of birth (52.8%), Essential Newborn Care (38.6%), and early initiation of breastfeeding (89.3%). ● Oromia region exhibited the least performance in a majority of the newborn health indicators, except for early initiation of breastfeeding, compared with other regions. Figure 12: Regional Performance of Key Newborn Health Indicators at Mid-term C. Difference-in-Difference (DID) Analysis on Key Newborn Health Indicators Although there is a significant performance improvement in Transform: PHC areas compared with the baseline values, the DID analysis did not show a statistically significant improvement in newborn indicators (in newborn care or early PNC) compared with non-Transform: PHC intervention areas. (Table 13). The non-significant increase in the DID is because non￾intervention sites also registered increases in proportions/improvements in newborn health indicators, as shown in Table 13 below. The improvements in non-intervention sites could be associated with other partners providing support to regions and woredas. 25.5 46.8 42.3 50.0 0.0 3.8 20.7 1.5 79.2 78.7 69.2 80.1 41.5 46.9 52.8 52.1 2.2 13.9 38.6 12.8 93.3 83.7 89.3 85.3 Oromia Amhara SNNP Tigray Oromia Amhara SNNP Tigray Oromia Amhara SNNP Tigray Early PNC for newborn within 2 days Essential Newborn Care Early initiation of breastfeeding Baseline Midterm Mid-term Performance Evaluation Report – Transform: PHC Activity Page 67 Table 12. Difference in Proportions and DID of Newborn Health Indicators, Disaggregated by Transform: PHC Intervention & Non-Transform: PHC Areas (Baseline, 2017 & Mid-term 2019). Indicators Baseline (Dec 2017) Mid-term (Dec 2019) DID Interve ntion (%) Non￾Interve ntion (%) Diff (95% CI) Interven tion (%) Non￾Interven tion (%) Diff (95% CI) Early PNC for newborns (n= 1251) 40.5 32.5 7.7 (0; 15.3)** 47.9 39.1 8.9 (1.2; 16.5)** 1.1 (-9.7; 11.9) Essential newborn care (n= 849) 6.3 3.2 3.1 (-2.6; 8.8) 15.4 11.7 3.7 (-1.6; 9.0) 0.6 (-7.2; 8.4) Early initiation of Breastfeeding (n= 1066) 76.5 61.5 15.0 (7.5; 22.4)*** 87.4 85.7 1.7 (-4.4; 7.8) -13.3 (-22.9; - 3.7)*** Note: The proportions for the DID analysis are calculated after matching and may slightly differ from the indicator observed proportions. *** p<0.01; ** p<0.05; * p<0.1; Diff – Difference in proportions; DID – Difference-in-Differences; CI – Confidence Interval 3.3.4 CHILD HEALTH This section presents findings on child health, including immunization, diagnosis, and treatment of essential childhood illnesses32 . A. Changes in the Performance of Key Child Health Indicators (Baseline. Mid￾term) in Transform: PHC intervention areas and non-Transform areas The following are key improvements in the use of life-saving child health measures. • Penta 3 coverage: Has shown a significant increase from the baseline by 10.9 percentage points with p-value 0.001 (47.7% at baseline and 58.6% at mid-term). ● Measles coverage: Significantly increased from the baseline by 29.9 percentage points (p-value <0.001), 37.5% at baseline and 67.4% at mid-term. ● Full immunization: The proportion of children fully immunized has shown a modest increase in the intervention period, i.e., 3.1 percentage points (p-value 0.197) ● Symptoms of ARI: Decreased significantly from 7.9% at baseline to 4.5% at mid-term (by 3.4 percentage points, p-value < 0.001). ARI treatment using antibiotics has shown a significant improvement in the mid-term evaluation over the baseline – from 36.2% to 52.3% (increased by 16.1 percentage points, p-value = 0.008). The improvements may indicate both a positive change in community behavior and improved ability and case management by the health providers of ARI cases. 32 The evaluation team collected and tabulated individual child health immunization by data on vaccination coverage for children 12-23 months in two ways: from vaccination cards shown to the interviewer and from mothers’ verbal reports. If the cards were available, the interviewer copied the vaccination dates directly onto the questionnaire. When there was no vaccination card for the child or if a vaccine had not been recorded on the card as being given, the respondent was asked to recall the vaccines given to her child. Childhood illnesses like diarrhea and ARI symptoms were asked for the two weeks preceding the survey. Mid-term Performance Evaluation Report – Transform: PHC Activity Page 68 ● Exclusive breastfeeding significantly increased from its baseline value of 59.4% to 76.8%, resulting in the highest significant increment of 17.4 percentage points (p￾value <0.01) compared with other indicators. ● Children under five years who slept under ITN (previous night) increased from 32.2 percent at baseline to 35.0 percent at mid-term (by 2.8 percentage points). Still, the change is not statistically significant at a 95% confidence level (p-value= 0.032). ● Children 24 to 59 months who were given drugs for intestinal worms significantly increased from 36.6% at baseline to 45.5% at mid-term (by 8.9 percentage points, p < 0.001). ● Treatment of diarrhea with ORS and Zinc increased from 28.4% at baseline to 31.4% at mid-term (by 3.0 percentage points). Still, the change is not statistically significant at a 95% confidence level (p-value= 0.256). There is a similar trend in non-intervention areas as well as with varying degrees of performance change between baseline and mid-term (Annex 1). ● However, diarrhea incidence (in the two weeks before the survey) significantly increased from 8.9% at baseline to 12.3% at mid-term (by 3.4 percentage points, p < 0.001). Figure 13: Child Health Indicator Performance of Transform: PHC Intervention and Non￾intervention Areas (Baseline, 2017 vs. Mid-term, 2019) 41.3 41.6 37.5 41.5 47.7 55.8 36.6 40.0 44.4 43.5 67.4 65.3 58.6 57.5 45.5 41.9 Transform PHC Non-Transform PHC Transform PHC Non-Transform PHC Transform PHC Non-Transform PHC Transform PHC Non-Transform PHC Full immunization Measles Penta 3 coverage Deworming Baseline Midterm Mid-term Performance Evaluation Report – Transform: PHC Activity Page 69 Figure 14: Child Health Indicator Performance of Transform: PHC Intervention and Non￾intervention Areas (Baseline, 2017 vs. Mid-term, 2019) B. Regional Variations in Performance of Child Health between Baseline and Mid-term Figure 15 below shows indicator performance variations by region. The proportion of children fully vaccinated is lowest in the SNNP and Tigray regions, but has the highest coverage rates (51.2%). Although Measles and Penta 3 coverage in the SNNP region is 72% and 52.9%, respectively33, the coverage for full immunization is only 34%, indicating high dropout rates. Measles vaccination and full immunization are highest in Tigray compared to other regions. Amhara region has the least coverage for measles but the highest coverage for Penta 3. Treatment of ARI with antibiotics was the highest in Oromia and the lowest in Amhara. The lowest diarrhea treatment was recorded in Tigray. Overall, diarrhea treatment with ORS and Zinc is deficient in all regions; this may be associated with the unavailability of Zinc tablets in Transform: PHC supported facilities reported to the evaluation team. 33 As reported by SNNP RHB staff, there were frequent outbreaks in the region and campaigns were organized at the community level and thus this has increased the coverage. 36.2 32 8.9 11.8 28.4 24 59.4 75 52.3 63 12.3 12.1 31.4 25.6 76.8 70.5 Transform PHC Non-Transform PHC Transform PHC Non-Transform PHC Transform PHC Non-Transform PHC Transform PHC Non-Transform PHC ARI treated with antibiotics Diarrhea episode Diarrhea treatment with ORS + Zinc Exclusive breastfeeding Base line Midterm Mid-term Performance Evaluation Report – Transform: PHC Activity Page 70 Figure 15: Regional Child Health Indicator Performance – Transform: PHC Intervention Areas (Mid-term, 2019) Figure 16: Regional Child Health Indicator Performance – Transform: PHC Intervention Areas (Baseline vs. Mid-term) C. Difference-in-Difference (DID) Analysis for Key Child Health Indicators As seen in Table 14 below, almost all indicators showed improvements from the baseline. Over the intervention period, the proportion of children aged 0-5 months who are exclusively breastfed significantly increased in Transform: PHC intervention areas as compared to non￾Transform: PHC areas. Similarly, symptoms of ARI declined in Transform: PHC intervention areas as compared to non-Transform: PHC areas. Transform: PHC interventions have contributed to a 20.8 percentage points (95% CI: 4.3; 36.1) increase in exclusive 24.1 67.4 29.7 44.2 36.1 37 33 44.2 37 58.4 37.9 57.4 29.8 35.5 45.3 33.1 43.8 48.8 34.5 51.2 66.1 50.4 72.4 80.6 53.7 67.5 52.9 63.1 44.9 37.4 45.6 52 Oromia Amhara SNNP Tigray Oromia Amhara SNNP Tigray Oromia Amhara SNNP Tigray Oromia Amhara SNNP Tigray Full Immunization Measles Penta 3 Deworming Baseline Midterm 34.6 28.6 25 50 8.1 9.6 9.5 8.6 27.5 18.4 38 27.3 59.7 44.7 66.7 60 61.8 31.3 60.9 47.4 12.2 15.4 14.8 6.9 28.1 41.8 29.9 20.6 59.2 76.5 75.9 93 Oromia Amhara SNNP Tigray Oromia Amhara SNNP Tigray Oromia Amhara SNNP Tigray Oromia Amhara SNNP Tigray ARI treated with antibiotics Diarrhea episode Diarrhea treatment with ORS + Zinc Exclusive breastfeeding Baseline Midterm Mid-term Performance Evaluation Report – Transform: PHC Activity Page 71 breastfeeding, and positively contributed to the reduction of ARI incidence among under-5 children by 4.2 percentage points (95% CI: -7.1; -1.3). Although there are improvements in most of the child health indicators, the DID analysis shows that the changes are not statistically significant as compared to non-Transform: PHC areas. Table 13. The Difference in Proportions & DID of Child Health Indicators, Disaggregated by PHC Intervention and Non-Transform Areas (Baseline, 2017 and Mid-term, 2019), Ethiopia. Indicators Baseline Survey (2017) Mid-term Evaluation Survey (2019) DID Transform : PHC Interventi on Areas (%) Non￾Transform Areas (%) Diff (95% CI) Transform PHC Interventi on Areas (%) Non￾Transfor m Areas (%) Diff (95% CI) Exclusive Breastfeeding (n= 528) 59.7 69.7 -10.0 (-21.9; 1.9) 76.8 66.6 10.2 (-0.2; 20.6)* 20.8 (4.3; 36.1)** Full immunization (n= 911) 40.9 45.3 -4.4 (-13.8; 5.0) 44.4 42.1 2.2 (-6.6; 11.0) 6.6 (-6.3; 19.5) ARI symptom (n= 4170) 8.7 5.9 2.9 (0.5; 5.3)** 4.4 5.7 -1.3 (-3.1; 0.5) -4.2 (-7.1; - 1.3)*** ARI treatment with antibiotics (n= 220) 42.0 46.2 -4.3 (24.1; 15.5) 52.3 54.6 -2.4 (- 20.4; 15.6) 1.9 (-24.9; 28.7) Diarrhea incidence (n= 4236) 10.5 13.9 -3.4 (-6.5; - 0.3)** 12.3 11.2 1.0 (-1.5; 3.5) 4.4 (0.5; 8.3)** Diarrhea treatment with ORS and Zin (n= 500) 29.2 28.8 0.4 (-12.7; 13.5) 31.4 26.5 4.9 (-5.3; 15.1) 4.5 (-11.9; 20.9) Children slept under ITN (n= 4236) 33.7 32.4 1.2 (-3.3; 5.7) 35.0 30.5 4.5 (0.8; 8.2)** 3.2 (-2.5; 8.9) Note: The proportions for the DID analysis are calculated after matching and may slightly differ from the indicator observed proportions. *** p<0.01; ** p<0.05; * p<0.1; Diff – Difference in proportions; DID – Difference-in-Differences; CI – Confidence Interval Mid-term Performance Evaluation Report – Transform: PHC Activity Page 72 D. Determinants of Child Health Service Utilization Full immunization is significantly associated with mothers who are employed and paid a wage, religion, and enrollment in the CBHI34 plan. Children whose mothers have paid jobs are 1.65 times (95% CI: 1.03 – 2.64) more likely to receive full immunization. Children from a family enrolled in CBHI are 1.55 times (95% CI: 1.00 – 2.41) more likely to have full immunization. However, the CBHI enrollment association with full immunization could improve the number of families visiting health facilities, and this could be an opportunity to get their children vaccinated through service integration and to get information about routine immunization services 35. Seeking treatment for fever within 24 hours from the onset of fever is also significantly associated with women’s education, exposure to MCH messages, and religion. Accordingly, women with secondary or above education are 1.3 times more likely to seek medical advice for their children’s fever compared to non-educated mothers. Similarly, women who heard MCH messages in the past few months are 1.3 times more likely to take their children for medical attention compared to their counterparts (Annex 4). A Key informant from a non-Transform: PHC, South West Showa Woreda in Oromia had this to say on the service utilization: “Awareness creation-related activities were implemented among the HEWs regarding the treatment of children at the health posts level. Given that there are gaps in this regard, cases of pneumonia have increased at the health center level. If we can treat these cases at the health posts level, patient load at the health centers reduced. On the other hand, what we consider as one of the problems is improving the community’s awareness related to service uptake. Concerning service utilization, awareness creation is needed. Uptake of services, particularly child health services, maternal health services during pregnancy, antenatal care, and others, is good. However, we believe that community awareness has to increase and is slowly improving concerning institutional delivery.” 3.3.5 SOCIAL BEHAVIOR CHANGE AND COMMUNICATION (SBCC) A. Change in Proportion of Key SBCC Indicators in the Transform: PHC Intervention Sites (Baseline vs. Mid-term) Compared to Non-Transform: PHC areas Two indicators were identified to assess the performance on MNCH and FP/RH messaging for the Transform: PHC Activity (Figure 17). The findings showed that in Transform: PHC intervention areas, the performance on MNCH messaging significantly increased from 24% at baseline to 48.3% at mid-term (by 24.3 percentage points, p-value<0.01), whereas FP/RH messaging remained the same over the two periods (53.7 % at baseline versus 53.6% at mid￾34https://www.researchgate.net/publication/332999123_Effects_of_community_based_health_insurance_on_modern_family_planning_ utilization_in_Ethiopia 35 Transform: PHC KI Mid-term Performance Evaluation Report – Transform: PHC Activity Page 73 term). In non-Transform: PHC areas, while MNCH messaging increased significantly, FP/RH messaging showed significant decline over the two periods. Figure 17. Performance on Key SBCC Indicators between Baseline and Mid-term in Transform: PHC Intervention Areas and Non-Transform: PHC Areas. In Transform: PHC intervention areas, further disaggregation by sources of information showed health workers to be the main source of information for MNCH and/or FP/RH messaging (Figure 18). A majority (83.4%) reported that they have heard the message from health workers, 29.7% heard from radio episodes, and 28.2% heard the messages during a community event. Figure 18: MNCH and/or FP Messaging by Source of Information in Transform: PHC Intervention Areas at Mid-term, Match 2019 B. Region Level Performance on Key SBCC Indicators at Mid-term The findings showed that there was considerable variation in MNCH and FP/RH messaging in Transform: PHC intervention regions. All regions showed improvement on MNCH messaging, where the highest improvement was seen in Oromia (by 38.2 percentage points) and the 53.7 60.4 24 26.8 53.6 47.7 48.3 41.4 Transform PHC Non-Transform PHC Transform PHC Non-Transform PHC Percentage of audience who recall hearing or seeing a specific FP/RH message in the last few months Percentage of audience who recall hearing or seeing MNCH message/ information in the last few months Base line Midterm 29.7 7.7 2.5 6.0 28.2 83.4 1.0 Radio Television Newspaper /Magazine Pamphlet /Poster/Leaflets Community Event Health Workers Other Mid-term Performance Evaluation Report – Transform: PHC Activity Page 74 lowest in Amhara (by 13 percentage points). On the other hand, performance on FP/RH messaging declined in Amhara and SNNP but improved in Oromia. Hence, Oromia exhibited better performance in both indicators compared with other regions. Figure 19: Performance on MNCH and FP/RH Messaging by Region at Mid-term, 2019 3.3.6 CROSS-CUTTING A. Sanitation and Hygiene Transform: PHC is involved with interventions related to hygiene and basic sanitation and the promotion of infection prevention activities at the facility level. At the household level, as part of the on-site support and model kebele creation interventions, the Activity’s communication strategy focused on disseminating information on handwashing with soap and water and the proper use of latrines and water treatment. During emergencies, such as cholera outbreaks, the Activity provided focused support in social mobilization and water treatment supplies. At the facility level, the Activity supported target sites’ CASH programs through its peer education program as well as IP as part of compliance and facility reform interventions36 . ● The major findings related to sanitation and hygiene, as documented in Figure 20 below, are households that have access to basic sanitation facilities significantly increased from 5.9% at baseline to 8.3% at mid-term (by 2.4 percentage points, p￾value < 0.001). ● Households that have handwashing facilities with soap and water significantly increased from 0.8 % at baseline to 4.2% (by 3.4 percentage points, p-value < 0.001). ● Households using water treatment technology increased from 10.4% at baseline to 11.7% at mid-term (by 1.3 percentage points, p-value= 0.067). 36 Transform: PHC FY 2020 Annual Workplan 40.4 50.6 62.3 62.2 12.9 26.1 20.1 37.2 55.6 41.9 55.6 61.9 51.1 39.2 50.5 52.6 Oromia Amhara SNNP Tigray Oromia Amhara SNNP Tigray Percentage of audience who recall hearing or seeing a specific FP/RH message in the last few months Percentage of audience who recall hearing or seeing MNCH message/ information in the last few months Baseline Midterm Mid-term Performance Evaluation Report – Transform: PHC Activity Page 75 Figure 20: Sanitation and Hygiene Indicator Performance in USAID Transform: PHC Intervention Area (Baseline vs. Mid-term) Regional Comparisons: The change in performance in accessing basic sanitation facilities is highest in Tigray region (by 6.6 percentage points), while the performance has declined in the Amhara region (by 1.7 percentage points) between the baseline and mid-term. Although the performance in using appropriate water treatment technology is higher in Tigray at mid-term (23.0%), the change in performance from baseline to mid-term is most significant in the Oromia region (by 6.0 percentage points (Figure 21). Figure 21: Regional Sanitation and Hygiene Indicator Performance in USAID Transform: PHC Intervention Areas (Baseline vs. Mid-term) B. Enrollment in Community-Based Health Insurance (CBHI) Scheme Transform: PHC operates alongside a separate USAID health financing Activity that promotes CBHI enrollment. Figure 22 shows that households enrolled with the CBHI scheme have significantly increased between baseline and mid-term. ● Households enrolled in CBHI schemes increased significantly by 21.7 percentage points, p-value < 0.001 in Transform: PHC intervention areas. ● A similar improvement was observed in non-Transform areas, with a significant 5.9 12.9 0.8 1.7 10.4 7.4 8.3 5.6 4.2 2.2 11.7 12.9 Transform PHC Non-Transform PHC Transform PHC Non-Transform PHC Transform PHC Non-Transform PHC Basic Sanitation Hand washing facility with Soap and Water Water treatment technology Base line Midterm 3.3 6.2 6.3 7.9 1 0.6 1.3 0.3 5 6.9 9.5 20.2 5.5 4.5 8.8 14.5 2.3 4.9 9.1 0.3 11 6.8 6.1 23 Oromia Amhara SNNP Tigray Oromia Amhara SNNP Tigray Oromia Amhara SNNP Tigray Basic Sanitation Hand washing facility with Soap and Water Water treatment technology Baseline Midterm Mid-term Performance Evaluation Report – Transform: PHC Activity Page 76 increase of 17.4 percentage points, p-value < 0.001. ● There is remarkable progress in all regions between baseline and mid-term. However, the evaluation team also noted the existence of a stand-alone health care finance project funded by USAID that supports CBHI-related interventions in the non-Transform: PHC sites. SNNP and Oromia regions, however, may need more support to increase CBHI enrollment, as it appears to be one of the women’s empowerment approaches that are also linked to positive changes in MCH performance. Figure 22: Household CBHI Enrollment in Transform: PHC Intervention Areas & Non￾Intervention Areas Figure 23: Household CBHI by Region (Baseline, 2017 & Mid-term, 2019) 28.8 30.2 50.5 47.6 0 10 20 30 40 50 60 Transform PHC Non-Transform PHC CBHI Enrollement Base line Midterm 9.9 43.1 14.6 47.6 37.8 67.6 31.2 64.5 Oromia Amhara SNNP Tigray CBHI Enrollement Baseline Midterm Mid-term Performance Evaluation Report – Transform: PHC Activity Page 77 At mid-term, of those households enrolled in CBHI schemes, 77.4% and 76.3% of them renewed their membership in Transform: PHC intervention and non-intervention areas, respectively, during EFY 2011 or after. The proportion of households that renewed their CBHI membership is relatively higher in Amhara region, both in Transform: PHC intervention and non-intervention areas (90.5% and 82.7%, respectively). Women with CBHI have more access to health services, and whenever they go for services37, health education is provided. CBHI has also empowered women to demand health services, which has led to increased service utilization. Figure 24: Household CBHI Membership Renewal in Transform: PHC Intervention Areas and Non-Intervention Areas; and by Region (Mid-term, 2019) C. Gender - Women Decision-Making and Male Involvement • Women accompanied by their spouse during delivery for their last birth slightly increased from 82.9% at baseline to 84.6% at mid-term (by 1.7 percentage points, p￾values= 0.263). The change, however, is not statistically significant at a 95% confidence level. • Women's participation in decisions regarding their health care showed no progress between baseline and mid-term (84.2% at baseline and 83.4% at mid-term). • Women accompanied by their spouse during ANC visits for their last birth appeared to decline from 62.2% at baseline to 48.6% at mid-term (by 13.6 percentage points, p-value< 0.001). The regression analysis also showed that women accompanied by their spouses at ANC clinic are 1.5 times more likely to have 4+ visits compared to women who have not been accompanied by their spouses (see Annex 4). Thus, encouraging spouses to accompany their partners for ANC could go a long way in encouraging women to complete the recommended 4+ ANC visits. 37 Transform: PHC gender analysis report Mid-term Performance Evaluation Report – Transform: PHC Activity Page 78 • Non-intervention areas have a similar trend between baseline and mid-term, but the actual proportion is higher in Transform: PHC intervention areas compared to non￾intervention areas. Figure 25: Women’s Participation in Decision-Making and Male Involvement Regionally, women's participation in decisions regarding their health is highest in the Amhara region (93.2%) at mid-term. However, the Tigray region achieved the highest performance difference (7.4 percentage points) between baseline and mid-term – from 76.0% to 83.4%. A decline in performance was observed in the other two regions, Oromia and SNNP. Males accompanying their wives/partners during at least one ANC visit have declined in all the four regions with Oromia region showing a 24.8 percentage points decline. Men in Tigray region are less likely to accompany their wives during delivery (77.7%) while men in Oromia region are more likely to accompany their wives (93.5%). Similarly, the decline in performance in accompanying wives/partners during delivery is relatively higher in Tigray region (by 2.0 percentage points). In comparison, Oromia region scored the highest increase (6.2 percentage points) between baseline and mid-term. 84.2 84.2 62.2 59.2 82.9 81.1 83.4 80.2 48.6 42.9 84.6 89.1 Transform PHC Non-Transform PHC Transform PHC Non-Transform PHC Transform PHC Non-Transform PHC Women's participation in decisions regarding their own health Women accompanied by their spouse during ANC Women accompanied by their spouse during the delivery Base line Midterm Mid-term Performance Evaluation Report – Transform: PHC Activity Page 79 Figure 26: Regional Women’s Decision-Making and Male Involvement in USAID Transform: PHC Intervention Areas (Baseline vs. Mid-term) 3.3.7 COMPARISON OF MID-TERM TRANSFORM: PHC ACTIVITY TARGETS VS. MID￾TERM PERFORMANCE Transform: PHC’s performance on key MNCH/FP indicators was reviewed, in part, to determine the likelihood of the Activity reaching its Life of Project/Life of Activity (LOP/LOA) targets. Table 15 shows indicator performance at the mid-term against the LOA targets. Family Planning ● The target for MCPR for the mid-point was 49%, and the achievement at mid-term is 45.8%. This indicates that the MCPR performance is less than the targets, though it is within the acceptable performance margin, and may require attention if the LOP target has to be achieved. ● MCPR attained is 15% for LAFP against a mid-term target of 15.4%. The likelihood of achieving its LAFP target by the end of the project is very high if similar efforts are exerted and assumptions remain the same. ● The target for PPFP use is 41% at the mid-point, and the achievement at mid-term is 40.3%. This shows the performance on PPFP use is on track to be achieved by the end of the project. Maternal Health ● Transform: PHC projected to increase the early initiative of ANC to 38% at the mid￾point of the intervention. The mid-term evaluation findings show early initiation of ANC is at 55%, which significantly exceeded the LOP target, and this may require revision of the target. ● Skilled birth attendance and Early PNC performance are within the acceptable 88.5 91.3 79.7 76 69.2 68.3 52.6 60.4 87.3 81 85.2 79.7 81.7 93.2 76.4 83.4 44.4 56.9 35.8 59.4 93.5 80.2 88.6 77.7 Oromia Amhara SNNP Tigray Oromia Amhara SNNP Tigray Oromia Amhara SNNP Tigray Women's participation in decisions regarding their own health Women accompanied by their spouse during ANC Women accompanied by their spouse during the delivery Baseline Midterm Mid-term Performance Evaluation Report – Transform: PHC Activity Page 80 performance margin and are more likely to achieve the LOP target. ● The remainder of the maternal health results (4+ ANC coverage, essential components of ANC, and iron folic supplementation within 90 days) are below the targets set for FY2019. This may necessitate that USAID/Ethiopia either revise the targets downward or reassess what it will take to reach these ambitious changes over the next two years. Newborn Health ● Transform: PHC met all set targets for newborn health for 2019, except for Newborns that received Essential Newborn Care (target of 20% with a performance at mid-term of 15.4%). This achieved below the acceptable margin. More effort is required or target revision may be necessary. Child Health ● Achieved its target set for Full Immunization, Penta 3, and Deworming. ● The target for Measles, Exclusive breastfeeding, and Children who had ARIs treated with antibiotics achieved above 110%. Thus, it may require a revision of the target (upward). ● The treatment of diarrhea with ORS and Zinc achieved below the acceptable margin. More effort is required. ● The targets set in the child health result area are modest, given the current positive pace of performance. Gender: Women’s Decision-Making & Male Involvement: The findings indicated that Transform: PHC achieved its FY2019 targets of women being accompanied by their spouse during delivery for their last birth and women's participation in decisions regarding their health. The performance is within the acceptable performance margin and, thus, is more likely to achieve the LOP target. However, less progress was realized on women accompanied by their spouse during ANC visits. (The achievement is below the acceptable margin. More effort is required.) Sanitation and Hygiene: Transform: PHC achievement on the Proportion of Households (HHs) with basic sanitation facilities and the use of appropriate water treatment technologies is below the acceptable margin. Thus, more effort may be required to achieve the LOP target or revise the target. Transform: PHC Activity Performance on Set Targets at Mid-term Per USAID’s standard and recommendations + or – 10% of the target is an acceptable margin. Anything beyond 110% requires a revision of the target or halting of implementation as per other criteria, and anything below 90% may require more effort or a target revision (see Table 15 for details). Mid-term Performance Evaluation Report – Transform: PHC Activity Page 81 Table 14. Transform: PHC Activity Performance on Set Targets at Mid-term Thematic Area KPI Activity Target at Mid-term (%) A Perfor mance at Mid￾term (%) B % Achieved (B-A)/A LOP Targets Likelihood of Achieving LOP Targets Family Planning MCPR among married women 49 45.8 93% 55 It is within the acceptable performance margin. More likely to achieve the LOP Target LAFP among married women 15 15.4 103% 19 On track- It is within the acceptable performance margin. More likely to achieve the LOP Target PPFP 41 40.3 98% 52 On track- It is within the acceptable performance margin. More likely to achieve the LOP Target Maternal and Newborn Health Early Initiation of ANC 38 55 145% 53 Achieved above 110%. Requires revision of target (upward) ANC 4+ Coverage 64 52 81% 75 Achieved below the acceptable margin- More effort required or target revision IFA for 90 days 36 27 75% 55 Achieved below the acceptable margin- More effort required or target revision Essential components of ANC 52 38 73% 81 Achieved below the acceptable margin- More effort required or target revision Skilled Birth Attendance 75 68 91% 90 It is within the acceptable performance margin. More likely to achieve the LOP Target Early PNC 57 55 96% 74 It is within the acceptable performance margin. More likely to achieve the LOP Target Newborn health Early PNC for a newborn within two days 48 48 100% 65 It is within the acceptable performance margin. More likely to achieve the LOP Target Essential Newborn Care 20 15 75% 49 Achieved below the acceptable margin- More effort required or target revision. Early initiation of breastfeeding 86 88 102% 90 It is within the acceptable performance margin. More likely to achieve the LOP Target Child Health Full Immunization 49 44.4 91% 56 It is within the acceptable performance margin. More likely to achieve the LOP Target Measles 46 67.4 147% 60 Achieved above 110%. Requires revision of target (upward) Penta 3 56 58.6 105% 70 It is within the acceptable performance margin. More likely to achieve the LOP Target ARI treated with antibiotics 45 56.9 126% 60 Achieved above 110%. Requires revision of target (upward) Mid-term Performance Evaluation Report – Transform: PHC Activity Page 82 Thematic Area KPI Activity Target at Mid-term (%) A Perfor mance at Mid￾term (%) B % Achieved (B-A)/A LOP Targets Likelihood of Achieving LOP Targets Diarrhea treatment with ORS + Zinc 38 33.6 88% 50 Achieved below the acceptable margin- More effort required or target revision Deworming 45 44.9 100% 64 It is within the acceptable performance margin. More likely to achieve the LOP Target Exclusive breastfeeding 63 76.8 122% 71 Achieved above 110%. Requires revision of target (upward) Cross-cutting: Gender Women's making decisions regarding their health 88 83.4 95% 92 It is within the acceptable performance margin. More likely to achieve the LOP Target Women accompanied by a spouse during ANC visits 66 48.6 74% 74 Achieved below the acceptable margin- More effort required or target revision Women accompanied by a spouse during delivery 83 84.6 102% 89 It is within the acceptable performance margin. More likely to achieve the LOP Target Sanitation and Hygiene The proportion of HHs with Basic Sanitation Facilities 17 8.3 49% 32 Achieved below the acceptable margin- More effort required or target revision The proportion of HHs using appropriate Water treatment technology 21 11.7 56% 36 Achieved below the acceptable margin- More effort required or target revision Mid-term Performance Evaluation Report – Transform: PHC Activity Page 83 3.3.8 COMPARISON OF USAID/ETHIOPIA TRANSFORM: PHC-ONLY AND TRANSFORM: PHC-WASH OVERLAP AREAS AT MID-TERM Comparison of Transform: PHC-only areas and Transform: PHC and WASH overlap areas at mid-term showed that access to basic sanitation is relatively lower in Transform: PHC-only areas as compared to the Transform: PHC-WASH overlap areas (8.69% versus 11.9%) with the difference not significant at a 95% confidence level, p-value= 0.069. There is also no significant difference (p-value= 0.296) in Transform: PHC-only (3.6%) and Transform: PHC-WASH overlap areas (2.43%) in households having handwashing facilities with soap and water at a 95% level of confidence. There is a relatively lower performance in the Transform: PHC-only areas (12.0%) over the Transform: PHC-WASH overlap areas (13.37%) in using appropriate water treatment technologies. The proportion of diarrhea episodes in the two weeks before the survey, though not statistically significant (p￾value=0.387), is moderately higher in the PHC-only areas (12.3%) as compared to the Transform: PHC￾WASH overlap (10.27%) areas. Similarly, the treatment of diarrhea with ORS and Zinc is slightly lower in Transform: PHC-only areas (27.2%) than the PHC-WASH overlap areas (30.4%), p-value= 0.742. Overall, there does not appear to be a significant difference in performance between Transform: PHC-only and Transform: PHC-WASH overlap areas for the selected set of sanitation and hygiene indicators. Figure 27: Comparison of USAID Transform: PHC-only and Transform: PHC and WASH Overlap Areas at Mid-term Although there is better performance in the Transform: PHC and Transform: WASH overlap areas, the difference in proportion is not statistically significant. 3.3.9 SUSTAINABILITY ELEMENTS The Transform: PHC Activity has integrated the following strategies/approaches into its interventions to help ensure the sustainability of its intended results after the project ends: ● Transform: PHC Activity supports government priorities for health systems. Transform: PHC implementation modalities are integral and part of the government health sector, so there is no parallel implementation structure. This promotes self-reliance and the sustainability of MNCH/FP interventions. ● Other interventions mentioned to enhance sustainability include LAFP back-up support to 8.7 12 12.3 11.9 13.4 10.3 0 5 10 15 Basic sanitation facility Use of appropriate water treatment Diarrhea incidence Transform PHC sites only Transform PHC & WASH overalp sites Mid-term Performance Evaluation Report – Transform: PHC Activity Page 84 health posts, woreda-level planning exercises, on-site training arrangements, clinical skill labs, creation of local pull of trainers, and training capacity38 . ● Designed and implemented coaching and mentorship support using the public system structures – Woreda-level interventions, either at woreda health offices or at primary hospitals, are essential to sustaining gains on service delivery and program management. Catchment based clinical mentorship, leadership development coaches, public finance management mentorship, connected woreda mentorship, and quality improvement mentorship are all about establishing and building sustainable capacity at the RHBs and the woreda level. ● The GOE has adopted some of the innovative approaches introduced by Transform: PHC, including mentorship guidelines, sub-grant management to woredas, and twinning partnerships. ● With regards to twinning, some local governments (e.g., the Oromia special zone of the Amhara region) have adopted the approaches and allocating of resources for peer-to-peer learning among woredas within their zones. The twinning of high-performing woredas to low￾performing woredas has allowed the low-performing woreda to learn from the high performing counterpart. A twinning participant from Machakel Woreda Health Office had this to say; “We brought back model kebele tracking tools from Bibugn Woreda and started to implement them in Machakel Woreda. We believe that the partnership helped us to accelerate toward our common goal of becoming a transformed woreda in Amhara Region.” The regional and woredas KIIs all believe that the strategies, elements, and approaches Transform: PHC has put in place reinforce the sustainability of the Activity results. A KII respondent from Huleteju Woreda Health Office in Amhara region said what was heard in other woreda and regional KIIs: “All the approaches applied by Transform: PHC consider sustainability. They are training an individual in a health institution to sustain the training agenda in that institution. Onsite training is given to workers in health centers; Transform: HPC is mainly supporting us in the transformation agendas; we never rely on their presence to keep the activities on track. So, it is possible to say it is mainly supporting us to be sustainable. We oppose the Family Guidance Association, as we evaluated the organization at the woreda level and found outdated approaches such as motivating health extension workers with allowance, where sustainability is not ensured if the compensation goes away. But Transform: PHC is quite different from that. For instance, if you conduct onsite coaching and onsite supervision, the institutions will sustain. Furthermore, if we identify training gaps in a specific health institution, then the training will be given there, and the gaps become filled; in this way, the institution becomes capable of providing all services properly. Therefore, Transform: PHC is playing a vital role in sustainability. If you take previous partners, they came with their agenda, whether it was long-term or not, and endorsed funds to run that agenda only.” 38 Transform: PHC technical staff KIs Mid-term Performance Evaluation Report – Transform: PHC Activity Page 85 Although Transform: PHC has incorporated the above sustainability elements into its programming, sustainability remains a challenge, as regions have limited budgetary and resource mobilization capacity to become self-reliant. Rolling out of the graduation strategy for high-performing woredas is key and should be done now. Likewise, engaging the FMOH about what to do in low-performing woredas is also critically important over the next year. This may require longer-term engagement, improved GOE resource mobilization, and management capacity to ensure the continuity of the results beyond the life of the Transform: PHC Activity. 3.3.10 TRANSFORM: PHC’S PROMISING APPROACHES AND STRATEGIES NOTED BY STAKEHOLDERS The Transform: PHC Activity designed, piloted, and adopted several innovative approaches/strategies and processes in collaboration with the FMOH, and Regional Health Bureaus. These approaches/strategies and processes have the potential to catalyze the achievement of intended results and ensure sustainability and deserve to be scaled up. Some of these include: 1. Twinning among high- and low-performing woredas and facilities. Twinning (peer-to￾peer learning) among high- and low￾performing woredas seems to be a promising way to share experiences and improve service delivery in low-performing woredas. The twinning partnership is an adopted strategy to enhance performances at woredas as well as primary level care facilities. The implementation of the strategy requires an orientation, joint problem analysis and solution identification, and project development. The strategy is being implemented in all four target regions, covering 112 woredas in 56 partnerships. This approach also strengthens partnerships and positive competition among woredas. The strategy was included in the woreda transformation strategy and the draft of the Health Sector Transformation Plan II. Also, the FMOH, as part of performance-based financing, has extended budgetary support to start the partnership in 24 woredas and zonal health departments to scale-up the strategy within their zones, (e.g., Oromia zone and Amhara region). This strategy created a genuine and healthy collaboration between woredas, whereby woredas, which used to be competitors, now work more collaboratively to share experiences to help each other perform well. 2. Theory of Change (TOC): The Activity uses its TOC as a planning tool to identify preconditions that lead to desired changes concerning the transformation agenda and preventing maternal and child deaths. The TOC exercises have enabled the teams across all four regions to identify regional and national priorities for the following year, common challenges among woredas, (e.g., model kebele graduation criteria and documentation, IPOS to address equity gaps), interventions that require modification (e.g., public finance management) and additional interventions (e.g., public health emergency). It also helped the regional teams identify issues for national-level advocacy, such as the criteria for woreda management standard, identified during an Amhara TOC exercise. 3. Woreda grants have provided additional resources to finance woreda health sector priorities and “We sent five staff to a performance-review meeting, to visit health facilities, and bring back lessons to our health center...the twinning team mobilized resources for creating a clean and safe health facility and establishing a sanitation committee.” –Case study document- Digo-tsion Health Center staff, Amhara Mid-term Performance Evaluation Report – Transform: PHC Activity Page 86 needs. Stakeholders appreciated the mechanism at different levels since it fills woreda budget gaps. Furthermore, KII respondents also acknowledged the experience of grants management, including the oversight of these grants, which has provided valuable expertise for woredas to implement future performance-based financing that may be financed by the GOE. The FMOH has replicated many aspects of the woreda subgrant management in other non- Transform: PHC woredas using government financing. 4. Coaching and mentorship support using the public system structures: Woreda level interventions, either at woreda health offices or at primary hospitals, are essential to sustaining gains on service delivery and program management. Catchment based clinical mentorship, leadership development coaches, public finance management mentorship, connected woreda mentorship, and quality improvement mentorship are all about establishing built-in capacity at the woreda level. So far, the expansion of these initiatives through public sector resources as well as other partners are going well, which signals that the strategy is useful and a candidate for scale￾up. 5. Scale-up of “Her Space” Initiative”: “Her Space” Is a strategy that focuses on empowering adolescent girls aged 11-14 years through building their knowledge and confidence in Sexually Reproductive Health (SRH) and other social and life skill issues. The strategy is implemented in 37 selected kebeles in Transform: PHC intervention regions. The steps involved community sensitization and stakeholders’ meetings and establishing a committee that selects mentors to be trained and adolescent girls to be enrolled in the program; about 32 mentors have been trained, and 800 adolescent girls are enrolled in the program. The Activity uses a curriculum adopted from the previous program called “Girl Effect” to educate adolescents. The adolescent girls meet once a week with their mentors for 40 consecutive weeks before they graduate. As part of their training, the girls visit social institutions such as health facilities and police stations so that they are aware of where to go when they need SRH services and have issues to discuss with these social institutions. Some of the girls are already graduated, and a few others will graduate very soon. It has the potential to empower adolescent girls and build their knowledge and confidence in SRH and other social and life skill issues and should be scaled up. 6. Addressing emergencies while working on systems-strengthening: The crisis-modifier budget is primarily intended to manage outbreaks/emergencies, saving lives, and avoiding back-sliding of health development gains. For example, the current measles and cholera outbreaks and landslides in SNNPR, Oromia, and Amhara regions are being managed using the crisis modifier budget in parallel with the program’s service delivery interventions. 7. Innovative clinical skill lab at the health center level strengthens peer-to-peer education and provides practical hands-on experience to practitioners and interns. 8. Random follow-ups provide real-time data to inform technical support and a useful internal monitoring system. The follow-up visits are a strategy used to standardize on-site technical support and an internal monitoring system -- using real-time data -- to improve program activities. As a result of data from the random follow-ups, the checklist and programming have evolved based on feedback from users of the checklist and collected data. Other essential tools, such as Mid-term Performance Evaluation Report – Transform: PHC Activity Page 87 the quality grid for IMNCI, were revised to take into account ways to fill gaps in the disparity between classification and treatment for malaria. Additionally, program managers have also improved their programming and focus. Random follow-up visits are a good way to rapidly assess program strengths and weaknesses and appear to promote evidence-based decisions and a more targeted allocation of resources. 9. The provision of solar suitcases by the Transform: PHC Activity is solving power problems at health centers and increasing the delivery of services. Regional respondents said this technology was found to be effective. The installations have resulted in an increase in institutional delivery and reduced maternal referrals for delivery services in almost all health facilities. By ensuring the availability of power, the intervention also improves the quality of prenatal and delivery services through fetal heart rate monitoring that requires a steady power supply to operate the equipment. This has the potential to reduce both maternal and infant death at health facilities. This vital input further enhances the capacity of healthcare providers to diagnose and treat obstetric complications, minimizing unnecessary night referrals, and reducing unnecessary costs associated with these emergency referrals. The continuous supply of power was also observed to be improving health workers' motivation and morale and is key to improvements in the quality of delivery of health services by reducing errors while conducting assisted deliveries. 10. Leadership development (building data-savvy leaders): Leaders at primary healthcare facilities using data properly leads to appropriate decision-making and enhances the process of addressing problems. The connected woreda strategy, supported by the Activity, regularly measures the use of quality data at the primary level of healthcare and is led by the woreda and facility leadership. In addition to this, the leadership development program, which focuses on the use of data to identify issues, jointly solve problems, and monitor changes, is another opportunity to develop a culture of data use. 3.3.11 ANALYSIS OF THE COMBINED EFFECTS BETWEEN USAID TRANSFORM: PHC AND GROWTH THROUGH NUTRITION (GTN) ACTIVITY This evaluation analyzed the combined effects of USAID-funded Growth through Nutrition (GtN) and Transform: PHC interventions in selected areas of overlap. Data were collected from 20 overlap areas and included interviews with 598 households. The following overlap indicators were used for calculating percentages and observing if there is a significant difference in proportions between these two types of areas: Iron folate supplement during pregnancy; Early initiation of Breastfeeding; Vitamin A supplementation; Deworming; Exclusive Breastfeeding; Diarrhea treatment with ORS and Zinc; and the use of appropriate water treatment technologies. The results are shown in Table 16, below: ● The overlap areas showed better results in Iron folate supplement use, early initiation of breastfeeding, Vitamin A supplementation, and deworming compared to Transform: PHC alone. However, the results show no significant difference between the combined effect of Transform: PHC and GtN compared to Transform: PHC alone. Mid-term Performance Evaluation Report – Transform: PHC Activity Page 88 ● Performance in children treated for diarrhea with ORS and Zinc was the only indicator found to be statistically significant in overlap areas as compared to Transform: PHC only areas (p=0.02). Table 15: Combined Effects between USAID Transform: PHC Activity & Growth through Nutrition (GtN) Indicator Transform: PHC and GtN Overlap Areas (%), Transform: PHC only Intervention Areas (%) P-Value Iron folate supplement for three months during last pregnancy 33.3 24.3 0.11 Early initiation of breastfeeding 90.7 87.4 0.10 Exclusive breastfeeding (children <=5 months) 73.8 77.4 0.62 Vitamin A supplementation in the last six months 35.1 31.4 0.07 Children 2-5 yrs. given a drug for Intestinal worms (Deworming) 46.4 45.2 0.73 Complementary feeding (children 6-8 months) 60.0 72.4 0.24 Diarrhea Treatment with ORS and Zinc 43.9 27.6 0.02* Households using appropriate water treatment technology 9.7 12.2 0.09 Although positive changes in performance in the Transform: PHC and GtN overlap areas are observed, analyzing the net effect of the two may require further isolation of the contribution of other confounding factors, which was not done in this evaluation. 3.3.12 ANALYSIS OF HEALTH FACILITIES READINESS AND PRACTICE IN PREVENTION AND MANAGEMENT OF POST-PARTUM HEMORRHAGE (PPH)39 Postpartum Hemorrhage (PPH) is commonly defined as a blood loss of 500 ml or more within 24 hours after birth. PPH is the leading cause of maternal mortality in low-income countries and the primary cause of nearly one-quarter of all maternal deaths globally40. According to the WHO estimate, PPH occurs in 2% of deliveries. This evaluation assessed health facilities’ readiness, practice in prevention and management of PPH cases including the use of a childbirth checklist, providing Basic Emergency Obstetric and Newborn Care (BEmONC) signal functions, presence of Maternal Death Review Committees, Availability of Oxytocin, availability of ferrous sulfate and folic acid, and women’s stay in HFs for at least 24 hours after delivery. In assessing facilities’ readiness in the prevention and management of PPH cases, the findings are presented below and in Table 17: ● 43% of the health facilities reported that they had at least one PPH case in the last three months, including referrals. Overall, 133 PPH cases were reported from 48 health facilities (31 health centers and 17 primary hospitals) in the last three months prior to the health facility assessment. 39 This is an additional analysis request made by USAID, hence no comparison between baseline and mid-term. 40 WHO recommendations for the prevention and treatment of postpartum hemorrhage. Mid-term Performance Evaluation Report – Transform: PHC Activity Page 89 ● A safe childbirth checklist was used by 69% of health facilities surveyed. ● 72% of health centers provide all BEmONC signal functions. Majority of the health centers provide most of the signal functions, where administration of Anticonvulsant drug being the least provided service compared with the other interventions; 97% administer Parenteral Antibiotics, 94% administer parenteral uterotonics, 85% administer Anticonvulsant drug, 98% manually remove placenta, 97% remove retained product, 96% mange assisted vaginal delivery, and 99% perform newborn resuscitation. ● Oxytocin was available in 96% of the health facilities assessed. In the absence of Oxytocin, only a few health facilities use misoprostol. ● Women stayed in health facilities for at least 24 hours after delivery: 44% of women who had live births at health facilities in the last year before the survey reported that they had stayed at least 24 hours after delivery, with 41% and 53% of women delivered at HC and hospitals, respectively. ● Iron and folic acid supplements were available in 79% of health facilities. However, the proportion of women who took iron tablets for at least three months during their last pregnancy was only 27%. ● Maternal Death Review Committees were present in 84% health facilities. Table 16: Health Facilities Readiness, Practice in Prevention & Management of PPH Cases Indicators % N 1. The proportion of health facilities which are using safe childbirth checklist 69 113 2. The proportion of health centers providing BEmONC signal functions 72 94 3. Proportion of facilities with the presence of maternal death review committees 84 113 4. Availability of Oxytocin at health facilities 96 113 5. Ferrous sulfate and folic acid availability 79 113 6. The proportion of women who took iron tablets for at least three months during their last pregnancy 27 320 7. The proportion of women who stayed in HFs for at least 24 hours after delivery 44 334 8. The proportion of HFs reporting at least one PPH case in the last three months (including referrals) 43 113 It was reported that the major challenges related to the management of PPH include: • Lack of skilled providers; • Mothers not willing to stay for 24 hours at a health facility after delivery; • Lack of ambulance services; • Communication barriers for timely referral; and • Shortage of infrastructures such as running water and electricity. Thus, to improve health facilities readiness and the ability to manage PPH cases, interventions should focus on increasing the supply of skilled health service providers, devise context-specific interventions and messaging that motivates mothers to: 1) stay for 24 hours after delivery; 2) take iron tablets, and 3) improve referral linkages. Mid-term Performance Evaluation Report – Transform: PHC Activity Page 90 4 KEY IMPLEMENTATION CHALLENGES Despite the various achievements observed and documented, as with any activity, there are always areas for improvement. The evaluation team identified the following key challenges: • Continuous health sector leadership and technical staff turnover remain problems. KIs reported that there continues to be high health worker staff turnover and very high vacancy rates for key clinical positions across all four regions. • Security has remained an implementation challenge in some parts of the regions, for example, West Amhara, specifically Gondar area as part of the security issues and West Oromia and south of SNNP. This affected the regular technical support and monitoring activities. • Adequate provision of service delivery has also been impacted by the presence of 27 hard-to￾reach areas. The integrated periodic outreach approach for hard-to-reach areas, while working, maybe less practical than the model of intensive technical assistance provided in other regions. In many cases, the harder-to-reach areas were found to require more intensive day-to-day hands-on technical support. 5 KEY LESSONS LEARNED DURING THE IMPLEMENTATION OF THE TRANSFORM: PHC ACTIVITY ● The twinning partnership approach created collaboration and experience-sharing between high- and low-performing woredas and facilities; this approach has scaled up from 4 to 56 woredas twinned. Additional resources may be required to expand this approach further. Based on the findings from this evaluation, the twinning partnership exercise helps both woredas to accelerate progress toward their respective transformation targets, and it includes on- and off￾site trainings, which are vital mentorship elements. Going forward, GOE/FMOH may want to consider using twinning as a viable approach for technical assistance across regions. ● Transform: PHC has supported the government to adopt and roll out several of its guidelines and manuals such as the Primary Health Care Performance Standards, Mentorship Guidelines, and Gender Mainstreaming manuals. It is too early to assess the impact of these promising new standards, although the evaluation team observed the consistent roll-out of the mentorship guidelines in all RHBs. ● Sub-granting to woredas has proved to be an essential mechanism to advance woredas’ MNCH priority activities. However, the grants development and approval process were found to have drawbacks in some woredas that faced sector budget reductions in part due to the receipt of the Transform: PHC sub-grant. For example, Woliso woreda in Oromia Region reported its budget was cut as a result of benefiting from the Transform: PHC support. However, the majority of the woredas visited that received subgrants didn’t face sector budget reductions, and three woredas in Oromia seemed to be outliers. Ensuring that the receipt of a small grant does not displace or replace GOE funding should be considered as a criterion for Transform: PHC support. ● Decentralized mentorship by teaming and twinning woredas between high-performing health centers with low-performing Health Centers are effective in sharing good practices and improving service delivery. Networking through GOE/FMOH coordination platforms has also proved to be necessary for sharing tools and lessons learned. Mid-term Performance Evaluation Report – Transform: PHC Activity Page 91 ● Allocation of the Crisis Modifier Fund allows/ensures there are funds available for crisis-prone regions when an unforeseen crisis emerges. Mid-term Performance Evaluation Report – Transform: PHC Activity Page 92 6 CONCLUSIONS The Activity has achieved impressive health results and has successfully built a management and assistance platform that is responsive to the GOE/FMOH’s health sector transformation agenda. Thus, the Activity is on track to achieve intended results provided the suggested recommendations are executed, and there are no emerging or unforeseen emergencies that impede service delivery for a protracted period. The Activity is positively regarded across the country by RHBs. It is providing tangible and meaningful assistance to step up and improve the GOE’s desired goal to end preventable maternal, newborn, and child deaths. Overall, there is an increase in performance related to Family Planning, Maternal, Newborn and Child Health, Hygiene and Sanitation, and CBHI. CBHI enrollment positively contributed to women’s use of family planning, and some maternal health services have made an important contribution to improved service utilization. The Activity has met or exceeded some targets and is on track to achieve the intended results. Several indicators have demonstrated performance improvement between the baseline (2017) and mid-term (2019). Although some of the performances reached statistical significance, some have not. Given that these indicators are at the outcome and a few at impact level, it is not surprising that the increments in these indicators are not statistically significant given that results at this level of the results chain often yield significant changes over a longer period. However, the overall performance of Transform: PHC on key performance indicators registered marked improvements, and these improvements are also supported by qualitative data. Evidence from this evaluation has shown that FP, Maternal Health, and Child Health outcomes have common determinants. These include literacy level, religious beliefs, access to health messages, and CBHI enrollment. Improving health outcomes may require designing integrated interventions that take all these variables into account. It may require developing a comprehensive pilot intervention that incorporates key technical areas such as health, education, etc. This could provide valuable learning on programming inter-dependencies, thus articulating a need for a coordinated, multi-sectoral award to improve MNCH/FP sustainably. Regions have varied performance in MNCH outcomes. Regions and woredas with low performance could benefit from cross-learning. That said, the observed gap at baseline between the lowest and highest, and between the lowest and highest at mid-term has widened. For example, with MCPR, Oromia was the lowest with 33.3%, and Amhara was the highest with 39.1%, resulting in a gap of 5.8%. At mid-term, Tigray was the lowest with 33.3%, and Amhara was the highest with 46%, giving a gap of 12.7%. The same pattern was observed in SBA, ITN, and ANC 4+ differences between the highest and lowest at baseline and are smaller than the mid-term. It is clear that there is much to be learned from high-performing woredas, and particularly those that have succeeded in moving from low- to high￾performing in resource-scarce situations. There are many ways to either directly or indirectly incentivize a few select centers of excellence that might be explored. Other local technical assistance options should also be explored so that woredas can readily access their financing to strengthen performance. Mid-term Performance Evaluation Report – Transform: PHC Activity Page 93 The achievement of improved health outcomes and attainment of health results (FP utilization, maternal health, child health, and newborn child outcomes) require a positive partnership and collaboration across multiple partners and stakeholders. The evaluation found that lower-performing woredas with standalone services for health were not as effective as those woredas that had more robust financing for all of the transformation agenda. The team observed that the most significant health impact was taking place where there was dynamic leadership at the woreda who could harness enough resources to achieve mutually reinforcing interventions. With the limited time remaining in the Transform: PHC Activity, it would be useful to carry out some “deep-dive” analysis on specific interventions that are still lagging,such as the treatment of diarrhea with zinc or the insertion of IUCDS by level IV HEWs at health posts. A lot of data is being generated at all levels of the health system right from the Transform: PHC to FMOH, RHB, and woredas. A visit by the evaluation team to sampled health facilities revealed that less than 60% of the HFs were using data for reporting purposes. However, the facilities reported the data was being reported regularly. There is a need to turn the needle beyond reporting and move towards “accountability for results.” There was not much evidence reported that data is driving resource allocation decisions within health facilities. Given that this evaluation was designed as a mid-term performance evaluation, it focused on assessing “progress towards” or the “likelihood of achieving intended results” with regards to the effectiveness question, as opposed to “measuring the full achievement of intended results and attribution.” The analysis also assessed Transform: PHC contributions to different thematic area results. Based on this premise, the evaluation concluded that the Activity is on track to achieve the intended results. It is important to note this when interpreting results from this mid-term performance evaluation. Mid-term Performance Evaluation Report – Transform: PHC Activity Page 94 7 KEY RECOMMENDATIONS FOR STAKEHOLDERS: Recommendation/Action Actor to Address Timing Priority # USAID/Ethiopia 1 USAID/Ethiopia and the GOE should revisit some of Transform: PHC’s targets to be achieved over the life of the Activity. Some targets appear to have been set high, and the target setting should consider factors like past experiences in achieving similar targets, exert opinions, resources available, contextual factors, etc. Reporting should then be linked to result areas and revised performance targets. USAID/ Ethiopia Immediate 2 USAID/Ethiopia should consider realigning the budget, if feasible, to provide additional resources for interventions that are under￾performing (e.g., Maternal Health, ANC4+, SBA), and scaling up others that were found to be successful. USAID/ Ethiopia Immediate 3 USAID/Ethiopia should ensure that context monitoring and reporting is mandatory for Transform: PHC. The success of any initiative requires gaining a sound understanding of the two-way interaction between Activity and context, taking quick and effective actions to minimize negative impacts and maximize the positive effects of interventions. A review of Transform: PHC performance reports revealed that, although the context is being monitored, there was no systematic reporting or a section dedicated to context monitoring, mitigation, and reporting in the Activity performance reports. In the Ethiopian context, there is a need to deliberately monitor and document context monitoring and mitigation concerning its effect on programming and how programming is shaping it. USAID/ Ethiopia Immediate 4 For future programming, USAID/Ethiopia may want to consider competitively selecting woredas that have multisectoral investments that support the achievement of MNCH/FP objectives. Future PHC activities should be designed in the same regions, ensuring that the woredas have a robust transformation agenda that will complement and reinforce USAID/Ethiopia’s health interventions. Consideration might be given to competitively selecting the next round of woredas for participation in such programming. USAID/ Ethiopia Future Design/ Programming (Beyond the life of Transform: PHC) 5 Continue using co-creation in designing new projects. This approach should also include bottom-up co-creation with the regions and woredas. USAID/ Ethiopia Future Design/ Programming (Beyond the life of Transform: Mid-term Performance Evaluation Report – Transform: PHC Activity Page 95 Recommendation/Action Actor to Address Timing Priority PHC) For Transform: PHC Activity Implementer(s) 6 Consider the approach to technical assistance and the gaps in thematic areas where there are low or, as yet, no discernible performance improvements in some of the maternal health indicators such as 4+ANC, and women who received essential components of ANC. Transform: PHC should also intervene in identified determinants that affect performances. In some thematic areas, this may necessitate revisiting strategies and approaches to be more productive. The approaches considered should also be geared towards narrowing regional gaps in key outcomes. Transform: PHC Immediate 7 Provide a comprehensive package for systems strengthening support to a selected number of woredas for better efficiency and to improve the quality of services being provided, hence, scaling up the achievement of desired outcomes. Transform: PHC Immediate 8 Assess the effectiveness of the woreda grant towards improving MNCH/FP outcomes in the supported woredas. Transform: PHC Immediate 9 Consider integrating community interventions in a few woredas, targeting women to improve empowerment and roll out learnings to scale up to other areas. Transform: PHC Immediate 10 Work with the FMOH and RHBs, including woredas, to ensure that there is a roll-out of a graduation plan for high-performing woredas and transition them to “model” woredas. On the other hand, as there are varying regional performances, mechanisms should be devised to improve the performance of low-performing woredas to medium and high. Transform: PHC Immediate 11 To increase utilization of family planning methods and maternal health services, Transform: PHC should increase/intensify its support to more women to register in CBHI and interventions modeled around working more with communities, community leaders, and religious leaders and FP/Reproductive Health messaging. Transform: PHC Immediate 12 Continue and strengthen collaboration with Transform: MELA to enhance the effectiveness of the operations research studies conducted by the Activity. This will help improve the relevance and help to maximize the use of vital information and recommendations made in those studies. Transform: PHC Immediate Mid-term Performance Evaluation Report – Transform: PHC Activity Page 96 Recommendation/Action Actor to Address Timing Priority 13 Strengthen/forge strategic partnerships with key actors, recognizing their niches and comparative advantages; provide support to government and mobilize additional resources around outcomes. This should include a joint gap analysis of regions/woredas, agreement on interventions with the highest potential to achieve given results, including who is responsible for what intervention. Sharing of planned activities and joint outcome monitoring should be central to these partnerships. Transform: PHC Immediate 14 RHBs and Transform: PHC should ensure that the receipt of a small grant does not displace or replace GOE funding and should be considered as a criterion for Transform: PHC support in all supported regions and woredas. Transform: PHC Immediate Recommendations for GOE/FMOH, RHBs 15 Strengthen the coordination mechanisms with key stakeholders to guide and tailor actions towards improving synergistic efforts related to MNCH. GOE/FMOH, RHBs Immediate 16 RHBs and Transform: PHC should clearly define sustainability elements that should be integrated into ongoing programming to ensure that results are sustained beyond the life of the Activity. This should include a deliberate effort made now to identify and define resiliency elements to be integrated into Activity interventions and strengthened in crisis-prone regions. GOE/FMOH, RHBs Immediate 17 Continue to strengthen the health extension program as the HEWs are vital to the primary care system. This may require revisiting the Health Extension Program (HEP) management and motivation strategies, and ensuring the continued acceptability of the program to the communities served. Besides, integrating the HEW fully into the health system without losing its unique identity could help reduce inequalities to access and increase the utilization of essential, high-impact health interventions. GOE/FMOH, RHBs Immediate/ Long-term 18 Improve resource mobilization and information use to support enhanced evidence-based decision-making and ensure leadership stability. GOE/FMOH, RHBs Immediate/ Long-term 19 Going forward, GOE/FMOH may want to consider using twinning as a viable approach for technical assistance across regions. GOE/FMOH, RHBs Immediate/ Long-term 20 Consider scaling-up the number of health facilities that have access to a solar power supply. This system was found to be useful in supplementing the power supply at health facilities, thus improving staff morale and the quality of care. GOE/FMOH, RHBs Immediate/ Long-term Mid-term Performance Evaluation Report – Transform: PHC Activity Page 97 ANNEXES Annex 1: USAID Transform: PHC Activity: Performance by Result Area at Mid-term (2019) Compared with the Baseline (2017) in Transform: PHC Intervention and Non-intervention Areas Key Performance Indicators Transform: PHC Intervention areas (%) Non-Transform: PHC areas (%) Oromia Amhara SNNP Tigray Total N Total N Expanded access and uptake of family planning 1 Modern Contraceptive Prevalence Rate (MCPR) among all women Baseline 33.3 39.1 37.5 33.6 35.9 2872 32.7 719 Midterm 36.8 46 41.9 33.3 39.6 2664 35 961 2 Modern Contraceptive Prevalence Rate (MCPR) among currently married women Baseline 42 48.2 45.2 41.4 44.3 2157 42.1 515 Midterm 42.5 52.7 47.1 40.1 45.8 2096 41 753 3 CPR for LAFP methods among all women Baseline 8.3 10.5 10.3 11.9 10.2 2872 9 719 Midterm 12.5 9.3 16 16 13.4 2664 9.3 961 4 CPR for LAFP methods among currently married women Baseline 10.5 13.1 11.9 14.8 12.5 2157 12.5 515 Midterm 14.4 9.1 18.2 20.5 15.4 2096 10.9 753 5 Unmet Need for Family Planning Baseline 33.2 26.2 34 31.7 31.2 2872 28.9 719 Midterm 33.3 26.9 34.6 30.1 31.2 2664 34 961 6 Use of modern contraception after birth (PPFP) Baseline 28.2 33 41.7 34.4 34.5 568 32.6 132 Midterm 35.6 44.6 48.8 30.2 40.3 484 38.5 182 7 Family planning counseling after birth (PPFP counseling) Baseline 21.5 25.7 39.1 39 31.9 568 30.3 132 Midterm 28.1 17.7 42.3 32.3 29.8 484 30.8 182 Increased numbers of healthy mothers-successful birth outcomes 8 The proportion of women who had their first ANC within the first 16 weeks of pregnancy Baseline 31.3 35.8 21.9 35.7 30.8 559 38 129 Midterm 57.6 59.2 34.5 73.9 55.3 468 51.8 168 9 Women who took Iron and Folic Acid supplement for at least 90 days during their last pregnancy Baseline 7 34.9 21.7 21.3 20.9 302 27.4 73 Midterm 20.9 25.5 28.4 30.6 26.6 320 29.1 117 10 Pregnant women who slept under ITN the previous night Baseline 24.4 46.3 35.7 43.3 37.9 195 49 49 Midterm 38.5 57.1 24.2 43.4 39.2 209 20.2 89 11 Baseline 53.4 55 59.5 60.4 57.3 564 57 568 Mid-term Performance Evaluation Report – Transform: PHC Activity Page 98 The proportion of women who had four or more ANC for their last birth in the last year Midterm 42.1 46.5 51.5 70.8 52 421 52.3 155 12 Women who received essential components of ANC (Essential components include: BP measured, blood and urine sample were taken, nutritional and danger sign counseled) Baseline 23.9 50.5 28.1 55.9 39.4 530 47.6 126 Midterm 21.5 46.7 39.8 46.9 38.2 435 36 161 13 Skilled Birth Attendance Baseline 50.3 59 63.2 85.7 66.4 568 67.4 132 Midterm 65.9 71.5 52 88.5 68.4 484 54.4 182 14 Postnatal Care for the mother within two days of birth Baseline 28.2 59.6 46.8 64.3 49.1 568 44.7 132 Midterm 45.2 57.7 52.8 66.7 54.8 484 42.9 182 15 % of women who delivered in a health facility and stayed at a health facility for at least 24 hrs. Baseline- not considered at baseline Midterm 30.3 38.3 57.6 54.1 44 334 46.5 99 Increased numbers of healthy newborns (birth to 28 days) 16 Early Postnatal Care for the newborn within two days of birth Baseline 25.5 46.8 42.3 50 40.8 568 31.1 132 Midterm 41.5 46.9 52.8 52.1 47.9 484 36.8 182 17 Postnatal Care for the newborn within seven days Baseline 27.5 46.8 44.2 50.6 42.1 568 32.6 132 Midterm 41.5 47.7 55.3 57.3 49.8 484 37.4 182 18 Newborns who received Essential Newborn Care (Vit K, TTC eye ointment, cord care with ointment - among births that occur in a health facility) Baseline 0 3.8 20.7 1.5 6.3 379 2.2 93 Midterm 2.2 13.9 38.6 12.8 15.4 357 11.9 101 19 Early initiation of breastfeeding (within 1 hour of birth) Baseline 79.2 78.7 69.2 80.1 77.6 393 64.9 93 Midterm 93.3 83.7 89.3 85.3 88.1 480 85.6 181 20 The proportion of infants 0–5 months who are fed exclusively with breast milk Baseline 59.7 44.7 66.7 61.8 59.4 219 75 60 Midterm 59.2 76.5 75.9 93 76.8 228 70.5 78 Child health (<5) 21 Full immunization Baseline 24.1 67.4 29.7 44.2 41.3 368 41.6 77 Midterm 43.8 48.8 34.5 51.2 44.4 372 43.5 161 22 Measles coverage Baseline 36.1 37 33 44.2 37.5 368 41.5 77 Midterm 66.1 50.4 72.4 80.6 67.4 372 65.3 161 23 Penta 3 coverage Baseline 37 58.4 37.9 57.4 47.7 368 55.8 77 Mid-term Performance Evaluation Report – Transform: PHC Activity Page 99 Midterm 53.7 67.5 52.9 63.1 58.6 372 57.5 161 24 Children with symptoms of ARI in the past two weeks Baseline 5.9 11.4 4.8 10.2 7.9 1755 6.6 380 Midterm 5.7 3.7 4.4 3.9 4.5 1,973 6.2 745 25 Children with Symptoms of ARI treated with antibiotics Baseline 34.6 28.6 25 50 36.2 138 32 25 Midterm 60 31.3 60.9 47.4 52.3 88 63 46 26 Diarrhea incidence in the past two weeks Baseline 8.1 9.6 9.5 8.6 8.9 1928 11.8 425 Midterm 12.2 15.4 14.8 6.9 12.3 1,973 12.1 745 27 Diarrhea treatment with ORS and Zinc Baseline 27.5 18.4 38 27.3 28.4 172 24 50 Midterm 28.1 41.8 29.9 20.6 31.4 242 25.6 90 28 Children under 5 who slept under ITN the previous night Baseline 20.1 43.6 24.8 42.9 32.2 1942 30.6 428 Midterm 38.9 44.8 25.9 31.6 35 1,973 31.7 745 29 Children 2-5 years who were given a drug for intestinal worms in the last six months Baseline 29.8 35.5 45.3 33.1 36.6 552 40 339 Midterm 44.9 37.4 45.6 52 45.5 1,157 41.9 422 30 Percentage of children aged 6-59 months who received vitamin A supplementation Baseline 37.2 43.5 48.7 48.7 44.5 766 33.3 171 Midterm 27.9 31.7 30.0 40.1 32.3 2874 34.0 1100 31 Percentage of children under five years who seek treatment in the first 24 hours of onset of fever among those who reported fever in the last two weeks Baseline 45.3 39.4 38.5 42.9 41.7 223 40.8 49 Midterm 34.2 41.3 36.1 34.8 36.8 283 31.9 91 SBCC 32 Percentage of audience who recall hearing or seeing a specific FP/RH message in the last few months Baseline 40.4 50.6 62.3 62.2 53.7 2872 60.4 719 Midterm 55.6 41.9 55.6 61.9 53.6 2664 47.7 961 33 Percentage of audience who recall hearing or seeing MNCH message/ information in the last few months Baseline 12.9 26.1 20.1 37.2 24 2859 26.8 714 Midterm 51.1 39.2 50.5 52.6 48.3 2664 41.4 961 Cross-cutting: 34 Proportion of households that have access to basic sanitation facility Baseline 3.3 6.2 6.3 7.9 5.9 2871 12.9 719 Midterm 5.5 4.5 8.8 14.5 8.3 2,664 5.6 961 Mid-term Performance Evaluation Report – Transform: PHC Activity Page 100 35 The proportion of households that have a handwashing facility with soap/ash and water Baseline 1 0.6 1.3 0.3 0.8 2872 1.7 719 Midterm 2.3 4.9 9.1 0.3 4.2 2,664 2.2 961 36 The proportion of households using water treatment technology Baseline 5 6.9 9.5 20.2 10.4 2872 7.4 719 Midterm 11 6.8 6.1 23 11.7 2,659 12.9 961 37 Women's participation in decisions regarding their own health Baseline 88.5 91.3 79.7 76 84.2 2138 84.2 507 Midterm 81.7 93.2 76.4 83.4 83.4 2,096 80.2 753 38 Women accompanied by their spouse during ANC visits for their last birth Baseline 69.2 68.3 52.6 60.4 62.2 518 59.2 125 Midterm 44.4 56.9 35.8 59.4 48.6 484 42.9 182 39 Women accompanied by their spouse during the birth of their last child Baseline 87.3 81 85.2 79.7 82.9 414 81.1 95 Midterm 93.5 80.2 88.6 77.7 84.6 357 89.1 101 40 Percentage of households that are enrolled in CBHI schemes Baseline 9.9 43.1 14.6 47.6 28.8 2871 30.2 719 Midterm 37.8 67.6 31.2 64.5 50.5 2,664 47.6 961 Mid-term Performance Evaluation Report – Transform: PHC Activity Page 101 Annex 2: Transform: PHC Three-Year (3) Resource Allocation/Expenditure (USD) Thematic area Transform PHC Three Year (3) Resource Allocation/ Actual expenditure (USD) Amhara Oromia SNNP Tigray Total Expendi￾ture, USD Average expenditu re per woreda # of indicators perform￾ance showed an increment Expendi￾ture, USD Average expenditur e per woreda # of indicators performanc e showed an increment Expendi￾ture, USD Average expenditu re per woreda # of indicators performan ce showed an increment Expendi￾ture, USD Average expendit ure per woreda # of indicators performance showed an increment Maternal health $16,383,0 89.12 $4,175,004. 35 45,879 2 $6,904,014.5 6 $42,617.37 4 $3,844,099. 19 $ 32,034.16 4 $1,459,97 1.02 $63,477.0 0 5 Newborn health $10,531,9 85.86 $2,683,931. 37 29,494 3 $4,438,295.0 8 $27,396.88 2 $2,471,206. 62 $ 20,593.39 3 $938,552. 80 $40,806.6 4 3 Child health $8,191,54 4.56 $2,087,502. 17 22,940 6 $3,452,007.2 8 $21,308.69 5 $1,922,049. 60 $ 16,017.08 5 $729,985. 51 $31,738.5 0 7 Family planning/RH $18,138,4 20.10 $4,622,326. 24 50,795 2 $7,643,730.4 1 $47,183.52 2 $4,255,966. 96 $ 35,466.39 3 $1,616,39 6.48 $70,278.1 1 1 Malaria $5,265,99 2.93 $1,341,965. 68 14,747 2 $2,219,147.5 4 $13,698.44 2 $1,235,603. 31 $ 10,296.69 1 $469,276. 40 $20,403.3 2 1 Gender $2,223,83 2.69 $563,692.2 6 6,194 1 $942,381.59 $5,817.17 1 $523,702.53 $ 4,364.19 1 $194,056. 31 $8,437.23 1 Total $60,734,8 65.27 $15,474,42 2.07 170,049 16 $25,599,576 .46 $158,022.0 8 16 $14,252,628 .22 $ 118,771.90 17 $5,408,23 8.52 $235,140. 81 18 Mid-term Performance Evaluation Report – Transform: PHC Activity Page 102 Annex 3: Adjusted Odds Ratios and 95% CI from the Logistic Regression Model for the Determinants of FP Use Among Currently Married Women, Transform Mid-term Survey (2019) Indicators CPR (n=2664; event=1054) LA methods (n=2664; event=357) Unmet need (n=2664; event=830) OR 95% CI OR 95% CI OR 95% CI Region Amhara (Ref) 1.0 1.0 1.0 Oromia 0.63 0.48−0.82 1.33 0.92−1.92 1.17 0.88−1.56 SNNP 0.72 0.51−1.01 1.91 1.34−2.72 0.86 0.59−1.24 Tigray 0.55 0.43−0.71 2.05 1.45−2.89 1.17 0.89−1.52 Age 15 - 19 (Ref) 1.0 1.0 1.0 20 – 24 0.74 0.38−1.46 0.54 0.26−1.13 0.43 0.22−0.83 25 – 29 0.56 0.29−1.09 0.39 0.19−0.82 0.33 0.17−0.63 30 – 34 0.49 0.24−0.98 0.33 0.15−0.72 0.21 0.11−0.42 35 – 39 0.43 0.21−0.86 0.20 0.09−0.44 0.13 0.06−0.26 40+ 0.27 0.13−0.55 0.23 0.10−0.51 0.0 0.01−0.06 Education No education (Ref) 1.0 1.0 Primary 1.24 1.01−1.52 0.95 0.76−1.18 Secondary + 1.36 0.98−1.87 1.10 0.79−1.54 Religion Muslim (Ref) 1.0 1.0 Orthodox 1.13 0.87−1.45 1.17 0.89−1.54 Protestant and others 1.42 1.01−1.99 1.82 1.26−2.62 Child ever born 1 - 2 (Ref) 1.0 1.0 1.0 3 – 4 1.25 0.96−1.63 1.25 0.88−1.78 2.19 1.64−2.91 5 – 6 1.41 1.02−1.96 1.66 1.1−2.52 2.77 1.92−3.99 7+ 1.16 0.78−1.72 1.15 0.69−1.92 4.77 3.06−7.43 The ideal number of children 0 - 3 (Ref) 1.0 1.0 4 – 5 0.87 0.66−1.15 1.00 0.74−1.36 6 – 9 0.69 0.5−0.95 1.10 0.78−1.56 10+ 0.43 0.26−0.70 0.83 0.5−1.38 RH/FP message No (Ref) 1.0 Yes 1.41 1.18−1.68 Non-inclusion of zero in the 95% CI indicates statistical significance. Shaded cells indicate non-significant variables in the bivariate analysis. Mid-term Performance Evaluation Report – Transform: PHC Activity Page 103 Annex 4: Adjusted Odds Ratios and 95% CI from the Logistic Regression Model for the Determinants of Maternal Health Services Utilization, Transform Mid-term Survey (2019) Indicators Early ANC (n=468; event=259) ANC 4+ (n=421; events=219) SBA (n=484; unmet=331) OR 95% CI OR 95% CI OR 95% CI Region Amhara (Ref) 1.0 1.0 1.0 Oromia 1.00 0.51−1.98 0.78 0.4−1.52 1.74 0.31−9.83 SNNP 0.45 0.19−1.09 4.25 1.67−10.84 0.52 0.08−3.47 Tigray 2.42 1.27−4.63 3.17 1.74−5.78 1.71 0.48−6.08 Age 15 - 19 (Ref) 1.0 20 – 24 0.52 0.16−1.73 25 – 29 0.68 0.2−2.34 30 – 34 0.83 0.22−3.14 35 – 39 1.10 0.28−4.32 40+ 0.88 0.19−4.2 Education No education (Ref) 1.0 1.0 1.0 Primary 0.94 0.57−1.55 1.12 0.7−1.77 0.26 0.07−0.97 Secondary + 1.53 0.73−3.19 2.22 1.19−4.12 0.54 0.08−3.55 Paid work No (Ref) 1.0 1.0 1.0 Yes 1.66 0.99−2.78 1.57 0.95−2.59 2.7 0.73−9.66 Religion Muslim (Ref) 1.0 1.0 Orthodox 0.45 0.24−0.85 0.45 0.24−0.85 Protestant and others 0.62 0.27−1.44 0.12 0.05−0.32 Child ever born 1 - 2 (Ref) 1.0 1.0 3 – 4 0.51 0.27−0.96 0.26 0.07−0.98 5 – 6 0.35 0.15−0.82 0.07 0.01−0.58 7+ 0.40 0.14−1.11 0.09 0.01−1.17 The ideal number of children 0 - 3 (Ref) 1.0 1.0 4 – 5 1.37 0.68−2.75 2.48 0.57−10.7 6 – 9 0.76 0.35−1.69 3.96 0.69−22.64 10+ 0.73 0.23−2.33 MCH message Mid-term Performance Evaluation Report – Transform: PHC Activity Page 104 No (Ref) 1.0 1.0 1.0 Yes 1.39 0.91−2.11 1.29 0.84−1.97 1.83 0.69−4.86 Spouse accompany ANC No (Ref) 1.0 1.0 1.0 Yes 2.23 1.46−3.41 1.5 0.99−2.27 0.85 0.3−2.43 Spouse accompany delivery No (Ref) 1.0 Yes 1.03 0.26−4.14 Women's participation in health care decisions No (Ref) 1.0 Yes 0.86 0.19−3.95 CBHI No (Ref) 1.0 1.0 Yes 1.01 0.64−1.58 6.66 1.81−24.53 Non-inclusion of zero in the 95% CI indicates statistical significance. Shaded cells indicate non-significant variables in the bivariate analysis. Mid-term Performance Evaluation Report – Transform: PHC Activity Page 105 Annex 5: Adjusted Odds Ratios (OR) and 95% CI from the Logistic Regression Model for the Determinants of Child Health Services Utilization, Transform Mid-term Survey (2019), PHC Intervention Woredas. Indicators Diarrhea treatment (n=242; event=76) Treatment ARI (n=88; event=46) Fever treatment (n=283; event=104) Full Immunization (n=372; event=165) OR 95% CI OR 95% CI OR 95% CI OR 95% CI Age 15 - 24 (Ref) 1.0 1.0 25 – 29 0.24 0.02 - 2.71 0.77 0.35−1.68 30 – 34 0.24 0.02 - 2.74 1.01 0.4−2.55 35 - 39 0.06 0.0 - 0.77 0.44 0.14−1.4 40+ 0.10 0.0 - 1.13 2.06 0.56−7.65 Education No education (Ref) 1.0 1.0 Primary 0.76 0.4−1.43 1.2 0.76−1.91 Secondary + 1.33 0.52−3.43 1.87 0.96−3.66 Paid work No (Ref) 1.0 Yes 1.65 1.03−2.64 Religion Muslim (Ref) 1.0 1.0 1.0 1.0 Orthodox 0.44 0.23 - 0.87 0.25 0.07 - 0.98 0.35 0.18−0.71 0.71 0.43−1.18 Protestant and others 0.55 0.25 - 1.19 0.55 0.11 - 2.67 0.56 0.25−1.22 0.43 0.22−0.83 MCH message No (Ref) 1.0 1.0 Yes 1.29 0.74−2.23 1.48 0.96−2.29 Full immunization No (Ref) Yes CBHI No (Ref) 1.0 1.0 Yes 0.42 0.16 - 1.10 1.55 1−2.41 Mid-term Performance Evaluation Report – Transform: PHC Activity Page 106 Annex 6: Institutional Deliveries Before and After the Installation of a Solar Suitcase (Source Transform: PHC Annual Report) Region Facility Name Monthly delivery at the time of installation Monthly delivery after six months of installation Amhara Ahtiy HC 45 101 Segnogebeya HC 12 26 Berera Getera HC 18 50 Chatwarka HC 30 94 Oromia Sede HC 30 170 Beleti HC 35 64 Challo HC 25 78 Harsu Wokilo HC 26 99 SNNP Zhazh HC 25 49 Yadota HC 71 112 Jeba HC 10 20 Kibish HC 5 15 Tigray Debre Genet HC 10 33 Egela HC 39 80 Adi Nebri Eid HC 44 81 Hadnet HC 3 32 Mid-term Performance Evaluation Report – Transform: PHC Activity Page 107 Annex 7: Statement of Work (SOW) and Evaluations Protocol for Mid-Term Performance Evaluations of Transform Implementing Mechanisms MELA-Midterm Evaluations Design _Revised SOW-31_05_19.docx Annex 8: Misterm Evaluation tools Household Survey Questionnaire_Final.docx Health Facility Assessment tool for Health Centers and Primary Hospitals_10182019.docx Health Facility Assessment tool for Health Posts_Final.docx Annex 8: List of Relevant Documents Reviewed Annex 9: List of Key Informants Mid-term Performance Evaluation Report – Transform: PHC Activity Page 108 U.S. Agency for International Development USAID/Ethiopia Entoto Road Addis Ababa, Ethiopia