i | P a g e | P a g e
Submission Date: August 19, 2022
Revised Version: September 9, 2022
Final Version: November 1, 2022
USAID/Ethiopia
Submitted to:
Gebeyehu Abelti, COR
Submitted by: 
The Mitchell Group, Inc. (TMG)
Contract Number: OAA-I-15-00028 Task Order No.: AID-663-TO-17-00001
Transform MELA Activity, Addis Ababa, Ethiopia
Michael Midling, PhD, Chief of Party/Principal Investigator
Email: mmidling@ethiopiatmela.com 
Phone: +251-986356914
Principal Contacts:
Abi Fasosin, Senior Advisor 
Email: abif@the-mitchellgroup.com
Phone: +001-202-557-9412 (Direct)
This document was produced for review by the United States Agency for International Development Ethiopia (USAID/Ethiopia). It was prepared by The 
Mitchell Group, Inc. (TMG) for USAID/Ethiopia for the “Transform Monitoring, Evaluation, Learning, and Adapting (Transform MELA) Activity."
TRANSFORM MONITORING, EVALUATION, LEARNING AND ADAPTING 
(TRANSFORM MELA) ACTIVITY
FINAL PERFORMANCE EVALUATION
TRANSFORM: PRIMARY
HEALTH CARE (PHC) ACTIVITY
ACTIVITY INFORMATION 
Activity Title TRANSFORM MONITORING, EVALUATION, LEARNING 
AND ADAPTING (TRANSFORM MELA): ACTIVITY
Contract/Task Order Numbers
Contract No.: OAA-I-15-00028
Task Order No.: AID-663-TO-17-00001
Name of Prime Implementing Partner
The Mitchell Group, Inc. (TMG)
1816 11th Street, NW, Washington, DC 20001
Tel: 202-745-1919
Activity Start Date March 7, 2017
Activity End Date
August 31, 2022
Period of Planning and Conducting the 
Evaluations 
September 1, 2021 – August 31, 2022
TABLE OF CONTENTS
ACRONYMS AND ABBREVIATIONS .................................................................................i
ABSTRACT.............................................................................................................................1
1) EXECUTIVE SUMMARY..............................................................................................1
2) BACKGROUND AND CONTEXT..............................................................................5
Activity Background 5
3) EVALUATION PURPOSE AND EVALUATION QUESTIONS ..............................6
Evaluation Purpose 6
Evaluation Questions 7
Audience and Intended Uses 7
4) EVALUATION METHODS, ETHICAL CONSIDERATIONS AND LIMITATIONS
.........................................................................................................................................7
4.1. Sampling of Woredas 7
4.2. Sampling of Households 8
4.3 Sampling For the Health Facility survey 8
4.4 Selection of Key Informants 9
4.5. Data Collection Methods 9
4.6 Data Analysis 10
4.6 Ethical considerations 11
4.7 Limitations 12
5) FINDINGS....................................................................................................................12
5.1 How effective were Transform approaches in contributing to improving MNCH/FP 
outcomes? 13
5.1.1 Were Transform activities implemented as planned? If not, why not?...............14
5.1.2 To what extent did the activity improve MNCH/PH outcomes and reduce 
inequities in different groups and intervention areas?............................................................14
Family planning performance 15
Maternal health performance 17
Newborn health performance 19
Child health performance 20
5.1.3 What interventions are replicable or can be scaled up in different settings?..29
5.1.4 What is the effect of context (conflict, COVID-19 pandemic, internal 
displacement) on the Transform interventions and achievements?................................30
5.2 How and to what extent did transform program activities facilitate local 
ownership, sustainability and coherence? 32
5.2.1 What mechanisms are in place to ensure government ownership of 
Transform health system interventions?.......................................................................................32
5.2.2 To what extent did the government staff perceive that the approaches 
established by the project will be sustained by the government and continue to 
sustain improvements?...........................................................................................................................34
5.2.3 To what extent did the project improve community involvement and its role 
in accountability, transparency in decision-making, and advocacy for community 
health needs?................................................................................................................................................35
5.2.4 To what extent did the project improve community engagement for 
advocacy and health service oversight? What capacity for community engagement 
did Transform build?................................................................................................................................35
5.2.5 How does TPHC coordinate with other development partners and their 
activities?........................................................................................................................................................36
5.3 How did the activity consider gender dynamics in implementation? 37
5.3.1. To what extent did the government staff/people perceive that the project 
established processes are functional to track and address gender and inequitable 
health outcomes?.......................................................................................................................................39
5.3.2. What gender equity results did Transform Program Activities’ interventions 
achieve?...........................................................................................................................................................40
5.3.3. What gaps exist in the current USAID/ Transform intervention 
design/programming in terms of addressing the key gender issues in the targeted 
regions of Ethiopia?...................................................................................................................................40
5.4 Indicator Trends at Baseline, Midline, and Endline 41
5.5 How and to what extent has the Transform IP been able to adapt their 
interventions about learnings and new evidence? 41
5.5.2. How did respondents perceive that these adaptations contributed to 
MNCH/FP outcomes?..............................................................................................................................44
5.5.3. What lessons were learned for project future design, management, and 
monitoring?...................................................................................................................................................44
6) CONCLUSIONS..........................................................................................................45
7) RECOMMENDATIONS..............................................................................................46
ANNEXES.............................................................................................................................50
ANNEX 1. EVALUATION QUESTIONS AND DATA SOURCES.................................50
ANNEX 2. DETAILED EVALUATION METHODS 52
ANNEX 3. DATA ANALYSIS 56
ANNEX 4. PHC TRANSFORM ACTIVITY RESULTS FRAMEWORK 57
ANNEX 5. GLOSSARY OF TERMS ...................................................................................58
List of Table and Figures
List of Tables
Table 1: Coverage of Transform PHC Activity by Region................................................................................6
Table 4: Summary of Activity Performance by Key Performance Indicators.............................................13
Table 5: Family planning program performance at baseline and endline by region. .................................15
Table 6: Maternal health program performance outcomes at baseline and endline.................................18
Table 7: Newborn health performance at baseline and endline by region.................................................19
Table 8: Baseline and endline comparison of child health indicators by region.........................................21
Table 9: Percentage of sampled health facilities that reported performing each BEmONC Signal 
function on the day of the survey.........................................................................................................................23
Table 10: Percentage of key adolescent and youth health interventions reported to be provided in 
sampled facilities........................................................................................................................................................24
Table 11: Sub-grants by health facility and year in Kembata and Halaba Zone (in Ethiopian Birr –
ETB). ............................................................................................................................................................................28
Table 12: Baseline and endline comparison of gender indicators by region...............................................38
List of Figures
Figure 1: Family planning program performance at baseline and endline, and LOA target (Top) and 
performance trend (bottom) in Transform PHC intervention areas...........................................................15
Figure 2: Reported stockouts of FP commodities in transform PHC intervention and non-transform 
intervention three months prior to the survey.................................................................................................16
Figure 3: Baseline and endline comparison of FP services by intervention and non-intervention areas.
.......................................................................................................................................................................................16
Figure 4: Maternal health outcomes at baseline and endline with LOA targets (Top) and performance 
trend (Bottom) in Transform PHC intervention areas. ..................................................................................17
Figure 6: SBA, early PNC, and ITN outcomes at baseline and endline by Transform intervention and 
non-Transform intervention areas. ......................................................................................................................18
Figure 7: Newborn health outcomes at baseline and endline with LOA targets (Top) and 
performance trend (bottom) in Transform PHC intervention areas...........................................................19
Figure 8: Newborn health performance outcomes at baseline and endline by transform intervention 
and non-transform intervention areas.................................................................................................................20
Figure 9: Skill lab in SNNP......................................................................................................................................20
Figure 10: Expanded Program on Immunization and Vitamin A supplementation at baseline and 
endline with LOA targets (top) and full immunization performance trend (bottom) in Transform 
PHC intervention areas...........................................................................................................................................20
Figure 11: Baseline, endline and LOA comparison of access to preventive and curative childhood 
illness treatment (top) and performance trend (bottom) in Transform intervention areas. .................21
Figure 12: Child immunization performance outcomes at baseline and endline by intervention and 
non-intervention areas. ...........................................................................................................................................22
Figure 13: Child health performance outcomes at baseline and endline by intervention and non￾intervention areas.....................................................................................................................................................22
Figure 14: Comparison of available adolescent and youth health services in intervention and non￾intervention areas.....................................................................................................................................................24
Figure 15: Number of health workers who received training provided by the transform program by 
thematic area (Source: TPHC training report)..................................................................................................25
Figure 16: CBHI enrollment at baseline and endline by region. ....................................................................28
Figure 17: Health facility performance in Transform PHC intervention areas..........................................29
Figure 18: Baseline and endline comparison of selected gender-related indicators.................................37
Figure 19: Percent of Transform PHC intervention and non-transform intervention health facilities 
providing specific GBV services at the time of the survey. ............................................................................39
Figure 20: Indicator trends at baseline, midline, and endline .........................................................................41
i | P a g e
ACRONYMS AND ABBREVIATIONS
ANC Antenatal Care
ARI Acute Respiratory Infection 
BEmONC Basic Emergency Obstetric and Newborn Care
CBHI Community-Based Health Insurance
CBNC Community Based Newborn Care
COR Contracting Officer’s Representatives
CPR Contraceptive Prevalence Rate 
CRC Compassionate and Respectful Care
CSC Community Score Card
DHSI2 District Health 
EHCRIG Ethiopian Health Center Reform Implementation Guidelines 
EHSIG Ethiopian Hospital Services Transformation Guidelines
EPCMD Ending Preventable Child and Maternal Deaths
EPHI Ethiopian Public Health Institute
EQs Evaluation Questions
FP Family Planning
GBV Gender Based Violence
GIA Greatest Impact Analysis
GIS Geographic Information Systems
GOE Government of Ethiopia
HC Health Center
HDR Health in Developing Regions
HF Health Facility
HH Household
HMIS Health Management Information System
HP Health Post
HSTP Health Sector Transformation Plan 
iCCM Integrated Community Case Management
ICMNCI Integrated Community management of Newborn and Childhood Illnesses
IFS iron-folic acid supplementation 
IMNCI Integrated management of Newborn and Childhood Illnesses
IP Implementing Partner
IRB Internal Review Board
ISS Integrated supportive supervision
KI Key Informant
KII Key Informant Interview
KPI Key Performance Indicators
LAFP Long-Acting Family Planning
LMG Leadership, management, and governance 
LOA Life of Activity
MELA Monitoring, Evaluation, Learning, and Adapting
M&E Monitoring and Evaluation
L6Ms Last 6 Months
L2Ws Last 2 Weeks
MCPR Modern Contraceptive Prevalence Rate
MCV Measles coverage
MNCH/FP Maternal, Newborn, Child Health, and Family Planning
MOH Ministry of Health
NGO Non-Governmental Organization
NICU Newborn intensive care unit
ii | P a g e
OR Operational Research
ORS Oral Rehydration Salts
PCMD Preventing Child and Maternal Deaths
PHC Primary Health Care
PHCU Primary Health Care Unit
PPFP Post-Partum Family Planning 
PMP Performance Management Plan
RH Reproductive Health
RHB Regional Health Bureau
RMNCH Reproductive, Maternal, Newborn, and Child Health 
SBA Skilled Birth Attendant
SDG Sustainable Development Goal
SNNPR Southern Nations, Nationalities, and People’s Region 
TMG The Mitchell Group, Inc.
TPHC Transform Primary Health Care
USAID United States Agency for International Development
USG United States Government
WHO World Health Organization
WorHO Woreda (district) Health Office
WMS Woreda Management Standards 
ZHD Zonal Health Department
1 | P a g e
ABSTRACT
The final performance evaluation of the Transform Primary Health Care (TPHC) activity was 
conducted to assess achievement of results and inform the design of future activities. The evaluation 
was designed to answer questions of effectiveness and quality of the continuum of care, ownership, 
sustainability and coherence, gender dynamics and learning and adaptive management. The results 
showed that the activity implemented effective interventions such as woreda twinning, FP program 
planning, skills labs and V-Scans, leadership management training, sub-granting, coaching and 
mentoring, “Her Space” program, community-based health insurance (CBHI) enrollment, etc. Most 
health outcomes have declined after the mid-term and marked regional difference persist, and the 
activity did not meet most of its end of activity targets, largely due to contextual factors, especially 
conflict and Covid, and systemic weaknesses beyond the scope of the activity (stockouts of 
commodities, high staff turnover, etc.). However, declines were worse in non-Transform intervention 
areas, indicating that the Transform program likely prevented greater deterioration as measured by 
key performance indicators.” The findings also indicate the need for concerted and continued health 
systems strengthening interventions that support the Ministry of Health (MOH) to improve human 
resource management and retention, reinforce coordination of all development partners and 
revitalize and restore community-based health programs and “last mile” logistics. Gains were 
achieved in supporting twinned woredas, but further work is necessary to implement improvements 
and conduct operational research to sort out bottlenecks in service delivery and ensure the 
continuum of care. USAID/Ethiopia was recommended to continue supporting community-based 
behavior change interventions and to revisit facility-based target setting to increase the number of 
woredas that demonstrate to sustainably deliver quality MNCH/FP services through an efficient 
network of hospitals, health centers and restored health posts in alignment with health system goals. 
1) EXECUTIVE SUMMARY
Background
The final performance evaluation of the Transform Primary Health Care (TPHC) activity was 
conducted from May to August 2022. The purpose of the evaluation was to measure achievements 
of Transform PHC in terms of its Activity-level results framework, to explore underlying reasons for 
achievement, assess inhibiting factors that caused non-achievement of anticipated results, as well as 
to review intervention approaches used by the implementing partner (IP). The performance 
evaluation contributed to evaluate the impact of the entire Transform program, which included two 
other PHC activities and the Transform Monitoring, Evaluation, Learning and Adaptation (MELA) that 
conducted the evaluations, and inform USAID/Ethiopia’s decisions related to future investment in 
supporting MNCH/FP interventions. 
TPHC was designed to improve PHC outcomes in almost all zones and 434 intervention woredas in 
four regions: Amhara, Oromia, Southern Nations, Nationalities and People’s Region (SNNPR) and 
Tigray. Due to recent unrest in Tigray, the performance evaluation did not include this region. TPHC 
was to achieve four intermediate results:
1) Improved management and performance of health systems. 
2) Increased sustainable quality of service delivery across the primary health care unit’s 
continuum of care.
3) Improved household and community health practices and health-seeking behavior. 
4) Enhanced program learning, and impact policy and programming related to preventing child 
and maternal deaths (PCMD). 
Evaluation Methodology
The evaluation used a pre-post cross-sectional design that compared endline performance to baseline 
and midline, regional and intervention and non-intervention areas performance to answer four 
questions:
2 | P a g e
1. How effective were the Transform PHC Activity approaches in contributing to improving 
maternal, newborn, child health, and family planning (MNCH/FP) outcomes? What were the 
drivers of the observed changes? What constraints affected the achievements?
2. How and to what extent did Transform PHC Activity facilitate local ownership, sustainability, 
and coherence? 
3. How did Transform PHC consider gender dynamics in activity design and implementation?
4. How and to what extent has the Transform IP been able to adapt their interventions in 
response to learnings and new evidence?
Quantitative and qualitative mixed data collection methods used in the evaluation included: a 
household survey of 2,703 randomly selected households in 80 intervention woredas and 692 
households in 23 non-intervention woredas; a health facility survey of 181 health facilities in 
intervention areas and 49 facilities in non-intervention areas; review of activity documents developed 
in the life of the activity; and key informant interviews with selected stakeholders at USAID/Ethiopia, 
Ministry of Health (MOH), Regional Health Bureaus (RHBs), Zonal Health Departments (ZHD) and 
Woreda Health Offices (WorHO). 
Performance was measured in terms of the activity’s results in the following technical areas: family 
planning, maternal health, newborn health, child health and immunization, and adolescent and youth 
health. Activity interventions were evaluated for their effectiveness to improve outcomes as well: 
communication of MNCH/FP messages, behavior change and treatment seeking practices, women’s 
participation in decision-making, community-based health insurance, woreda twinning and quality 
improvement, leadership, management and governance training and coaching, and use of information 
and lessons to adapt and improve. 
Findings
Overall performance declined after midterm for most performance outcomes and few life of activity 
(LOA) targets were met. Several factors explained the decline, most notably conflict and Covid-19. 
Performance is presented below by technical area. 
● Family Planning. Long-acting FP was the only outcome that increased slightly from baseline 
(from 11.8 to 12.7%) and whose LOA target was met. After having increased at midline, modern 
contraceptive prevalence rate (MCPR) and postpartum family planning (PPFP) indicators declined
in both intervention and non-intervention areas, though no difference was statistically significant. 
There were differences by region but no significant differences between intervention and non￾intervention woredas. Interviewed women reported being turned away from facilities due to lack 
of FP commodities or of trained providers. They also reported wanting to have more children, 
religious restrictions, fear of side effects and partner refusal as reasons for non-use. 
● Maternal Health. Except for early antenatal care (ANC) and skilled birth attendance (SBA), the 
performance of all the other maternal health indicators declined, and no LOA target was met. 
ANC4+ has declined from baseline and even more after midline indicating the difficulties in 
ensuring continuity of care in the intervention woredas. There were differences by region, with 
Oromia and SNNP showing improvement in the implementation of essential components of 
ANC. Early ANC improved in both intervention and non-intervention areas. SBA increased in 
intervention areas and declined in non-intervention. All other indicators declined in both. No 
difference was statistically significant. KIs reported the following reasons for the decline: conflict, 
COVID, weak community health program, high leadership turnover, long distance to facilities, 
lack of counseling, and mothers’ beliefs they do not need to go to a health facility if they feel well. 
● Newborn Health. All the four performance indicators (essential newborn care, early PNC for 
newborns, exclusive breast feeding and early breastfeeding) improved from baseline, but when 
compared to midline, only exclusive breastfeeding improved from midline, while other indicators 
declined. No LOA targets were met, though. Regional differences showed Amhara had the lowest 
performance. Newborn health outcomes have increased in both intervention and non-
3 | P a g e
intervention areas, except exclusive breastfeeding that declined in non-Transform intervention 
areas. This difference was statistically significant (p = 0.01). 
● BEmONC. More health facilities in intervention areas provide all Basic Emergency Obstetric and 
Newborn Care (BEmONC) signal functions than health facilities in non-intervention areas. Except 
for parenteral oxytocin and antibiotics administration, other BEmONC signal functions are 
provided in more health facilities in intervention than non-intervention areas
● Child Health. Measles coverage and Vitamin A supplementation increased significantly (p<0.001) 
in both intervention and non-intervention areas. However, full vaccination has significantly 
declined (p<0.001) to 26.8% in intervention areas, even more after midterm. No LOA targets 
were met except for Vitamin A supplementation in all regions except for Penta 3 that increased 
in Oromia and SNNP and markedly decreased more than 20 percentage points in Amhara. 
● Deworming and ARI treatment with antibiotics have increased significantly (p<0.001) in both 
intervention and non-intervention areas. Fever treatment within 24 hours of onset declined 
significantly (p<0.001). Diarrhea treatment decreased in SNNP which is a concern because its 
incidence increased from baseline. Only the deworming LOA target was met, which had a marked 
increase after midterm. 
● Gender. Overall, higher percentage of health facilities in Transform PHC intervention areas 
provide different types of gender-based violence (GBV) services than non-Transform intervention 
areas. The proportion of women accompanied by their spouses during delivery increased slightly, 
while being accompanied for ANC declined. Women participation in health care decision making 
and women accompanied by their spouses during ANC showed a statistically significant decline 
(p<0.001) from baseline and more when compared to the midterm. No LOA targets were met. 
There were no statistically significant differences between intervention and non-intervention 
areas. 
● Adolescent and Youth Health. Overall, a higher proportion of health facilities intervention 
areas provide adolescent and youth friendly services than non-intervention areas. About one out 
of four surveyed facilities in intervention areas provide all adolescent and youth friendly service 
components, while only one in ten did in non-intervention areas.
● Community-based health insurance (CBHI). Households’ enrollment in CBHI significantly 
increased (P<0.001) and the LOA target was surpassed at endline. Enrollment increased in both 
intervention and non-intervention areas and there was no statistically significant difference. 
● Quality improvement. At the endline, 59.2% of the sampled facilities had a performance score 
of greater than 80% as per the Ethiopian Hospital Services Transformation Guidelines/Ethiopian 
Health Center Reform Implementation Guidelines (EHSTG/EHCRIG) and implementation of 
quality improvement plan of health facilities was 74.8%, which exceeded the respective LOA 
targets. 
● Ownership, Sustainability and Coherence. TPHC was reported to have improved ownership 
through several interventions, namely leadership, management and governance (LMG) training, 
quality improvement interventions; improved planning and budgeting, especially for family 
planning; and woreda twinning and sub-granting. 
● Learning and Use of Evidence. TPHC implemented the midterm evaluation recommendations 
and conducted research to inform policies and intervention design and implementation. “For 
example, the activity conducted research to integrate post-partum FP services in maternity 
waiting homes and explored ways to improve quality of care.”
Conclusions 
Contextual factors, particularly conflict and Covid-19 affected the capacity of the activity to achieve 
its LOA targets, as evidenced by the decline in achievements since the midterm. Despite these 
challenges, the final evaluation demonstrated that TPHC has improved early ANC and skilled birth 
attendance, long-acting FP, measles vaccination and vitamin A supplementation. It also established 
innovative processes, such as twinning woredas and FP program planning; helped to improve the 
functioning of regional public health systems and programs; and strengthened the leadership, 
management, and governance skills of staff in selected woredas and facilities. TPHC also 
4 | P a g e
demonstrated an iterative learning program that, among other results, helped expand CBHI and 
inform policy and programming. 
The Activity achieved these results through a combination of technical support interventions to the 
public health system. The interventions included: establishing skill labs; provision of on-the-job 
training, mentoring, and coaching; post training follow-up visits and supervision; provision of 
equipment and supplies; and sub-grants. TPHC’s technical support was reported to have improved 
quality and thereby increased coverage of high impact health interventions. However, despite a wide 
range of such interventions, lack of optimum quality is still a roadblock to the consistent delivery of 
MNCH/FP services. As described below, and likely due to contextual headwinds, household and 
community health practices and health-seeking behaviors declined, and continuity of MNCH/FP 
services remains a challenge. 
Gender outcomes were generally positive but require further efforts. As a positive outcome, the 
availability of post GBV interventions increased at health facilities supported by intervention when 
compared to non-intervention areas. The activity was also able to increase the availability and range 
of adolescent and youth health services. However, the utilization rate and impact of such services on 
gender inequalities, such as early marriage and early pregnancy, requires further study. 
Recommendations 
Recommendations for the Ministry of Health
▪ Reinforce coordination of development partners to effectively implement woreda 
transformation.
▪ Revitalize community health programs. 
▪ Assess woreda and health facility twinning partnership bottlenecks and make path correction 
measures before scale-up. 
▪ Integrate FP counseling across all contacts in the continuum of care.
▪ Commission supplemental operational research to explore the underlying factors contributing to 
the decline in performance of selected key performance indicators, such as ANC4+, which has 
been declining since baseline. 
▪ Add and retain additional health facility personnel to ensure consistent provision of services, 
especially in difficult-to-reach areas. 
Recommendations for USAID
▪ Continue support for innovative approaches on social and behavior communication change to 
bring sustained behavioral change within MNCH/FP practices.
▪ Revisit target setting methodology and assumptions and ensure performance is monitored 
continuously by a third party for course correction and target adjustments. 
▪ Reinforce leadership capacity building.
▪ Develop a dual health system strengthening strategy that continues support for the Health Sector 
Transformation Plan and implements a recovery strategy to build resilience in conflict zones. 
▪ Strengthen support for health posts and health centers with additional supplies and equipment.
▪ Improve ownership, sustainability, and coherence by providing specific metrics to progressively 
demonstrate ownership by woredas and facilities. 
▪ Continue support for participatory learning and adapting to address ongoing challenges within 
the health system.
Recommendations for the current TPHC activity
5 | P a g e
Given that this is the final evaluation of the TPHC activity, the recommendation to the IP is to ensure 
the effective transfer and handover of its research reports, tools and training manuals to all its GOE 
counterparts. 
2) BACKGROUND AND CONTEXT
This is the report of the final performance evaluation of the Primary Health Care Transform Activity 
(Transform PHC). In 2017, USAID/Ethiopia launched the Transform Program, a five-year (2017–
2021) integrated reproductive, maternal, newborn, and child health (RMNCH) and water, sanitation 
and hygiene (WASH) program to reduce maternal and child morbidity and mortality in Ethiopia. The 
Transform Program operated across ten regions in Ethiopia and comprised three Activities: 
Transform PHC, Transform Health in Developing Regions (HDR), and Transform Water, Sanitation, 
and Hygiene (WASH).1 
In 2017, USAID/Ethiopia also awarded a contract to The Mitchell Group, Inc. (TMG) to implement 
the Transform Monitoring, Evaluation, Learning, and Adapting (MELA) Activity. The role of Transform 
MELA was to provide and synthesize high-quality Monitoring and Evaluation (M&E) data for the 
activities, USAID/Ethiopia, and the Ministry of Health (MOH) of Ethiopia and to guide them in learning 
and adaptive health system management. As part of its contract, Transform MELA also conducted 
the baseline, midline and now the final performance evaluation of the Transform PHC and Transform 
HDR Activities, the Transform Program Impact Evaluation, and the development of selected case 
studies to document important lessons learned that highlight the contribution of the Transform 
program. The evaluations took place from May to August 2022. The WASH activity was not included 
in the final evaluations. Annex 5 presents a glossary of the definitions used in all the Transform 
evaluations. This report outlines the objectives and methods to conduct the final performance 
evaluation of the Transform PHC activity, and it presents the findings, conclusions, and 
recommendations for sustaining current performance and for the future improvement of Primary 
Health Care (PHC) programs in Ethiopia. 
ACTIVITY BACKGROUND
The Transform PHC (TPHC) Activity was implemented by Pathfinder International (prime), John 
Snow International, Encompass, Abt. Associate, Ethiopian Midwife Association, and Malaria 
Consortium. TPHC aimed to contribute to preventing child and maternal deaths (PCMD) through 
supporting the implementation of the Health Sector Transformation Plan (HSTP)-Government of 
Ethiopia (GOE) at different levels of the health system. Annex 4 presents the results framework for 
this activity and reflects the theory of change that the Activity was designed to implement. By 
supporting the GOE’s strategic initiatives, TPHC aimed to achieve four high-level intermediate results 
(IRs):
1) Improved management and performance of health systems. 
2) Increased sustainable quality of service delivery across the primary health care unit’s 
(PHCU) continuum of care.
3) Improved household (HH) and community health practices and health-seeking behavior. 
4) Enhanced program learning, and impact policy and programming related to PCMD. 
Maternal newborn, child health and family planning (MNCH/FP), reproductive health (RH), and 
malaria represented the primary intervention areas of Transform PHC. The activity operated in five 
major regions in the country– Amhara, Oromia, Southern Nations, Nationalities, and People (SNNP), 
1 Beginning from September 2021, the Transform program is being implemented in ten regions. This 
evaluation however will not include a separate analysis for the newly formed South-West Ethiopia region. 
6 | P a g e
Sidama, and Tigray – and targeted a total of 434 woredas over the five-year lifespan of the program.2
Woredas in Tigray were not included in the evaluation due to security reasons. Table 1 presents the 
coverage of the Transform PHC activity in the evaluated regions, representing a total sampling frame 
of 381 woredas:
Table 1: Coverage of Transform PHC Activity by Region
Regions Intervention Woredas Total Woredas in 
Each Region
Percent of 
woredas covered 
by Transform Start End
Amhara 76 94 143 66
Oromia 133 162 325 50
SNNP 98 111 186 60
Sidama 13 14 37 38
Tigray 17 53 88 60
Total 337 434 691 63
Throughout the life of the activity, TPHC implemented several interventions to assist and support 
the MOH and its regional health bureaus—RHBs to improve MNCH/FP outcomes. TPHC was 
designed to support the institutionalization of health care quality; to implement performance 
standards at different levels of the health system, and for various health service performance 
improvement initiatives, to strengthen public financial management and health management 
information systems (HMIS); to build the capacity of the health workforce; and to provide medical 
equipment and supplies. 
3) EVALUATION PURPOSE AND EVALUATION QUESTIONS
EVALUATION PURPOSE
The final performance evaluation of the Transform PHC Activity was intended to inform 
USAID/Ethiopia’s decisions related to future investment in supporting MNCH/FP interventions. The
purpose of the evaluation was to measure achievements of Transform PHC in terms of its Activity￾level results framework (see results framework in Annex 4) and explore associated underlying 
reasons for achievement or non-achievement of anticipated results, as well as to review intervention 
approaches used by the implementing partner (IP). 
The evaluation assessed the degree to which the IP proactively adapted to contextual changes during 
the intervention period. In addition, the evaluation report documents lessons learned and best 
practices—incorporating illustrative vignettes on selected case studies based on field visits—to 
inform the future programming of similar program activities by USAID/Ethiopia. The evaluation also 
assessed factors that contributed to the observed results and those which may have inhibited 
achievement, focusing on the following key areas:
● Effectiveness, which is defined as the extent to which the interventions achieved their 
objectives and activity results.
● Gender equity, which is defined as fairness of treatment for women and men, according to 
their respective needs.
● Adaptive management, which is defined as an intentional approach to making decisions 
and adjustments in response to new information and changes in context.
2 The number of target woredas increased from 360 in the second year to 434 (including Tigray =53 woredas) in the final 
year due to administrative re-structuring. 
7 | P a g e
● Sustainability, which is defined as the extent to which interventions and their benefits will 
continue beyond the life of the program.
EVALUATION QUESTIONS
The performance evaluation answered the following questions related to Activity performance:
 
1. How effective were the Transform PHC Activity approaches in contributing to improving 
MNCH/FP outcomes? What were the drivers of the observed changes? What constraints 
affected the achievements?
2. How and to what extent did Transform PHC Activity facilitate local ownership, sustainability, 
and coherence? 
3. How did Transform PHC consider gender dynamics in activity design and implementation?
4. How and to what extent has the Transform IP been able to adapt their interventions in 
response to learnings and new evidence? 
Specific sub-questions were identified under each key EQs as presented in Annex I. The Impact 
Evaluation of the Transform program ascertained the extent to which the Activity improved 
MNCH/FP outcomes in Transform intervention woredas compared to non-Transform intervention 
woredas (see the Impact Evaluation Report 2022). The impact of the overall Transform Program is 
addressed in a separate impact report. 
AUDIENCE AND INTENDED USES
The audiences of this final evaluation report are USAID/Ethiopia, specifically the Health Office, MOH 
authorities, RHBs and other country-level partners, and healthcare providers. For these audiences, 
the report also provides documented evidence of the lessons learned and best practices that can 
inform new policies and MNCH/FP programs. 
4) EVALUATION METHODS, ETHICAL CONSIDERATIONS AND LIMITATIONS
4.1. SAMPLING OF WOREDAS
All regions were included, except Tigray, and a purposive sample of zones and woredas was 
developed to gather qualitative data. Quantitative data were also collected in all visited zones. In 
order to measure the effectiveness of the TPHC interventions, woredas were selected based on their 
performance classification as high, moderate, or low3 performing, using the results of the Greatest 
Impact Assessment (GIA) which used MNCH/FP performance indicators from HMIS 2016 and that 
has been used for high level monitoring.4 The reason for using the GIA was to categorize woredas 
based on their recent RMNCH performance data and prioritize woredas where the results of the 
PHC interventions could be evaluated. 
Simple random sampling was used to select woredas in the TPHC targeted regions with other 
development partner intervention data in the five selected MNCH/FP thematic areas; that is, maternal 
3 Coverage greater or equal to 80% is high; coverage 60% to 79% is moderate; and coverage < 60 % 
is low. 
4 Source: USAID GIA analysis summary (2016). Transform MELA used the woreda performance 
categorization in the GIA analysis to ensure that there will not be sample selection bias for the endline 
evaluation between Transform program intervention and non-Transform intervention woredas. GIA criteria 
used were: Early institutional neonatal mortality rate; Proportion of surviving infants vaccinated with Penta-3; 
Proportion of people of all ages tested positive for malaria; Proportion of children under 5 years who tested 
positive for malaria; Proportion of children with moderate malnutrition; Proportion of children with severe 
malnutrition; Institutional maternal mortality rate; Proportion of pregnant women that received at last ANC4; 
Proportion of births attended by skilled health worker; Early postnatal are coverage; Proportion of pregnant 
women tested for HIV; and Contraceptive acceptance rate.
8 | P a g e
health, child health, neonatal, family planning, and immunization, and woredas that have GIA 
performance categorization information. A total of 80 intervention woredas and 23 non-intervention 
woredas were selected for the performance evaluation. 
All intervention woredas that received three or more interventions from other development 
partners were excluded. All woredas with no GIA woreda performance categorization information 
and woredas in Sidama Region, which does not have data on development partners, were also 
excluded from the sampling frame. Conflict-affected woredas were excluded due to security reasons. 
Table 2 (below) shows the sampled woredas by GIA performance. 
Table 2: Distribution of sampled intervention woredas by GIA performance.
Region Low performance Moderate performance High Performance Total
Amhara 8 12 4 24
Oromia 12 18 5 35
SNNP5 7 11 3 21
Total 27 41 12 80
4.2. SAMPLING OF HOUSEHOLDS 
A household (HH) survey employed a three-stage cluster sampling technique. At the first stage, the 
evaluation team drew a random selection of kebeles (the lowest administrative units) in the 
Transform intervention woredas as primary sampling units, drawing from the list of kebeles in the 
respective regions stratified by administrative zones and performance before 2017. In this manner, 
woredas, in which the randomly selected kebeles were located, were automatically included in the 
evaluation. The second stage of the sampling entailed the random selection of Gotts (neighborhoods) 
from the selected kebeles. Because of the difficulties and time-consuming exercise of developing a 
fresh list of households in a kebele, 2-3 Gotts were selected randomly from each kebele, to generate 
an average list of 150 households from those selected Gotts. 
The third stage of the sampling was the selection of HHs from the randomly selected Gotts. A list of 
households from each selected Gott served as a frame to select 30 HHs per kebele. The key eligibility 
criterion for selecting the HHs was the availability of a woman aged 15-49 in the HH, regardless of 
marital status, who had residential status there for at least 6 months before the date of data collection. 
When more than one such eligible respondent was found in a selected household, the enumerator 
randomly selected one respondent from among the eligible women. A total of 2,703 households were 
surveyed and eligible respondents were interviewed in TPHC woredas, and 692 households in non￾intervention woredas.
4.3 SAMPLING FOR THE HEALTH FACILITY SURVEY
All the primary health care facilities designated to serve the population of the sampled kebeles -
health posts (HPs), health centers (HCs), and primary hospitals - were identified and included in the 
final performance evaluations. That is, the health post serving the population in the sampled kebele 
was automatically included in the study, and the catchment HC to which the selected health post is 
linked was also included. Similarly, if a primary hospital is attached to the selected HC, it too was 
included. Non-functional health posts were not included in the health facility survey. A total of 191 
facilities were sampled: 80 HCs, 78 HPs, and 23 hospitals from Transform intervention woredas. 
Moreover, 49 facilities were surveyed in the non-intervention woredas: 23 HCs, 21 HPs, and 5 
hospitals. 
5
Sidama became a separate region in 2020, but for purposes of baseline and endline comparison, its data were 
analyzed together with Southern Nations, Nationalities and Peoples (SNNP) regional data.
9 | P a g e
4.4 SELECTION OF KEY INFORMANTS 
Interviewees for KIIs were selected using purposive sampling. Gender balance was considered in the 
selection process, as was representation of various stakeholder groups: USAID, central and regional 
IPs technical staff, relevant directorates at MOH, Regional Health Bureaus (RHBs), and selected 
WorHO. A total of 32 KIIs were conducted as part of this performance evaluation (Table 3). 
Table 3. Distribution of KIIs 
Stakeholders 
USAID/Ethiopia Interview with USAID Agreement Officer’s 
Representative (AOR) for Transform 
activities 
Ministry of Health Interview with MOH Directorates 
● Health System Special Support; 
● Health Extension Program and Primary Health Care; and
● Policy, Planning, Monitoring, and Evaluation 
IP’s central office KIs Transform PHC management and technical team (2)
IP’s regional offices Interview with Transform PHC four regional project 
coordination team
Regional Health Bureaus – MCH 
unit
KIs (1 KI per region) MNCH Department 
Woreda health offices Transform Intervention woredas (4): 7 KIs – Heads of Woreda 
Health Offices and/or MNCH officers 
Non-Transform Intervention woredas (2): 2 KIs – Heads of 
Woreda Health Offices or MNCH officers
Total 32 KIIs
4.5. DATA COLLECTION METHODS
4.5.1. Data Sources
This final performance evaluation used two main sources of qualitative data – document review and 
KIIs. Quantitative data came from several sources: household (HH) surveys, health facility (HF) 
surveys, and monitoring data as reported by Transform PHC. A wide range of documents were 
reviewed, including IP performance reports, IP-led research, MOH annual reports, the HSTP midterm 
review, Mini-Demographic and Health Survey reports, Transform baseline and midterm evaluation 
reports, and other relevant documents. The primary data collection involved HH and HF surveys, 
along with KIIs from USAID/Ethiopia, MOH, RHBs, IP, and WorHOs in all regions. 
4.5.2. Data Collection Tools
The desk review was conducted on a list of documents. A data extraction spreadsheet was developed 
to gather the most significant findings. The findings from the desk review were then coded and 
triangulated with data from KIIs and HF and HH surveys. 
The KIIs followed a guide to gather the views and experience of the interviewee in relation to the 
performance of the Transform PHC Activity. The questions were broad and open-ended to facilitate 
unbiased responses and were accompanied by probing questions to gather more detail under each 
EQ. Trained bilingual interviewers conducted the KII in the language of preference of the interviewee 
and took notes in English. The KIIs were recorded to facilitate notetaking. The KII notes were emailed 
daily to the focal person for review and quality control. 
Household and HF surveys were conducted using pre-tested structured questionnaires that were 
uploaded to smartphones, generating electronically collected HH and HF data. Moreover, some open-
10 | P a g e
ended questions were used to collect relevant data from KIs at health facilities. In addition, Gott 
listing, HH listing, enumerator assignment and follow-up tools were used to guide the sampling and 
the data collection process.
The household survey tool included sections on: 
• Household and women’s demographics, household living conditions 
• Enrollment in community-based health insurance plans 
• Household decision-making practices 
• Health service uptake including family planning, antenatal care, delivery and postnatal care, 
and newborn health child immunization and child health services 
The health facility survey tool enabled collection of data on: 
• Availability of MNCH/FP services and water and sanitation facilities 
• Access to communication and power supply
• Health facility management and performance, human resources
• Gender responsiveness 
• Provision of selected family planning, antenatal care, delivery and postnatal care, immunization 
services, and adolescent and youth health services
4.5.3. Data Collectors’ Training and Quality Assurance
To assure high-quality data, a day-and-a-half training was conducted for qualitative data collectors and 
a three-day training took place for all survey coordinators and supervisors in Addis Ababa. 
Enumerators who administered HH surveys had at least a BA/BSc degree and were fluent in the local 
languages. Importantly, all data collectors had prior experience collecting similar data and 
understanding the culture and traditions of the communities they were visiting. Supervisors had an 
MA/MSc in health or social science fields and previous experience in similar activities. Given that the 
respondents were mothers of reproductive age, and to ensure that they spoke comfortably with 
interviewers, the survey team took gender balance into account. In addition, qualitative data 
collectors had a minimum of an MA/MSc in health or social science fields and extensive experience 
in qualitative data collection, transcription, and analysis. 
The training covered topics such as research ethics, rights of human subjects during research, 
sampling procedures, informed consent, data collection tools, interviewing techniques, data handling, 
confidentiality and quality, and gender considerations during data collection. The structure of the 
training included presentation of and detailed discussions on survey instruments followed by a role￾play activity. Data collectors received training on both the paper-based and electronic versions of the 
data collection tools. The survey tools were pre-tested by data collectors in the field in all languages 
at the end of the training, before the commencement of the actual data collection, to ensure 
consistency with the baseline and midline evaluation. 
Before traveling to each of the selected woredas, supervisors and coordinators communicated with 
regional and local leaders about the evaluation. To ensure the quality of collected data, supervisors 
conducted spot-checks and reinterviewing, especially at the beginning (first three days) of the data 
collection. Moreover, supervisors reviewed a sample of completed questionnaires daily before 
uploading them to the server. All KIIs were audio-recorded and transcribed and translated to English. 
Five percent of the transcriptions were checked against the audio file for accuracy. In addition to the 
KIs, Transform: MELA conducted limited interviews for nested case study vignettes on topics relevant 
to program goals of reducing preventable maternal and child morbidity and mortality.
4.6 DATA ANALYSIS
All HH and HF survey data were cleaned, checked, and validated to immediately identify and address 
any issues during data collection, a process that takes place directly with supervisors and data 
collectors. Once data collection was complete, an intensive cleaning was conducted that included 
11 | P a g e
coding and recoding, consistency checks, addressing refusals and non-responses, and validating 
contents.
4.6.1. Quantitative Data Analysis: 
Descriptive (frequency and percentage calculations) analysis was conducted to describe the 
characteristics of a select set of MNCH/FP key performance indicators (KPIs). The analysis of HH 
data compared KPI values between periods (baseline vs. endline), across regions, and against Life of 
Activity (LOA) targets for Transform intervention areas. It also assessed the difference in endline KPI 
values between Transform PHC and non-Transform intervention areas. Data management and 
analysis were done using STATA 14 (StataCorp LLC, USA).
4.6.2. Outcome measures
The evaluation used the results framework of Transform PHC and its KPIs to measure performance. 
Some of the KPIs considered in this analysis were: family planning indicators (contraceptive 
prevalence rate (CPR), long-acting family planning (LAFP) use, and postpartum family planning (PPFP) 
counseling and use), maternal health indicators (antenatal care (ANC), skilled birth attendance (SBA), 
early postnatal care (PNC), iron-folic acid supplementation (IFS), neonatal health indicators (early 
PNC, essential newborn care and early initiation of breastfeeding), child health indicators (Penta3 
coverage, measles coverage (MCV1), exclusive breastfeeding, vitamin A supplementation, deworming, 
ARI treatment with antibiotics, diarrhea treatment with oral rehydration salts (ORS) and Zinc, and 
fever treatment within 24 hours of onset) and cross cutting indicators (access to basic sanitation, 
spouse accompany to ANC, spouse accompany to delivery, and women participation in decisions 
regarding their health). These KPIs were used in the measurement of achievement of intermediate 
results of the Transform PHC Activity. 
4.6.3. Qualitative Data Analysis
Coding: A codebook was developed based on evaluation questions. In addition, part of the interview 
transcripts was reviewed, and the codebook was further refined as specific themes emerged. Next, 
the transcripts were coded using the codebook. Development of the coding framework was an 
iterative process based on an initial list of the codes developed by the analysis team, as identified 
from the evaluation questions. Through a series of meetings with technical specialists, preliminary 
codes were further refined by adding emergent codes and modifying the preliminary codes after 
careful reviews of a sample of field notes based on KIIs.
Analysis: Thematic analysis was conducted using ATLAS.ti, a qualitative data analysis software. The 
selection of themes and the analysis of the data were framed by the EQs and key thematic areas 
related to MNCH/FP. The synthesis of various qualitative information was used to understand factors 
facilitating or hindering implementation as well as achievement of results.
4.6 ETHICAL CONSIDERATIONS
The evaluation received institutional review board (IRB) approval from the Ethiopian Public Health 
Institute (EPHI) at the outset of research, and all data collectors and supervisors were trained in 
research ethics. Key informants (KIs) were asked for their consent to be interviewed and for the 
interview to be recorded, and interview notes and recordings were coded to protect respondent 
confidentiality. Survey respondents were also asked for their informed consent; during that process, 
they were informed about choices regarding participation, responding or not to specific questions, 
and their right to request termination of the survey. Informed consent also covered the privacy and 
confidentiality of participants—respondent names were not captured; instead, coding was used to 
conceal their identity. All interviews were conducted privately, ensuring that information collected 
remained confidential. For women under the age of 18, additional parental permission and participant 
assent was obtained before data collection.
12 | P a g e
4.7 LIMITATIONS
Logistical limitations. Security issues in the country related to conflict limited access to certain 
regions. Security-related limitations were anticipated and were mitigated by careful planning. The 
household and health facility data collection took longer than initially anticipated because of security 
and conflict issues which required adjustments to the survey and interview schedules in all three 
regions. Despite these challenges, the selected informants were interviewed, facilities were visited, 
and the sampled data were gathered.
Methodological limitations. Non-Transform intervention areas selected to serve as the 
comparison areas should not be regarded as true controls due to complex factors. One of the 
principal ways that the design sought to control for “contamination” effects was to ensure that 
Transform intervention woredas were not contiguous with non-Transform intervention woredas. 
Such a design does mitigate contamination to some degree. Although these design issues pose 
methodological challenges, from a methodological standpoint the spillover would likely attenuate 
differences between the intervention and non-intervention areas, thus leading to conservative findings 
rather than overstated ones. Furthermore, because Transform program interventions are designed
to support nationwide health-system building for improved MNCH/FP outcomes, from a “real-world” 
implementation perspective such “spillover” effects can be viewed as positive outcomes for the 
program. 
The evaluation also designed rigorous controls based on Transform: MELA’s partner mapping studies, 
and for the evaluation of Transform PHC, it limited woreda selection in both Transform intervention 
and non-intervention areas to those that had two or fewer intervention types of a total of five 
MNCH/FP thematic interventions. However, this design allows for only an approximation of the
effects of other MNCH/FP-focused development partners intervening in the non-intervention areas. 
There are limited data available, for example, on the level of financial investments that these other 
development partners made in both Transform-intervention woredas and non-Transform 
intervention woredas. Thus, measuring the actual effects of other partner investments remains a 
challenge. 
In terms of triangulation with other data sources such as DHIS2 and IP random follow up visits, the 
evaluation recognizes that there are methodological and data quality issues with these sources. For 
this reason, comparisons of these data sources, and others, such as PMA, with the Transform 
evaluation data, particularly with the quantitative data obtained through household and facility 
surveys, were not used as they may demonstrate divergent results due to different methodologies.
6
Furthermore, because the qualitative data depends on the views and comments of those persons that 
made themselves available to be interviewed, selection bias can result. However, the careful selection 
and training of highly qualified qualitative data collectors and analysts helped to mitigate data quality 
bias to a large extent. Data from various sources were carefully triangulated to confirm the evidence 
and to mitigate recall bias as well as potential social desirability bias, whereby respondents give 
favorable or positive answers to please the interviewer.
5) FINDINGS
This section presents overall performance findings by evaluation question, by intermediate result and 
where appropriate, by region and by technical area: maternal health, newborn health, child health, 
family planning, and gender. 
6 This evaluation does not triangulate findings with those from PMA because PMA survey data include large cities such as 
Addis Ababa. In contrast, the data from the current evaluation was collected primarily in rural areas which showed evidence
of larger declines compared to PMA data.
13 | P a g e
5.1 How effective were Transform approaches in contributing to improving MNCH/FP outcomes?
In general, there was consensus among all KIs, who
reported that TPHC implemented effective 
approaches that contributed to improved MNCH/FP
healthcare delivery, and probably even helped prevent 
further deterioration due to contextual factors. Key 
MOH informants also reported that they appreciated 
the program’s contributions and were able to list 
several effective interventions that contributed to 
improved healthcare delivery (Box 1). 
Investments in system strengthening interventions 
such as woreda twinning, quality improvement, family 
planning program woreda planning, and linkages with 
hospital services were reported as most effective, 
along with the provision of equipment and other 
supplies. These are discussed in detail later in the 
corresponding section of the report.
Notwithstanding the positive perceptions and 
appreciation for the support received, survey data 
showed that the activity did not meet its targets for most indicators. Table 4 shows a summary of 
the activity’s performance by key performance indicators. 
The activity effectively contributed to improving skilled birth attendance (SBA), early Antenatal Care
(ANC), newborn care, measles and Vitamin A coverage and long-acting family planning (LAFP). Other 
indicators showed a decline in comparison to midterm values (dark shaded figures), except for five 
indicators, some related to outreach campaigns (Vitamin A, deworming,) and the others being early 
ANC and exclusive breastfeeding along with Acute Respiratory Infection (ARI) treatment. In total, 
11 out of 24 indicators showed an improvement in comparison to baseline values (light shaded 
figures). Each thematic area is discussed in detail later in the report. 
Table 2: Summary of Activity Performance by Key Performance Indicators.
Shaded cells show indicators that improved from baseline, darker shades show improvement from midterm. 
Thematic 
Areas Key Performance Indicators Baseline 
(%)
Midterm 
(%)
Endline 
(%)
Family 
Planning
MCPR among all women 36.6 41.6 32.6
MCPR among currently married women 45.1 47.3 36.5
LAFP methods among all women 9.7 12.5 11.4
LAFP among currently married women 11.8 14.1 12.7
PPFP counseling after birth 29.2 29.1 19.4
PPFP use 34.6 42.8 27.7
Maternal 
Health
Early initiation of ANC 28.9 50.8 51.8
IFA supplementation for at least 90 days 21.7 25.1 18.5
ANC4+ 56.1 46.5 40.6
Essential components of ANC 32.8 35.7 28.4
Skilled birth attendance 57.5 63.4 59.1
Early PNC for mothers 43.5 51.8 41.9
Newborn 
Health
Early PNC for newborns 37.4 46.9 39.9
Essential Newborn Care 8.9 16.4 12.0
Box 1: Most appreciated interventions by Key 
Informants
• Twinning partnerships of woredas and 
health centers 
• Establishment of the clinical skill 
laboratories 
• LMG local trainer pools 
• Introduction of the V-scan ultrasound 
• Coaching and mentorship support using 
the public system structures
• On-site training 
• Woreda grants 
• Family Planning Exercises
• Peer education approach 
• “Her Space” initiative 
• installation of the solar suitcases in 
selected health centers 
14 | P a g e
Thematic 
Areas Key Performance Indicators Baseline 
(%)
Midterm 
(%)
Endline 
(%)
Early initiation of breastfeeding 74.9 88.8 83.7
Child 
Health 
(under 5 
years)
Full immunization 40.3 42.4 26.8
Vitamin A supplementation in last 6 mo. 43.1 46.4 72.2
Children with Symptoms of ARI 7.0 4.7 3.0
ARI treatment 29.4 53.6 55.3
Diarrhea incidence 9.0 14.1 12.1
Diarrhea treatment 28.9 33.2 12.1
Exclusive breast feeding 58.3 71.4 75.0
Deworming in the last 6 months 36.9 43.2 60.5
Fever treatment within 24 hrs. of onset 40.9 37.1 22.9
5.1.1 Were Transform activities implemented as planned? If not, why not?
At the beginning of the evaluation process, evaluators asked the IP to describe the implementation 
status of the TPHC activity. The IP indicated that most of the planned activities were implemented 
as planned, but certain activities that were not implemented as well as hoped included gender and 
community level activities, which were limited due to security problems. From the perspective of the 
IP, the government was not able to allocate a budget dedicated to implementing gender-related 
activities. Different external factors frequently mentioned as barriers to implementation were staff 
and leadership turnover, political unrest and security issues, and a lack of motivation and commitment 
among service providers and leaders. 
KIs also frequently mentioned COVID-19 as one of the factors hindering program activities. COVID￾19 disrupted most of the basic health services and TPHC activities. It also prevented health seeking 
among communities because of fear of acquiring the infection. As a result, most of the maternal and 
child health services were reported to have significantly dropped and, in some areas, they were 
completely interrupted. KIs also mentioned that in some areas COVID-19 seem to have minimal 
impact on health care provision and program activities. Despite this perception, quantitative data 
showed a decline of MNCH/FP outcomes, as well as significant differences by region. These 
differences in performance are noted below.
5.1.2 To what extent did the activity improve MNCH/PH outcomes and reduce 
inequities in different groups and intervention areas? 
Despite the positive perception of how the TPHC Activity performed by most stakeholders 
interviewed, progress was not achieved in all technical areas and inequalities across regions persisted. 
The evaluation showed marked differences across regions that have not been addressed yet. The 
activity does not have an intermediate result or an indicator to measure reduction of inequalities, so 
this dimension remains to be addressed. 
The summary (Table 4) shows that 11 indicators improved from baseline and five from midterm 
values. KIs reported several reasons for this, notably political and social unrest and COVID, which 
limited mobility and access to health facilities. Almost all KIs mentioned that war/conflict affected the 
program’s implementation and overall health service provision. It was mentioned that the program’s 
achievements were “wiped out due to the civil-war,” and that several health facilities in war-affected 
areas were either destroyed or not functioning properly due to a shortage of the necessary supplies. 
As a result, the program activities were focused on emergency response and humanitarian assistance 
in lieu of planned activities. The effectiveness of the emergency response activities was not part of 
the final evaluation design. 
15 | P a g e
FAMILY PLANNING PERFORMANCE
TPHC did not meet its life of activity (LOA) targets in family planning (FP) services. By the end of 
activity, it achieved an overall MCPR of 36.5% among currently married women in the interventions 
woredas which is about nine percentage points below baseline (Figure 1). Interestingly, the 
performance of the LAFP indicator slightly increased in comparison to the baseline, possibly showing 
preference for these methods in the current conflict context.7 In summary, overall MCPR declined, 
and one out of four FP indicators showed improvement in comparison to baseline: LAFP (Figure 1).
Figure 1: Family planning program performance at baseline and endline, and LOA target (Top) and performance 
trend (bottom) in Transform PHC intervention areas.
Most family planning indicators declined in all regions except for LAFP in Oromia and SNNP (Table 
4). One explanation for this difference is that 13% of women in the intervention areas reported being 
turned away from health facilities when seeking FP services (data not presented). The percentage of 
women being turned away from health facilities was highest in Amhara (21.5%), followed by 
SNNP/Sidama (13.9%) and Oromia (8.3%). The main reason for turning away clients were commodity 
stockouts, facilities that were closed, or that service providers were unavailable. Further research at 
the facility-level is needed to confirm the finding that women reported to be turned away due to lack 
of commodities and trained staff. FP utilization by sampled facility would need to be compared with
household data in the coverage area of each facility.
In addition, stockouts were frequent in surveyed facilities; 89% of them reported stockouts of at least 
one type of FP commodities in the last three months preceding the survey (Figure 2). FP commodities 
that were out of stock in many of the facilities included Progestin only injectable (Depo-Provera), 
Implanon and combined oral contraceptives.
Table 3: Family planning program performance at baseline and endline by region.
Indicator Amhara Oromia SNNP
Baseline (%) 48.2 42.0 45.2
7 A strong hypothesis for this result is that LAFP results are less affected by unrest in the country than other modern 
contraceptive methods; once LARC are inserted they typically remain for 3 to 5 years and thus are not as affected by external 
factors compared to short-acting methods such as pills and injectables. 45.1 11.829.2 36.5 34.6 12.7 19.427.748.0 18.031.043.0
M C P R L A F P P P F P 
C O U N S E L L I NG
P P F P U S E 
Baseline (%) Endline (%) LOA Target (%)
Baseline Midterm Endline
MCPR
LAFP
PPFP
16 | P a g e
Indicator Amhara Oromia SNNP
MCPR among currently 
married women Endline (%) 39.8 33.9 38.3
LAFP among currently 
married women
Baseline (%) 13.1 10.5 11.9
Endline (%) 11.5 12.5 14.6
PPFP counselling
Baseline (%) 33.0 28.2 41.7
Endline (%) 30.1 24.8 32.2
PPFP Use 
Baseline (%) 33.0 28.2 41.7
Endline (%) 28.3 18.9 26.0
Figure 2: Reported stockouts of FP commodities in transform PHC intervention and non-transform intervention 
three months prior to the survey.
The health facility survey showed that 6.4% of surveyed health posts in intervention areas did not 
provide any modern FP methods. Aside from being turned away, the most frequently mentioned 
reasons for women not to use FP methods in rank order included: wanting to get pregnant, religious 
restrictions, fear of side effects/infertility, and partner refusal (limited autonomy women have over 
health care decisions). Despite the decline in FP coverage, KIs reported that attitudes towards FP 
services have improved.
“There were many women who have a fear to use family planning services during the previous time. 
However, they are currently using this service without any fear. Significant progress has also been made 
on youth Adolescent health through providing a youth friendly service program, especially for women.” 
WorHO, SNNP)
Family planning indicators declined in both intervention and non-intervention areas except for LAFP, 
where they increased in intervention areas and remained the same in non-intervention areas (Figure 
3). There were no statistically significant differences between intervention and non-intervention areas. 
TI = Transform Intervention; NTI = Non-Transform Intervention
Figure 3: Baseline and endline comparison of FP services by intervention and non-intervention areas. 28.2 17.5 51.5 11.7 32.0 12.6 22.3 13.6 10.7 28.6 14.3 57.1 10.7 32.1 14.321.4 14.3 17.9
COC POP Injectables Male
condom
Implanon Jadelle ECPs IUCD No
stockout
Intervention (%) Non-intervention (%)
45.1
43.7
11.8
11.9
29.2
31.3
34.6
30.3
36.5
34.2
12.7
11.9
19.4
17.1
27.7
24.7
TI NTI TI NTI TI NTI TI NTI
MCPR LAFP PPFP counselling PPFP Use
Baseline (%) Endline (%)
17 | P a g e
MATERNAL HEALTH PERFORMANCE
Maternal health performance showed improvement in early ANC and SBA coverage. All other 
indicators showed a decrease in coverage (Figure 4). The biggest decline in outcomes, for ANC4, 
showed a 16-percentage point decrease and has been declining since baseline. The decrease in 
maternal health coverage was explained by reduced access to services due to unrest and COVID.
Further operational research is necessary to fully understand the reasons for the decline and how to 
develop an effective causal pathway to increase ANC4+ coverage. No LOA targets were achieved. 
Figure 4: Maternal health outcomes at baseline and endline with LOA targets (Top) and performance trend 
(Bottom) in Transform PHC intervention areas.
Performance varied by region (Table 5). In particular, early ANC and SBA improved between baseline 
and endline in all regions, and the essential components of ANC and early PNC improved in Oromia 
and SNNP but declined in Amhara. ANC4+ declined in all regions. IFA coverage improved in Oromia 
only. Some of the reasons mentioned by KIs for the observed low performance for maternal health 
KPIs were:
• Weak community health programs; interruption of community structures that would have 
enhanced tracing and follow-up of pregnant women 
• Weak and continuous reshuffling of health system leadership
• Long distance to the health facilities and disruption of maternity waiting homes during the 
COVID-19 pandemic8
• Recurring conflict in some parts of the regions e.g., east and northern Amhara, west and 
south Oromia, south-west SNNP 
• Mothers’ low health literacy (poor risk perceptions)
• Lack of counseling contributed to dropout from the continuum of MNCH care
Factors reported to have affected implementation such as high staff turnover and long distance to 
facilities were not easily amenable to TPHC interventions, but can be mitigated through MOH 
policies, as well as program support for ongoing orientation of new staff, mobile clinics,
8 Although the incidence of SBA improved, KIs said that COVID prevented women from use maternity waiting homes and 
affected SBA in some woredas. It is likely that SBA may have increased even more if COVID had not affected service delivery 
and access to maternity waiting homes.
28.9 32.8
19.9
56.1 57.5
43.5
51.8
28.4
18.5
40.6
59.1
41.9
58.0
42.0
34.0
60.0
71.0
58.0
Early ANC Essential ANC IFA ANC4+ SBA Early PNC for
mothers
Baseline (%) Endline (%) LOA Target (%)
Baseline Midterm Endline
IFA (>=90 days)
ANC4+
SBA
Early PNC
18 | P a g e
teleconsultations, and strengthening of local health posts. The lack of counseling and mothers’ 
beliefs they do not need to go to a facility if they feel well are amenable to further interventions.
Table 4: Maternal health program performance outcomes at baseline and endline.
Indicator Amhara Oromia SNNP
Early ANC
Baseline (%) 35.8 31.3 21.9
Endline (%) 49.6 52.5 52.1
Essential ANC 
components
Baseline (%) 50.5 23.9 28.1
Endline (%) 29.2 26.1 32.9
IFA
Baseline (%) 34.9 7.0 21.7
Endline (%) 16.8 18.6 19.9
ANC 4+ 
Baseline (%) 55.0 53.4 59.5
Endline (%) 41.6 34.9 52.1
SBA
Baseline (%) 59.0 50.3 63.2
Endline (%) 63.7 54.1 66.4
Early PNC for 
mothers
Baseline (%) 59.6 28.2 46.8
Endline (%) 59.3 32.4 49.3
Early ANC increased in both intervention and non-intervention woredas, probably as the result of 
spillover due to systemic improvement that covered both. The improved performance observed in 
intervention woredas (Figure 5) was not statistically significant, however.
TI = Transform Intervention; NTI = Non-Transform Intervention
Figure 5: Antenatal care performance outcomes at baseline and endline by transform intervention and non￾transform intervention areas. 
Figure 5: SBA, early PNC, and ITN outcomes at baseline and endline by Transform intervention and non￾Transform intervention areas. 28.9 29.2 32.841.9 19.925.5 51.8 48.0 28.4 21.9 18.5 13.7
TI NTI TI NTI TI NTI
Early ANC Essential ANC IFA
Baseline (%)
56.1
55.8
57.5
59.6
43.5
40.6
34.9
37.4
59.1
51.4
41.9
39.0
TI NTI TI NTI TI NTI
ANC 4+ SBA Early PNC for mothers
Baseline (%) Endline (%)
19 | P a g e
NEWBORN HEALTH PERFORMANCE
No newborn health outcomes met the LOA targets, but they all improved when compared to 
baseline (Figure 7). The rate of increase in newborn health intervention coverage was higher before 
the midline than after the midline. The most probable reason that LOA targets were not met was 
because IPs set targets based on the progress observed within the first two years of program 
implementation. It is important to note that 50% of the respondents did not know if their baby had 
received essential newborn care. Although improved, coverage of Essential Newborn Care was 
reported at 12%, which would leave about 88% of newborns not receiving the essential newborn 
care package. Early initiation and exclusive breastfeeding continued to be high. 
Figure 6: Newborn health outcomes at baseline and endline with LOA targets (Top) and performance trend 
(bottom) in Transform PHC intervention areas.
Despite improvements, regional differences in coverage deserve further study. Oromia region 
performed worse than others in the provision of early postnatal care (PNC) and Amhara’s 
performance was the lowest (4.4%) in the provision of essential newborn care services (Table 7) 
when compared to the other regions. This was so because the baseline rate was the lowest in 
Oromia. However, the rate of improvement from the baseline is almost the same for all regions
Table 5: Newborn health performance at baseline and endline by region.
Indicator Amhara Oromia SNNP
Essential Newborn 
Care
Baseline (%) 3.8 0.0 20.7
Endline (%) 6.8 10.2 19.2
Early PNC for 
newborn 
Baseline (%) 46.8 25.5 42.3
Endline (%) 52.2 30.5 50.7
Early initiation of 
breastfeeding
Baseline (%) 78.7 79.2 69.2
Endline (%) 84.1 84.3 82.2
Exclusive breastfeeding Baseline (%) 44.7 59.7 66.7
Endline (%) 75.5 68.8 86.4
8.9
37.4
74.9
58.3
12.0
39.9
83.7
75.0
23.0
54.0
90.0
79.0
Essential Newborn 
Care
Early PNC for 
n ewb orn 
Early initiation of 
b reastfeedin g
Exclusive 
b reastfeedin g 
Baseline (%) Endline (%) LOA Target (%)
Baseline Midterm Endline
Early PNC
Essential Newborn Care
Early breastfeeding
Exclusive breast feeding
20 | P a g e
As Figure 8 shows, newborn health outcomes increased in both Transform PHC and non-Transform 
intervention areas, except exclusive breastfeeding in non-Transform intervention areas. At the 
endline, a statistically significant difference was detected in early initiation of breastfeeding between 
Transform PHC and non-Transform intervention areas (p=<0.01). 
TI = Transform Intervention; NTI = Non-Transform Intervention
Figure 7: Newborn health performance outcomes at baseline and endline by transform intervention and non￾transform intervention areas.
Figure 8: Skill lab in SNNP
The quality of newborn care was reported by KIs to have 
improved due to the creation of newborn intensive care units 
(NICU) in all district hospitals found in Transform intervention 
areas and the creation of 33 skills labs: 11 in Oromia, 9 in 
Amhara, 9 in SNNP and 4 in Tigray (Figure 9).
CHILD HEALTH PERFORMANCE
Preventive services: The LOA of child health services 
targets were not met except for Vitamin A supplementation
(Figure 10) and deworming (Figure 11).
Figure 9: Expanded Program on Immunization and Vitamin A supplementation at baseline and endline with LOA 
targets (top) and full immunization performance trend (bottom) in Transform PHC intervention areas.
Measles vaccination coverage increased significantly (p<0.001) by 24.3 percentage points from 
baseline, which KIs attributed to vaccination campaigns coupled with deworming that was also found 8.9 3.3 37.4 28.3 74.9 55.2 58.375.5 12.0 6.4 39.9 35.6 83.7 74.7 75.0 70.0
TI NTI TI NTI TI NTI TI NTI
Essential Newborn Care Early PNC for newborn Early initiation of
breastfeeding
Exclusive breastfeeding
Baseline (%) Endline (%)
44.3
35.2
41.4
40.3
43.1
61.8
26.8
70.0
70.0
72.2
48.0
36.0
Pen ta 3 coverage MCV1 Fu ll immu n ization Vitamin A su p p .
Baseline (%) Endline (%) LOA Target (%)
40.3 42.4
26.8
Baseline Midterm Endline
Full immunization performance trend
21 | P a g e
to have increased about the same percentage points from baseline. On the other hand, the 
performance of Penta 3 declined from 44.3% to 41.4%. Full immunization declined significantly 
(<0.001) and covered 26.8% of the children under 5 years of age. This suggests that the routine 
vaccination program did not perform as well as expected, probably due to contextual factors; now 
there are over 70% of children who did not receive all basic vaccines, suggesting that there is now a 
health emergency in terms of the health system’s ability to save children’s lives. KIs reported that 
vaccine campaigns have been effective in the past and will need to continue to be conducted until a 
routine vaccination program is restored to make up for the shortfall in the performance of the routine 
immunization program. The routine vaccination system has not been strengthened enough to 
withstand current shocks. 
Coverage of antibiotic ARI treatment and deworming significantly increased significantly (p<0.001) 
from baseline (Figure 11). On the other hand, fever treatment within 24 hours of onset declined 
significantly (p<0.001). Diarrhea treatment with ORS and Zin decreased mostly due to a marked 
decrease in SNNP. This reflects the inequalities mentioned above. The decline in ORS and Zinc 
treatment is important due to the increase in diarrhea incidence. This finding deserves further study 
because the IP reported that 90% of health posts in intervention woredas provided CBNC service 
with trained staff at the end of 2021. A much better performance would have been expected. All 
hospitals/health centers also provide these services. The health facility survey found that 95% of HPs 
in intervention areas provide ICMNCI and 99% of hospitals/Health centers provide IMNCI services. 
Figure 10: Baseline, endline and LOA comparison of access to preventive and curative childhood illness 
treatment (top) and performance trend (bottom) in Transform intervention areas.
The performance by region showed a low fully vaccinated indicator across all three regions. Measles 
and deworming service indicators nevertheless reached about two-thirds of the children. Access to 
ARI treatment improved in all three regions (Table 8). Shaded figures highlight the differences in child 
health performance by region. 
Table 6: Baseline and endline comparison of child health indicators by region.
Indicator Amhara Oromia SNNP
Penta 3 coverage Baseline (%) 58.4 37.0 37.9
Endline (%) 34.7 41.8 46.4
Measles Baseline (%) 37.0 36.1 33.0 36.9 29.4 28.940.960.5 55.3 26.0 22.9 51.060.0 35.0 42.0
Dewormin g ARI treatmen t 
with an tib iotics 
Diarrh ea 
treatmen t 
(ORS+Zin c)
Fever treatmen t 
with in 2 4 h rs of 
on set
Baseline (%) Endline (%) LOA Target (%)
Baseline Midterm Endline
ARI treatment
Diarrhea treatment
Deworming
22 | P a g e
Indicator Amhara Oromia SNNP
Endline (%) 59.2 62.1 63.6
Full immunization Baseline (%) 67.4 24.1 29.7
Endline (%) 19.4 29.3 28.2
Vitamin A 
supplementation 
(6-59 months)
Baseline (%) 43.5 37.2 48.7
Endline (%) 74.9 72.9 68.6
Deworming Last 6 months 
(L6Ms)
Baseline (%) 35.5 29.8 45.3
Endline (%) 61.3 60.9 59.1
U5 children with ARI 
symptoms Last 2 weeks 
Baseline (%) 11.4 5.9 4.8
Endline (%) 3.2 3.1 2.8
ARI treatment with 
antibiotics 
Baseline (%) 28.6 34.6 25.0
Endline (%) 37.5 59.5 61.1
Diarrhea incidence_L2Ws Baseline (%) 9.6 8.1 9.5
Endline (%) 15.0 8.8 17.0
Diarrhea treatment with 
ORS+Zinc
Baseline (%) 18.4 27.5 38.0
Endline (%) 32.9 34.2 12.0
U5 children with fever for 
who advice/treatment was 
sought within 24 hrs.
Baseline (%) 39.4 45.3 38.5
Endline (%) 19.7 34.8 15.2
Measles coverage, Vitamin A supplementation, deworming, and ARI treatment increased in both 
Transform PHC and non-Transform intervention areas. However, no statistically significant difference 
was detected between Transform PHC and non-Transform intervention areas (Figure 12 and 13).
TI = Transform Intervention; NTI = Non-Transform Intervention
Figure 11: Child immunization performance outcomes at baseline and endline by intervention and 
non-intervention areas.
TI = Transform Intervention; NTI = Non-Transform Intervention
Figure 12: Child health performance outcomes at baseline and endline by intervention and non￾intervention areas. 44.3 50.4 35.2 42.3 40.3 48.6 43.1 35.5 41.4 41.661.8 61.8 26.8 23.6 72.2 73.7
TI NTI TI NTI TI NTI TI NTI
Penta 3 coverage MCV1 Full immunization Vitamin A Supp.
Baseline (%) Endline (%) 36.9 40.0 29.4
22.2
28.9
23.3
40.9
38.5
60.5
62.6
55.3
70.0
26.0
34.8
22.9
22.4
TI NTI TI NTI TI NTI TI NTI
Deworming ARI treatment with
antibiotics
Diarrhea treatment
(ORS+Zinc)
Fever Teatnment within
24hrs of onset
Baseline (%) Endline (%)
23 | P a g e
5.1.3 How effective are the Transform activity strategies and approaches? 
KIs in the regions were asked to identify what they believed were the most and least effective 
interventions. The most effective reported were: Woreda Twinning, FP planning, Clinical Skills labs, 
capacity building through onsite training, coaching and mentoring, and Woreda Sub-granting, as well 
as providing equipment (V-scans and solar suitcases). 
“The experience of the initiated Ultrasound service at the 27 health centers has been scaled up to 
cover a total of 70 health centers.” -- (RHB respondent, Amhara region)
Findings from the health facility sample also indicated that the BEmONC, Adolescent Youth Health 
(AYH) services and community-based health insurance were effective. The evidence of effective 
strategies and approaches is described in the sections below.
However, under certain circumstances, interventions that were reported as successful in some
regions were less successful in others. For example, some key informants described difficulties with 
the health information system that did not allow for continuous monitoring and quality improvement 
interventions that did not persist:
“We were not successful result in transforming kebeles into model kebeles. Of course, we have made 
some progress though it is not promising. In addition, there were major problems on ISS, data verification 
and connected woreda. In these areas, our performance was less than our target.” SNNP
“Regarding the Transform approaches which were the least effective, in our woreda context, we believe 
that the quality improvement training was the one we failed. We were proposing as it would be so 
productive and we got grant from the project, and the training was given as planned. But it did not work 
any longer. Our health workers who took the training did not get to action as proposed. Putting the 
reasons for the failure precisely demands researching and probing the workers, while in our opinion it can 
be linked with the lack of motivation. Surprisingly, when we communicate in person there are individuals 
replying as they lost the skills and demand more training of the type to rehearse.” Amhara
BEmONC
About 60% of the facilities reported being able to implement all seven BEmONC signal functions, 
representing a significant improvement from the baseline. Except for one signal function, the 
administration of corticosteroids, a large majority of facilities were able to perform all BEmONC
functions (Table 7). As one important example, only 35% of facilities reported being able to perform 
an assisted vaginal delivery at baseline; this figure has now increased to 73%. 
Table 7: Percentage of sampled health facilities that reported performing each BEmONC Signal 
function on the day of the survey.
BEmONC Signal Functions Facilities with Functional 
BEmONC
Administration of parenteral oxytocin 95.1
Administration of parenteral anticonvulsants 84.3
Administration of parenteral antibiotics 95.2
Manual removal of placenta 89.3
Removal of retained products of conception 88.4
Newborn resuscitation using bag and mask 92.2
Administration of corticosteroids 35.6
Assisted vaginal deliveries 72.8
All signal functions (except administration of corticosteroids) 61.2
24 | P a g e
Adolescent and Youth Health (AYH)
Most of the sampled facilities (over 50%) reported that they provide AYH services on all ten criteria 
(Table 10), and the activity-supported facilities scored higher in all service categories in the 
intervention areas, thus demonstrating the effectiveness of the strategy implemented (Figure 14).9
Table 8: Percentage of key adolescent and youth health interventions reported to be provided in 
sampled facilities.
Key adolescent and youth interventions Percent of facilities
Counselling through peer educators 61.1
Counselling through YFS trained health providers 63.9
Provision of modern contraceptives 90.3
Youth-friendly post-abortion care 84.7
Youth-friendly post-abortion contraceptives 88.9
Pregnancy testing 94.4
Treatment of sexually transmitted infections 93.1
HIV testing 94.4
Link to ART clinics 86.1
Life skill training 50.0
Figure 13: Comparison of available adolescent and youth health services in intervention and non-intervention 
areas.
Workforce Training
Key informants reported the usefulness of health-related workforce training. Because the activity 
implementer placed a high value on human resources as the “heart” of the healthcare system, about 
thirty-six thousand health workers received training (Figure 15). In addition, TPHC also reported to 
9 These data show the supply side and not the demand side. Despite the large proportion of sampled facilities that report 
to provide youth-friendly services, pregnancy testing and contraceptives, ANC and FP outcome indicators have declined 
since midline. The evaluation did not collect data on change in the number of ANC and FP users among adolescents and 
further study of use patterns among both adults and adolescents is included in the recommendations. 
61.1
63.9
90.3
84.7
88.9
94.4
93.1
94.4
86.1
50.0
25.0
52.6
47.4
68.4
68.4
79.0
79.0
84.2
79.0
68.4
36.8
10.5
Counselling through peer educators
Counselling through YFS trained HCPs
Provision of modern contraceptives
Youth-friendly post-abortion care
Youth-friendly post-abortion FP
Pregnancy testing
Treatment of STIs
HIV testing
Link to ART clinics
Life skill training
All components
Non-intervention (%) Intervention (%)
25 | P a g e
have developed skill-based mentorship platforms, which were implemented to ensure capacity was 
built, using on-site-orientation and skill labs in selected health facilities. 
Figure 14: Number of health workers who received training provided by the transform program by 
thematic area (Source: TPHC training report).
Training was also reported to have enabled selected health facilities to effectively utilize their budgets 
towards quality improvement activities such as to purchase medications, to improve patient registries, 
and expand building activities. Moreover, training was also reported to have helped to ensure more 
effective utilization of health facility financial resources for improving quality of service, instead of 
what one IP regional manager termed the “old tradition of accumulating capital.”
“The [health care financing] training enabled the health facilities to utilize their budgets … to improving 
the quality of their services, instead of the old tradition of accumulating capital.” (IP regional office, 
Amhara region).
The activity did not have a target that required it to measure the number of facilities that have trained 
and well-performing staff who apply what they learned. Consequently, the evaluation could not
measure how effective this training was to improve health care provider performance, except 
indirectly through the views of KIs and the measurement of health outcomes, which are also 
influenced by other systemic variables (medicine supply, etc.) and contextual factors. Nevertheless, 
there is consensus that the activity effectively built capacity. 
Woreda Twinning
KIs reported that TPHC successfully facilitated the 
implementation of twinning partnerships between high and low 
performing woredas10. TPHC facilitated the creation of win-win 
relationships between woredas that until now had worked 
independently of each other, thereby helping not only to create 
and strengthen health system linkages at the regional level, but 
also to create opportunities for collaboration and joint 
continuous improvement within the region. A total of 116 
mentor and mentee woredas were included in the twinning 
program. The main goal of the twinning intervention was to 
help the low performing woredas to improve. However, it also helped the higher performing woreda 
to sustain their achievements and motivated their staff to continue improving and sharing their 
expertise. The twinning process took about five years and included a continuous cycle of repeating 
the implementation steps (Box 1) at higher levels of performance. TPHC provided the training, grant 
funding, supplies and facilitated the process of relationship building among twinned woredas.11 An 
example of site observation of woredas in the SNNP Region is discussed below.
10 Argaw et al. BMC Health Services Research (2020) 20:892. Accelerating the performance of district health systems 
towards achieving universal health coverage via twinning.
11 The final evaluation did not gather specific primary quantitative data related to improvements of this intervention.
However, at the baseline, TPHC reported that “of 81 standards set for health centers, performance increased from 81 to 88 
percent in one mentor woreda and from 58 to 78 percent in its mentee. (Transform PHC, Technical Brief, 2018).
8089
4281
12470
6959
4690
FP/RH service
provision
Nutrition related
Issues
AYHD related
issues
Child Health Maternal and
Newborn Health
Box 2: Implementation Steps
1. Partnership Development
2. Needs Assessment
3. Gap Analysis
4. Action Planning
5. Action Implementation
6. Review and Evaluation 
26 | P a g e
Site visit observations: The Case of Kembata Tembaro Zone 
As part of the evaluation, the Transform MELA team travelled to the SNNP Region to interview the 
local TPHC team and the local health authorities who implemented the Twinning program in Kembata
Tembaro Zone and to document their experience. This Zone manages 11 woredas, five of which
participated in the twinning program. The participating woredas signed Memoranda of Understanding 
(MOUs) to implement the steps in Box 2 and meet a number of standards (Box 3). 
A district health system performance score was calculated that showed improvement a year later
based on several programmatic outcomes, such as: 
a. Contraceptive prevalence rate (CPR)
b. Proportion of pregnant women attending antenatal care clinics tested for syphilis
c. Proportion of births attended by skilled health personnel
d. Proportion of women who attended postnatal care at least once during the early post￾partum period (within 7 days after delivery) 
e. Proportion of children who had Penta 1, but dropped from Penta 3
f. Fully immunization coverage for children under one year
g. Iron-folic acid supplementation
h. Children attended Growth Monitoring and Promotion sessions
i. Proportion of all forms of TB (New and relapse) cases detected during a specified period
The Kembata Tembaro Zone authorities emphasized the importance of the intersectoral solidarity 
and collaboration across woredas that created a true partnership among all the professionals that 
participated in the program. The Twinning allowed the zone to create model facilities and woredas 
that will be able to twin with and support the improvement of the remaining six woredas in the zone:
“This health center was in “red” in all indicators at the start of the program and now it is a model 
facility that has received national awards. The transformation is fantastic. Now over 80% of the 
facilities are meeting standards. We learned the importance of partnerships and solidarity to improve 
healthcare. It works because it is adapted to our culture.” (ZHD, SNNP)
Box 3: Woreda Performance Standards
The woreda health office has core processes and case teams responsible to execute the 
following district management functions: 
1. Planning, monitoring and evaluation of health promotion, disease prevention and curative health 
care activities in the woreda.
2. Coordinating mentoring and technical support among primary hospital, health centers, and health 
posts.
3. Planning and coordinating supportive supervision of primary health care facilities and monitoring
quality of service.
4. Coordinate resource mobilization for primary health care.
5. Ensuring community engagement and ownership.
6. Disease surveillance and coordinating and planning and emergency response for public health.
emergencies.
7. Conducting regulatory functions.
8. Coordination with other sectors at the woreda level.
9. Provision of oversight on finance, human resources infrastructure and supplies to Primary Health 
Care Facilities.
10. Ensures quality service delivery (referrals and linkages, staffing and human resource management, 
medicine supply, etc. to provide an essential package of basic health care services at Primary Health 
Care Facilities 
11. Community engagement activities and coordination with other sectors in the woreda (education, 
agriculture, etc.)
12. Performance management, monitoring and supervision
27 | P a g e
Family Planning Exercises
Despite the decline in FP outcome indicators in the endline 
survey, it is also reasonable to assume with some KIs, that the 
decline would have been bigger without the support of TPHC. 
Moreover, the decline was not the same in all woredas. Some 
facilities in selected woredas like Halaba Woreda in SNNP 
showed improvement because of facilitated planning. TPHC
developed an approach to help woredas plan and ensure that 
facilities can provide quality FP/RH services. The approach 
included a planning workshop (Box 4) in which each facility 
team used data and planning tools developed by the activity to 
estimate the resources required to deliver the planned FP services. The planning tools also allowed 
facility teams to estimate and budget for FP commodity needs. Facility planning data were then 
consolidated into woreda, zone and regional plans and budgets. The planning workshop and tools 
were reported to improve efficient planning and budget allocation and spending of public resources.
Other woredas also reported 
improvements in FP outcomes, especially 
long-acting methods:
“We have seen remarkable progress in terms 
of increasing the number of women who use 
long-acting family planning services to 60%. 
In addition, the program was also effective in 
increasing the counselling skills of health 
professionals on family planning services.” –
(WorHO, SNNP)
Subgrants 
Among the most frequently cited intervention approaches, sub-grants to the Woredas were often 
reported as a mechanism that helped to close financial gaps and empowered local interventions. KIs 
indicated that the sub-grants eased the prevailing budget shortage of the woredas, and as a result, 
woredas were able to accomplish the goals they established, especially when they were twinned with 
a better performing “mentor” woreda. Sub-grants were given based on the priority needs identified 
by woredas themselves, differing from the approach of other projects, in which donor partners made 
decisions on who receives the funding and for what purpose. For this reason, the woredas felt 
empowered to make financial decisions. In addition, KIs reported that the Activity improved their 
grant proposal writing and grant management capacity and created a sense of ownership, and that 
they were able to draw on these skills to apply for funding from other donors. 
In the case of Kembata Tembaro Zone, the Zone Director attributed their results and improvement 
to having received technical support and training to apply for and receive sub-grants. Several of their 
health centers were recognized at the national level for achieving over 90% of KPIs. Similarly, a 
respondent in Amhara region noted the contribution of the sub-grants in improving the capacities of 
the health system. 
“The contribution of the Transform project brings a positive change compared to other partners, in that 
it performs the activities starting from the lower level at the community. For instance, the sub-grants are 
directly delivered to the woredas and [distributed from woredas to] the health facilities. It also attempted 
Box 4: Planning Workshop Agenda
1. Definition of activities
2. Roles and responsibilities
3. Facility assessment
4. Service planning
5. Commodity quantification
6. FP/RH activity organization
7. Action Plan and budget
Box 5: Site observation: Halaba Health Center, SNNP
The Transform MELA team visited Halaba Zone and 
interviewed zonal staff and staff at Halaba health center. 
The health center covers an urban population of over 
100,000 and 12 health posts for a total of over 360,000 in 
the woreda. FP outcomes reported by this facility 
included significant improvements: 
• Adolescent and youth FP users increased from 8% to 
40%
• Total women of reproductive age using modern FP 
methods also increased from 40% to 78%
• Immediate post-partum FP increased from 0% to 24%
28 | P a g e
to improve the technical capacities and the leadership along all levels, including those at the lower 
hierarchy. -- (WorHO, Amhara region)
Table 9 shows the distribution of subgrants to Kembata and Halaba Zones in the LOA. Hadero TZ 
woreda, received subgrants for three years in a row and were able to improve and sustain their 
performance. 
Table 9: Sub-grants by health facility and year in Kembata and Halaba Zone (in Ethiopian Birr –
ETB).
Grantees 2018 2019 2020 Total
Damboya 483,042 501,020 455,420 1,439,482 
Hadero TZ 635,334 326,310 400,874 1,362,518 
Tembaro 506,931 481,145 988,076 
Modula PHL 209,930 209,930 
Angacha 214,890 510,275 725,165 
Kacha birra 607,815 453,810 1,061,625 
Shinshicho PHL 205,606 205,606 
Kembata T. ZHD 574,415 1,001,970 1,576,385 
Atoti Ullo WorHo 275,212 275,212 
Werra WorHo 352,054 352,054 
Wera Dijo WorHo 360,438 360,438 
Halaba Kulito TA 131,525 131,525 
Kulito PHL 203,791 203,791 
Besheno PHL 206,061 206,061 
Halaba ZHD 1,146,793 580,120 508,145 2,235,058 
Total 2,265,169 3,311,501 5,756,256 11,332,926 
Note: Approximately 50 Ethiopian Birr = $1 USD at time of study.
Community-based Health Insurance (CBHI)
CBHI and public finance trainings for health center staff were reported to help improve 
outcomes. The endline survey found that the activity significantly increased (p<0.001) and far 
exceeded the CBHI enrollment targets in the three regions. Of course, this achievement cannot be 
attributed to the activity only, but its contribution was recognized by KIs as significant (Figure 16).
Figure 15: CBHI enrollment at baseline and endline by region.
43.1
9.9 14.6 22.6
83.5 74.0 62.8 73.8
53.0
Amhara Oromia SNNP overall
Baseline (%) Endline (%) LOA Target (%)
29 | P a g e
In addition, community mobilization efforts were reported to have helped raise funds from 
community members for health facilities to improve quality of health services. TPHC also gave training 
and technical support on how to prepare effective budget proposals and how to tap available 
resources. According to one KI from Amhara:
“The activities of the Transform program include involving the community to contribute funds and 
improving their health facilities. For instance, this is notably done in one of our woreda called Gonji￾Kollela. In this woreda, each member of the community contributes one hundred birr [approximately 
USD $2] yearly. Then, the money collected from the community is utilized to improve the health 
center, given that the members of the community have believed in their ownership of the health 
facilities. As a result, we have a plan to appoint the experience of this woreda for regional 
accreditation. The project has also helped us to activate the health care financing system.” (ZHD, 
Amhara region)
Health facility performance
The total number of facilities that meet all MNCH/FP performance indicators is not known as this 
was not a performance target. However, numerous health facility performance indicators show the 
contribution of the TPHC activity. First, about one out of five surveyed facilities in TPHC areas 
provided all adolescent and youth friendly service components, unlike facilities in non-Transform 
intervention areas where fewer facilities provided similar services (less than one in ten). Overall, a
higher proportion of health facilities in TPHC intervention areas provided adolescent and youth 
friendly services than non-Transform intervention areas. 
Second, more health facilities in TPHC intervention areas provided all BEmONC signal functions than
health facilities in non-Transform intervention areas. Except for parenteral oxytocin and antibiotics 
administration, other BEmONC signal functions were provided in more health facilities in TPHC than
non-Transform intervention areas. 
Lastly, at endline, the proportion of health facilities with a performance score of greater than 80% as 
per the EHSTG/EHCRIG (Figure 17) reached 59.2 % slightly higher than the LOA target (50%), as 
well as the implementation of quality improvement plan of health facilities (well over the threshold 
of 48%) exceeded the respective LOA targets, 74.8%. At endline, the percentage of surveyed health 
facilities women represented was around 79.8%, which is slightly lower than the LOA target of 86.0%. 
Figure 16: Health facility performance in Transform PHC intervention areas.
5.1.3 What interventions are replicable or can be scaled up in different settings?
TPHC worked closely with the MOH to institutionalize and scale up of all the interventions cited 
above. These interventions were widely reported effective and as having replicable approaches that 
should be further scaled up and sustained. For example, woreda twining was included as part of 
HSTP-II and scale up is imminent. In addition, key informants also reported that learning how to apply 
for grants empowered woreda staff to successfully apply for fundings from other partners and 
59.2
79.8 74.8
50.0
86.0
48.0
Performance score > 80%
as monitored using EHCRIG
& EHSTG
Women representation in
the governing board
Quality improvement plan
implemented
Performance at endline (%) LOA target (%)
30 | P a g e
continue improving. It was also reported that many of the TPHC interventions have been included in 
the MOH’s budget for 2023 and will be continued or scaled up. When considering scale up, it is 
important to consider that it was reported that what drives improvement is not training alone but 
training in combination with other interventions such as mentoring, coaching and effective 
supervision, as well as other systemic strengthening (financial management, HIS, supply chain, etc.) 
The combination with effective health system performance improvement interventions was reported 
to ensure what is learned in training is applied and incorporated in healthcare delivery, and that it 
empowers health care providers to achieve better health outputs and outcomes on their own. 
On-the-job training, BEmONC, woreda twinning, sub-grants, and clinical audit were among the most 
replicable. In fact, some KIs reported that all approaches employed by the Transform program were 
effective and potentially transforming:
“The first and important [effective approach] is the leadership and management training (LMG) 
given. Second, the open house meeting, a discussion which involves all stakeholders, was a good 
lesson that I must mention. The third and the most important initiative by this program is the increase 
the capacity of the professional staff on preparation of project proposal to get sub grant.” (WorHO, 
SNNP region)
5.1.4 What is the effect of context (conflict, COVID-19 pandemic, internal 
displacement) on the Transform interventions and achievements?
Several factors affected program implementation. The most frequently mentioned factors by KIs in 
in-depth interviews included war and conflict; political instability or unrest; COVID-19; and staff 
turnover. Almost all KIs mentioned that war and conflict affected activity implementation and overall 
health service provision. KIs mentioned that in many cases the activity’s achievements “were wiped 
out due to the civil-war”—health facilities in war affected areas were either destroyed or not 
functioning properly due to shortage of the necessary supplies. As a result, because of the crisis, the 
TPHC focused on emergency response in conflict-affected areas, where implementation of previously 
planned activities was not possible. 
COVID-19. The initial spread of COVID-19 was not as severe as many had feared. In Ethiopia, 1,054 
COVID cases per million were reported and only 16 people per million had died from the condition 
in 202012. However, many KIs feared that by disrupting the health system, the pandemic could have 
devastating consequences on the public health system.
13 A reduction in utilization of essential health 
services could have long-term effects by increasing preventable morbidity and mortality14. Some of 
the COVID-19 prevention measures taken in Ethiopia could have had adverse impacts on the 
utilization of essential health services. Prior to the pandemic, health care utilization and health 
outcomes were relatively poor in Ethiopia compared to other countries. For example, only 48% of 
women gave birth in health facilities at that time, and the neonatal mortality rate was 30 per 1,000 
live births15. Restriction of movement, the conversion of selected health facilities into COVID-19 
treatment centers and the redeployment of health workers to COVID-19 care likely further reduced
access to and provision of essential health services, thereby adversely affecting population health.
16
12 Our World in Data. Ethiopian: Cumulative confirmed COVID-19 cases per million people. Our World in 
Data. Available from: https://ourworldindata.org/coronavirus/country/ethiopia
13 Inzaule SC, Ondoa P, Loembe MM, et al. COVID-19 and indirect health implications in Africa: IMPACT, 
mitigation measures, and lessons learned for improved disease control. PLOS Medicine. 2021;18(6).
14 Assefa N, Sié A, Wang D, et al. Reported barriers to healthcare access and service disruptions caused BY 
COVID-19 in Burkina Faso, Ethiopia, and Nigeria: A telephone survey. The American Journal of Tropical 
Medicine and Hygiene. 2021;105(2): 323–30.
15 EPHI and ICF. Ethiopia Mini Demographic and Health Survey 2019: Final Report. Rockville, Maryland, USA: 
Ethiopian Public Health Institute and ICF; 2021
16 Workicho A, Kershaw MJ, Berhanu L, et al. Essential health, and Nutrition service provision during the 
COVID-19 Pandemic: Lessons from Select Ethiopian Woredas. Current Developments in Nutrition. 2021;5(4).
31 | P a g e
Moreover, due to limited awareness and lack of proper preparedness of health providers, the 
community could fear exposure to the virus while visiting health facilities and thus avoid or delay 
needed care.
The pandemic caused a decline in reproductive, maternal and child health services. However, 
disruptions in these services rebounded to pre-COVID levels within a short period of time.
17
The endline evaluation of the Transform Program in Ethiopia asked several key informants in the 
health system to describe the impacts of the COVID-19 pandemic on the health system. Almost all 
KIs said that there was a decline in the uptake of MNCH/FP interventions within the first two months 
after the index Coronavirus Disease was reported. The decline occurred because of the disruption 
of the health system as health workers were panicking to provide services and maternal waiting 
homes were closed to avoid COVID infections. Some facilities also served as COVID-19 isolation 
and treatment centers. KIs indicated that, while COVD-19 had a lesser impact on health care 
provision and program activities in some areas, particularly in the year following the outbreak of the 
pandemic, COVID-19 disrupted most of the basic health services and Transform PHC activities in 
many areas. The pandemic also limited health seeking behaviors among communities because of fear 
of potential patients acquiring the infection. As a result, most of the maternal and child health services 
dropped, and in some areas, services were completely interrupted. A health center director stated 
the following:
“The early days of the pandemic drastically compromised the service provision compared to the pre￾COVID-period. Many of those who require the service did not visit the facility fearing the pandemic” 
(Haik Health Center, Amhara region).
Similarly, an Oromia RHB respondent mentioned that COVID-19, along with conflict and political 
instability, were bottlenecks MNCH/FP service delivery in the region:
“Conflict and political instability, the COVID-19 pandemic, and staff turnover were the major factors 
that potentially affected the implementation of the Transform program and other government activities.”
The impact of conflict. Health systems in fragile and conflict affected states (FCAS) often struggle 
to provide basic health services. Apart from direct casualties, war and civil strife disrupt health care 
delivery and thereby increase morbidity and mortality.
18 Ethiopia was impacted by war, communal 
violence, and instability over the last two years of the Transform Activity. Most of the regions covered 
by the Activity experienced some form of disruption due to conflict and instability. A devastating war 
broke out in Tigray at the end of 2020, and later expanded to Afar and Amhara. Insurgencies were
also reported in Oromia and Benishangul Gumuz regions. Many areas in Tigray and pocket areas of 
Amhara, Afar, Benishangul Gumuz and Oromia remained inaccessible to humanitarians because of 
conflict. In addition, repeated inter-communal violence and unrest was reported in Oromia, Sidama 
and SNNP. In Sidama, disruptions due to unrest reached a peak in 2019 and 2020. Conflict and 
violence across the intervention areas disrupted the movement of ambulances; health workers were 
displaced; health facilities were looted; and in the worst case, the entire infrastructure of health 
facilities was destroyed. 
Key informants from RHBs, woreda health offices and health facilities reported the impact of the 
continuous conflict and unrest on the health system in their areas. There was a major health system 
disruption, and the impact was worse over the last two years and affected all regions, though the 
scale of the impact varied. A key informant from Oromia region reported that the health system in 
17 Arsenault C, Gage A, Kim MK, Kapoor NR, Akweongo P, Amponsah F, et al. COVID-19 and resilience of 
healthcare systems in ten countries. Nature Medicine. 2022; 28: 1314–1324
18 Barry S. Levy, Victor W. Sidel. War and public health. 2008. 2nd ed
32 | P a g e
many parts of the region was disrupted due to conflict. “Security problems are more damaging to the 
health system in Oromia than the COVID-19 pandemic,” he stated. Key informants from several health 
facilities in Amhara also noted that health care delivery was disrupted for months because of the war 
in northern Ethiopia. A hospital director, for example, stated that the facility was out of service for a 
month because of war. Even after the war was over, much of the infrastructure was destroyed and 
the hospital was performing below its capacity, and even now, the hospital does not work in its full 
capacity. The situation has been particularly acute in the Amhara region, where war affected service 
delivery and stalled the progress within the health system:
“We should tell you our opinion along with the current contextual situations of our woreda. Our woreda 
is in a post-war time right now. We have lost all the progress we have made to improve the MNCH 
services in our woreda so far. As a result, many mothers are not coming to the health facilities” (Woreda 
health office head, Amhara region).
Even woredas in Amhara far from war affected areas reported that the conflict impacted service 
delivery in their areas because it shifted the attention of the government. In addition, health workers 
deployed to the war front and health facilities in non-conflict affected areas were unable to perform 
in their pre-war capacities:
“Many of our health workers working in the different health facilities were moved to the temporary 
stations to treat injured soldiers. … The health professionals were leaving their regular duties, so affecting 
the service. There were also our health professionals assigned to go to the battlefields in order to treat 
wounded people” (WorHO, Amhara region).
Because of severe headwinds in the health system, including conflict, COVID-19 and other disease 
outbreaks, which shocked the health system and disrupted health system functions, many KIs who 
commented on the impacts of these contextual factors indicated that the contribution of TPHC 
should be interpreted in terms of how things would have been worse, if the activity had not been 
there.
5.2 HOW AND TO WHAT EXTENT DID TRANSFORM PROGRAM ACTIVITIES 
FACILITATE LOCAL OWNERSHIP, SUSTAINABILITY AND COHERENCE?
Improvements in management and in the performance of the health system were reported to have 
improved local ownership and facilitated measurable spillover to non-intervention areas. KIs also 
reported that sustainability has been built in as part of systemic interventions, but some question its 
success due to contextual factors. Coherence and collaboration with other partners working in the 
same technical areas and regions were reported to have been effective. 
5.2.1 What mechanisms are in place to ensure government ownership of Transform 
health system interventions? 
In addition to the approaches to ensure ownership and empowerment mentioned above, two main 
health system strengthening mechanisms were put in place to strengthen government ownership of 
PHC interventions: capacity building, especially in leadership, management and governance; and 
quality improvement. 
Health workforce capacity building
Capacity building and mentorship represented the largest investment and included various MNCH/FP 
technical services (Box 5). KIs agreed that the TPHC activity significantly contributed to improving 
the leadership and governance.
“One of the supports provided by the Transform program was strengthening leadership and governance 
to ensure woreda transformation. They supported creating awareness among PHCU directors and HEWs 
33 | P a g e
on Woreda transformation agendas. Orientations and training were provided on key performance 
indicators (KPI). Woreda health office management staff were training on management indicators.”
(WorHO, Oromia Region)
According to KIs, Leadership, Management and 
Governance (LMG) training helped leaders to 
become more competent, and lead woredas to 
become better model health centers and model 
woredas. For instance, a respondent in Amhara’s 
West Gojjam zonal health department gave the 
examples of Bahir Dar Zuria, Degadamot and Finote 
Selam woredas, which succeeded in becoming model 
woredas for several consecutive years. The KI 
attributed these successes to improvement of the 
leadership at all levels of the health system. KIs 
mentioned that, after the trainings and orientations,
most woredas regularly assessed their own woreda 
management standards using woreda management 
standards tool. Similarly, health centers and hospitals 
regularly assessed their performance standards. 
Most KIIs noted that PHC has contributed to 
building facility-level capacity. As a result of LMG 
training, one Amhara KI (Estie Woreda Health 
Office, Amhara Region) pointed to improvements in 
the management standards of the woreda health 
office. TPHC respondents also indicated effective
support to MOH to implement the woreda 
standards.19 A TPHC regional manager in Oromia outlined a range of improvements because of
Transform support as follows:
“Ninety-four [all Transform intervention woredas] assess their own woreda management standards. [All] 
517 health centers [in Transform intervention areas] are assessing their own health center performance 
standards, except for those health centers located in conflict areas of the region which were interrupted 
during the last two quarters. All the 2,190 Kebeles [found in Transform intervention areas] can measure 
their own community score card standards. There are [also] improvements that have made regarding 
using of the tools of identifying and solving own problems, and in assessment of the job satisfaction of 
[health] workers due to the training given.” (IP Regional manager, Amhara region).
However, although other KIs agreed that such practices brought performance improvement in the 
health system and that Transform PHC has built capacity at the facility level, sustainability will depend 
on future availability of resources for health workforce capacity building. 
Quality improvement 
According to KIs, the activity also supported the government’s reform agenda by improving quality 
through on-job trainings and orientations on the minimum national standards, such as the Ethiopian 
Health Center Reform Implementation Guidelines (EHCRIG),
20 the Ethiopian Hospital Services 
19 The evaluation did not collect primary quantitative data to measure and compare trends in facility performance. Most 
KIIs noted that PHC has built facility capacity.
20 The EHCRIG has 10 chapters and 87 sets of minimum standards with 209 validation criteria. The tools were 
used to assess the availability of essential resources and services. The checklist used consists of ten chapters: 
(1) leadership and governance, (2) health center and health post linkage, (3) patient flow and service 
organization, (4) medical records management, (5) pharmacy services, (6) laboratory services, (7) clean, and 
Box 6: Technical Training Topics
• ICCM
• IMNCI
• BEmONC
• Training on how to use mobile 
ultrasound (V-Scan)
• Malnutrition screening and 
management 
• Family planning counseling
• Family planning provision (LAFP 
methods e.g., Implanon and IUCD 
insertion and removal)
• Compassionate and respectful care 
(CRC)
• Effective vaccine management, Cold 
chain 
• Immunization micro-planning 
• Training on youth friendly services 
• Training on quality improvement 
• Training on fistula management
• Training on health emergencies 
(COVID-19)
• Training on GVB service provision
34 | P a g e
Transformation Guidelines (EHSTGs), and Woreda Management Standards (WMS). Selected health 
facilities reported to be linked via quality improvement projects and to share best practices. Linkages
between primary hospitals and health centers were established through an initiative called 
“coordination for quality improvement.” Quality of care audit committees were reported to have 
contributed to improvements on client-provider communication while undertaking the regular 
observation of the service provision, such as the interaction between patients and pharmacist at 
health facility dispensaries. 
5.2.2 To what extent did the government staff perceive that the approaches established 
by the project will be sustained by the government and continue to sustain 
improvements?
KIs indicated that the Activity’s achievements were likely to continue. They reported that the ways 
in which Transform PHC implemented its major strategies and approaches would ensure their
sustainability and referred to specific elements of the approach that increased the likelihood of 
sustainability. For example, KIs emphasized that primary stakeholders from all levels of the health 
system were actively involved in the designing of the program and their suggestions were 
incorporated in the program plan. This created a sense of ownership among stakeholders and the 
program enjoyed widespread acceptance by stakeholders working in the public health sector. KIs 
also pointed out to the fact that Transform activities were in line with government plans and strategies
and directly supported the activities of the public health system rather than implementing new or 
parallel activities implementing various capacity building activities. Because some primary stakeholders 
participated from the beginning of the program design and throughout the program implementation, 
certain KIs reported that despite government underfunding for health and declining health indicators 
since 2020, they believed that innovations such as sub-grants contributed to the likelihood of 
sustaining some of the program’s activities: 
“In fact, as the program’s agendas are part of our responsibilities, we have no choice but to sustain the 
improvements. Therefore, we believe we will keep on sustaining the improvements, even with their 
burdens and shortage of resources. As well, we also believe that the improvements made, and the activities 
initiated by the Transform’s program will continue.”
“The woreda sub-grant is the major mechanism in place to ensure government ownership. In sub-grants, 
Woredas identify health problems in their respective Woredas, prepare plans, implement them, and 
make evaluations.” (Oromia region WorHO).
Furthermore, co-creation with stakeholders from all levels of the health system involved the designing 
of the program with the government’s suggestions incorporated in Activity plans, resulting in a sense 
that Transform PHC facilitated achievement of the government’s strategic plan on MNCH/FP, and 
increased ownership and acceptance among stakeholders in the public health sector.
Woreda technical support was reported to gradually shift from Transform PHC to Zonal Health 
Departments (ZHDs), providing ZHDs a leading role and responsibility in supporting the woredas. 
Coaching and mentorship support also helped in establishing a cadre of mentorship experts who are 
working within government structures to improve and sustain the capacity of health facilities and 
public health staff. However, documentation of standard operating procedures (SOPs) and other 
tools is not widespread. 
Despite having a wide range of interventions to ensure sustainability, KIs at woreda health 
offices reported that they still have capacity gaps and insufficient resources to sustain some of the 
safe health facility, (8) medical equipment management and biomedical engineering, (9) human resource 
management, and (10) quality improvement and health information system.
35 | P a g e
interventions, e.g., training and clinical mentorship of healthcare providers. Sub-granting program and 
Woreda Twinning were reported as instrumental, but the activity lacks indicators and targets that 
would provide the evidence to prove it. In addition, there was a lack of a handover and sustainability 
strategy and plan for each region, zone, and woreda. 
KIs at RHBs noted that, despite some remarkable interventions to ensure sustainability, some 
woredas may face challenges, and some of the achievements may not be sustained. As turnover of 
health workers and managers at district health offices is high, the TPHC program organized training 
more frequently than previously anticipated, but it has not been able to develop a mechanism to 
ensure the orientation and training of new staff on an ongoing basis yet. In short, the TPHC activity 
became a gap-filling instead of system-strengthening program, and its sustainability is thus questioned. 
Key informants mentioned that some woredas developed dependency instead of working to fill their 
own gaps, such as strengthening of on-the job-training mechanisms and development of in-house 
capacities. In addition to turnover, lack of resources may challenge the sustainability of the 
achievements. Key informants noted that sustainability cannot be realized unless the government 
allocates adequate budget resources to the health sector. 
5.2.3 To what extent did the project improve community involvement and its role in 
accountability, transparency in decision-making, and advocacy for community health 
needs?
TPHC supported the use of community score cards (CSC). This is a social accountability mechanism 
that was implemented by the government nationwide to improve transparency, accountability and 
responsiveness within the health system was the CSC. The CSC was aimed at engaging community 
members in the management oversight of the public health facilities by promoting the engagement of 
community members in the planning, development, implementation, and monitoring and evaluation 
processes of health service delivery. Every quarter, client councils worked together with health 
service providers and local government officials to discuss problems and come up with solutions. 
These client councils were comprised of selected community members who identified problems 
related to health and provided performance ratings of health care facilities. 
KIs reported that CSC results improved the health system responsiveness and performance. They 
also pointed to specific activities that they believed improved community involvement oversight of 
the health system:
“CSCs served us as one of the platforms for community engagement. A community forum has been 
organized to collect feedback on the health service provision. We have organized a community forum 
this year. There is a community council established in each kebele and [a typical] council has six members. 
Health center directors are trained on CSC, who in turn establish community councils and orient them. 
Orientation has been provided at the zonal and woreda levels.” (ZHD, Oromia Region) 
Despite these efforts, however, challenges including conflict, civil unrest, and shifts in government 
priorities prevented Transform PHC and Ethiopia’s flagship community health program from 
achieving many of its MNCH/FP targets at the community level. Disruptions of the community health 
program was a factor for the slowdown of progress in key MNCH/FP interventions. For example, 
KIs in SNNP noted that the Women Development (WDA) and Health Extension Program (HEP) did 
not work as expected, and setbacks were observed in many of the community-based interventions, 
such as the pregnant mother’s conference. 
5.2.4 To what extent did the project improve community engagement for advocacy and 
health service oversight? What capacity for community engagement did Transform 
build? 
36 | P a g e
TPHC was reported to support community engagement activities through trainings, supportive 
supervision, and materials support. The KIs also reported that PHC conducted advocacy to improve 
awareness of the community about their rights. The Activity supported the “pregnant mothers’ 
conference” initiative. This is a forum where pregnant women gather regularly to discuss health and 
health-related issues during in pregnancy. KIs reported that the program support was critical to 
strengthening the functionality of the pregnant women’s conference. The Activity also supported an 
“open house health post” initiative where community members and HEWs jointly identified problems 
and proposed solutions to improve the health posts’ services. Numerous KIs believed that this 
approach improved the RMNCH/FP service utilization and quality.
“The health post open house program was also among the community engagement activities; it was 
launched in two of our kebeles. In our opinion, the health post open house program has made immediate 
improvements in the number of institutional deliveries and other indicators in those kebeles at the start 
of the program.” (Amhara, IP regional manager) 
KIs also discussed other Activity interventions that improved community engagement: the community 
action cycle model, school engagement, male engagement in family planning, mass media engagement, 
and partnership defined for quality. Despite the positive views of KIs, the Activity did not establish 
indicators that would measure the effectiveness and coverage of these community-engagement 
interventions.
5.2.5 How does TPHC coordinate with other development partners and their activities?
The fact that the Transform program activities were aligned with the government plan, which 
facilitates a natural partnership between the program and government, is a sign in favor of alignment 
and coherence. Transform staff shared offices with the government staff, which was reported to have 
facilitated communication and collaboration. In addition, Transform staff actively participated in NGO 
forums and technical working groups, and they used these platforms to share research, experience,
and resources. Partnership forums were reported to have been established at national, regional,
zonal, and woreda levels to facilitate coordination among partners, harmonize activities, and avoid 
duplication of efforts. The forums were chaired by the government leadership, and in some of the 
forums the transform program served as a co-chair. According to KIs, these forums were functional, 
and regular meetings were conducted to harmonize plans, present reports, share experiences, and 
avoid duplication of effort: 
“We were working with other development partners. There was a ‘Partner Forum’, a platform through 
which all development partners meet every six months. We are the co-chair for this forum. In addition, 
we were strongly working with the regional health bureau in several activities like gap strength analysis, 
woreda base plan and development of annual work plan. Creating a strong partnership with the regional 
health bureau helped us use their resources whenever we had budget limitations.” (SNNP, IP regional 
manager).
“The Transform program has been closely working with other partners operating in the same intervention 
areas. Woreda-based planning, technical working groups, joint planning, and participatory planning 
sessions are the mechanisms in place to strengthen the partnership among development partners. The 
RHB is involved in the selection of intervention sites to avoid duplication of efforts. Shifting of intervention 
sites is also one of the mechanisms we use to avoid duplication of efforts.” (RHB, Oromia).
At the zonal health department level, different partners were given responsibility over different areas 
based on need and avoid duplication of efforts. There were also government-NGO forums at zonal 
level where partners present their activity reports and harmonize activities.
37 | P a g e
“The zonal health department facilitates activities to harmonize the programs of other partners with the 
Transform’s program activities. For instance, if Engender Health plans to give training for health 
professionals at a given health center, we will check whether the training was given by another partner 
at that health center or not. Likewise, we are attempting to avoid duplication of efforts in any way. 
Besides, we use governmental-nongovernmental organizations (G-NGO) forum at the zonal level. In the 
forum, partners are allowed to present their activity reports. The G-NGO forum is also useful to avoid 
duplications of efforts.” (IP Amhara regional manager) 
Despite these effective partnership interventions, KIs reported several challenges to effective 
partnerships that require effective leadership, planning and monitoring to be addressed coherently 
and collaboratively: 
• Partners have different objectives, strategies and methodologies and are not open to changing 
their priorities or project implementation.
• The partnership forums are not strong enough to guide coordinated action and mutual 
accountability. Some are not active.
• NGOs sign project agreements with the RHB without coordinating with other partners, 
which creates overlapping of programs and gaps in some areas. 
• There is no database of the health staff who have received training on specific topics. It is 
possible that the same person attends training on the same topic repeatedly. 
• The Woredas to be supported were selected by the RHB. The problem is that the RHB did 
not involve the Zones in the Woreda selection process.
• Lack of accountability of development partners to follow through with their commitments 
and meeting community expectations.
• Reluctance among some of the development partners to cooperate and work together as 
they may be competitors for future projects.
• These inefficiencies lead to resources being left unutilized and significant duplication of efforts.
5.3 HOW DID THE ACTIVITY CONSIDER GENDER DYNAMICS IN 
IMPLEMENTATION?
Figure 18 shows that the activity did consider gender in its activities, but had limited effect: 
• Women accompanied by their spouses during delivery increased slightly. 
• Women’s participation in health care decision-making and women accompanied by their 
spouses during ANC showed a statistically significant decline (p<0.001).
• Men accompanying their spouses during delivery remained the same, whereas women 
accompanied by their husbands during ANC showed a declining trend compared to the 
midterm. 
Figure 17: Baseline and endline comparison of selected gender-related indicators. 86.5 62.784.9 75.5 40.4 85.0 86.6 60.0 88.0
Particip ation in own 
h ealth care decision s
Accomp an ied b y sp ou se 
du rin g last ANC visit
Accomp an ied b y sp ou se 
du rin g last dellivery 
Baseline (%) Endline (%) LOA Target (%)
38 | P a g e
As Figure 19 shows, women’s participation in own health care decision making and the proportion 
of women accompanied by a spouse during delivery increased both in Transform intervention and 
non-Transform intervention areas. 
TI = Transform Intervention; NTI = Non-Transform Intervention
Figure 19: Selected gender related indicators at baseline and endline
The endline survey uncovered regional differences that deserve further study. First, women’s 
participation in health care decision-making increased in Amhara, while it declined markedly in the 
other regions; in Amhara and SNNP women who were accompanied by their spouses during delivery 
increased; and women accompanied by their spouses during ANC declined across all regions (Table
10).
Table 10: Baseline and endline comparison of gender indicators by region.
Indicator Amhara Oromia SNNP
Women participated in their health care 
decisions
Baseline (%) 91.3 88.5 79.7
Endline (%) 91.5 75.0 58.1
Women accompanied by their spouse 
during at least one ANC visit
Baseline (%) 68.3 69.2 52.6
Endline (%) 40.7 40.6 39.7
Women accompanied by their spouse 
during last delivery 
Baseline (%) 81.0 87.3 85.2
Endline (%) 84.2 86.2 89.0
TPHC conducted an assessment to identify gender issues and chose to also address the health needs 
of gender-based violence (GBV) victims and the provision of integrated GBV services in all woredas. 
By 2021, TPHC reported that 73% of the supported facilities provided post-GBV services and had 
created nine demonstration GBV centers. KIs reported that these centers provide a “One Stop Shop’ 
service for GBV victims where professionals from justice, police, and health work together to help 
victims. 
“Gender issues were one of the activities that we have in our checklist whenever we went to woredas for 
monitoring. Besides, gender issue was one of our routine activities. For instance, we have conducted gender 
analysis in more than 50 woredas and provided a psychological support for victims of gender-based 
violence (GBV), especially in the areas where there are IDPs. We were closely working with different 
governmental and non-governmental organizations which works on gender issues.” SNNP 86.5 87.7 62.7 54.6 84.9
92.3
75.5
77.4
40.4
39.0
86.6
87.8
TI NTI TI NTI TI NTI
Participated in own health care
decisions
Accompanied by spouse during last
ANC visit
Accompanied by spouse during last
dellivery
Baseline (%) Endline (%)
39 | P a g e
The health facility survey during the endline showed that 67% of surveyed health centers/hospitals in 
the Transform intervention areas conducted GBV screening, and close to 60% referred victims for 
psychosocial support. TPHC also supported the training of health professionals on GBV service 
provision, including counseling. Moreover, the program provided materials such as “Dignity Kits” for 
GBV victims during the war. Figure 19 shows improvement for GBV indicators in intervention areas 
when compared to non-intervention.
Figure 18: Percent of Transform PHC intervention and non-transform intervention health facilities 
providing specific GBV services at the time of the survey.
TPHC also introduced the “Her Space” initiative aimed at improving the knowledge, attitude, and 
practice of young girls regarding reproductive health and teaching essential life skills.
“The other one is the Her space program and the approaches involving the participations of the brothers 
and parents of the female members (the girls) in the learning sections. This program was launched in 
four kebeles of our woreda. This approach increases the knowledge, attitude, and practice of young girls 
in many aspects of reproductive health. This program helps young girls to acquire essential life skills which 
are valuable to their futures. And, looking at the biological brothers and parents of the member girls 
taking part in the learning section, tells us the increase in awareness of the brothers and the parents of
the girls.” (WorHO, Amhara Region)
The further expansion and sustainability of these GBV centers and gender activities was unknown, as 
they were too new to be able to predict future programming.
5.3.1. To what extent did the government staff/people perceive that the project 
established processes are functional to track and address gender and inequitable health 
outcomes? 
73% of the facilities surveyed as part of this evaluation reported providing some form of post-GBV 
services. KIs perceived several gender-related improvements such as the post-GBV guidelines and 
the establishment of nine GBV centers. However, the Activity faced challenges in deepening its 
contributions to gender-related improvements: First, it had only one gender expert to address this 
technical area and second, reportedly did not have gender-specific indicators, nor did it use HMIS 
indicators to monitor gender interventions to tracking the effectiveness of gender-related 
interventions. Use of such data would have better allowed for comparing regions, zones or woreda 
performance to support weaker areas to improve. 
For some KIs, these challenges posed barriers to implementing deeper gender-based interventions 
that could contribute to effectively addressing gaps. For example, KIs reported that it was unclear
how many victims have been treated at the GBV centers or how many woredas or facilities have 
improved their gender-related performance because of the TPHC activities. Moreover, MOH 
currently does not provide sufficient resources to tackle gender-based issues, particularly at lower 
levels. According to one regional IP manager:
67.0 59.2 44.7 49.5 60.2 50.5 60.7 46.4 42.9 39.3 46.4 46.4
GBV screening Emergency
contraceptives
Post exposure
prophylaxis
Psychosocial
counseling
Referal for
psychosocial
services
Referal for legal
services
Intervention (%) Non-intervention (%)
40 | P a g e
“Tracking gender issues is still needs more work. It has many barriers for the implementation of gender. 
One thing there is no clear plan and guideline about gender integration in the system. The other is 
community’s perception about gender is low. Though gender seems getting better at the top with equal 
representations in Ministerial cabinet, the problem is still there when it comes to the lower level.” --- (IP 
Regional Manager, Amhara Region) 
5.3.2. What gender equity results did Transform Program Activities’ interventions 
achieve?
Informants reported that the gender assessment and landscaping analyses TPHC conducted helped 
inform policies and intervention. The Activity’s recent (2022) landscape analysis described its activities 
as “entry point" for integrate GBV prevention and treatment, which were intended to provide 
evidence for MOH to “propel GBV interventions.” Key informants also widely reported effective 
activities for youth. One respondent in South Wollo Zone in Amhara region stressed the contribution 
of the Activity to generating youth awareness as follows:
“There are effective processes that address gender issues. For example, there are activities done to 
address young girls. For instance, there was a program called ‘Act with Her.’ The program was given at 
schools to foster peer-discussion and build life-skills of young girls mainly on reproductive health issues. 
This youth friendly approach was effective. In the process, trainings were given to the youth; and peer￾to-peer discussions were encouraged. In this regard, there were activities done to make both girls and 
boys become capable of utilizing the youth friendly services fairly. Besides, there were activities done to 
raise awareness of the girls, so as to enable them to prevent gender-based violence. Overall, the 
processes were assessed by the program at quarterly basis.” -- WorHO health officer, South Wollo Zone,
Amhara region.
5.3.3. What gaps exist in the current USAID/ Transform intervention 
design/programming in terms of addressing the key gender issues in the targeted 
regions of Ethiopia?
There are many gaps that remain to be addressed, both in terms of service provision and in 
monitoring the quality and effectiveness of gender related services. With only nine centers providing 
post-GBV services nationwide, access to services GBV victims is limited. A number of challenges 
were reported such as he still-prevalent practice of early marriage, the lack of gender equity 
indicators to monitor the Activity’s progress towards reducing gender inequity in supported woredas, 
and barriers to access to FP/RH services for unmarried and out of school youth represent current 
gaps, 
The scale of “entry point” interventions conducted by TPHC is also not well documented; gender
documents do not typically specify how many woredas or facilities are targeted, nor are 
improvements resulting from gender equity activities documented. KIs did realize that broader 
gender issues were a multi-sectoral issue needing more concerted effort and investment and more 
collaboration with other stakeholders. KI reported the following major challenges to addressing 
gender gaps in the health sector: 
• Lack of documentation of gender interventions in woreda based planning, including use of 
MOH standard operation procedures (SOP) and manuals 
• Lack of a set of targets to monitor effectiveness and coverage of implementing gender 
interventions
• Lack of technical staff to implement gender-related activities at large scale and to support 
scale up
41 | P a g e
For these reasons, clear gender targets for gender-based activities are needed and future baseline 
research disaggregated by locality is necessary to collect data on scale of investment, number of 
woredas using the MOH SOP and manuals, and the results of the use of these guiding materials. 
5.4 INDICATOR TRENDS AT BASELINE, MIDLINE, AND ENDLINE
Figure 20 below summarizes the progress of the TPHC activity and its outcomes by technical area
ay baseline, midline, and endline. The downward trends in most indicators since midline indicate the 
need for further operational research and revised SOPs to ensure effective service delivery and 
coverage expansion during periods of external shocks. 
Figure 19: Indicator trends at baseline, midline, and endline
5.5 HOW AND TO WHAT EXTENT HAS THE TRANSFORM IP BEEN ABLE TO 
ADAPT THEIR INTERVENTIONS ABOUT LEARNINGS AND NEW EVIDENCE? 
5.5.1. To what extent did the project make adaptive changes? Based on what evidence? 
Thematic 
Areas Key Performance Indicators Baseline 
(%)
Midterm 
(%)
Endline 
(%)
MCPR among all women 36.6 41.6 32.6
MCPR among currently married women 45.1 47.3 36.5
LAFP methods among all women 9.7 12.5 11.4
LAFP among currently married women 11.8 14.1 12.7
Family planning counseling after birth (PPFP counseling) 29.2 29.1 19.4
PPFP use 34.6 42.8 27.7
Early initiation of ANC 28.9 50.8 51.8
IFA for at least 90 days 21.7 25.1 18.5
ANC4+ 56.1 46.5 40.6
Essential components of ANC 32.8 35.7 28.4
Skilled birth attendace 57.5 63.4 59.1
Early PNC for mothers 43.5 51.8 41.9
Early PNC for newborns 37.4 46.9 39.9
Essential Newborn Care 8.9 16.4 12.0
Early initiation of breastfeeding 74.9 88.8 83.7
Full immunization (at anytime before the survey) 40.3 42.4 26.8
Vitamin A supplementation in the last 6 months 43.1 46.4 72.2
Children with Symptoms of ARI 7.0 4.7 3.0
ARI treatment 29.4 53.6 55.3
Diarrhea incidence 9.0 14.1 12.1
Diarrhea treatment 28.9 33.2 12.1
Exclusive breast feeding 58.3 71.4 75.0
Deworming in the last 6 months 36.9 43.2 60.5
Fever treatment within 24 hrs of onset 40.9 37.1 22.9
Women who participate in decisions regarding their own 
healthcare 86.5 83.5 75.5
Women accompanied by their spouses/partner to at least 
one ANC visit 62.7 45.9 40.4
Women accompanied by their spouses/partner to a facility 
during birth 84.9 87.1 86.6
CBHI Households enrolled in CBHI scheme 22.6 45.9 73.8
Trend
Child Health 
(under 5 years)
Gender and 
women 
empowerment
Family 
Planning
Maternal 
Health
Newborn 
Health
Time frame
42 | P a g e
The Activity’s drew heavily on its operational research based on the thesis that a strong and sustained 
quality health system that uses evidence-based decision-making requires evidence in health program 
interventions. Without such research, it is less likely to build successful practices, make evidence￾based judgments, and design policies that can improve population health outcomes. 
Transform PHC therefore conducted numerous studies and operational research surveys intended 
to improve and adapt its interventions in different thematic areas. 
These included research on a wide variety of topics such as male engagement, leadership 
management, family planning, and post-partum family planning. The activity developed over 50 articles 
for publication in peer-reviewed journals to document evidence and share learning nationally and 
globally. These studies were used to improve programming and adapt implementation approaches. 
Examples of the utility of operational research and studies includes the use of research evidence to 
contribute to intervention planning in a number of areas, including supportive supervisions, referrals, 
integrated periodic outreach services, emergency response, service integration, strengthening of the 
health system, instillation of a culture of performance and quality improvement, facilitation of the 
introduction of technology and innovations, and informing manuals and policy at the national level.
The IP also conducted quarterly and annual review sessions to assess results and determine where 
gaps remain. Each technical review meeting begins with a discussion of the project's theory of change
(TOC). During these meetings, government representatives discuss priorities and the role of 
stakeholders. As a result of these meetings the Activity confers with USAID on questions related to 
shifting resources, expanding budgets, and to seek advice on the extent of adaptation the project can 
support. During the annual TOC exercise, the IP conducted a strength-gap analysis and partners 
mapping to leverage resources and prevent duplication of effort before finalizing yearly 
implementation work plans. A comprehensive plan was then created in conjunction with the public 
sector and partners operating in each implementation region, and a final plan is created at the national 
level. Disseminating information from research through websites, journals, webinars, and national and 
international conferences was also a key important element to Transform PHC’s approach:
“Doing research is only half the battle; without publishing the findings of research studies or program 
documentation, other researchers or decision-makers will not recognize the importance of the evidence 
produced, won't be able to see it or expand on it, and science won't advance as a whole. Without written 
records, knowledge is lost, and it cannot be broadly disseminated without an efficient process.” 
From the IP’s perspective, the primary and crucial step in the generation of evidence was choosing 
an appropriate and pertinent research topic. Prior to choosing the precise research topic, program 
staff were given training on how to identify relevant issues from prior experience, field observations, 
and critical reading of quarterly and annual reports, and how they prioritized areas of health 
promotion and the most pressing issues relating to reproductive, maternal, newborn and child health 
(RMNCH). The IP also provides training and orientation on how to identify the research questions 
that need to be addressed and how to communicate these to the scientific community. To facilitate 
the process, the IP created a guide for the implementation of research, which was distributed to all 
technical personnel so they can understand how and why we undertake research. Technical personnel 
then created concept notes that outline what will be explained or answered, why the evidence is 
significant and the process by which the issue will be investigated. Before creating a developed 
proposal, USAID was provided with a draft concept note to solicit additional input. The criteria that 
the IP used for selection of topics were:
• Will the research answer a relevant and important project intervention problem?
• Is the new knowledge potentially worth the cost of the research?
• Is there are clear research objectives and questions related to the TPHC project, and does 
the proposed research design answer these objectives?
• Does the research fit with a theory of change or causal chain in a coherent way? If not, 
what is the potential for generating new theories or questions?
43 | P a g e
• Will the research produce results that can be acted on in a timely way by the intended 
audiences?
• Does the research design reflect an understanding of whether the intervention is stable and 
simply replicable, or whether the intervention is expected to change?
Because the activity was carried out in collaboration with the government at all levels, data and 
program learning were shared with partners through a variety of venues and sources. These were 
perceived to have an impact on plans and policies, many of which are now owned and used by 
government. Important examples include the FP quality standards, which were tested at healthcare 
facilities and were found to be successful in raising the quality of care. Based on implementation at 
healthcare facilities, these standards were later incorporated into national health planning. 
Similarly, the Activity introduced V-Scan obstetric ultrasound in a few health centers as a pilot,
increasing service quality and usage. Later, as a prelude to wider scale up, MOH added V-Scan 
restricted obstetric ultrasonography to its HSTP-II. Other examples of changes noted by the IP 
because of Activity support include the revision of health care performance implementation 
guidelines, the creation of new job aids that took into consideration complex requirements and 
implementation of the twinning partnership model as a pilot, which enhanced partnership 
engagement. Later, to scale up twinning, the government also added this concept to the HSTP-II. 
Similarly, based on pilot projects, the community scorecard intervention was scaled up to bigger 
primary healthcare facilities. The Activity also used specific recommendations from the mid-term 
evaluation to modify implementation approaches and to improve results. For instance, a planning 
exercise on family planning (FP) was introduced based on the findings of the mid-term evaluation. 
In addition to publication in peer-reviewed journals and presenting conference abstracts, the Activity 
developed webinar presentation on such topics as:
• Leveraging maternity waiting homes to increase the uptake of immediate postpartum family 
planning in primary health care facilities
• Accelerating the performance of district health systems towards achieving universal health 
coverage via twinning partnerships
• Practices and prospects for adolescent transition to adulthood through very young 
adolescent interventions
• Health system strengthening through investment activities across woreda FP/RH care 
systems
• Life expectations in early adolescence and the impact of clubs on reproductive health 
knowledge 
• Coordinating efforts to decreased maternal death due to postpartum hemorrhages 
In addition, the activity participated in three learning forums with the other Transform activities 
organized by the Transform MELA team. They also reported using information from different sources, 
including operational research, and DHIS2 data, to monitor progress and adjust strategies. According 
to KIs, findings from random follow-up visits provided gist for discussion between program staff and 
staff in the public health system. 
“Every year, we conduct random follow up/visits to sampled woredas and health centers under these 
woredas. We used to conduct a forum and share these results for the public sector staff. Second, we 
have published much research in different thematic areas (male engagement, leadership management, 
FP, and post-partum FP). We have presented these findings in different international journals and at the 
Ethiopian Public Health Association Forum. Besides, we have been sharing all our data for different 
stakeholders found in our region. In general, we have been making continuous improvements on our 
program based on the evidence generated from our own work plan, MELA mid-term finding, random 
follow up and the evidence generated from different research findings.” (IP Regional Office, SNNP Region) 
44 | P a g e
5.5.2. How did respondents perceive that these adaptations contributed to MNCH/FP 
outcomes? 
KIs reported on several effective interventions that contributed to MNCH/FP outcomes. They 
emphasized the most important contributions made by TPHC: the leadership, management and 
governance (LMG) training, increasing the capacity of the professional staff on preparation of project 
proposal to apply for grants, the woreda twinning, the FP program planning exercises, the creation 
of linkages between hospitals and health facilities, and the provision of ultrasound machines:
“Among the best practices of the Transform program; the woreda-to-woreda twinning approach, the 
linkage created among health faculties such as health centers with health posts, again the approach 
called EPHCAQ (Ethiopia Primary Health Care Alliance for Quality) where hospitals are supporting the 
health centers, the budget allocation or the sub-grants approach, the activities done to improve the 
leadership such as leadership management system (LMS) and the woreda management standard 
trainings, the Her Space program, the custom of working in integration with the governments system, 
and generally, all the attempts made to increase accessibility, quality and continuum of care regarding 
MNCH service were all best.” (ZHD, Amhara Region) 
“The provision of ultrasound [v scan] is the first intervention that should be considered a best practice. 
In the past, ultrasound service was provided at private clinics and hospitals only, and women should travel 
a long distance to get an ultrasound service. [Because of the success at Transform intervention areas], 
other health centers not supported by the Transform PHC are also requesting us to organize and provide 
training to midwives and that they will allocate the budget needed to purchase ultrasounds. This is the 
result of the interventions and the motivation created. I have been talking this to the RHB. This is a good 
practice the program introduced.” (ZHD, Oromia Region).
5.5.3. What lessons were learned for project future design, management, and 
monitoring? 
After five years, numerous lessons emerged. In the view of the interviewed KIs, having a focused 
strategy for improving measurable MNCH/FP outcomes allowed for the success of the twinned
woredas and facilities. Selected twinned woredas contributed importantly to improving measurable 
performance-based outcomes, according to the KIs. Woredas were able to apply for sub-grants from 
other sources and improve services. Some have been recognized as model woredas, as well. 
“Those strategies that have the greatest effect on the MNCH/FP outcomes could be taken as a good 
opportunity for scale up and strengthen learning. For example: Ethiopian Public Health Institute (EPHI) 
of the Amhara region has taken twinning strategy as one of the agendas for its weekly meetings and 
contributed for MNCH/FP outcome. Through twinning, resources are shared across woredas. Example, 
through the twinning strategy woredas share V-scan and get technical support from one woreda to 
another or One PHCU to another within the woreda.” – (IP Regional Manager, Oromia Region) 
From the design point of view, having measurable targets and indicators that capture the effectiveness 
and coverage of every intervention may help inform implementation decisions. For instance, and as 
recommended below, it would help to conduct operational research to improve coverage of ANC4+ 
and barriers to FP services. It would help monitor how many woredas are able to perform according 
to standards and thus “graduate” to be able to be twinned with lower performing ones. Ideally, the 
number of high-performing woredas should increase every project year. New indicators and 
monitoring can help to track the rate of improvements across various woredas to help those where 
the rate of improvement is slowing down. 
45 | P a g e
6) CONCLUSIONS 
Contextual factors affected the capacity of the activity to achieve its targets, as evidenced by the 
significant decline in achievements since the midterm. The final performance evaluation of TPHC 
evaluated the endline outcomes in relation to the baseline and midline and in relation to the 
Activity’s intermediate results. The evaluation noted progress despite severe contextual constraints 
since the midterm and can be briefly summarized as follows: 
• IR1. Qualitative evidence indicates that the activity has improved management but not 
enough to improve the performance of the health system as shown by the decline in 
performance on MNCH/FP indicators. 
• IR2. The activity has improved the quality of services in several woredas, but this was not 
enough to overcome contextual and systemic factors that would ensure that quality 
improvements could be sustained. 
• IR3. Household and community behaviors have had limited improvement, mostly due to 
contextual factors that require further study. 
• IR 4. The activity has enhanced learning but faced with contextual headwinds, this did not
prevent the decline observed since midline. 
Despite challenges, the final evaluation demonstrated that TPHC has established innovative 
processes; helped to improve the functioning of regional public health systems and programs; and 
strengthened the leadership, management, and governance skills of staff in selected woredas and 
facilities. TPHC also demonstrated to have developed an iterative learning program that, among other 
results, helped expand CBHI and inform policy and programming. Sometimes, successful interventions 
such as training and twinning did not work as well in other woredas indicating the need to adapt 
interventions to different contexts. 
The Activity achieved these results through a combination of technical support interventions to the 
public health system. The interventions included: establishing skill labs; provision of on-the-job 
training, mentoring, and coaching; post training follow-up visits and supervision; provision of 
equipment and supplies; and sub-grants. TPHC’s technical support was reported to have improved 
quality and thereby increased coverage of high impact health interventions. Relatively few KIIs 
discussed least effective interventions—generally, these related to cases in which interventions that 
were reported as successful in some regions were less successful in others; difficulties with the health 
information system that did not allow for continuous monitoring and quality improvement was cited 
as one example. This demonstrates that despite a wide range of interventions, lack of consistent 
quality across woredas and facilities is still a roadblock to the consistent delivery of MNCH/FP 
services As described below, and likely due to contextual headwinds, household and community 
health practices and health-seeking behaviors declined, and continuity of MNCH/FP services remains 
a challenge. Nevertheless, comparisons between Transform intervention areas and non-Transform 
intervention areas were favorable:
● Contextual factors including conflict, COVID, and systemic issues negatively 
impacted outcomes. Conflict led to important regional and intra-regional variations in 
performance as measured by KPIs. Declines in KPIs at endline were, for example, most 
pronounced in the Amhara region, and particularly in conflicted areas of Eastern Amhara 
such as North Wollo, South Wollo, and North Shewa. COVID-19 disrupted many basic 
health services. Household and community health practices and health-seeking behaviors 
outcomes (IR3) also declined because of patients’ fear of acquiring the infection at health 
facilities. Stockouts of medicines and equipment and high rates of staff turnover also 
negatively impacted progress towards targets.
● Ownership and sustainability gains due to Transform interventions are at-risk 
because of high turnover of trained staff, which was compounded by the effect of 
war and insecurity. Understanding the causes of this challenge warrants further study to 
46 | P a g e
develop more effective and sustainable recovery and improvement interventions in the 
affected conflict zones.
 
● Transform PHC intervention areas had better performance for facility-based 
services at endline when compared with the non-Transform intervention areas.
As examples, outcomes for SBA, LAFP, essential newborn care either improved or remained 
the same. However, most community-based service KPIs (MCPR, immunization, basic 
sanitation, diarrhea treatment) declined in both Transform and non-Transform intervention 
areas. 
● Transform PHC contributed to improvement in performance in some key 
maternal health outcomes (SBA, early ANC coverage), and likely prevented 
deterioration in coverage of other maternal health outcomes (e.g., ANC4+).
● Transform PHC contributed to improving some newborn and child health 
outcomes. Treatment for acute respiratory infection (ARI) increased, contributing to 
improved newborn health outcomes as compared to non-intervention areas. Measles 
coverage also increased, but full vaccination and Penta 3 coverage remained low, indicating 
that immunization was not conducted as expected due to conflicts and COVID. 
● Transform PHC contributed to improving the coverage and quality of selected 
services of the PHCU’s continuum of care (IR2). These include early ANC, SBA, 
newborn healthcare, ARI treatment for children below five years of age, and LAFP use.
● LAFP use increased, whereas PPFP and MCPR performance declined. The decline 
in MCPR resulted from the decrease in use of short-acting methods, probably because of 
weak performance of health posts. 
● Continuity of MNCH/FP services remains a challenge (e.g., ANC 1 to ANC4+
dropout). Lack of counseling on the importance of continuity of care was identified as the 
main driving factor for dropouts. A key informant, for example, stated that women usually 
come to health facilities for ANC and fail to return for the next visit because of the lack of 
good quality of care. 
● Transform PHC established and implemented innovative processes (IR1.1) such as 
the twinning of woredas and FP program planning; improved the functioning of regional public 
health systems and programs in several woredas (IR 1.2), and strengthened the leadership, 
management, and governance skills of staff (1.3) in selected woredas and facilities. 
● Transform PHC demonstrated a high capacity for learning and adaptation in 
implementation (IR3.1). The activity demonstrated good practices in use of evidence from 
studies and operational research in collaboration with MOH and integrated this evidence in 
their activity planning and implementation. As one example, Transform-funded research 
informed policy and programming which facilitated the expansion of CBHI in both 
interventions and non-intervention areas (IR 3.3). 
● Gender outcomes were generally positive but require further efforts. As a positive 
outcome, the availability of post GBV interventions increased at health facilities supported by 
Transform PHC intervention when compared to non-Transform intervention areas. The 
activity was also able to increase the availability and range of adolescent and youth health 
services. However, the utilization rate and impact of such services on gender inequalities, 
such as early marriage and early pregnancy, requires further study. 
7) RECOMMENDATIONS 
47 | P a g e
Recommendations for MOH
▪ Reinforce coordination of development partners to effectively implement 
woreda transformation. HSTP II set goals for woreda transformation in the context of 
multisectoral development and a set of targets. It is essential that MOH work with
development actors to demonstrate an effective model that can be implemented in a pre￾determined number of model woredas in each region. 
▪ Revitalize community health programs. Priority should be given to rehabilitating and 
revitalizing community health programs, especially in war-affected woredas. Health posts 
should be functional and provide services on all weekdays, and various community structures 
and platforms should be utilized to mobilize the community and ensure whole-of-society 
engagement. Key informants reported that community mobilization and engagement 
platforms have weakened, and this has resulted in declines in community-based performance 
due to weak community platforms. Community health programs such as ICCM and ANC 
service exist but community mobilization and follow-up is weak due to the non-functioning 
of the Health Development Army (HDR) or similar structures.
▪ Assess woreda and health facility twinning partnership bottlenecks and make 
path correction measures before scale-up. Transform PHC adapted the twinning 
partnership strategy in Ethiopia to improve health system performance. The result of the 
twining strategy gave mixed results. While twinning was successful in some woredas and 
facilities, it did not result in significant contributions in other woredas and facilities. In some 
areas, lead woredas were unable to provide adequate technical support to their peers. In 
other areas, mentee woredas were not ready to accept recommendations from lead 
woredas. Future interventions in this initiative should properly support and strengthen both 
the roles of the lead and mentee woredas, closely monitor the proper implementation of the 
twinning program, and conduct path correction when implementation shortcomings are 
identified.
▪ Integrate FP counseling across all contacts in the continuum of care: Contraceptive 
use is low, and few women are counseled on FP during the postpartum period. Integrating 
family planning counseling during all contacts in the continuum of care (ANC, childbirth, 
postnatal care, immunization, etc.) will likely lead to increases in family planning use. 
▪ Commission supplemental operational research to explore the underlying 
factors contributing to the decline in performance of selected key performance 
indicators. Determining the root causes for the decline in performance for selected 
performance indicators requires further research. With continued support from 
development partners, the outcome of such research can inform future design and 
programming such as ANC4+, which has been declining since baseline
▪ Add and retain additional health facility personnel to ensure consistent provision 
of services, especially in difficult-to-reach areas. Staff turnover can create gaps in 
service, and the evaluation suggests that distance is a factor in affecting activity performance. 
MOH should therefore develop personnel retention plans that motivate qualified health staff 
to remain in their posts.
Recommendations for USAID
▪ Continue support for innovative approaches on social and behavior 
communication change (SBCC) to bring sustained behavioral change to 
MNCH/FP practices: The findings of this evaluation showed that coverage for key 
48 | P a g e
MNCH/FP indicators in Transform PHC regions is low despite some progress. Many women 
who participated in the survey noted that they did not visit health facilities during pregnancy 
and childbirth largely because it is still not customary for them to go for ANC visits. Many 
believe a pregnant woman does not need to have ANC visits unless there is a specific problem 
or disorder. These findings demonstrate that there are still gaps in women's awareness of 
the importance of ANC. Further work is therefore required to develop and implement 
effective SBCC approaches designed to change such perceptions.
▪ Revisit target setting methodology and assumptions. Target against 
performance should be monitored continuously by a third party for course 
correction and adjustment of targets. Most of the Transform PHC Life of Activity 
targets have not been fully achieved. Thus, continuous monitoring and timely adjustment are 
required to increase the likelihood of achieving targets. 
▪ Reinforce leadership capacity building: This evaluation pinpointed that leadership is 
critical to performance of the health system. However, leadership turnover at woreda and 
regional levels remains a challenge. Thus, support for establishing a continuous leadership and 
management program should be further strengthened
▪ Develop a dual health system strengthening strategy that 1) continues its support 
to the Health Sector Transformation Plan II (HSTPII) to keep advancing towards 
EPCMD goals, and 2) implements a recovery strategy to build resilience in 
conflict zones. A focused and targeted health system strengthening strategy that 
measurably contributes to HSTP II and strengthens the health system to recover from recent 
shocks in the most affected woredas, will allow USAID to achieve greater impact. This health 
system strengthening recommendation follows USAID’s health system strengthening strategy 
principles and the WHO's six building blocks model.21
▪ Strengthen support for health posts and health centers with additional supplies 
and equipment. Several factors have contributed to a continued deterioration of service 
delivery at the lowest levels of the health system, namely health posts. Despite challenges, 
important improvements came because of the provision of critical supplies and equipment to 
health facilities such as ultrasounds and V-scans. However, stockouts of supplies and lack of 
equipment continue to impede the effective delivery of MNCH/FP services, especially at 
health posts. Therefore, USAID should work with the MOH to prioritize supply provisions 
and to provide critical supplies and equipment.
• Improve Ownership, Sustainability and Coherence: the design of future activities 
should build ownership, sustainability, and coherent interventions should provide specific 
metrics to progressively demonstrate that woredas and facilities are “owners” and capable 
of sustaining improvements in coordination with all stakeholders and partners. 
• Continue Participatory Learning and Adapting: The current activity recognized that 
these problems require study and prioritization and that solutions to emerging problems 
should be addressed as new challenges arise. It also recognized that the continuous cycle of 
learning and adapting is a “team sport” that needs to be participatory. There are however
still many challenges and bottlenecks to the health system that vary among regions, zones, 
woredas, and facilities, including high staff turnover, that prevent the effective and efficient 
operation of health care delivery. Because each region has unique characteristics, health 
21 Using the World Health Organization health system building blocks through survey of healthcare 
professionals to determine the performance of public healthcare facilities 
https://archpublichealth.biomedcentral.com/track/pdf/10.1186/s13690-017-0221-9.pdf
49 | P a g e
system structures and resources, further operational research is needed to identify and 
monitor the factors related to staff turnover as well as family planning utilization and ANC 
4+ uptake for adults and adolescents, and the impact of woreda twinning and gender-based 
services. Similarly, although CBHI has greatly expanded, it will be important to understand 
the effectiveness of CBHI and how it contributes to expansion of access to MNCH/FP 
services. Future activities should therefore develop an operational research agenda that 
includes these chronic problems and identifies the appropriate causal pathways to address
them, taking into consideration regional differences. The expected outcome of such 
operational research should be: “An efficient health facility management model that solves 
operational problems in the supported regions, zones, woredas, and facilities.”
Recommendations for the current TPHC activity
Given that this is the final evaluation of the TPHC activity, the recommendation to the IP is to ensure 
the effective transfer and handover of its research reports, tools, and training manuals to all its GOE 
counterparts. 
50 | P a g e
ANNEXES
ANNEX 1. EVALUATION QUESTIONS AND DATA SOURCES
Key Evaluation Questions Specific and Probing Questions Design Data Source
1. How effective were 
the Transform Program 
Activities’ approaches in 
contributing to improving 
MNCH/FP outcomes?
1.1. Were Transform activities 
implemented as planned? If not, 
why not?
1.2. To what extent did the 
Transform improve MNCH/FP 
outcomes and reduce outcome 
inequities in different groups in 
intervention areas compared to 
control areas?22
• To what extent did activities 
achieve their targets? 
• Which targets were met or 
exceeded?
• Which targets were not met? 
1.3. How effective are the Transform 
Activities strategies and 
approaches? 
1.4. In your opinion which of the 
Transform Activity interventions 
are replicable or can be scaled 
up/applied to a different setting?
1.5. What is the effect of context 
(Conflict, Covid 19 pandemic, 
internal displacement) on 
Transform Activity interventions 
and achievement of intended 
results?
Mixed methods - HH respondents
- HF respondents 
and observation
- KIs from USAID, 
MOH, IPs, RHBs
- Transform IP 
reports and 
documents from
MOH
2. How did Transform 
Program Activities 
facilitate local 
ownership, 
sustainability, and 
coherence? 
2.1. What mechanisms are in place to 
ensure government ownership of 
Transform health system 
interventions? 
2.2. To what extent did government 
staff/community perceive that 
the approaches established by 
the project will be maintained by 
the government and continue to 
sustain MNCH/FP improvements 
in the next 3 and 5 years?
2.3. To what extent did the project 
improve community involvement 
and its role in accountability, 
transparency in decision-making, 
and advocacy for community 
health needs?
2.4. To what extent did the project 
improve community engagement 
Qualitative - KIs from USAID, 
MOH, IPs, RHBs
- Transform IP 
reports and 
documents from 
MOH
22 Note: We intend to measure inequities in terms of geographic variations among regions, residential locations 
(urban/rural), age, educational status, and others.
51 | P a g e
Key Evaluation Questions Specific and Probing Questions Design Data Source
for advocacy and health service 
oversight? 
• What capacity for community 
engagement did Transform build? 
2.5. How do Transform Activities 
coordinate with other 
development partners and their 
activities?
3. How did Transform 
Activities consider 
Gender dynamics in 
activity 
implementations? 
3.1. To what extent did the 
government staff/people perceive 
that the project established 
processes are functional to track 
and address gender and 
inequitable health outcomes? 
• Provide examples to substantiate 
perceptions. 
3.2. What gender equity results did 
Transform Program Activities’ 
interventions achieve?
3.3. What gaps exist in the current 
USAID/ Transform intervention 
design/programming in terms of 
addressing the key gender issues 
in the targeted regions of 
Ethiopia?
Qualitative - KIs from USAID, 
MOH, IPs, RHBs
- Transform IP 
reports and 
documents from 
MOH
4. How and to what 
extent have the 
Transform IPs been 
able to adapt 
implementation 
approaches based on 
learnings and new 
evidence? 
4.1. To what extent did the project 
make adaptive changes?
Based on what evidence? (e.g., 
responses to Transform midterm 
recommendations and IP-led 
studies)
4.2. How did respondents perceive 
that these adaptations 
contributed to MNCH/FP 
outcomes?
4.3. What lessons were learned for 
project future design, 
management, and monitoring? 
Qualitative - KIs from IPs, 
- Transform IP 
reports 
5. To what extent did 
the Transform 
program improve 
MNCH/FP outcomes 
in Transform 
intervention woredas 
compared to non￾Transform 
intervention 
woredas?
5.1. To what extent did the 
Transform program ensure 
improvement in MNCH/FP 
outcomes? 
5.2. Are there any unintended 
results (positive or negative) 
specific to Transform 
program interventions?
- HH respondents
- HF respondents 
and observation
- KIs from USAID, 
MOH, IPs, RHBs
- Transform IP 
reports and 
documents from 
MOH
52 | P a g e
ANNEX 2. DETAILED EVALUATION METHODS
DATA COLLECTION METHODS
Data Sources
The final performance evaluation used two main sources of qualitative data – document review, and 
KIIs, and drew quantitative performance data from HH survey and HF survey of the Impact Evaluation. 
A range of documents were reviewed including IP performance reports, MOH annual reports, Health 
System Transformation Program (HSTP) midterm review, Mini-Demographic, and Health Survey 
report, Transform baseline and midterm evaluation reports, and other relevant documents. The 
primary data collection involved a HH survey, HF survey, and KIIs from sampled woredas in all regions, 
except Tigray. 
Data Collection Tools
KII followed a guide to gather the views and experience of the interviewees in relation to the 
performance of the PHC Activity. The questions are broad and open-ended to gather unbiased 
answers and were accompanied of probing questions to gather more detail on the themes raised by 
the interviewee. Trained bilingual interviewers conducted the KII in the language of preference of the 
interviewee and took notes in English. The KIIs were recorded to facilitate notetaking and report 
checking. The KII notes were emailed daily or on another pre-agreed basis to the Transform MELA 
Focal person for review and quality control. Data were coded and preliminary summarize were 
performed by a team led by a qualitative analyst. 
Household and health facility surveys were conducted using pre-tested structured questionnaires. 
Structured questionnaires were uploaded to smartphones and the HH and HF data were collected 
electronically using Mobile phones. KIIs were conducted using semi-structured interview guides. 
The household survey included sections on: 
• Household and women’s demographics, household living conditions 
• Enrollment in community-based health insurance plans 
• Household decision-making practices 
• Health service uptake including family planning, antenatal care, delivery and postnatal care, and 
newborn health child immunization and child health services 
The health facility survey tool was designed to collect data on: 
• Availability of MNCH/FP services and water and sanitation facilities 
• Access to communication and power supply
• Health facility management and performance, human resources
• Gender responsiveness 
• Provision of selected family planning, antenatal care, delivery and postnatal care, immunization 
services, and adolescent and youth health services
Data Collector Training and Quality Assurance
To assure high-quality data, a day and half training was conducted for qualitative data collectors and a 
three-day training for all HH data enumerators and supervisors. Enumerators who would administer 
HH surveys had at least a BA/BSc degree, were fluent in both the local language and English, had prior 
experience collecting similar data, and understood the culture and traditions of the community they 
were visiting. Supervisors, qualitative data collectors, and those who administered the HF surveys had 
an MA/MSc in the health or social science field and previous experience in similar research activities. 
Given that the respondents were mothers of reproductive age, and to ensure that they could 
comfortably speak with interviewers, the field team took gender balance into account.
The training covered topics such as research ethics, rights of human subjects during research, sampling 
procedures, informed consent, data collection tools, interviewing techniques, data handling, security 
53 | P a g e
and quality, and gender considerations during data collection. The structure of the training included a 
review of survey instruments and role-play. Once the data collectors completed the training on the 
paper-based data collection tools, they were trained on how to collect data using the electronic data 
collection template and how to upload the collected data onto the server. 
The survey tools were pre-tested by data collectors in the field in all languages at the end of the 
training, that is, before the commencement of the actual data collection, to ensure consistency with 
the baseline and midline evaluation. 
Before traveling to each of the selected woreda, supervisors and coordinators communicated with 
regional and local leaders about the evaluation. To ensure the quality of collected data, supervisors 
conducted spot-checking, and reinterviewing, especially at the beginning (first 3 days) of the data 
collection. Moreover, supervisors also reviewed a sample of completed questionnaires daily before 
uploading them onto the server. All KIIs were audio-recorded, and interviewers’ notes were translated 
to English. Five percent of the transcriptions were checked against the audio file for accuracy. 
SAMPLING
Site Selection for Qualitative Data Collection 
All regions were included, except Tigray, and a purposive sample of zones and woredas was developed 
to gather qualitative data. Quantitative data were also collected in all visited zones. 
In order to measure effectiveness of the Transform PHC Interventions for IRs 1, 2, 3 and 4, woredas 
were selected based on their performance classification: high-, moderate,- and low-performing using 
the results of the Greatest Impact Assessment (GIA), which used a select set of 12 MNCH/FP 
performance indicators from HMIS 2016 that has been used for high level monitoring. 23
The purpose of the GIA is to categorize woredas based on their recent RMNCH performance data 
and prioritize woredas where the results of the PHC interventions could be evaluated. It was also a 
way to measure progress by analyzing the number of woredas that have moved up to a higher 
performance level and compare it to the initial percentage in each category. 
Simple random sampling was used to select woredas in the Transform PHC targeted regions, with 
other development partner intervention data in the five selected MNCH/FP service areas: Maternal 
health, Child Health, Neonatal, Family Planning, and Immunization, and woredas that have GIA 
performance categorization information. Table 1 (below) shows the sampled woredas by GIA 
performance. 
Distribution of sampled woredas by GIA performance.
Region Low performance Moderate performance High Performance Total
Amhara 8 12 4 24
Oromia 12 18 5 35
SNNP 7 11 3 21
Total 27 41 12 80
A total of 80 woredas were selected for the performance evaluation. All intervention woredas that 
received three or more interventions from other development partners were excluded. All woredas 
with no GIA woreda performance categorization information and woredas in Sidama Region, which 
23 Source: USAID GIA analysis summary (2016). Transform MELA used the woreda performance 
categorization in the GIA analysis to ensure that there will not be sample selection bias for the endline 
evaluation between Transform program intervention and non-Transform intervention woredas. 
54 | P a g e
does not have data on development partners were excluded from the sampling frame. Conflict-affected 
woredas were excluded due to security reasons. 
Sampling of Households 
The household (HH) survey employed a three-stage cluster sampling technique. The first stage was a 
random selection of kebeles (wards) in the Transform intervention woredas as primary sampling units, 
drawing from the list of kebeles in the respective regions stratified by administrative zones and 
performance before 2017. Thus, woredas, where the randomly selected kebeles are located, were 
automatically included in the evaluation. The second stage of the sampling entailed the random 
selection of Gotts from the selected kebeles. Because of the difficulties and time-consuming exercise 
of developing a fresh list of households in a kebele, 2-3 Gotts were selected randomly from each 
kebele, to generate an average list of 150 households from those selected Gotts. 
The third stage of the sampling was a selection of HHs from the randomly selected Gotts. A fresh list 
of HH for each selected Gott served as a frame to select 30 HHs per kebele. The key eligibility 
criterion for selecting the HHs was the availability of a woman aged 15-49 in the HH, regardless of 
marital status, who has residential status there for at least six months before the date of data 
collection. If there were more than one such eligible respondent in a selected household, the 
enumerator used a Kish grid method to randomly select one respondent among the eligible women. 
Sampling for Health Facility Survey
Purposive sampling was employed to select zones and woredas for KIIs, with the following additional 
criteria:
● Zones and woredas in the sample were also selected for the HH survey. However, one 
exception was made to ensure representation (Silte Zone of SNNP was selected for KII even 
though the zone was not part of the Household Survey).
● Heterogeneity was one of the criterium to select woredas (We avoided selection of more 
than one woreda in a zone to ensure heterogeneity of woredas)
● All zones and woredas with security concerns were excluded from the sample.
● Resource implications were considered in the sampling process (even though heterogeneity 
of woredas was one of the criterium employed to select woredas; we balanced the distances 
between woredas and zones to reduce resource requirements during data collection).
All of the primary health care facilities designated to serve the population of the sampled kebeles -
health posts (HPs), health centers (HCs), and primary hospitals - were identified and included in the 
final performance evaluations. That is, the health post serving the population in the sampled kebele 
was automatically included in the study, the catchment HC linked with the selected health post was 
also included and the primary hospital attached to the selected HC, if any, was also included. Primary 
hospitals were included whenever there are such facilities in the selected woredas. 
Selection of Key Informants 
KIs were enrolled using purposive sampling. Gender and representation of various stakeholder groups
were considered in the selection process: USAID, central and regional IPs, relevant directorates at 
MOH, RHBs, ZHDs, selected WorHO, and selected HFs. A total of 65 KIIs were conducted as part 
of the performance evaluation (See Table 14 for distribution of KIIs by stakeholder and gender). 
Proposed distribution of KIIs by stakeholder type across all Transform evaluations
Stakeholders Key Informants
USAID/Ethiopia 1 KI 
Ministry of Health 7 KIs from MOH
− MOH directorates –
55 | P a g e
Stakeholders Key Informants
Note: KIs from four directorates and 
one MOH agency will be interviewed 
on both PHC and HDR Activities. 
Therefore, the total number of KIs at 
MOH will be Seven (five directorates 
and two MOH agencies). 
− Health Extension Program and Primary Health Care, 
Maternal, Child Health and Nutrition, Policy Plan Monitoring 
and Evaluation, and Women, Children and Youth Affairs
− Human Resources for Health Development 
Directorate
− MOH agencies (Ethiopian Health Insurance Agency 
and Pharmaceuticals Supply Agency)
IPs central office 2 KIs 
IP’s Regional offices 4 KIs – in four regions
Regional Health Bureaus – MCH unit 4 KIs (1 KI per region) MNCH Department 
Zonal Health Department 9 KIs (1 KI per zone)
Woreda health offices − Transform Intervention woredas (9): 18 KIs – Heads 
of Woreda Health Offices and MNCH officers (2 KI per 
woreda) 
Health Facility – HC 20 KIs (1 per HC) 
Total 65 KIIs
56 | P a g e
ANNEX 3. DATA ANALYSIS
All HH and HF survey data were cleaned, checked, and validated to identify and address any issues 
during data collection, a process that was undertaken directly by supervisors and data collectors. Once 
data collection was complete, an intensive cleaning process was conducted that included coding and 
recoding, consistency checks, addressing refusals and nonresponses, and validating contents.
Quantitative Data Analysis 
Descriptive (Frequency and percentage calculations) analysis was conducted to describe the 
characteristics of evaluation participants; and to estimate a select set of Transform program 
intervention indicators. The analysis of HH data also focused on comparisons of final performance vs. 
baseline key performance indicator (KPI) values. Data management and analysis were done using 
Outcome Performance Measures
The evaluation used the Results Framework of Transform PHC. The Results Framework was used to 
identify key performance indicators (KPIs) and design qualitative and quantitative data collection tools 
to measure achievement of the IRs and sub-IRs. A results-indicator matrix was used to link the 
key performance indicators to measure intermediate results (IR) areas. The key performance 
indicators considered in this analysis are:
• Family planning indicators: contraceptive prevalence rate (CPR), long-acting (LA) 
contraceptive methods use, postpartum contraceptive use, and unmet need for family planning 
(FP), 
• Maternal health indicators: (antenatal care (ANC), skilled birth attendance (SBA), 
postnatal care (PNC), iron-folic acid supplementation (IFS), cross-cutting issues (basic 
sanitation, spouse accompany to ANC, spouse accompany to delivery, and community-based 
health insurance (CBHI), and 
• Child health indicators: (measles vaccine (MCV), exclusive breastfeeding). 
Number of Woredas enrolled by Transform PHC in the two activity phases.
Regions Phase I (FY 2017) Phase II (2018) Total 
Amhara 76 18 94
Oromia 133 29 162
Sidama 13 1 14
SNNP 98 13 111
Total 320 61 381
Qualitative Data Analysis
Coding: A codebook was developed based on evaluation questions and the interview guides. In 
addition, interview transcripts were reviewed, and the codebook was further refined as specific 
themes emerged. Next, the codes were uploaded to the qualitative analysis software by the qualitative 
data analyst. Then, the interview transcripts were imported to the data analysis software and then 
thematic coding and summarization of preliminary KII findings was conducted. 
Analysis: Thematic analysis was conducted using Atlas.ti, a qualitative data analysis software. Selection 
of themes and the analysis of the data was framed by the evaluation questions. In addition, the 
evaluation team used the results framework of the PHC activity to understand and assess whether the 
intervention theory of change had held in the implementation of activities, and whether there were 
explanatory contextual factors at play. Various qualitative information was synthesized to understand 
factors facilitating or hindering implementation as well as achievement of results. Other analytical 
methods employed included a comparison of specific themes between Transform intervention and 
non-Transform intervention woredas. 
57 | P a g e
ANNEX 4. PHC TRANSFORM ACTIVITY RESULTS FRAMEWORK
Goal: Healthy, productive, prosperous Ethiopians
Objective: Health Sector Transformational Plan's agenda and EPCMD achieved
R1: Improved 
management and 
performance of health 
systems
R2: Increased 
sustainable quality of 
service delivery across 
the PHCU’s continuum 
of care
R3: Improved 
household and 
community health 
practices and health￾seeking behaviors
R4: Enhanced program 
learning to impact 
policy and 
programming related 
to ending preventable
child and maternal 
deaths (EPCMD)
R1.1 Established 
and strengthened 
innovative 
processes to 
sustainably 
enhance health 
system 
management and 
performance
R1.2 Enhanced 
functionality of 
the health system 
within the 
context of 
primary level care
R1.3
Strengthened 
transformational 
leadership, 
governance, and 
management at 
the woreda and 
PHCU level
R2.1
Strengthened 
skills for delivery 
of quality and 
integrated 
RMNCAH-N 
services
R2.2 Improved 
provider 
behaviors and 
communication 
skills toward a 
compassionate, 
respectful, and 
caring health 
workforce
R2.3 Improved 
management of 
health service 
delivery and 
oversight of 
service quality
R2.4 Innovative 
service delivery 
interventions to 
impact EPCMD; 
introduced and 
scaled up
R3.1 Increased 
individual- and 
household-level 
care-seeking 
behavior and 
uptake of healthy 
practices
R3.2
Strengthened 
enabling 
environment for 
health-seeking 
behavior, 
including 
community 
engagement in 
health service 
oversight
R4.1
Strengthened 
health system 
capacity to 
generate learning 
and evidence
R4.2 Evidence of 
what works in 
EPCMD informed 
by results from 
program learning 
and iterative 
adaptation
R4.3 Evidence 
utilized to inform 
programming and 
policy with local 
and global 
stakeholders
58 | P a g e
ANNEX 5. GLOSSARY OF TERMS
Terms Definition
Adaptive management An intentional approach to making decisions and adjustments in response to 
new information and changes in context. 
Adaptive management integrates approaches throughout the Program Cycle as the 
second of four core principles. (Program Cycle Operational Policy. ADS 201.6; 
09/21/2021).
Case Study For the purpose of the Transform Program evaluation, we defined a case 
study as the study of public health problem-solution situation and the 
process to implement, evaluate, and learn from it. All the interventions 
selected to solve the problems identified by the PHC and HDR activities 
have demonstrated their efficacy elsewhere to prevent or treat disease, 
death, or disability. They had just not been adapted to be implemented in 
the Primary Health Setting in Ethiopia and are therefore considered 
innovations. 
Coherence The compatibility of the intervention with other interventions in a country, 
sector or institution. 
Collaboration with the 
Ministry of Health 
Collaboration is a practice whereby individuals, communities and 
organizations work together for a common purpose to achieve a shared 
benefit. For the purposes of this evaluation, collaboration between MOH 
and the Transform program encompasses working relations between 
central bureaus, regional, zonal and woreda health offices. 
Effectiveness The extent to which the intervention achieved its objectives and results, 
including any differential results across groups.
Gender equity Fairness of treatment for women and men, according to their respective 
needs. 
In the development context, a gender equity goal often requires built-in 
measures to compensate for the historical and social disadvantages of 
women. (USAID Gender Terminology)
Health care quality The degree to which health care services for individuals and populations 
increase the likelihood of desired health outcomes and are consistent with 
current professional knowledge. (Institute of Medicine: Crossing the Quality 
Chasm, 2001)
Health workforce 
capacity building
Actions intended to improve human resources for health, contributing to 
healthy lives and well-being, effective universal health coverage, resilience 
and strengthened health systems at all levels. 
WHO (2016:4). Global strategy on human resources for health: Workforce 
2030
Impact The extent to which the intervention has generated or is expected to 
generate significant positive or negative, intended or unintended, higher￾level effects.
Intervention A process developed with the objective of improving a health outcome. 
Interventions can be classified as preventive interventions such as vaccines, 
or curative, such as an improved diagnostic or treatment protocol or a more 
effective and efficiently streamlined service delivery, or both preventive and 
curative interventions such as CEmONC that detects and treats 
complications in a timely manner, thus preventing morbidity and mortality. 
Medical equipment and 
supplies provision
Medical equipment is defined as a capital equipment and durable items that 
last for several years. Medical supplies are items that need replacing on a 
routine basis, including disposables and single use items. 
Provision encompasses procurement, delivery, and management of 
equipment and supplies based on requirements for preventive care, 
diagnostic tests, and treatment. 
59 | P a g e
Terms Definition
Non-Transform 
intervention woredas 
Woredas which do not receive USAID investment through the Transform 
Program.
Ownership The extent to which national partners have a stake in the implementation 
of programs and activities. 
Performance 
improvement
The goal or benefit of focusing on individual and organizational change and 
business results. 
Caiola and Sullivan for USAID, Paper #9, page 2 (2000). Performance 
Improvement: Developing a Strategy for Reproductive Health Services.
Performance monitoring 
indicators 
Measures intended to detect progress towards the results included in a 
Results Framework. (USAID: Performance monitoring indicators)
Performance standards 
for public health 
Organizational or system standards, targets, and goals to improve public 
health practices. (Public Health Foundation: Performance Standards)
In the Ethiopian context, this refers to performance standards at different 
levels of the health system. These are included in references such as the 
Ethiopian Hospital Services Transformation Guidelines (EHSTG).
Public financial 
management
The set of laws, rules, systems, and processes used by sovereign nations 
(and sub-national governments) to mobilize revenue, allocate public funds, 
undertake public spending, account for funds, and audit results. 
Lawson (2015). Public Financial Management
Relevance The extent to which the intervention objectives and design respond to 
beneficiaries, global, country, and partner/institution needs, policies, and 
priorities, and continue to do so if circumstances change. 
Solution A solution is a set of interventions that health managers have selected to 
solve a healthcare delivery or public health program problem. Solutions are 
usually implemented by development projects with donor support to 
improve the quality of healthcare and expand utilization and access to 
services and thus improve health outcomes. In the case of The Transform 
Program, both activities have implemented numerous solutions that have 
demonstrated improved outcomes and impact on the health of mothers and 
children in Ethiopia.
Sustainability The extent to which interventions and their benefits will continue beyond 
the life of the program. 
Transform intervention 
woredas
Woredas in which the USAID Transform Program implemented through the
Transform PHC and HDR Activities.