USAID/GLOBAL HEALTH EVALUATION AND LEARNING SUPPORT ACTIVITY (GH EvaLS) USAID ORGANIZED NETWORK OF SERVICES FOR EVERYONE’S HEALTH (ONSE) IN MALAWI ENDLINE PERFORMANCE EVALUATION APRIL 2023 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by ME&A, Inc., and the evaluation team of Martha Benezet, Tim Clary, Joseph Inungu, Shabnam Shahnaz, and Robert Waswaga. GH EvaLS Malawi ONSE Endline Performance Evaluation / 1 USAID Contract No. GS-10F-154BA/7200AA20M00003 Period of Performance: February 22–June 30, 2022 Submitted by: Andrea Camoens, Chief of Party GH EvaLS 4350 East-West Highway, Suite 210 Bethesda, MD 20814-4410 Tel: (301) 652-4334 Email: acamoens@engl.com Cover Photo: Hector Malaidza: A focus group discussion session with Men Champions in Karonga District DISCLAIMER The views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. This document is available online. Online documents can be located on the GH EvaLS website at https://ghevals.meandahq.com. Documents are also made available through the Development Experience Clearinghouse (http://dec.usaid.gov). Additional information can be obtained from: Global Health EvaLS 4350 East-West Highway, Suite 210 Bethesda, MD 20814-4410 Tel: (301) 652-4334 Website: https://ghevals.meandahq.com GH EvaLS Malawi ONSE Endline Performance Evaluation / 2 ABSTRACT PROGRAM PURPOSE/DESCRIPTION GH EvaLS conducted an endline performance evaluation of the USAID-funded Organized Network of Services for Everyone’s Health (ONSE) Activity in Malawi, which was implemented from November 15, 2016, to June 30, 2022. ONSE’s purpose was to support efforts by the Government of Malawi (GOM) to reduce maternal, newborn, and child mortality in 16 districts and bring essential health care services to more than half of Malawi’s population. This evaluation sought to assess the extent to which the Activity achieved its objectives, provide insights into factors that facilitated and limited increased service utilization, and document progress made toward building the capacity of the GOM to deliver quality health services. KEY QUESTIONS 1. To what extent did the Activity's interventions achieve their intended results? 2. To what extent did ONSE improve access to priority high-impact Essential Health Package interventions in the target health facilities and community service delivery points? 3. What are the most significant accomplishments, best practices, and lessons learned from the ONSE Activity? 4. What were the contextual factors such as socioeconomic factors, gender, demographic factors, environmental characteristics, baseline health conditions, health services characteristics, and so forth that affected implementation and outcomes of the ONSE Activity? METHODOLOGY The evaluation used a mixed-methods approach. The evaluation team conducted a desk review of materials that provided secondary quantitative data for analysis, along with a limited amount of analysis from two primary data sets. Qualitative data came from interviews with key informants, focus group discussions, and group interviews for a total of 89 individuals participating. Qualitative data is subjective and susceptible to a number of biases; the team sought to mitigate and minimize this bias through the use of multiple data sources, a robust and systematic analysis of information collected, and triangulation of results. KEY FINDINGS, CHALLENGES AND MITIGATION STRATEGIES In general, ONSE achieved its intended results, including improving access to high-priority services, with most of targets met or exceeded. However, it fell short in several other areas (e.g., nutrition interventions, couple-years of protection, etc.). Regardless, participants appreciated ONSE’s efforts, especially interventions that were aimed at health systems strengthening and specific, measurable, achievable, realistic, and timed (SMART) capacity building. These were noted as ONSE’s main accomplishments and best practices. A number of challenges arose during ONSE’s implementation, some of which were within its manageable interests (e.g., gaps in early collaborative efforts) and some of which were unforeseen and outside its ability to manage (e.g., the COVID-19 pandemic). ONSE mitigates many of these issues possible, but it remains to be seen whether the implemented interventions will be sustained. KEY RECOMMENDATIONS Because this was an endline evaluation, recommendations are structured around potential future activities. They include ensuring the use of a health systems strengthening approach, a focus on capacity GH EvaLS Malawi ONSE Endline Performance Evaluation / 3 building, and that potential participating facilities are adequately assessed prior to implementation. Further, efforts should focus on better communication with partners, ensuring that sustainability and accountability are addressed in design, and that communities are well-engaged at all stages. GH EvaLS Malawi ONSE Endline Performance Evaluation / 4 TABLE OF CONTENTS ABSTRACT ................................................................................................................................... 2 ACKNOWLEDGMENTS.............................................................................................................. 6 ACRONYMS AND ABBREVIATIONS ....................................................................................... 7 EXECUTIVE SUMMARY.............................................................................................................. 8 1. EVALUATION PURPOSE AND EVALUATION QUESTIONS..................................... 14 1.1. ASSIGNMENT PURPOSE ............................................................................................... 14 1.2. ASSIGNMENT QUESTIONS .......................................................................................... 14 1.3. ASSIGNMENT AUDIENCE............................................................................................. 15 2. BACKGROUND................................................................................................................... 16 2.1. RESULTS FRAMEWORK ................................................................................................ 16 2.2. GEOGRAPHIC COVERAGE ........................................................................................... 17 3. METHODS AND LIMITATIONS....................................................................................... 19 3.1. METHODOLOGY............................................................................................................. 19 3.2. DATA SOURCES.............................................................................................................. 19 3.3. SITE AND SAMPLE SELECTION................................................................................... 21 4. DATA ANALYSIS................................................................................................................ 22 5. ETHICAL CONSIDERATIONS ......................................................................................... 23 5.1. HUMAN SUBJECT PROTECTION ................................................................................ 23 6. LIMITATIONS ..................................................................................................................... 24 6.1. RELIABILITY AND VALIDITY ....................................................................................... 24 6.2. QUALITATIVE APPROACH .......................................................................................... 24 6.3. QUANTITATIVE DATA FROM PROJECT REPORTING............................................ 24 6.4. SELECTION AND REPRESENTATION BIASES.......................................................... 24 6.5. RESPONSE BIAS .............................................................................................................. 24 6.6. EVALUATION SUBSEQUENT TO PROJECT CLOSE................................................ 25 7. FINDINGS ............................................................................................................................ 26 7.1. EVALUATION QUESTION 1 ......................................................................................... 26 7.2. EVALUATION QUESTION 2 ......................................................................................... 30 7.3. EVALUATION QUESTION 3 ......................................................................................... 36 7.4. EVALUATION QUESTION 4 ......................................................................................... 41 8. CONCLUSIONS.................................................................................................................. 44 9. RECOMMENDATIONS ...................................................................................................... 46 9.1. HIGH PRIORITY............................................................................................................... 46 9.2. FOR CONSIDERATION .................................................................................................. 47 ANNEX 1: SCOPE OF WORK............................................................................................... 48 GH EvaLS Malawi ONSE Endline Performance Evaluation / 5 ANNEX 2: DATA COLLECTION TOOLS ........................................................................... 85 ANNEX 3: LIST OF DOCUMENTS REVIEWED ............................................................... 105 ANNEX 4: QUALITATIVE RESEARCH ACTIVITIES....................................................... 107 ANNEX 5: LENGTH-OF-ACTIVITY DATA INDICATOR TARGETS AND RESULTS BY USAID IR AND SUB-IR......................................................................................................... 108 ANNEX 6: SMART CAPACITY BUILDING, HSS, AND CONTEXTUAL FACTORS ACCOMPLISHMENTS.......................................................................................................... 119 ANNEX 7: DISCLOSURE OF ANY CONFLICTS OF INTEREST ................................... 122 ANNEX 8: EVALUATION TEAM........................................................................................ 128 FIGURES Figure 1: ONSE Results Framework ......................................................................................... 17 Figure 2: Map of ONSE Interventions ....................................................................................... 18 Figure 3: Package of Services..................................................................................................... 18 Figure 4: Provision of Oxytocin for Emergency Obstetric Care ............................................ 27 Figure 5: Under-1 Childhood Vaccinations............................................................................... 27 Figure 6: Women Making More than Five ANC Visits ............................................................ 28 Figure 7: Early ANC Visits.......................................................................................................... 28 Figure 8: Newborns Resuscitated (ONSE Versus Non-ONSE Districts)............................... 32 Figure 9: Newborns Resuscitated via Airway Cleaning and Stimulation............................... 33 Figure 10: Newborns Resuscitated via Bag and Mask Ventilation.......................................... 33 Figure 11: Basic Conceptualization of SMART Capacity Building Approach........................ 36 TABLES Table 1: Evaluation questions and sub-questions..................................................................... 14 Table 2: Evaluation sampling for KIIs, FGDs, and GIs ............................................................. 20 Table 3: Indicator performance success rate by ONSE intermediate results....................... 26 Table 4: Top Interventions that Contributed Most to ONSE Achieving Its Results ............ 29 Table 5: Top Three Interventions that Increased Demand for Health Services per KIIs.... 29 Table 6: Top Five Interventions that Improved the Quality of EHP Services, per KIIs....... 31 Table 7: LOA Changes in Provision of Selected EHP Services............................................... 31 Table 8: Interventions Affecting Systems-Level Issues at the MOH Central and District Levels............................................................................................................................................ 34 Table 9: Top 10 HSS Contributions by ONSE, per KIIs.......................................................... 35 Table 10: Top Three Most Significant Accomplishments of ONSE, per KIIs ....................... 37 Table 11: Top Five Best Practices that Emerged from ONSE, per KIIs................................ 37 Table 12: Top Five Practices for Ensuring Activity Sustainability, per KIIs .......................... 39 Table 13: Five Greatest Challenges for Ensuring Sustainability, per KIIs ............................. 40 Table 14: Challenges that ONSE Could Have Mitigated......................................................... 40 Table 15: Contextual Factors Addressed by the ONSE Activity ........................................... 42 Table 16: Top Five Contextual Factors that Constrained ONSE, per KIIs ........................... 43 GH EvaLS Malawi ONSE Endline Performance Evaluation / 6 ACKNOWLEDGMENTS The evaluation team of the USAID-funded Organized Network of Services for Everyone’s (ONSE) Health Activity in Malawi acknowledges the support received for this assignment from the following: • Representatives from Malawi’s Ministry of Health who showed support for the Activity and provided useful insights from the government’s perspective; • USAID/Malawi team members who provided insights and guidance from the donor’s perspective; and • The ONSE team and implementing partners in the field who gave valuable time to answer questions and provided support through program documents. The team is also grateful to the GH EvaLS team for their assistance and direct support during the preparation and field implementation of this evaluation and preparation of this report. The team thanks all the staff of the local health facilities that were visited, as well as the local resource persons and groups contacted. They graciously gave their time to respond to interview questions, share experiences and concerns, and make suggestions for future improvements. Finally, the team expresses its deepest appreciation to all those who generously reviewed the findings, conclusions, and recommendations of this report and provided feedback. GH EvaLS Malawi ONSE Endline Performance Evaluation / 7 ACRONYMS AND ABBREVIATIONS Acronym/ Abbreviation Definition AMEP Activity Monitoring and Evaluation Plan ANC Antenatal Care CHAG Community Health Action Group COVID-19 Coronavirus Disease 2019 DHIS2 District Health Information System 2 DHO District Health Office (now the Director of Health and Social Services) EHP Essential Health Package FGD Focus Group Discussion FHP Family Health Package FP Family Planning GH EvaLS Global Health Evaluation and Learning Support Activity GI Group Interview GIZ Deutsche Gesellschaft für Internationale Zusammenarbeit (German Development Agency) GOM Government of Malawi HIS Health Information Systems HMIS Health Management Information System HSS Health Systems Strengthening HSSP Health Sector Strategic Plan IP Implementing Partner IR Intermediate Result ISS Integrated Supportive Supervision KII Key Informant Interview LMIS Logistics Management Information System LOA Life of Award/Activity MCHN Maternal and Child Health and Nutrition MMR Maternal Mortality Ratio MOH Ministry of Health MSH Management Sciences for Health NGO Non-governmental organization ONSE Organized Network of Services for Everyone’s Health PENTA III Pentavalent Vaccine Third Dose PY Project Year RH Reproductive Health RMNCH Reproductive, maternal, newborn, and child health SMART Specific, Measurable, Achievable, Realistic, and Timed USAID United States Agency for International Development USG United States Government WASH Water, Sanitation, and Hygiene YFHS Youth-Friendly Health Services GH EvaLS Malawi ONSE Endline Performance Evaluation / 8 EXECUTIVE SUMMARY INTRODUCTION The Global Health Evaluation and Learning Support (GH EvaLS) Activity conducted an endline performance evaluation of the United States Agency for International Development (USAID)-funded Organized Network of Services for Everyone’s (ONSE) Health Activity in Malawi (also referred to as “the Activity”), implemented from November 15, 2016, to June 30, 2022. ONSE’s overall purpose was to support Government of Malawi (GOM) efforts to reduce maternal, newborn, and child mortality in 16 districts and bring essential health care services to more than half of Malawi’s population. The aim of the evaluation was to assess the extent to which Activity objectives were achieved regarding the quality of and access to care, provide in-depth insights into the factors that facilitated and limited increased service utilization at each level of service delivery, and document progress toward building the capacity of the GOM to deliver quality health services. In addition, the evaluation helped identify promising best practices and lessons learned. The evaluation findings will inform the Malawi Ministry of Health (MOH), USAID, and other development partners in the design and prioritization of future investments to strengthen Malawi’s health system. It will also feed into adaptations of current activities, as appropriate. The target audiences for the ONSE Activity performance evaluation include: 1. The USAID Malawi Mission team; 2. GOM MOH units involved in prevention and care services at all levels; 3. Donors, collaborators, and partners; 4. Health care providers, managers, and beneficiaries (including clients, community members, and other stakeholders) at the country level; and 5. ONSE’s Activity staff and implementing partners (IPs): prime implementer Management Systems for Health (MSH) and sub-award implementers. ACTIVITY BACKGROUND Although maternal and child health has improved in Malawi, the country faces several challenges. According to the Demographic and Health Survey1 report (2015–2016), knowledge of family planning (FP) is almost universal in Malawi, with 98 percent of women and nearly 100 percent of men ages 15–49 knowing at least one method of contraception. Modern contraceptive use by currently married women has increased steadily during the last 25 years, from 7 percent in 1992 to 58 percent in 2015–2016. The total fertility rate declined markedly, from 6.7 children in 1992 to 4.4 in 2015–2016. The proportion of women aged 15–49 in Malawi who received antenatal care (ANC) from a skilled provider rose from 90 percent in 1992 to 95 percent in 2015–2016. Since 1990, the maternal mortality ratio (MMR) decreased from 1,100 maternal deaths per 100,000 live births to 439 per 100,000 in 2015–2016. The percentage of children aged 12–23 months who received all basic vaccinations declined from 82 percent in 1992 to 64 percent in 2004 before rebounding to 81 percent in 2010 and then dropping again to 76 percent for 2015–2016. Of children under age 5, 37 percent are stunted (short for their age), 3 percent are wasted (thin for their height), 12 percent are underweight (thin for their age), and 5 percent are overweight (heavy for their height). The number of children dying before reaching their fifth birthday 1 For a full list of documents reviewed, see Annex 3, List of Documents Reviewed. GH EvaLS Malawi ONSE Endline Performance Evaluation / 9 declined from 234 deaths per 1,000 live births in 1992 to 63 deaths per 1,000 in 2015–2016. The infant mortality rate dropped to 31 deaths per 1,000 live births in 2021,2 down from 234 deaths per 1,000 live births in 1992. ONSE supported the GOM to provide essential health care services across 16 districts3 in several priority health areas—maternal and child health and nutrition (MCHN); malaria; FP and reproductive health (RH); nutrition; and water, sanitation, and hygiene (WASH). The population covered by these interventions was between 11 million and 12 million.4 ONSE’s overall purpose was to support GOM efforts to reduce maternal, newborn, and child mortality through four main objectives: 1. Increasing access to priority health services; 2. Improving the quality of priority health services; 3. Strengthening the performance of health systems; and 4. Increasing demand for priority health services. ONSE was implemented by MSH and its partners Banja La Mtsogolo, Development Innovations Group, Dimagi, Overseas Strategic Consulting, Ltd., and VillageReach. ASSIGNMENT PURPOSE AND KEY QUESTIONS The overall purpose of this endline performance evaluation is to assess the extent to which Activity objectives have been achieved regarding quality of and access to care; provide in-depth insights into the factors facilitating and limiting increased service utilization at each level of service delivery, and document progress made toward building the GOM’s capacity to deliver quality health services. USAID posed four primary questions (with several sub-questions) that guided this evaluation’s inquiry and that were presented as specific objectives of the assignment: 1. To what extent did the Activity's interventions achieve their intended results? 1a. What changes, if any, were made to activities, and how did those changes positively or negatively affect the project’s achievement of its results (e.g., gaps and opportunities in retention and training of health workers)? 2. To what extent did ONSE improve access to priority high-impact Essential Health Package (EHP) interventions in the target health facilities and community service delivery points? 2a. Of the capacity strengthening approaches employed by ONSE, which have most enabled the MOH to address systems-level issues at central and district levels? 3. What are the most significant accomplishments, best practices, and lessons learned from the ONSE Activity? 3a. How were these best practices systematically scaled up? 3b. What strategies, approaches, or methodologies did the Activity use to ensure sustainability? 2 https://data.unicef.org/country/mwi/ 3 The 16 districts were Balaka, Chikwawa, Chitipa, Dowa, Karonga, Kasungu, Lilongwe, Machinga, Mangochi, Mchinji, Mulanje, Nkhata Bay, Nkhotakota, Ntcheu, Salima, and Zomba. 4 The 2018 Malawi Census Preliminary Report estimated the total population for the 16 districts at 11,687,285. GH EvaLS Malawi ONSE Endline Performance Evaluation / 10 3c. How were scale-up activities monitored and measured, and what level of sustainability was achieved? 3d. What were the challenges faced and how were they addressed/overcome? 4. What were the contextual factors, such as socioeconomic factors, gender, demographic factors, environmental characteristics, baseline health conditions, health services characteristics, and so forth that affected implementation and outcomes of the ONSE Activity? 4a. What key findings emerged? 4b. What were the challenges faced and how were they overcome? 4c. What more can be done in future activities of similar nature? METHODS AND LIMITATIONS This endline performance evaluation used a mixed-method design (mostly qualitative, as well as quantitative) to produce evidence and insights to answer the evaluation questions. Qualitative data comprised the majority of the analysis, drawing on reviews of documents (including Activity reports), planning meetings, key informant interviews (KIIs), focus group discussions (FGDs), and Mission debrief meetings. At the national level, the evaluation team collected data in person from representatives of USAID/Malawi; the GOM MOH; ONSE and its IPs, and other agency representatives working in maternal and neonatal health, the Family Health Package (FHP), supply chain management, and other relevant areas. At the district level, officers and representatives from the public sector provided insights into health system strengthening (HSS) efforts, including supply chain systems and capacity building strategies. Likewise, health service providers from the public and private sectors participated in interviews regarding training and other support received through the Activity; they also offered perspectives on improvements in access to and quality of services for the Malawian population. At the community level, adult male and female clients, as well as youth clients of FHP services, were divided into homogeneous groups for discussions related to their health knowledge, experience of care, and community efforts to create demand and mobilize change. Finally, different types of service providers were interviewed to learn about service delivery to various clients. The evaluation team conducted 65 KIIs in nine evaluation districts and Lilongwe with representatives of major public and private institutions, communities, and other stakeholders involved in the ONSE Activity. The team also organized 13 FGDs and 11 group interviews (GIs) with district officials and community members in four districts. (For details, see Annex 4, Qualitative Research Activities). Quantitative data was derived from both Activity monitoring data and the Digital Health Information System 2 (DHIS2). The evaluation team employed several methods for qualitative data analysis (e.g., coding and thematic identification and synthesis) and quantitative data analysis (e.g., longitudinal and comparative). Findings were triangulated and used to populate a findings, conclusions, and recommendations matrix. While there were several potential limitations due mainly to the qualitative nature of the data, the evaluation team believes it has mitigated those issues through a variety of analytical techniques and the triangulation of its findings through multiple sources. GH EvaLS Malawi ONSE Endline Performance Evaluation / 11 KEY FINDINGS Evaluation Question 1: To what extent did the Activity's interventions achieve their intended results? Finding 1.1: As measured by achieving its targets throughout the life of the award/activity (LOA), in general ONSE achieved its intended results. However, the results for HSS and increased demand for quality services fell short. Finding 1.2: The implementation of a single intervention versus a full package created tensions between districts receiving a single intervention (e.g., for malaria) and those receiving larger packages of support. Finding 1.3: Several foreseen (e.g., similar interventions) and unforeseen (e.g., the COVID-19 pandemic’s effects) changes to ONSE implementation affected results. Evaluation Question 2: To what extent did ONSE improve access to priority high-impact EHP interventions in the target health facilities and community service delivery points? Finding 2.1: Per key informants, ONSE improved access to priority high-impact EHP interventions; however, whether this finding was significantly different than in non-ONSE districts merits further examination. Finding 2.2: ONSE’s support of health information systems (HIS) contributed to the improvement of EHP service provision at the community level. Finding 2.3: ONSE conducted numerous well-received HSS interventions to address ongoing issues at the central and district levels. Of note were improvements in supportive supervision, planning, and coordination. Finding 2.4: Public sector strengthening by the ONSE Activity improved sustainable knowledge and skills among providers delivering services. Evaluation Question 3: What are the most significant accomplishments, best practices, and lessons learned from the ONSE Activity? Finding 3.1: Specific, measurable, achievable, realistic, and timed (SMART) capacity building and HSS interventions emerged as the most significant accomplishments of the ONSE Activity. Similarly, HSS interventions were considered among the best practices. Finding 3.2: While key informants noted many lessons learned, only one (integrated supportive supervision [ISS] and mentorship) stood out. Finding 3.3: In accomplishing its results, ONSE also experienced a few unintended consequences (e.g., potential changes in gender norms, participant discontent based on the method of payment). Finding 3.4: ONSE, in collaboration with other relevant stakeholders, relied primarily on the rollout of information systems to monitor the progress of interventions, ensure the reach of services was scaled up and used, and mitigate any challenges to scale-up. Finding 3.5: While ONSE incorporated measures to ensure greater sustainability of interventions, these did not always align precisely with what key informants believed were either the top factors for ensuring sustainability or factors that may pose risks to long-term adoption of practices. Finding 3.6: ONSE relied on routine Activity performance indicators to measure and monitor scale-up of activities. However, no defined set of indicators was identified to track quantifiable progress on sustainability efforts. Finding 3.7: Key informants cited numerous challenges to ONSE’s, and there was no clear consensus on which were the most significant. ONSE could have mitigated some of the challenges fully or partially, but others were outside of its manageable interests. Evaluation Question 4: What were the contextual factors such as socioeconomic factors, gender, demographic factors, environmental characteristics, baseline health conditions, health services characteristics, and so forth that affected implementation and outcomes of the ONSE Activity? Finding 4.1: There was no one significant contextual factor cited by key informants as facilitating the implementation of ONSE. Finding 4.2: Key informants cited several contextual factors as having limited ONSE’s ability to implement its interventions. The Activity addressed them through responsiveness to changing needs, initiating cost-cutting measures, or fostering greater involvement by community counterparts. However, not all limiting factors could be addressed. Finding 4.3: Future activities can support the systematic use of data at the facility and community levels in each district to address some of the limiting factors. GH EvaLS Malawi ONSE Endline Performance Evaluation / 12 CONCLUSIONS ONSE for the most part achieved its intended results, as demonstrated by the monitoring data analyzed. While there were some shortcomings in certain categories (e.g., HSS and service quality), the Activity met or exceeded most targets. The onset of the COVID-19 pandemic did not appear to substantially reduce the provision of services or achievement of results, pointing to ONSE’s ability to adapt interventions to the changed circumstances. While some of its targets may have been affected due to the redirection of funding for COVID-19 and other unforeseen events, it appears to have successfully navigated most of those challenges. Examining solely the monitoring indicators focused on EHP, it can be concluded that ONSE substantially improved access to EHP services. Per key informant feedback, what appears to have had the greatest effect in improving EHP services provision were interventions aimed at HSS, notably improvements in supportive supervision, planning, and coordination, and efforts to strengthen HIS and use of data for decision-making. By using the SMART5 capacity building approach, ONSE enhanced overall systemic capacity within the health sector at various geographic levels and improved the performance of individuals, teams, and structures working in and with Malawi’s health system. What should be considered and examined moving forward, though, is whether these EHP improvements were significantly better in ONSE districts than non-ONSE districts as the (limited) data analysis around the issue for this evaluation was inconclusive. Stakeholders indicated that ONSE’s HSS interventions were its most significant accomplishment, along with emerging best practices. They also noted that the SMART capacity building approach was a notable achievement of the Activity. It can reasonably be concluded that future approaches to improving the health of Malawians (whether funded externally or domestically) should give serious consideration to incorporating a comprehensive design and hands-on methods for improving health services. Of particular note was the main lesson learned from ONSE: ISS and mentorship leads to improved skills in all service areas. While ONSE tracked the progress and scale-up of its interventions, primarily through the rollout of information systems and use of data, and incorporated measures to better ensure sustainability, these efforts could have been improved. ONSE, like any other activity, faced numerous challenges in its implementation, both within and outside of its manageable interests. Mitigation measures only partially addresses most of the manageable challenges. Although ONSE both addressed and measured a number of contextual factors related to its implementation, per key informants, no one conclusive factor aided its success. Several elements of a generally enabling environment point to the need to ensure that these types of elements are in place for successful implementation. Respondents were more specific and provided a greater number of responses noting contextual factors that impeded ONSE’s implementation. Most of those factors were either somewhat or wholly outside of ONSE’s manageable interests or related to unforeseen events (e.g., the COVID-19 pandemic). Thus, while it is commendable that ONSE achieved a substantial portion of its intended results despite these challenges, there remain possibilities for improvement and recommendations for future activities and interventions. 5 The SMART approach acknowledges that practitioners are critical to effective management and that providing them with tools to collect, analyze, and use information is important, but not enough by itself. Thus, SMART provides comprehensive and ongoing training and support in addition to the tools. GH EvaLS Malawi ONSE Endline Performance Evaluation / 13 RECOMMENDATIONS IN BRIEF Because the performance evaluation was conducted at the end of the ONSE Activity, the recommendations below are directed at future activities. They are based on the findings and are divided into those that should be considered high priority and others that are for consideration. Finally, because external funding should be used to support Malawi’s Health Sector Strategic Plan (HSSP), in general the recommendations are aimed at all interested stakeholders. Where possible, the evaluation team provides guidance as to which stakeholders should, perhaps, be the primary leads. HIGH PRIORITY 1. Adopt an integrated HSS approach, giving priority to three specific building blocks (health workforce development, particularly supportive supervision and mentoring; support to HIS; and strengthening of the procurement and supply chain management systems). Lead stakeholders: USAID and the MOH equally. 2. Continue to use the SMART capacity building approach when appropriate. Lead stakeholders: USAID and its IPs primarily; potentially the MOH. 3. Conduct initial harmonized health facility assessments to ensure participating health facilities are fit for purpose. Lead stakeholders: USAID and development partners primarily; potentially the MOH. 4. Continue to engage with communities, particularly around issues of mutual accountability for the quality of health services while non-financially incentivizing members for their participation. Lead stakeholders: USAID and MOH equally. 5. Define sustainability prior to implementing interventions and build it into any future activities. Lead stakeholders: USAID and development partners primarily; potentially the MOH. FOR CONSIDERATION 6. Build into the planning and programming documentation contingency plans to enable pivots, if needed, to respond to unforeseen events. Lead stakeholders: USAID; potentially the MOH. 7. Delineate between all partners areas of manageable interest and lines of accountability. Lead stakeholders: USAID and MOH equally. 8. Communicate early and widely, and describe the reasoning for any differentiation in support that will be provided. Lead stakeholders: USAID and MOH equally. 9. If resources are available, conduct an impact evaluation for any future activity that can also include any possible spillover effects. Lead stakeholders: USAID and development partners primarily; potentially the MOH. GH EvaLS Malawi ONSE Endline Performance Evaluation / 14 1. EVALUATION PURPOSE AND EVALUATION QUESTIONS 1.1. ASSIGNMENT PURPOSE The endline performance evaluation of ONSE aimed to assess the extent to which the Activity achieved its objectives regarding the quality of and access to care, provide in-depth insights into the factors facilitating and limiting increased service utilization at each level of service delivery, and document progress toward building GOM capacity to deliver quality health services. The findings of this evaluation will inform MOH, USAID, and other development partners regarding the design and prioritization of future investments in strengthening Malawi’s health system and supporting the adaptation of current activities where possible. USAID will also use this evaluation to learn whether and to what extent ONSE met the stated objectives of improved access to and quality of a broad range of services (e.g., MCHN, FP/RH, malaria, and WASH); strengthened district health systems in support of these services; and increased community demand for them. The evaluation will also provide in-depth insights into the facilitating and limiting factors of increased service utilization at each level of service delivery. USAID will also use this evaluation to identify activities that warrant continued investment and any additional recommendations to strengthen future program implementation. 1.2. ASSIGNMENT QUESTIONS The evaluation considered four broad themes: effectiveness, access, quality, and demand. To do so, the evaluation team assessed four main evaluation questions (see Table 1). The evaluation questions are further divided into sub-questions to help answer the main questions. Table 1: Evaluation questions and sub-questions Evaluation Questions Sub-questions I. To what extent did the Activity's interventions achieve their intended results? 1a. What changes, if any, were made to activities, and how did those changes positively or negatively affect the Activity's achievement of its results? 2. To what extent did ONSE improve access to priority high-impact EHP interventions in the target health facilities and community service delivery points? 2a. Of the capacity strengthening approaches employed by ONSE, which have most enabled the MOH to address systems￾level issues at central and district levels? 3. What are the most significant accomplishments, best practices, and lessons learned from the ONSE Activity? 3a. How were these best practices systematically scaled up? 3b. What strategies, approaches, or methodologies did the Activity use to ensure sustainability? 3c. How were scale-up activities monitored and measured, and what level of sustainability was achieved? 3d. What were the challenges faced and how were they addressed/overcome? 4. What were the contextual factors such as socioeconomic factors, gender, demographic factors, environmental characteristics, baseline health conditions, health services characteristics, and so forth that affected implementation and outcomes of the ONSE Activity? 4a. What key findings emerged? 4b. What were the challenges faced and how were they overcome? 4c. What more can be done in future projects of similar nature? GH EvaLS Malawi ONSE Endline Performance Evaluation / 15 1.3. ASSIGNMENT AUDIENCE The target audiences for the ONSE Activity performance evaluation include: • The USAID/Malawi Mission; • GOM MOH units involved in prevention and care services at all levels; • Collaborators/partners and donors; • Health care providers, managers, and beneficiaries (including clients and community members and representatives) at the country level; and • ONSE Activity staff and IPs. GH EvaLS Malawi ONSE Endline Performance Evaluation / 16 2. BACKGROUND The MOH of the GOM, despite making progress over the last decades in several areas of MCHN and FP, continued to face many challenges in meeting the Millennium Development Goals (now known as the Sustainable Development Goals). These included low immunization rates, high MMRs, adolescent pregnancies, endemic malaria, gaps in nutrition coverage, and diseases such as diarrhea and respiratory infections. In 2014, USAID's Health Office developed integrated health activity strategies to support Malawi’s Country Development Cooperation Strategy. The goal was to support the improvement of Malawians’ quality of life by enabling them to access and use those services in a joint and complementary way. A request for proposals was developed for the ONSE Activity. The overall purpose of ONSE is to reduce maternal, newborn, and child morbidity and mortality. USAID expected this result to be achieved through close coordination with the MOH and other stakeholders to achieve four key intermediate results (IRs): 1. Improved access to priority health services; 2. Improved quality of priority health services; 3. Strengthened performance of health systems; and 4. Increased demand for priority health services. In addition, because of the unexpected onset of the COVID-19 pandemic, ONSE added the provision of specific support to Malawi’s response—in particular, in the 16 districts where ONSE was implemented. To improve access to high-quality EHP services and improve competence and performance, ONSE adopted an innovative way to build competence and capacity among health and related staff. Rather than traditional classroom-type training, ONSE focused on using the SMART approach to capacity building, along with more sustainable methods such as guided simulation responses, mentoring, coaching, and on￾the-job ISS. To meet those objectives, ONSE focused providing technical assistance rather than relying only on direct delivery of services. This required working on the system of governance, leadership, and service management, necessitating close coordination and collaboration with government actors from the central and district levels, as well as related partners and stakeholders. Finally, ONSE worked with communities, not only to mobilize demand and change health-seeking behavior, but also to engage with community leaders and members to self-identify their health and environmental conditions and demand the health system’s accountability for the availability and quality of services. During the final years of implementation, ONSE experienced several changes in its activities and finances, as well as occurrences in the national context. Two changes were a no-cost extension from January to June 2021 and a cost extension from July 2021 to the end of June 2022. Another was the COVID-19 pandemic, which brought several challenges to and changes in implementation procedures. This evaluation considered all major shifts and changes to explore and measure their effects on processes and achievements or results. 2.1. RESULTS FRAMEWORK ONSE’s results framework (Figure 1) presents the four IRs leading to the main result, Maternal, Newborn, and Child Morbidity and Mortality Reduced. The IRs complement each other to ensure both increased access to the priority health services targeted in the Activity and the delivery of those services with improved quality of care. Additionally, to achieve and sustain such results, ONSE embarked on a number of key activities to strengthen Malawi’s health system. Finally, the Activity aimed to empower clients and GH EvaLS Malawi ONSE Endline Performance Evaluation / 17 communities and integrate them into the Activity’s efforts to increase demand for such priority health services. Figure 1: ONSE Results Framework 2.2. GEOGRAPHIC COVERAGE ONSE conducted activities in 16 districts: Balaka, Chikwawa, Chitipa, Dowa, Karonga, Kasungu, Lilongwe, Machinga, Mangochi, Mchinji, Mulanje, Nkhata Bay, Nkhotakota, Ntcheu, Salima, and Zomba. As shown in Figure 2, intervention strategies varied by district. Malaria interventions were not included in the cost extension period from July 1, 2021, to June 30, 2022. GH EvaLS Malawi ONSE Endline Performance Evaluation / 18 Figure 2: Map of ONSE Interventions Legend: Package of Services . . Packages . . Full Family Health Malaria Services Family Health* Blue Red White Malaria Blue White Grey Health Systems Strengthening Blue Red Grey . *Maternal, newborn and child health; FP/RH; nutrition; WASH GH EvaLS Malawi ONSE Endline Performance Evaluation / 19 3. METHODS AND LIMITATIONS 3.1. METHODOLOGY This section describes the overall evaluation approach, along with the evaluation questions and sub￾questions and evaluation methods. All GH EvaLS assignments follow the GH EvaLS Data Management Plan. The plan details the process for data collection and reporting; data storage and security; data privacy; data flow; data analysis and dissemination; and reviewing and ensuring data quality, as well as posting data to the Development Data Library. The evaluation used a mixed methods approach to collect data, given that activities were conducted at different levels and in an expanded sequence for nearly six years. The collection of quantitative and qualitative data ensured that the evaluation could answer questions regarding not only access to health care by clients and communities, but also the quality of care and the strengthening of the capacity of health personnel and facilities to provide such care. 3.2. DATA SOURCES The evaluation methods and data sources depended on the evaluation questions. Qualitative data sources included KIIs, FGDs, and GIs conducted at multiple levels of implementation (national, regional, district, community, and health facility). Sources of quantitative data included a document review, including monitoring data and LOA results, along with some limited analysis of data from the Malawi DHIS2. The data collection approach was inductive to enable the evaluation to delve into complex topics while obtaining multiple perspectives on Activity strategies and interventions. 3.2.1. Document and Desk Review The evaluation team conducted a desk review to contribute to a thorough understanding, analysis, and interpretation of evaluation findings. The information came from a wide range of documents including the original request for proposals, national- and Activity-specific surveys and assessments, monitoring and evaluation plans and work plans, annual and other time-specific reports, organizational charts, management information systems, and other generic or specific Activity-related reports. Annex 3 is the list of the documents reviewed. The evaluation team used this information to identify key themes for KII and FGD questions. The desk review provided information concerning ONSE’s intended and actual implementation efforts as well as its documented achievements and challenges. 3.2.2. KIIs, GIs, and FGDs This evaluation employed purposive sampling to select evaluation participants. Purposive sampling is widely used in qualitative research for the identification and selection of information-rich cases.6 The evaluation team identified and selected individuals and groups with firsthand, expert knowledge of the Activity at the central, regional, and community levels. Based on this sampling, the team conducted KIIs (i.e., with one individual), GIs (specific questions directed to two or more individuals), or FGDs (facilitated general discussions with groups of individuals).7 At the central level, key individuals included USAID representatives, government officials, Activity staff, key partners, and other development actors. The evaluation team interviewed these participants to 1) discuss and, where possible, validate Activity approaches, interventions, and achievements; 2) elicit 6 Patton, M. Q. (2002). Qualitative research & evaluation methods (3rd ed.). Thousand Oaks, CA: Sage. 7 Group discussions or interviews were envisioned as more efficient in obtaining similar or overlapping Activity implementation information from staff with different roles and responsibilities. GH EvaLS Malawi ONSE Endline Performance Evaluation / 20 opinions and perceptions of the effectiveness of Activity implementation and missed opportunities, if any; 3) obtain firsthand reports on training, supportive supervision, and data and management systems; 4) determine how stakeholders and beneficiaries interacted with the Activity in terms of accountability, partnership, and collaboration; and 5) determine how ONSE contributed to increased uptake of quality health services and behaviors in nearby and hard-to-reach rural communities. Representatives of GOM ministries provided perspectives on collaboration among bilateral agencies and coordination between central and district public entities during Activity implementation. ONSE’s implementing organizations played essential roles in describing Activity implementation successes, challenges, and recommendations for sustained positive outcomes. IPs added clarity to the evaluation team’s understanding of information from the desk review. Other agency representatives working in maternal and neonatal health, FHP, supply chain management, and related areas were interviewed concerning coordination and leveraging efforts to reduce duplication. Finally, USAID staff described Activity implementation rollout and results relative to USAID’s Country Development Cooperation Strategy and its vision. At the district level, public sector officers and representatives provided insights into HSS efforts, including supply chain systems and the SMART capacity building strategy. This was especially important as much of the district planning and decision-making occurred at this level, because the government has undergone a decentralization process in recent years. Health service providers from the public and private sectors were interviewed regarding training and other support received from ONSE and on their perspectives on improvements to access and quality of services for Malawians. The evaluation team made efforts to understand differences in the types of district-level staff from ONSE implementing organizations as crucial key informants to explain how the Activity provided supportive supervision and coordination during the LOA. Finally, the evaluation team interviewed local stakeholders8 whose activities influenced ONSE. At the community level, clients of FHP services were separated into homogeneous groups (male, female, and youth) for discussions related to their health knowledge, care service experiences, and community efforts to create demand and mobilize change (e.g., community champions). The evaluation team worked with implementing organizations to find youth who took part in intervention activities to assess their viewpoints during FGDs. IPs also assisted in finding health volunteers to understand demand creation at the community level. Finally, the evaluation team interviewed different types of service providers to learn about service delivery to varied districts and clients. Table 2 lists the estimated evaluation sampling numbers for FGDs and KIIs. Annex 2 presents data collection tools, and Annex 4 provides details of the final evaluation participant groups, geographic breakdown of participants, and the domains of inquiry. Table 2: Evaluation sampling for KIIs, FGDs, and GIs Data Collection Level Method Number Central Level MOH and other public representatives KIIs 4 USAID KIIs 2 Prime IP KIIs 5 Other IPs KIIs 4 8 Including district health officials and officers, health providers, community leaders, health volunteers, and adult and youth recipients of ONSE services. GH EvaLS Malawi ONSE Endline Performance Evaluation / 21 Data Collection Level Method Number Other development partners KIIs 2 Total (all KIIs) 17 District Level*, ** and Lilongwe . KIIs 48 FGD participants 13 GI participants 11 Total participants (all methods) at the district level 72 Total number of individuals participating 89 * This included district health officials and officers, health providers, community leaders, health volunteers, adult and youth clients of services. ** The districts were Balaka, Chikwawa, Chitipa, Karonga, Kasungu, (Lilongwe), Machinga, Nkhata Bay, Nkhotakota, and Ntcheu. The evaluation team coordinated with the prime IP, MSH, to plan practical and smooth data collection activities with evaluation participants at the district and community levels. A team of local consultants worked in pairs to conduct KIIs and GIs and take notes. The field notes were then completed and transcribed for analysis. The core evaluation team and field team used WhatsApp for daily check-ins and weekly debriefings on data collection status, observations, and findings. 3.3. SITE AND SAMPLE SELECTION Given the anticipated evaluation timeframe, the evaluation team, in collaboration with USAID/Malawi, selected nine districts and the capital city, Lilongwe, for the endline performance evaluation. The selection of sampling sites took into account the types of intervention packages (full versus single intervention package), geographic areas (Northern, Central, and Southern districts), as well as the presence of the Christian Health Association of Malawi as a key partner. The team also coordinated the selection of smaller geographic zones (villages) with MSH to complete the sample and include hard-to￾reach intervention areas. The nine districts sampled were Balaka, Chikwawa, Chitipa, Karonga, Kasungu, Machinga, Nkhata Bay, Nkhotakota, and Ntcheu, along with Lilongwe. Although the findings may not necessarily be generalized to the rest of the country, the diverse types of respondents and study sites should inspire some confidence that the results reflect the typical experience in rest of the ONSE districts. GH EvaLS Malawi ONSE Endline Performance Evaluation / 22 4. DATA ANALYSIS To answer the evaluation questions, the evaluation team cross-referenced information obtained during the desk review with primary data (interview and group discussion data) and quantitative analysis to develop conclusions and recommendations. The team used multiple quantitative and qualitative methods, and existing data (Activity performance indicator data, etc.) to triangulate findings and produce more robust evaluation results. The team reviewed KII, GI, and FGD transcripts and organized responses to align with the evaluation questions, prevalent themes, and perspectives. They also cross-checked responses to verify data. This facilitated the transition to the interpretative phase to support the development of evidence to reflect emerging themes. The team used the resulting qualitative data to corroborate and triangulate performance indicators, gaining further insights and clarifying the narrative behind the quantitative data. Quantitative data analysis used existing service performance indicator data extracted from districts, facilities, community health management information system (HMIS), Activity-developed data tools, and data compiled in the Activity Monitoring and Evaluation Plan (AMEP) LOA indicators from the Final Activity Report (see Annex 5). This quantitative data reflecting the extent of Activity service indicators achieved is presented in simple cross-tabulations and service targets achievement levels, as well as IR￾related bar charts for visualization. The data validated access to and use of services at the public and private institutional levels, including aspects of coverage of MCHN, youth-friendly health services (YFHS), post-abortion care, intermittent preventive treatment in pregnancy, WASH, and malaria care demand and uptake, as well as HSS activities at the central and district levels. Additionally, using two data sets derived from the DHIS2, the evaluation team performed longitudinal and comparative analysis for a limited number of indicators. The team used the quantitative data to corroborate findings from the qualitative data, which was the primary source of analysis. The evaluation team used NVIVO software to analyze qualitative data for the ONSE performance evaluation. The data analysis entailed setting up an analysis project file in the software, defining primary or main themes (parent nodes) as reflected in evaluation questions, importing interview transcripts (data sources) into the project, reading through transcripts and identifying sub-themes (child nodes), and coding each theme. Each theme was developed or coded to pre-defined precise levels for better insight into the qualitative data and findings. The team coded 75 transcripts9 in NVIVO. Upon completion of the coding, themes (codes) were expanded, and the list was exported to a Microsoft Excel spreadsheet summarizing the number of data sources (files or transcripts) reporting on a given theme and the number of references or direct quotes per each specific theme. Qualitative analysis summary tables were extracted from the Excel spreadsheet for easy sorting according to a number of data sources and references. This was intended to identify themes that involved a large number of interviews (informants). It also enabled the extraction of corresponding references (direct quotes) by theme. Emerging themes were synthesized to gauge how different themes linked to one another. Where themes were related, they were merged into a broad theme or finding during report writing. The direct quotes were read and key messages identified and reported either independently (standalone findings) or jointly with quantitative findings. A selection of direct quotes in the report gives voice to evaluation participants and validates synthesized findings. 9 This is fewer than the number of participants, as each FGD and GI resulted in only one transcript. GH EvaLS Malawi ONSE Endline Performance Evaluation / 23 5. ETHICAL CONSIDERATIONS 5.1. HUMAN SUBJECT PROTECTION The evaluation team developed protocols to ensure privacy and confidentiality during data collection. Primary data collection included a consent process that described the purpose of the assignment; risks and benefits to participants and communities; the right to refuse to answer any question; and the right to refuse to participate in the assignment at any time, without consequences. Only adults could consent as part of this assignment. Youths recruited for the evaluation were at least 18 years of age. During the evaluation process, if data from existing documents included unique identifiers, those data were abstracted without including the identifying information. The data collectors sought and obtained verbal consent prior to data collection. All data collectors were trained in informed consent. GH EvaLS Malawi ONSE Endline Performance Evaluation / 24 6. LIMITATIONS The evaluation team anticipated that it would need to mitigate several possible biases and other data limitations through methodological and/or analytical means. 6.1. RELIABILITY AND VALIDITY This evaluation employed qualitative data methods to complement existing quantitative data. Qualitative methods, by their nature, are less structured than quantitative methods to allow for greater exploration and depth in answers. As such, data quality depends on the data collector rather than quantitative or structured survey tools. To help ensure the reliability and validity of qualitative data, local data collectors underwent remote training to learn or refresh data collection techniques and become familiar with the evaluation tools. The evaluation team and data collectors conducted regular debriefs to help ensure the reliability and validity of the qualitative data. 6.2. QUALITATIVE APPROACH The primary approach for this evaluation was qualitative data collection via KIIs and FGDs and, to a lesser degree, the review and analysis of secondary qualitative and quantitative data. The opinions of stakeholders are, by their nature, subjective, and may reflect certain vested interests. Further, team members may not have accurately recorded or correctly transcribed important data for a variety of reasons (such as fatigue and difference in understanding). Therefore, the team instituted data checks that were both internal (e.g., team members cross-checking each other’s notes) and external (e.g., triangulating among various data sources) to mitigate this issue as much as possible. 6.3. QUANTITATIVE DATA FROM PROJECT REPORTING The evaluators conducted secondary analysis of the quantitative data collected and reported to USAID through project performance indicators. The integrity and quality of project data reported was assumed. Additionally, the team performed limited analysis of data supplied via DHIS2. The analysis and results, when presented, involve a number of assumptions and caveats that are noted. 6.4. SELECTION AND REPRESENTATION BIASES The evaluation team identified a large and diverse group of stakeholders at the central and peripheral levels. It is possible that some key stakeholders may have been excluded inadvertently and that some participants in KIIs, FGDs, or GIs may have introduced self-selection bias (either beneficial or detrimental) into the results. Persons with stronger vested interests in the results of the assessment (either negative or positive) may have been willing to spend more time with the interviewers. As a result, it may not have been possible to obtain a complete representation of all stakeholders, although the team believes that this was at least partially mitigated by the broad extent of those interviewed (see Table 2). 6.5. RESPONSE BIAS Response bias is a common problem for assessments, evaluations, and reviews that takes several forms. For instance, respondents may make positive remarks to interviewers about an activity because they have a vested interest in seeing it succeed and continue. Respondents may be inclined to shape their responses according to gender or other social norms, or to say what they think an interviewer wants to hear. The team anticipated this possible bias and took several measures to mitigate the issue. Those measures included similar but not identical questions throughout interviews; carefully reviewing the wording of questions to ensure that language was culturally appropriate and did not include any inherently biased wording; and providing open-ended questions, including the option not to respond. GH EvaLS Malawi ONSE Endline Performance Evaluation / 25 6.6. EVALUATION SUBSEQUENT TO PROJECT CLOSE As the ONSE Activity ended before the evaluation process began, there were additional limitations to Activity data resources and staff available to further define, triangulate, and validate qualitative and quantitative data or details of Activity successes and challenges. The evaluation team mitigated some of the limitations described above through a combination of thorough training of interviewers; consistent development and pilot testing of tools; consistent fieldwork supervision, including sample reviews of first transcripts to ensure comprehensive questioning and accurate notetaking during interviews; and triangulation of methods from different and complementary sources of data. Transcription errors were reduced through joint reviews of field notes among the team, especially at the start of fieldwork, to seek internal agreement and clarify any questionable or conflicting notes. These measures were intended to reduce the appearance and subsistence of biases and limitations, rendering more complete and valid data collection and results. GH EvaLS Malawi ONSE Endline Performance Evaluation / 26 7. FINDINGS This section presents the key findings of the endline performance evaluation based on the four evaluation questions and corresponding sub-questions. 7.1. EVALUATION QUESTION 1 To what extent did the Activity’s interventions achieve their intended results? 1a: What changes, if any, were made to activities, and how did those changes positively or negatively affect the Activity’s achievement of its results (e.g., gaps and opportunities in retention and training of health workers)? Finding 1.1: As measured by achieving its targets throughout the life of the Activity, in general, ONSE achieved its intended results; however, the results for HSS and increased demand for quality services fell short. As shown in Table 3, several IRs were achieved wholly (e.g., integration of interventions) or substantially (e.g., quality of priority health services, and cross-cutting interventions), or had reasonable success (e.g., increasing access to priority health services). Where ONSE fell short was in improving the performance of the health system and increasing demand for quality priority health services. Further details of specific interventions, their actual results, and targets are provided in Annex 5. The impact of the COVID-19 pandemic must be accounted for within the life of ONSE—specifically, whether it affected implementation of interventions during the period 2020–2022. Therefore, the evaluation team conducted additional analysis using DHIS2 data to examine whether the effects of COVID-19 were noticeable for a limited number of interventions within the 16 ONSE districts. Those results are depicted in Figures 4–7. As noted, except for the provision of oxytocin, which may have been due to supply chain issues early in the pandemic, no other indicators showed declines and, indeed, most showed consistent improvement. Table 3: Indicator performance success rate by ONSE intermediate results ONSE Intermediate Result Total # of Performance Indicators # of Indicators with LOA Target 90%+ Achieved # of Indicators with LOA Target <90% Achieved % of Indicators with LOA Target Achieved at 90%+ (Success Rate) IR 1: Access to priority health services increased 39 28 11 72% IR 2: Quality of priority health services improved 6 5 1 83%* IR 3: Demand for quality priority health services increased 8 4 4 50% IN-1: Number of integration interventions completed 1 1 0 100% Cross-cutting indicators 4 3 1 75% *Data for five training indicators not included for IR 2 as LOA data was not available to evaluators. This may affect the overall success rate for IR 2. “Through the ONSE Activity, these funds helped fill a large gap in the health sector and made it possible for district councils to deliver essential health services … It enabled improvement in a number of areas, including equitable access and quality health care services.” —Key informant GH EvaLS Malawi ONSE Endline Performance Evaluation / 27 Figure 3: Provision of Oxytocin for Emergency Obstetric Care Figure 4: Under-1 Childhood Vaccinations * Data for two districts (Lilongwe and Ntcheu) are missing for 2017, accounting for the decline. 0 50000 100000 150000 200000 250000 300000 350000 400000 2015 2016 2017 2018 2019 2020 2021 2022 ONSE Districts: Reproductive Health Department Emergency Obstetric Care Oxytocin (2015-2022) 0 50000 100000 150000 200000 250000 2015 2016 2017 2018 2019 2020 2021 2022 ONSE Districts: EPI DPT-HepB-Hib3 Childhood Vaccination Statistics, Under 1 (2015-2022) GH EvaLS Malawi ONSE Endline Performance Evaluation / 28 Figure 5: Women Making More than Five ANC Visits Figure 6: Early ANC Visits Key informants reported many reasons for ONSE’s achievement of its results. Table 4 lists the most frequently cited interventions, many of which were structured around either improving infrastructure and logistics or increasing access to quality services. The evaluation team observed no consistent feedback in KII responses on interventions that may have negatively affected ONSE’s ability to achieve its results (e.g., KII responses did not include enough mentions to be significant). 0 10000 20000 30000 40000 50000 60000 70000 80000 2015 2016 2017 2018 2019 2020 2021 2022 ONSE Districts: Reproductive Health Department ANC visits per Woman with 5+ Visits (2015-2022) 0 10000 20000 30000 40000 50000 60000 70000 80000 2015 2016 2017 2018 2019 2020 2021 2022 ONSE Districts: Reproductive Health Department ANC visits during weeks 0-12 (2015-2022) GH EvaLS Malawi ONSE Endline Performance Evaluation / 29 Table 4: Top Interventions that Contributed Most to ONSE Achieving Its Results (at Least 10 KII Mentions) Intervention KII Respondents10 Improving infrastructure and medical equipment 27 Increased access to FP services 23 Organized outreach clinics 17 COVID-19 vaccination provision 16 Increasing ANC attendance 14 Logistics support to health facilities 14 Integration of health services 13 Improved quality of MCHN services 12 Logistical support (general) 11 Increased access to WASH services 10 Re-allocation of medical supplies 10 It was, of course, not sufficient for ONSE to increase the supply (and quality) of services to achieve its results; the Activity also had to create demand among potential clients. While respondents mentioned 14 ONSE interventions as increasing demand, only three received more than three KII mentions (see Table 5). Table 5: Top Three Interventions that Increased Demand for Health Services per KIIs Intervention KII Respondents Improved sanitation in facilities 11 Increased involvement of community 6 Youth uptake of YFHS 6 Finding 1.2: The implementation of a single intervention versus a full package created tensions between districts receiving a single intervention (e.g., malaria) and those receiving larger packages of support. Key informants commented that some districts did not understand why ONSE provided them a full package of support, while in other districts it focused solely on malaria. Some respondents noted that this strained the relationships between districts and resulted in suboptimal interactions. In districts that received a single intervention, the general perception was that the Activity’s interventions partly achieved their intended results, while its efforts were more recognized and appreciated in districts receiving the full package. Further, as one key informant noted, because ONSE was implementing malaria interventions in the sub￾10 “KII Respondents” represents the number of key informants who mentioned a particular intervention at least once during their interview. In other words, a key informant may have mentioned the intervention multiple times throughout the discussion, but it was recorded only once for reporting purposes. This is believed to be a more accurate representation than the number of mentions (e.g., one respondent may have mentioned an intervention several times). “… we were able to support 3.8 million couple-years of protection with the FP services. We resuscitated over 40,000 newborn babies born with difficulties in breathing.” —Key informant “… malaria interventions were not in all districts. So, where you have inputs on one intervention like malaria, achieving optimal results, you will observe gaps in other districts … Communities were not benefitting for accessing those services. This is where the mismatch has been, and ONSE has left that mark.” —Key informant GH EvaLS Malawi ONSE Endline Performance Evaluation / 30 group of districts, those same districts were not included in the most recent Global Fund grant application. This affected them when ONSE’s resources became scarcer. Finding 1.3: Several foreseen and unforeseen changes to the ONSE intervention implementation affected results. Changes that affected ONSE positively and negatively included: • Widespread power outages during the first year of ONSE diverted funds from planned interventions for the purchase of fuel and generators. • As early as ONSE’s second year, there were important changes in its scope as needs arose to support the MOH in developing its national performance-based finance strategy, conduct contact investigation of tuberculosis cases in Machinga, and support an effective response to an outbreak of cholera. • In Activity Year 3, the United States government (USG) shutdown put the Activity into temporary slow-down mode. • During years 4–6, ONSE adapted SMART capacity building approaches to address the impact of the COVID-19 pandemic on the health and safety of ONSE staff and of health workers. • Key informants noted that overlap with a Canadian-funded program led to duplication of efforts, and discontent among ONSE participants, as the Canadian project paid higher allowances. 7.2. EVALUATION QUESTION 2 To what extent did ONSE improve access to priority high impact EHP11 interventions in the target health facilities and community service delivery points? 2a: Of the capacity strengthening approaches employed by ONSE, which have most enabled the MOH to address systems-level issues at central and district levels? Finding 2.1: Per key informants, ONSE improved access to priority high-impact EHP interventions; however, whether this was significantly different than in non-ONSE districts merits further examination. Given that only 20 percent of the population lives within 25 kilometers of a hospital, ONSE worked with the MOH to increase the proportion of health facilities offering priority EHP services, strengthen existing outreach clinics, establish new ones, and conduct community services campaigns, such as for vaccination and FP, so services were brought closer to the people. Health facility respondents highlighted ONSE’s support as helping to ensure regular and continued supplies of commodities, equipment, and fuel for transportation; data management; and training of providers that were prerequisites for the enhancement of EHP services in health facilities. Respondents noted that this assistance enabled clients to better access vital services. Support by ONSE to organize community outreach clinics was critical in increasing access to EHP services for community beneficiaries by taking services to their doorsteps. ONSE strengthened Community Champions, community health workers, and Community Health Action Group (CHAG) members who played important roles in disseminating information about services and referrals, eventually creating demand for services at 11 Per the Health Sector Strategic Plan II: 2017–2022, the EHP covers several categories under which there are numerous intervention packages and interventions. The main categories are reproductive, maternal, newborn and child health (RMNCH); vaccine preventable diseases; malaria; integrated management of childhood illnesses; community health; neglected tropical diseases; HIV/AIDS; nutrition; tuberculosis; non-communicable diseases; and oral health. Historically, the EHP has been financially unobtainable and unsustainable. GH EvaLS Malawi ONSE Endline Performance Evaluation / 31 community-level clinics and health facilities. Table 6 lists the five top interventions that improved the quality of EHP services. Table 6: Top Five Interventions that Improved the Quality of EHP Services, per KIIs Intervention Integrated supportive supervision at the facility level 31 Social accountability (scorecards) 15 Improved skills of health workers 10 Strengthened management of human resources 7 Improved quality of baby friendly services 5 A cross-section of relevant performance indicators for the provision of EHP reflected gains in the targeted districts over the Activity timeline. six of the seven indicators reached or exceeded their LOA targets (see Table 7). Only 81% percent of pregnant women reached with nutrition interventions underperformed, and only by 19 percent below its LOA target. Table 7: LOA Changes in Provision of Selected EHP Services Indicator Baseline LOA Target Endline Result LOA target Achieved LOA from Change Baseline Number of newborns not breathing at birth who were successfully resuscitated in USG supported program 3,500 34,500 49,821 144% +46,321 newborns (1,324% increase) Number of pregnant women who initiated ANC visits in the first trimester 36,500 259,730 233,906 90% +197,406 women (541% increase) Number of women giving birth who received uterotonics in the third stage of labor (or immediately after birth) through USG-supported programs (EPMM indicator) 155,000 1,269,00 1,200,068 95% +1,045,608 women (544% increase) % of pregnant women reached with nutrition interventions through USG-supported programs 242,000 1,916,800 1,557,978 81% +1,315,578 women (544% increase) Number of children who received DPT3 PENTA III by 12 months of age in USG-assisted 0 1,070,600 1,155,504 108% 0-> 1,155,504 children programs Number of children ages 12–23 months who received measles￾rubella second dose via USG- 0 599,759 542,672 90% 0-> 542,672 children assisted programs % of non-public health facilities (including the Christian Health Association of Malasi, NGOs, and private for-profit institutions) supported by USG to provide priority health service 46% 70% 77% 110% 67% increase Source: AIDS-612-C-00001 ONSE Final Project Draft 2022 GH EvaLS Malawi ONSE Endline Performance Evaluation / 32 One final point of analysis was a cursory examination of one of the EHP interventions to see whether there was a significant difference in the performance of the 16 ONSE districts versus the 12 districts in which ONSE was not implemented, as reported by district health offices (DHOs) in the DHIS2. The evaluation team selected data for neonatal resuscitation12 for this analysis, as it presented a complete data set from all the districts for the period 2015–2022. As shown in Figures 8, 9, and 10, while an overall slightly greater number of resuscitations was performed in ONSE districts, there were no significant differences in those done by bag and mask ventilation versus airway cleaning and stimulation. Indeed, for 2020, when resuscitations increased in ONSE districts, fewer newborns were resuscitated by either method compared to non-ONSE districts. This may be due to supply chain issues in 2020 that marked the onset of the COVID-19 pandemic or other methods used during that year (e.g., dry and wrapping). Regardless, this initial exploration of the data should be further investigated to identify any comparative differences between the ONSE and non￾ONSE districts. Figure 7: Newborns Resuscitated (ONSE Versus Non-ONSE Districts) 12 Per the Health Sector Strategic Plan II: 2017-2022, neonatal resuscitation is part of the intervention package for deliveries as a sub-group of the RMNCH category. 0 2 4 6 8 10 12 2015 2016 2017 2018 2019 2020 2021 2022 Babies resuscitated in health facilities as a percentage of deliveries: DHOs only (2015-2022) ONSE Non-ONSE “ONSE ensured the availability of a system for tracking stocks of drugs at the health center, which gave timely data required for the ordering the EHP drugs needed … and continually requested reports and provided gadgets for capturing data … which improved the information system.” —Key informant GH EvaLS Malawi ONSE Endline Performance Evaluation / 33 Figure 8: Newborns Resuscitated via Airway Cleaning and Stimulation (ONSE Versus Non-ONSE Districts) Figure 9: Newborns Resuscitated via Bag and Mask Ventilation (ONSE Versus Non-ONSE Districts) Finding 2.2: ONSE’s support of the HIS contributed to the improvement of EHP service provision at the community level. Although none of the HIS interventions (see Table 8) met their targets, feedback received by MOH staff at all levels indicated that these interventions were crucial in supporting EHP service provision. Notably, monthly data review meetings provided clinicians and health workers with essential indicators to monitor gaps and shortfalls in health facilities. ONSE also supported public and private sector officials with appropriate capacity building and infrastructure development in HIS. In turn, this assisted these officials to assess the needs of the facility and take essential and appropriate measures to rectify them. 0 10 20 30 40 50 60 2015 2016 2017 2018 2019 2020 2021 2022 Percentage of successful newborn resuscitations done by bag and mask ventilation: DHOs only (2015-2022) ONSE Non-ONSE 0 10 20 30 40 50 2015 2016 2017 2018 2019 2020 2021 2022 Percentages of successful newborn resuscitations done by clearning airway and stimulation: DHOs only (2015-2022) ONSE Non-ONSE GH EvaLS Malawi ONSE Endline Performance Evaluation / 34 Respondents also appreciated the introduction of the COMCARE tool for supportive supervision and direct feeding of data into the HIS, which led to faster responses by health managers. ONSE also provided health service assistants with tablets to update the DHIS2 database, along with fuel, vehicles, and, on occasion, allowances to facilitate the collection of data at the community level in ONSE￾sponsored districts. This improved the quality and timeliness of the data collection and review process. Strengthening the functionality of existing MOH systems at the central and district levels to continue supporting decentralization policies also demonstrated gains as a result of ONSE interventions. Table 8: Interventions Affecting Systems-Level Issues at the MOH Central and District Levels Indicator Baseline LOA Target Endline Result LOA Target Achieved LOA from Change Baseline Number of District Implementation Plan (DIP) reviews conducted to revise DIP activities or budgets with stakeholders 16 161 139 86% +123 DIP reviews (769% increase) % of health facilities supervised based on performance standards in ISS tools that completed follow￾up actions 40% 89% 75% 85% 86% increase Number of health facilities +102 health submitting data in national standards line with (DHIS2) 205 308 307 98% facilities (50% increase) Number of health facilities submitted data in line with +72 health national standards (Open logistics management information system [LMIS]) 308 388 380 98% facilities (23% increase) Source: AIDS-612-C-17-00001 ONSE FINAL PROJECT DRAFT 2022 Finding 2.3: ONSE conducted numerous well-received HSS interventions to address ongoing issues at the central and district levels. Of note were improvements in supportive supervision, planning, and coordination. ONSE supported the decentralization model in Malawi, devolving MOH decision-making authority for budgets, planning, and management of health services from the central level to the district level. To do so, the Activity coordinated and funded regular capacity building meetings and provided tablets, community-focused tools and methodologies, and electronic resources in the 16 districts to set and achieve operational agendas in support of high-quality, in-demand health services that prioritized community needs. This was well-received by district and central managers and supported ongoing functionality and productivity during the ONSE Activity timeframe, as well as stronger collaborative relationships with central MOH technical managers and working groups. Of more than 50 beneficial HSS interventions provided by ONSE as noted by key informants, Table 9 lists those that key informants cited most often. “The monthly data review meetings provided clinicians and other health workers with the required indicators to monitor the gaps and shortfalls in their health facilities. The DHO, GOM, health workers, and other partners now have the necessary infrastructures like the DHIS2 to help them know what each facility needs.” —Key informant GH EvaLS Malawi ONSE Endline Performance Evaluation / 35 Table 9: Top 10 HSS Contributions by ONSE, per KIIs Themes KII Respondents Supportive supervision 22 Improved planning by districts 21 Improvement in data quality 20 Coordination of the health system 17 Improved health programming 17 Integrated supportive supervision (digital tools) 17 Mentorship of health workers 15 Collecting and analysis of DHIS2 and LMIS data 14 Improved reporting rates 11 Assistance in the implementation of projects at district level 8 To promote a robust systemic capacity building approach, ONSE intervened at multiple levels of Malawi’s health system and targeted various components. Many respondents highlighted positive outcomes, from interaction with technical working groups at the central level to the facilitation of district-level planning. A key interviewee pointed out that ONSE’s revival of district￾level committees and decentralization of supervision and monitoring from the central MOH to the district level contributed to strengthening district-level capacity and efficiency in addressing community health needs. Several respondents underscored ONSE’s input to the strengthening of health center management committees, and establishment and revitalization of CHAGs to enhance communities’ roles in supporting the implementation of community-led approaches, including the involvement of village health committees. Respondents also expressed appreciation for ONSE’s contribution to empowering and building the capacity of community members and service providers to disseminate information on the availability of services and referrals. Finding 2.4: Public sector strengthening by the ONSE Activity improved the sustainable knowledge and skills of providers delivering services. To enable stronger overall systemic capacity, ONSE intervened at multiple levels to improve the performance capacity of individuals, teams, and structures working in and with Malawi’s health system. Recognizing that traditional classroom-based training had a limited impact on outcomes, increased workplace absences, and distorted incentives, ONSE catalyzed a shift toward approaches that develop skills in day-to-day working environments through simulation-based practice and drills, coaching, and mentoring based on the SMART capacity building approach first formulated by Potter and Brough (see Figure 11).13 13 https://www.researchgate.net/figure/Capacity-building-pyramid-Potter-and-Brough-7_fig1_264055541 “ONSE engaged and mobilized communities to disseminate critical health information and messages, increased demand for and utilization of high-quality services, and strengthened and elevated the voices of citizens and communities to ensure that health services are accountable to their users.” —Key informant “As a provider, the mentorship has helped me to attain much knowledge and skills on how to manage complicated cases or how to deliver the service in general to clients or mothers or patients who come to our facility.” —Key informant GH EvaLS Malawi ONSE Endline Performance Evaluation / 36 Figure 10: Basic Conceptualization of SMART Capacity Building Approach Most of the SMART capacity building approaches that ONSE successfully implemented during Years 1–4 relied on face-to-face interaction and on-the-job skills building to drive change. With the onset of the COVID-19 pandemic in 2020, ONSE recognized the need to strategically adjust these approaches to virtual or remote platforms, while still facilitating knowledge sharing, skills building, and improvement in overall quality of care at the district level. 7.3. EVALUATION QUESTION 3 What are the most significant accomplishments, best practices, and lessons learned from the ONSE Activity? 3a: How were these best practices systematically scaled up? 3b: What strategies, approaches, or methodologies did the Activity use to ensure sustainability? 3c: How were scale-up activities monitored and measured, and what level of sustainability was achieved? 3d: What were the challenges faced and how were they addressed/ overcome? To better understand ONSE’s accomplishments, the evaluation team conducted a brief review of the Malawi health sector in the years prior to the Activity. According to the 2010 Demographic and Health Survey, infant and child mortality rates decreased from 76 per 1,000 in 2004 to 66 per 1,000 in 2010 and from 133 per 1,000 to 112 per 1,000, respectively. The MMR decreased from 984 per 100,000 in 2004 to 675 per 100,000 in 2010. Despite those gains, a number of factors negatively impacted the health of Malawians. These included the availability and quality of health services, access to health services, and environmental and behavioral issues. The HSSP (2011–2016) was designed to address these factors to improve the health status of the people of Malawi. An EHP was agreed upon, covering diseases and conditions affecting the majority of the population and especially the poor. The EHP was updated in the HSSP II (2017–2022), categorizing services into priority classes using relevant criteria, expanding coverage for high-priority services to all Malawians, and ensuring disadvantaged groups would not be left behind. This was necessary because, the HSSP noted, “… the EHP has consistently been financially unobtainable and unsustainable.” Finding 3.1: SMART capacity building and HSS interventions emerged as the most significant accomplishments off the ONSE Activity. HSS interventions were considered among the best practices. GH EvaLS Malawi ONSE Endline Performance Evaluation / 37 Of the 27 accomplishments noted by key informants, three received two or more citations (see Table 10). Of the 22 best practices noted, key informants cited five multiple times (see Table 11). In both cases, the highlighted accomplishments and best practices were drawn from a longer list of responses that focused primarily on either capacity building conducted by ONSE or its efforts to strengthen the health system.14 This is confirmed by ONSE’s monitoring data, which indicated that the Activity met or exceeded nearly all the targets for SMART capacity building, HSS, and addressing contextual factors (see Annex 6). Table 10: Top Three Most Significant Accomplishments of ONSE, per KIIs Intervention KII Respondents Improved WASH at the facility level 10 Improving competencies of health workers 6 Integrated support supervision 6 Table 11: Top Five Best Practices that Emerged from ONSE, per KIIs Intervention KII Respondents DHIS2, data collection, and reporting 5 Intersectoral collaboration 4 Supportive supervision 4 Community engagement 3 Joint planning 3 Highlights from ONSE’s monitoring data include the following: • Of the target of 102 facilities, 141 (153 percent) were provided with improved infrastructure. • ONSE supported 10,057 integrated family health outreach clinics to provide critical preventive and curative services. • ONSE established and revitalized CHAGs and community champions to enhance communities’ role in supporting community-led approaches with the involvement of village health committees. The LOA target for community champions was 100; the LOA achievement was 127. • More than 3.8 million youth used youth-friendly health services—253 percent of the target of approximately 1.5 million. Finding 3.2: While key informants noted many lessons learned, only one stood out. Through its analysis of the KII transcripts, the evaluation team derived 32 lessons learned, as cited by participants. However, more than two respondents cited only one of those lessons (ISS and mentorship leads to improvement in skills in all service areas)— noted by seven respondents. Nearly all the other lessons learned received only one reference. Examples included: 14 The World Health Organization describes the six building block components of HSS as 1) leadership and governance; 2) service delivery; 3) health system financing; 4) health workforce; 5) medical products, vaccines, and technologies; and 6) health information systems. https://extranet.who.int/nhptool/BuildingBlock.aspx “The project has mentored many MOH personnel in multi-tasked areas such as planning, budgeting, report writing, supervisory skills, etc. ONSE improved infrastructure in targeted health facilities such as pit latrines, placenta pits, drilling of boreholes, etc.” —Key informant GH EvaLS Malawi ONSE Endline Performance Evaluation / 38 • A large project requires massive preparation before it starts full implementation. • Communities need to be supervised regularly to achieve maximum results of the health interventions they conduct. • Effective empowerment involves bringing community and health facilities together to find solutions to challenges. • The more people use data, the more the data improves. • It is important to provide allowances on time to keep morale high. • Program managers and coordinators need to be engaged to avoid implementation gaps. Finding 3.3: In accomplishing its results, ONSE experienced several unintended consequences. Unintended consequences differ from expected outcomes. They can be either positive or negative.15 ONSE’s commitment to strengthening gender parity and engaging women as leaders at the community level may have affected community power structures. ONSE used CHAGs as an avenue to promote women’s leadership within communities, give women and girls voices, and provide a platform for women to join district health facility management teams. This may have contributed to changes in gender norms in Malawi. Implications for the future are not clear. ONSE envisioned the mobile money system16 as the main vehicle for per diem and transportation disbursements, daily subsistence allowances, and other payments at the district and community levels. Delayed payment of lunch allowances was a major issue that health service assistants and CHAGs mentioned several times in community-level FGDs. Participants who shared their experiences described the negative impacts of late payments on their morale. Although the system was thought to be innovative, it did not work as expected. Delayed payments (for as long as two months), created discontent, affected participants’ motivation, and may have affected service delivery (for example, data collectors observed data entry staff sitting at their desks instead of entering data). Finding 3.4: In collaboration with other relevant stakeholders, ONSE relied primarily on the rollout of information systems to monitor the progress of interventions and ensure that services were scaled up and used and that any challenges to scale-up were mitigated. ONSE conducted baseline (2016–2017) and endline assessments (March 2022) to measure changes in the availability of health services offered by facilities in Activity districts. The baseline assessment included a census of every facility in the ONSE districts, and the endline assessment included a sampled subset of the facilities. The assessments found a substantial increase from baseline to endline in public facilities offering child health, FP, and maternal and newborn services (for example, facilities offering FP services increased from 82 percent at baseline to 100 percent at endline). Key informants reported that ONSE expanded services through routine collaboration with other stakeholders—particularly USAID IPs such as Health Communication for Life, Breakthrough Action and the Global Health Supply Chain– Procurement and Supply Management Activity. ONSE also collaborated with other partners, such as Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ), Last Mile Health, and UNICEF to address specific issues. This collaboration involved routine sharing of data and other information; joint district-level planning via informal and formal mechanisms (working groups) to avoid or limit duplication of efforts and a comprehensive and cohesive approach; joint participatory site visits, when possible (e.g., 15 Merton, R. (1936) “The Unanticipated Consequences of Purposive Social Action.” American Sociological Review 1 (6): 894. https://doi.org/10.2307/2084615. 16 The mobile money system was ONSE’s preferred payment method for district activities because it is traceable, user-friendly, quick, secure, and transparent, with robust built-in controls and checks. GH EvaLS Malawi ONSE Endline Performance Evaluation / 39 before and after COVID-19 lockdowns); discussions of emerging best practices, innovations, and lessons learned; and the cultivation of shared community champions to benefit all activities. Finding 3.5: While ONSE incorporated measures to ensure the greater sustainability of interventions, these did not always align precisely with what key informants believed were the top factors for either ensuring sustainability or those that might pose risks to long-term adoption of practices. One definition of sustainability in public health is the capacity to maintain program services at a level that will provide ongoing prevention and treatment for a health problem after the termination of major financial, managerial, and technical assistance from an external donor.17 Some ONSE approaches may contribute to the continued use of interventions after the LOA. These included: • Use of District Health Management Team Physical Asset Management units to provide technical support and Activity oversight. • Reinforcing MOH technical and managerial capacity to ensure a meaningful transition as the ONSE Activity ended. • Transitioning to the use of digital health tools for better data collection, feedback, and planning. • Intensive skills transfer in integrated supervision, data collection, patient care, and other areas. • Empowering communities through community champions and community action groups to identify challenges in their catchment areas and find sustainable solutions to local issues. • The development of infrastructure such as maternity facilities, incinerators, and boreholes. • Support to a sustainability task force (however this support was initiated too late to provide any notable benefit). The general perception among key informants was that capacity building of MOH personnel will contribute to sustainability because they will continue to use their new skills in Malawi. However, as seen in Table 12, perhaps some additional practices should receive greater emphasis to ensure sustainability. Additionally, as presented in Table 13, ONSE may have needed to survey its stakeholders at the start of implementation to ensure that the greatest perceived challenges to sustainability were further mitigated (although some of those challenges were outside of ONSE’s manageable interest). Table 12: Top Five Practices for Ensuring Activity Sustainability, per KIIs Practice KII Respondents Ownership and empowerment of community 10 Community engagement 8 Mentorship of health workers 7 Economic empowerment of CHAGS 6 Training of officials at the health facility level to collect and input DHIS2 data 4 17 Claquin, P. (1998). Sustainability of EPI: Utopia or Sine Qua Non Condition of Child Survival. Arlington, Virginia: Resources for Child Health Activity. “Probably a few of the initiatives will be maintained, but obviously not all of them … I remember some of the districts already started supporting some of the initiatives … possibly they are going to be sustained, but most of them that have been costly, I think it will be doubtful [those will be sustained] if support is not coming forth.” —Key informant GH EvaLS Malawi ONSE Endline Performance Evaluation / 40 Table 13: Five Greatest Challenges for Ensuring Sustainability, per KIIs Challenge KII Respondents No smooth or well-planned exit 9 No follow-on funding by GOM 8 Need for allowances 6 Lack of medical supplies 5 Cultural barriers 5 Finding 3.6: ONSE relied on routine Activity performance indicators to measure and monitor the scale-up of activities. However, no defined set of indicators was identified to track quantifiable progress on sustainability efforts. Although sustainability was a clear goal in the Activity design, clear and distinct indicators demonstrating progress (or lack of progress) on sustainability were not always available. Instead, ONSE primarily used existing measurement systems (HMIS, DHIS2, and LMIS) to collect routine data to measure health indicators and monitor their progress. Through these indicators, ONSE monitored the progress of its work to determine whether the Activity was meeting its targets in areas such as HSS, access to EHP, quality of health services, and others. Including Activity-specific indicators in its AMEP would have been an informative addition to the Activity’s monitoring and, in conjunction with contextual indicators, could have reflected a better sense of the sustainability of interventions and actions that were needed to mitigate any manageable challenges during the life of the Activity. Finding 3.7: Key informants cited numerous challenges to ONSE’s implementation, but there was no clear consensus on which were the most significant. In all likelihood, ONSE could have fully or partially mitigated some of the challenges, but others were outside of its manageable interests. Challenges cited by key informants that ONSE could have fully or partially mitigated are listed in Table 14. Table 14: Challenges that ONSE Could Have Mitigated Challenge Addressed? (Y/N/Partially) How? Sub-optimal collaborative efforts by ONSE at the Activity’s outset Yes/partially • Repeated engagement with relevant stakeholders • Change of approach to soliciting ideas (within the bounds of its agreement) Overpromising support/interventions prior to implementation No • N/A Delayed allowance payments Yes/partially • Adoption of an electronic or mobile payment approach, although it did not work as expected • Limiting participants to those with phones (which may have had unintended inequity consequences) Discontent with allowance provided Yes/partially • Onsite trainings, although these may have actually decreased participation Stockouts of medicines and commodities at facilities Yes/partially • Transportation provided to supply commodities to health facilities facing stockouts when possible GH EvaLS Malawi ONSE Endline Performance Evaluation / 41 Challenge Addressed? (Y/N/Partially) How? Lack of MOH/GOM active participation Yes/partially • Repeated engagement, including, as possible, providing leadership roles in discussions (although some GOM officials expected direct financial benefits from participation) Decreasing interest by community participants (CHAGs and youth) because of a lack of or limited incentives No • N/A Lack of contextual understanding Yes/partially • Improved throughout the life of the Activity, although COVID-19 limited site visits for an extended period Abrupt ending of some interventions (for reasons of funding or otherwise) Yes/partially • Improved communications with stakeholders, particularly at the district level Attrition of health workers Yes • Repeated trainings, as well as follow-up mentoring Transition of project activities (and sustainability) Yes/partially • Addressed in Finding 3.5 Challenges cited by key informants that were essentially outside ONSE’s manageable interest included: • Staff shortages at facilities; • Lack of roads to access hard-to-reach areas; • Limited resources; implementation limited to 16 districts; • People seeking services outside the normal health facility catchment; • Ongoing issues with utilities, primarily electricity; • Scarcity of medicines and commodities; • Lack of GOM procurement capacity; and • Cultural barriers (e.g., early marriage and age of sexual debut). 7.4. EVALUATION QUESTION 4 What were the contextual factors, such as socioeconomic factors, gender, demographic factors, environmental characteristics, baseline health conditions, health services characteristics, and so forth that affected implementation and outcomes of the ONSE Activity? 4a: What key findings emerged? 4b: What were the challenges faced and how were they overcome? 4c: What more can be done in future activities of similar nature? Finding 4.1: There was no single significant contextual factor cited by key informants as facilitating the implementation of ONSE. Although ONSE addressed contextual and other enabling environment factors through its interventions (see Table 14), key informants did not mention what the most important factors were. Key informants did not mention any response more than three times. Some factors noted (of the 13 options) were: • The flexibility and responsiveness of ONSE’s approach and activities; • CHAGs linkages between communities and health facilities; • An established and coordinated MOH framework; GH EvaLS Malawi ONSE Endline Performance Evaluation / 42 • The overall enabling policy environment; • Having experienced coordinators within districts; • Health workers with a positive attitude toward learning and using new knowledge; and • The use of existing partnerships with the government. Table 15: Contextual Factors Addressed by the ONSE Activity Finding 4.2: Key informants cited several contextual factors as limiting ONSE’s ability to implement interventions, and which the Activity addressed through its responsiveness to changing needs, initiating GH EvaLS Malawi ONSE Endline Performance Evaluation / 43 cost-cutting measures, or greater involvement by community counterparts. However, not all limiting factors could be addressed. As a result of these limiting factors, ONSE had to pivot on some interventions to ensure, for example, service continuity during the COVID-19 pandemic and the supportive supervision and mentoring that were foundational to its assistance. The Activity had to reprioritize interventions with a focus on district support, especially activities that facilitated service delivery such as FP and integrated family health outreach clinics; reduce central-level activities; and postpone resource-intensive activities such as facility renovation and WASH. In recognizing that further community engagement was needed, per key informants, ONSE increased its outreach efforts in at the local level with relevant stakeholders, especially to reduce cultural barriers to accessing the COVID-19 vaccine when it became available. Finally, the Activity initiated several cost-cutting measures such as reducing central level travel, and office rentals, and other line items when possible. However, several contextual factors were outside ONSE’s manageable interests such as the level of funding (both Activity-specific and for the health sector in general), general macroeconomic conditions, and lack of facility access during the rainy season. Table 16: Top Five Contextual Factors that Constrained ONSE, per KIIs Factors KII Respondents Slow logistics 16 Shortage of funds 13 Exclusion of local structures 7 COVID-19 pandemic 6 Drug stockouts 6 Finding 4.3: Future activities can support the systematic use of data at the facility and community levels in each district to address some of the limiting factors. Malawi is moving steadily toward unifying and strengthening HIS, including DHIS2. The ONSE project correctly relied on and built capacity around existing data systems rather than creating a parallel data system for its AMEP and routine performance monitoring. Ensuring that these unified systems capture not only intervention-specific indicators and corresponding data, but also contextual factors should help to mitigate some of the issues which may impede implementation. Further, the continued strengthening of data-drive decision-making using a unified information system should provide continued progress in the areas which ONSE address as part of its Activity. “ONSE also aimed at complementing the DHIS2 effort in terms of the programs that they were supporting … There were improvements in terms of consistency of the reports that from the community level and district level for use at central level.” —Key informant GH EvaLS Malawi ONSE Endline Performance Evaluation / 44 8. CONCLUSIONS ONSE supported the GOM to provide essential health care services across 16 districts in several priority health areas covering a population estimated at 11 million to 12 million. To achieve its results, the Activity focused on four main objectives: 1. Increasing access to priority health services; 2. Improving the quality of priority health services; 3. Strengthening the performance of health systems; and 4. Increasing demand for priority health services. Implemented from November 2016 to June 2022, ONSE was considered one of USAID/Malawi’s flagship health activities and therefore merited an endline performance evaluation. The evaluation sought to assess the extent to which the Activity achieved its objectives regarding the quality of and access to care, provide insights into factors that facilitated and limited increased service utilization at each level of service delivery, and document progress toward building GOM capacity to deliver quality health services. Using the four primary questions below, and several sub-questions, the evaluation used a mixed methods approach to collect data for analysis. This resulted in the 17 findings discussed in Section 7, from which the following conclusions are derived. Evaluation Question 1: To what extent did the Activity's interventions achieve their intended results? For the most part, ONSE achieved its intended results as demonstrated by the monitoring data analyzed. Despite some shortcomings in certain categories (e.g., HSS and service quality), the Activity met or exceeded most targets. ONSE achieved LOA targets for 82 percent of its core indicators and at least 90 percent of LOA targets for 35 of 51 core performance indicators. While some key informants noted discord within the Activity because of differences in support packages that districts received, this did not significantly affect ONSE’s ability to implement interventions, as demonstrated by the end results. Likewise, the onset of the COVID-19 pandemic did not appear to substantially reduce provision of services or achievement of results, pointing to ONSE’s ability to adapt interventions to the changed circumstances. While some targets may have been affected due to the redirection of funding for COVID-19 and other unforeseen events, it appears to have navigated most of those challenges successfully. Finally, because of ONSE’s focus on engagement with multiple stakeholders, abundant information provided by key informant feedback is part of this evaluation. This feedback points to future directions that USAID/Malawi might consider if it decides to continue with efforts similar to ONSE. Evaluation Question 2: To what extent did ONSE improve access to priority high impact EHP interventions in the target health facilities and community service delivery points? Examining only monitoring indicators focused on EHP, it can be concluded that ONSE substantially improved access to EHP services. Even though only three of the seven indicators listed in Table 7 met or exceeded their targets, three of the four remaining indicators did not underperform significantly: All were in the 90 percent to 95 percent range (the remaining indicator was at 81 percent of its target). Per key informant feedback, interventions that appeared to have the greatest effect on improving EHP services provision were aimed at HSS. Notably, these included improvements in supportive supervision, planning, and coordination, as well as efforts to strengthen the HIS and use of data for decision-making. Other interventions of note were ONSE’s renovation of facilities, including provision of some GH EvaLS Malawi ONSE Endline Performance Evaluation / 45 equipment, and strengthening of the supply ordering system. By utilizing the SMART capacity building approach, ONSE enhanced the health sector’s overall systemic capacity at various geographic levels and improved the performance of individuals, teams, and structures working in and with Malawi’s health system. Per key informant feedback, this increased a sense of programmatic ownership among Activity participants. For future activities, USAID/Malawi should consider and examine whether these EHP improvements were significantly better in ONSE districts than non-ONSE districts, as the evaluation’s limited data analysis around this issue was inconclusive. Evaluation Question 3: What are the most significant accomplishments, best practices, and lessons learned from the ONSE Activity? Similar to the conclusion for Evaluation Question 2, stakeholders indicated that ONSE’s HSS interventions were its most significant accomplishment, along with emerging best practices. They also identified the SMART capacity building approach as a notable achievement of the Activity. Given this recurring theme for these two areas, it can reasonably be concluded that future approaches to improving the health of Malawians (whether funded externally or domestically) should seriously consider incorporating a comprehensive design and hands-on methods for improving health services. Of particular note was the main lesson learned from ONSE: ISS and mentorship lead to improved skills in all service areas. Similarly, future approaches should address the unintended negative consequences of payment methods and incorporate better understanding of local gender dynamics. While ONSE tracked the progress and scale-up of its interventions, primarily through the rollout of information systems and use of data, and utilized measures to better ensure sustainability, these efforts could have been improved. For example, ONSE could have been more proactive in ensuring scale-up in ways other than solely monitoring—for example, through focused and collaborative advocacy efforts with GOM officials. Likewise, the Activity could have better mitigated sustainability challenges by devoting measures to this issue and by seeking information at the start from a broad range of stakeholders regarding current and future challenges to sustainability. Finally, while ONSE, like any other activity, faced numerous implementation challenges both within and outside its manageable interests, it addressed most manageable challenges only partially through mitigation measures. Whether this was due to the Activity’s management or other factors is unknown. Evaluation Question 4: What were the contextual factors such as socioeconomic factors, gender, demographic factors, environmental characteristics, baseline health conditions, health services characteristics, and so forth that affected implementation and outcomes of the ONSE Activity? Although ONSE addressed and measured a number of contextual factors related to implementation, key informants could not cite any one conclusive factor that aided in its success. However, several elements of a generally enabling environment (e.g., policies, coordination frameworks and mechanisms, enthusiastic and experienced personnel, and others.) point to the need to ensure that such elements are in place to ensure successful implementation. Respondents were more specific and cited more contextual factors that impeded implementation, although most of those were somewhat or wholly outside of ONSE’s manageable interests (e.g., supply chain issues, lack of funding) or unforeseen events (e.g., the COVID-19 pandemic). Thus, while it is commendable that ONSE achieved a substantial portion of its intended results despite these challenges, opportunities for improvement (and, consequently, recommendations for future activities and interventions) remain. GH EvaLS Malawi ONSE Endline Performance Evaluation / 46 9. RECOMMENDATIONS Because this performance evaluation was conducted after the ONSE Activity ended, the recommendations listed below are directed at any future activities, whether funded domestically or by development partners. They are based on the findings discussed in this report and are divided into those that should be considered high priority and others for consideration. Finally, because external funding should be used to support Malawi’s HSSP, in general the recommendations are aimed at all interested stakeholders. However, the recommendations indicate which stakeholders might be the primary leads. 9.1. HIGH PRIORITY 1. Regardless of the health issue to be addressed, an integrated HSS approach should be adopted. Within the six components of HSS, the following building blocks should be prioritized; • Health workforce development, particularly supportive supervision and mentoring; • Support to HIS, including the routine use of data for decision-making; and • Ensuring medical products, vaccines, and other technologies are available by strengthening the procurement and supply chain management systems. Lead stakeholders: USAID and the MOH equally. 2. Continue to use the SMART capacity building approach (as shown in Figure 11 and noted throughout the report). As reflected in the findings, participating stakeholders appreciated this approach, which should increase the sustainability of interventions (e.g., through enhanced skills and trainings). Lead stakeholders: USAID and its IPs primarily; potentially the MOH. 3. Balance demand for quality services with supply by ensuring that any participating health facilities are fit for purpose. To do so, conduct initial harmonized health facility assessments18 to learn whether basic services (e.g., sanitation, infection prevention and control, electricity, clean water sources, waste disposal, etc.) are in place prior to the implementation of interventions and allow resources to be directed to resolving any infrastructure-related issues. Lead stakeholders: USAID and development partners primarily; potentially the MOH. 4. Continue to engage with communities, particularly around issues of mutual accountability for the quality of health services while offering members non-financial incentives for participating. The use of community scorecards has been shown to increase health service responsiveness and quality in numerous countries19 and provides communities a tool to advocate for their health needs. However, as has also been shown, while they appreciate the improved services, community members often need other incentives. These can be purpose-driven (e.g., fostering a sense of shared goals), status-related (e.g., conferring prestige and recognition), and socially based (e.g., developing forums and events that are enjoyable and strengthen relationships). Lead stakeholders: USAID and MOH equally. 5. Define “sustainability” before implementing interventions and build it into any future activities. This involves including sustainability-related indicators in project documents and developing a sustainability, transition, and exit strategy that should be disseminated to and discussed with all counterparts (e.g., government officials, participating facilities, community members/clients of 18 https://www.who.int/data/data-collection-tools/harmonized-health-facility￾assessment/introduction#:~:text=The%20Harmonized%20Health%20Facility%20Assessment,at%20required%20standards%20of %20quality. 19 CDC/ATSDR Committee on Community Engagement. (2011). Principles of Community Engagement, Second Edition. GH EvaLS Malawi ONSE Endline Performance Evaluation / 47 facilities, and development partners) within the first year of implementation. This would allow for necessary joint planning to occur early and would define responsibilities for ensuring the short-, medium-, and long-term sustainability of activities. Lead stakeholders: USAID and development partners primarily; potentially the MOH. 9.2. FOR CONSIDERATION 6. Build contingency plans into planning and programming documentation to allow for pivots, if needed, to respond to unforeseen events. While this evaluation found that the COVID-19 pandemic did not have a decisive impact on ONSE’s ability to reach its goals, this was not the case for other activities in other countries. Nor was the COVID-19 pandemic the only type of unforeseen event that an activity might encounter. Thus, it is imperative that contingency and mitigation planning occur during the design stage. Further, development partners such as USAID should consider the types of agreements (e.g., grants, cooperative, agreements, contracts) that would best allow an activity to respond quickly to changing circumstances. Lead stakeholders: USAID; potentially the MOH. 7. Differentiate among partners’ areas of manageable interest and lines of accountability. While, for example, ONSE’s AMEP included indicators for contextual factors, many issues that the Activity encountered were only partially within (and, in some cases wholly outside) its manageable interest and ability to respond. A similar future activity could include a governing body comprising not only the donor organization and the IP, but also government counterparts, local participants, and community members. The terms of reference of such a group might include overseeing implementation, identifying bottlenecks, and, when possible, leveraging its members’ positions to address and/or remove any impediments to implementation. Lead stakeholders: USAID and MOH equally. 8. Communicate early and widely, and explain the reasoning behind any differences in support to be provided. Whether funding is provided domestically or from a development partner, there are almost always gaps in support. Some geographic areas will be prioritized over others. Thus, the core stakeholders (the donor, the IP, technical working groups, etc.) must communicate to all participants the criteria for selecting participating districts and facilities and explain why levels of support may differ. While communication may not mitigate all issues, it allows participants opportunities to provide additional feedback and better understand the analysis that underlies the decision-making process. Lead stakeholders: USAID and MOH equally. 9. If resources are available, conduct an impact evaluation for any future activity, including any possible spillover effects. Impact evaluations can require years of planning and can be costly and labor￾intensive. However, they can provide invaluable insights into the true extent of an activity’s reach and effects. This endline evaluation offers some insights into ONSE’s performance and its strengths and challenges, and attempts some cursory comparative analysis. However, the only robust method to ensure full understanding of the Activity’s effects would have been the establishment of a counterfactual. While Malawi is geographically small, and spillover is possible of (e.g., interventions having effects beyond geographic boundaries), well-designed counterfactuals can incorporate these effects by, for example, including temporal elements and/or spatial analysis using geographic information systems. Lead stakeholders: USAID and development partners primarily; potentially the MOH. GH EvaLS Malawi ONSE Endline Performance Evaluation / 48 ANNEX 1: SCOPE OF WORK GH EvaLS Malawi ONSE Endline Performance Evaluation / 49 GH EvaLS Malawi ONSE Endline Performance Evaluation / 50 GH EvaLS Malawi ONSE Endline Performance Evaluation / 51 GH EvaLS Malawi ONSE Endline Performance Evaluation / 52 GH EvaLS Malawi ONSE Endline Performance Evaluation / 53 GH EvaLS Malawi ONSE Endline Performance Evaluation / 54 GH EvaLS Malawi ONSE Endline Performance Evaluation / 55 GH EvaLS Malawi ONSE Endline Performance Evaluation / 56 GH EvaLS Malawi ONSE Endline Performance Evaluation / 57 GH EvaLS Malawi ONSE Endline Performance Evaluation / 58 GH EvaLS Malawi ONSE Endline Performance Evaluation / 59 GH EvaLS Malawi ONSE Endline Performance Evaluation / 60 GH EvaLS Malawi ONSE Endline Performance Evaluation / 61 GH EvaLS Malawi ONSE Endline Performance Evaluation / 62 GH EvaLS Malawi ONSE Endline Performance Evaluation / 63 GH EvaLS Malawi ONSE Endline Performance Evaluation / 64 GH EvaLS Malawi ONSE Endline Performance Evaluation / 65 GH EvaLS Malawi ONSE Endline Performance Evaluation / 66 GH EvaLS Malawi ONSE Endline Performance Evaluation / 67 GH EvaLS Malawi ONSE Endline Performance Evaluation / 68 GH EvaLS Malawi ONSE Endline Performance Evaluation / 69 GH EvaLS Malawi ONSE Endline Performance Evaluation / 70 GH EvaLS Malawi ONSE Endline Performance Evaluation / 71 GH EvaLS Malawi ONSE Endline Performance Evaluation / 72 GH EvaLS Malawi ONSE Endline Performance Evaluation / 73 GH EvaLS Malawi ONSE Endline Performance Evaluation / 74 GH EvaLS Malawi ONSE Endline Performance Evaluation / 75 GH EvaLS Malawi ONSE Endline Performance Evaluation / 76 GH EvaLS Malawi ONSE Endline Performance Evaluation / 77 GH EvaLS Malawi ONSE Endline Performance Evaluation / 78 GH EvaLS Malawi ONSE Endline Performance Evaluation / 79 GH EvaLS Malawi ONSE Endline Performance Evaluation / 80 GH EvaLS Malawi ONSE Endline Performance Evaluation / 81 GH EvaLS Malawi ONSE Endline Performance Evaluation / 82 GH EvaLS Malawi ONSE Endline Performance Evaluation / 83 GH EvaLS Malawi ONSE Endline Performance Evaluation / 84 GH EvaLS Malawi ONSE Endline Performance Evaluation / 85 ANNEX 2: DATA COLLECTION TOOLS GH EvaLS Malawi ONSE Endline Performance Evaluation / 86 GH EvaLS Malawi ONSE Endline Performance Evaluation / 87 GH EvaLS Malawi ONSE Endline Performance Evaluation / 88 GH EvaLS Malawi ONSE Endline Performance Evaluation / 89 GH EvaLS Malawi ONSE Endline Performance Evaluation / 90 GH EvaLS Malawi ONSE Endline Performance Evaluation / 91 GH EvaLS Malawi ONSE Endline Performance Evaluation / 92 GH EvaLS Malawi ONSE Endline Performance Evaluation / 93 GH EvaLS Malawi ONSE Endline Performance Evaluation / 94 GH EvaLS Malawi ONSE Endline Performance Evaluation / 95 GH EvaLS Malawi ONSE Endline Performance Evaluation / 96 GH EvaLS Malawi ONSE Endline Performance Evaluation / 97 GH EvaLS Malawi ONSE Endline Performance Evaluation / 98 GH EvaLS Malawi ONSE Endline Performance Evaluation / 99 GH EvaLS Malawi ONSE Endline Performance Evaluation / 100 GH EvaLS Malawi ONSE Endline Performance Evaluation / 101 GH EvaLS Malawi ONSE Endline Performance Evaluation / 102 GH EvaLS Malawi ONSE Endline Performance Evaluation / 103 GH EvaLS Malawi ONSE Endline Performance Evaluation / 104 Key Informant Interview Guide (FINAL additional interviews – district level) INTERVIEW GUIDE (Note to data collectors: Please feel free to probe for more in-depth information if the interviewee states something of interest or something which is unusual) 1. (Background question): What has been your engagement in the ONSE Project and what are the ways in which you have interacted with it? 2. (EQ2) What types of support has ONSE provided to you? What do you believe were the most beneficial activities / technical assistance that ONSE provided for your work? 3. (EQ1) During the life of the ONSE project, what changes, if any, were made to activities? Did those changes positively or negatively affect your interaction with the project and any goals you had wanted to achieve? 4. (EQ4) What were the main challenges the ONSE project faced and how were they overcome? 5. (EQ4) Are there any greater contextual factors (e.g., socioeconomic factors, political factors, environmental issues, etc.) that affected implementation and your interaction with the ONSE project? If so, how did these factors affect the results achieved? 6. (EQ3) What are the most significant accomplishments, best practices, and lessons learned from the ONSE Activity? GH EvaLS Malawi ONSE Endline Performance Evaluation / 105 ANNEX 3: LIST OF DOCUMENTS REVIEWED 1. Activity Monitoring and Evaluation Plan: Organized Services for Everyone’s Health (ONSE Health). March 2017. 2. ONSE Health Activity Malawi: Annual Activity Report- PY1. October 2017. 3. ONSE Health Activity Malawi: Annual Activity Report- PY2. October 2018. 4. ONSE Health Activity Malawi: Annual Activity Report- PY3. October 2019. 5. ONSE Health Activity Malawi: Annual Activity Report- PY4. October 2020. 6. ONSE Health Activity Malawi: Annual Activity Report- PY5. October 2021. 7. ONSE Quarterly reports, 2016-2022 8. Organized Network of Services for Everyone’s Health (ONSE) Activity Malawi: Annual Workplan FY17. January 2017. 9. Organized Network of Services for Everyone’s Health (ONSE) Activity Malawi: Annual Workplan Y2. September 2018. 10. Organized Network of Services for Everyone’s Health (ONSE) Activity Malawi: Annual Workplan Y3. August 2019. 11. Organized Network of Services for Everyone’s Health (ONSE) Activity Malawi: Annual Workplan Y4. August 2020. 12. Organized Network of Services for Everyone’s Health (ONSE) Activity Malawi: Annual Workplan Y5. August 2021. 13. ONSE Health Activity: Health Facility Assessment- District Assessments. January 2018. 14. Together For Everyone’s Health: ONSE Final Project Report- Malawi, 2016-2022. June 2022. 15. ONSE Health Activity Malawi: Endline Report. July 2022. 16. Request for Proposal (RFP) No: SOL-612-16-000001- Organized Network of Services for Everyone’s (ONSE) Health. February 2016. 17. Government of the Republic of Malawi: Health Sector Strategic Plan II (2017-2022). April 2017. 18. A cluster randomized trial of delivery of intermittent preventive treatment of malaria in pregnancy at the community level in Malawi: IPTP in Malawi. 2020. 19. Management of Health Information in Malawi: Role of Technology. Advances in Science Technology and Engineering Systems Journal. January 2017. 20. Malawi Government: National Health Communication Strategy 2015-2020. 2020. 21. Draft Baseline Report: Malawi Organized Network of Services for Everyone’s Health (ONSE) Impact. November 2017. 22. Government of the Republic of Malawi: Ministry of Health- National Digital Health Strategy 2020-2025. May 2020. 23. Government of the Republic of Malawi: Ministry of Health- Malawi National Health Information System Policy. September 2015. 24. Government of the Republic of Malawi: Ministry of Health- National Community Health Strategy 2017- 2022. July 2017. 25. ONSE Health Activity Malawi: WASH Strategy Document. June 2018. 26. Impact Evaluation of Malawi’s Organized Network of Services for Everyone’s (ONSE) Health Project: Baseline Report. July 2018. 27. The Republic of Malawi Ministry of Health: The Malawi COVID-19 Vaccine Deployment Plan. February 2021. GH EvaLS Malawi ONSE Endline Performance Evaluation / 106 28. ONSE Health Activity: Addressing Malaria in Pregnancy in Malawi-Community-Based Delivery of IPTp: Study Brief. December 2021. 29. ONSE Health Activity: SMART CAPACITY BUILDING Brief. 30. ONSE Health Activity: ONSE Joint-Planning and Resource Allocation Process Brief. 31. Unpacking our Impact: The ONSE Health Activity Approach to Planning Presentation. June 2020. 32. Civil Society Organization Capacity Assessment. January 2017. 33. ONSE Partner Mapping 2017. 34. ONSE Health Activity: ONSE Highlights Brief. 35. ONSE Health Activity: Youth-Friendly Health Services (YFHS) Assessment. 36. Assessment of EMR Systems in Malawi: Draft Initial Landscape Assessment. Vital Wave. February 2019. 37. Organized Network of Services for Everyone’s (ONSE) Health Activity, Malawi: Strategic Communications and Outreach Strategy. June 2017. 38. ONSE Health Activity: Community Mobilization and Engagement Brief. 39. ONSE Health Activity: Fixed Fee Comments. 40. ONSE Health Activity: Annual Survey Approach. GH EvaLS Malawi ONSE Endline Performance Evaluation / 107 ANNEX 4: QUALITATIVE RESEARCH ACTIVITIES Organization and Districts Planned Completed USAID (Malawi Mission) 4 2 KIIs 4 2 ONSE (Central and District Level) 4 5 KIIs 4 5 Central Government (Central Level) 4 4 KIIs 4 4 Implementing Partners 3 4 KIIs 3 4 International Development Actors 4 2 KIIs 4 2 Lilongwe 12 14 KIIs 10 GIs 2 FGDs 2 Balaka 12 12 KIIs 7 GIs 2 FGDs 3 Machinga 12 11 KIIs 5 GIs 3 FGDs 3 Chikwawa 12 14 KIIs 10 GIs 2 FGDs 4 Karonga 12 14 KIIs 9 GIs 2 FGDs 1 Nkhata Bay 2 KIIs 2 Ntcheu 1 KIIs 1 Chitipa 1 KIIs 1 Kasungu 1 KIIs 1 Nkhotakota 2 KIIs 2 Total 79 89 GH EvaLS Malawi ONSE Endline Performance Evaluation / 108 ANNEX 5: LENGTH-OF-ACTIVITY DATA INDICATOR TARGETS AND RESULTS BY USAID IR AND SUB-IR Evaluators conducted secondary review of quantitative data collected, analyzed and reported by the USAID Malawi ONSE Project to assess LOA performance. Data were sourced from the ONSE Project Final Report (Draft), as provided to more accurately and comprehensively analyze trends and changes in performance indicators over the full 5+ years of project implementation as well as to evaluate whether planned targets were met. Below are tables of ONSE Key Performance Indicators for each ONSE IR and Sub-IR approved by USAID, their targets, their endline results, and the LOA quantitative change in final results from the baseline. Baseline data was not available for some indicators, as noted in the LOA Change from Baseline column, and quantitative change from baselines could not be determined for those indicators. A “traffic light” shading is applied to % of LOA targets achieved, with Green indicating 100% or higher, Yellow – 90-99% target achievement, and Red – an 89% or lower target result (legends included at the bottom of each table). LOA Change from Baseline results cells shaded gray indicate a decrease in achievement from the related indicator target set for the LOA. For this evaluation, an LOA Target Achieved score was considered for indicator results that met 90% or above of the approved target by the conclusion of project activities and funding and as presented in the Draft Final Report of the ONSE Project. LOA Change from Baseline results cells shaded in gray indicate a decrease in achievement from the related indicator target set. GH EvaLS Malawi ONSE Endline Performance Evaluation / 109 GH EvaLS Malawi ONSE Endline Performance Evaluation / 110 GH EvaLS Malawi ONSE Endline Performance Evaluation / 111 GH EvaLS Malawi ONSE Endline Performance Evaluation / 112 GH EvaLS Malawi ONSE Endline Performance Evaluation / 113 GH EvaLS Malawi ONSE Endline Performance Evaluation / 114 GH EvaLS Malawi ONSE Endline Performance Evaluation / 115 GH EvaLS Malawi ONSE Endline Performance Evaluation / 116 GH EvaLS Malawi ONSE Endline Performance Evaluation / 117 GH EvaLS Malawi ONSE Endline Performance Evaluation / 118 GH EvaLS Malawi ONSE Endline Performance Evaluation / 119 ANNEX 6: SMART CAPACITY BUILDING, HSS, AND CONTEXTUAL FACTORS ACCOMPLISHMENTS GH EvaLS Malawi ONSE Endline Performance Evaluation / 120 GH EvaLS Malawi ONSE Endline Performance Evaluation / 121 GH EvaLS Malawi ONSE Endline Performance Evaluation / 122 ANNEX 7: DISCLOSURE OF ANY CONFLICTS OF INTEREST GH EvaLS Malawi ONSE Endline Performance Evaluation / 123 GH EvaLS Malawi ONSE Endline Performance Evaluation / 124 GH EvaLS Malawi ONSE Endline Performance Evaluation / 125 Martha Benezet Consultant GH EvaLS Malawi ONSE Endline Performance Evaluation / 126 GH EvaLS Malawi ONSE Endline Performance Evaluation / 127 GH EvaLS Malawi ONSE Endline Performance Evaluation / 128 ANNEX 8: EVALUATION TEAM Joseph Inungu: Co-Team Lead Joseph Inungu is a professor of global health in the School of Health Sciences at Central Michigan University (CMU). He is the founding Director of the Master of Public Health Program at CMU. Before joining CMU, he served as the Director of socio-behavioral research for the International AIDS Vaccine Initiative (IAVI) in Southern Africa. IAVI is a New York-based NGO specializing in HIV vaccine research. Prior to joining IAVI, he worked as the Regional Researcher for Population Services International (PSI) in West and Central Africa. He provided technical assistance and oversaw the design and implementation of social marketing research in six African countries including Benin, Burkina Faso, DR Congo, Liberia. Nigeria, and Senegal dealing with the prevention of HIV/AIDS and Malaria, RH, water, sanitation and hygiene. He conducted program evaluations and capacity building in conducting research in participating countries. Dr Inungu authored more than 50 articles in peer-reviewed journals and presented several papers at national and international conferences. He co-authored the textbook titled Foundation of Rural Public Health in America published by Jones & Barletts Learning in 2022. He holds a Master’s and a Doctor of Public health degrees from Tulane University School of Public Health and a Doctor of Medicine degree from the University of Kinshasa in DR Congo Tim A. Clary (M.A., EMBA, M.S./Ph.D.), Co-Team Lead Dr. Tim A. Clary is an independent consultant who focuses on international health and development. An infectious diseases epidemiologist by training, he has provided consulting services for organizations such as USAID, PEPFAR, the Global Fund, several United Nations agencies, the World Bank, the International Finance Corporation, GIZ, the U.K. Department for International Development, and several nongovernmental organizations. His consulting assignments have taken him to more than 65 countries in all geographic regions of the world. From 2011–2013, Dr. Clary was the Director of Health for the Millennium Challenge Corporation. During that time, he oversaw a portfolio of approximately $300 million in USG investments in health, including a $131.5 million nutrition, maternal, and child health and results-based financing initiative in Indonesia; a $120.5 million HSS and HIV/AIDS project in Lesotho; and a $38.5 million noncommunicable disease and injury program in Mongolia. From 2000–2006, he was with USAID, first as the HIV/Infectious Diseases Epidemiologist for Europe and Eurasia overseeing several grants, cooperative agreements, and contracts, and then as a Senior Public Health Advisor for USAID/Ukraine. During his time in Ukraine, Dr. Clary designed, developed, and oversaw several projects covering infectious diseases, RH, maternal and child health, and birth defects for Ukraine, Belarus, and Moldova. Prior to his work with USAID, Dr. Clary was with the U.S. Centers for Disease Control and Prevention’s Office of Research Methodology. Martha Benezet, Evaluation Specialist Martha Benezet is a senior technical advisor and activity leader with 18+ years strengthening national and sub-national systems and building capacity for quality, evidence-based health service delivery in 17 African and Asian countries. She has held leadership roles in the targeted design, management, assessment and evaluation of complex national and sub-national Global Fund, USAID-, PEPFAR-, and MCC-funded health programs, with focus on innovative, cost-effective solutions and expertise in HSS. Prior to her current role with USAID GH EvaLS as an Evaluation Specialist, she was the Director of Global Monitoring, Evaluation, Learning and Communications for the USAID Infectious Disease Diagnostics and Surveillance Activity at ICF International and also a Senior MEL Advisor at University Research Co., LLC (URC). Earlier, she spent 7 years at Abt Associates as a Senior Associate and TB GH EvaLS Malawi ONSE Endline Performance Evaluation / 129 Strategic Lead, providing cross-activity team leadership and technical support for health systems, nutrition, and infectious disease research, health policy development and service delivery activities. Martha began her international career as the TB-HIV Program Manager in South Africa for Medical Care Development International. She has an M.S. in Food Policy and Applied Nutrition from the Friedman School of Nutrition Science and Policy at Tufts University in Boston, Massachusetts. Shabnam Shahnaz, Evaluation Specialist Shabnam Shahnaz is a senior manager and technical consultant with more than 30 years of experience in developing and managing international Health, Population and Nutrition programs and activities. She is a specialist in reproductive health particularly in the activities related to the reduction of maternal morbidity and mortality (with special focus on Maternal, Neonatal and Child Health; Adolescent health, FP; Birth Spacing, Post-Abortion Care, Emergency Obstetric Care, and STDs/ HIV/ AIDS) and Child Survival, Primary Health Care (Immunization, ARI, CDD, integrated management of childhood illness, Nutrition), Infectious diseases (tuberculosis and malaria). Her experience entails conceptualizing, designing, implementing, and evaluating complex national, regional and global programs in more than 10 countries in Asia, Europe, and the Arab World during her work with Pathfinder International, UNICEF, Plan International, Marie Stopes International. She worked for Pathfinder International in different capacities Since her work with Pathfinder, she has served as the activity officer for the Women’s Right to Life and Health Activity with UNICEF and then Regional Director of South Asia, Arab World and East Europe Programs with Marie Stopes International. Shabnam holds a MD from the Lady Hardinge Medical College, University of Delhi, India and an MPH from the University of North Carolina at Chapel Hill, USA with the focus on Health Policy and Administration. She is also a Fellow of the Royal Society for Public Health, United Kingdom. She also has training in leadership, management, quality and performance improvement, service improvement and systems development. Before her current role with USAID GH EvaLS as a Subject Specialist, she has been involved with Chemonics, RTI as technical advisor to support and contribute to business development activities for USAID and DFID activities. Robert Waswaga, Data and Evaluation Specialist Mr. Waswaga is an economist and evaluation practitioner with more than 20 years of experience in international development. He has provided consultancy services in monitoring and evaluation for 18 years, including 97 program evaluations, 53 baseline surveys, and 47 organizational assessments amongst other assignments. Mr. Waswaga’s sector consultancy experience covers areas such as maternal, neonatal and child heath, adolescent SRHR, food security, WASH, basis (primary) education, and other sectors. He has provided consultancy services to government and non-governmental agencies operating in Uganda, South Sudan, DR Congo, Kenya and Tanzania. Some of his clients have included the Government of Uganda, Norwegian Refugee Council, World Vision International, Build Africa Uganda, Save the Children International, ChildFund International, SightSavers, Plan International, the Belgium Technical Cooperation Agency, USAID GH EvaLS, AMREF Health Africa, and War Child Holland and United Kingdom. Mr. Waswaga is currently finalizing a Master of Philosophy in Programme Evaluation (dissertation stage) at University of Cape Town, South Africa. He holds Masters of Arts in Economic Policy Management, Bachelors of Arts Degree (Economics/Social Anthropology)-Second Class Upper Division, Certificate in Health Systems Research, July 2022 (Makerere University College of Health Sciences’ School of Public Health).