FINAL EVALUATION REPORT MIDTERM EVALUATION HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) ACTIVITY Ethiopia Performance Monitoring and Evaluation Service July 2017 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Abebe Alebachew, John Osika, Workie Mitiku, Esubalew Demissie, and Nigusu Aboset, hired by the Ethiopia Performance Monitoring Evaluation Service (EPMES), the USAID/Ethiopia Contractor. JOHN OSIKA FOR SOCIAL IMPACT FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) | i PERFORMANCE EVALUATION FOR HEALTH SECTOR FINANCING MIDTERM PERFORMANCE EVALUATION HEALTH SECTOR FINANCING REFORM/ HEALTH FINANCE AND GOVERNANCE (HSFR/HFG) ACTIVITY July 7, 2017 Activity Award # AID-OAA-A-12-00080 Evaluation Mechanism Number: #AID-663-C-16-00010 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. ii | HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT ABSTRACT Health financing remains one of the major challenges to increasing access and improving the quality of healthcare in Ethiopia. HSFR/HFG aims to address this challenge. This midterm evaluation used mixed methods to assess the activity’s performance based on two evaluation questions: a) To what extent are HSFR/HFG’s theory of change and objectives adequate, relevant, and viable? and b) What progress has been made towards achieving the activity’s performance objectives by focusing on relevance, effectiveness, efficiency, and sustainability? The activity design and interventions were found to be relevant, adequate, and viable in improving the quality and responsiveness of care by facilities, addressing community needs, and being fully aligned with government policies and strategies. The interventions effectively reduced financial barriers to healthcare, especially for women and the very poor, and in enhancing their voices to demand for accountability. Outpatient service utilization by community-based health insurance (CBHI) members and by the public surpassed the set targets. This was achieved with a declining share of out-of￾pocket health care spending by households and improved patient satisfaction rates. In most facilities assessed, lack of financing is no longer the prime cause of inadequate medical supplies. A total of 2.41 million households were enrolled in CBHI. About 1.8 million poor households benefited from increased protection. Moving forward, revenue retention and utilization at facilities, governance boards, fee waivers, private wings, and outsourcing of non-clinical services can be sustained without significant activity support. However, some design and implementation challenges pose risks for the sustainability of CBHI. The activity should focus on implementing a supply-side exit strategy that prioritizes building local institutional capacities and systems, particularly at the regional level. FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) | iii TABLE OF CONTENTS ABSTRACT.........................................................................................................................................................................II TABLE OF CONTENTS.................................................................................................................................................III ACKNOWLEDGMENTS ................................................................................................................................................V ACRONYMS..................................................................................................................................................................... VI EXECUTIVE SUMMARY..............................................................................................................................................VIII Evaluation Purpose..................................................................................................................................................... viii Background.................................................................................................................................................................. viii Evaluation Methods.................................................................................................................................................... viii Key Findings...................................................................................................................................................................ix Conclusions..................................................................................................................................................................xii Key Recommendations.............................................................................................................................................xiii I INTRODUCTION...................................................................................................................................................1 II PURPOSE AND SCOPE OF EVALUATION.....................................................................................................3 Evaluation Purpose........................................................................................................................................................3 Methodology ..................................................................................................................................................................3 Sampling...........................................................................................................................................................................5 Analysis............................................................................................................................................................................7 Limitations.......................................................................................................................................................................8 III FINDINGS..................................................................................................................................................................9 Evaluation Question 1..................................................................................................................................................9 Evaluation Question 2................................................................................................................................................10 IV CONCLUSIONS ....................................................................................................................................................28 Relevance ......................................................................................................................................................................28 Effectiveness.................................................................................................................................................................28 Efficiency........................................................................................................................................................................29 Sustainability .................................................................................................................................................................29 V RECOMMENDATIONS .......................................................................................................................................30 Recommendations for the Remaining Period of the Activity...........................................................................30 ANNEXES .........................................................................................................................................................................31 ANNEX A: EVALUATION STATEMENT OF WORK..........................................................................................32 ANNEX B: EVALUATION DESIGN MATRIX.........................................................................................................43 ANNEX C: DATA SOURCES......................................................................................................................................49 ANNEX D: LIST OF PEOPLE CONSULTED...........................................................................................................50 ANNEX E: DATA COLLECTION SCHEDULE.......................................................................................................61 ANNEX F: INTERVIEW PROTOCOLS AND GUIDES ........................................................................................64 ANNEX G: FGD GUIDE...............................................................................................................................................85 ANNEX H: PROFILES OF THE EVALUATION TEAM MEMBERS.....................................................................89 ANNEX I: REFERENCES ...............................................................................................................................................91 ANNEX J: HSFR EVALUATION SITES......................................................................................................................93 iv | HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT LIST OF TABLES Table 1: HSFR/HFG Impact and Outcome Targets...................................................................................................2 Table 2: Number of Sample Regions, Woreda, and Facilities.................................................................................6 Table 3: Internal Revenue as Share of Recurrent Budget to Health Facilities..................................................15 Table 4: Percentage of Health Facilities Managed with Boards............................................................................23 Table 5: Examples of Outsourcing of Non-Clinical Services in Some Facilities................................................26 LIST OF FIGURES Figure 1: Map showing the regions, woredas, and facilities sampled for the midterm evaluation .................7 Figure 2: Consolidated TOC based on project description and work plans....................................................10 Figure 3.1: Achievements of outcome targets contributed to by the activity interventions.........................12 Figure 3.2: Achievements of outcome targets contributed to by the activity interventions.........................13 Figure 4: Achievements in patient satisfaction..........................................................................................................13 Figure 5: Average annual revenue retention per HC (USD).................................................................................15 Figure 6: Average annual revenue retention per hospital (USD).........................................................................15 Figure 7: Utilization of retained revenue in HCs.....................................................................................................17 Figure 8: Utilization of retained revenue in hospitals .............................................................................................17 Figure 9: Number of functioning schemes and enrollment rates.........................................................................21 FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) | v ACKNOWLEDGMENTS The evaluation team would like to acknowledge the support and inputs from several colleagues from Social Impact, USAID, and the HSFR/HFG Activity during the evaluation process. The evaluation team would like to thank USAID/Ethiopia for its guidance throughout the evaluation. In particular, the team is thankful to Mr. Awoke Tilahun and Mr. Eshete Yilma from USAID/Ethiopia for their valuable comments on the evaluation design, tools, and report preparation, all of which improved the quality of this evaluation report. Our thanks also go to Kerry Bruce, Executive Vice President of Social Impact (SI), and Francis O. Okello, Chief of Party for the Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity, for their leadership and guidance of the evaluation team during the design of this evaluation and the preparation of the report. We also acknowledge Worku Ambelu and Dereje Getahun from the EPMES Activity for their invaluable inputs during all stages of this evaluation. Dereje Getahun also participated in data collection. We are grateful to SI Headquarters staff, Tasneem Nahar, Deputy Director, Program Management Unit, and Mike Pressl, Senior Program Assistant, Program Management Unit, for their administrative support and formatting of this evaluation report. The evaluation team would like to thank Luelseged Ageze, Chief of Party for the Health Sector Finance Reform/Health Finance and Governance (HSFR/HFG) Activity, and his staff, Zelelem Abebe and Tiliku Yeshanew, for sharing secondary data and coordinating between the evaluation team and the HSFR/HFG regional staff during field data collection. Without their support, data collection would not have been successful. The evaluation team would also like to express their gratitude to HSFR/HFG Activity staff at the national and regional levels for their support in making vital appointments with key informants at the federal, regional, woreda, and community levels. Finally, we are very grateful to all the key informants and focus group discussion participants at the federal, regional, woreda, facility, and community levels for their open and honest views on the performance of the activity. All errors remain the responsibility of the authors of this evaluation report. Abebe Alebachew (Team Leader) John Osika Workie Mitiku Esubalew Demissie Nigusu Aboset vi | HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT ACRONYMS ANC Antenatal care BOFED Bureau of Finance and Economic Development CASH Clean and Safe for Health CBHI Community-based health insurance CHAI Clinton Health Access Initiative CVC Central venous catheter DFID Department for International Development DP Development partner DRS Developing regional state EDHS Ethiopia Demographic and Health Survey EEA Ethiopian Economic Association EHIA Ethiopian Health Insurance Agency EPMES Ethiopia Performance Monitoring and Evaluation Service FGB Facility Governance Board FGD Focus group discussion FMOH Federal Ministry of Health GoE Government of Ethiopia HC Health center HCF Health care financing HDA Health Development Army HF Health facility HFG Health finance and governance HMIS Health Management Information Systems HR Human resources HSDP Health Sector Development Plan HSFR Health Sector Financing Reform HSSSD Health System Special Support Directorate HSTP Health Sector Transformation Plan IP Implementing partner KII Key informant interview LOE Level of Effort M&E Monitoring and evaluation MoFED/C Ministry of Finance and Economic Development/Cooperation MSD Medical Service Directorate NHA National Health Accounts OECD Organisation for Economic Co-operation and Development OOP Out-of-pocket PD Project description PEPFAR President’s Emergency Plan for AIDS Relief PHC Primary health care PNC Prenatal care FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) | vii PW Private wing RHB Regional Health Bureau RMD Resource Mobilization Directorate RR Revenue retention RRU Revenue retention and utilization SI Social Impact, Inc. SNNPR Southern Nations Nationalities and Peoples’ Region SHI Social health insurance SOW Statement of work TA Technical assistance TOC Theory of change UHC Universal health coverage USAID United States Agency for International Development WHO World Health Organization WOFED Woreda Office of Finance and Economic Development viii | HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT EXECUTIVE SUMMARY EVALUATION PURPOSE The main purpose of this midterm performance evaluation was to assess the continued viability of the Health Sector Financing Reform/Health Finance and Governance (HSFR/HFG) Activity’s design and its progress made towards achieving results. It was also expected to document best practices and key challenges faced by the activity. The evaluation addressed two questions: 1. To what extent are the HSFR/HFG theory of change and objectives adequate, relevant, and viable? and, 2. What progress has been made towards achieving the activity’s performance objectives by focusing on relevance, effectiveness, efficiency, and sustainability? The statement of work (SOW) noted that all intermediate results (IRs) of the activities, except for IR 4 (improved program learning), were the focus of this exercise. The findings and recommendations of the evaluation are expected to provide input (i) towards improving the implementation of the activity’s interventions for its remaining life; (ii) to chart out medium- and long-term recommendations to inform the development of future support; and (iii) to strengthen health financing reform in Ethiopia. BACKGROUND The United States Agency for International Development (USAID) has been supporting the Government of Ethiopia (GoE) through the Federal Ministry of Health (FMOH) to implement health financing reforms since the late 1990s. The current activity, Health Sector Financing Reform/Health Finance and Governance (HSFR/HFG), runs from August 2013 to September 2018. The period of the focus for the evaluation spans from August 1, 2013 to June 30, 2016 to account for Ethiopian fiscal year. The overall objective of the activity is to increase access to and utilization of health services through improved quality of care (supply￾side) and reduced financial barriers (demand-side). The specific objectives of the interventions are to: (i) improve the quality of health services; (ii) improve access to health services; (iii) improve governance of health insurance and health services; and (iv) improve program learning. USAID commissioned the HSFR/HFG Activity midterm evaluation to the mission-wide M&E Contract, Ethiopia Performance Monitoring and Evaluation Service (EPMES). EVALUATION METHODS The evaluation used mixed methods, including document review, key informant interviews, focus group discussions, facility-level direct observations, and secondary data analysis. The main data sources for national-level achievements were secondary sources. The information from secondary sources was analyzed in conjunction with the findings from primary data collected during field visits to provide evidence on best practices and challenges. The HSFR/HFG intermediate results were examined against criteria of relevance, effectiveness, efficiency, and sustainability, and with respect to cross-cutting issues of capacity development. Data were collected at the federal level and in five regions, 14 woredas, 24 health facilities, and 8 community-based health insurance (CBHI) schemes. FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) | ix KEY FINDINGS EVALUATION QUESTION 1 “To what extent are HSFR/HFG’s theory of change and objectives adequate, relevant, and viable?” While there was no theory of change (TOC) in the project description (PD), the evaluation team reviewed the activity documents available to them and extracted from them elements that constitute a theory of change. The HSFR/HFG design was found to be relevant, adequate, and viable. The major challenges outlined in the activity design document are relevant to removing the financial hurdles that facilities face and the financial barriers that poor communities face in accessing health facility care. The TOC included interventions that empower facilities to invest internally-generated resources in interventions that improve quality. The design was plausible, and there was no evidence of “theory failure” (wherein the activity was delivered effectively but outcomes were not achieved). All interventions have been implemented, and most of the assumptions in the work plans have been realized. However, emerging health care financing agendas, such as tax-based domestic resource mobilization for health (e.g., innovative financing), financing-exempted services, and enhancing efficiency did not materialize until after the design of the activity, and were thus not included in the activity design. EVALUATION QUESTION 2 “What progress has been made towards achieving the activity’s performance objectives by focusing on relevance, effectiveness, efficiency, and sustainability?” 2A: RELEVANCE The interventions of the HSFR/HFG activity were relevant to the needs and priorities of different beneficiaries, as demonstrated by several key findings. First, supply-side financing reforms enabled facilities to generate, retain, and use revenues, and have mitigated challenges related to shortages in operational budgets. Second, the establishment of CBHI schemes reduced financial barriers to health services and led to increased utilization. CBHI members consider insurance “a gift from God,” as it enables them to seek care immediately when they feel sick and enables women to visit health facilities without seeking money from their husbands. The activity provided better coverage and protection for the poor through local government financing. Third, community forums organized by boards were instrumental in enhancing community participation in improving governance and accountability. Woredas with CBHI confirmed that communities are exercising their rights to access acceptable quality health services, and facilities have become increasingly responsive and accountable to communities. Finally, the interventions are priorities in Ethiopia’s health financing strategy and align well with the priorities of the health sector transformation plan. 2B: EFFECTIVENESS The activity was effective in achieving its targeted outcomes at the national level. The activity surpassed its target for outpatient utilization of services by CBHI members, achieving 0.78 per capita visits per year against the target of 0.60 visits per capita per year. For the general population, outpatient visits increased from a baseline of 0.30 visits per capita per year to 0.56 visits per capita per year, which nearly met the x | HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT activity target of 0.60 visits per capita per year. The percentage of deliveries assisted by skilled health personnel increased from 23.1% in 2012/13 to 72.7% in 2015/16.1 While all this increase cannot be directly attributed to the activity, the activity contributed to this increase which occurred during the activity timeframe. According to the recent National Health Accounts (NHA) report, the percentage of household spending on health as a share of total health expenditures declined from 37% to 34% in 2015/16, despite the fact that CBHI has mobilized 2.4 million households to contribute to health spending.2 While inpatient satisfaction increased from a baseline of 80% to 87%, only a 1% increase in satisfaction was observed among outpatient clients (87% at baseline compared with 88%). Patients bypassing lower-level facilities due to shortages of pharmaceutical products declined from a baseline of 56% to 33% in outpatient services and from 33% to 29% in inpatient services, which signals an improvement in the quality of health services at both the outpatient and inpatient levels. The activity also achieved the following outputs: • The number and percentage of health centers (HCs) and hospitals retaining and utilizing internally generated revenues has increased from a baseline of 2,184 (68%) and 103 (79%) to 3,244 (93%) and 225 (92%), respectively. The average amount of retained revenue for HCs and hospitals is now estimated to be USD 15,843 and USD 133,908 per year, respectively. Internally generated revenue accounted for, on average, 31% of total financial resources at health centers (varying from 18% to 41%). In hospitals, internally generated revenue accounted for, on average, 50% of total revenue (varying from 21% to 79%).3 Most of these resources were spent on ensuring the availability of medicines and medical supplies, which accounted for 73% and 52% of spending in HCs and hospitals, respectively. Most facility-level respondents reported that lack of financing is no longer the major cause of inadequate supply of medicines and medical supplies. • The provision of free delivery at hospitals and health centers, together with effective demand￾generation mechanisms, has contributed to increased skilled delivery in health facilities. This was made possible in part by the ability of health facilities to procure medicines and medical supplies using internally generated revenue. It is estimated that delivery alone accounted for about 21% of expenditures of retained revenue generated, with health centers and hospitals, on average, paying as much as ETB 75,000 [USD 3,260] and ETB 635,000 [USD 27,608] per year, respectively, to cover the costs of these supplies. Facilities are stepping in with internally generated revenue to pay for the costs of delivery services which the government or development partners are currently financing for other exempted services. • Nationally, 236 woreda CBHI schemes have been established in six regions, enabling the enrollment of 2.41 million households (37% of eligible households) by June 2016. This indicator surpassed the activity’s target of establishing 198 schemes, but fell below the activity’s household target of enrolling 50% of households. Of the 236 CBHI schemes, 204 had started provision of services to their members. The distribution of established schemes (77 in Amhara, 68 in Oromia, 41 in SNNPR, and 18 in Tigray) and enrolled households varied among regions. For instance, Amhara accounted for 50%, Oromia for 16.4%, SNNPR for 21.5 %, and Tigray for 12.1% of the total national household enrollments. Enrollment rates also varied widely among 1 FMOH, 2016, Annual Performance Report 2008EFY (2015/16). 2 FMOH, 2017, Revised health care financing strategy, quoting the draft NHA VI Report 3 Abebe Alebachew, Yasmin Yusuf, Carolyn Mann, Peter Berman. 2014, Ethiopia’s Progress in Health Financing and the Contribution of the 1998 Health Care and Financing Strategy in Ethiopia, Tracking and Management Project. FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) | xi schemes, ranging from 5% household enrollment in Kercha, Oromia to universal coverage in Dembecha, Amhara. CBHI schemes contributed to the mobilization of ETB 672.5 million (USD 29.24 million) over the last three years. The major reason for variation in enrollment among and within regions is the difference in ownership and commitment of regions and woreda cabinets to CBHI activities. • Regional and woreda governments continued to finance the protection of the very poor through fee-waiver and CBHI indigent4 support mechanisms. A total of 276,444 households were covered through CBHI indigents financing, and 1.5 million households benefited from fee waivers. • The activity exceeded its targets in establishing facility governance boards at the hospital and health center levels by five and two percentage points, respectively. The activity established boards at 92% of HCs (target 90%) and 95% (target 90%) of hospitals throughout the country. • FMOH, regions, and woredas confirmed that the technical assistance (TA) provided was effective and responsive. The activity staff were widely viewed by stakeholders as “government” staff. The activity’s role in regions such as Afar, and its work to accelerate the CBHI scale-up strategy, was especially commended. There was evidence of skill transfer during the NHA VI development and its institutionalization into government structures. The evaluation identified the following major gaps and issues around effectiveness: • Isolated instances of using internally generated resources for unintended purposes, including use for unauthorized items such as the procurement of bonds and investing revenue retention and utilization (RRU) on long-term training; • Budget offsets in some woredas, where the woreda administrator and finance offices were not closely involved in health care financing initiatives; weak capacity of the procurement and finance personnel in health facilities; and lack of capacity to regularly undertake facility audits by woreda finance offices; • CBHI schemes’ effectiveness had challenges, including varying degrees of ownership by regions and woredas; variation in institutional arrangements; inadequate numbers and types of scheme staff, combined with high levels of turnover; lack of kebele-level structures and staff; lack of periodic auditing; forcing CBHI members to follow referral procedures that are not required for non-CBHI members; and the challenges of availability of good service (diagnosis, medicines, attitude of staff) in many of the health centers visited; • The rollout of a social health insurance scheme that covers formal-sector employees was postponed three times by the government, adversely affecting the realization of the coverage rate of 20% of the population with health insurance. • Governance challenges included high turnover of governance board members and lack of systematic training for new members; varying levels of competency of the governance boards; and, in some regions, individuals who were too occupied with other duties to perform their governing board duties. • Despite the continuous provision of technical assistance, there was heavy reliance on the activity to lead and implement the reforms. The capacities of the Regional Health Bureaus (RHBs) to lead and manage the health care financing reforms remain weak due to a lack of well-defined 4 Household without land, houses or any valuable assets xii | HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT structure. Further, most of the training was conducted by activity staff rather than regional training institutions and universities. 2C: EFFICIENCY The HSFR/HFR activity is delivering some of its outputs at a reasonable cost. Unlike other development support activities, where resources are used to finance interventions, this support helps the sector to generate its own resources to finance quality care. Health facilities mobilized USD 25 through RRUs with an average cost of USD 1 for health care financing trainings, which is a high return on investment by any standard. During the first year of the activity, the average cost of health care financing training in supply￾side interventions decreased from USD 13 to USD 3 per $100 collected retained revenue in 2016. The cost of enrolling an informal-sector household into the community-based health insurance scheme was ETB 8.65 [USD 0.37]. The average cost for establishing a CBHI scheme ranges from USD 1,899.70 to USD 3,750.26, depending on the types of capacity-building equipment provided. 2D: SUSTAINABILITY This evaluation documented solid evidence that the supply-side interventions (RRU, boards, fee waivers, private wings, outsourcing, etc.) can be sustained without significant activity support in the consolidating regions. The guidelines and manuals developed are the most important instruments for sustaining gains made. Regions and woredas have begun budgeting for health care financing reform implementation and refresher trainings, and have expressed willingness and demonstrated the ability to take ownership during the current activity period, provided proper exit strategies enable a smooth handover. There are several design and implementation challenges that threaten the sustainability of CBHI. For example, there is a lack of clarity on CBHI institutional arrangements—the role of the Ethiopian Health Insurance Agency (EHIA) and which entity (woreda administration or health office) should take the lead on CBHI implementation. The absence of kebele-level structures and employees results in a campaign￾based approach of regularly renewing membership. Moreover, some schemes cannot cover their costs, and there is no high-level risk-sharing pool. The reduction of the general subsidy by the government from 25% to 10% also presents a challenge to sustainability. CONCLUSIONS Based on the evidence generated, the HSFR/HFG activity design and interventions implemented over the last three years were found to be relevant in terms of providing high-quality and responsive care to the communities and being fully aligned with government policies and strategies. The interventions not only reduced financial barriers to the communities, including to the very poor, but also enhanced voices to demand accountability from service providers. The activity’s theory of change was relevant to, and adequate for, helping Ethiopia achieve its health care financing targets. At the midterm, the activity is on track to achieve its strategic objectives in terms of improving quality of care, reducing financial risks, and improving accountability and responsiveness of health providers. There was solid evidence to suggest that supply-side health care financing reforms can sustain themselves in the short-term through the activity’s systematic exit strategy. FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) | xiii KEY RECOMMENDATIONS RECOMMENDATIONS FOR THE REMAINING PERIOD OF THE ACTIVITY The activity, in its remaining period, should focus on ensuring that it has put the proper capacities and systems in place before exiting from the supply-side reforms (e.g., revising guidelines and conducting a training of trainers for regional and woreda administrators and local training institutions). The focus of the activity during the remaining period in demand-side reforms should be: a) laying the groundwork for regional risk pooling for CBHI; b) supporting the Government in the development of career paths for scheme staff to ensure sustainability in staffing; c) facilitation audits of the schemes to ensure financial sustainability; and d) building the capacity of the RHB/EHIA to ensure the growth and sustainability of the program. 1 | FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) I INTRODUCTION The Government of Ethiopia endorsed the Health Care Financing Strategy in 1998 and implemented it thereafter. According to the Ethiopia’s Progress in Health Financing and the Contribution of the 1998 Health Care and Financing Strategy in Ethiopia (2014) and the revised draft health financing strategy 2015– 2025 (2017), Ethiopia has made significant gains in this sector. The most recent documented health care per capita expenditure for 2013/2014 was USD 28.40, compared to USD 7.10 per capita in 2000. 5 Furthermore, the government’s general revenue contribution to the health budget increased by 238% from 1998 to 2013/14.6 Out-of-pocket (OOP) spending has been decreasing as a share of the total health expenditures (from 53% in 2000 to 34% in 2013/14). 7 Additionally, there has been a dramatic scale-up in external financing for the health sector, which made up 50% of total health expenditures in 2010/11 but has reduced to 34% in the recent 2013/14 estimates. 8 Other initiatives of the health financing strategy include “fee-exempt” services, which minimize the disease burden on households for key health services, and the “fee-waiver” program, which benefits the poor. USAID/Ethiopia has been supporting the Government of Ethiopia on health sector financing reform since 1998 and continues to contribute significantly to such efforts. This is exemplified by the current work of the Health Sector Financing Reform/Health Finance and Governance (HSFR/HFG) activity. HSFR/HFG is a five-year, $42 million USAID-financed activity awarded to and implemented by Abt Associates with award # AID-OAA-A-12-00080. The activity supports Ethiopia’s government at the federal, regional, woreda, and health facility levels. Having started its operations on August 1, 2013, the activity supports each of the nine regions and two city administrations in the country, and it is scheduled to end on July 31, 2018. The overall objective of the activity is to increase access to and utilization of health services through improved quality of care and reduced financial barriers. The specific objectives of the activity are to: (i) improve quality of health services; (ii) improve access to health services; (iii) improve governance of health insurance and health services; and (iv) improve program learning. The activity is expected to achieve five outcome targets, shown in Table 1. USAID commissioned this midterm performance evaluation to EPMES to assess project performance and document key successes, good practices, gaps, and constraints in implementing the HSFR/HFG intermediate results. A team of five consultants from Social Impact, Inc. (SI) conducted this evaluation from March 2017 to June 2017. The report is organized into five sections. Section I provides an overview of the background and context. Section II presents the purpose and scope of the evaluation, as well as the methodology employed in undertaking this evaluation. Section III describes the findings—achievements, success factors, and challenges—of each of the intermediate results of the activity for each evaluation question. Section IV presents the overall conclusions of the report. The recommendations for the remainder of the activity and the design of the next activity are presented in Section V. The annexes to this report comprise the Evaluation Statement of Work (Annex A), Evaluation Design Matrix (Annex B), Sources Reviewed (Annex C), List of People Consulted (Annex D), Data Collection 5 FMOH, 2017, Draft Revised Health Care Financing 2015–2025. 6 Abebe et al. 2014. 7 FMOH, 2017, Draft Revised Health Care Financing 2015–2025. 8 FMOH, 2017, Draft Revised Health Care Financing Strategy, Addis Ababa. HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT | 2 Schedule (Annex E), Interview Protocols and Guides (Annex F), FGD Guide (Annex G), Profiles of the Evaluation Team Members (Annex H), References (Annex I), and HSFR Evaluation Sites (Annex J). NO INDICATOR BASELINE DATE OF BASELINE TARGET 2018 1 Increase health service utilization in 185 CBHI districts/woredas (13 current pilot and 172 second phase pilot woredas) 0.3 June 2013 0.6 2 Health facilities managed with boards where communities are represented No data June 2013 90% 3 Public health facilities retaining and using their revenue HCs: 68% Hospitals: 79% June 2013 90% 4 Share of out-of-pocket (OOP) expenditures to total health budget reduced 37% June 2013 30% 5 Proportion of people enrolled in health insurance increased 1% June 2013 20% Source: HSFR/HFG Activity Description and the SOW of the evaluation TABLE 1. HSFR/HFG IMPACT AND OUTCOME TARGETS 3 | FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) II PURPOSE AND SCOPE OF EVALUATION EVALUATION PURPOSE The main purpose of the midterm performance evaluation was to assess the continued viability of the HSFR/HFG activity’s design and the progress that the activity has made towards achieving expected results (outputs and outcomes). The period of performance evaluated for this midterm evaluation was from August 01, 2013 to June 30, 2016 to align with the completed Ethiopian fiscal year; the evaluation focused on intermediate results (IRs), excluding IR 4 (“improved program learning”). The evaluation was expected to document best practices as well as implementation successes and challenges. The evaluation was also expected to provide input towards (i) improving the implementation of the HSFR/HFG activity’s interventions for the remaining life of the activity; (ii) charting out medium- and long-term recommendations to inform the development of future support; and (iii) strengthening health financing reform in Ethiopia. EVALUATION QUESTIONS: This evaluation sought to answer two questions: 1. To what extent are the HSFR/HFG theory of change and objectives adequate, relevant, and viable? 2. What progress has been made towards achieving the activity’s performance objectives by focusing on: • Relevance? 9 • Effectiveness? 10 • Efficiency? 11 and • Sustainability? 12 In addressing these questions, the evaluation report documented the challenges faced in implementing the activity and identifies both short-term (within the remaining period of the activity) and long-term (informing the potential follow-on activity design and implementation) priority action-oriented recommendations. METHODOLOGY A five-member evaluation team, assisted by one EPMES staff, conducted fieldwork from March to June 2017. The team utilized a mixed-methods approach, involving (i) a desk review of available secondary documents and data; (ii) structured key informant interviews at federal, regional, woreda, and facility levels (iii) focus group discussions with communities, CBHI, and facility boards; and (iv) site visits to federal, regional, and district hospitals and health centers. The main methods used are the following: 9 Relevance: the extent to which the HSFR/HFG interventions are suited to the priorities and policies of the communities and facilities as well as the Ethiopian government’s health financing priorities and its capacity-building needs; 10 Effectiveness: the extent to which HSFR/HFG intervention and capacity building attain their stated objectives and results. 11 Efficiency: how well are the HSFR/HFG project converting inputs into outputs (considering both the quality and quantity of these outputs). 12 Sustainability: measuring whether the benefits of HSFR/HFG project are likely to continue with ownership of government even after project funding has been withdrawn. HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT | 4 • A review of documents, including government health financing strategies and national and regional achievements. Health financing–related assessments, strategies, and guidelines were also reviewed. The team reviewed activity implementation reports, supportive supervision and monitoring visit reports, best practice documents, and reports to provide evidence on best practices and possible gaps. The national and regional HSFR/HFG data set was reviewed, verified, and used for the main source of quantitative data analysis to generate evidence on the identified results of the program. The facility and CBHI scheme monthly and quarterly reports were also reviewed to understand activity successes, gaps, and challenges. • Key informant interviews (KIIs) using semi-structured questionnaires were conducted with key policymakers; development and implementing partners; federal, regional, and woreda administrators; and health facility managers to explore their perceptions on the relevance, effectiveness, efficiency, and sustainability of the HSFR/HFG activity and to understand the successes and the challenges of implementing the intermediate results. A total of 99 key KIIs were carried out at the federal level and in the sample visits. (See Annex D for persons consulted and interview tools.) • Focus group discussions (FGDs) at the community level. FGDs were used to identify the perceptions of the beneficiaries, the health facility, and CBHI governance boards on the performance, relevance, and effectiveness of the activity support to meet their needs and expectations. FGDs helped generate recommendations for improving HSFR/HFG implementation and the design of any follow-up activity. A total of 31 FDGs with CBHI and facility boards as well as communities were carried out. (See Annex D for a list of communities visited.) FGD in Hanto Kebele community, Gedeb Asasa woreda, Oromia region, April 2017 • Health facility direct observation. A health facility checklist was used to assess the value of supply-side interventions (any facility improvement due to investment made by RRU) on health centers and district and regional hospitals. The assessment was carried out in the facilities within the referral chain. The evaluation team reviewed the availability of the asset and financial management guidelines and manuals at the facility to implement the reform. This helped the 5 | FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) evaluation team to understand how far the health facilities were capacitated/trained to undertake the activities in the future per the guidelines without the activity support. SAMPLING Regions were selected based on two categories of health care financing (HCF) reform implementation: (1) consolidation regions (regions that started health care financing reforms earlier in Ethiopia) and (2) expansion regions (regions that joined later as implementers of the reforms). Among the consolidation regions, the team chose Amhara and Oromia, which represent the two most populous regions in the country for visits at regional, woreda, and facility levels, and SNNPR (for CBHI implementation variations) only for visits at the woreda and facility levels. The team selected Afar from Developing Regional States (DRSs), as it represents expansion regions and pastoralist communities. Addis Ababa city administration was selected to represent a unique reform implementation context of a large urban environment. The selection of woredas in the five regions was facilitated by the regional staff of HSFR/HFG and the regional health bureaus. Health facilities were classified based on their performance in the implementation the health care financing reform. Consequently, the CBHI schemes of each region evaluated (Tigray, Amhara, Oromia, and SNNPR) were categorized into three groups: high performers (Group 1); medium performers (Group 2); and low performers (Group 3) based on their performance. Those districts (woredas) with more than 50% household enrollment were classified as high performers, those with 30% to 50% household enrollment were classified as medium performers, and those with less than 30% household enrollment were classified as low performers. Woredas were sampled by selecting “high￾performing”13 as well as “low-performing.” 14 Accordingly, in consolidation regions (Amhara and Oromia), and within their respective zones identified above, four woredas were selected in each region (of these, three were woredas implementing CBHI, while one was a woreda that is not implementing CBHI). Each of the four selected woredas fall into one of the following categories: • One woreda with the best health center in health care financing reform implementation and the best CBHI performance; • One woreda with the best health center in health care financing reform implementation and weak CBHI performance; • One woreda with a poor performing health center in health care financing reform implementation and weak CBHI performance; • One woreda with a poor performing health center in health care financing reform implementation and that is not implementing CBHI. In the non-consolidation regions (Afar and Addis Ababa), only two woredas in each region were selected because they are not implementing CBHI. These two types of woreda in non-consolidation regions were as follows: • One woreda with the best health center in health care financing reform implementation; and • One woreda with a weak health center in health care financing reform implementation. 13 A “high-functioning” woreda in implementing a first-generation reform is one that has at least one health center that has demonstrated strong performance in implementing HCF reform implementation such as RRU, health facility governance, etc. A strong woreda using second-generation reforms is one that has higher performance of CBHI coverage rates. 14 A “not-high functioning” woreda is one that has a health center that does not perform well in first-generation reforms and/or that has either weak performance or has not started implementing CBHI schemes. HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT | 6 Fieldwork took place in five regions (Afar, Amhara, Oromia, SNNPR, and Addis Ababa), 14 woredas, and 24 health facilities from April 2–28, 2017, as presented in Table 2, Figure 1, and Annex E. (For more detail, see Annex E: Data Collection Schedule.) TYPE OF SAMPLED ENTITIES AFAR AMHARA OROMIA SNNPR ADDIS ABABA TOTAL Woredas with CBHI 0 3 3 2 0 8 Woredas without CBHI 2 1 1 0 2 6 Health centers 2 4 4 2 2 14 District hospitals 1 2 2 0 0 5 Regional hospitals 1 1 1 0 1 4 Federal hospital 0 0 0 0 1 1 TABLE 2. NUMBER OF SAMPLE REGIONS, WOREDA AND FACILITIES 7 | FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) Figure 1. Map showing the regions, woredas, and facilities sampled for the midterm evaluation ANALYSIS The team used three approaches to analyze the data. First, a review of the secondary data obtained from the HSFR/HFG activity, the Federal Ministry of Health (FMOH), and other studies explored trends, percentages, shares, unit costs, etc. to understand change in key indicators over time. Analysis of secondary data such as Ethiopia Demographic and Health Survey (EDHS) and NHA household surveys were also carried out. Given the national scope of the support activity, the analysis of the secondary information was the main source of the evidence for making judgments on performance in terms of meeting set targets for effectiveness. Some evidence from other research in areas outside the sample regions was also reviewed and included as success/and or challenges in the report. Second, the team carried out a rolling analysis of the qualitative data generated from the field. At the end of each day of fieldwork, the team members met and reviewed the field notes. The team discussed and identified the findings and emerging issues and categorized them into a findings, conclusions, and recommendations matrix that was developed on an ongoing basis during fieldwork. The findings of the field visits were largely used to generate evidence on what works, what doesn’t work, and what needs to be done in the future, but they did not generate data on the effectiveness of performance. At the conclusion of the fieldwork, the team produced the preliminary findings, conclusions, and recommendations for all the key evaluation questions that were prepared in the form of a preliminary findings matrix and PowerPoint presentation. HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT | 8 LIMITATIONS This evaluation depended on secondary data from the activity and did not generate primary quantitative data. The team collected evidence from the activity implementation documents of the implementing partner and the FMOH. Other quantitative data came from secondary sources. Given the increasing number of facilities implementing the supply-side (3,244 health centers and 225 hospitals) and demand￾side (236 woredas) health financing reforms and time constraints related to data collection, the evaluation covered only limited areas where the activity is operating. Although three of the four consolidated regions as well as one pastoral and one urban region are included in the sample, the number of woredas in each region was limited to four in the consolidated regions and two in the other regions. As a result, best practices and challenges presented in this report may not fully represent the entire country. The health facility board members in Afar were all in training and were not available for FDGs with the evaluation team. Efficiency analysis is complex, requiring separate work of its own, and the analysis in this evaluation is limited in its ability to show illustrative case material. 9 | FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) III FINDINGS EVALUATION QUESTION 1 “To what extent are HSFR/HFG’s theory of change and objectives adequate, relevant, and viable?” A theory of change (TOC) is a tool to help describe an activity’s pathway from the need that it is trying to address to the results that it wants to achieve (outcomes). The TOC outlines the interventions (project activities) that the activity intends to implement to achieve those results. It also articulates the assumptions that lie behind the activity’s reasoning along the pathway and addresses the question of why project activities will lead to the results that are targeted. It is often represented in a diagram or chart that shows the pathway from the need (challenge) to the results targeted by the activity. The activity has a two-year project description (PD) document which clearly outlines the health care financing challenges, the interventions to be implemented to address these challenges, and the expected results during this timeframe—from outputs to outcomes. After the two-year initial implementation period covered by the HSFR PD, the HSFR activity became part of the global HFG activity, and the subsequent implementation was guided by the annual work plans and the overall global HFG PD. All the elements of the TOC chart can be traced in the PD and work plans. The evaluation team has consolidated and abstracted the available elements from the PD and work plans into the TOC chart shown below (see Figure 2). This process was undertaken partially to help the team approximate a TOC based on existing documents but also to help the team systematically examine and better understand the key elements of the HSFR activity. While the evaluation team developed the TOC chart, the elements therein are abstractions from the activity documents. The evidence generated during this evaluation, and views of all stakeholders, confirm that activity interventions (as conceptualized in the PD and the implicit TOC) were: • Relevant. The major HCF challenges that were outlined in the TOC in Figure 2 had the funding and technical assistance that were relevant to address those challenges. The funding and core priorities of the activity interventions helped to remove the financial challenges that facilities face; to improve access to health facilities by the communities, including the very poor; and to empower the facilities to allocate their resources and invest it on quality-improving interventions. • Adequate. The review of the activity design and implementation clearly showed that the activity achieved almost all deliverables. There was no evidence of “theory failure” (wherein the activity was delivered effectively but outcomes were not achieved). Despite the lack of the chart, the TOC, as outlined in the PD, was viable and adequate for getting increased quality, access, and accountability. • Viable. Most of the interventions have been implemented on the ground. There was no evidence of “implementation failure” (delivery failure) that affected the realization of results (outcomes and outputs). Hence, the design was viable within the Ethiopian context. • Assumptions and risks. Most of the assumptions of the work plans, with the exception of the launching of social health insurance (SHI), were realized. Both the supply- and demand-side reforms are owned and driven by the federal and regional governments. The EHIA managed to establish 24 branches and recruited more than 700 staff to manage the insurance schemes. HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT | 10 Figure 2. Consolidated TOC based on project description and work plans The only gap observed was that some of the recently emerging health care financing agendas are not included in the TOC interventions, such as tax-based domestic resource mobilization for health, including innovative financing, and sustainable financing of exempted services, as well as enhancing efficiency in use of resources. EVALUATION QUESTION 2 “What progress has been made towards achieving the activity’s performance objectives by focusing on relevance, effectiveness, efficiency, and sustainability?” 2A: RELEVANCE This performance evaluation used the OECD definition of relevance to assess the extent to which the HSFR/HFG interventions were suited to the priorities and needs of facilities to provide quality care as well as the priorities of communities. The evaluation also assesses the extent to which interventions fit well with the policies and strategies of Government of Ethiopia (GoE) health financing priorities, as well as the capacity-building needs of different government counterparts at federal and regional levels. The supply-side financing reforms of the activity not only enabled facilities to generate, retain, and use revenue to improve quality of care as per their priorities, but also improved facilities’ ownership of control HCF & Governance Challenges Poor quality of care due to budget shortages High health worker attrition rate in hospitals Administrative burden + inefficiency of non-clinical services in hospitals Centralized and delayed decision-making: service, resource, & client relations Health facilities not accountable or responsive to local context and clients Lack of governance knowledge and weak capacity to govern CBHI schemes OOPs can be catastrophic and prohibitive Inequity in health service utilization at service delivery point Training on HCF & PFM & planning and budgeting Adapt guidelines on Private wing and outsourcing Training for CBHI executives, woredas, facilities, journalists, & facility staff Develop directives and scale up plans Training (CBHI executives, boards, woreda, kebeles and facilities) Facilitate periodic review TA for HCF and mentoring TA for EHIA and its branches on CBHI implementation and fee waivers Awareness creation through mass media and community mobilization Facilitate and monitor the functioning of these boards Provide TA (governance & EHIA Branches) INPUTS Funding TA for FMOH/Regions/EHIA ACTIVITIES OUTPUTS Increased # of health facilities retaining revenue and the amount they retained & used for priority services and commodities Increased # of established private wings & outsourced non￾clinical services Increased CBHI geographic & population coverages Increased protection of the poor through targeted subsidies Functional facility governing boards in place Enhanced health insurance leadership and governance Improving quality of services Increasing access to health services Improving of governance of insurance and health services OUTCOME Increased demand & use of quality services by all, especially by insurance members and the poor IMPACT Improved health outcomes In Ethiopia 11 | FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) leakages (misappropriation of retained revenue). The generation and use of revenue was facilitated by the reduction of administrative and management complexities at the facilities through the establishment boards that fostered autonomy of decision making. FGDs with the communities indicated that community forums organized by the facility governance boards and the presence of a community representative as members of the facility governance boards were instrumental for community participation in the governance and increasing accountability of the facilities. Private wings in public hospitals improved the quality and timeliness of services for those who can afford to pay, especially on weekends and holidays, reduced the turnover of skilled manpower though additional compensation, and helped to motivate and retain staff members. These supply-side interventions were found to be relevant to ensuring that the operational budget was a minimal issue for the facilities, improving their ability to respond to community needs and provide alternative care. On the demand side, the expansion of exempted services15 and increasing uptake of the previously exempted services assisted the population in accessing health services. The establishment of CBHI schemes reduced financial barriers at the point of use. FGD discussions with the community characterized CBHI as a “gift from God,” as it enabled them to seek care immediately as soon as they feel sick (improved utilization), access care up to hospital levels, costing as much as ETB 30,000 [USD 1,304] per visit with little contribution (about ETB 200 [USD 9] per household annually). More specifically, CBHI enabled more women to visit health facilities without seeking money from their husbands, which empowered women to better control their health care. The activities also strengthened the protection of the poor by putting in place a process for the selection of the very poor and by advocating for increased financing of their cost/premiums through local government financing. Fee-waiver beneficiaries 16 were fully covered by woredas, and CBHI indigents17 were covered by the regional government (70%) and woreda (30%). Most importantly, CBHI has strengthened communities’ engagement with health facilities, woreda administration, and the health office to ensure responsive service delivery. Respondents reported that as they began to exercise their rights to access proper health services, facilities were forced to be more responsive and accountable. The training provided to CBHI stakeholders at the woreda and kebele levels were found to be relevant in creating awareness and facilitated the implementation of CBHI. The current benefit packages covered in CBHI schemes also addressed the needs of CBHI members, but quality of service was a challenge, as some of the services that are included in the benefit package are not being provided, and there is a lack of drugs and diagnostic facilities in health facilities. All these interventions were aligned with the priorities of Ethiopia’s health financing strategy, and are stated as priorities of the Health Sector Development Plan (HSDP) (2010/11 – 2014/15 and Health Sector Transformation Plan (HSTP) (2015/16 – 2019/20). They fit well with the government’s agenda for moving towards universal health coverage (UHC) and improving the quality of care. With the revision of the current financing strategy, however, new emerging priorities are not yet included in the activity interventions. These include evidence generation and strategic engagement for policy development on increased tax-based resource mobilization for health, including innovative financing, pushing progress towards UHC, and taking increasing responsibility of financing-exempted service commodities. 15 Exempted services are public health services that any Ethiopian should access free of charge regardless of income. This includes immunization of children; family planning; treatment for HIV/AIDS, malaria, TB, etc. 16 Fee waivers are implemented in non-CBHI woredas and their subsidy is budgeted and paid by the woreda finance office per the services they obtained from facilities. 17 CBHI indigents are the very poor in the woredas that have established CBHI and whose premiums are paid annually by the woreda and regional governments. HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT | 12 2B: EFFECTIVENESS The activity’s main expected outcomes are increased utilization of services (both total and members of CBHI), reduction of the share of OOP spending from total health spending, and an increased percentage of health facilities retaining and using their resources under the guidance and leadership of their boards. There is evidence that some of the targeted outcomes have been realized, in part due to the activity’s interventions. For instance, the target for increasing outpatient utilization rate was surpassed by the CBHI members (0.78 per capita visit per year, against the target of 0.60). 18 Although enrollment into insurance (CBHI) didn’t meet the target, there is 12 percent increase in enrollment since the baseline, which is a considerable improvement (See Figure 3.1.) The activity almost attained the target for per capita visits by the general population (0.56 per capita visit per year against the target of 0.60) 19 (See Figure 3.2). The share of household spending on health from total health expenditures declined from the baseline of 37% to 34 % in 2015/16, despite CBHI having mobilized 2.4 million households to contribute to health spending. 20 The recent NHA household survey documented that government health facilities, where the activity support is being implemented, provided 75% and 78% of the total outpatient and inpatient services covered in this survey but obtained about 50% of OOP payments (ETB 9.1 billion [USD 395,653,000] paid out by households in 2015/16. This demonstrated that government health providers remain financial affordable to the people as compared to the private sector. It also documented that only 6% of households sold household assets and another 4% borrowed from families to pay for health services. 21 18 HSRF/HFG data set 19 FMOH, 2017, NHA VI Household Survey Draft Report 20 FMOH, 2017, Revised Health Financing Strategy 2015–2025 21 FMOH, 2017, NHA VI Household Survey, 2015/16 Source: NHA V and VI; HSFR/HFG data base Figure 3.1. Achievements of outcome targets contributed to by the activity interventions 20% People enrolled into insurance (CBHI) 1% 13% Baseline 2018 Target Achievement 13 | FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) Figure 3.2. Achievements of outcome targets contributed to by the activity interventions Source: NHA V and VI; HSFR/HFG data base The evaluation team’s analysis of the NHA V and VI household surveys revealed that patient satisfaction has increased. Outpatient patient satisfaction increased marginally from a baseline of 87% to 88%, while inpatient patient satisfaction increased from a baseline of 80% to 87%. (See Figure 4.) There is evidence that patients’ bypassing of lower-level facilities due to a shortage of pharmaceutical products has declined from a baseline of 55.6% to 33% in outpatient services. Similarly, bypassing of lower￾level facilities by patients due to a shortage of pharmaceutical products among inpatients has also declined from 33% to 29%. The increased inpatient satisfaction and reduction of bypassing of lower-level facilities by patients due to a shortage of pharmaceutical products all point to an increase in quality of health services at both the outpatient and inpatient levels. Figure 4. Achievements in patient satisfaction Source: FMOH, NHA V and VI 0.30 0.37 0.60 0.60 0.56 0.78 0.00 0.20 0.40 0.60 0.80 1.00 Outpatient visit per capita per year (total population) Per capita service utilization rate (CBHI members) Baseline 2018 Target Achievement 87% 80% 56% 33% 88% 87% 33% 29% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Outpatient Service Satisfaction Rate Inpatient Service Satisfaction rate % of outpatients bypassing nearest facility % of inpatients bypassing nearest facility Baseline Status Current HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT | 14 INTERNAL REVENUE RETENTION AND UTILIZATION Evidence reviewed by the evaluation team showed that domestic resource mobilization through user fees, including retention and utilization of user fees, was effective, both in terms of the number of facilities in which it is implemented and in the magnitude of resources being generated and used for quality improvement at the facility level. The number and percent of HCs and hospitals retaining and utilizing internally generated revenues increased from a baseline of 2,184 (68%) and 103 (79%) to 3,244 (93%) and 225 (92%), respectively. The share of HCs’ internally generated revenue from the facility’s total budget has increased from a baseline of 14% to 20% (HSFR/HFG database). Detailed data collected from sample health centers showed that internally generated resources accounted for, on average, 31% of resources at health facilities (varying from 18% to 41%). In hospitals, internally generated resources accounted for, on average, 50% of resources (varying from 21% to 79%). 22 The share of RRU from the total operational budget of health facilities was between 60% and 80%, showing that the availability of operational budget at the point of service delivery has increased. The average amount of retained revenue for HCs and hospitals was found to be USD 15,843 and USD 133,908 per year (see Figures 5 and 6). Because many newly built hospitals have only recently begun accepting patients (increased by about 50% in terms of number of hospitals), the trend of average hospital revenue collection has reduced due to the lower performance of late joiners. 22 Abebe Alebachew et al., 2015. Box 1: Some income-generation approaches (outside user fees) applied in health facilities Health facilities are now trying to be creative to generate additional income through different mechanisms; the following are some of the examples documented: • Adama Hospital constructed buildings for shops and rented them out, which turn in a monthly rent income of about 43,000 birr [USD 1,870]. • Bishoftu hospital earned more than 800,000 birr [USD 34,782] per annum from renting out space in a hall. • Gendeberet hospital started fattening of cattle and selling them at higher price, earning a large profit. • Fee-waiver reimbursement from woredas as provided in HCF became one of the major sources of revenue to hospitals. Most hospitals have raised more than half a million birr from this source. • Assassa HC is engaged in providing cafeteria services. The HC constructed a cafeteria with the purpose of serving its customers and getting additional revenue for the facility. The initial construction and facility cost was about 400,000 birr [USD 17,391] but currently the cafeteria building and assets are valued at more than one million birr. The HC also carries out farming and sells its produce. Source: Abebe et. al 2014. 15 | FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) Figure 5. Average annual revenue retention per HC (USD) Figure 6. Average annual revenue retention per hospital (USD) A 2014 HCF study analyzed the internal revenue (user fee and other internal revenues) generated from 84 health facilities as a share of non-salary recurrent budget and documented that it ranges from a low of 43% to a maximum of 146% in hospitals and between 55% and 153% in health centers. Health facilities are raising revenue, which is significant compared to resources that they receive from the government budget (see Table 3). Health facilities have also started being creative about their income-generating approaches, in addition to the user fee retention (see Box 1). This has enabled health facilities to expand service scope and coverage as well as improve quality by investing internal resources where needed (see Box 2). TABLE 3. INTERNAL REVENUE AS SHARE OF RECURRENT BUDGET TO HEALTH FACILITIES AMHARA OROMIA SNNPR ADDIS ABABA User fee as % of recurrent budget HC 35% 17% 19% 26% Hospital 48% 19% 18% 52% Total revenue as % of recurrent budget HC 41% 18% 26% 39% Hospital 57% 21% 43% 79% Total revenue as % of non-salary recurrent budget HC 153% 55% 71% 204% Hospital 146% 43% 59% 194% Source: Abebe et al, 2014, page 67 INDICATORS FACILITY TYPE REGION Source: HSFR/HFG database Source: HSFR/HFG database HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT | 16 Facilities have the right to budget and use their retained revenue per the guidelines provided in the form of positive (e.g., medicines and supplies, medical equipment, HMIS, etc.) and negative23 lists (expenditure on any foreign trip and training, donations, hiring of permanent staff, etc.) and on activities that improve quality of care. Accordingly, in most of the facilities, the largest share of the internally generated revenue is spent on ensuring availability of medicines and medical supplies, which accounted for 73% and 52% for HCs and hospitals, respectively (see Figures 7 and 8). Most facility-level respondents claim that lack of financing is no longer the major cause of inadequate supply of medicines and medical supplies for non-exempted services. This is much more visible in Addis Ababa, where shortages of pharmaceuticals and laboratory reagents are comparatively less significant compared to hospitals and health centers located outside of Addis Ababa that the evaluation team visited. This is due to financing and greater availability of pharmacy outlets in Addis Ababa to procure from. 23 Positive lists are lists that indicate a list of goods and services that can be procured using retained revenue. Negative lists are lists of services and items that facilities are not allowed to procure using retained revenues. Box 2: Best practices in using RRU in SNNPR and Amhara Areka Health Center Areka health center is in Wolaita zone of SNNPR. In 2016, the health center collected USD 49,542 from internally generated revenue. The facility utilized 57.4% of the retained revenue for purchasing medicines and medical supplies. Other investments made possible through retained revenue included the following: • 2 tuk-tuks purchased for prevention activities • 4 IPD rooms renovated • 2 medical record & security rooms built • 1 Olympus microscope purchased • 1 centrifuge purchased • 1 photocopier purchased • 2 desktop computers with printers purchased Felege Hiwot Hospital, Amhara The regional referral is currently able to provide 87% of diagnostic tests and 93% of services expected of a regional hospital. It invested the resources as follows: • Medicines and medical supplies are 85% available at store and 95% at dispensary in the hospital. • Invested about USD 435,000 on various RRU resources on procuring diagnosis equipment, including microbiology sensitivity, central venous catheter (CVC) machine, chemistry machine, hormone analyzer, slit lump, laparoscopy, Lazier machine, and ultrasound. • Rehabilitated buildings used for private wings, laundry room, pediatric inpatient and outpatient services, and emergency services. • Currently building orthopedic service, internal medicine, and prenatal care (PNC) service providing buildings. • The major issues raised on improving RRU are the revision of user fees and an outdated list of positive and negative lists. 17 | FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) Figure 7. Utilization of retained revenue in HCs Figure 8. Utilization of retained revenue in hospitals 5% 7% 73% 3% 3% 3% 4% 2% Office supplies Printing Services Drugs & Medical Supplies Fuel & Lubricant Miscellaneous equipment Maintenance and Repair of Building and Furniture Contracted Services All Others 12% 11% 52% 5% 12% 4% 1% 3% Office supplies Printing Services Drugs & Medical Supplies Fuel & Lubricant Miscellaneous equipment Maintenance and Repair of Building and Furniture Contracted Services All Others HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT | 18 However, there are isolated instances of the use of resources for unintended purposes. A few facilities used the retained revenue in inappropriate ways, including the procurement of bonds and investing it in long-term training, etc., which are on the negative list and therefore the use of retained revenue is prohibited. However, all the KIIs at the facility, woreda, and regional levels confirmed that these positive and negative lists are outdated, as they were developed when resources were limited. Furthermore, the fees in some regions such Amhara have never been revised, and some of the service charges do not adequately cover costs of services. The activity has not institutionalized a systematic revision of user fees, and regions lack the necessary capacity to carry out cost-based revision. While the regulations and guidelines clearly indicated that the retained revenue should be additional to the government’s own operational budget allocation, there are mixed findings in different woredas. Some woredas report budget offsets (reduction due to increased mobilization of RRU), while most woredas ensure that RRU continues to be additional to the government budget. The evaluation team observed that when the woreda administrator and woreda finance offices are engaged and knowledgeable about the health care financing initiatives, budget offsetting is not an issue. The evaluation team found that health facility procurement and finance personnel are not adequately supported due to weak links between the procurement and finance personnel in the health facilities. The woreda- and zonal-level health office managers lack capacity in financial management, and the procurement and finance personnel in facilities report feeling excluded during supportive supervision visits. Due to the limited capacity of Woreda Offices of Finance and Economic Development (WOFEDs), many health facilities are not regularly audited. PRIVATE WING The second aspect of the supply-side reforms is the establishment of private wings. Over the last three years, the activity planned to provide TA for federal referral, teaching, and regional/zonal hospitals to help them establish 63 private wings and achieved 86% of the plan. These private wings mainly provide outpatient services during off-hours. KIIs with Facility Governance Boards (FGBs) with hospital staff participating in private wings found that private wings have contributed to the reduction in patient waiting lists in the regular ward, especially with regard to minor operations. Private wings in some of the government hospitals, such as Ras Desta and St. Paul in Addis Ababa, have become not only a point of revenue retention, but also a point of attraction, where some qualified health professionals working in private for profits and government hospitals are applying to be engaged. There was evidence that health workers indicated improvements in the working environment, pay, and opportunities to use their skills in public hospitals. Focus group discussions with patients found the main reasons they chose private wings over regular wards were (i) the advice they received by health professionals when they meet them during the regular time; (ii) ability to personally choose their physicians; (iii) shorter wait time and more convenient appointments; and (iv) the convenience of service provision after working hours and on holidays. Clients also preferred public facilities’ private wings over private, for-profit providers due to cheaper prices (38%), better quality of the service (31%), and shorter wait times (19%). On average, 45% of specialized doctors’ total monthly income was earned from private wings. 24 In general, the results of private wings vary greatly by region. Federal, Addis Ababa and Oromia regions have relatively good success, while there is average performance in Tigray and Amhara (EEA, 2016). 24 Ethiopian Economic Association, 2015, “Assessment of private wings in public hospitals in Ethiopia: Enhancing the role of academia in evidence generation and action in Ethiopia,” USAID-funded independent research. 19 | FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) Operational challenges of private wings include inappropriate implementation of guidelines, inadequate oversight, and weak standardization. All these challenges have impacted further progress of private wings and this might require a different approach to financing of hospitals and retention of human resources. GRADUATION OF FACILITIES During this activity, a major strategic intervention was the graduation of facilities from support. During the activity period evaluated, a total of only 59 health centers graduated: 28 in Amhara, 4 in Benishangul Gumuz, and 27 in Southern Nations Nationalities and Peoples’ Region (SNNPR), per the HSFR/HFG database. The second intermediate result of the activity involves creating access to health services, which comprised two main interventions: provision of exempted services and enrollment in an insurance scheme, namely CBHI. PROVISION OF EXEMPTED SERVICES The provision of exempted services, especially delivery at hospital levels, was successful in increasing skilled deliveries over the last few years. The percentage of deliveries assisted by skilled health personnel increased from 23.1% in 2012/13 to 72.7% in 2015/16, according to the FMOH Health Management Information Systems (HMIS), 25 and from 10% to 28%, according to EDHS surveys.26 While the government has invested in many different interventions to achieve this gain, there is evidence that free health care delivery at the facility level supported through the health financing reform has contributed to this improvement. In addition to those services at the hospital level, 27 a few additional exempted services are now provided by some health facilities. These include emergency/accidents services and STD prevention. All the human resource costs of these exempted services are currently paid through government salary. Most of the commodities for exempted services are financed through donor support, which undermines long-term financial sustainability until the government takes over. Currently, unlike other exempted services, there is no major financier (donor or government) for antenatal care (ANC), delivery, and PNC commodities used at health facility level. The health facilities are forced to finance these costs using their internally generated revenue, which is not its intended use. According to the EEA assessment and data obtained from HSFR/HFG, health centers and hospitals on average paid annually as much as USD 3,327 and USD 28,120, respectively, to cover the costs of the supplies for these services in 2015/16. Health facility delivery alone consumes about 21% of retained revenue. As a result, the gains made by RRU (increasing funding for facility operational costs) are compromised, as some of these funds are being used to offset this unfunded mandate. With an increasing number of institutional deliveries, the percentage and the amount of resources that health facilities will spend is expected to increase. For example, in Amhara region, the number of patients who utilized exempted services in the areas of ANC, delivery, and PNC at hospitals and clinics increased from 1,065,304 in 2014 to 8,687,001 in the first three quarters of 2016 (for which data is available). The cost of these services to the hospitals and clinics in the region also increased from ETB 43,349,234 [USD 17,391] in 2014 to ETB 91,463,539 [USD 3,977] in the first three quarters of 2016 (for which data is available). There is no clearly defined party responsible for financing the provision 25 CSA, Ethiopian Demographic and Health Survey 2010 and 2015. 26 FMOH, Health Sector Transformation Plan I (2015/16), p. 24. 27 Previously endorsed exemption services include: family planning; prenatal, delivery, and postnatal; immunization of mothers and children; diagnosis, treatment, and follow-up of tuberculosis; voluntary counseling and testing of HIV/AIDS and prevention of HIV/AIDS transmission from mother to child; leprosy management; epidemic follow-up and control; fistula management; and immunization and treatment of health professionals to reduce risk related to occupational hazards. HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT | 20 of these exempted services free of charge. Therefore, hospitals and health centers have had to step in with RRU to address this unfunded mandate, thus compromising the objectives of RRU. Despite this, the evaluation team found that some regions plan to reimburse facilities for commodities for ANC, delivery, and PNC (e.g., Amhara, SNNPR) and have begun to charge for services. HEALTH INSURANCE SCHEMES At the country level, 236 woreda schemes had been initiated in six regions as of June 30, 2016, and of these, 204 had started provision of services to their members. Of the visited regions, Afar has not yet initiated CBHI, Addis Ababa is in the preparatory phase, and the other three regions are implementing the scheme. This achievement is above the activity target of establishing 198 woreda schemes. Although CBHI has not yet started, most of the preparatory works were carried out in Addis Ababa regions to launch the scheme by July 2017. The regulation, directives, and implementation have been endorsed; CBHI boards have been established in 10 woredas with 20 HCs selected to provide service. 6.8% of indigents will be selected. Unlike other regions, Addis Ababa is going to establish two levels of risk pools: at the woreda (90%) and city (10%) levels. In these schemes, about 2.41 million households (37% of eligible households) were enrolled (see Figure 9), which is lower than the set target of 50% of eligible households. The distribution of established schemes and enrolled households among regions is uneven. For example, 77 schemes were established in Amhara, while only 68 were established in Oromia, 41 in SNNPR, and 18 in Tigray. Amhara region alone accounted for 50% of the total households enrolled in CBHI schemes at the national level, while Oromia, SNNPR, and Tigray accounted only for 16.4%, 21.5%, and 12.1%, respectively, in 2015/16 (HSFR/HFG database). Box 3: UHC CBHI enrollment in Dembecha woreda— success factors Dembecha woreda is one of the most successful CBHI schemes visited by the evaluation team. It has achieved almost 100% enrollment of the eligible 19,468 households in 2015/16. The major success factors were the following: • The woreda administration and woreda health office working together as a team; new officials replacing previous ones also continue to be committed to take it as priority. • High level of ownership of the scheme by the woreda and kebele administration; commitment of woreda health office and facilities and WOFED’s commitment to finance its costs. • Proclamation of CBHI week in the woreda. • The woreda cabinet and all the woreda and kebele professionals lead continuous awareness creation. • An incentive mechanism introduced for both providers and mobilizers created a sense of competition. • WOFED budgeted ETB 81,000 allocated to motivate kebeles (2,000 ETB award for each kebele for 100% enrollment); three individuals were also rewarded with 500 ETB each. • Providers were motivated through the provision of a generator and televisions. • CBHI enrollment was one of the core performance measures for each woreda cabinet member in each quarter and is closely followed weekly and biweekly. • Leaders allocated the necessary budget to recruit human resources and other inputs to address community complaints on service quality. • All the leaders use their mobile phones to take passport-size pictures of members, to be used on the CBHI membership cards, to reduce the opportunity cost to the farmers who have to travel to towns to get their pictures taken. 21 | FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) Figure 9. Number of functioning schemes and enrollment rates The performance of schemes also varies in enrollment rate within and among regions. For example, in Dembecha woreda, Amhara region, the scheme has reached 100% of the eligible households (see Box 3). There is a very wide range of enrollment rates among schemes ranging from 5% in Kercha (Guji zone in Oromia region) to the highest in Dembecha. The variation of enrollment rate across schemes is mainly attributed to a varying degree of ownership and commitment on the establishment and subsequent follow￾up of CBHI schemes by the respective regional and woreda stakeholders. This in turn is attributed to the unclear institutional arrangement of CBHI and especially the varied level of engagement of the health office at the woreda and kebele levels. The activity complemented increased access through CBHI with the protection of the poor through targeted subsidies. Currently, the government is supporting 276,444 households through CBHI indigents, and 1.5 million fee-waiver beneficiaries. Some facilities have started special innovations to protect the ultra-poor (see Box 4). The number of fee waiver beneficiary families selected in 94 woredas of Addis Ababa reached 210,850 (2016), with a total budget of ETB 52,114,888 (USD 2,265,865). Six hospitals requested ETB 12,894, 451 (USD 560,628) from woredas and only 57% of that amount was reimbursed. According to the recent NHA Household Survey, 27% of individuals in Ethiopia did not pay for services, 80% of whom because they are exempted, 4% because they are financed as fee-waiver beneficiaries, and 3% because they are members of CBHI. 28 However, fee-waiver services and exempted services are a greater challenge in some regions like Addis Ababa because of the greater number of people in Addis Ababa who qualify for such protection. The sub-city budget allocations in Addis Ababa are inadequate to match the number of people who qualify for these services. The number of poor protected through fee 28 FMOH, 2017, National Health Accounts: Household service utilization and expenditure survey EFY 2008 (2015/16) 13 31 66 204 48 37 33 37 0 50 100 150 200 250 2012/13 2013/14 2014/15 2015/16 # of Functioning Schemes Enrollment Rate (%) Box 4: Innovation to protect fee waivers from shortage of pharmaceutical supplies at Ras Desta Hospital Ras Desta Hospital, realizing that indigents do not have the resources to purchase pharmaceuticals at private pharmacies (when these are out of stock at the hospital), has established pharmacy specifically for indigents. This pharmacy is the best stocked as it is prioritized. In case of shortages of pharmaceuticals at the hospital, the indigents’ pharmacy is protected from stock-outs by being the last pharmacy in the hospital to run out of pharmaceuticals. HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT | 22 waivers and CBHI schemes has reduced over time. In woredas where CBHI is established, the number of indigents covered in CBHI was reduced by 20,692 households between 2014/15 and 2015/16, and the number of fee-waiver beneficiary households decreased by 300,000 between 2014/15 and 2015/16 (HSFR/HFG database). CBHI schemes also contributed to increased financial mobilization and use at the facility level through increased reimbursements from members with high utilization. Over the last three years, ETB 672.5 million [USD 29,239,130] in resources have been mobilized. Of these, ETB 524 million [USD 22,782,609] was mobilized from members (78% of the total resources mobilized) and ETB 89.1 million [USD3, 873,913] through targeted subsidy (13%); the remaining 9% (ETB 58.9 million [USD 2,560,870]) was through general subsidy. The share of general subsidy has decreased from 25% of the total resources mobilized to 9%. During 2015/16, of the total health service visits by CBHI members, about 91% of the visits were made to health centers, while the remaining 9% were to hospitals. In line with health service utilization, of the total reimbursement made to health facilities by schemes, 71% were made to health centers, and the remaining 29% went to hospitals (HSFR/HFG database). Through KIIs with woreda health offices and FGDs with CBHI boards, the evaluation team found varying degrees of CBHI ownership on the part of regions and woredas, which has led to variations in performance. As repeatedly reflected in KIIs with the CBHI scheme staff, in addition to institutional arrangement variations, the number of scheme staff at the woreda level was inadequate, particularly in its data management and medical auditing staff. At the kebele level, the scheme did not have its own structure and staff. Kebele managers perform scheme-related activities in addition to the many duties they currently have. There is also high turnover of scheme staff due to low salaries compared to others with comparable educational backgrounds and work experience. There was no clear career path defined to inspire staff to stay in CBHI schemes. The CBHI staff was also found to lack adequate training in managing financial transactions and medical auditing. A system of periodic auditing of CBHI schemes was not established in most of the regions. This was primarily due to the limited capacity of the woreda finance office to undertake audits of different government offices. The only region that undertook auditing of schemes for the period 2015/16 was Amhara. According to FGDs with the CBHI members and KIIs with scheme managers and facilities, another challenge was the double standard of the use of referrals to primary hospitals. While CBHI members are requested to follow the referral procedures by initially going to HCs, non-members can directly go to primary hospitals. The CBHI members felt discriminated against, and there was a need to develop a strategy to introduce a bypassing fee for non-CBHI members or to allow members to directly access similar types of treatment. While respondents reported improvement in the availability of diagnostics, medicines, and attitude of staff, challenges remain in meeting the higher expectations of CBHI members. Another major area of the activity support was to establish a social health insurance (SHI) scheme for the formal sector in Ethiopia, supported by a previous USAID project. The SHI scheme had not yet begun during the period of this evaluation, as the government postponed its commencement on more than three occasions. According to the KIIs with the EHIA and its branches, the issues raised during consultations with the general public on SHI implementation have contributed to the postponement. The issues raised by the public included, but were not limited to, the readiness of health facilities to provide quality health care; the perceived high contribution level requested for SHI; and the perceived unreasonable demand for both husband and wife to contribute as individuals when they are both formal sector employees. Currently, it is estimated that only 13% of the total population is covered in health insurance through 23 | FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) CBHI (HSFR/HFG database). However, the government has established the Health Insurance Agency, and its 24 branches are spread across the country, employing about 750 staff. GOVERNANCE The other intermediate result of the activity was increased governance of facilities and insurance schemes. The activity exceeded its targets by 5% and 2% in establishing facility governance boards at hospital and health center levels, respectively (see Table 4). These governance boards were found to function in accordance with their expected roles. Discussions with stakeholders show that the composition and degree of competency of the facility governance board members, and hence their effectiveness in supporting the facility, varied among, and sometimes within, regions. Some boards were reportedly competent and committed, while others were less so. The Dubti and Dangila hospitals, as well as the Dembecha health center management boards, were identified as some of the best examples that others could learn from (see Box 5), since they go beyond their obligations and try to mobilize resources to improve quality. High turnover of members in the facility governance boards contributed to the reduction in the level of competency of governance boards. On the other hand, governance boards with a relatively stable composition of members over time functioned more competently and more routinely. TABLE 4. PERCENTAGE OF HEALTH FACILITIES MANAGED WITH BOARDS INDICATOR TARGET ACHIEVED % of hospitals managed with boards where communities are represented 90% 95% % of HCs managed with boards where communities are represented 90% 92% Source: HSFR/HFG Database For example, the governance board of Meshualekia Health Center, in Kirkos sub-city in Addis Ababa, has been stable for over six years, with all the seven governing board members having been trained by the activity and still functioning in the board. Meshualekia Health Center is considered a model health center in the area, particularly regarding the health care financing reforms. In the Oromia region, for example, Chefedonsa Health Center, in Gimbichu woreda, has a stable facility governance board that is functioning well at the facility and involves the community in the governance of the health center. Criteria for membership of governing boards was already prescribed in regional directives and manuals. However, people listed as members of the governance boards are usually those with other responsibilities outside of the boards. This creates challenges for these members, as they may not have time allocated for governance board duties. For some boards, the prescribed board members did not have the management capacity to address governance board responsibilities (e.g., the chairs of boards in rural HCs in Amhara are the kebele chairs). High-performing boards are characterized by making timely decisions. The evaluation team found that boards led by one member of the woreda cabinet (as opposed to having all woreda cabinet members on the board of a single facility) were more likely to function well (see Box 5). At the other extreme, facility governance boards led by Kebele administrators were less likely to have the level of leadership skills required to function effectively. In areas where there has been a high level of commitment to the boards HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT | 24 at different levels (region, woreda, facility, and kebele level), this commitment has contributed to the success of the boards. Respondents reported that the initial training provided by the activity to the facility governance boards served as a foundation for governance boards to operate with minimal support from the activity. The guidelines and manuals that the activity helped develop have now become key learning tools for new governance board members who did not benefit from the initial training. However, these guidelines and manuals have not been revised for some time, and are not well aligned to the realities of the field, such as issues around composition of boards, the positive and negative lists, aligning with the health center and hospital reforms, and working with CBHI boards. The major gaps observed in governance include: (i) the high turnover of governance boards and the lack of systematic training to new members on the manuals and guidelines; (ii) the varying levels of competency of the governance boards, affecting their effectiveness in supporting the facility; and (iii) in some regions, individuals who are too busy with other duties or may not have the right level of commitment become the governing board members, which affects its functionality. PROVISION OF TECHNICAL ASSISTANCE The provision of technical assistance (TA) to FMOH, regions, and woredas was another area of activity support. The activity had one central, four regional, and seven satellite offices with 98 staff, of whom 59 were technical staff. Of these technical staff, only seven were embedded TA staff. Overall, the TA was considered effective, as it achieved most of the targets set. The FMOH, regions, and woredas confirm that the technical assistance they received from the activity was effective and responsive. The activity did not have its own parallel plan for TA; rather, it jointly planned with its regional and federal counterparts. Because of the level of coordination between the activity and government counterparts, the activity staff were widely viewed by stakeholders as “government staff.” Respondents in regions such as Afar clearly stated that the health financing reform “would not have moved forward without the TA support.” Some woreda CBHI scheme managers and woreda health office officials in Amhara stated that CBHI enrollment and its functioning improved when the activity staffs came to discuss issues directly with woreda administration. They also worked directly with zonal leaders to encourage ownership and commitment. Respondents also reported good practice of skills transfer during the National Health Account VI development, which FMOH respondents say helped to institutionalize it into government structures. With that skills transfer, the FMOH felt confident that it could lead the next NHA with minimal external support. The development of the health care financing manuals and guidelines developed through the activity’s TA Box 5: Best practice example—the Dubti hospital board in Afar The hospital board was ineffective until June 2016. With the arrival of a new CEO, who nominated members, it was restructured. So far, the board has: • Met every month for four months to set their priorities; • Endorsed the 2009 EFY plan and budget; • Conducted two pubic conferences to engage the community and improve the quality of care; • Negotiated with the Bureau of Finance and Economic Development (BOFED) and RHB to get additional budget for the hospital this financial year promised; • Ensured the hospital got 24-hour water services and started to dig a well; • Negotiated with the RHB to pay the hospital’s electric cost debt of ETB 500,000; • Enabled the hospital to access houses for doctors and critical staff from Tendaho farm; • Maintained the road around the hospital; and • Been proactive in the Clean and Safe for Health (CASH) program. 25 | FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) helped the facilities to continue implementing the reforms, despite the turnover of managers and board, which has positive implications for sustainability. However, the capacity of the RHBs to lead and manage the health care financing reform remains weak, mainly due to (i) lack of well-defined structure to lead and manage the process; (ii) lack of staff to take the lessons from the activity; and (iii) inadequate capacity outside the activity staff, as most of the training are conducted by the activity staff rather than working through regional training institutions and universities. In DRSs like Afar, there is only one activity person providing TA, with no decentralized operational budget or vehicle, which undermines effectiveness and functionality. Overall, there is heavy reliance on the activity staff in all the regions. 2C: EFFICIENCY Efficiency in this evaluation was measured by comparing the outputs in relation to the inputs and estimating the support used to achieve the desired targets. Efficiency analysis generally requires substantial quantitative data and a detailed analysis; however, this evaluation was able to generate some limited evidence on the performance of the activity around efficiency. First, by investing in HCF trainings with an average cost of USD 1, health facilities were able to mobilize USD 25 of internal revenue, which indicates a high return on investment. Analysis of the HSFR/HFG cost data by the evaluation team documented that the average cost of HCF training per USD 100 of retained revenue mobilized has decreased from USD 13 during the first year of the activity to USD 3 in 2016. The major issues around efficiency in the first-generation reforms, especially of the private wing, include poor implementation of guidelines, inadequate oversight and vested interest, and not reimbursing utilities and other inputs used by the private wings from the general ward prior to distribution of revenue among the different cadres of workers. The cost of enrolling an informal sector household into the CBHI scheme was USD 0.37. 29 The average cost per CBHI trainee (woreda and kebele cabinets and scheme staff) was USD 21.78 during the last three years. The average cost for establishing a CBHI scheme was USD 2,827.87 but varied when different components of capacity building were included. For instance, if all of the 321 schemes were given desktop computers, printers, and motorbikes, the average cost of establishment would increase to USD 3,750.26; if the support of desktop computers, printers, and motorbikes is excluded from establishment cost, the average cost of establishing a scheme decreases to USD 1,899.7 (HSFR/HFG database). The high turnover of members of facility governing boards across the regions had a negative effect on the efficiency of training offered to facility governance board members. During the early stages, when the facility governance boards were being formed, the activity offered training on the functions of the board to governance boards. However, subsequent trainings were not offered to governing boards and therefore new members of the governance boards have not benefited from that training. The new board members rely on guidelines and manuals produced by the activity to update themselves on the functions of the governing boards. Outsourcing of non-clinical services was designed to enhance efficiency of service providers by ensuring that they focus on their core business (service delivery) and obtain better quality and competitive costs for the services outsourced. Currently, the activity assisted 71 health facilities in outsourcing non-clinical 29 These costs don’t include the salaries of the project staff. HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT | 26 services. There is no consolidated or comprehensive evidence on its impact. However, limited evidence from some health facilities show that some realized cost reductions due to outsourcing while at the same time getting better food and security (see Table 5). 2D: SUSTAINABILITY This evaluation has also examined the extent to which the gains of the HSFR/HFG activity will continue without external support and with the ownership and leadership of government. The evaluation team finding is that the supply-side interventions (RRU, fee waivers, boards, private wing, outsourcing, etc.) can be sustained with a responsible exiting strategy. This is demonstrated by the fact that: • The activity only provided training for these interventions when the reforms started in health facilities, yet the interventions have continued with minimal subsequent support since then; • The guidelines and manuals developed are the most important instruments for sustaining gains with high turnover of facility managers and boards; • Regions and woredas have started budgeting for HCF reform implementation and for refresher trainings; the RHBs and woredas expressed willingness and ability to take over if a proper exit strategy exists and handover takes place during the activity period. As described above, the governing boards are currently operating with minimal support of the activity, after the initial support that they received during the formation of the governing boards. Most of the governing board members (particularly the new ones) rely on guidelines and manuals to guide their functioning. The evaluation team found a high commitment to facility governance boards by the regions, 30 The name of the hospital has now changed to Gebertsadik Shawo Memorial General Hospital 31 Formerly Hawasa HC (upgraded to hospital) TABLE 5. EXAMPLES OF OUTSOURCING OF NON-CLINICAL SERVICES IN SOME FACILITIES REGION NAME OF HOSPITAL OUTSOURCED SERVICES MONTHLY COST INCURRED FOR THE SERVICES (ETB) YEARLY COST REDUCTION (ETB) % OF COST REDUCTION BEFORE OUTSOURCING AFTER OUTSOURCING Amhara Dessie Cleaning 108,333.33 [USD 4,710] 83,333.33 [USD 3,623] 300,000.00 [USD 13,043] 23% Security 27,212.43 [USD1,183] 25,200 [USD 1,096] 24,149.16 [USD 1,050] 7% SNNPR Bonga30 Food 68,577 [USD 2,982] 55,540 [USD 2,415] 156,444 [USD 6,802] 19% Adare31 Cleaning 96,787 [USD 4,208] 28,600 [USD1,243] 818,244 [USD35,576] 70% Adare Catering 71,535 [USD 3,110] 40,000 [USD 1,739] 378,420 [USD 16,453] 44% Total annual cost reduction 1,677,257.16 [USD 72,924] 27 | FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) woredas, facilities, and communities; all these stakeholders would like the governance boards to continue functioning, with or without activity support. Regarding the demand-side interventions, particularly CBHI, currently the institutional arrangement of CBHI schemes varies from region to region. In some regions, it is housed under woreda administration, and in others, under the woreda health office. There is also no clarity on the role of EHIA and who should take the lead on CBHI implementation—the woreda administration or woreda health office or EHIA. This institutional arrangement, coupled with inadequate staff at the scheme/woreda level and absence of its own structure at the kebele level has forced renewing of membership regularly through a campaign-based approach, which raises doubts about its sustainability. There was evidence showing that some schemes are not financially sustainable. The Amhara CBHI scheme audit showed that among the 61 audited schemes, 23 had excess expenditure over income. This resulted in a need to revise the level of contributions in some of the schemes in the region without undertaking a clear analysis of the implications. For instance, the different levels of contribution within the region, if not corrected, might create a challenge of establishing a single regional CBHI risk pool and staying with the current model of woreda-based fragmented pools. According to the KII with HSFR/HFG, EHIA, and CBHI schemes, the financial sustainability of schemes is also compromised by the reduction of the FMOH general subsidy from 25% of the total contribution to only 10% of the contribution. The main challenge of ensuring the existence of sustainable capacity at the regional, woreda, and facility levels is the lack of having a structure or a case team at the RHB and woreda levels that spearheads HCF reform. Currently, health care financing is being implemented under the curative and rehabilitative process. There is a capacity and knowledge gap in identifying implementation gaps, prioritization, and the proper planning and budgeting. There is no systematic effort to introduce managers to financial and human resource management (skills and awareness) before taking their respective managerial/leadership posts. HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT | 28 IV CONCLUSIONS RELEVANCE The HSFR/HFG activity design and the interventions being implemented over the last three years were found to be relevant in terms of providing quality and responsive care to the community, and they were fully aligned with government policies and strategies. The activity not only reduced financial barriers to the community, including the very poor, but also enhanced their voices to demand accountability from service providers about their responsiveness through facility-community dialogue forums and also when they went to health facilities to access services. The theory of change at the time of design was relevant and adequate to help Ethiopia achieve the set health care financing targets set in the HSDP/HSTP. The various components were relevant to the challenges of the health sector and translated into action. The supply-side financing reforms enabled facilities to generate, retain, and use revenue to improve quality of care per their priorities as well as to improve facilities’ ownership of the reform. The governance interventions were relevant for community participation and for increasing accountability of the facilities. Operational budget shortages at the facility level were no longer the major challenge as was the case in the past before the health care financing reforms. The demand-side reforms facilitated the increasing uptake of the previously exempted services. Enrollment in CBHI schemes reduced financial barriers at the point of use of health service; enabled communities to seek care immediately when they feel sick; enabled women to visit health facilities without seeking money from their husbands; and enabled more protection of the very poor through fee waivers and CBHI indigent provisions. The CBHI also strengthened communities’ voice and engagement, leading to more demand for good and responsive service delivery. The interventions fit well with the government’s agenda of moving towards UHC and improving the quality of care. The follow-on activity may benefit the country if it also includes evidence generation and strategic engagement for policy development on increased tax-based resource mobilization for health, including innovative financing, which would push progress towards UHC and take increasing responsibility of financing-exempted service commodities as one of the strategic support to be provided. EFFECTIVENESS The HSFR/HFG activity is on track to achieve its strategic objectives in terms of improving quality of care, reducing financial risk protections, and improving accountability and responsiveness of health providers. More than 3,244 health centers and 225 hospitals have started retaining and using retained funds to improve quality of care through the oversight of the established facility boards, more than the target set. Although the national coverage rate for health insurance has not reached 20% as planned due to the delay in providing social insurance, there is clear evidence that CBHI interventions are moving towards that objective with the number of woredas implementing the scheme exceeding the target. Some best practices, such as in Dembecha woreda, have demonstrated the possibility of achieving UHC for the non￾formal sector in the agrarian context. Although graduation of health facilities from support seems to fall far short of targets, the activity is only supporting health facilities that are now starting the reform and is no longer providing substantial support to the health facilities that began the reform earlier. 29 | FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) EFFICIENCY The review of the activity’s implementation documented important gains around efficiency and value for money. However, the frequent turnover of facility and woreda CBHI boards and facility heads had a negative effect on the efficiency of the activity as new members must be re-oriented, and additional investments are required to acquaint them with the guidelines and manuals. The training mechanisms by the activity could also be enhanced if it were managed through regional-based training institutions (from activity-based to local training institution–based). SUSTAINABILITY There is solid evidence showing that supply-side health care financing reforms (RRU, boards, fee waivers, provision of exempted services) were being provided with very little support from the activity and can sustain themselves in the short term with implementation of a systematic exit strategy. The proposed establishment of the health care financing structures at the regional levels will facilitate ownership and sustainability. Most of the CBHI schemes can function sustainably. However, the lack of clear institutionalization mechanisms, especially at the kebele level; inadequate incentive mechanisms for scheme staff; the delay in establishing larger risk pools; scheme-based revision of contributions; the decline in general subsidy; the lack of regular auditing of schemes; and the inability to improve the quality of services and the ability of the facilities to respond to increased community demands present major risks to the sustainability of CBHI. HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT | 30 V RECOMMENDATIONS RECOMMENDATIONS FOR THE REMAINING PERIOD OF THE ACTIVITY Develop and implement exit strategies for the supply-side reforms. In its current phase, the activity should focus preparing an exit strategy from supply-side interventions. Regional health bureaus and their structures should be able to take over most of the activities with a proper and agreed-upon exit strategy. Revenue retention, fee waivers, governance boards, outsourcing, and private wards can be implemented without significant activity support in the consolidating regions. However, in its remaining period, the activity should focus on ensuring that it puts the right capacities and systems in place before exiting from the first-generation reforms. These include: • Developing/revising training modules for each component of the health care financing reforms that have been implemented; • Preparing selected trainers to take over future trainings for both new facility implementers and existing staff, to reduce the impact of staff and governance board turnover; • Revising and updating the health care financing guidelines and manuals, printing and distributing them to RHBs, and handing them over as facility property; • Recommending structural arrangements at the woreda and regional levels on how to build capacities at all levels; • Ensuring the development of a health financing information system (a data system) to bring standardized information to all levels of the health system; • Revising user fees. This activity has not been carried out systematically, and the regions have therefore found it difficult to revise user fees. In its remaining period, it is essential that the activity refocus on implementing some of its previously-developed models and building regional capacity to generate evidence, undertake revisions, and advocate for its endorsement by the regions. Regarding the demand-side reforms, the focus of the activity during the remaining period should be: • Exploring options for institutional arrangements of CBHI schemes at the regional level that take the unique nature of CBHI implementation into consideration and lay the groundwork for regional risk pooling; • Assessing the number, composition, and salaries of scheme management staff, with a possibility of developing alternative career paths for these cadres; • Proactively engaging WOFEDs and other institutions to allocate resources and conduct regular audits of schemes. The activity can then act on the results of these audits by providing support to weak schemes and taking corrective measures to ensure their financial sustainability; • Building the capacity of the RHB/EHIA and engaging with relevant stakeholders to improve the quality of health services, with a focus on addressing drug shortages and ensuring the program’s sustainability and growth. Training and capacity building: The activity should explore options for providing all training under the leadership of RHBs. There are regional management institutes and universities and 56 in-service training centers available as options. These institutions should be assessed to ensure that they can provide sustainable mechanisms for training and capacity building. 31 | FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) ANNEXES Annex A: Evaluation Statement of Work Annex B: Evaluation Design Matrix Annex C: Sources Reviewed Annex D: List of People Consulted Annex E: Data Collection Schedule Annex F: Interview Protocols and Guides Annex G: FGD Guide Annex H: Profiles of the Evaluation Team Members Annex I: References Annex J: HSFR Evaluation Sites HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT | 32 ANNEX A: EVALUATION STATEMENT OF WORK STATEMENT OF WORK MIDTERM PERFORMANCE EVALUATION Support to Health Sector Finance Reform (HSFR)/Health Financing and Governance (HFG) Activity, Award Number AID-OAA-A-12-00080 I. INTRODUCTION. This performance evaluation is broadly designed to examine what the Health Sector Finance Reform (HSFR)/Health Financing and Governance (HFG) field support activity has achieved at the mid-way point in its implementation; how well it is being implemented; whether expected results are occurring or are likely to occur before the end of the activity. The findings, analysis and recommendations will inform strategies to improve implementation of HSFR/HFG in the remainder of the activity’s life. Activity Name Health Sector Finance Reform/Health Financing and Governance (HSFR/HFG) Global Field Support Mechanism Cooperative Agreement Number AID-OAA-A-12-00080 Start Date – End Date August 1, 2013 – July 31, 2018 Total Estimated Cost $42,000,000.00 Activity Funding Health, PEPFAR Implementing Partner Abt Associates Activity Manager Eshete Yilma USAID’s evaluation policy encourages independent external evaluation to increase accountability to inform those who develop programs and strategies, and to refine designs and introduce improvements into future efforts. In keeping with that aim, this external evaluation will be conducted to review and evaluate the performance of the USAID-funded Health Sector Finance Reform/Health Financing and Governance (HSFR/HFG) field support activity being implemented by Abt Associates. The evaluation will focus on assessing the activity’s performance in achieving its goal, objectives, and results from its starting period to the time of the evaluation. Theory of change By improving the quality of health services and reducing financial barriers, health service utilization will 33 | FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) be increased. Specifically, activities are aimed at: ● developing schemes for improving health financing options, ● increasing availability of budgets at service delivery points through revenue retention and utilization, ● communicating on/promoting health insurance and protection of the poor, ● increasing community-based health insurance coverage, ● improving the institutional capacity of the health insurance agency (including its networking and governance), ● establishing performance measurement systems, ● updating policies and strategy documents. The activities will improve access to health services, the quality of health services, the governance of health insurance, and program learning. Ultimately, these improvements will lead to increased utilization of health services. Results Framework of the Activity Increased utilization of health services IR 1. Improved quality health services IR 2. Improved Access to Health Services IR 3. Improved governance of health insurance and health services IR 4. Improved program learning Availability of operational budget at a point of service delivery increased Promotion of motivation schemes including private wing in public hospitals Health facilities that successfully implemented HCF reforms recognized and graduated 1.4 Coverage of CBHI and SHI increased Resource mobilization for health insurance increased Protection mechanism for the poor through waiver, CBHI and other mechanisms expanded Communication and mass media coverage on health insurance enhanced Management of exempted programs strengthened Facility Governance Boards management capacity improved Networking of health insurance schemes and boards strengthened Institutional capacity of health insurance agency improved Availability of evidence for decision making including Routine financial and beneficiary data improved NHA and other surveys conducted Updated policy and strategy documents, success stories and documentation improved M&E systems established HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT | 34 BACKGROUND The Ethiopian health system is characterized by extreme underfinancing, low protection mechanisms of the poor, and a lack of risk pooling and cost sharing mechanisms, all of which result in low access to and poor quality of health services. To address the growing need for health services and ensure sustainable health financing, the Ethiopian Ministry of Health, in collaboration with development partners, initiated and institutionalized several reforms. The first generation health sector finance reforms include six health financing reform components: revenue retention and utilization (RRU), establishment of private service wings in public health facilities, reform of fee waivers and exempted health services, user fee setting and revision, establishing facility governance structures in public health facilities, and outsourcing of non￾clinical services. The second-generation health reforms are focused on community-based health insurance and social health insurance schemes. USAID is closely working with the Ethiopian Ministry of Health to strengthen the health system in the country and is funding its flagship activity in health finance and governance, called Health Sector Finance Reform/Health Financing and Governance (HSFR/HFG). The Health Sector Finance Reform/Health Financing and Governance (HSFR/HFG) field support project is a five-year activity implemented by Abt Associates in all of the nine regions and two city administrations of Ethiopia. The health care financing (HCF) reform components are implemented in all regions targeting existing and new health facilities (hospitals and health centers). Health insurance schemes are being piloted in the four larger regions (Amhara, Oromia, Southern Nations and Nationalities, and Tigray) in over 200 districts (or approximately 20-25% of the districts in the regions). The objective of HSFR/HFG is to support the GoE’s health sector finance reform strategy that aims to improve access to and quality of health services in alignment to GoE and U.S. government health system strengthening priorities and development goals. III. Objectives of the Activity The overall objective of the activity is to increase health services utilization through improved quality of health services and reduced financial barriers. The specific objectives of the activity are to: ● Improve quality of health services ● Improve access to health services ● Improve governance of health insurance and health services ● Improve program learning USAID’s strategy to expand HSFR and health insurance shall consolidate the achievements of the reforms implemented at health facilities and support the implementation of health insurance to improve access, equity and quality of health care services. The following are the key expected results/outcomes of the HSFR/HFG. ● Increase health service utilization in 185 CBHI Districts/Woredas (13 current pilot and 172 second phase pilot Woredas) from 0.3 to 0.6 new consultations per person per year. ● Health facilities managed with boards where communities are represented increased to 90%; ● Public health facilities retaining and using their revenue increased to 90%; ● Share of OOP expenditures to total health budget reduced from the current 37% to 30%; 35 | FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) ● Proportion of people enrolled in health insurance increased from the current 1% to 20% at the end of the project. So far, the activity has been providing technical support to health facilities, regional health bureaus, zonal/district health offices and federal agencies of the Ministry of Health in scaling up the first generation health sector finance reforms (revenue retention and utilization, fee waiver systems, exempted services, private wings in public facilities and outsourcing of non-clinical services) and the second generation reforms (community-based health insurance and social health insurance schemes) in the country. IV. PURPOSE AND USE OF THE EVALUATION The main purpose of the midterm evaluation is to assess the viability of the HSFR design, the progress towards expected results, and the implementation successes and challenges. The midterm evaluation will be focused examining the activity relevance, efficiency, and effectiveness. It will also ascertain the potential for sustaining the implementation and the results of HSFR after USAID funding ends. IR 4: “Improved Program learning,” is not the focus of this evaluation. The findings, analysis and recommendations are intended to inform USAID and the GoE to improve implementation of HSFR/HFG in the remainder of its life. V. EVALUATION QUESTIONS The intent of the evaluation questions is to assess the feasibility HSFR/HFG activity theory of change, results framework and objectives. The evaluation must therefore examine the relevance, efficiency and effectiveness of the elements of the activity theory of change and the objectives. The key evaluation questions include the following: 1. To what extent is the HSFR theory of change and its objectives adequate, relevant, efficient, effective and viable? a. Are the IRs and their associated activities sufficient and necessary to meet the project’s objectives? If not why?32 b. Are there additional indicators that should be identified and tracked that will assist in measuring the short and long-term project outcomes? What new or substitute indicators would be especially useful for future results management? 2. What progress has been made at mid-way towards achieving the key elements of the HSFR theory of change and performance objectives (including establishing strategies for sustainability)? a. Establishing schemes for improving health financing options b. Increasing budgets at service delivery points, revenue retention and utilization c. Communicating on/promoting health insurance and protection of the poor d. Expanding/increasing community-based health insurance coverage e. Strengthening institutional capacity for governance of health insurance and health services (including the capacity of the health insurance agency in governance and networking) f. Establishing performance measurement systems g. Updating policies and strategy documents h. Improving the quality of health services 32 Note that IR4: Improved program learning, is excluded from this evaluation HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT | 36 i. Increasing access to health services 3. What measurable actions have been taken to identify and resolve activity implementation challenges? a. How timely and effectively were activity successes and lessons learned disseminated? b. Is there a need for USAID, the implementing partner and the relevant local government counterparts to improve their management, collaboration and communication modalities in the implementation of this activity? If so, what and how? c. Looking beyond HSFR performance against indicator targets, what are the key performance accomplishments and limitations of the activity to date? 4. Which intervention(s) should the activity prioritize in its remaining period to maximize achievements? VI. METHODOLOGY The evaluation team will be responsible for developing an evaluation strategy and methodologies that include a mixed-method approach. The team should present an evaluation design matrix showing the source of data, method of data collection and the tools to be used to answer each of the evaluation questions. The methodology will be presented as part of the draft work plan as outlined in the deliverables below, approved by USAID/Ethiopia and included in the final report. The evaluation team will have available for their analysis a variety of activity implementation documents, baseline surveys and reports. Methodology strengths and weaknesses should be identified and measures taken to address those weaknesses. All data collected and presented in the evaluation report must be disaggregated, as appropriate, by sex and geography. The regions and woredas where the evaluation team will collect data will be determined in consultation with the implementing partner and USAID. Use of quantitative data, includes, but is not limited to: ● Comparison of current indicator values to baseline data for selected output and outcome indicators depending on availability of primary and secondary data. ● Map out the activity results against performance measure indicators to show the total number of indicators under each result and whether performance is met/on target (90-110%), exceeded (>110%), or not achieved (<=89%) ● Conducting exit interviews (with CBHI enrollees and non-enrollees) in CBHI facilities Use of qualitative data, which includes, but is not limited to: ● Document Reviews ● Key Informant Interviews (KIIs) ● Focus Group Discussions (FGDs) VII. EXISTING SOURCES The consultants will review the following documents: a) Activity descriptions b) Work plan c) Quarterly reports d) Annual reports e) Budget and financial reports f) M&E plan g) Government 37 | FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) h) Evaluations/studies done by the implementing partner i) Project performance data j) Project-generated assessments k) Government of Ethiopia Health Sector Reform background reports VIII. TEAM COMPOSITION AND REQUIRED SKILLS The evaluation team shall consist of a Team Leader who leads the team, one independent international expert, two high level Ethiopian experts in health financing and one Monitoring and Evaluation specialist. The evaluation Team Leader, in consultation with other team members, will be responsible for team coordination and performance and for ensuring the timeliness and quality of deliverables. The Team Leader is also responsible to ensure that the evaluation report meets the USAID Evaluation Report Standards (see Annex A) USAID may propose internal staff members from USAID/Washington or other Missions to accompany the team during site visits or participate in key parts of the evaluation (specific event participation to be determined in conjunction with the contractor and the Team Leader), and they are expected to provide written inputs to the draft report prior to their departure from country. A statement of potential bias or conflict of interest (or lack thereof) is required of each team member. Team Qualifications: 1. National/International Team Leader (one): The Team Leader should have at least eight years work experience in health sector financing reform efforts in Ethiopia or other African countries and led at least two similar evaluations. The evaluation Team Leader should have expertise in health economics or in related areas and also should have strong team management skills, and sufficient experience with evaluation standards and practices to ensure a credible product. It is preferable if the Team Leader has the experience with Health Insurance Models/particularly in Community-based health insurance. The Team Leader must also be fluent in English and have strong writing skills. 2. International Expert (one): With at least five years of work experience in evaluation and performance monitoring, the international expert should be a senior-level technical analyst specialized in the areas of social and community-based health with a preference for those with experience with community based insurance models in developing countries. This expert must be fluent in English and have strong writing skills. 3. Local Ethiopian Experts (Two): The Ethiopian experts should have a solid understanding of Ethiopian health sector reform issues with a minimum of five years of work experience in monitoring and evaluation of health sector development. The local experts should have expertise in health economics or in health related areas, including experience with Community-based health insurance. Experience in they should also be proficient in English and Amharic. 4. M & E specialist (one): The M&E specialist should have a minimum of five years of work experience especially in the health sector and preferably in health sector financing. IX. EVALUATION SCHEDULE The estimated period for undertaking this evaluation is 75 working days including time for review of products. The ideal start time for the evaluation team is March 2017, however, the date will be finalized between USAID and the EPMES Contractor. HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT | 38 The evaluation team is required to work six days a week. The team is required to travel to selected provinces in each region where the activity is being implemented until data saturation is achieved. At least 50% of the consultants’ time will be spent outside Addis Ababa to conduct interviews with activity staff, government partners, and project beneficiaries. The evaluation team will prepare an exit briefing and presentation of the interim findings, which it will deliver to USAID staff before the consultants depart Ethiopia. Following from this briefing, the consultants will put together the draft report. USAID will have 2 weeks to provide comments and suggestions to the report. If appropriate, the draft report may also be shared with the implementing partner for comment. Comments from USAID will be incorporated before the submission of the final draft. The final report should be submitted in the beginning of June 2017. The findings from this report will be used in the development of the HSFR/HFG 2017 work plan. Illustrative Level of Effort (LOE) in person-days (to be finalized in Work Plan) Activity Team Lead (1) International Consultant (1) Local Consultant (3) Desk review of documents and analysis of indicator and other secondary data 7 7 21 Draft evaluation work and survey instruments; plan logistics 8 8 24 Travel to country 0 2 0 In-brief with USAID (Planning Meeting w/ HAPN team and Program Office) 1 1 3 Team finalizes evaluation work plan, design, methodology, and data collection tools. 2 2 6 Organize logistics for field work 2 2 6 National Level Meetings in Addis Ababa 1 1 3 Test data collection instruments and adjustments, if any 3 3 9 Field work including travel days 18 18 54 Preliminary Data analysis 7 7 21 Interim Findings Meetings (with preparation) 2 2 6 Debriefing with mission staff with draft findings and recommendations 1 1 3 Expats depart country 0 2 0 Write 1st draft of report 14 10 12 Final exit presentation to USAID (with PowerPoint presentation and draft evaluation report) 1 1 3 Final exit presentation to relevant partners (with PowerPoint presentation) 1 1 3 Finalize report and submit for final approval 6 6 6 One-page briefer 1 01 0 Total LOE 75 75 180 X. MANAGEMENT Social Impact, the Contractor managing the Ethiopia Monitoring and Evaluation Service (EPMES) activity will identify and hire the evaluation team, pending the Contracting Officer’s Representatives (COR’s) and 39 | FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) HIV/AIDS, Population and Nutrition Office/Health Systems Strengthening Team (HAPN/HSS) concurrence and CO approval, assist in facilitating the work plan, and arrange meetings with key stakeholders identified prior to the initiation of the fieldwork. The evaluation team will organize other meetings as identified during the course of the evaluation, in consultation with the EPMES Contractor and USAID/Ethiopia. The EPMES Contractor is responsible for all logistical support required for the evaluation team, including arranging translation, accommodation, security, office space, computers, internet access, printing, communication, and transportation and may hire additional temporary personnel to provide these functions as necessary. The evaluation team will officially report to the Ethiopia Monitoring and Evaluation Service (EPMES) Contractor, Social Impact. The EPMES Contractor is responsible for all direct coordination with the USAID/Ethiopia Program Office through the EPMES COR. From a technical management perspective, the evaluation team will work closely with the Health, HIV/AIDS, Population and Nutrition (HAPN) Office/Health Systems Strengthening Team, in particular with the Activity Manager seated in this office. In order to maintain objectivity, all final decisions about the evaluation will be made by the Program Office. XI. LOGISTICS The contractor will be responsible for all travel and logistics associated with conducting the evaluation. XII. REPORTING REQUIREMENTS AND DELIVERABLES 1. In-briefing: Within 48 hours of the availability of the evaluation team in the EPMES’s Contractor Office, the evaluation team will have an in-brief meeting with USAID/Ethiopia’s Program Office and the HAPN Office for introductions; presentation of the team’s understanding of the assignment; initial assumptions. Following this, the evaluation team shall develop evaluation work plan/evaluation design and make a presentation to USAID within five (5) working days of the initial in-brief. 2. Evaluation Work Plan: Within three working days following the in-brief presentation, the Evaluation Team Leader shall provide a detailed revised evaluation work plan to USAID/Ethiopia’s Program Office and the HAPN Office. The revised work plan will include: (a) the overall evaluation design, including the proposed methodology, data collection and analysis plan, and data collection instruments; (b) a list of the team members and their primary contact details while in￾country, including the e-mail address and mobile phone number for the Team Leader; and (c) the team’s proposed schedule for the evaluation. USAID offices and relevant stakeholders are asked to take up to two working days to review and consolidate comments through the EPMES COR. Once the evaluation team receives the consolidated comments on the revised work plan, they are expected to return with a final work plan within two working days. At this stage the evaluation team will be approved to implement the Work Plan, unless the EPMES COR raises a concern that needs resolution. 3. Interim Findings Meeting: After the fieldwork and preliminary data analysis, the Evaluation team will schedule a briefing with USAID to review the status of the evaluation’s progress, with a particular emphasis on addressing the evaluation’s questions and a brief update on potential challenges and emerging opportunities. The team will also provide the COR for EPMES and Activity Manager for Health Sector Finance Reform/Health Financing and Governance (HSFR/HFG) with periodic weekly briefings by phone with the Program Office and the HAPN Office to provide updates on field progress and any problems encountered. HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT | 40 4. Draft Evaluation Report: The content of the draft evaluation report is outlined in Annex A, and all formatting shall be consistent with the USAID branding guidelines. The focus of the report is to answer the evaluation questions and may include factors the team considers to have a bearing on the objectives of the evaluation. Any such factors can be included in the report only after consultation with USAID. USAID’s Program Office and the HAPN office will have ten working days in which to review and comment and the Program Office shall submit all comments to EPMES. The evaluation team will then have 10 working days to make appropriate edits and revisions to the draft and re-submit the revised final draft report to USAID. The HAPN Office and the Program Office will have 10 working days after the submission of the second revised draft to again review and send any final comments. 5. PowerPoint and Final Exit Presentation to USAID and relevant partners that will include a summary of key findings and key conclusions as these relate to the evaluation’s questions and recommendations to USAID. A final presentation from the evaluation team can be scheduled remotely for expatriate team members and presented by the contractor or an alternative team member. A copy of the PowerPoint file will be provided to the Program Office prior to the final exit presentation. 6. One-page briefer on key qualitative and quantitative findings and conclusions relative to the evaluation questions included in the evaluation’s scope so that readers can quickly review evaluation findings. Each briefer will be reviewed by the Program Office and the HAPN Office prior to distribution. The Final Evaluation Report will incorporate final comments provided by the Program Office. The length of the final evaluation report should not be more than 30 pages, not including Annexes and Executive Summary. All project data and records will be submitted in full and shall be in electronic form in easily readable format; organized and fully documented for use by those not fully familiar with the project or evaluation; and owned by USAID and made available to the public, barring rare exceptions, on the USAID Development Experience Clearinghouse (http://dec.usaid.gov). 41 | FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) Annex A: USAID Evaluation Report Standards 1. Identify the evaluation as either an impact or performance evaluation per the definitions in ADS 201. 2. Include an abstract of not more than 250 words briefly describing what was evaluated, evaluation questions, methods, and key findings or conclusions. The abstract should appear on its own page immediately after the evaluation report cover. 3. Include an Executive Summary 2–5 pages in length that summarizes key points (purpose and background, evaluation questions, methods, findings, and conclusions). 4. State the purpose of, audience for, and anticipated use(s) of the evaluation. 5. Describe the specific strategy, project, activity, or intervention to be evaluated including (if available) award numbers, award dates, funding levels, and implementing partners. 6. Provide brief background information. This should include country and/or sector context; specific problem or opportunity the intervention addresses; and the development hypothesis, theory of change, or simply how the intervention addresses the problem. 7. Identify a small number of evaluation questions. 8. In an impact evaluation, identify questions about measuring the change in specific outcomes attributable to a specific USAID intervention. 9. Describe the evaluation method(s) for data collection and analysis. 10. Describe limitations of the evaluation methodology. 11. In an impact evaluation, use specific experimental or quasi-experimental methods to answer impact evaluation questions. 12. Include evaluation findings and conclusions. 13. If recommendations are included, separate them from findings and conclusions. 14. Address all evaluation questions in the Statement of Work (SOW) or document approval by USAID for not addressing an evaluation question. 15. Include the following annexes: ● Evaluation SOW. If the SOW is revised, the evaluation report should include the updated SOW as an Annex rather than the original SOW. ● A description of evaluation methods (if not described in full in the main body of the evaluation report). ● All data collection and analysis tools used, such as questionnaires, checklists, survey instruments, and discussion guides. ● All sources of information—properly identified and listed. ● Any “statements of differences” regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team. ● Signed disclosures of conflicts of interest from evaluation team members. ● Abridged bios of the evaluation team members, including qualifications, experience, and role on the team. 16. Include enough information on the cover of the evaluation report so that a reader can HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT | 42 immediately understand that it is an evaluation and what was evaluated. The evaluation cover should: ● Include a title block in USAID light blue background color. ● Include the word “Evaluation” at the top of the title block and center the report title underneath that. The title should also include the word “evaluation.” ● Include the following statement across the bottom of the cover page: “This publication was produced at the request of the United States Agency for International Development. It was prepared independently by [list authors and organizations involved in the preparation of the report].” For an internal evaluation team, use the following statement: “This publication was produced at the request of [USAID/Mission] and prepared by an internal evaluation team comprised of [list authors and affiliation].” ● Feature one high-quality photograph representative of the project being evaluated and include a brief caption on the inside front cover describing the image with photographer credit. ● State the month and year of the report. ● State the individual authors of the report and identify evaluation Team Leader. Annex B: USAID Criteria for Quality Evaluation ● Evaluation reports should represent a thoughtful, well-researched, and well-organized effort to objectively evaluate the strategy, project, or activity. ● Evaluation reports should be readily understood and should identify key points clearly, distinctly, and succinctly. ● The Executive Summary should present a concise and accurate statement of the most critical elements of the report. ● Evaluation reports must address all evaluation questions included in the SOW, or the evaluation questions subsequently revised and documented in consultation and agreement with USAID. ● Evaluation methodology must be explained in detail and sources of information properly identified. ● Limitations to the evaluation must be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). ● Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or simply the compilation of people’s opinions. ● Findings and conclusions should be specific, concise, and supported by strong quantitative or qualitative evidence. ● If evaluation findings address person-level outcomes and impact, they should be assessed for both males and females. ● If recommendations are included, they should be supported by a specific set of findings and should be action-oriented, practical, and specific 43 | FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) ANNEX B: EVALUATION DESIGN MATRIX Evaluation criteria Indicators/Issues of Interest Data Collection Instruments Sources of Information Data Analysis Method 1. To what extent is the HSFR theory of change (TOC) and its objectives adequate, relevant, and viable? Relevance To what extent the HSFR/HFG design interventions continue to address a demonstrable need; is it responsive to the identified challenges and fit well with the new government health financing strategies and priorities? Is the structure and design of interventions still appropriate? How far the programming context and assumptions still relevant and valid? Review of documents/second ary data Project documents (TOC, program descriptions and its logic model); contextual information from government documents and other studies Comparison of the current facts and data with baseline (the facts during the design period) Key informant interviews FMOH and DP health financing specialists HSFR/HFG advisors 2. What progress has been made towards achieving the projects performance objectives by focusing on the 4 attributes? Relevance How important is the implementation of RRU to increase availability of operational budget at facility level? Are there other interventions more relevant and more appropriate than RRU? How important is the implementation of Private wings to improve the quality and timeliness of services and help retain and motivate health staff? HOW important are exemption services to meet community’s public health needs and address equity in access and does the implementation of exempted services comply with legal framework established? How Relevant is accreditation of facilities for improved quality of health services? Key Informant Interviews and Document/data Review Woreda/RHBs/FMoH managers HFs managers HSFR/Implementing partners Finance offices Does the introduction of CBHI reduce the community’s financial barriers to access health services? Are trainings provided, production and dissemination of mass media programs relevant to improve awareness of the public on health insurance? Do the current capacity building approach/ trainings provided benefited stakeholders (woreda cabinet, kebele cabinet, CBHI executive organs) to promote CBHI schemes? Do the CBHI communication interventions reach the target population? Are the benefit packages defined fits well to the health needs of the community? Document/utilizati on of health services review HSFR/HFG data base Analysis of utilization of health services between CBHI members and non-CBHI members KII EHIA, HSFR/HFG project staff, CBHI scheme Management staff FDGs HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT | 44 Evaluation criteria Indicators/Issues of Interest Data Collection Instruments Sources of Information Data Analysis Method Community, CBHI boards Analysis of produced communication materials by type and, region Does the establishment of Facility Governance Boards (FGB) enable facilities to have a decentralized and timely decision making about their priorities and on their own resources allocations? Does the community have a mechanism for influencing decision-making? KII, FGD Woreda/RHBs managers Analysis of capacity building activities by area, level (Head quarter, region); Data tabulation (frequency distributions & percept distributions) HFs managers Are the technical assistances provided to strengthen the institutional capacity of EHIA (in designing, implementation and monitoring and evaluation of health insurance) are demand driven and prioritized as per EHIA’s capacity gaps? KII Members of governance EHIA Data disaggregation Effectiveness % Facilities retaining and utilizing internally generated revenues Qualitative data analysis % of RRU expenditures on quality improving activities Descriptive data Number of PW established PW review studies EEA, FMOH Reviews hospital records Identification, examination, and interpretation of patterns and themes in textual data KII RHBs/ FMoH managers HFs managers Number of facilities graduated from the project support Review documents RHBs managers HFs managers KII HSFR/Implementing partner agents Number of public facilities accredited Review documents % of facilities operating as per the standards of accredited facilities and KII Number of facilities undertaking client satisfaction surveys Review documents, KII, Review of facility surveys RHBs managers % of facilities using the results of survey for prioritizing interventions HFs managers Number of Regional, Zonal and woreda cabinet members that received TOT on CBHI scale-up Review documents Number of Zonal Council or Steering committee members that received orientation on CBHI scale-up Number of Kebele Cabinet members trained on CBHI implementation in pilot expansion woredas Number of CBHI Executive staff trained on CBHI & M & E 45 | FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) Evaluation criteria Indicators/Issues of Interest Data Collection Instruments Sources of Information Data Analysis Method Number of regions implementing CBHI HSFR/HFG data base Data tabulation (frequency distributions & per cent distributions) by sex Descriptive data Data disaggregation Qualitative data analysis The identification, examination, and interpretation of patterns and themes in textual data Number of woredas that established CBHI schemes Number of schemes that started covering the cost of service provision/functioning CBHI schemes Number of household covered in CBHI schemes % of households overfed in CBHI Amount of Resources generated from paying members Review documents HSFR/HFG central & regional data base Amount of Resources generated from regions and woredas (targeted subsidy for the indigents) EHIA, HSFR project Amount Resources generated from federal MOH (general subsidy) CBHI management staff % of CBHI members whose premiums are paid by government (sex disaggregated) Review documents, HFSR/HFG documents RHBs/FMoH documents HFs documents Number of waiver beneficiaries from non CBHI woredas (sex disaggregated) Woreda level CBHI scheme documents Number and types of communication materials produced, Review of documents/Review of Mass media materials and products; Exit interview HSFR/HFG documents Number of Disseminations/ sensitization activities conducted HFG/HSFR staffs Media personnel Communities # of services that are legally part of exempted services Review of legal documents HFSR/HFG documents KII RHBs/FMoH managers HFs managers # of new services included in this category HSFR/Implementing partner agents HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT | 46 Evaluation criteria Indicators/Issues of Interest Data Collection Instruments Sources of Information Data Analysis Method % facilities that established facility governing boards (sex disaggregated) KII and FGDs RHBs/FMoH managers Data tabulation (frequency distributions & per cent distributions) by sex HFs managers Descriptive data HSFR/Implementing partner agents Data disaggregation Board members Qualitative data analysis % of functioning facility governing boards Communities The identification, examination, and interpretation of patterns and themes in textual data Number of health facility governing board and CBHI scheme networks established; Number of health facility governing board and CBHI Board of Directors scheme networks established; Number of people who participated in the CBHI review meetings organized at zonal (cluster) level in collaboration with zonal health department (ZHD), zonal administration and EHIA branch offices Number of People participated in the CBHI National Level Experience Sharing and Networking workshop Number of CBHI Inter-Regional Experience Sharing visits organized with project support for new CBHI initiating regions KII and projects reports KII with RHB, EHIA and HSFR/HFG performance reports Analysis of capacity building activities by area, level (Head quarter, region, woreda facility), etc. and also its relevance. Number of people participated in experience sharing and networking workshop at regional level; Number of people participated in experience sharing between existing and expansion CBHI woredas; Number of Regions conducted experience sharing visits in CBHI schemes; Number of CBHI schemes adopting medical audit manual; Number of CBHI schemes undertaking financial audit; Number of CBHI schemes that received in Kind Support Number of networking meetings/workshops conducted, KII with RHB and EHIA and HSFR/HFG performance reports Number of capacity building meetings/workshops for the networks conducted KII with RHB, EHIA and HSFR/HFG performance reports Number of CBHI Regional Steering Committees review meetings conducted KII with RHB, EHIA and HSFR/HFG performance reports 47 | FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) Evaluation criteria Indicators/Issues of Interest Data Collection Instruments Sources of Information Data Analysis Method Number of people participated in regional level experience sharing and networking workshop in CBHI implementation KII with RHB, EHIA and HSFR/HFG performance reports Number of people participated in CBHI bi-annual review meetings organized at zonal level KII with RHB, EHIA and HSFR/HFG performance reports Reported improvements in the quality of health services that are attributed to the strengthened networks Communities Number of capacity building trainings provided to EHIA KII and reports EHIA, HSFR project # of policies and strategies updated and revised KII and review of # of success stories documented and shared with stakeholders documentation Number of M&E tools/ICT-based systems, protocols, guidelines, manuals developed and utilized Document review. KII HSFR/HFG data base and project performance reports; RHBs/FMoH managers Number of supportive supervision programs designed and conducted. KII HFs managers Efficiency of program implementation Revenue collected by facilities per cost of TA Document review HSFR result and expenditure report Cost of accreditation process Document review HSFR result and expenditure report Average cost per CBHI establishment Document review HSFR result and expenditure report Total resources generated per cost of CBHI Document review HSFR result and expenditure report Average cost of CBHI enrollment Document review HSFR result and expenditure report Average cost per mass media session/sensitization Document review HSFR result and expenditure report Quantitative analysis supported by evidence from qualitative analysis Cost per functioning board Document review HSFR result and expenditure report Cost per strategies revised Document review HSFR result and expenditure report Cost per success stories documented Document review HSFR result and expenditure report Average cost per training Document review HSFR result and expenditure report Sustainability of program implementation How far can RRU lead to self-financing of facilities over the long term? Can the reform program be taken over fully by government without external TA? Document review HSFR/HFG documentation FMOH reviews and reports KII RHBs/FMoH managers Woreda HO heads HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT | 48 Evaluation criteria Indicators/Issues of Interest Data Collection Instruments Sources of Information Data Analysis Method HFs managers Woreda Administration HSFR/Implementing Are PWs the long-term solution for ensuring retention of health workers? Can the schemes management arrangement and financial generating capacity make sustainable? Is the financing of exempted services adequate not to compromise the gains made by RRU? Are the CBHI schemes financially sustainable? What can be done to make them more self-sustaining? How are they integrated into government structures and systems? Is CBHI schemes institutional arrangement sustainable? Can the government continue financing it? Document Review, KII HSFR/HFG documentation; FMOH reviews and reports; KII with EHIA, HSFR/HFG, and CBHI scheme Management staff Are ongoing insurance communicating methods and strategies sustainable (are they integrated and part of government activity) to create awareness and promote health insurance regularly? Can they continue without TA support? KII RHBs/FMOH managers; Woreda HO heads; Health Facility (HF) Managers, Woreda Administration; HSFR/Abt Has the TA and capacity building support provided to EHIA enabled it to lead and manage health insurance on its own? Is the training provided being supported by other capacity building measures? KII Can the management of facilities by boards continue without technical and financial support from the project? Can the regions take over the provision of TA and support? KII RHBs/FMOH managers; Woreda HO heads; HFs Managers, Woreda Administration; HSFR/Abt Can the government take over the supportive supervision and other monitoring functions if and when the TA support ends? KII RHBs/FMOH managers; Woreda HO heads; HFs Managers, Woreda Administration; HSFR/Abt How sustainable is the capacity building effort made by the TA to FMOH, RHBs, woredas and HFs? KII RHBs/FMOH managers; Woreda HO heads; HFs Managers, Woreda Administration; HSFR/Abt 49 | FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) ANNEX C: DATA SOURCES No. Source category Sources 1 Gov’t and other partners’ documents • GoE (Health financing strategy, review, HSTP) • FMOH documents • HSFR/HFG • Surveys • Other studies 2 Project and health facility database • HSFR/HFG database • Health facility documentation KII • Federal Level: FMOH (RMD, MSD, HSSSD), EHIA • Regional level: RHBs, BOFED, EHIA branches, HSFR/HFRG reg offices • Woreda level: WoHO, WOFED, Woreda Admin, CBHI Admin 4 FGDs • Health facility boards/management committees • CBHI boards • Communities Site visits • Regional hospital • District hospitals • Health centers 3 5 HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT | 50 ANNEX D: LIST OF PEOPLE CONSULTED List of People Interviewed at the Federal Level R./S Name Title Sex Remark 1 Dr. Mizan Kiros Director, Directorate for Resource Mobilization M FMOH 2 Mideksa Adugna HEFA Case Team Leader, Resource Mobilization Directorate M FMOH 3 Eyerusalem Animut Public Private Partnership Case Team Leader, Resource Mobilization Directorate FMOH 4 Belay Urgessa Resource Mobilization Directorate FMOH 5 Dr. Desalegn Clinical Service Directorate Director M FMOH 6 Dr. Sofonias Getachew WHO M WHO 7 Raphael Hurley Clinton Health Access International M CHAI 8 Bob Frayatt Director, HSFR/HFG Global activity M Abt Associates, USA 9 Luelseged Ageze Chief of party, HSFR/HFG activity M Abt Associates, Ethiopia 10 Hailu Zelelew HSFR/HFG activity M Abt Associates, USA 11 Zelelem Abebe HSFR/HFG activity M Abt Associates, Ethiopia 12 Tiliku Yeshanew HSFR/HFG activity M Abt Associates, Ethiopia 13 Habtamu Tadesse HSFR/HFG activity M Abt Associates, Ethiopia 14 Yenehun Tawye HSFR/HFG activity M Abt Associates, Ethiopia 15 Abduljelil Reshad Deputy Director M Ethiopian Health Insurance Agency List of People Interviewed in Amhara Region R./S Name Title Sex Remark 1 Bezuayehu Gashaw Deputy Bureau Head M RHB 2 Bayehe Atnafu HCF Focal Person M RHB 3 Tilahun Eshet Deputy Bureau Head M BOFED 4 Worku Gashaw Budgeting Director M BOFED 5 Genet Anteneh Regional Director M HSFR/HFG Project Office 6 Abay Akalu Deputy Regional Director and Senior Health Insurance Specialist M HSFR/HFG project office 7 Molalegn Tarekegn CEO M Felegehiwot Regional Hospital 8 Wasehun Walelegn CEO M Dangela Primary Hospital 9 Yehuala Desta Office Head M Dembecha WOFED 10 Yajeb Semachew Woreda CBHI Executive Coordinator M Dembecha CBHI scheme 11 Adugna Demissie Representative of the Head of the HC M Dembecha Woreda Health Center (HC) 12 Belay Endeshaw Acting Head M Dembecha Woreda Health Office 13 Amlaku Belay CEO of the Hospital M Finote Selam Hospital 14 Muluken Dagnachew CBHI Coordinator M Guagusa Shikudade CBHI Scheme 15 Getaneh Gashu Health Officer, Medical Auditor M Guagusa Shikudade CBHI scheme 16 Germiew Melese Zone Coordinator, CBI schemes M West Gojjam, 17 Miherete Kelemu Head of Gagusa Shekudad Woreda Health Office M Gagusa Shekudad Woreda Health Office 51 | FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) 18 Abeje Desse Health Center Head M Wanjela Health Centre 19 Amanuel Aynalem Head, Procurement and Finance Process M Gugusa Shekudad Woreda 20 Meseret Berhe Head, Guagusa Shekudad Woreda Finance Office M Gugusa Shekudad Woreda 21 W/o Debretu Ashagre CBHI Coordinator F Fogera Woreda CBHI scheme 22 Ato Asnake Tarekegn Representative of the Head of the HC and Head Nurse M Woreta Town Health Center 23 W/ro Yealemwork Wassie Procurement and Finance Process Owner Coordinator F Fogera Woreda Health Office 24 Worku Melesse Acting Head and malaria and other communicable diseases proves owners M Fogera Woreda Health Office 25 Adane Ambaye Curative Care Process Owner M Fogera Woreda Health Office 26 Aemiro Tenagne Head of the Woreda Health Office M Yilmana Densa Woreda Health Office 27 Jejaw Miheret Head of Health Center M Adet 28 Abraham Asnake Head of Providers and Member Registration M EHIA Branch office, Bahir Dar List of People Interviewed in Afar Region 1 Ato Osman Mequbul Osman Bureau Head M BOFED 2 W/ro Fatima Mohammed Director, Procurement and Disbursement F BOFED 3 Ato Abdu Musa Hussein Head WOFED and Member of the Meganta HC Board M Dubti 4 Hussein Mohamed Acting head and Curative and Rehabilitative Process Owner M Chifera Woreda Health Office 5 Hussein Mohamed Medical Service Head M RHB 6 Ahmed Mustafa Medical Service officer and HCF Focal Person M RHB 7 Ato Sultan Abdela Health Center Head M Chifera Health Center 8 Hussen Mohamed Head, Curative and Rehabilitative Core Process M Chifera Woreda Health Office 9 Anteneh Yirdaw Senior Finance Officer M Chifera Woreda Finance Office 10 Ato Nuru Mohamed CEO M Dalifagi District Hospital 11 Dr. Mohamud Mustefa Medical Director M Dalifagi District hospital 12 Ato Mussa Salehe Finance Officer M Dalifagi District hospital 13 Dr. Mahee Ali CEO M Dubti Regional Hospital 14 Ato Melaku Hagos Senior Finance Officer M Dubti woreda Health Office 15 Ato Mohamed Hassen Finance Officer M Dubti Woreda Health Office 16 Elsa Abdu Representative of the Head of the Health Office F Dubti Woreda Health Office 17 Girum Amene Health Officer – Prevention M 18 Keder Abate Head, Megenta Health Center M Megenta Health Center Head, Dubti Woreda List of People Interviewed in Addis Ababa 1 Fistume Haile Addis Ketema Woreda 04 Executive Head M Addis Ketema Woreda 04 2 Abebe Tsegaye Kirkose Woreda 09 Executive Head M Kirkose Woreda 09 3 Getachew Worku HSFR/HFG project, Oromia Branch M Addis Ababa 4 Masay Bela Manager EHIA, Addis Ababa branch M EHIA, Addis Ababa branch HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT | 52 5 Fekadu Worku Medical Director, Addis Katema Health Center M Addis Katema Health Center 6 Endale Bezu Tadesse Medical Director, Meshualekia Health Center M Meshualekia Health Center 7 Nega Kidanu CEO, Ras Desta Hospital M Ras Desta Hospital 8 Abrehet Ylak Giwot Head Woreda 4 Health Office F Woreda 4 health office 9 Faisal Safawo Deputy Head Oromia RHB M Addis Ababa 10 Yenenesh Engida Members Affairs and contributions F FinFin Branch Oromia 11 Tewodros Tekle Addis Ababa Head of BOFED M Addis Ababa BOFED 12 Alemanchu Adane Head of Woreda Finance and Economic Development Office M Addis Ketema sub city woreda 04 13 Genet Gebre Yohannes Head of Woreda Finance Economic Development Office F Kirkose sub city finance office woreda 09 List of People Interviewed in Oromia Region 1 Fejiso Babsa Head, Boset Woreda Administration M Boset Woreda 2 Jiru Chummier CBHI Coordinator M Gedeb Asasa Woreda 3 Abush Arage Gimbichu Woreda Administrator M Gimbichu Woreda Administration 4 Kemal Tolla GedebAsasa Woreda Administrator M GedebAsasa Woreda Admin 5 Mohammed Eboo GedebAsasa Woreda Health Office M GedebAsasa Woreda 6 Diriba Degefa Adama Town Health Office M Adama Town 7 Kemal Kasso Head of Woreda health Office M BosetWoreda Health office 8 Teketele Kebede CBHI Coordinator M GimbichuWoreda CBHI scheme 9 Debissa Beyene Head of Woreda Health Office M GimbichuWoreda Health Office 10 Tesfaye Mulugeta CBHI Coordinator M Boset Woreda CBHI scheme 11 Abay Solomon PHCU Director M Chefedonsa Health Center 12 Sister Genet Ibrahim Head of Geda Health Center F Geda Health center 13 Haji Sheka Gedaeb Assasa Health center M Gedaeb Assasa Health Center 14 Fedadu Aduga Adama Head of Finance and Economic Development Office M Adama, WoFED 15 Shebeshe Kefalew Planning and Budget Expert M Adama 16 Gebi Hussien Auditor M Adama 17 Kelele Aciyu Head of Bosat Woreda Finance and Economic Development Office M Boset Woreda 18 Getu Roba Head of Gimbicho Woreda Finance and Economic Cooperation Office M Gimbicho Woreda 19 Mhidin Tufaa CEO, Dodolla Hospital M Dodolla 20 Kedir Serkio Gedebe Assasa Woreda Finance Office Accountant M Gedebe Assasa 21 Hedeto Hasanaa CEO, Shashemene Hospital M Shashemene 22 Gindo Lemma CEO, Welnchti Hospital M Welnchti Hospital 23 Dr. Eyob Mamo Welnchti Hospital Medical Director M Welnchti Hospital List of People Interviewed in SNNPR 1 Tsegaye Geneto Damboya Woreda Head, Finance and Economic Development Office M Damboya Woreda 2 Mihret Arega Yirgalem Town Health Office Head F Yirgalem Town 3 Asemelelashe Kebede Head of woreda Finance, Economic and Development Office M Yirgalem Town 4 Asefa Dansamo SNNPR, EHIA Awassa Branch Manager M Awassa 5 Yonas Petros Damboya Head of Health Center M Damboya Health center 53 | FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) 6 Teshale Yohannes Damboya Woreda Chief Administrator M Damboya, KembataTembaro Zone of SNNPR 7 Bederu Botoko CBHI Coordinator M Damboya Woreda CBHI scheme 8 Tamene Head of Woreda Health Office M Damboya Woreda Health Office 9 Yohannes Letamo Hujawa SNNPR, RHB Medical Services Core Process Owner M SNNPR RHB 10 Sarmiso Samuel Yirgalem Town Mayor M Yirgalem 11 Aschalew Lidetu CBHI Coordinator M Yirgalem 12 Ashenafi Wagiso SNNPR HSFR/HFG Project Director M SNNPR HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT | 54 Focus Group Discussions Facility Governance Boards – CBHI Participants of Facility Board Focus Group Discussions Amhara Region Name Name of Facility Woreda Yilkal Dagne Demebecha HC Demebecha Eshete Mekinen Demebecha HC Demebecha Lake Alemeneh Demebecha HC Demebecha Adugan Demissie Demebecha HC Demebecha Mantiwab Sahilu Demebecha HC Demebecha Mergeta Bekalu Meweye HC Yilman Densa Nega Yigzaw Meweye HC Yilman Densa Jejaw Miheret Meweye HC Yilman Densa Ato Kendealem Chane Woreta Town HC Fogera Sr. Medferiash Haileselasie Woreta Town HC Fogera Ato Asnake Tarekegn Woreta Town HC Fogera Adamu Negatu Wanjela HC Gagusa Shekudad Habtamu Abera Wanjela HC Gagusa Shekudad Gebeyaw Wanjela HC Gagusa Shekudad Abeje Desse Wanjela HC Gagusa Shekudad Oromia Region Name Name of Facility Woreda Kemal Tolla Gedeb Asasa HC Gedeb Asasa Sara Hajor Gedeb Asasa hC Gedeb Asasa Kedire Ebrahim Gedeb Asasa hC Gedeb Asasa Mohammed Hebbo Gedeb Asasa hC Gedeb Asasa Sofia Kedir Gedeb Asasa hC Gedeb Asasa Feyiso Dube Gedeb Asasa hC Gedeb Asasa Abay Solomon Chefe donsa HC Gimbichu Debisa Beyene Chefe donsa HC Gimbichu Abush Arage Chefe donsa HC Gimbichu Getu Roba Chefe donsa HC Gimbichu Roman Dawe Chefe donsa HC Gimbichu Teketel Kebede Chefe donsa HC Gimbichu Asres Gebisa Chefe donsa HC Gimbichu Zewdu Gemechu Chefe donsa HC Gimbichu Fejiso Babssa Wolenchiti HC Boset Mekonnen Assefa Wolenchiti HC Boset 55 | FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) Kemal Kasso Wolenchiti HC Boset Kidde H/mariam Wolenchiti HC Boset Tesfaye Mulugeta Wolenchiti HC Boset Zeine Mohammed Wolenchiti HC Boset Sonan Tesfaye Wolenchiti HC Boset Keril Aliye Wolenchiti HC Boset Bayyane Zrane Wolenchiti HC Boset Zaqiri Muhammed Wolenchiti HC Boset SNNPR Name Name of Facility Woreda Teshale Yohannis Damboya HC Damboya Yonnas Petros Damboya HC Damboya Tamene Tesfaye Damboya HC Damboya Solomon G/Meskel Damboya HC Damboya Bediru Butako Damboya HC Damboya Aberash Abo Damboya HC Damboya Tegaye Geneto Damboya HC Damboya Sermiso Samuel Yirgalem HC Yirgalem Town Asmelash Kebede Yirgalem HC Yirgalem Town Mihret Arega Yirgalem HC Yirgalem Town Yantu Simion Yirgalem HC Yirgalem Town Aschalew Lidetu Yirgalem HC Yirgalem Town Tsegaye Gatibu Yirgalem HC Yirgalem Town Tigist Wororo Yirgalem HC Yirgalem Town Addis Ababa Region (Not Applicable – CBHI not yet implemented in Addis Ababa region) Participants of Facility Board Focus Group Discussions Amhara Region Name Name of Facility Woreda Yilkal Dagne Demebecha HC Demebecha Eshete Mekinen Demebecha HC Demebecha Lake Alemeneh Demebecha HC Demebecha Adugan Demiisie Demebecha HC Demebecha Mantiwab Sahilu Demebecha HC Demebecha Mergeta Bekalu Meweye HC Yilman Densa Nega Yigzaw Meweye HC Yilman Densa Jejaw Miheret Meweye HC Yilman Densa Ato Kendealem Chane Woreta Town HC Fogera Sr. Medferiash Haileselasie Woreta Town HC Fogera Ato Asnake Tarekegn Woreta Town HC Fogera HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT | 56 Adamu Negatu Wanjela HC Gagusa Shekudad Habtamu Abera Wanjela HC Gagusa Shekudad Gebeyaw Wanjela HC Gagusa Shekudad Abeje Desse Wanjela HC Gagusa Shekudad Oromia Region Name Name of Facility Woreda Kemal Tolla Gedeb Asasa HC Gedeb Asasa Sara Hajor Gedeb Asasa hC Gedeb Asasa Kedire Ebrahim Gedeb Asasa hC Gedeb Asasa Mohammed Hebbo Gedeb Asasa hC Gedeb Asasa Sofia Kedir Gedeb Asasa hC Gedeb Asasa Abay Solomon Chefe Donsa HC Gimbichu Debisa Beyene Chefe Donsa HC Gimbichu Abush Arage Chefe Donsa HC Gimbichu Getu Roba Chefe Donsa HC Gimbichu Roman Dawe Chefe Donsa HC Gimbichu Teketel Kebede Chefe Donsa HC Gimbichu Asres Gebisa Chefe Donsa HC Gimbichu Zewdu Gemechu Chefe Donsa HC Gimbichu Fejiso Babssa Wolenchiti HC Boset Mekonnen Assefa Wolenchiti HC Boset Kemal Kasso Wolenchiti HC Boset Kidde H/mariam Wolenchiti HC Boset Tesfaye Mulugeta Wolenchiti HC Boset Zeine Mohammed Wolenchiti HC Boset Sonan Tesfaye Wolenchiti HC Boset Keril Aliye Wolenchiti HC Boset Bayyane Zrane Wolenchiti HC Boset Zaqiri Muhammed Wolenchiti HC Boset SNNPR Name Name of Facility Woreda Teshale Yohannis Damboya HC Damboya Yonnas Petros Damboya HC Damboya Tamene Tesfaye Damboya HC Damboya Solomon G/Meskel Damboya HC Damboya Bediru Butako Damboya HC Damboya Aberash Abo Damboya HC Damboya Tegaye Geneto Damboya HC Damboya Sermiso Samuel Yirgalem HC Yirgalem Town Asmelash Kebede Yirgalem HC Yirgalem Town Mihret Arega Yirgalem HC Yirgalem Town 57 | FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) Yantu Simion Yirgalem HC Yirgalem Town Aschalew Lidetu Yirgalem HC Yirgalem Town Tsegaye Gatibu Yirgalem HC Yirgalem Town Tigist Wororo Yirgalem HC Yirgalem Town Teshale Yohannis Damboya HC Damboya Addis Ababa Region Name Name of Facility Woreda Tsehay Mulugeta Addis Ketema HC Woreda 4 Takelu Woldie Addis Ketema HC Woreda 4 Abrehet Yishak (S/r) Addis Ketema HC Woreda 4 Dereje Bekele Addis Ketema HC Woreda 4 Alemayehu Adane Addis Ketema HC Woreda 4 Fitsum Haile Addis Ketema HC Woreda 4 Fikadu Worku Addis Ketema HC Woreda 4 Endaw Bizu Meshualekia HC Woreda 9 Genet G/yohannis Meshualekia HC Woreda 9 Abebe Tsegaye Meshualekia HC Woreda 9 Yohannis G/Michael Meshualekia HC Woreda 9 Tolfe Awoke Meshualekia HC Woreda 9 Community Focus Group Discussions Amhara Region Name Community’s Name Woreda Gebru Ayele Abuwa-Kokit Fogera Negatu Teshome Abuwa-Kokit Fogera Awoke Ejigu Abuwa-Kokit Fogera Zelalem Sendeke Abuwa-Kokit Fogera Mebratu Adigo Abuwa-Kokit Fogera Teshome Admassu Abuwa-Kokit Fogera Haymanot Abebe Abuwa-Kokit Fogera Zenebe Mare Abuwa-Kokit Fogera Alemu Bogale Askuna Agza Gagusa Shekudad Kefale Ayene Askuna Agza Gagusa Shekudad Tsengaye Shiferaw Askuna Agza Gagusa Shekudad Zewde Bogale Askuna Agza Gagusa Shekudad Kebbe Alemu Askuna Agza Gagusa Shekudad Yesalemu Dememme Askuna Agza Gagusa Shekudad Fentanesh Shiferaw Askuna Agza Gagusa Shekudad Zeyene Kahenne Askuna Agza Gagusa Shekudad Tiruneh Taye Askuna Agza Gagusa Shekudad Zenebu Weddu Askuna Agza Gagusa Shekudad HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT | 58 Asmamaw Yitawe Mehel Dembecha Andualem Azale Mehel Dembecha Gubaye Azale Mehel Dembecha Amera Sineshaw Mehel Dembecha Tirfe Walle Mehel Dembecha Zelalem Taye Mehel Dembecha Bedelu Azale Mehel Dembecha Woubante Aderaw Mehel Dembecha Channe Azale Mehel Dembecha Workie Alemneh Mehel Dembecha Gedefe Desalegn Mehel Dembecha Mulusew Akalu Mehel Dembecha Mulinesh Flate Debre Meawi Yilmna Densa Aynalem Mengistu Debre Meawi Yilmna Densa Workinesh Shibabaw Debre Meawi Yilmna Densa Bekalu Adamu Debre Meawi Yilmna Densa Gorebet Ayenew Debre Meawi Yilmna Densa Emunesh Musche Debre Meawi Yilmna Densa Mengistu Mucheye Debre Meawi Yilmna Densa Yetemare Yismaw Debre Meawi Yilmna Densa Abeje Abebe Debre Meawi Yilmna Densa Awoke Menge Debre Meawi Yilmna Densa Tewilign Chale Debre Meawi Yilmna Densa Oromia Region Name Community’s Name Woreda Meka Ismael Hanto Kebele Gedeb Asasa Halima Kedire Hanto Kebele Gedeb Asasa Kedija Bedaso Hanto Kebele Gedeb Asasa Fatuma Camola Hanto Kebele Gedeb Asasa Mohammed Sado Hanto Kebele Gedeb Asasa Shambel Haji Hanto Kebele Gedeb Asasa Misiraa Jima Hanto Kebele Gedeb Asasa Medina Yahaya Hanto Kebele Gedeb Asasa Rehima Dule Hanto Kebele Gedeb Asasa Zeineaba Adem Hanto Kebele Gedeb Asasa Mengistu Abebe Adadi Gole Gimbichu Workitu Demu Adadi Gole Gimbichu Azalu Maru Adadi Gole Gimbichu Berike Wolde Adadi Gole Gimbichu Lomi Kassa Adadi Gole Gimbichu Beletu Hatene Adadi Gole Gimbichu 59 | FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) Zenebeche Deribew Adadi Gole Gimbichu Woyneshet Zegeye 02 Kebele Boset Geteneshe Eshete 02 Kebele Boset Tegi Yirgu 02 Kebele Boset Alemi Haile 02 Kebele Boset Teej Ami 02 Kebele Boset Dumbeshe Gudeta 02 Kebele Boset Leika Burena 02 Kebele Boset Askale Demo 02 Kebele Boset SNNPR Name Community’s Name Woreda Adanceh Aynuru Yebu Damboya Amanche Youhannes Yebu Damboya Bexeleche Jama Yebu Damboya Rohel Dawit Yebu Damboya Etapem Abose Yebu Damboya Kebebushe Eyuale Yebu Damboya Mulu Addis Yebu Damboya Bogaleche Legesse Yebu Damboya Abebeche Boltano Yebu Damboya Dembel Semerdein Yebu Damboya Barte Ahmed Yebu Damboya Abebayehu Achiso Yebu Damboya Kebebush Wariche Yebu Damboya Bertukan Solomon Mehal Ketema Yirgalem Addise Mengistu Mehal Ketema Yirgalem Meseleche H/mariam Mehal Ketema Yirgalem Tilahune Beshir Mehal Ketema Yirgalem Mulu Assefa Mehal Ketema Yirgalem Abebe Setegn Mehal Ketema Yirgalem Amare G/tsadik Mehal Ketema Yirgalem Afar Region Name Community’s Name Woreda Asefa Abadru Megenta Dupti Hamu Mehommed Megenta Dupti Meriem Awol Megenta Dupti Kassahun Siraw Megenta Dupti Belete Damite Megenta Dupti Ager Mohammed Megenta Dupti Akmed Issie Megenta Dupti Aminat Mohammed Megenta Dupti HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT | 60 Addis Ababa Region Name Community’s Name Woreda Abrehet Kebede Meshualekia 09 Nure Indo Meshualekia 09 Seble Zerihun Meshualekia 09 Nigatwa Mekonnen Meshualekia 09 Abezu Wubetu Meshualekia 09 Hawa Mohammed Meshualekia 09 Alemtsehay Fetene Addis Ketema 04 Zergi Belayneh Addis Ketema 04 Almaz Tiruneh Addis Ketema 04 Tsehay Mulugeta Addis Ketema 04 Hirut Bekele Addis Ketema 04 Aynalem Zeleke Addis Ketema 04 61 | FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) ANNEX E: DATA COLLECTION SCHEDULE Sub-team 1 (Amhara, Afar, Tigray) Abebe Alebachew, Workie Mitiku, Dereje Getahun Date Morning I Morning II Afternoon I Afternoon II 4/2/2017 Arrival (Air Travel) at Bahir Dar Meeting with Regional HSFR Director 4/3/2017 Meeting with Regional HSFR Team EHIA BOFED 4/4/2017 RHB Regional hospital (Feleghiowt) 4/5/2017 District hospital (Dangela) Guagussa shikudad Woreda health office (tilili) KII with Guagussa Shikudad woreda finance office 4/6/2017 Wonjela HC visit (direct observation)- near Tililli town, and HC Governing Board FDG with the Community FGD with CBHI board - Tilili KII with Guagussa Shikudad CBHI scheme 4/7/2017 KII with Denbecha Woreda health office and Woreda Administration KII with Denbecha woreda finance office FGD with CBHI board –Denbecha FGD with Health facility governance 4/8/2017 Denbecha HC visit (direct observation), and FGD community KII with Denbecha CBHI scheme Travel to Markos 4/9/2017 Weekend 4/10/2017 KII with Sinan Woreda health office and Woreda Administration KII with Sinan woreda finance office, and FGD with Health facility governance 4/11/2017 Yetsed HC visit (direct observation), and FGD community Travel to Bahir Dar Visit Feneto selam district hospital 4/12/2017 KII with Fogera Woreda health office and Woreda Administration KII with Fogera woreda finance office FGD with CBHI board –Fogera FGD with Health facility governance 4/13/2017 Woreta HC visit (direct observation), and FGD community KII with Fogera CBHI scheme Travel to Bahir Dar 4/14/2017 Travel to Addis (Air Travel from Bahir Dar) 4/15/2017 Weekend 4/16/2017 Weekend 4/17/2017 National Level Interview 4/18/2017 National Level Interview 4/19/2017 National Level Interview HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT | 62 Date Morning I Morning II Afternoon I Afternoon II 4/20/2017 Travel (Air) to Mekele, and KII Tigray RHD Regional HSFR and Travel (Air) to Addis 4/21/2017 National Level Interview 4/22/2017 National Level Interview 4/23/2017 Travel (Air) to Semera Afar 4/24/2017 KII with HSFR/HFG satellite office KII with RHB BOFED, Dubit WHO, Woreda Administration Dubti WoFED 4/25/2017 Megenta HC Megenta HC facility Board FGD community Dubti Regional Hospital 4/26/2017 Travel to Chiefra and Chiefra Woreda Health office Woreda Finace Chefra, Chifera HC visit 4/27/2017 Chifera Facility FGD FGD community 4/28/2017 Travel to Addis by Air from Kombolcha Sub-team 2 (Oromia, Addis Ababa, SNNPR) John Osika, Esubalew Demissie, Nigusu Aboset Dates Morning I Morning II Afternoon I Afternoon II March 31, FRI KII with HSFR/HFG Oromia office (Getachew) April 2 SUN Travel to Shashemene April 3 MON KII with Gedeb Assasa Woreda health office Assassa HC visit/ KII with Gedeb Assassa Woreda finance office KII with CBHI scheme in Gedeb Assasa woreda Shashemene regional hospital visit (direct observation) April 4 TUES FGD with CBHI board - Assasa FGD with community- Health Development Army (HDA) leaders Dodolla district hospital visit (direct observation) FGD with Gedeb Assassa health facility governance April 5 WED Travel to Damboya KII with Damboya Woreda Health office; Damboya HC visit KII with CBHI Scheme KII with Danboya Woreda Finance April 6 THU FGD with CBHI board in Damboya/FGD with health facility board of Damboya FGD with community –HDA leaders Travel to Yirgalem April 7 FRI KII with Yirgalem city health office/yigralem HC visit KII with CBHI scheme / FGD with CBHI board KII with Yigralem Woreda Finance KII with Yigralme city administration April 8 SAT FGD with Yirgalem health facility board FGD with community – HDA leaders 63 | FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) Dates Morning I Morning II Afternoon I Afternoon II April 9 SUN WEEKEND April 10 MON KII with HSFR/HFG Staff KII with SNNPR RHB April 11 TUE Travel to Adama KII with Adama Woreda health office FGD with facility governance April 12 WED Geda HC visit (direct observation)/KII with Adama Woreda finance office FGD with Geda facility governance FGD with community – HDA leaders KII with boset Woreda health office /wolenchitic HC visit April 13 THU Wolenchiti district hospital visit (direct observation)/KII with Boset Woreda finance office FGD with CBHI board of Wolenchiti FGD with facility governance FGD with community – HDA leaders April 14 FRI Travel to Addis Ababa April 20 TH KII with EHIA branch office of Oromia region KII with A/Ababa HSFR/HFG focal person - Wondwosen KII with EHIA branch office of Addis Ababa city administration April 21 FRI KII with Oromia RHB/KII with Addis Ababa RHB KII with Oromia BoFED/ KII with Addis Ababa BoFED Ras Desta Regional Hospital visit (direct observation) St. Paul federal hospital visit (direct observation) April 25 TU KII with kirkose, Woreda 9 Health office Meshualekia HC visit (direct observation) KII with kirkose, Woreda 9 finance office/ FGD with community –HDA leaders FGD with kirkose, woreda 9 facility governance April 26 WED KII with Addis Ketema, Woreda 3 health office /Woreda 3 health center visit (direct observation KII with Addis Ketema, Woreda 3 finance office FGD with Addis ketema, Woreda 3 facility governance FGD with community – HDA leaders April 27 TH Travel to Bishoftu KII with Gimbichu Woreda health office Chefe Donsa HC visit (direct observation) FGD with Community -HAD leaders April 28 FRI FGD with Facility governance KII with CBHI scheme in Gimbichu FGD with CBHI baord -Chefe donsa KII with Gimbichu Woreda finance office/admin office April 29 SAT Travel to Addis Ababa HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT | 64 ANNEX F: INTERVIEW PROTOCOLS AND GUIDES Annex F.1: Key Informant Interviews Instructions: At the beginning of each interview with a participant, please introduce your team and explain the purpose, confidentiality and ethics context of the evaluation as stated below. Introduction: Good morning/afternoon! Thank you for sparing your time today to speak with us. The team here is working to conduct a midterm evaluation of the HSFR/HFG project. The purpose of this project evaluation is to provide an informed assessment of progress to date, and recommendations for the remaining period of the project and any future follow-on project. Our team has had the opportunity to review project documents to get a sense of the design and implementation of the project. However, review of the documents alone is not enough. Therefore, we would like to speak with you today to hear about your experience, in your own words, in order to help us better understand how this project is working. Confidentiality: ● The information that we will collect will include individuals’ names, organizations, and positions. The annex of the evaluation report will include a list of key informants, but the findings or statements in the report will not be associated to any particular name of the key informant. ● Quotes from respondents will be included in the evaluation report, but there will be no link between the quotes and the names of the individuals who provided the quote. In the event that the team desires to use any personally identifiable information in the report (such as a photograph of the person), the evaluators will first contact the respondent(s) to seek permission to do so. ● The information that we shall collect during this evaluation will be used for the sole purposes of this evaluation. This information will not be used for any other purpose. ● Your participation in this interview is voluntary. If you do not feel comfortable answering any particular question, please let us know and we will simply go on to the next question. ● Thank you once again, for taking the time to speak with us today. If you have any questions for us, you can ask now, before we get started. 65 | FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) Annex 1.1 Key Informant Interview Guide – FMOH-RM Directorate Position of Interviewee: Location: Date: 1. Who are the major players in supporting health financing reform? (Probe for: role of the HSFR/HFG project and each of the other players in this effort; What resources what is the government of Ethiopia is investing in health financing reform) 2. What are the achievements of health financing reform over the last three years? Probe the following? (EFFECTIVENESS and get examples and explanations)) ● Revenue retention and utilization ● Fee waivers ● Fee revision ● Exemptions ● Governing boards ● Outsourcing ● Private wing ● The accreditation system ● Networks of insurance schemes and FGBs and their capacity ● Community based health insurance scheme ● Social Health Insurance ● Policy and strategy Updates 3. What are the efforts being made to institutionalize and foster ownership of the above health financing reforms at all levels ● At federal ● Regional ● Woreda levels Please explain (SUSTAINABILITY) 4. What are the success factors on achievements made in the above components? If yes, please describe them for us?(EFFECTIVENESS) 5. Are there components of the reform program (see above) that has not been able to achieve the set goals and targets, such as CBHI/SHI-coverage rate of 20%? If so, what are the main challenges?(EFFECTIVESNESS) Probe for: a) Health sector constraints (e.g., commitment, policy gaps, capacity, staff turnover, institutional) b) Constraints outside the health sector? 6. How far do you think, government has realized major assumptions of HSFR project design? If not, why?(TOC) (Notes for the team: increased government ownership and leadership of the health financing reform program (e.g. Operationalize the EHIA by opening 24 branch offices across the country, HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT | 66 strengthening of the EHIA and CBHI executive organs, MoFED/MOFEC, the FMOH, and regional health bureaus (RHBs) to allocate budgets for indigents and general subsidy) 7. Has the reform contributed to their intended outcomes? (EFFECTIVENESS)Probe for (details of in which way? How? At levels below) a) Improved quality of health services at facility levels (increased resourcing, better retention of human resources, increasing focus and efficiency) b) Enhance access to health (reduce financial barriers, protecting the indigent) at point of use and during seeking care c) Improved accountability (functioning oversight mechanisms, community say on service delivery) at facility, and woreda levels? d) Improved evidence generation and learning at regional and federal levels? 8. How are you coordinating the different development partners working on health financing initiatives? Probe: Is the project support showing duplications, inefficiency or complementarily to other initiatives? If yes, what is the FMOH doing to reduce this? What is the FMOH sustainability plan for gains from partner support? (SUSTAINABLITY) 9. How has the project TA performed in strengthening your capacity? (RELEVANCE)Probe for: a) How do you see the relevance (demand driven nature-consistent with the FMOH needs of the sector at all levels), effectiveness (getting the right TAs support at the right time) and sustainability of the technical support? b) What were the focus of TA support –development of tools, skills, staff and infrastructure, as well as structures, systems and roles? Which components of the investment of capacity building (individual, organizational and enabling environment levels) were effective? c) What are the major successes and challenges? What do think the HFG/HSFR team should change or improve to provide responsive and effective TA for capacity building for the remaining period of the project and beyond? d) What are the Strengths (relevance, appropriateness, timeliness, supporting the government own plan)? e) What are the Weaknesses (relevance appropriateness, timeliness, supporting the government own plan) are observed in the TA provisions; f) What Suggestions (including another approach for the provision of the technical assistance?) do you have to further improve HCF performance? 10. Do you think the interventions of the project being supported continue to be relevant to the context and adequate to meet the health financing reform objectives of the country? (Given the recent development in the health financing reform landscape -HCF strategy, Universal Access, HSTP, envisioning document; potential fatigue of support by USAID after financing some of the interventions for over 15 years)? If not (SUSTAINABLITY) a) Which reform areas should be phased out from project support? b) Which reform should be adjusted? c) What new initiatives should be included to supportive you better? 11. What do you think the focus of the future Health financing reform should be? (Given the new HCF 67 | FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) strategies and Ethiopian desire to move towards PHC led UHC)? (SUSTAINABLITY)Probe a) What does the current support program need to focus on and support in its remaining period? b) What should be the focus of the new USAID TA support in the near future? Any recommendation on nature and composition of the most appropriate HSFR interventions? c) What should USAID and other partners do to support HCF strategy implementation? d) What should the government of Ethiopia commit to make the health financing reform successful and attract more investment by partners as well as sustain the gains made so far? 12. Is there anything we missed that you would like to talk about? Annex 1.2: Key Informant Interview Guide – FMOH-Medical Service Directorate Interviewee (Title): Location: Date: 1. Please give us a brief overview of what you do to improve quality of care at hospital and health center levels? (INTRODUCTION) 2. What are the areas that you are working together with HSFR/HFG project? (TOC) 3. Are there any initiatives by the Ministry or directorate regarding institutionalization and ownership of health financing reforms at all levels (federal to Woreda)? Please explain (SUSTAINABLITY) 4. What do you think are the major effects of health care financing reform in improving quality of care? (EEFECTIVENESS)Probe for examples and explanations on the effects of: ● Revenue retention and utilization ● Fee waivers ● Fee revision ● Exemptions ● Governing boards ● Outsourcing ● Private wing ● Community based health insurance scheme ● Social health insurance ● Policy and strategy Updates 5. How is quality of care being monitored at facility levels? (EFFECTIVENESS)Probe for: a) The role of facility boards, b) CBHI boards and communities c) Client satisfaction surveys? d) Regional, woreda level support and monitoring e) FMHACA 6. What efforts have been made to prepare health facilities to meet the preconditions of CBHI/SHI HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT | 68 enrollment? (EFFECTIVENESS) 7. In your opinion, how has increasing CBHI enrollment affected health facilities services provision? (RELEVANCE) 8. What do you think should be done to ensure facilities receive resources for the services they render (note for the team members: (Probe: Under financing of health facilities (inadequate operational budget; Major areas concern in improving quality. Fee waiver beneficiaries further undermine this, exempted services provided without third party payer etc.)? (SUSTAINABLITY) 9. Please elaborate for us the types of TA that you are receiving from the HSFR/HFG project? (Probe for (seconded staff, tools/guideline development, etc.;). In what ways has the technical assistance from HSFR supported the FMOH with strengthening capacity for health sector finance reform? (RELEVANCE)Probe for: a. How do you see the relevance (demand driven nature), effectiveness (getting the right TAs support at the right time) and sustainability of the technical support? b. What were the focus of TA support –development of tools, skills, staff and infrastructure, as well as structures, systems and roles? Which components of the investment of capacity building (individual, organizational and enabling environment levels) were effective? c. What do think the HFG/HSFR team should change or improve to provide responsive and effective TA for capacity building for the remaining period of the project and beyond? d. What are the Strengths (relevance, appropriateness, timeliness, supporting the government own plan) e. What Weaknesses (relevance appropriateness, timeliness, supporting the government own plan) are observed?; what do you think the major reasons for these weaknesses are? f. What Suggestions (including another approach for the provision of the technical assistance?) do you have to further improve performance? 10. Do you think the current interventions of the HSFR project continue to be relevant in the short and medium term? (SUSTAINABLITY); Probe: a) If yes, which one of the reform components should continue as is? What are the reasons for continuing these reforms? b) If no, which components need to be phased out from project support? What are the reasons for phasing out these reforms from the project support? c) Which one of the reform components should be adjusted? What are the reasons for adjustment? d) What new initiatives should be included to supportive you better? 11. What do you think the focus of the future Health financing reform should be to improve quality of care?:(SUSTAINABILITY) a) The current support program need to focus on and support in its remaining period? 69 | FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) b) What should be the focus of the new USAID TA support in the near future? Any recommendation on nature and composition of the most appropriate interventions? c) What should USAID and other partners do to support HCF strategy related interventions that promote quality of care? Annex1.3: Key Informant Interview Guide – FMOH-Health System Special Support Directorate Interviewee (Title):_____________________ Location: ________________________________ Date: ___________________________________ 1. What do you think are the major achievements in the health care financing reform?? (10-15 mins) (EFFECTIVENESS: Probe for examples and explanations on the following) ● Revenue retention and utilization ● Fee waivers ● Fee revision ● Exemptions ● Governing boards ● The accreditation system ● Networks of insurance schemes and FGBs and their capacity Outsourcing ● Private wing ● Community based health insurance scheme ● Health insurance ● Policy and strategy Updates 2. What do you think are the major reasons for Developing Regional States low Health Care Financing performance or their lagging as compared to other regions?(TOC) (Probe: What should be done to improve HCF performance in these developing regional states?) (5-10 mins) 3. Do you think the current interventions of the HSFR project continue to be relevant in the short and medium term? (SUSTAINABLITY); Probe: e) If yes, which one/s of the reform components should continue as is? What are the reasons for continuing the support? f) If no, which components need to be phased out from project support? What are the reasons for phasing out the support? g) Which one/s if any, of the reform components should be adjusted? Why? h) What new initiatives should be included to support you better? 4. What should Government of Ethiopia, and USAID do to support HCF strategy related interventions to bring Developing Region States to similar levels with other regions? (EFFECTIVENESS) Annex 1.4: Key Informant Interview Guide – Ethiopian Health Insurance agency and its Regional HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT | 70 Branches Interviewee (Title): Location: Date: 1. Please tell us what the EHIA has done to support CBHI and Social Health Insurance over the last three years. (EFFECTIVENESS) Probe: geographic and population coverage; service uptake; financial sustainability of the CBHI; Role of the pilot woreda schemes; lessons learnt in design of the scaling up exercise. 2. Has there been any progress made in CBHI and SHI over the last three years? Probe in terms of examples in: ● Establishment/expansion of schemes ● Enrolment (both paying and non-paying); ● Resource mobilization (contribution, general and targeted subsidy); ● Access to health services (Service provision) ● Accreditation and quality assurance ● Reimbursement to facilities ; ● Establishment of SHI branches 3. Are there any areas in which progress has not been made in CBHI and SHI over the last three years? (Probe for examples, lessons learnt from this) 4. (For any progress mentioned by respondent above only) - Are there any major contributing factors to that progress? (EFFECTIVENESS) Probe; Any success factors on CBHI scaling up that are documented or communicated? 5. Could you please explain us the mechanisms that are being practiced to monitor the progress of CBHI and share experience between the schemes? ((EFFECTIVENESS) 6. Are there components of the reform program (see above) that has not been able to achieve the set goals and targets?(one of them being CBHI/SHI-coverage rate of 20%). If so, what are the main challenges?(EFFECTIVENESS) Probe for: a) Health sector constraints (e.g., commitment, policy gaps, capacity, staff turnover, institutional) b) Constraints outside the health sector? 7. Has the government and your agency realized the project assumption of increased government ownership and leadership of the health financing reform program? (Probe for examples). (TOC)(Note: e.g. Operationalize the EHIA by opening 24 branch offices across the country, strengthening of the EHIA and CBHI executive organs, BoFEDs/WOFEDs, the FMOH, and regional health bureaus (RHBs) to allocate budgets for indigents and general subsidy and EHIA’s capacity to communicate with the public and providers); Probe: If yes, how? If not, why? 8. What type of TA has been provided, if any, to you by the HSFR/HFG project? 9. How has the project TA performed in strengthening your capacity? (RELEVANCE)Probe for: 71 | FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) a) How do you see the relevance (demand driven nature- consistent with the needs of the sector at all levels)), effectiveness (getting the right TAs support at the right time) and sustainability of the technical support? b) What were the focus of TA support – development of tools, skills, staff and infrastructure, as well as structures, systems and roles? Which components of the investment of capacity building (individual, organizational and enabling environment levels) were effective? c) What do think the HFG/HSFR team should change or improve to provide responsive and effective TA for capacity building for the remaining period of the project and beyond? d) What are the Strengths (relevance, appropriateness, timeliness, supporting the government own plan) e) What Weaknesses (relevance appropriateness, timeliness, supporting the government own plan) are observed?; what do you think the major reasons for these weaknesses are? f) What Suggestions (including another approach for the provision of the technical assistance?) do you have to further improve performance? 10. Do you think the interventions of the project being supported continue to be relevant to the context of the country? (SUSTAINABLITY) (Note for team: Probe in the context of the recent developments in the health financing reform landscape - HCF strategy, Universal Access, HSTP, envisioning document; potential fatigue of support by USAID after financing some of the interventions for over 15 years)? If not a) if yes, which one/s of the reform components should continue as is? What are the reasons for continuing the support? b) If no, which reform areas should be phased out from project support? What are the reasons for phasing out the support? c) Which reforms, if any, should be adjusted? What are the reasons for adjustment? d) What new initiatives should be included to support you better? 11. What do you think the focus of the future Health financing reform should be to improve health insurance? (SUSTAINABLITY)Probe: a) What the current support program needs to focus on and support in its remaining period? b) What should be the focus of the new USAID TA support in the near future? Any recommendation on nature and composition of the most appropriate interventions? c) What should USAID and other partners do to support HCF strategy related interventions that promote health insurance? HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT | 72 Annex 1.5: Key Informant Interview Guide – HSFR/HFG Project Interviewee (Title): Location: Date: 1. Please tell us what HSFR/HFG has done in the context of health care financing reforms in the country. 2. Have there been any major achievements in the implementation of the health care financing reform at national, regional and Woreda levels? (EFFECTIVENESS) (probe for getting examples and explanations the following) ● Developing legal frameworks ● Revenue retention and utilization ● Fee waivers ● Fee revision ● Exemptions ● Governing boards ● Outsourcing ● Private wing ● Community based health insurance scheme ● Social Health Insurance ● Coverage of the reform program in terms of facility, and Woredas? 3. What are the major contributing factors for such progress stated above? (EFFECTIVENESS)Probe: Are these success factors documented and communicated? If yes how? 4. Are there components of the reform program (see above) that have not been able to achieve the set goals and targets?(EFFECTIVENESS) Probe: What are the main challenges and constraints in this regard on the following: ● HSFR/HFG Project constraints ● Health sector constraints (e.g., commitment, policy gaps, capacity, staff turnover, institutional) ● Constraints outside the health sector 5. Please tell us about the performance of CBHI and SHI over the last three years. (EFFECTIVENESS) Probe: geographic and population coverage; service uptake; financial sustainability of the CBHI; Role of the pilot woreda schemes; lessons learnt in design of the scaling up exercise 6. Has the government assisted you to realize project objectives by turning major project design assumptions into reality?(TOC) Probe: If not, why? What has been its impact? (Notes for the team: Focus on increased government ownership and leadership of the health financing reform program (e.g. Operationalize the EHIA by opening 24 branch offices across the country, strengthening of the EHIA and CBHI executive organs, MoFED/MOFEC, the FMOH, and 73 | FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) regional health bureaus (RHBs) to allocate budgets for indigents and general subsidy) 7. What have been the effects of the above HCF reforms? (EFFECTIVENESS)Probe areas of: ● Increased resources? ● Increasing access? ● Quality of care? ● Equity? ● Evidence generation for planning and M&E? ● Sustainability? 8. What has been the effect of the health finance reforms, including the establishment of private wings in hospitals, on the motivation and retention of health workers? (EFFECTIVENESS)Probe the following: ● Retention of highly qualified health professionals ● Improvement of quality of service ● Equity in service availability in both the general and private wards ● Any other intended and unintended effects 9. Have you observed any unintended consequences (both positive and negative) of the health finance reform program? If yes, what did you do to address these unintended consequences?(SUSTAINABILITY) 10. Do you think that the components of the health care financing reform project and their assumptions are adequate to deliver the result chain expected from the project? (RELEVANCE/EFFECTIVENESS)Probe for: ● Completeness of the result chain ● Completeness and objectiveness of the assumptions used ● Basis of identified interventions ● Definition of measurement indicators 11. Do you think the current interventions of the project being supported continue to be relevant in the short and medium term? (SUSTAINABILITY)If not; probe: Which ones of the reform components should be phased out from project support? Which ones of the reform components should be adjusted? What new initiatives should be included to supportive you better? 12. What HSFR interventions in the consolidating regions can be institutionalized and implemented by these regions without further projects support? (SUSTAINABLITY)Probe for: a) What are areas that the regional government can take full responsibility? b) What should be the exit strategy should this be considered? c) Which of the above HCF reform programs are still relevant and should continue for the next five years? d) In which of these areas should the strategy of implementation be modified and why? 13. What are the successes and challenges in providing technical assistance at federal, regional and woreda levels? (SUCCESS and SUCCESS FACTORS) Probe for: a) How do you determine what type of TA to provide HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT | 74 b) How soon can you provide when requested? c) What were the focus of TA support –development of tools, skills, staff and infrastructure, as well as structures, systems and roles? Which components of the investment of capacity building (individual, organizational and enabling environment levels) were effective? d) Sustainability of capacity building effort? e) Acceptance and readiness of government to use TA and build sustainable capacity? f) What Suggestions (including another approach for the provision of the technical assistance?) do you have to further improve HCF performance? 14. How far have you used the main implementation strategies to enhance efficiency and effectiveness? (EFFICIENCY/EFFECTIVENESS)Probe for any successes and challenges? Probe for getting examples for each of the following: ● Policy dialogue with all stakeholders? ● Capacity building of health sector managers? ● Provision of international technical assistance? ● Partnership with other DP financed projects ● Program learning for policy development? 15. What do you think should be done within the current project time frame by different stakeholders? (EFFECTIVENESS)Probe: Main actions by: ● Regional and Woreda health administration? ● Regional and Woreda finance offices? ● FMOH? ● HSFR/HFG project /Abt associates/? ● USAID? ● Other partners 16. What do you think should be done in the next five years to inform the next project design? (SUSTAINABLITY)Probe for the roles of: ● Government of Ethiopia? ● FMOH? ● Regional and Woreda health administration? ● Regional and Woreda finance offices? ● Development partners including USAID? Annex1.6: Key Informant Interview Guide – Development and implementing partners /DPs/IPs Interviewee (Title): Location: Date: 1. Please brief us what you know about the health finance reform in Ethiopia and your organization’s role, if any, in supporting it? (INTRODUCTION/RELEVANCE) 2. What effects of health financing reform have you observed over the last 3 three years? (EFFECTIVENESS)Probe for examples and explanations in the following areas: 75 | FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) ● Revenue retention and utilization ● Fee waivers ● Fee revision ● Exemptions ● Governing boards ● The accreditation system ● Networks of insurance schemes and FGBs and their capacity Outsourcing ● Private wing ● Outsourcing ● Community based health insurance scheme ● Health insurance ● Policy and strategy Updates 3. Have you observed any achievements from the health care financing reform? (EFFECTIVENESS)Probe in terms of: ● Access? ● Quality? ● Equity? ● Evidence generation? ● Contributing factors to any achievements 4. What are the major a) challenges and b) gaps in the implementation of the health sector financing reform program in Ethiopia? Note for team: gaps may be challenges. (CHALLENGES AND MAIN REASONS) Probe the following, with examples: ● Policies and strategies ● Gaps in implementation ● Gaps in capacity ● Institutionalization ● Sustainability 5. What do you think should be done to strengthen the health sector financing reform in the next two years? (SUSTAINABILITY)Probe: ● By Government? ● By development partners, including USAID? ● By implementing partner like HSFR/HFG /Abt associates/? 6. What do you think will be the major long-term health financing challenges in the next five years and beyond? (CHALLENGES AND MAIN REASONS) Probe the following including examples of how to overcome those challenges: ● Access ● Quality ● Sustainability ● Effectiveness and Efficiency ● Ownership and institutionalization of reforms ● Local level leadership and governance HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT | 76 7. What do you think government should do to address /overcome these longer-term challenges in health financing? (TOC)Probe: ● Policy and strategy ● Expanding Universal Access ● Equity ● Efficiency ● Effectiveness and efficiency ● Ownership and institutionalization of reforms ● Local level leadership and governance 8. What do you think development partners should do to address /overcome these longer-term challenges in health financing? (TOC)Probe: ● Policy and strategy ● Expanding Universal Access ● Equity ● Efficiency ● Effectiveness and efficiency ● Ownership and institutionalization of reforms Local level leadership and governance 9. Does your organization have any interest in supporting/ financing health financing reforms, (including SHI and CBHI) in the future? If yes, in what areas and form? (SUSTAINABILITY) Probe: 10. Given the nature and performance of its health financing support through HSFR/HFG project so far, what do you recommend for USAID to do in its next follow-on project design? (SUSTAINABILITY) a. In intervention area focus? b. In the modality of TA provision? c. In levels of focus (federal, regional, implementation or combinations)? Annex 1.7: Key Informant Interview Guide – Regional and Woreda Health Office Heads Interviewee (Title): Location: Date: 1. Can you please briefly describe your roles in implementing the health finance reforms in your region/woreda? (INTRODUCTION/RELEVANCE)Probe: What is the structural and functional delineation and linkage between the region and woredas? 2. Have there been any major achievements in the implementation of the health care financing reform in the region/ Woreda? (EFFECTIVENESS) (Probe for examples and explanations in the following areas :) a. Developing legal frameworks (only for regional level) b. Revenue retention and utilization 77 | FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) c. Fee waivers d. Fee revisions e. Exemptions f. Governing Boards g. Networks of health insurance schemes and FGBs h. Accreditation i. Outsourcing j. Private Wing k. CBHI l. Client satisfaction surveys 3. What are the major contributing factors for major achievements stated above (refer to respondent’s response to question 2)? (EFFECTIVENESS) Probe: Are these success factors documented and communicated? If yes how? 4. Could you please explain us the mechanisms that are being practiced to monitor the progress of CBHI and share experience between the schemes? (EFFECTIVENESS) 5. Are there components of the reform program (Refers responses given in question 2 for the different components) that have not been able to achieve the set goals and targets? (EFFECTIVENESS) Probe main challenges and constraints in the following: a) HSFR/HFG Project constraints b) Health sector constraints (e.g., commitment, policy gaps, capacity, staff turnover, institutional) c) Constraints outside the health sector 6. What has been the effect of the reform program in your region/Woreda? (EFFECTIVESS); Probe for examples in the following core areas: a) Increased resources? b) Increasing access? c) Quality of care? d) Equity? e) Evidence generation for planning and M&E f) Sustainability g) Any other unintended effects (positive or negative) 7. Please describe us on the availability of health professionals as per the national standard, in the facilities in your woreda? Probe: What are the major issues in retaining and motivating the health workers in the facilities in the Woreda ? facility? What do you think are the major reasons for any staff turnover in the woreda?? 8. How has retention of staff been in your region/woreda during the project period? (EFFECTIVENESS) Probe areas: a) Private wing role in retention b) Retention of doctors c) Improvement in quality of service d) Equity in service availability in both the HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT | 78 general and private wards e) Any other intended and unintended effects 9. Has the implementation of CBHI contributed to improving access to health services in your woreda? Probe: How? (EFFECTIVENESS) 10. Do you think the interventions of the project being supported continue to be relevant to the region? (SUSTAINABILITY) (Note for team: This is in the context of the recent developments in the health financing reform landscape - HCF strategy, Universal Access, HSTP, envisioning document; potential fatigue of support by USAID after financing some of the interventions for over 15 years) a) Which reforms, if any, are good and need to continue? What are the reasons for continuing the support? b) Which reform areas, if any, should be phased out from project support? What are the reasons for phasing out the support? c) Which reforms, if any, should be adjusted? Why? d) What new initiatives should be included to support you better? 11. Has the health sector reform project made any contribution to capacity building in your region/woreda?(RELEVANCE) In what ways? Probe areas a) Relevance of the training programs (push and pull) provided by the project b) Scope of depth of capacity building (training, supervision, guidelines, mentoring, legal frameworks, decision making capacity) c) Possible contribution of capacity building to motivation and retention and motivation of health workers? 12. For only consolidating regions (Oromia, SNNPR, and Amhara): Given that you have implemented HSFR project for more than ten years, do you think the region can now implement the reform on its own without the projects support? (SUSTAINABILITY) Probe: a) Are there areas that the regional government can take full responsibility? b) Any suggests about the exit strategies for USAID? 13. What are your perceptions of the technical support provided by the HSFR project?(RELEVANCE) Probe areas: a) Strength (relevance appropriateness, timeliness, supporting the government own plan) b) Weaknesses (relevance appropriateness, timeliness, supporting the government own plan); what do you think the major reasons for these weaknesses are? c) Suggestions (including another approach for the provision of the technical assistance?) 14. In order to take the health financing reform program forward, what do you think different actors should do? Probe (short term and longer term action by) (SUSTAINABLITY) a) Regional and woreda health administration? b) Federal, Regional and woreda finance offices? c) FMOH? d) HSFR/HFG project or any other support? e) USAID? f) Other partners Annex 1.8: Key Informant Interview Guide – Regional/Woreda Finance Bureau 79 | FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) Interviewee (Title): Phone no: Location: Date: 1. Have there been any major achievements of the different health financing reform activities in your region?(EFFECTIVENESS) (probe the following) b. Developing legal frameworks (only for regional level) c. Revenue retention and utilization d. Fee waivers e. Fee revisions f. Exemptions g. Governing Boards h. Networks of health insurance schemes and FGBs i. Accreditation j. Outsourcing k. Private Wing l. CBHI m. Client exit interviews 2. What is the perceived or actual effects/contribution of these reforms on the following:(EFFECTIVENESS) n. Access to health care? o. Improving Quality of care? p. Enhancing Equity? q. Strengthening health financing M &E and learning? r. Financial sustainability of health facilities? s. Accountability t. Affordability 3. Please tell us about the structure of health facility governance (governing bodies) in terms of :(RELEVANCE) a) Composition of board members b) Role and responsibility of board members c) Functionality d) Effectiveness in HCF reform implementation 4. How do you assess the facility’s capacity to properly utilize in proper utilization the resources generated and retained? Probe in: In terms of: a) Proper prioritized planning and budgeting? b) Focus on quality - improvements? c) Responding to client needs? d) Proper utilization and reporting? Any audit issues? 5. How regular and complete is the financial management reporting system? (EFFICIENCY) Probe: a) How are you strengthening the capacity of the health facilities in financial management? HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT | 80 b) Does the use of modern information technology have a role? 6. As per regional regulation, is retained revenue of health facilities appropriated before utilization? (RELEVANCE)(Probe: major complaints of Health Facilities (HF):, how many times should HFs can appropriate; strategies to deal with problems) 7. What do you think about budget offsetting during annual budget allocation as a result of revenue retained by facilities? (EFFICIENCY) 8. How often do you investigate /audit/ the accounting records of health facilities? (EFFICIENCY) (probe: misappropriation of RR, inappropriate prioritization,) 9. Based on your experience with financing reforms so far, what modifications would you suggest for the next two and five years? (SUSTAINABILITY) Probe the following: a) Retention of fee and its additionality at the facility levels b) Utilization of revenues at the facility level c) Monitoring and Evaluation d) Governance e) Waivers and exemptions f) Activities or programs to be scaled up or contracted g) Activities and programs that have become self-sustaining h) CBHI in terms of enabling environment and readiness of facilities 10. How effective is your targeting of the poor? (EFFECTIVENESS) Probe: covering the cost of services for the indigents, those with wavers, and those eligible for subsidies under CBHI? 11. How do you assess the TA provided through the project? (RELEVANCE) Probe: effectiveness, efficiency and sustainability of the TA provided to RHBs, woredas, facilities and CBHI schemes? Annex 1.9: Key Informant Interview Guide –Heads of Health Facilities (Hospitals and Health Centers) Interviewee (Title): Location: Date: Telephone No. 1.Tell us a little about your health center or hospital (beds, services, area of service, population covered, number of staff, size and makeup of the board, etc.,)? (INTRODUCTION) A. Section guide for CEO/head of HF 1. Are there any major a) achievements and b) constraints with regard to implementing the health care finance reforms in this health facility? (EFFECTIVENESS) Probe for examples and explanation on the following: ● Revenue Retention and Utilization: ● Financial Management Improvements: ● Fee Waivers and Exempt Services: ● Revision of User Fees: ● The Accreditation system ● Functioning of Governing Board (see specific questions): 81 | FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) ● Outsourcing of Non-Clinical Services: ● Implementation of a Private Wing (see specific questions): ● Working with Community Based Health Insurance: ● Relationships and Referral Linkages. 2. What is the experience of your facility in working with of CBHI schemes and members? (EFFECTIVENESS)Probe: ● Increasing utilization by members? ● Reimbursement of funds? ● Burden to the facility? ● Increasing the revenue retention ● Community’s claim to get better and responsive service? ● Any complaints of readiness and quality of service in the facility? 3. Has the facility ever done client satisfaction survey?(RELEVANCE) If yes, probe: how the findings were used.(e.g. to influence prioritization in using facility funds). 4. What are some effects of health financing reform on your facility?: (EFFECTIVENESS)Probing areas: ● Increasing Access to Services ● Quality of Care ● Equity ● Sustainability ● Monitoring and Evaluation ● Unintended consequences (both positive and negative) 5. Do you think the current interventions of the project being supported continue to be relevant to your facility in the short and medium term? (SUSTAINABLITY) probe:: ● If yes, which ones need to continue as they are? What are the reasons for continuing the support? ● If not, which ones of the reform components should be phased out from project support and what are the reasons for phasing out? ● Which ones of the reform components, if any, should be adjusted? ● What new initiatives should be included to support you better? 6. Tell us about the type of technical support you get from HSFR/HFG Project? (RELEVANCE) Probing areas: ● Communication methods and strategies ● Relevance of training programs: ● Capacity Building among staff: ● Motivation and retention of staff: ● Lack of incentives for staff: HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT | 82 7. Do you have any recommendations for future USAID investments in the GoE’s health sector financing reform efforts for the next 2 years? (RECOMMENDATIONS) 8. What areas should USAID focus on over the next 5 years that will benefit the GoE’s health reform efforts? (SUSTAINABILITY) 9. Are there any other comments or suggestions that you have on the health financing reform efforts? B. Section guide for clinical staff (one person) 1. Tell us what you know about the health care financing reforms that have been taking place for the past few years? (EFFECTIVENESS) (Probe: revenue retention, private wing, community insurance scheme) 2. What do you think is the effect of health financing reform activities on the quality of care you provide to patients? (EFFECTIVENESS) 3. Have any of the health financing reform activities affected your motivation to work in this facility? (RELEVANCE)Probe: How? Have you worked in the private wing and how does the operation of private wing in this facility affect you? 4. What are the challenges of the activities under the health care financing reform? (CHALLENGES AND MAIN REASONS) 5. What are your recommendations to improve activities under the health care finance reform? (RECOMMENDATIONS) Probe: Revenue retention; Private wing and CBHI.) Annex 1.10: Key Informant Interview Guide –CBHI Scheme Management Staff Interviewee Title): Location: Date: Telephone No. 1. Please tell us about the performance of the CBHI scheme in the woreda? Probe for success stories and their drivers: a) Enrolment and renewal of membership? b) Availability and quality of services and members’ perceptions and complaints if any c) Woreda and regional support for the indigents d) FMOH support for general subsidy and readiness of facilities e) Woreda’s strategy to increase enrollment? 2. Please tell us about on the financial status of the CBHI scheme? Probe: a) The amount of resources collected (from premiums, targeted subsidies and general subsidies) b) The total reimbursements paid out for providers c) Financial balance of the scheme and prospect to covering its own cost? 83 | FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) d) Cash-flow status 3. Can you tell us about the capacity of the scheme management to manage CBHI funds? (Probe for: Trained Financial management staff; Financial management systems; monitoring and evaluation systems; Communication of financial information to members; Communication of financial information with facilities; Collection of premiums; reimbursement to facilities) 4. Could please inform us how the scheme presents the financial status of the scheme to its members? Probe: though general assembly meeting once in a year or other forum? 5. What type of training and capacity building assistance, if any, have you received from the project for strengthening the management of the scheme? Probe: Areas of training; relevance of the training; mode of conduct of the training.? What is your opinion on these TAs? 6. How do you assess the capacity of the scheme management in managing the CBHI in terms of: a) Registration of members and renewal of membership; b) Timely collection of premiums? c) Financial management d) Reimbursement management e) Medical audit (Control of facilities from requesting unnecessary reimbursement) Generating, reporting and utilizing information for decision making)? Probe for the areas of scheme management that needs strengthening) 7. How convenient is the institutional arrangement of the scheme (CBHI are housed either in Woreda administration or Woreda health office) to undertake you day to day operation and achieve your targets? Do you alternative idea on the institutional arrangement? If yes, could explain it? 8. In your woreda, what is best modality of communication/reaching out to the community to enrolled into CBHI? Please probe for the relevance and effectiveness of : a) The role of the mass media activities and its communication mechanisms? b) The role of the Health development army and its networks? c) The role of the woreda and kebele cabinet and its instruments? 9. How far have project interventions been effective in terms of mobilizing the population into enrolling in schemes? (EFFECTIVENESS) 10. How can the communication modalities be improved? 11. Are the benefit packages included relevant and worth the contributions levied to attract members and expand coverage in health insurance? (RELEVANCE) 12. Are the interventions/ methods used (Note for team: i.e. membership policy - Individual based, household based, method of registration of members, and collection of contributions for expanding coverage of the population in schemes) appropriate? (RELEVANCE) 13. Is the relation between the schemes and providers efficient in providing services and reimbursement of service fees? (EFFICIENCY) Probe: Have you adopted the prototype medical audit manual? 14. How frequently do you undertake financial audits of the scheme and who is undertaking that?(EFFICIENCY) . Probe – internal audits? External audits? Both? HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT | 84 15. Could you please describe the enabling and constraining environment for implementation of health insurance in general and CBHI in particular? (RELEVANCE/ENABLING ENVIRONMENT) 16. What are the challenges for the implementation of health insurance in general and CBHI in particular? (CHALLENGES AND MAIN REASONS) 17. What do you recommended for the improvement of health insurance implementation? (RECOMMENDATIONS) 85 | FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) ANNEX G: FGD GUIDE Instructions: At the beginning of each interview with participants, please introduce your team and explain the purpose, confidentiality and ethics context of the evaluation as stated below. Introduction: Good morning/afternoon! Thank you for sparing your time today to speak with us. The team here is working to conduct a midterm evaluation of the HSFR/HFG project. The purpose of this project evaluation is to provide an informed assessment of progress to date, and recommendations for the remaining period of the project and any future follow-on project. Our team has had the opportunity to review project documents to get a sense of the design and implementation of the project. However, review of the documents alone is not enough. Therefore, we would like to speak with you today to hear about your experience, in your own words, in order to help us better understand how this project is working. Confidentiality: ● The information that we will collect will include individuals’ names, organizations, and positions. The annex of the evaluation report will include a list of key informants, but the findings or statements in the report will not be associated to any particular name of the key informant. ● Quotes from respondents will be included in the evaluation report, but there will be no link between the quotes and the names of the individuals who provided the quote. In the event that the team desires to use any personally identifiable information in the report (such as a photograph of the person), the evaluators will first contact the respondent(s) to seek permission to do so. ● The information that we shall collect during this evaluation will be used for the sole purposes of this evaluation. This information will not be used for any other purpose. ● Your participation in this interview is voluntary. If you do not feel comfortable answering any particular question, please let us know and we will simply go on to the next question. ● Thank you once again, for taking the time to speak with us today. If you have any questions for us, you can ask now, before we get started. HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT | 86 ANNEX G.1: FGD Guide –CBHI Board Names and titles of FGD participants: Date:_________________________ 1. Could you explain what the major roles of the woreda CBHI Board are? (INTRODUCTION/RELEVANCE) Probe: Is the CBHI Board undertaking its role as stated in the regional CBHI Directive? 2. Are the interventions/ methods used in implementing CBHI Board relevant? (RELEVANCE)Probe: membership policy (Individual based, household based); method of registration of members; collection of contributions for expanding coverage: benefit packages; 3. How is the relationship between the scheme and providers in providing health services and reimbursement of service fees? (EFFECTIVENESS) Probe: mechanism to ensure transparency and accountability for the financial status of the scheme. 4. Is CBHI scheme institutional arrangement sustainable?(SUSTAINABILITY) If yes, why and if no, why not? Are there any success stories in the implementation of CBHI? Probe for (involvement of beneficiaries, service providers, other stakeholders) 5. Could you please describe the enabling/constraining environmental factors for implementation of health insurance in general and CBHI particular? (RELEVANCE/ENABLING ENVIROMENT) 6. Are there any recommendations that you have for the improvement of the implementation of health insurance in general and CBHI in particular? (RECOMMENDATIONS) ANNEX G.2: FGD Guide –COMMUNITY-HDA LEADERS Names and titles of FGD participants: Date:_________________________ 1. Tell us about the Facility Governing Boards (FGBs) of your health facilities. (INTRODUCTION/RELEVANCE) Probe: a) Do you think that these boards are needed (relevance)?; b) Are they functioning/working (effectiveness); c) Are they working as well as they should (efficiency); d) Can they continue to work if the project ends (sustainability); e) Any success stories; f) Any constraints to an enabling environment). g) Any challenges; and h) Any recommendations; 2. Tell us about the CBHI and CBHI board of your scheme. (RELEVANCE) Probe: a). Does the communication interventions on CBHI reach you and are they relevant ? b). Do the benefit packages of CBHI address the needs of the community? 87 | FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) c). Does CBHI Improved access to health services for CBHI members? d). Do you think that these boards are needed (relevance)?; e). Are they functioning/working (effectiveness); f). Are they working as well as they should (efficiency); g). Can they continue to work if the project ends (sustainability); h). Any success stories; i). Any constraints to an enabling environment). j). Any challenges; k). Any recommendations 3. Tell us about utilization of health services in your facilities since HSFR/HFG project implementation, particularly during the last three years.(EFFECTIVENESS) (Probe: perception of more or less utilization of health services at the facilities; perception of why this is happening) 4. Tell us about the trend of out of pocket payments at health care since HSFR/HFG project commenced? (EFFECTIVENESS) (Probe: perception of more or less out of pocket payments since HSFR; perception of why this is happening) 5. Tell us about the quality of services at health facilities since HSFR/HFG project implementation in the last three years.(EFFECTIVENESS) (Probe: perception of improvement of quality or not; perception of why this is happening.) 6. If you were to design the project afresh to increase access and utilization of health services, what would it look like? (EFFECTIVENESS) (Probe: Areas for improvement) Annex G.3: Focus Group Discussion Guide: Facility Governance Names and titles of FGD participants: Date:_________________________ 1. Please describe your role in overseeing the performance of the health facility? (INTRODUCTION/RELEVANCE) 2. Does the governing board have sub-committees? (EFFICIENCY) (Probe: such as Quality committee, Audit committee, Finance committee etc.) If yes, how do they support you in your functions? 3. How do you carry out your functions? (RELEVANCE) (Probe: meetings, facility performance supervision, management consultations; major decisions taken in the last six months?) HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT | 88 4. Is there any internal document of the GB that outlines board proceedings?(EFFECTIVENESS) (Probe: ask to see it) 5. Have the GB members received any training regarding their roles? (RELEVANCE) (Probe: By whom? Content of training; frequency of training) 6. Has there been any attrition of the governing board members? If so, tell us about it.(EFFECTIVENESS) 7. How do you bring the views and concerns of the community into your decision-making process? (EFFECTIVENESS) 8. What is the gender mix of GB members? (RELEVANCE) Probe: What is the professional mix of GB members? 9. What has been the outcome of setting up governance bodies on service delivery in your HF? (EFFECTIVENESS) Probe: access, quality, equity and sustainability? 10. Does the facility follow-through on your decisions and advise?(EFFECTIVENESS) (Probe: Examples of such. 11. What challenges have you faced in the operation of governing bodies? Probe: resources, leadership and management skills, etc.) (CHALLENGES AND MAIN REASONS) 12. What recommendations do you have to improve the functioning of health facility governance?(RECOMMENDATIONS) 89 | FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) ANNEX H: PROFILES OF THE EVALUATION TEAM MEMBERS Name Position Short Profile Abebe Alebachew Team Leader He is a health economist and health system specialist with over thirty years of working experience in formulation, review and undertaking health sector strategic plans and policies in many countries in Africa. He is also extensively worked in health systems strengthening in general and health financing reforms and aid architectures in particular. He appraised the costing and financing of health sector and sub sector strategic plans in Ethiopia, Kenya, Rwanda, Ghana and Sudan. He undertook an evaluation of the health sector development/strengthening plans in Kenya, Rwanda (twice),Sierra Leone. He also evaluated GAVI and Global Fund health system support in Ethiopia and Sudan.; He led the annual joint health sector review missions of 2009, 2011, and 2012 In Ethiopia; He was a Deputy Team Leader for the team that appraised the MDG pooled fund, & its related financial assessment of 2011 that mobilized $150 million per year. He has worked extensively with international organizations including U.S. Agency for International Development, UNICEF, DFID, DANIDA, Sida, WHO and other DP funded implementing partners. John Osika International consultant Dr. Osika has over two decades of high-level international experience in different health systems. His global experience includes researching, monitoring and evaluating different reforms and interventions in different health systems. Key experiences include health system and health finance reforms that include health insurance; innovative health sector financial reforms. Dr. Osika is skilled in liaising effectively with government and civilian organizations, as well as varied development agencies, including USAID, The World Bank, WHO, Global Fund for AIDS. He has worked on global and bilateral USAID-funded health systems projects with significant health financing components. Dr. Osika has academic research and teaching experience, having held a faculty position at the University of Wales College of Medicine. Workie Mitiku National consultant Workie is an economist and has a competence in the area of health financing in general and health insurance in particular. He has served as a consultant in Sudan to advise the National Health Insurance Fund on the design of CBHI and in Namibia to assist the Universal Health Coverage Advisory Committee of Namibia in the development of the road map to Universal Health Coverage (UHC). He has acquired experiences on health insurance through study tours in Africa (Ghana, Rwanda), Asia (Thailand, Vietnam, and China), Europe (Germany) and in Latin-America (Mexico) between 2006 and 2015. He has worked in the area of health financing for more than 10 years. Workie has also a competence in research and analytical skills and currently works as freelance consultant in health insurance and financing. Esubalew National Esubalew is a health economist with over 25 years of work experience HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT | 90 Demissie Consultant in health, finance and education sectors of the economy. Key competencies include health care financing; health policy analysis; assessment and analysis of health system constraints to scaling up of high impact HNP interventions; decentralized planning and budgeting; National Health Accounts /NHA/; health facility governance and management; establishment and operation of private rooms/wings in public hospitals; outsourcing of health services; survey design and implementation. Esubalew served both in government and bilateral development agencies in Ethiopia. He served as a Country Technical Director for Family Health Efficiency Effectiveness and Equity (FHE3) Initiative Project, Bill & Melinda Gates funded & MSH implemented; Senior Health Care Financing Advisor and health care financing specialist for USAID funded & Abt Associates implemented Health Sector Financing Reform Projects; Esubalew did his M.Sc. Degree in Economics at Addis Ababa University and MBA in Health Care Services (MBA-HCS) at Sikkim Manipal University, India. Nigusu Aboset National Consultant Mr. Nigusu has more than 10 years of experience in public health programs and social welfare development, monitoring and evaluation, training, research and program development and management. With a background of Sociology and Social Anthropology and Public Health Specialist, Nigusu has experience in designing studies, developing data collection instruments and report write-up. He has extensive experience in Surveillance, Humanitarian Emergency Programs, Public Health and Nutrition, Livelihood and Food security and Social Welfare Development. He has also ample experience in coordinating and conducting public health and social welfare researches for various international and national NGO. He has also been working in South Sudan and Uganda as Monitoring and Evaluation Manager. He is well acquainted in various donor reports like USAID, ECHO, HRF, EU￾SHARE, UNICEF, The Netherlands Funds, OFDA, DEC, DFID etc. He has three masters (Master of Public Health, Master in Sociology and Social Anthropology and Master’s in International Cooperation and Humanitarian Aid). 91 | FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) ANNEX I: REFERENCES Abebe Alebachew, Yasmin Yusuf, Carolyn Mann, Peter Berman (2014). Ethiopia’s Progress in Health Financing and the Contribution of the 1998 Health Care and Financing Strategy in Ethiopia, Tracking and Management Project, Harvard school of public health and Breakthrough International Consult PLC CSA (2010). Ethiopian Demographic and health Survey CSA (2016). Ethiopian Demographic and health Survey EEA (2015). Assessment on the Operation of Private Wings in Public Hospitals, Addis Ababa, Ethiopia. EEA (2016). Revenue Retention and Utilization in improving Quality of Health care in Ethiopia, Addis Ababa, Ethiopia. EHIA (2014). Revised CBHI Pilot Prototype Directive, Addis Ababa, Ethiopia. EHIA (2015). Community Based Health Insurance Scale-up Strategy, Addis Ababa, Ethiopia. FMOH (2017). Draft Revised Health Care Financing Strategy, Addis Ababa, Ethiopia. FMOH (2017). NHA VI Household Health Service Utilization and Expenditure Survey Draft Report, Addis Ababa, Ethiopia. EHIA (2015). Evaluation of Community Based Health Insurance Pilot Schemes in Ethiopia, Final Report, Addis Ababa, Ethiopia. FMOH (1998). Health Care and financing Strategy, Addis Ababa, Ethiopia. FMOH (2015). CBHI Scale-up Directive, Addis Ababa, Ethiopia. FMOH (2015). Health Sector Transformation Plan (2015/16-2019/20), Addis Ababa, Ethiopia. FMOH (2016). Ethiopian National Health Care Quality Strategy 2010-2016, Addis Ababa, Ethiopia. FMOH (2016).Health Sector Transformation Plan - I Annual Performance Report, Addis Ababa, Ethiopia. FMOH (2012) Fifth National Health Account: Household Health Service Utilization and Expenditure Survey, Addis Ababa Ethiopia HSFR (2009). Establishing Private Wings in Public Health Facilities: Operational Manual, Addis Ababa, Ethiopia. HSFR/HFG (2013). Year One Work Plan (August 1, 2013-June 30, 2014). Addis Ababa, Ethiopia. HSFR/HFG (2013). Two-Year Work Plan (August 2013-September 2015), Addis Ababa, Ethiopia. HSFR/HFG (2013). Year One (1st August 2013-30th June 2014) Performance Monitoring Plan (PMP) Matrix, Addis Ababa, Ethiopia. HSFR/HFG (2014). Year One Annual Performance Report (August 1, 2013-June 30, 2014), Addis Ababa, Ethiopia. HSFR/HFG (2014). Year One Annual Performance Plan (August 1, 2013-June 30, 2014), Addis Ababa, Ethiopia. HSFR/HFG (2014). Year Two Annual Performance Report (July 1, 2014-June 30, 2015), Addis Ababa, Ethiopia. HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT | 92 HSFR/HFG (2014). Year Two Performance Monitoring Plan (PMP) Matrix (July 1, 2014-June 30, 2015), Addis Ababa, Ethiopia. HSFR/HFG (2014). Year Two Work Plan (July 1, 2015-June 30, 2016), Addis Ababa, Ethiopia. HSFR/HFG (2014). Year Three Annual Performance Report (July 1, 2015-June 30, 2016), Addis Ababa, Ethiopia. HSFR/HFG (2015). Year Two Annual Performance Plan (July 1, 2014-June 30, 2015), Addis Ababa, Ethiopia. HSFR/HFG (2015). Year Three Performance Monitoring Plan(PMP) Matrix (July 1, 2015-June 30, 2016), Addis Ababa, Ethiopia. HSFR/HFG (2015). Year Three Work Plan (July 1, 2015-June 30, 2016), Addis Ababa, Ethiopia. HSFR/HFG (2015). Health Facilities Graduation Guideline: Prototype, Addis Ababa, Ethiopia. HSFR/HFG (2016). Year Three Annual Performance Plan (July 1, 2015-June 30, 2016), Addis Ababa, Ethiopia. HSFR/HFG (2017). HSFR/HFG Midterm Evaluation Briefing Power Point Presentation, Addis Ababa, Ethiopia. HSFR/HFG (2017). Power Point Presentation for the HSFR/HFG Midterm Evaluation Team, Bahir Dar, Ethiopia HSFR/HFG (2017). Amhara and Benishangule Gumuz Regional Office Health Insurance Performance Report (2013/14-2015/16), Bahir Dar, Ethiopia HSFR/HFG (2017). CBHI Data Base for the Period 2012/13-2015/16, Addis Ababa, Ethiopia. Keith Mcinnes (1993). Local retention of user fees in government health facilities, Health Financing and Sustainability project, MD, USA. Ricardo Bitrán and Ursula Giedion (2003). Waivers and Exemptions for Health Services in Developing Countries, The World Bank, Washington, D.C., USA. USAID (2013). Health Sector Finance Reform (HSFR) Bridging Program Description (PD)-Addis Ababa. 93 | FINAL EVALUATION REPORT HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) ANNEX J: HSFR EVALUATION SITES REGION WOREDA CBHI STATUS NAME OF HOSPITAL TYPE OF FACILITY Amhara Gaugusa Shekudad CBHI Amhara Dembecha CBHI Amhara Adet Non-CBHI Amhara Finoteselam Finoteselam Primary Hospital Amhara Dangila Dangila Primary Hospital Amhara Bahir Dar Town Felegehiwot Regional Hospital Amhara Fogera CBHI Oromia Adama Non-CBHI Oromia Boset CBHI Oromia Gimbichu CBHI Oromia Gedeb Asassa CBHI Oromia Dodolla Primary Hospital Oromia Shashemene Shashemene Regional Hospital SNNPR Danboya CBHI SNNPR Yirgalem CBHI Afar Chifera Non-CBHI Afar Dubti Non-CBHI Afar Dalifagi Dalifagi Primary Hospital Addis Ababa Addis Ketema Non-CBHI Addis Ababa Kirkos Non-CBHI Addis Ababa Gulelle St. Paul Referral Hospital Addis Ababa Arada Ras Desta Regional Hospital HEALTH SECTOR FINANCING REFORM/HEALTH FINANCE & GOVERNANCE (HSFR/HFG) FINAL EVALUATION REPORT | 94 U.S. Agency for International Development Entoto Street, P.O. Box 1014 Addis Ababa, Ethiopia