This publication was produced at the request of the United States Agency for International Development. It was prepared independently by the Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity. November 2018 EVALUATION REPORT Strengthening Human Resources for Health Activity Ethiopia Performance Monitoring and Evaluation Service DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. EVALUATION REPORT Strengthening Human Resources for Health Activity Final Performance Evaluation Report November 2018 Evaluation Mechanism Number: AID-663-C-16-00010 Ethiopia Performance Monitoring and Evaluation Service (EPMES) for USAID/Ethiopia Cover photo by Social Impact, Inc. taken during data collection. Authors/Evaluation Team Abebe Alebachew Asfaw, Team Leader Fiseha Terefe Yayehyirad Kitaw John Osika Isaac Akanbi iii Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity ABSTRACT The purpose of the evaluation of Strengthening Human Resources for Health (SHRH) Activity was to assess Activity progress and inform design for future Activities. The evaluation used mixed methods to answer five evaluation questions: (1) In what ways did the SHRH Activity contribute to improving human resources for health (HRH) and strengthening the regulatory environment in Ethiopia? (2) Did the Activity interventions facilitate local ownership, coordination and partnerships, and sustainability? (3) Were gender issues addressed in pre-service and in-service training? (4) Have Activity result/sub-result areas and interventions continued to be relevant to HRH gaps and challenges in Ethiopia? (5) Were there any facilitating and hindering factors? The SHRH Activity contributed to improved human resource management and quality of training in Oromia. Currently, Debre-Tabor University is ahead of its peers in the quality of its training. The SHRH Activity supported the federal regulatory agencies. However, the private sector continued to be more highly regulated than the public sector. Many of the Activity’s interventions were sustainable, but research and program learning interventions had limited local ownership and will not be sustained. The Activity made contributions to address gender issues in pre-service education, but not with in-service training. Human resource management, quality improvement, and research were found to be the most relevant interventions for HRH. The SHRH Activity support was a key driver of change for HRH in Ethiopia, and where there were high levels of ownership, commitment, and leadership among the different levels of management at the Ministry, Regional, University, and Health Science Colleges, there was better performance. iv Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity TABLE OF CONTENTS Abstract ................................................................................................................................................................................iii Table of Contents ..............................................................................................................................................................iv List of Tables ........................................................................................................................................................................v Acronyms.............................................................................................................................................................................vi Acknowledgments ............................................................................................................................................................vii Executive Summary .........................................................................................................................................................viii I. Evaluation Background .................................................................................................................................................. 1 2. Methods, Sampling, Analysis, and Limitations ......................................................................................................... 3 3. Findings on the Activity Effectiveness....................................................................................................................... 7 4. Findings of the Evaluation on the Sustainability of SHRH Interventions ........................................................ 25 5. Findings of the Evaluation on Gender .................................................................................................................... 29 6. Findings of the Evaluation on Continued Relevance of SHRH Interventions ............................................... 31 7. Findings on Evaluation on Facilitating/Hindering Factors ................................................................................... 35 Annex 1: SHRM MEL Targets ....................................................................................................................................... 38 Annex 2: Evaluation Matrix ........................................................................................................................................... 39 Annex 3: Statement of Work End of Project Performance Evaluation .............................................................. 47 Annex 4: List of Documents Reviewed ...................................................................................................................... 57 Annex 5: List of Key Informants Interviewed During Data Collection at Different Levels ........................... 58 Annex 6: Data Collection Tools .................................................................................................................................. 59 Annex 7: Focus Group Discussion Guides ................................................................................................................ 76 Annex 8: Observations Guide ...................................................................................................................................... 81 Annex 9: Sampling Frame by the Three Regional Contexts ................................................................................. 83 Annex 10: Quantitative Data Collection Formats ................................................................................................... 84 Annex 11: List of KII Participants ................................................................................................................................ 93 Annex 12: Post evaluation Action Plan ...................................................................................................................... 96 v Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity LIST OF TABLES Table 1: Evaluation Questions ........................................................................................................................................ 2 Table 2: Achievement of HRM ....................................................................................................................................... 7 Table 3: Ratio of HRM Positions to Health Workers by Region ........................................................................... 8 Table 4: Total Number of Human Resource Management Positions in the Sample Regions Combined: Health Centers, Hospitals, Woredas, Zonal, RHB) ......................................................................................... 9 Table 5: HRH Forum Meetings ..................................................................................................................................... 11 Table 6: Performance of the SHRH Activity in Increasing Availability of Targeted HWs .............................. 12 Table 7: Curricula Developed/Updated with SHRH Activity Support for the Three Types of essential HWs, 2018 ............................................................................................................................................................... 13 Table 8: Performance against MEL Targets ............................................................................................................... 16 Table 9: Topics Addressed by Technical Reports (TRs) and Research Publications (RPs) ........................... 21 Table 10: Participation in PLAs and RPs ..................................................................................................................... 21 Table 11: Proposed Study Topics and Reasons for Change/Dropping Proposed Studies ............................. 22 Table 12: HRM Positions Created and Filled (by Year).......................................................................................... 84 Table 13: Performance of HRM, 2012 - 2017 ........................................................................................................... 84 Table 14: Number of Midwives, Anesthetists, and HEWs..................................................................................... 85 Table 15: Performance on Quality Improvement (2012 – 2017) ......................................................................... 86 Table 16: Data on Program Learning Activities Undertaken ................................................................................ 87 Table 17: Data on Research Activities Undertaken ................................................................................................ 88 Table 18: Program Learning Activities by Number and Type of Participants ................................................... 89 Table 19: Research Activities by Number and Type of Participants ................................................................... 90 Table 20: Publications by Number and Type of Participants ................................................................................ 91 Table 21: Post evaluation Management ...................................................................................................................... 97 Table 22: Post-evaluation Action Plan ........................................................................................................................ 99 LIST OF FIGURES Figure 1: Results and Sub-Result Areas of the SHRH Activity (2012 - 2017) ..................................................... 2 Figure 2: Four Steps of the SHRH Activity Performance Evaluation ..................................................................... 3 Figure 3: Training Institutions Visited ........................................................................................................................... 5 Figure 4: HRM Positions in FMOH, RHBs, Zonal and Sub-city (All Regions) ................................................... 10 Figure 5: HRM Positions in Hospitals and Health Centers (All Regions) ........................................................... 10 Figure 6: Health Workforce Density in 2012/13 and 2017 in Ethiopia .............................................................. 14 Figure 7: Health Professionals per 1000 Population (2013-2016)........................................................................ 15 Figure 8: HRM Interventions Sustainability Matrix .................................................................................................. 25 Figure 9: Staff Turnover Rates by Region in Afar, Amhara, Oromia, and SNNP from 2013 - 2017 ........... 32 vi Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity ACRONYMS BOFED Bureau of Finance and Economic Development CIRHT Centre for International Reproductive Health Training CLA Collaborating, Learning, and Adapting CPD Continuing Professional Development CRC Caring, Respectful, and Compassionate CTO Cognizant Technical Officer DHIS2 District Health Information System 2 EAA Ethiopian Association of Anesthetists EDC Education Development Center EMA Ethiopian Medical Association EMT Emergency Medical Technicians EMWA Ethiopian Midwives Association EPMES Ethiopia Performance Monitoring and Evaluation Service FGD Focus Group Discussion FMHACA Food, Medicine and Human Care Administration and Control Authority FMOE Federal Ministry of Education FMOH Federal Ministry of Health FOG Fixed Obligation Grant GoE Government of Ethiopia HEI/TI Health Education Institutions/Training Institutions HERQA Higher Education Relevance and Quality Assurance HEW Health Extension Worker HR/D/M Human Resources/Development/Management HRH Human Resources for Health HRIS Human Resource Information System HSC Health Sciences College HSEDC Health Science Education Centers HSTP Health Sector Transformation Plan HW IR Health Worker Intermediate Result IST In-service Training IT Information Technology KII Key Informant Interview M&E Monitoring and Evaluation MEL Monitoring, Evaluation, and Learning MOU Memorandum of Understanding MSH Management Sciences for Health PLA Program Learning Activity PSE Pre-service Education RHB Regional Health Bureau RHSC Regional Health Science College SBM-R Standards-Based Management and Recognition SHRH Strengthening Human Resources for Health SI Social Impact, Inc. SNNP South Nations, Nationalities, and Peoples SOW Statement of Work SPA Service Provision Assessment SARA Service Availability and Readiness Assessment TA Technical Assistance USAID United States Agency for International Development WHO World Health Organization vii Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity ACKNOWLEDGMENTS The evaluation team would like to thank USAID/Ethiopia for its guidance throughout the evaluation. In particular, the team would like to express special thanks to Mr. Awoke Tilahun, Dr. Helena Worku, and Mr. Eshete Yilma from USAID/Ethiopia for their invaluable comments on the evaluation design, tools, and reports, which have improved the quality of the evaluation report significantly. Similarly, the evaluation team would like to acknowledge the support and inputs from several colleagues from Social Impact Inc. (SI). From SI, thank you to Francis Ogojo Okello, Kerry Bruce, Mike Pressl, Nora Chamma, Worku Ambelu, and Dereje Getahun for their guidance and constructive comments during the evaluation. The evaluation team would like to thank all the Strengthening Human Resources for Health (SHRH) Activity staff at federal and regional levels for having provided valuable support by identifying secondary data and coordinating the logistics during field data collection. Without their support in facilitating access to information, as well as arranging meetings both at the national and regional levels, the depth of evidence would have been limited. Finally, we are very grateful to all the key informants and focus group discussion participants at the federal, regional, facility, and training institution levels for their open and honest views on the performance of the Activity. viii Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity EXECUTIVE SUMMARY PURPOSE AND BACKGROUND The purpose of the performance evaluation of the Strengthening Human Resources for Health (SHRH) Activity was to assess the Activity’s progress and inform the design of future similar activities, with a focus on its theory of change/results framework based on successes, gaps, and bottlenecks identified. The SHRH Activity was implemented by JHPIEGO and received $55 million in funding. The evaluation findings will inform the United States Agency for International Development (USAID)/Ethiopia and the Government of Ethiopia (GoE) at the federal and regional levels and development partners on implementation progress, results achieved, and any policy and strategy implications for human resources for health (HRH) programming in the future. EVALUATION QUESTIONS This evaluation was conducted from February 2018 to April 2018. It generated evidence to answer five evaluation questions: (1) In what ways did the SHRH Activity contribute to improving HRH in Ethiopia and to strengthening the regulatory environment? (2) To what extent did the SHRH Activity interventions facilitate local ownership, coordination and partnerships, and sustainability? (3) To what extent were gender issues addressed in pre-service and in-service training (ISTs)? (4) To what extent have SHRH Activity result/sub-result areas and interventions continued to be relevant to HRH gaps and challenges in Ethiopia? (5) What facilitated or hindered the achievements of planned results? EVALUATION METHODS Quantitative and qualitative methods were used o answer these evaluation questions. An evaluation design matrix was developed and used to guide the generation of evidence and analysis of findings. A document review, 138 key informant interviews (KIIs), 39 focus group discussions (FGDs), and 30 observations were analyzed to answer the evaluation questions. Five regions representing agrarian, pastoralist, and urban contexts were selected for primary qualitative data collection. The first three regions (South Nations, Nationalities, and Peoples [SNNP]; Amhara; and Oromia) represent the agrarian context; the Afar region, the pastoralist context; and Addis Ababa, the urban context. The team visited eight universities, seven regional health science colleges (RHSCs), and one private college. KEY FINDINGS The SHRH Activity has contributed to improving HRH in Ethiopia particularly in the areas of human resource management (HRM) and training quality. Human resources fora have been formed at the national and regional levels to facilitate stakeholder leadership and coordination. Positions for HRM have been created and great strides have been made in filling them. The SHRH Activity also contributed to the increased availability of HRM policy documents at various levels of the health system to support day￾to-day HRM. The Oromia region, in particular, stands out as a region in which significant improvements in the area of HRM were observed. Ethiopia is moving in the right direction in ensuring the availability of different cadres of health professionals. The SHRH Activity’s contributions, however, have been overshadowed by massive government investment and effort in developing human resources in the health sector. These efforts have “crowded-out” all external support, including support from this Activity. SHRH Activity support to establishing IST centers was appropriate but has not yet reached its intended outcome of shifting training from hotels to these centers. Furthermore, IST centers generally have poor infrastructure and service provision compared to hotels, which affects the quality of training provided. Continuing ix Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity professional development (CPD) has not yet been implemented as a means to ensure the quality of professional competence during practice. All stakeholders agreed that SHRH Activity support contributed to improved quality of training through its investments in (i) internal quality improvement systems for training institutions (TIs); (ii) the review and updating of curricula; (iii) the provision of training materials, including skills-lab equipment; (iv) the creation of education development centers (EDCs) within TIs to lead quality improvement within the colleges; and (v) providing staff development training that focuses on the quality of training. Debre￾Tabor University stood out as a TI that was ahead of its peers in moving towards a competency-based curriculum. SHRH Activity research on the program learning activity (PLA) (e.g., capacity assessments) was effective in informing regional health bureaus (RHBs) and the Federal Ministry of Health (FMOH) on how to establish effective structures, guidelines, and strategies. Activity research related to evidence generation largely prioritized publications. There were minimal local ownership and leadership of the research process, which limited the overall influence of the research on HRH policies and strategies. The SHRH Activity supported and made some strides in building the capacity of the two key federal regulatory agencies, the Food, Medicine, and Health Care Administration and Control Authority (FMHACA) and the Higher Education Relevance and Quality Agency (HERQA), to strengthen HRH training and practice regulation. However, efforts made by the SHRH Activity were found to be too limited to bring fundamental change to the regulation process. A double standard in regulation was observed, as the private sector is still being regulated more stringently than the public sector. FMHACA and HERQA have limited capacity to enforce standards set by themselves, particularly within the public sector. The SHRH Activity worked through federal and regional institutions in HRH interventions to foster local ownership, coordination, partnerships, and sustainability. However, the level of sustainability of the SHRH Activity’s interventions varied. Some interventions were highly sustainable (such as the HRM positions created), while others were unsustainable (such as the research and program learning interventions that had limited local ownership). The SHRH Activity contributed to addressing gender issues in pre-service education (PSE). The Activity focused mainly on supporting female students in health science colleges (HSCs) with both life skills and provision of limited financial support. However, at the IST level, the SHRH Activity hardly contributed to addressing gender issues. Three of the SHRH Activity’s result areas (HRM, quality improvement, and research) were found to be relevant to HRH support in Ethiopia now and for the future. Increasing the availability of health workers (HWs) is also highly relevant as support to HRH, but not for external partners, given the focus and commitment of government investment. There were several key factors that facilitated the achievement of planned results. The central government commitment was a particularly key factor in increasing the production of healthcare workers through investing in the expansion of HSCs and enrolling students. At the regional level, the commitment of the Oromia region, for example, was instrumental in serving as a best practice example of HRM in the country. The high demand by stakeholders for quality improvement in TIs and the Activity’s responsive programming therein facilitated strong achievement in this area. The involvement of high-capacity researchers facilitated the publication of research findings in the area of HRH in the country. On the other hand, there were also factors that hindered the achievements of some planned Activity results. The limited involvement of the Bureau of Finance and Economic Development (BOFED) and the Public/Civil Service Commission in some regions, for example, hindered the approval, ownership, and financing of HRM posts at regional levels. In the production of health care workers, low enrollments at level-five training in anesthesia, and slow expansion in anesthesia courses at colleges x Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity contributed to a low achievement in producing trained anesthetists. The double standards in the enforcement of regulations (the public sector is regulated less stringently than the private sector) meant the lower quality of training, especially in public colleges. The limited involvement of local researchers in the actual research (beyond data collection and identification of research topics), limited the more significant potential that the research would have had to influence policy and strengthen local research capacity. CONCLUSIONS The USAID SHRH Activity is the only comprehensive HRH support provided to the GoE by development partners. It responded to the needs of the communities by supporting compassionate, respectful care through improved quality of human resources and HRM. However, the Activity’s influence on increasing the supply of HWs and improving retention and motivation was limited. Strengthening the capacity of training and regulating professionals and provider institutions remain major gaps in the Ethiopian HRH agenda. The Activity effectively brought the HRH agenda to the forefront of regional and federal governments through the establishment of HRM structures, and fora at the federal and regional levels, as well as through the development of HRH guidelines and policies. Even though the number of healthcare workers is increasing faster than the resources and capacity building supported by external partners, the skills composition of the healthcare workers produced so far does not yet adequately cover the diverse needs of the Ethiopian health system. The Activity was the main external technical support provider and a major source of the new thinking behind quality improvements observed in the TIs. This is particularly true in the area of developing new innovative curriculum and the shift observed by a few TIs to competency-based integrated learning. Despite Activity support, regulation of both pre-service TIs and regulation of professional practice during service provision remains weak. HRM structures and increased availability of human resources are priorities of the GoE and are financed mainly by the government, and hence, can be well-sustained by the resources of the government. However, the quality of ISTs, the enforcement capacity of the two regulatory agencies, and research activities cannot be sustained without external support. The efforts towards integrating gender into the Activity design and implementation was limited to pre￾service education. The Activity was effective in establishing gender units within HSCs and providing financial, tutorial, and life skills training support to mainly female students. Ownership, commitment, and leadership among the different levels of management (FMOH, RHBs, and universities, and HSCs) was the major factor supporting performance among the different result areas, regions and TIs. KEY RECOMMENDATIONS FOR POLICY DIALOGUE WITH GOVERNMENT OF ETHIOPIA The major areas where USAID should engage on HRH issues with the government are: (i) Scaling up the Oromia region’s experience of reviewing remuneration packages for HRM professionals and making the HRH forum more functional; (ii) Removing double standards in accreditation and licensing of public and private TIs and service providers and ensuring standardized regulation of all public and private training and professional practice sites; (iii) Developing and tracking quality standards against inputs and enrollment numbers to ensure that there is a match between increasing enrollment and available infrastructure to ensure quality; xi Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity (iv) Developing implementation strategies for and investing in institutionalizing IST operations including continuing professional development; and (v) Advocating for inclusion of the Human Resource Information System (HRIS) as a part of District Health Information System 2 (DHIS2) to foster interoperability. KEY RECOMMENDATIONS FOR THE DESIGN OF ANY FUTURE SHRH ACTIVITIES In the design of any future Activity, USAID should:  Include a clear results framework/logframe, outlining risks and assumptions, the role of the government, and include a sustainability strategy.  Consider designing the Activity to support HSCs to shift towards a competency-based curriculum.  Consider including strengthening leadership and management through provision of skill training and mentoring as part of any future SHRH Activity to enhance ownership and commitment to HRM, EDCs, and competency-based curriculum at all levels.  Focus investments for any future support on third- and fourth-generation universities that have proven themselves receptive to new ideas and can effect greater change.  Consider the inclusion of strategic experts (that know how to influence political actors, know program formulation and how to work in the government system) in addition to HRM technical experts. These strategic experts should assist with charting out the fiscal, political, and management implications of future HRH implementation activities and their sustainability.  Reduce the Activity’s role in increasing the availability of human resources as the government has committed to this.  Include tasks to address improved motivation and retention as an HRM sub-result area.  Include strengthening the regulation of TIs, professional practice, and providers, particularly in the public sector. 1 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity I. EVALUATION BACKGROUND 1.1 ACTIVITY BACKGROUND The Government of Ethiopia (GoE)’s Growth and Transformation Plan (2010/11 to 2014/15) envisages growth in all sectors, including the health sector. Producing better-trained and qualified health system human resources and distributing them equitably among regions has long been a major challenge for the Ethiopian health system. Human resource shortages became critical after major investments were made in the expansion of health facilities (2005–2010) and efforts were made to meet Millennium Development Goals. The Strengthening Human Resources for Health (SHRH) Activity was a cooperative agreement (AID-663-A-12-00008) with a budget of USD $55 million initiated in June 2012 and is planned to end in September 2018. The Activity aims to support government efforts to strengthen the availability, distribution, quality, and management of human resources in the health sector. It is implemented by a JHPIEGO-led consortium that includes Management Sciences for Health (MSH), the Ethiopian Midwives’ Association (EMWA), the Ethiopian Association of Anesthetists (EAA), and the Open University. Before the Activity was initiated, Ethiopia’s human resources for health (HRH) system faced various strategic and competency-related challenges. There was a lack of human resource management (HRM) structures in all regions. Shortages of human resources at all levels was a norm rather than an exception. There was a lack of human resource policy, strategy, plan, and budget. There was also a lack of in-service training (IST) opportunities, and hence, limited staff competence. These problems were further compounded by weak and fragmented partnership, as well as weak human resource information systems (HRIS). The SHRH Activity was designed to address shortages in critical human resource categories, especially at the primary health care level, and to fill gaps in the quality and management of human resources. The Activity also aimed to address the challenges of poor-quality pre-service education (PSE) and IST, rapid turn-over of skilled health care providers, the weak regulatory capacity to sustain HRH quality, and weak HRM. Special focus was given to removing critical human resource shortages for midwives and anesthetists (at an unmet need of 63 percent and 44 percent, respectively) and health extension workers (HEWs), where there were an unmet need and high levels of attrition. The four main result areas and the thirteen sub-result areas are identified in Figure 1. 2 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity Figure 1: Results and Sub-Result Areas of the SHRH Activity (2012 - 2017) The SHRH Activity developed a Performance Management/Monitoring, Evaluation, and Learning (MEL) Plan with targets in 2012. The evaluation team used this plan to evaluate the Activity’s performance against key indicators. Annex 1 presents performance on the major targets set in the MEL Plan. 1.2 PURPOSE OF THE EVALUATION The SHRH Activity evaluation was designed to assess Activity progress and inform the design of future Activities. There was a special focus on the successes, gaps, and bottlenecks identified in the current SHRH Activity. The evaluation findings will inform the United States Agency for International Development (USAID)/Ethiopia and the GoE at the federal and regional levels on the implementation progress, the results achieved, and any policy and strategy implications for strengthening HRH in the future. The users of the findings of this evaluation will be USAID/Ethiopian technical office teams, the GoE at the federal and regional levels, and other development partners that have an interest in strengthening HRH in Ethiopia. 1.3 EVALUATION QUESTIONS The evaluation generated evidence to answer five evaluation questions which are presented in Table 1. Table 1: Evaluation Questions # Evaluation Question 1 In what ways did the SHRH Activity contribute to improving HRH in Ethiopia? Questions should examine the Activity’s contribution to the four result areas. (Effectiveness) How effective were the approaches used to strengthen the regulatory environment related to HRH activities? 2 To what extent did the SHRH Activity interventions facilitate local ownership, coordination and partnerships, and sustainability? (Sustainability) 3 To what extent were gender issues addressed in pre-service and in-service training? (Gender) 4 To what extent have SHRH Activity result/sub-result areas and interventions continued to be relevant to HRH gaps and challenges in Ethiopia? (Relevance) 5 What facilitated or hindered the achievements of planned results? Result 1: Improving Human Resource Management Improved HRM capacity (Federal Ministry of Health, Regional Health Bureaus) Improved HRM motivation and retention schemes Improved HRH policies and practices Enhanced HRH fora Improved management of staff training Result 2: Increased Availability of Midwives, Anesthetists, HEWs, and Other Essential Health Workers Increased availability of midwives and anesthetists Skills of HEWs enhanced through blended career development schemes Trained HEWs available for replacement schemes PSE of essential non￾clinical health workers promoted Result 3: Improved Quality of Training of Health Workers Improved quality of PSE of health workers Improved quality of IST of health workers Result 4: Program Learning and Research Conducted Research and evaluation evidence on HRH issues generated Local capacity on HRH M&E and research built 3 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity 2. METHODS, SAMPLING, ANALYSIS, AND LIMITATIONS 2.1. METHODS This evaluation was conducted from February 2018 to April 2018. The evaluation team applied the evaluation criteria of effectiveness, sustainability, addressing gender dimensions, and relevance to answer the five evaluation questions as presented in the evaluation matrix (Annex 2). Given that the SHRH Activity was planned to work through government systems and was mixed with government investments, the team used contribution analysis as a method as it was difficult to isolate and attribute the direct impact of Activity support. In keeping with the Statement of Work (SOW) (Annex 3) for this evaluation, the evaluation team used a mixed-methods approach for data collection and analysis, aligned to the evaluation questions. The mixed-methods approaches included document review, key informant interviews (KIIs), focus group discussions (FGDs), and physical observation of training facilities. The overall approach is described below, in Figure 2. Figure 2: Four Steps of the SHRH Activity Performance Evaluation The evaluation also documented the challenges faced in implementing the Activity and identified priority action-oriented recommendations for informing a future design and implementation of a similar Activity. Consideration was given to partnerships, coordination, and improvement in the HRH regulatory environment. For each result area, and for each evaluation question, the evaluation generated evidence on the following:  Innovative approaches in implementing the Activity  drivers of success or reasons for not meeting targets  Role of the federal and regional governments (i.e., ownership, facilitation, guidance, partnership)  Role of other implementing and development partners (i.e., coordination)  Challenges encountered during implementation of the Activity •National and regional visits and data collection using mixed methods (quantitative data, KIIs, FGDs, observations) •Context updates •Preliminary feedback to USAID on the findings (PowerPoint) •Analysis of secondary and Activity data •Triangulated analysis of qualitative data •Desk review •Development of tools and Evaluation Design Matrix •Develop sampling framework •Stakeholder communication •Logistical arrangements •Compilation of draft report: feedback and finalization •Revision of the finalrReport based on the comments provided •Two-page summary Step 4 Report + Follow Up Step 1 Inception Phase Step 2 Data Collection Phase Step 3 Analysis 4 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity Evaluation Design Matrix Annex 2 presents the evaluation matrix developed and used to provide a structured framework and guidance for the evaluation team throughout all stages of the evaluation process. It identifies the five evaluation questions and specifies the methods, tools, and data sources used to answer each evaluation question. 2.2. SOURCES AND ANALYSIS OF INFORMATION The results of the Activity progress towards achieving the intended outcome and output indicators were collected from secondary sources. The main sources of data were the Activity database, Federal Ministry of Health (FMOH) reports, and the data generated from FMOH health and health-related indicators document, which is published annually. The team also reviewed recent surveys and studies, including the Service Provision Assessment (SPA) and the Service Availability and Readiness Assessment (SARA) to explore how well the health facilities are staffed with the necessary human resources. The team verified routine data sources during interviews and field visits using information collected in the KIIs, FGDs and observations at the Federal level, regions and training institutions. Document Review Documents collected and reviewed included: (a) all relevant GoE, USAID/Ethiopia, and SHRH Activity documents, including background documents; (b) the health sector transformation plan; (c) the HRH strategic plan; (d) the Activity MEL Plan; (e) annual plans; (f) annual reports; (g) research papers produced by the Activity; and (h) other relevant HRH studies produced. Best practices documents from training institutions (TIs) and regional health bureaus (RHBs) as well as progress reports were also collected at regional levels. The secondary document review was national in scope and was triangulated with secondary data from the sampled regions. The team also accessed and reviewed the different HRM tools developed and used in the regions. A list of all the relevant documents appears in Annex 4. Key Informant Interviews The evaluation team developed and used detailed KII guides covering the specific evaluation questions for the federal and regional levels as well as training institutions (TIs) supported by the Activity. These included specific questions for the federal, regional, zonal, and woreda health workforce managers; university and regional health science college (RHSC) managers; and development and implementing partners (see Annex 5 for a list of key informants). Information from KIIs was used to complement quantitative findings, and to draw lessons on the successes and weaknesses of Activity implementation. The interviews were also used to identify the main achievements of the Activity, lessons learned and best practices, challenges, and recommendations for similar activities in the future. The KII guides are attached in Annex 6. Focus Group Discussions FGDs were conducted in the sampled TIs with trainers/instructors, trainees/students, and alumni of the three major professions (HEWs, midwives, and anesthetists) to identify progress, gaps, and challenges for both the short term and the long term as well as perceptions about the performance of the Activity in improving the quality and readiness of health workers (HWs) to provide responsive services. An FGD guide appears in Annex 7. Physical Observations The evaluation team undertook direct observations of clinical practice sites, skills labs, and IST centers in HSCs and universities. The observation guide shown in Annex 8 structured the process. 5 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity 2.3 ANALYSIS METHODS The performance evaluation used contribution analysis to compile the evaluation findings and conclusions. The analysis built the evidence on the contributions made by the SHRH Activity while highlighting the importance of other influences on the outcomes. The team used two approaches to analyze data during the evaluation: (i) undertook different types of quantitative analysis (trend, percentages, and shares) on secondary data obtained from the Activity, FMOH database, and other document reviews; and (ii) used thematic analysis for qualitative data generated from KIIs, FGDs, and observations. The analysis and write-up of the findings, conclusions, and recommendations were organized around the evaluation questions and result areas. Whenever it was appropriate, data were disaggregated by sex, region, and other relevant variables. 2.4. SAMPLING The SOW specified that the team should sample at least two agrarian regions, one pastoral region, and one urban region. The team used a purposive sampling technique with the aim to include regions and institutions considered by the implementing partner as having demonstrated best practices and weak performance. The evaluation team used the Activity performance ranking of regions, universities, and RHSCs (as low or high performers) made by JHPIEGO for its sampling. The Activity also ranked regional TIs as high and low performing, which was used as a sampling frame. From this sampling frame, South Nations, Nationalities, and Peoples (SNNP), Oromia, and Amhara regions were selected to represent the agrarian regions, while Addis Ababa was selected to represent the urban context, and Afar to represent the pastoralist context. The justification for the selection of these regions is presented in Annex 9. Except in Oromia, the evaluation team selected one strong and one weak TI per region based on the rankings of JHPIEGO. In Oromia, only Jimma University was visited due to time constraints. The sampling was carried out with the goal of reducing travel time from one site to another. The sampled TIs visited are presented in Figure 3. Figure 3: Training Institutions Visited 2.5. TEAM COMPOSITION AND ORGANIZATION The team had five consultants and one member from the staff of the Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity. This team was divided into two sub-teams with three individuals per team. Within each team, one person was tasked with undertaking the interview, another team member was tasked to take notes, and the third member assisted in both interviewing and note￾taking. Sub-team 1 traveled to Afar and SNNP and Sub-team 2 traveled to Amhara and Oromia and conducted interviews in Addis Ababa. SNNP •Wolkite University & Arba Minch University •Hawassa and Hossalina & Arba Minch HSC Amhara •Debre Tabor University & Bahir Dar University •Debre Tabor & Bashir HSCs Oromia •Jimma University Addis Ababa •Menelik RHSC, Addis Continental & St. Paul Millennium Afar •Semera University •Semera RHSC 6 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity 2.6. LIMITATIONS Four main limitations to the evaluation were identified as follows: 1. The Activity program description did not have a full-fledged logical framework, and its results framework lacked a detailed description of the risks and assumptions. However, the Activity did have a MEL Plan with associated targets for indicators. The team used these MEL targets for evaluating the Activity performance. 2. The available time constraint required the team to purposively sample regions (rather than visit all the regions in the country) that would represent regions in which the Activity was implemented. Thus, this might have an impact on the rigor of the findings. However, to mitigate this limitation, the evaluation team designed the selection criteria to ensure comparability of the selected regions and institutions with those that were not selected. 3. Quantitative data were obtained only from secondary sources, particularly from Activity’s performance monitoring reports. A survey was not required in this evaluation, therefore, the findings reported are largely qualitative descriptions of the Activity results. 4. Qualitative data collection is by nature a subjective form of data collection. Therefore, results are based on the opinions of the persons interviewed, which may not accurately reflect the views of other people involved with the Activity who were not part of the evaluation. 7 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity 3. FINDINGS ON THE ACTIVITY EFFECTIVENESS Evaluation Question 1: In what ways did the SHRH Activity contribute to improving HRH in Ethiopia? 3.1 HUMAN RESOURCE MANAGEMENT (ACTIVITY INTERMEDIATE RESULT [IR] 1) The Activity aimed to improve HRM through strengthening of the Directorate of HRM and RHBs by developing staff requirement plans, filling HRM positions (100 percent), undertaking 15 capacity assessments, providing training on HRM and establishing HRH forums (12). Table 2 presents the Activity’s performance against the targets as detailed in the MEL Plan. Of the five MEL indicators, three were achieved. Targets for development of staff requirement plan and filling HRM posts were not met. According to the secondary data obtained, which was subsequently verified by the evaluation team, the Activity delivered effectively on capacity building of HWs, having achieved 106 percent of the target in this area. The establishment of 12 HRH fora at the regional and federal levels contributed to realizing the set MEL target through improved involvement and ownership of the HRH agenda by stakeholders and strengthening the leadership capacity of RHBs and FMOH. The evidence gathered during field visits shows variations in the performance of the fora across the regions visited, particularly regarding a commitment to meeting schedules and internal communication among stakeholders. Regarding HRM positions, the Activity achieved 53 percent of the target. Moreover, the information gathered from verified secondary documents, which were obtained from the implementing partners, indicated that 44 percent of HRM positions were yet to be filled nationwide. Table 2: Achievement of HRM There was clear evidence from FGDs that the Activity contributed to improving awareness and understanding of the importance of HRM as a profession within the health sector, especially at the federal and regional levels. This opinion was consistent in all regions visited, although the level of commitment for translating understanding to action varied when examined regarding budget allocation for HRM unit operations. There was also increased commitment of regional governments for creating 1 The target and achievements provided in the MEL document to the evaluation team did not disaggregate data as was the case in the baseline. MEL Indicators Baseline Target Achieved % % of organizations with staff requirement plans 77% 100% NA NA % HRM post filled with qualified staff1 0% FMOH, 100% RHBs, 58% ZHD; 15% WorHOs 100% 53% 53% No. of HRM capacity assessments conducted 0 15 15 100% No. of HWs that successfully completed IST in HRM 0 2,560 2,717 106% No. of functional HRH forums 0 12 12 100% “The number of personnel￾related issues and complaints being brought to RHB has significantly diminished as they are mostly being addressed at hospitals and zonal levels.” RHBs, Oromia Region 8 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity decentralized HRM units and approving increased numbers of HRM positions. Analysis of information gathered across regions visited depicted better utilization of manpower due to decentralized HRM units; health professionals now have better access to personnel related services within the health facilities, woredas, and zones, thereby allowing more time and energy at their duty posts. Table 3: Ratio of HRM Positions to Health Workers by Region Due to variations in the establishment of HRM positions and structures among regions, the ratio of approved HRM positions to HWs also varied (see Table 3). The HRM personnel in the Tigray region are dealing with the highest number of HWs (ratio of HRM to HWs 1:145. Also, approval for new positions in Amhara has yet to be finalized, as the document has been presented to the cabinet for consent. Despite creating new HRM positions, regions have difficulty filling them. This was consistent across all the regions visited and was confirmed during FGDs with FMOH Humanitarian Requirements Document (HRD) personnel. The major reason for this was that there is limited availability of qualified candidates in the labor market and the remuneration package is not enough to attract candidates. Overall, about 43 percent of the positions across the regions and FMOH had not been filled at the time of the evaluation. For instance, 80 percent of the HRM positions created in Amhara had been filled, whereas in Oromia, where the most HRM positions (2,675) were created and where better payment was offered, 58 percent of the positions had been filled with qualified candidates, 600 of whom were degree holders (see Table 4). According to data gathered from FGDs in the regions visited, better awareness of HRM functions within the health system was reported, evidenced by the larger proportion of sites (Hospitals, RHBs, FMOH) having budgetary allocations in place for HRM operations (beyond salaries for personnel). For instance, data obtained from the Bureau of Finance and Economic Development (BOFED) Amhara showed that 15 percent of the budget was allocated to the health sector and a larger portion was spent on HRM. The findings of this evaluation revealed that the commitment of the government to fill HRM positions Region # of Health Workers (2017) Approved HRM positions Ratio Addis Ababa 16,778 700 1:24 Afar 5,244 158 1:33 Amhara 43,397** 1,593 1:27 Beneshangul Gumuz 3,327 215 1:15 Dire Dawa 1,693 20 1:85 Gambela 2,826 158 1:18 Harari 1,242 43 1:29 FMOH 5,937 135 1:45 Oromia 48,782 2,675 1:18 SNNP 33,347 1,528 1:22 Somali 6,414 414 1:15 Tigray 14,372 99 1:145 **The Amhara figure was obtained directly from RHB because FMOH data on Amhara is not consistent. “The Activity helped a lot to make HRM a focus area and bring about improvements, which would have been impossible without it.” Director, Amhara Region 9 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity varied by region. The number of new HRM positions approved and filled since the Activity support began was 2365 and 1308 respectively in Oromia which was the highest among regions to the lowest of 125 and 116 in FMOH respectively (see Table 4). Looking at the federal and all regions visited by the evaluation team, a total of 5623 new positions were approved, and 3675 (65 percent) positions were filled. Table 4: Total Number of Human Resource Management Positions in the Sample Regions Combined: Health Centers, Hospitals, Woredas, Zonal, RHB) Regions Baseline (Human Resources [HR] Positions) Status (HRM positions) Difference between baseline and current HRM positions Approved Occupied % Approved Occupied % Approved Occupied FMOH 10 9 90% 135 125 93% 125 116 Oromia 310 239 77% 2,675 1,547 58% 2365 1308 SNNP 631 386 61% 1528 887 58% 897 501 Amhara 199 14 7% 1,593** 1,281 80% 1394 1267 Addis Ababa 9 5 56% 700 359 51% 691 354 Afar 7 5 71% 158 134 84% 151 129 ** Numbers do not include the newly created positions that are yet to be approved by the regional cabinets Figure 4 below shows variation between HRM positions that have been approved against those that are currently filled (occupied) at FMOH, RHB, Zonal and Sub-city, whereas Figure 5 shows HRM positions in hospitals and health centers by regions, due mainly to the shortage of budget to fully employ.2 2 The rationale for presenting information separately is because the SHRH Activity initially excluded Hospitals & Health Centers from its support. 10 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity Figure 4: HRM Positions in FMOH, RHBs, Zonal and Sub-city (All Regions) Figure 5: HRM Positions in Hospitals and Health Centers (All Regions) According to evidence obtained during FGDs with the FMOH HRD team, the Activity interventions have helped to inform the government on the importance of a strong and functional HRM system and have also changed government attitudes about the critical need of competent and qualified HRM professionals at different levels. Similarly, across the regions visited, there were clear indications of improved recognition for HR managers. For instance, in Oromia, HRM positions have now been included in the category of technical roles. As a result, benefits such as duty allowance have been extended to HR managers; which will motivate HRH managers to fill positions. Regarding performance management and staff motivation, the Activity built the capacity of 2717 HRM personnel on administration/application of the balanced scorecard – a system which the Ethiopian government uses to manage the performance of its initiatives. Some of the HR managers interviewed perceived the training and supportive supervision provided by the Activity to have contributed 621 119 1278 156 14 101 26 1582 835 216 84 Addis Ababa Afar Amhara Ben-Gum D.D Gamb Har Oromia SNNP Somali Tigray HR Position Available HR Position filled 79 39 315 59 6 135 57 17 1093 693 198 15 Addis Ababa Afar Amhara B-G D.D FMoH Gambella Harari Oromia SNNP Somali Tigray # of HR positions available # of HR positions filled 11 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity immensely to improving their performance. Evidence gathered during FGDs with HRM teams in the regions showed that capacity building component of the Activity played a critical role in creating new directives and strategies for organizational HR planning, procedures, and practices including a guide for exit interviews when employees leave their positions, new hire orientation guides, and operations manuals. This intervention was being implemented in all the regions visited and has contributed to improved efficiency and effectiveness in overall HRM operational activities. The establishment of the HRH forum is perceived, by key informants, as another significant intervention of the Activity because it created a platform for collaboration and greater partnership in the health sector across HRH issues. Some evaluation participants believe the forum was a good initiative as a strategic approach to addressing the HRH agenda. However, the level at which the fora function varied across the regions visited, regarding following up on strategic issues. The fora have also served as the platform for presenting HRH issues and improving collaboration between RHBs and TIs. In some regions, such partnerships have yielded better working relationships between two pillars of the health sector (i.e., service providers [hospitals] and TIs). The fora have been helpful in improving engagement among other government agencies such as education bureaus and civil service commissions. Some of the results include HRM decentralization to zonal and woreda levels, integration of services and training, and collaboration to address the internal migration of staff (professionals). Overall, the regularity of fora meetings has been inconsistent with most regions visited not achieving their meeting frequency targets (Table 5). Table 5: HRH Forum Meetings Despite its achievements, the fora were perceived by some key informants as non-inclusive of some key government agencies such as the Civil Service Bureau, and weak regarding influencing policies in some regions. Although its membership includes key relevant government agencies, its operations in engaging those agencies have been passive, causing dissatisfaction and loss of confidence in the forum. Some key informants interviewed also expressed the capability of the forum to influence policy issues is limited. Moreover, its focus was perceived to be more on technical areas rather than on policy and strategic areas, which could be addressed without such a platform. The FMOH HRD Directorate, for instance, clearly stated that there is a need to ensure the forum has key decision makers outside the FMOH, and TIs to ensure it drives policy and strategic agendas. In this regard, the Oromia Region is making the best use of this forum to address HRH issues and other health sector challenges. Conclusions The Activity’s contribution in improving retention and motivation was very minimal because the government, through its public service agenda, drives job grading and sets national levels of payment. Moreover, the Activity emphasized public sector approach to staff performance recognition, which, according to 71 percent of the KII respondents, is outdated and needs to be reviewed. Although no innovative approaches were deployed by the Activity to address the motivation of HWs, an exception was found in the Oromia region, where the leadership made concerted efforts to address such issues. The Activity was credited for providing technical support for those efforts. The process led to the identification of real factors that enhance staff motivation and improve retention rates, which have been sent to the cabinet for approval and could soon be implemented across the region. Implementing this Region Frequency of Meetings Number of Meetings Held in the Last Year Afar Bi-annually 1 Amhara Bi-annually 2 FMOH Quarterly 3 Oromia Quarterly 4 SNNP Bi-annually 1 “We have not received any communication regarding the HRH forum in over three years.” City Civil Service Bureau, Addis Ababa 12 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity motivation and retention scheme is estimated to increase the annual budgetary commitment to 300 million Ethiopian Birr (ETB) (USD 10.8 million), and the RHB has already made provisions for it. 3.2. INCREASED AVAILABILITY OF HUMAN RESOURCES (ACTIVITY IR 2) This result area of the SHRH Activity aims to increase the availability of midwives, anesthetists, HEWs, and other essential HWs to support the GoE’s priority of increasing the availability of mid-level HWs in the country. The Activity was initiated in an environment in which government-led expansion of higher TIs was already booming, both regarding infrastructure and enrollment, as demonstrated by: (i) The number of public institutions of higher education grew from eight to 36; (ii) A rapid expansion of private institutions of higher education; wherein 2012/13 the number of private medical and HSCs reached 45 from about 26 in 2000; and (iii) The expansion of medical and health science TIs also increased, with the number of midwife TIs increasing from five in 2000 to 46 in 2012. This expansion of higher education institutions created the space that allowed for a rapid increase in enrollment in both undergraduate and postgraduate programs. Overall, the MEL targets set by the Activity under HRH production are on course to be achieved: eight of the 11 indicators have achieved at least 75 percent of their targets. The Activity managed to surpass targets on four indicators, achieved more than 80 percent on 3, and reached between 65 and 76 percent on another three. However, the rate of HW certification and training, especially of anesthetists, falls short of the population ratio targets outlined in the MEL Plan (see Table 6 and refer to subsequent paragraphs for details). Table 6: Performance of the SHRH Activity in Increasing Availability of Targeted HWs Indicators Baseline Plan Achievement Ratio of midwives to population 0.56 1.99 1.48 (74%) Number of new midwives graduated 4,709 14,419 12,633 (88%) Ratio of anesthetists to population 0.03 0.17 0.07 (41%) Number of anesthetists graduated from PSE institutions 252 1,415 1070 (76%) Number of healthcare workers who successfully completed in-service training (IST) 0 3,875 4,136 (107%) Percentage of HEWs who advanced to higher level 5% 54% 44% (81%) Number of new HEWs who graduated from a pre‐service TI 0 8,230 10,532 (128%) Number of curricula developed/updated 0 22 40 (182%) Number of new health care workers who graduated from a pre‐service TI 0 38,891 37,159 (96%) Number of healthcare workers who completed a postgraduate program 0 90 62 (69%) Number of new health care workers who graduated from a pre‐service TI 0 1,238 1403 (113%) A significant area of focus for the Activity was strengthening the availability of mid-level health workforce in the Ethiopian health system. By strengthening the infrastructure and capacity of 52 public higher TIs, the SHRH Activity contributed to the government’s ability to meet the targets set for increasing the availability of midwives, HEWs, anesthetists, and other essential HWs. Furthermore, curriculum development surpassed the target (182 percent) because the Activity was responsive to the government’s emerging requests and demand for new and revised curricula. The Activity is considered as a center of excellence in the Ethiopian context for curriculum development. The major role the Activity played in contributing to the increased availability of HWs has been through: (i) increasing students’ interest in enrolling in a midwifery program through different forms of advocacy and promotional activities; (ii) provision of special support to TIs (public universities and RHSCs) as a 13 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity part of expanding the training of anesthetists; and (iii) support to open new pre-service training programs of emergency medical technicians and paramedics, biomedical technology, and HRM/health economics postgraduate training. In this regard, the SHRH Activity supported the pre-service education TIs to conduct a capacity assessment, to expand anesthesia training and with the provision of materials to select TIs to establish anesthesia skills labs. For example, the TIs and HSCs sampled stated that the Activity played a major role in establishing skills and built the capacity of faculty members and lab assistants through the provision of a technical training and mentorship program. The Activity also supported the faster implementation of different training programs by providing support for the development of curriculum reviews (see Table 7). Table 7: Curricula Developed/Updated with SHRH Activity Support for the Three Types of essential HWs, 2018 The Activity also supported the training of essential HWs across the country. To this end, 1,403 (113 percent of the target) essential HWs (emergency medical technician (EMTs), biomedical technicians, and nursing specialty)3 graduated from the supported pre￾service education institutions. Similarly, the Activity supported postgraduate training of 62 HRM/health economics professionals in Gondar University, Jimma University, Addis Ababa University and Addis Continental Institute of Public Health, which achieved 69 percent of the planned target In general, the SHRH Activity supported 52 TIs to increase their capacity to train HWs through skills lab development, faculty development (benefiting more than 90 TIs both private and public TIs), developing and distributing teaching materials, curriculum development (40 curricula developed for different HRH types including those shown in Table 7 – achieving 182 percent of the target) and supporting clinical training by donating 20 buses for select TIs to transport students to clinical training sites). The main reason for this over￾achievement was the higher than expected volume of requests by FMOH and HSCs for curriculum development support. The Activity responded favorably to these requests, which led to over-achievement of planned targets in this area. A total of 50 IST centers in different regional states were established by the GoE with the support of the Activity that allowed 4136 HWs to participate in different kinds of IST – achieving 107 percent of their target. An IST is 3 The evaluation team was not able to collect disaggregated training data for these three professional categories. Curricula Developed/Updated Number Midwifery Curricula Level IV 1 Bachelor of Science (BSc) 1 BSc upgrading of midwives 1 National harmonized modular BSc 1 Master of Science (MSc) and doctor of philosophy (Ph.D.) 2 Anesthesia Curricula Level V 1 BSc (generic, post-basic) 2 MSc 1 HEW Curricula HEW III and IV 2 Urban HEW level III & Level IV 2 Total 14 Box 1: Collaboration Learning and Adapting (CLA) opportunity to increase production of essential health workers Ethiopian universities typically offer little postgraduate distance learning but with the help of the Open University which worked with four universities to launch a postgraduate HRM and health economics training program, new opportunities have been created. With technical support and mentoring by Open University faculty members, Ethiopian academics and practitioners created the content by drawing on the Open University’s existing distance learning materials. Subsequently, having received training in how to write blended learning materials, the Ethiopian faculty applied their new skills to author drafts of modules that are part of both programs. Bringing subject matter experts from the Open University to co-teach the HRM and health economics courses created knowledge and skill transfer opportunities for Ethiopian academics on quality blended learning practices. 14 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity an in-service training center established with the support of the Activity to shift hotel based in-service training into institutional-based training in HSCs. The curriculum is developed mainly by the FMOH. The Activity supported IST centers by donating the necessary required materials such as furniture, air conditioners, and other essential materials. Additionally, technical support was provided to develop different IST curricula (e.g., basic emergency obstetric and newborn care training curricula). The Activity has also provided technical support to the FMOH to review and standardize the different IST training packages, including Nursing Care Standards, National Comprehensive HIV Care Training for Pharmacy Professionals, Oxygen Therapy in Children and Adults, and many more. It is important to note that the feedback from KIIs at all levels shows that the contribution of the government and other partners (i.e., Columbia University’s ICAP and I-TECH) was significant during the establishment process of the IST centers. These contributions were mostly related to capacity building through material support such as the provision of additional furniture, computers, video conference equipment, LCD/projectors, etc. The existence of IST centers is an important first step. However, the standardization and institutionalization of IST are not yet well in place. The IST centers training quality remains nascent due to inadequate infrastructure (quality training facility, hostels, and good food) and low motivation of trainers. For example, among those training centers visited by the evaluation team, only Menelik HSC, and Hawassa University have full-time staff managing the ISTs. The remaining are being managed by people with other responsibilities; moreover, the IST is recognized as one of several formal management structures only in these two TIs. In other institutions, the task of IST coordination is allocated to staff and combined with other primary responsibilities. Given that these IST centers have limited infrastructure of their own, they still depend on hotels and external venues, which limits their effectiveness. The exceptions are reported to be Hawassa and Oromia, however, Jimma University and Harar RHSC are in the process of building their own centers. Currently, the savings from ISTs are limited to conference hall rentals and cost of refreshments and supplies, whereas the quality of training is compromised due to lack of infrastructure and weak motivation and commitment by the top management of the different TIs. The total health workforce density in the country increased substantially over the past five years as indicated in Figure 6. Figure 6: Health Workforce Density in 2012/13 and 2017 in Ethiopia4 4 The evaluation team used WHO indicator when showing workers per population. We used per 10,000 for specific cadre and per 1000 for overall HR density. In keeping with the indicator stated in the MEL Plan of the Activity, the team used the same denominator to examine the extent of change. 0.56 0.03 0.97 1.48 0.07 1.63 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 Midwifery/10,000 pop. Anesthetist/10,000 pop. Overall/1000 pop. 2012/13 2017 15 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity Despite the numbers reported in Figure 6, the number of health professionals reported by the Activity contradicts the official FMOH data, as reported in the recent national health indicator (2017) report. For example, the number of midwives in the FMOH report is almost half (7275) of the number reported by the Activity (14419), which gives a density of 0.8/10000 population (by far lower than 1.48/10,000 population indicated in the Activity report). As all key informants agreed, such discrepancies are a result of the lack of a functional HRIS, both at RHBs and FMOH, which creates challenges around getting reliable HRIS data. As a result, the evaluation team relied on the Activity report, and the evaluation team was unable to verify the figures reported by the Activity with the FMOH HRIS data. Nevertheless, despite the progress and efforts made by the government, the health professionals to population ratio is still behind the World Health Organization (WHO) benchmark of 2.3 per 1000 population for Sub-Saharan countries. However, some regions in Ethiopia meet or surpass the WHO benchmark, namely Benishangul Gumuz, Gambela, Hariri, Addis Ababa, and Dire Dawa (Figure 7). Figure 7: Health Professionals per 1000 Population (2013-2016) . Stakeholders interviewed in all regions confirmed that there is an acute shortage of pharmacists and laboratory technicians/technologists nationwide. For example, in Oromia, 25 percent of health centers are functioning without these professionals. To address this shortage, some regions, like Oromia, designed a strategy to fast-track the training and certification of this cadre to resolve the current scarcity. Within a year, the region was able to revise their curriculum, agree with the TI to cover the cost of the training and since April 2018, they have just enrolled close to 800 students for each of the two professional categories. In this regard, the commitment of the Oromia regional health bureau and existence of strong HRM was the major driver of this action to prioritize training of pharmacists and laboratory technicians. Region￾specific shortages of HWs were also observed in other areas during this evaluation. There is, for example, an acute shortage of environmental HWs in Afar. There is an acute shortage of anesthetists in all regions, where according to key informants in this evaluation, shortages are largely due to inadequate capacity of TIs, unsatisfactory anesthetist training expansion and inadequate enrollment of students. Moreover, programs designed to increase the “As FMOH we really feel outraged regarding human resource data. Honestly speaking, at present we don’t have reliable health workforce data which is a disaster for FMOH” HRM and HRD Directorates, FMOH, Addis Ababa 16 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity availability of anesthetists, such as a Level V training, have made little contribution because of the lack of demand from students, resulting in inadequate enrollment levels. For example, Hossana Health Science College already closed its program, in less than three years, due to inadequate enrollment. As a result of the lack of demand from students for Level V training, there was a slow expansion of anesthesia programs in regional HSCs and public universities. The Activity target did not adequately consider the annual number of anesthetists graduating from the existing TIs, the anticipated pace, and expansion of the anesthesia training program in the country and the extent of annual population growth. These factors contributed to the low performance on the anesthetist to population target (achieved only 41 percent of the target). The HRH Strategic Plan of the government focuses on the rapid increase of health professionals being trained, sometimes referred to as the “flooding” strategy. However, while the government has expanded the number of higher TIs, their infrastructure, teaching materials, and faculty have not grown equally to take on the increased number of enrollees. Therefore, the increase in the number of the enrolled students into health and medical science schools created a mismatch between student ratio and available resources, which is critically affecting the quality of training as evidenced by different key informants. Conclusions While increasing the availability of the health workforce was one big element (result area) of the SHRH Activity, it was difficult to determine its direct contributions to increasing availability of targeted health cadres as stipulated in MEL Plan. The difficulty was mostly because the government has invested heavily in the expansion of these higher TIs in the country and is making decisions on the number of enrollees into health and medical science programs. The visible contribution of the Activity was observed in the interest creation among enrollees and increasing the speed of initiating new programs through curriculum revision/development and support. 3.3. IMPROVING QUALITY OF EDUCATION (ACTIVITY IR3) The Activity’s program design and MEL Plan identified pre-service and in-service training quality as an essential element in supporting the GoE HRH strategic plan. Initially, the Activity’s priority was to ‘implement innovative strategies to improve the quality of health education for all cadres, with a focus on midwifery training,’ however, as the Activity moved into the implementation phase, the focus was expanded to involve addressing the needs of other cadres including anesthetists, nurses, and doctors. Table 8 summarizes the performance of the Activity against targets in improving the quality of education. The Activity, as shown in the table, has been able to meet or exceed MEL targets in this area. There were three key factors that led to high (and over) achievement in this area. First, there were frequent demands and request from the TIs and government for support to develop and review curricula, policies, and regulations. Second, there was a high level of acceptance and support for the Health Science Education Centers (HSEDC). Finally, the TIs and government commitment to shift hotel-based to institution-based IST enabled high numbers of cadre to be trained. Table 8: Performance against MEL Targets Baseline Target Achieved % Number of policies, regulations reviewed and/or developed 0 7 30 429% Number of TIs with internal quality improvement 3 50 50 100% Ratio of academic staff/students 1:24 1:20 1:14 142% Number of curricula developed/reviewed 0 22 40 182% Number of Health workers that completed an IST in Standards￾Based Management and Recognition (SBM-R) course 0 5,904 8,806 149% 17 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity Number of IST centers established 0 35 50 143% Number of HSEDC established 3 47 47 100% With Activity support, several universities began to make the shift from conventional curriculum to competency-based curriculum. Debre Tabor University has already implemented this new curriculum (see Box 2) while Jimma University is planning to make this shift during the next enrolment cycle. The FMOH and Ministry of Education (MOE) have also adopted a competency-based curriculum to be implemented at the TI when the TIs are ready. One of the interventions supported by the SHRH Activity is the establishment of 47 Health Science Education Development Centers (HSEDCs) at HSCs. These HSEDCs aim at supporting faculty development, quality improvement in teaching and learning as well as revision of curricula. The Activity supported the TIs to improve the technical and managerial skills of the Centers’ faculty. More than 8800 faculty members in these colleges were trained through Activity support, according to the data generated from Activity reports. KII faculty member respondents noted that the HSEDC training was particularly useful to help faculty gain more knowledge in the areas of effective teaching skills, gender￾sensitive pedagogy, skills-lab teaching, and preceptor teaching at clinical facilities. Moreover, about 40 curricula were revised, and internal quality improvement programs were established in 50 TIs. KIIs at the FMOH and TIs reported that the Activity had become the major center for curriculum development and revision. The Activity supported the national accreditation and quality improvement mechanisms of the Higher Education Relevance and Quality “The health science colleges are more orderly and advanced in moving towards better quality than other colleges, as a direct result, of the Activity support, such as support on education development centers (EDCs), Skills Lab, starting licensure, and faculty development.” FMOE and HERQA, Addis Ababa Box 2: Debre Tabor University: a center of excellence for pre-service education The evaluation team identified Debre-Tabor University as a center of excellence: It has shifted from conventional curricula to competency-based/problem-solving curricula (In international pedagogical practice, this is considered a superior pedagogy compared to conventional curricula being used in other universities in the country). It has integrated the three basic areas of healthcare training (biomedical, clinical, and community) from the first year of training to the final year of training, ensuring that students have access to clinical training and community training, from year one. It has institutionalized EDCs in all colleges of the University, learning from the experience of the HSC. It invests heavily in the hospital, where its students are being trained, including investments in infrastructure development, to ensure that facilities are available for quality training students. The bed to student ratio of clinical training is reported to be on average (one bed to six students). The University leadership is highly committed to learning and teaching and demonstrates this commitment both financially and in strategic thinking. It invests heavily on equipment for its highly developed skills-labs, (about seven million ETB /$252,000 invested to purchase equipment). The major drivers of the success in the university are the commitment and ownership of the curriculum by both the university and hospital; the commitment of the university leadership to invest its own resources as well as the advocacy; and the technical support provided by the Activity. There is also adherence to and respect for the memorandum of understanding (MOU) by the University and the hospital.      18 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity Assurance Agency (HERQA). HERQA is the federal government agency with a mandate to support and regulate quality and continuous improvement of education at higher institutions of learning. HERQA processes reinforced the internal quality assurance process of TIs. In addition to the training curricula, quarterly coaching and mentoring visits were provided by the Activity to strengthen the technical and managerial skills of the HSEDCs. The Activity also supported the Federal Ministry of Education (FMOE) Supervision Directorate, to encourage increased supervisory visits to the HSEDCs with an established HSEDC supervision check-list. The FMOE is also planning on developing health science technical standards, to be applied across all categories of colleges. This plan is in its infancy, and the Activity has not yet been invited by the Ministry to contribute substantially to the process. Activity resources have been utilized to improve and expand facilities for clinical training of students as evidenced by the observations of the evaluation team. For example, equipment in skills-labs5 that help prepare students for clinical practice was provided by the Activity in areas of midwifery, anesthesia, and nursing. The Activity also supported TIs to develop MOUs with clinical facilities, which strengthened the commitment of these clinical facilities to provide clinical training sites for the students. Information technology (IT) was provided to training sites by the Activity. The computers and other equipment contributed to the improved quality of training in HSCs. This was noted in KIIs especially with the FMOE and HERQA officials. The Activity supported the FMOH to adapt international courses to the Ethiopian context. Newly developed training manuals included, for example, Critical Incidents and Basic Life Support, Nursing Care Standards and Pharmaceutical Distribution Management. The Activity supported the development of a licensure examination for health professionals as a tool to ensure the quality of the HWs that are being newly deployed. The evaluation team found that the Activity supported the development of national continuing professional development (CPD) schemes for different categories of HWs. While CPD is not yet mandatory, the CPD guideline is in place to prepare the health sector for future compulsory CPD requirements. The CPD implementation strategy, including the cost and the mechanisms of financing at the federal, regional, zonal, and woreda levels are yet to be defined and instituted. The Activity has also supported the transition of the national CPD Accreditation Committee from the Food, Medicine, and Healthcare Administration and Control Authority (FMHACA) to the FMOH. While the Activity surpassed all of the MEL targets for improving the quality of training, there were some challenges faced that were reported through the KIIs and FGDs. These included:  Resistance from the conventionally-trained trainers (as evidenced initially even at Debre Tabor) to new methods of teaching.  Double-standards in regulations (see section below on regulations).  Increased enrollments in TIs having a negative impact on the quality of training due to a mismatch between infrastructure constraints and other inputs versus the number of trainees.  The Fixed Obligation Grant (FOG) system supports all TIs irrespective of their curriculum, providing little incentive for the TIs to move towards competency-based training.  There were too many TIs to support to the desired level of quality and quality gains across the institutions were mixed. 5 Skills-labs are learning laboratories at TIs, where students learn basic practical skills, before they are sent for further hands-on practical training at health facilities. 19 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity  High turnover of staff and the long lead-time necessary to build young faculty member expertise made the pace of teaching quality change slow.  Inadequate availability of preceptors by many TIs. Conclusions The Activity met or exceeded the MEL targets in the area of improving the quality of training across the board and their inputs were widely appreciated and valued among the HSEDCs and TIs. However, there were also major challenges associated with improving the quality of pre-service training, not all of which were in the manageable interest of the Activity to control. Sub Evaluation Question: How effective were the approaches used to strengthen the regulatory environment related to HRH activities? FMHACA is a regulatory authority in Ethiopia established to ensure the safety and quality of health and health-related products, food, professionals, health facilities, and services. The approach that the Activity used to strengthen the regulatory environment related to HRH activities was to work through the two established regulatory agencies, namely HERQA and FMHACA. This approach was aimed at building on the existing regulatory agencies and strengthen their regulatory practices. The Activity supported HERQA in the training of 700 HERQA assessors and the development of 14 HERQA program standards that are used to monitor quality at HSCs across the country. Through technical support, the Activity also increased the capacity of HERQA to develop guidelines and undertake spot and audit checks on TIs. The Activity supported the development of licensure examinations in the health sector. For FMHACA, the Activity supported the development of case-based material to train staff to avoid malpractices actions, using real-life experiences of malpractice cases as instruction tools. This training was seen as a remarkable achievement because, previously, the use of these cases was restricted due to their ‘confidential’ classification. The evaluation team’s interviews with training institutes, regulatory authorities, federal, and regional authorities found that weak regulation capacity remains a significant challenge for improving the quality of TIs, advancing the professionalism of HWs and ensuring responsiveness of health providers. The regulatory authorities, HERQA, for higher education institutions and FMHACA, for professionals, do not uniformly enforce regulation standards across public and private institutions or professionals. The capacity of both regulatory institutions, regarding information systems, staffing and structures at regional levels, is still too weak to adequately enforce regulations. In theory, public and private TIs can be managed under the same regulations. However, in practice, the enforcement of these regulations is more strictly adhered to in the private sector than in the public TIs. Data shows that enforcement of standards of training and professional practice are often either ignored or given minimal attention for public sector TIs/ professionals. This disparity between private and public sector regulation results in ‘double standards.’ For example, Addis Continental Training College - a private institution, submits its reports to HERQA and receives annual visits from HERQA. On the other hand, public TIs claim that HERQA does not have enforcement authority over them and it is a ‘lion without teeth.’ The proclamation that established HERQA has the same ‘double standards’ as it enforces accreditation as mandatory for private institutions and not for public institutions. The reason being that HERQA’s proclamation was created during a time of private sector ‘boom’ in the country. There was public concern that the standards in private institutions could be compromised without firm regulation and the focus on regulation was there. There is also limited capacity, both in terms of human resource and quality of skilled personnel, at the federal level to scale up the enforcement of regulation to all public institutions. Furthermore, there are no HERQA branches or offices at the regional level to ensure regulation enforcement at the many regional TIs. According to proclamation 661/2009, health professionals need to be licensed and registered before starting practice at any health facility. Registration and licensing of professionals should be carried out at 20 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity two levels; (i) at the federal level, for scarce professionals, like medical doctors and midwives; and (ii) at the regional level, for other professionals designated as non-scarce, such as nurses and health officers. Unfortunately, professionals are currently being deployed by FMOH to work at health facilities without the proper licenses. Although the proclamation clearly states that every professional must be re-licensed every five years, after undergoing CPD, this practice is not regulated and enforced. There are, therefore, health professionals across the region who are providing services without a license. For example, a visit to one hospital in one region confirms the existence of 16 doctors operating without a license. FMHACA, which has the authority to regulate health professionals, does not have a comprehensive information system that helps track health professionals working in both public and private facilities, along with their up-to-date regional and federal licensing status. The HRH strategy does not include regulatory staffing needs as part of the HRH development framework and plan, which limits the ability of regulatory institutions to build capacity at the regional and woreda levels. There is, for instance, no medicolegal TI in the country which is essential to develop competent regulators. Furthermore, the current regulatory arrangement at FMHACA is not only ineffective but also at a crossroads. FMHACA is planning to move towards a product regulator function, limiting its actions to regulating food and medicines, leaving provider and professional regulation services to FMOH. The division of tasks and responsibilities among federal and regional regulators is also ineffective. While federal regulators have their branches and offices, regional regulators maintain their distinct lower-level structures and institutions that work independently across regions resulting in lack of transparency and consistency of effort. In some regions, the regional regulators are semi-independent, and in others, they are within the RHBs. These challenges are well documented by FMHACA, and currently, there are plans for the Department for International Development (DFID) to design and support implementation for a strategy to strengthen regulatory procedures at FMHACA. The timing and scope of this support are not yet clearly defined. Conclusions Transforming the quality of pre-service and in-service training, as well as enhancing the professionalism of HWs and providers requires strengthening the two regulatory agencies (HERQA and FMHACA) in charge of implementing quality assurance. The Activity’s approach of working through these two agencies had good intentions regarding building on existing regulatory agencies. However, this approach was constrained by the limited capacity of these agencies to enforce quality regulations, particularly in the public sector. The double standards (where regulations were less stringently enforced in public institutions compared to private institutions) was a particular challenge of these regulatory institutions. 3.4. RESEARCH, MONITORING, EVALUATION, AND LEARNING (ACTIVITY IR4) The Activity planned to conduct 24 studies of which 22 have been carried out. Of these, eight were program learning (i.e., studies not published in scientific journals), and fourteen were research activities (i.e. published or to be published in scientific journals). The studies addressed issues related to availability, quality, motivation/retention, and regulation/legislation of HRH. Activity reports (Jhpiego6 and Damtew 20187) stress that development of study proposals, designs, data collection, analysis, and manuscript preparations was done with the broad participation of all partners. 6 JHPIEGO nd. HRH evidence generation and utilization 7 Damtew Woldemariam. “Strengthening Human Resources for Health in Ethiopia to Improve Health Services Delivery and Health Outcomes.” 2018, Addis Ababa. 21 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity Study topics addressed highly relevant, priority and topical issues in HRH in Ethiopia (Table 9) and reportedly informed subsequent development of scopes of practice (SOP) for midwives, anesthetists, IST centers, CPD guidelines, curricula, quality enhancing inputs (skill labs, improved practice sites, etc.) as well as for the development of documents such as the HRH strategic plan (FMOH 2015a) and the new health policy (FMOH 2015b). Table 9: Topics Addressed by Technical Reports (TRs) and Research Publications (RPs) Midwifery Anesthetists HEW Others* TR RP TR RP TR RP TR RP Baseline/Background 1 - 1 - 1 3 - Competency 2 4 1 2 - - - - Motivation/Retention, Turnover - - - - - 2 1 4 Task Analysis 1 1 1 1 1 1 1 Total number** 4 5 3 3 2 3 5 4 * Most categories HR but mostly medical doctors, health officers, nurses, medical laboratory professionals, and pharmacy professionals ** Higher than 22 because a number addressed multiple topic areas Overall, the Activity generated evidence on pre-service training, motivation/retention, quality of care and the state of the regulation of HRH in Ethiopia using acceptable standard methodologies. However, the participation of other stakeholders including higher education institutions (HEIs), in the conduct of studies, was limited. There appear to be gaps in the selection of topics as most FMOH and RHB interviewees had not been involved in the selection of the priority agendas of the research; a problem partly attributable to the high turnover of staff and absence of a formal handover process among government officials. Notably, ethical clearance was only obtained from Johns Hopkins, and no ethical clearance was solicited from local institutions. While participation in designing and carrying out the studies was limited, there is evidence that participation in program learning activities (PLAs) was greater. The evaluation team found that the majority (>70 percent) of participants in PLAs were from Ethiopian institutions (48 percent were from HEIs) and less than 20 percent were females. At the same time, participation in research publications seems to be dominated by the Activity staff, where 53 percent of the authors were from JHPIEGO (almost all Ethiopians), 43 percent were external (Netherlands, etc.), and there were no authors from HEIs (Table 10). KII and FGD participants at all levels confirmed this saying that any research they have heard of was conducted by JHPIEGO staff with almost no involvement of other stakeholders except the use of some staff from TIs as data collectors in a few studies. As expressed by one KII from FMOH: “They do all the research by themselves with no involvement of staff from FMOH or the RHBs. There has been no attempt at skill transfer. The logo of the Ministry is put on their publication only as a token/lip service.” Table 10: Participation in PLAs and RPs Activity JHPIEGO External (Outside of Ethiopia) MOH/ Regions EMWA EAA HEI (Ethiopia) Other (Specify) Total (Female) PLA # 87 4 38 9 8 159 28*** 333 (58) % 26.1 1.2 11.4 2.7 2.4 47.8 8.4 100 (17.4) RP # 49 40 2 0 2 0 0 93 (31) % 52.7 43.0 2.2 0 2.2 0 0 100 (33.3) * Ethiopian Midwife Association **Ethiopian Association of Anesthetists *** 2 MSH, 1 FMHACA, 10 from Professional Associations (report writing), 14 unspecified Dissemination of the study outcomes was mostly limited to reports and publications with very few of the planned workshops held. Little effort seems to have been made to disseminate findings to targeted audiences such as policymakers (e.g., no policy briefs prepared). 22 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity Capacity Building in Monitoring and Evaluation (M&E) and Research: All PLA and Rapid Assessments (RAs) were designed as capacity-building activities. The intention of these activities was to model monitoring of the availability, facility coverage, training and retention of HWs; show how to use data to strengthening the reporting process, to review the completeness and timeliness of routine HRIS; to use routine HRIS data for programmatic decision-making; and to measure the impact of specific training interventions. However, very few of the activities were implemented as envisaged in the original document (Table 11). To date, only one study was undertaken as intended in the original Activity plan (Table 11, item 8). A second one (Table 11, item 1) was undertaken with a slightly modified heading. Moreover, still another one (Table 11, item 4) was undertaken by integrating gender into different assessments and studies. All the other studies were canceled because they were either not feasible (items 2 and 6), covered by other donor-supported projects (items 3 and 7), or not a priority for the government (items 5 and 9). Having said this, the document was envisaged as a ‘living’ document and subject to adaptation to evolving contexts – so in this sense, it was successful. Table 11: Proposed Study Topics and Reasons for Change/Dropping Proposed Studies In Project Proposal Actual/modified/dropped* 1. Conduct a work climate assessment in sample regions, woredas and health facilities (IR1) Job satisfaction and factors affecting HW retention in Ethiopia’s public health sector 2. Examine the joint appointments of health professionals in private & public health facilities in terms of job satisfaction, motivation, and retention (IR1) Canceled because of feasibility on tracking of study participants 3. Examine the impact of leadership and management training on human resource management (IR1) Canceled because of the direction from donor to drop this Activity as it is implemented by other USID funded organization under Leadership, Management & Governance (LMG) project 4. Examine/identify gender issues impacting women satisfaction in the workplace (IR1) Conducted integrated with other broader study indicated above. 5. Conduct an assessment to examine the main reasons for students' attrition in midwifery and anesthesia (IR2) Canceled because of other priorities from the federal ministry of health 6. Examine whether HEWs' attrition has been effectively managed through the use of HRIS data by RHB (IR2) Canceled because of feasibility issues as HRH did not have a direct role in HRIS implementation 7. Explore the impact of the problem based accelerated learning approach using a cohort of medical students (IR3) Canceled because of the direction from the donor as this is covered by another project (New Medical Education Initiative [NMEI]) 8. Assess whether students have adequate exposure to core competencies during clinical practice in a selected cadre (IR3) Competency Assessment of Graduating midwifery and Anesthesia Students (pre-post study) 9. Examine whether performance recognition influences the retention of technical and academic staff (IR3) Canceled because of other government priorities. Some part addressed in the broader study of job satisfaction *Source: JHPIEGO Some headway has been made in building institutional capacity to monitor and evaluate competency￾based training through the development, with the FMOH and university partners, of a national M&E framework for HRH. Through this framework, the Activity supports the implementation of retention and blended learning assessments. Steps have also been taken in supporting program learning through systematically capturing and processing information, documenting lessons learned and contributing to the global body of evidence through publication. However, almost no activity was undertaken to build M&E capacity in the ISTs as planned in the Activity program description. This was dropped because it 23 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity was handled through another project. There are no clear indications that HRH M&E training has been strengthened in the regular curricula. Capacity building for PLAs and RA seemed to face several challenges documented from the document reviews, KIIs and FGDs. These challenges were: high turnover of staff, limited research culture, and inadequate preparedness for (and therefore ‘fear’ of) participation in research. The evaluation team also found a perception that participation in/undertaking research was not part of IST or other institution work and that specialized agencies (the Ethiopian Public Health Institute (EPHI) and Armauer Hansen Research Institute (AHRI) are mandated to do this. Some of the other challenges noted were that there were insufficient numbers of skilled researchers with experience; a situation shared with most African countries8 and that most university staff members have high workloads with teaching and other supplementary activities to make ends meet.9 Conclusions The SHRH Activity has played a major role in drawing attention to the pressing HRH agenda and to strengthening HRH and HRM function in Ethiopia. While the Activity met or surpassed the intended number of research assessments and publications, there was little focus or intention on capacity building for research and assessments with government counterparts. Most of the publications were authored by Activity staff or external consultants. There were changes to the research agenda over the life of the Activity, which is understandable, but the process of consultation driving these changes was not well documented. Finally, there were missed opportunities for capacity building with government counterparts that would have been complementary to Activity interventions. Recommendations on Effectiveness  The Government of Ethiopia (federal and regional) should review remuneration packages for HRM professionals to create interest from qualified candidates to apply for HRM positions and to improve retention of existing HRM staff. The experience of the Oromia region could be used as a best practice example for other regions to learn from and implement.  USAID should engage with the FMOH & RHB to prioritize, reconstitute, and reinvigorate the HRH forum, to strengthen its functioning as a platform for addressing cross-cutting HRH issues, especially strengthening it at the federal level to go beyond the technical to strategic policy discussions.  The Government of Ethiopia and USAID should set aside funds to promote the development of a clear IST implementation strategy, including infrastructure, technical capacity trainer motivation, generation and use of resources, etc.)  The Government of Ethiopia should review the effectiveness of IST expansion as a means for shifting hotel-based training to IST centers.  USAID should not include increasing the availability of health workers as a result area for any future HRH interventions as this has been prioritized by the Government.  USAID should support the FMOE’s strategy to develop health science qualifications standards for colleges across the country, by providing technical and financial support.  USAID should focus investments for any future support on third- and fourth-generation universities that have proven themselves receptive to new ideas and can affect more significant change. 8 Kebede, D., Zielinski, C., Mbondji, P. E., Sanou, I., Kouvividila, W., & Lusamba-Dikassa, P.-S. (2014). Human resources in health research institutions in sub-Saharan African countries: results of a questionnaire-based survey. Journal of the Royal Society of Medicine, 107(1 Suppl), 85–95. 9 EAS. Report on Mapping the Health Research Landscape in Ethiopia. Ethiopian Academy of Sciences (EAS), 2013, Addis Ababa. 24 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity  USAID should consider designing the activity to support HSCs to shift towards a competency￾based curriculum.  The Government of Ethiopia should develop and implement strategies to balance enrollment with available resources and infrastructure in TIs.  The Government of Ethiopia should develop and implement a continuing professional development strategy. USAID should provide support for this strategy when finalized.  USAID’s future Activity designs should explore ways of strengthening the consultation processes with donors, government and other partners in developing priority topics/activities for program learning and research.  USAID should ensure ownership and use of PLAs and RA findings through greater participation in the selection of topics, the design of studies, analysis of data and dissemination  USAID should consider using “theories of change”10 and methodologies that facilitate the uptake of the evidence base by policymakers.11  USAID should consider providing direct support for regulatory strengthening activities as a separate result area for any future HRH support. Some of the interventions in this area could include the following: o o o o o Support with building the capacity of the two regulatory agencies to ensure their increased ability to enforce regulations at both public and private institutions. Strengthen the accreditation and enforcement capacity of the two regulatory agencies. Develop concrete strategies to produce and train more professionals in the area of regulation (such as medicolegal professionals) as part of HRH strategy. Advocate for and support the development of a stronger information sharing system platform that provides greater linkages within and across regions for regulatory and accreditation information. Improve systems for licensing TIs, professionals, and providers 10 Paina, L. et. Al. Health Research Policy and Systems 2017-15 (Suppl 2) :109. 11 For example USAID could consider ‘discreet choices experiments’ which is “a quantitative research method that can measure the strength of preference and trade-offs of the health workers towards different job characteristics that can influence their decision to take up rural postings” (see Lagarde, M., & Blaauw, D. (2009). A review of the application and contribution of discrete choice experiments to inform human resources policy interventions. Human Resources for Health, 7, 62.) for which there are already some experiences in Ethiopia (Hanson, K., & Jack, W. (2010). Incentives could induce Ethiopian doctors and nurses to work in rural settings. Health affairs, 29(8), 1452-1460.) 25 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity 4. FINDINGS OF THE EVALUATION ON THE SUSTAINABILITY OF SHRH INTERVENTIONS Evaluation Question 2: To what extent did the SHRH Activity interventions facilitate local ownership, coordination, and partnerships and sustainability? (Sustainability) Of the six components contributing to improved HRM systems, two HRM policies and HRM structure were found by the ET to be completely sustainable. Two components, the HRM forum and the support for motivation and retention were found to be partially sustainable, and two components were found to be unsustainable, the HRIS and capacity building interventions (Figure 8). More detail is provided below. 4.1. HUMAN RESOURCE MANAGEMENT (ACTIVITY IR1) The approach implemented by SHRH Activity to establish HRM units, create and fill positions, was entirely locally owned and led in most of the RHBs and FMOH. According to the Activity document reviewed and verified by physical observation during field visits, the Activity utilized evidence-based advocacy; conducted organizational needs assessments, and the outcomes were utilized in securing stakeholders’ buy-in to create new HRM structures and positions. The Activity also provided technical assistance throughout the process, as reported by KII participants. Some HR managers interviewed stated that it was the involvement of government agencies in evidence generation that facilitated local ownership. This was not the same in all regions. HRM structures in regions like Afar are not wholly owned locally. During FGDs with the RHB team, participants stated that organizational needs assessments were only conducted in a few regional states, excluding Afar, and the outcomes were used as the benchmark for all regions. Therefore, ownership of the HRM structure (newly created HRM positions) by the regional government in Afar was weak. Regarding the HRH forum, evidence gathered through KIIs showed wide variation in the level of ownership by the government. While there were indications of keen interest, motivation, and commitment in Oromia, demonstrated by the frequency of meetings (usually quarterly) and document action taken on strategic issues, the experience of key informants in SNNP was that the forum has not been meeting regularly, and attendance was usually low (see Table 5 in Section 3 above). Key informants at the Addis Ababa Civil Service Commission reported that inconsistent communication with key stakeholders was responsible for the weak ownership of the forum in that region. This clearly shows that there is a gap in the interest and motivation of most of the regions to effectively use the HRH forum to solve some of the pressing HRH agenda. Improved HRM Figure 8: HRM Interventions Sustainability Matrix Light Blue Fully sustainable Dark Blue Partially sustainable Red Unsustainable 26 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity Furthermore, concerning capacity building initiative, the Activity documents reviewed did not show any mechanism for sustaining the capacity building intervention. Instead, the emphasis was placed on achieving the number of personnel trained. Also, evidence from FGDs revealed that the Activity prioritized strengthening individual capacities with limited attention to institutional capacity development. Respondents stressed that little or no effort had been made by the Activity to internalize capacity building process or create an institutional memory. This opinion was further confirmed by the high number of re-trainings due to high personnel turnover rate, as indicated in the quantitative data obtained from the RHB. It was also revealed that limited attention was given to on-the-job training, coaching, and mentoring, which were perceived by most key informants as effective ways to improve performance. Similarly, while the Activity trained HRIS focal persons (officers), according to some key informants interviewed, the application is not user friendly and has limited flexibility, thereby limiting its functionality. In establishing a decentralized HRM structure, the evaluation team found that the Activity engaged with the civil service bureau and other relevant agencies of the government, such as in Amhara and Oromia regions. The regional and federal governments have been continuously approving additional HRM positions and funding is being provided for recruitment and other HRM operational activities. One instance of ownership and engagement is the signature and stamp of the head of the civil service bureau in Oromia for the compiled HRM processes, procedures, and proclamation, which were distributed at all levels of the region’s health sector. Conclusions Although the evaluation team did not find any clear evidence of a sustainability strategy, either in the program description or MEL Plan there was a positive achievement in this area. Based on the findings described above, the HRM structure, staffing, and HRM policies and procedures were wholly owned by the government and can be sustained without further support. However, HRH forum and Motivation and Retention scheme, are deemed to be partially sustainable having identified some level of commitment demonstrated by the regional and national government, therefore would require substantial additional support to be owned and sustained at regional and federal levels. Conversely, the approach for capacity building support, cannot be sustained because the training conducted was not integrated into the core system. As personnel leave, there was a need to re-train new hires, a cost which is too high for the government to endure. “Engagement of Public Service Commission in the planning and implementation of the Activity was very minimal; the bureau had no adequate information on the annual plan of the Activity, which made it difficult to incorporate into our annual plan.” HRM Office, Afar “But the major driver of these achievements is the ownership and commitment and leadership of the region at all levels. The decision to create these positions was decided by the regions president office, who is the chairman of the HRH forum in the region.” RHB, Oromia 27 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity 4.2. INCREASED AVAILABILITY OF HUMAN RESOURCES (ACTIVITY IR 2) Increased availability of healthcare staff, including staff targeted by the Activity, has been a priority of the government at both federal and regional levels. Both federal and regional governments have made significant investments in the training of HWs. Such large investments by federal and regional governments have overshadowed the Activity’s contribution to increasing the availability of healthcare staff. Such large investments to train HWs by the federal and regional government are projected to continue in the short and medium terms, with the likelihood of overriding any future Activity support in this area. Conclusions The evaluation team concludes that the increased availability of healthcare staff is sustainable without external support, except new and emerging health work needs. However, the sustainability of some specific ISTs will be challenging without additional external support. For example, the capacity of implementing partners such as the Ethiopian Association of Anesthetists (EAA) and the Ethiopian Midwives’ Association (EMWA) to play a significant role in the provision of training is not sustainable without external support. Several capacity-strengthening activities were provided to EAA by the Activity, but they were not enough to bring the EAA to a sustainable level, as the organization is still highly dependent on the Activity inputs for its operations. 4.3. IMPROVING QUALITY OF EDUCATION (ACTIVITY IR 3) To improve the quality of pre-service education, the Activity worked through existing local institutions at the federal level (FMOH, FMOE, HERQA, FMHACA, professional associations), the regional level (RHBs, regional bureaus of education), and the TIs in all Activity interventions. This was to facilitate local ownership, coordination, partnership, and sustainability. However, the degree to which Activity interventions are sustainable varies from fully sustainable (e.g., PSE quality interventions, such as faculty development at TIs and a competency-based curriculum at Debre Tabor University) to unsustainable (e.g., IST quality interventions at the EAA). For example, Activity interventions in quality of PSE such as effective teaching skills, EDCs, and other faculty development and curriculum review activities are likely to continue with or without external support. However, they are likely to be at a reduced scale without support. The major constraint when it comes to the sustainability of PSE quality is the availability of equipment (such as skills-lab equipment) that is not readily available in the Ethiopian market and may not be easily procured by local TIs. Another constraint is the double standard in the regulation of TIs (private institutions being better regulated than public institutions). On the other hand, the sustainability of interventions in In-Service Training (IST) quality as a whole is questionable, because the IST centers are not fully functional at operational levels. CPD is not yet fully rolled out as a requirement, and IST standardization in all regions has not yet been fully implemented. Conclusions In working to improve quality, the Activity facilitated local ownership, coordination, partnership, and sustainability by working through existing local institutions at federal, regional and TIs levels. However, the sustainability of interventions in quality of IST is questionable. 4.4. RESEARCH, MONITORING, EVALUATION, AND LEARNING (ACTIVITY IR 4) Evidence of participation and ownership of local stakeholders in the primary studies conducted by the Activity was limited. Feedback received showed minimal achievements of the Activity in program learning and research capacity building. Low levels of achievement were compounded by the high staff 28 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity turnover within the evaluated institutions. To date, very few institutions have allocated their resources for research on HRH. Conclusions As a result of the insights shared by the participants, the evaluation team finds that the Activity could not build sustainable research initiatives, without active external support. If USAID values local research capacity, a significant investment would need to be made to build capacity in research design and implementation. There are currently few allocations of funding for research by the institutions themselves. Recommendations on Sustainability  USAID should focus on building leadership commitment among forum members in future Activity support. The case of Oromia region can be used as an example.  USAID should have the next SHRH Activity carry out organizational capacity assessments designed to engage stakeholders in not only assessing their organizational capacity but involve them in designing the action plan for the way forward.  USAID should focus on institutional capacity as equally or more important than the individual capacity to facilitate sustainability. USAID should continue to support the capacity building of TIs through technical support and ensuring the availability of adequate teaching materials.  USAID and the Activity should collaborate, learn and adapt by working with international TIs to implement best practices.  USAID should ensure that any future similar Activity should prioritize interventions that enhance sustainability in quality of IST by ensuring infrastructure development as a pre-requisite for providing future support.  USAID should direct the partner to work with and support FMOH, RHBs, and local research institutions need to build their research capacity rather than just having the implementing partner do the research. 29 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity 5. FINDINGS OF THE EVALUATION ON GENDER Evaluation Question 3: To what extent were gender issues addressed in pre-service and in-service training? (Gender) The SHRH Activity has been supporting gender-related activities in 52 TIs, with the aim of improving the retention, graduation rate, and performance of female students in pre-service and in-service training. It also supported 44 institutions to establish and strengthen gender offices at the college level, with the objective of addressing gender disparities in these TIs. Over 7,000 female students received SHRH support with life-skills training, and 150 faculty members were trained on gender-responsive pedagogy (GRP). Also, 1,163 economically disadvantaged female students benefited from the fixed-amount cash award, which was widely considered instrumental in the success of gender-related activities. The Activity made contributions to gender equity by supporting the establishment of a gender office at the health science school level, which brought an increased recognition and focus on gender-related interventions in pre-service training. Recognition awards from the SHRH Activity were given to 519 top-performing female students. Additionally, 2,972 female students were supported with tutoring above and beyond their regular school program, with the aim to improve their performance. However, the provision of tutorial classes for female students varied between TIs. For example, Menelik HSC had 3 percent of teaching hours allocated to tutoring female students, but others did not. The SHRH Activity also supported TIs to begin using a sexual harassment reporting form and monitoring tool to improve reporting of sexual harassment, take action against those that are harassing girls and document of gender-related activities, but the evaluation findings indicate that these tools were rarely implemented in those TIs visited. The above gender interventions were implemented only for pre-service training activities. The SHRH Activity did not undertake gender interventions at in-service training levels and other result areas such as HRM. Another area that was supported by the SHRH was the prioritization and implementation of gender-focused research. Program learning, as well as research topics and reporting explicitly, addressed gender issues. However, the participation of women in published reports was minimal. Funds allocated to gender offices for research by the Activity were too small (mostly ETB 2,000 / $72) to make a difference, even as start-up money for research. The evaluation team did not find that the SHRH Activity promoted gender awareness or address gender-related issues in ISTs. Although secondary data in training reports are gender-disaggregated, no assessment was conducted to prove that the increased number of women participating in training and other interventions in practice areas (healthcare delivery) was influenced by the Activity’s support. There were indications of an increased number of women in management positions at different levels and greater women’s participation in decision-making, but these observations could not be related to the interventions of the SHRH Activity, mainly because a similar trend is visible in other sectors of government in Ethiopia. During discussions with key informants, it was evident that gender-related interventions were still in a state of infancy in the TIs (that is, they lack a robust institutional foundation). There were no gender targets or interventions in ISTs, the budget was insufficient to support material assistance to most of the needy female students, and counseling services need more support for female students. Conclusions The results of the evaluation indicate that there was no concrete evidence that the SHRH Activity contributed to strengthening the necessary institutional and structural mechanisms for gender equality 30 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity that would increase the enrollment, retention, and performance of female students. The Activity undertook motivational approaches, such as the recognition awards, to encourage continued enrollment and success of female students. Overall, the efforts were spread too thin to have a significant impact on pre-service and in-service training. Recommendations on Gender  USAID should support future Activities to provide training and support the implementation of the sexual-harassment reporting system.  The GoE or USAID should evaluate the tutorial program to determine its effectiveness in increasing retention, grades, skills, and graduation rates of female students.  The Government of Ethiopia should consider providing financial assistance to needy female students and expand that assistance to IST. 31 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity 6. FINDINGS OF THE EVALUATION ON CONTINUED RELEVANCE OF SHRH INTERVENTIONS Evaluation Question 4: To what extent have SHRH Activity result/sub-result areas and interventions continued to be relevant to HRH gaps and challenges in Ethiopia? (Relevance) The SHRH evaluation found that the Activity interventions are highly relevant in addressing several challenges related to HRH and HRM. These include availability/accessibility, quality, motivation/retention, legislation/regulation, and proper management, which are considered critical by both national and international policy documents. 6.1. HUMAN RESOURCE MANAGEMENT (ACTIVITY IR 1) Most of the SHRH Activity interventions are critical in addressing HRH challenges in Ethiopia. The establishment of an HRH forum as a strategic platform and bringing political leaders into the picture to address HRH gaps are highly relevant activities. Both steps facilitate collaboration that is otherwise missing among key stakeholders and relevant government agencies. Further, the efforts in strengthening HRM processes and procedures have contributed towards addressing personnel-related issues and increasing trust in the system, according to some key informants. The staff turnover/attrition rate in government positions has remained low (less than 2% nationally)—the retention rates remain high across regions and FMOH. While the motivation of HWs remains relevant, the analysis of secondary data indicates that retention of human resources is not a significant challenge in Ethiopia. According to FMOH data, the national staff turnover rate between 2014 and 2017, was at an average of 1.2 percent. Figure 9 below demonstrates staff turnover rates in the regions visited over a 5-year period. The SHRH Activity has contributed to the development of staff satisfaction survey and staff exit interview procedures when employees leave their positions. However, KIIs with HR managers in most of the regions visited confirmed that these procedures have not been fully operationalized. To date, only the Oromia RHB has fully integrated staff satisfaction and exit surveys into its HRM procedures. The Oromia RHB has developed a unique motivation mechanism, highlighted in Box 3, that may be utilized by other regions. Box 3: Oromia Motivation Mechanism The Oromia Region has introduced differential duty pay and non-financial incentives to ensure human resources are motivated and retained, specifically in remote areas of the region. It also developed posting categorization by grouping health facilities into three categories according to locations. Facilities in urban areas of the region are categorized as A (good); those in less urbanized areas are categorized as B (mid-quality); and facilities in rural areas are categorized as C (distant & underserved). Strong incentives, such as timelines for transfer and earlier skills training opportunities, are provided for staff in underserved areas. We now have opportunity to unite [to] improve awareness of HRH issues because of [the] HRH forum” Regional Science College, Bahir Dar 32 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity Figure 9: Staff Turnover Rates by Region in Afar, Amhara, Oromia, and SNNP from 2013 - 2017 Personnel information management remains critical for decision-making on HRH issues (MoH, Health￾Related Indicators 2017). However, the current HRIS application has not addressed this need. Evidence obtained by a physical assessment of the system indicates that the current HRIS application provides users with limited flexibility. Users face difficulty controlling permissions, creating new cadres, and customizing system function. Conclusions The HRH forum as a strategic platform and bringing political leaders into the picture to address HRH gaps is a potential best practice when fully operationalized and should be strengthened. This is evidenced by the success in the Oromia region. Although our findings show low attrition rates, further investigation is necessary to determine the profiles of the people leaving the government system. It is possible that the departing employees possess skills that are in short supply and continuous departure of such staff will leave a hole in the public health system. While the SHRH Activity has contributed developing staff satisfaction survey and staff exit interview procedures, efforts to ensure the utilization of these tools is essential to generating critical data for improvement. 6.2. INCREASED AVAILABILITY OF HUMAN RESOURCES (ACTIVITY IR 2) Human resources are one of the six building blocks of all health care systems. Health priorities of the post-2015 agenda for sustainable development and progressing towards universal health care require an innovative attempt to supply a competent and qualified health workforce with mixed skills; seen as essential for the effective implementation of health program interventions. It is, therefore, important to note that the supply and training of the health workforce are recognized in the Ethiopian Health Sector Transformation Plan (FMOH 2015c) as a key pillar for ensuring satisfactory performance. Ethiopia currently faces a triple health burden: both communicable and non-communicable disease as well as injury. There is a need for a responsive system to equip train a mixed-skills health workforce to adequately respond to changing epidemiological and lifestyles conditions; the skills mix continues to be a challenge. Nurses and HEWs constitute the majority of the available supply of HWs, as suggested by key informants at both the national and regional levels. Moreover, the health workforce density (1.6 per 1,000 population) is low compared to the WHO benchmark for Sub-Saharan Africa, which is 2.3 per 7% 6% 3% 2% 5% 5% 2% 1% 1% 5% 2% 1% 3% 3% 1% 1% 2% 2% 1% 1% 0% 1% 2% 3% 4% 5% 6% 7% 8% Afar Amhara Oromia SNNP 2013 2014 2015 2016 2017 33 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity 1,000 population. Additionally, there is a shortage of anesthetists, midwives, laboratory technologists, and pharmacists in Ethiopia, which indicates that the training of these HWs by the SHRH Activity was a relevant activity. Additionally, they trained other essential HWs as appropriate. The Health Sector Transformation Plan (HSTP) of Ethiopia strongly posits HRH as the foundation of overall health improvement in the country. HSTP I put forward four planks of the transformation agenda that will help to achieve the HSTP’s targets: (1) transformation towards quality and equity in health services; (2) woreda transformation; (3) movement towards compassionate, respectful, and caring (CRC) health professionals; and (4) an information revolution. Each transformation agenda requires an adequate and competent health workforce to be realized, which makes continued training of HWs relevant now and in the future. However, the CRC agenda is purely HRH-related at present, where significant investment is required both for training new HWs and for boosting the morale of the existing health workforce. Conclusions The interventions undertaken by the SHRH Activity are relevant as they were designed to increase the training of anesthetists, midwives, HEWs, and other essential HWs. Support for the training of essential HWs, such as EMTs, BMTs, postgraduate HRM/Health Economics students, and those with a nursing specialty (neonatal nursing, emergency, and critical care, operating theater, pediatrics, or surgical nursing), was particularly relevant in addressing current gaps and future demands in Ethiopia. 6.3. IMPROVING QUALITY OF EDUCATION (ACTIVITY IR 3) Data collected from KIIs and FGDs across the sampled regions indicated that respondents unanimously found improving the quality of training to be relevant. As a respondent from Samara University said, without the support from the Activity, they would have been moving at ‘tortoise’ speed. Conclusions The evaluation team found that improving the quality of both pre-service and in-service training was highly relevant for addressing HRH gaps and challenges in Ethiopia. Improving the quality of both pre￾service and in-service training will remain highly relevant since the pressure to increase the production of HWs poses a significant challenge to maintaining high-quality training and services. 6.4 RESEARCH, MONITORING, EVALUATION, AND LEARNING (ACTIVITY IR 4) Conclusions Program learning and research remain highly relevant, as HRH in Ethiopia still faces several challenges requiring evidence-based responses. As indicated in Section 3.4 above, capacity building in M&E was dropped from the SHRH Activity as it was being handled by another Activity. However, the SHRH Activity has succeeded in putting research in HRH—usually a neglected field—squarely on the agenda in the Ethiopian context. This is in line with the GoE’s HRH strategy, which underscores that “information and research on HRH is also an important Activity that will become a priority area of the DHRM [Directorate of Human Resources]…” (MOH 2015a).12 The strategy further recommends the development of a mechanism of competitive research grant award to young and senior researchers actively employed in the public health care system. HSTP (MOH 2015c)13 also highlights “strengthening 12 MOH 2015a. Human Resource for Health Strategic Plan, Ethiopia 2009-2020. Addis Ababa. 13 MOH 2015c. Health Sector Transformation Plan HSTP 2015/16 - 2019/20 (2008-2012 EFY). Addis Ababa. 34 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity implementation research and quality of care across the board [with] evidence-based policy formulation and implementation … and participatory resource allocation.” Recommendations on Relevance  The Government of Ethiopia should approach motivation and retention efforts differently. In line with government priorities, staff satisfaction and motivation to improve on-the-job performance should be emphasized over retention.  The Government of Ethiopia (supported by USAID) should strengthen the human resource information management systems in alignment with the implementation of District Health Information System 2 (DHIS2).  USAID should support the strategic provision of technical assistance and material support to TIs for training essential HWs should be continued as relevant in future HRH activities.  USAID should consider continuing to fund activities that build the institutional capacity at all levels of the government to provide quality pre-service and in-service training.  The Activity and USAID should place an increased emphasis on developing capacity through collaborative studies and research with all stakeholders. This research capacity development approach is especially crucial for new TIs with young and inexperienced staff. 35 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity 7. FINDINGS ON EVALUATION ON FACILITATING/HINDERING FACTORS Evaluation Question 5: What facilitated and/or hindered the achievements of planned results? 7.1 IMPROVED HUMAN RESOURCE MANAGEMENT (ACTIVITY IR1) The Activity was recognized as a driver of change in human resource management in Ethiopia through its active engagement of regions and the FMOH and bringing new ideas and skills to the table. The SHRH Activity was referred to by some respondents from the FMOH as the only development Activity that has made direct contributions to HRM in the health sector of the country. The FGDs held with some HRM personnel in Oromia RHB reported that many international development programs had provided support to various health system building blocks, but none except the SHRH Activity has shown an interest in the management of human resources. In Oromia, the regional civil service bureau stated that due to this support the sector has better human resource management systems and structures compared to other sectors. Therefore, the fact that HRM initiatives were designed to respond to the needs and demands in the health sector must have facilitated the achievements recorded, said a KII from Addis Ababa city administration. The inclusion of highly influential personalities in the HRH forum reportedly helped to drive commitment and dedication among members and as well strengthened resolutions of the forum in some regions. For instance, the president of the Oromia regional government was involved in the HRH forum which according to some KII from the RHB was responsible for the easy passage of resolutions made by the forum. Conversely, the inability of the Activity to achieve set targets on the number of HRM positions filled could be attributed to the Activity’s approach of generalizing its rapid assessment survey findings to all regions in the country, whereas survey was only done in a few regions. For instance, according to respondents from Afar RHB, the proposed figures for HRM positions were based on the assessment conducted in some selected regions excluding Afar, therefore the initiative had limited buy-in of the regional government. Also, the BOFED was not involved at any stage of the process of assessing and proposing an increased number of HRM positions, which affected funding allocation for HRM. According to a KII from the BOFED in Amhara, there was no consideration for budgeting and availability/allocation of funds to make provisions for salaries, which affected approval of new positions in the region. Another factor that hindered recruitment (filling) of the approved HRM positions was poor remuneration. For instance, during the FGD with some HRM personnel, the evaluation team was informed that HRM positions are the lowest paid in the RHB as it was categorized as a support role. This view was consistent in most of the regions visited including FMOH, aside from Oromia region where specific measures have been put in place to address this situation. “The health sector in the region is getting maximum attention because most crosscutting issues are discussed and addressed at the HRH forum meetings” RHB, Oromia “HRM position grades and salaries are lower than other positions, which is demotivating to HR staff. All these need due attention” Civil Service Bureau Bahir Dar 36 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity 7.2 INCREASED AVAILABILITY OF HUMAN RESOURCES (ACTIVITY IR 2) The responsiveness of the Activity to the government’s emerging needs, in particular on curriculum development and IST related activities for expanding HSCs and ISTs, was the primary factor that supported success in increasing the availability of human resources. Engagement and mobilization of several stakeholders through advocacy and promotion activities at the initial implementation phase of the Activity was one factor which ensured buy-in from the Ministry of Health, RHBs, and TIs. The HRH Activity was consistent with the HRH Strategic Plan and specifically with its aim to increase the training of the targeted health cadres. Moreover, the Activity operated in an environment where there was a high level of government investment and commitment to expanding TIs in the country. However, the target of increasing the anesthetist to population ratio under IR2 was not achieved (only 41 percent achieved (0.07 against the target of 0.17). Low enrollment of students for Level V training and slow expansion of the anesthesia program in RHSCs and public universities were the primary factors that drove this outcome. There was also a gap in setting the anesthetist to population ratio target during the design stage of the Activity. The MEL assumption on the annual number of anesthetists graduating from the existing TIs, the anticipated pace, and expansion of anesthesia training program in the country, and extent of annual population growth were not adequately considered while setting the target of anesthetist to population ratio. 7.3 IMPROVING QUALITY OF EDUCATION (ACTIVITY IR 3) The Activity met or exceeded MEL planned targets (shown in Table 8 in section 3.4 above) in the area of improving quality of education. A number of factors facilitated the achievement and, in some instances, over-achievement of the targets in the area of quality of education: From the KIIs and FGDs, all stakeholders at national, regional, and TI level, were unanimous in their perception of the relevance of the Activity, and the need to invest more on improving the quality of HRH. The limitations in HRH created an enabling environment that allowed the Activity to meet or even exceed its planned targets in the area of quality. There was more extensive than anticipated demand from TIs for Activity support in improving quality of education. This increased demand led to increased support from SHRH, resulting in higher achievement of planned results and targets. The Activity’s response to the TIs’ demands for improving quality was appreciated by the TIs, thus creating a good working environment between the Activity and the TIs. The TIs saw the Activity as a vehicle to help them to meet the requirements set by HERQA. The institutions, therefore, saw the Activity as a needed solution to their quality challenges and tried to exploit its potential as much as they could. See section 3.4 above for more details. 7.4 PROGRAM LEARNING AND RESEARCH (ACTIVITY IR 4) Several factors contributed to high achievements in HRH-related research. A concerted effort was made by the Activity to identify high priority and immediate impact topics done in close consultation with local staff at the regional level in particular. Research conducted by relatively high capacity researchers, Activity staff and input from other institutions (out of the country) produced a relatively high-quality output which helped to put HR in the limelight despite limited consultation with local partners in designing and conducting the researches. Dissemination of study outcomes was mostly limited to reports and publications (14 published/to be published against 24 studies), and little effort seems to have been made to disseminate findings to targeted audiences such as policymakers (e.g., no policy briefs prepared. Thus, the researches reportedly informed some of the policy and program decisions, but wider dissemination was limited, and potential long-term influence (e.g., no policy briefs). Major hindering factors improve research and evidence generation included: weak participation of local staff in the selection, high turnover and limited experience of the young HEI staff in particular; weak ownership and involvement of FMOH and RHB in setting research agenda and guiding its implementation; little or 37 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity no local funds mobilized for research as almost all research expenditures were covered by Activity fund and no resource (even limited human) allocated by other local partners. Most of these (M&E related in particular) were dropped because either they were not feasible or handled by others or considered not a priority. These indicate that there were major gaps in the design of the Activity implying that proper vetting and consultation on the mapping of activities by stakeholders and consultation with the government about the priority of some of the research agendas was not carried out during the design. Capacity building in PLAs and RA faced several challenges including high turnover of staff, limited research culture; inadequate preparedness for (and therefore ‘fear’ of) participation in research; insufficient numbers of skilled researchers with experience to serve as mentors (3.4). Also, the Activity seemed highly task-oriented regarding carrying out quality research on time sidelining capacity building of local partners. All these hindered better achievements Conclusions Based on the findings listed above, it can be concluded that the involvement of policy-makers and highly influential personnel has a huge impact on how successful an Activity initiative can be. The involvement of policy makers needs to commence right from the planning stage and throughout implementation for the initiative to be sustainable. Also, much emphasis was given to proposing and getting approval for new HRM positions without paying due attention to the underlying factors such as professionalization and remuneration of HRM personnel. Operating in a favorable HRH policy & strategy climate and partnering with a responsive government were key factors that supported the increased availability of human resources. The Activity’s ability to pivot to meet emerging demands and engaging different stakeholders were also important factors that supported the outcomes under IR2. Recommendations  Ensure that a comprehensive stakeholders’ analysis is conducted before the introduction of the initiative that aims to increase the number of HRM positions. 38 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity ANNEX 1: SHRM MEL TARGETS Type Baseline (2012) Target for 2017 Result 1: Improved Human Resources for Health Management 1.1 Percentage of organizations that have staff requirement plans in place Outcome 77% 100% 1.2 Percentage of human resource management posts filled with staff who have a qualification in health or management or social science with a minimum of bachelor’s degree Outcome 20% 100% 1.3 Number of human resources management capacity assessments conducted Output 0 15 1.4 Number of HWs who successfully completed an in-service training program in HRM Output 0 2,560 1.5 Number of functional federal and regional HRH forums established Output 0 12 Result 2: Increased Availability of Skilled Midwives, Anesthetists, HEWs and Other Essential Health Workers 2.1 Ratio of midwives to population Outcome 0.56 per 10,000 1.99 per 10,000 2.2 Ratio of anesthetists to population Outcome 0.03 per 10,000 0.17 per 10,000 2.3 Percentage of health extension workers (HEWs) who advance to a higher level Outcome 5% 54% 2.4 Number of new health care workers who graduated from a pre‐service TI within the reporting period Output 0 38,892 HWs* 2.5 Number of HWs who completed a postgraduate program in human resource management and/or health economics Output 0 90 2.6 Number of health TIs (TIs) with expanded clinical sites Output 0 50 2.7 Number of curricula developed and/or updated Output 0 22 2.8 Number of healthcare workers who successfully completed an in-service training program in teaching skills, instruction design, and related topics Output 0 3,875 Result 3: Improved Quality of Training of Health Workers 3.1 Number of laws, policies, regulations reviewed and/or developed to improve the quality of education and training Output 0 7 3.2 Number of health science TIs with internal quality improvement programs Output 3 50 3.3 Number of healthcare workers who successfully completed an in-service training program in educational quality/Standards-Based Management and Recognition (SBM￾R)–related topics Output 0 5,904 3.4 Number of local organizations that provide in-service training Output 0 35 Result 4: Program Learning and Research Conducted 4.1 Number of program learning studies/research reports produced Output 0 16 4.2 Number of healthcare workers who successfully completed an in-service training program in data management and/or monitoring and evaluation (M&E) Output 0 981 *38,892 total HWs = 14,419 midwives; 1,415 anesthetists; and 8,230 HEWs (L3); 13,500 HEWs (L4+); 1,238 other essential HWs; and 89 postgraduates (PG) in Human Resource Management (HRM) and Health Economics (HE) 39 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity ANNEX 2: EVALUATION MATRIX EQ/Obj. Result Area Issues of Interest Data Collection Instruments Data Sources Data Analysis Methods 4 Relevance Result 1: Human Resources Management To what extent do the capacity building investment and in￾service training fit well to the needs of Federal Ministry of Health (MOH), RHBs, and Food, Medicine and Healthcare Administration and Control Authority (FMHACA) and HR managers Document reviews, KII JHPIEGO/FMOH annual Plans, performance reports KIIs with (JHPIEGO, MSH); FMOH, FMHACA; RHBs, RHSC] Qualitative analysis Triangulation among different sources and levels of the health system 4 Relevance Result 1 How relevant are the interventions implemented to improve motivation and retention? Document reviews, KII JHPIEGO/FMOH annual Plans, performance reports; KII with Implementing agencies (JHPIEGO, MSH); FMOH, FMHACA, RHBs, RHSC and Civil Service Administration 4 Relevance Result 1 How well do the HRH fora fit within promoting partnership, coordination, and alignment with government processes and procedure? Document reviews, KII JHPIEGO/FMOH annual Plans, performance reports; FMOH and FMHACA documents; KIIs with (JHPIEGO, MSH); FMOH, FMHACA; RHBs, RHSC] 4 Relevance Result 2: Increased Availability of Midwives, Anesthetists, HEWs How relevant was the focus of SHRH Activity on the identified health workers to workforce requirements in the country? Should the focus still be to continue the similar cadres for the coming years? Document reviews, KII JHPIEGO/FMOH annual Plans, performance reports, and studies; KII with Implementers, FMOH and RHBs HEW, Instructors, Trained HEW. 4 Relevance Result 2 How relevant was the intervention in SHRH Activity to increase Document reviews, KII JHPIEGO/FMOH annual Plans, performance reports; KII with 40 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity EQ/Obj. Result Area Issues of Interest Data Collection Instruments Data Sources Data Analysis Methods the supply of health workers? Implementers, FMOH and RHBs 4 Relevance Result 2 How relevant was the blended career development scheme to enhance the skill gaps identified on HEWs? Document reviews, KII JHPIEGO/FMOH annual Plans, performance reports KIIs with TIs 4 Relevance Result 2 How relevant was the SHRH Activity intervention to meet the non-clinical health workers need in the workforce Document reviews, KII JHPIEGO/FMOH annual Plans, performance reports KIIs with TIs 4 Relevance Result 3: Improved Quality of Training of Health Workers How much are PSE interventions appropriate to improve the quality of education of health workers? KII; FGD FMOH, RHBs, TIs, Students, and facilities 4 Relevance Result 3 How much are in￾service training (ISTs) interventions appropriate to improve the quality of education of health workers? KII; FGD FMOH, RHBs, TIs, Students, and facilities 4 Relevance Result 4: Program Learning and Research How fit was the research conducted to identify the following: (1.) HRH bottlenecks (supply, quality, and management) (2.) to inform government policies & strategies (3.) improve SHRH Activity planning and management via Collaborative Learning Approach? Document review KII Research products; government policies, strategies, and plans; KIIs with imp motors, FMOH and FMHACA 4 Relevance Result 4 How relevant are the SHRH efforts in building monitoring and research capacities at all levels? KII FMOH and RHBs 1 Effectiveness Result 1 How did regional Staff Needs Assessment inform regional Human Resource Development plan? Document review HRH strategic plan and Health Sector Transformational Plan (HSTP) Quantitative analysis (trend analysis, unit cost calculation, 41 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity EQ/Obj. Result Area Issues of Interest Data Collection Instruments Data Sources Data Analysis Methods 1 Effectiveness Result 1 How effective is HRM capacity building efforts in establishing skills, tools, process, and systems? (Skills: individual capacity, Tools: HR tools, System) Document review KII JHPIEGO/FMOH annual Plans, performance reports, and studies; KII with Implementers, FMOH and RHBs comparative analysis between target and actuals Triangulation of data among sources, JHPIEGO, FMOH, RHB, & TIs) Qualitative analysis of perceptions and views of beneficiaries (Govt structures, TI, and Trainees) 1 Effectiveness Result 1 How has the Activity been effective in Reducing attrition and improving motivation? Document review KII JHPIEGO/FMOH annual Plans, performance reports; KII with Implementers, FMOH and RHBs, civil service offices 1 Effectiveness Result 1 How has the Activity been effective in providing IST that improves the performance of HRM staff? Document review KII JHPIEGO/FMOH annual Plans, performance reports4and studies; KII with Implementers, FMOH and RHBs 1 Effectiveness Result 2 Improving workforce density and distribution and skill mix within the facilities Document review KII Activity and MOH documents; KIIs with Activity implementers, FMOH and RHBs 1 Effectiveness Result 2 Replacing and upgrading HEWs KII RHBs and facilities 1 Effectiveness Result 2 # of curricula developed and updated in promoting the PSE of essential non-clinical health workers KII RHBs, TIs, Jhpiego, FMOH 1 Effectiveness Result 2 # of other essential health workers trained Document review RHBs, TIs, Jhpiego, FMOH 1 Effectiveness Result 2 % of HEW progressed to level 3 Document review RHBs, TIs, Jhpiego, FMOH 1 Effectiveness Result 2 % of HEW progressed to level 4 Document review RHBs, TIs, Jhpiego, FMOH 1 Effectiveness Result 3 Number of laws, policies, regulations reviewed and/or developed to improve the quality of education and training, as well as accreditation Document review and KIIs Activity documents, KIIs with FMOH, RHB 1 Effectiveness Result 3 The ratio of academic staff to students Document review Activity and TIs 42 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity EQ/Obj. Result Area Issues of Interest Data Collection Instruments Data Sources Data Analysis Methods 1 Effectiveness Result 3 Number of Health Science TIs with internal quality improvement programs Document review Activities documents 1 Effectiveness Result 3 Person hours of training completed informal science or science-related training courses supported by the USG (New Indicator) Document review Activity documents 1 Effectiveness Result 3 Number of regions who link continuous professional development (CPD) to career development of health workers Document review Activity documents 1 Effectiveness Result 3 Number of in-service training centers established Document review & Observation Activity documents 1 Effectiveness Result 3 Number of regions with in-service training/continuing professional development plan Document review Activity documents 1 Effectiveness Result 3 Number & Quality of Practicum sites Observation & KII TIs 1 Effectiveness Result 3 Number of healthcare workers who successfully completed an in￾service training program in SBM-R related topics Document review Activity documents 1 Effectiveness Result 4 # of researches conducted and published Document review Activity documents 1 Effectiveness Result 4 # of researches informing improved SHRH planning and management (CLA)? Document reviews, KII Activity documents; KII with Activity managers 1 Effectiveness Result 4 What is the number of researches that is informing government policies and strategies Document reviews, KII Activity documents; KII with FMOH 1 Effectiveness Result 4 How has the SHRH Activity research been building the capacity of FMHACA, TIs and FMOH staff? KII, FGD Activity Documents 43 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity EQ/Obj. Result Area Issues of Interest Data Collection Instruments Data Sources Data Analysis Methods 1 Effectiveness Result 4 To what extent are the stakeholders aware of the research findings of the Activity? KII RHBs, FMOH Efficiency Result 1 What is the cost per person per CPD per year Document review Activity documents Quantitative analysis (trend analysis, unit cost calculation, comparative analysis between target and actuals Triangulation of data among sources, JHPIEGO, FMOH, RHB, & TIs) Qualitative analysis of perceptions and views of beneficiaries (Govt structures, TI, and Trainees) Efficiency Result 1 What is the Cost per IST per person per year Document review Activity documents Efficiency Result 1 Investment on Motivation and retention per additional worker retained per year Document review Activity documents Efficiency Result 1 Unit cost per in￾service HRM trainee per year Document review Activity documents Efficiency Result 1 Average unit cost per national and regional HRH forums per year Document review Activity documents Efficiency Result 2 Cost per additional workforce produced: Midwives, HEW & Anesthetists Document review Activity documents Efficiency Result 2 Unit cost per TI supported per year Document review Activity documents Efficiency Result 2 Unit cost per essential health worker trained per year Document review Activity documents Efficiency Result 3 Received tuition fees by TIs per targeted student population Document review Activity documents Efficiency Result 3 The annual cost of in￾service training supported per health worker Document review Activity documents Efficiency Result 3 Average cost per established skills lab per TI Document review Activity documents Efficiency Result 3 Average cost per curriculum revision Document review Activity documents Efficiency Result 4 Average unit cost per research conducted Document review Activity documents Efficiency Result 4 Research findings informing policy/strategy development and/or program learning Document review Activity documents 44 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity EQ/Obj. Result Area Issues of Interest Data Collection Instruments Data Sources Data Analysis Methods 2 Sustainability Result 1 How do the structures and processes established by HRH activities continue without Activity support? (Structure: HRH fora, HRM units) KIIs FMOH RHBs, TIs, Jhpiego Quantitative analysis (trend analysis, unit cost calculation, comparative analysis between target and actuals Triangulation of data among sources, JHPIEGO, FMOH, RHB, & TIs) Qualitative analysis of perceptions and views of beneficiaries (Govt structures, TI, and Trainees) 2 Sustainability Result 1 How would you rate the ability of RHBs to function without TA support? Can the HRM training continue without Activity support? KIIs FMOHRHBs, TIs, Jhpiego 2 Sustainability Result 1 What do you think about the continuity of HRM IST without Activity support? KII FMOH RHBs 2 Sustainability Result 1 How can the HRH fora (national and regional) continue without Activity support? KII FMOH RHBs, TIs, Jhpiego 2 Sustainability Result 2 What is the ability of TIs to continue producing quality human resources without support? KII FMOH, RHBs and TIs, Jhpiego, MOE 2 Sustainability Result 2 Can the Blended training continue without Activity support? KII FMOH, RHBs, TIs, Jhpiego 2 Sustainability Result 2 What is the level of readiness of RHBs and TIs to produce essential health workers by themselves? KII FMOH, RHBs, TIs, Jhpiego 2 Sustainability Result 3 % of Health Science TIs with the capacity to continue the current level of pre￾service education without further SHRH support KII FMOH, RHBs, TIs 2 Sustainability Result 3 What is the number of regions with the capacity to implement current levels of in￾service training without further KII FMOH, FMHACA, RHBs, TIs 45 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity EQ/Obj. Result Area Issues of Interest Data Collection Instruments Data Sources Data Analysis Methods support from SHRH Activity? 2 Sustainability Result 4 For the TI supported to start a new program, How would you describe their ability to undertake researches on their own? KII FMOH, RHBs, TIs 2 Sustainability Result 4 What is the ability of FMOH, RHBs, and Woredas to undertake routine HRH monitoring regularly on their own (without Activity support)? KII FMOH, RHBs, TIs 3 Gender Result 1 What new roles/positions have been created with the support of the HRH Activity? What is the proportion of women in the HR roles that were created and filled Document review KII Activity documents; KII with implementers, FMOH and RHBs Quantitative analysis (disaggregation sex) Qualitative analysis (views & perceptions of different stakeholders and beneficiaries on gender 3 Gender Result 1 What interventions have been implemented with the HRH support to make the workplace responsive to the needs of female workers? In what ways have these interventions contributed to improving the workplace environment for women, and with the retention of women? Document review KII Activity documents; KII with implementers, FMOH and RHBs 3 Gender Result 2 What has the HRH Activity done to increase the proportion of female workers? Document review KII Activity documents; KIIs with implementers, TIs 3 Gender Result 2 What is the increase in the percentage of female students in pre-service education? Document review KII Activity documents; KIIs with implementers, TIs 46 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity EQ/Obj. Result Area Issues of Interest Data Collection Instruments Data Sources Data Analysis Methods 3 Gender Result 2 What is the increase in the percentage of female students in in￾service training Document review KII Activity documents; KIIs with implementers, TIs 3 Gender Result 3 # and % of female students financially and technically supported Document review FDGs Activity documents; KIIs with implementers, TIs and FDGs with beneficiaries 3 Gender Result 4 How has the research conducted, incorporated gender as a priority topic? Document review FDGs Activity documents; KIIs with implementers, TIs and FDGs with beneficiaries 3 Gender Result 4 To what extent was there a concerted effort in building the capacity of women to undertake research? Document review FDGs Activity documents; KIIs with implementers, TIs and FDGs with beneficiaries 47 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity ANNEX 3: STATEMENT OF WORK END OF PROJECT PERFORMANCE EVALUATION SUPPORT TO STRENGTHENING HUMAN RESOURCES FOR HEALTH, CONTRACT NUMBER AID￾663-A-12-00008 I. INTRODUCTION Ethiopia faces a high burden of morbidity and mortality largely from communicable diseases, nutritional disorders, and poor maternal and child health outcomes. The maternal mortality (412/100,000) and under-five mortality (67 per 1,000 live birth) rates are unacceptably high,14 and only 26% of mothers deliver with the assistance of skilled birth attendants.15 The contraceptive prevalence rate is 35%, and 62% of pregnant women receive at least one antenatal care visit2 . The lack of access to care necessitates an increase in the production and retention of qualified and competent health professionals to meet the needs of a largely rural population. Several critical factors affect the current human resources for health (HRH) status in the country, including poor quality pre￾service education (PSE) and in-service training, rapid turn-over of skilled health care providers, inequitable distribution of the health workforce, underdeveloped regulatory capacity to sustain human resources for health quality, and ineffective management of human resources for health systems. Findings from the 2012 human resources for health rapid situational assessment16 suggested that there are critical human resource shortages for midwives and anesthetists in most regions. There is an unmet need of 63% (4,040) for midwives, with regional variations, and greatest unmet need observed in Amhara (84 %). For anesthetists, the unmet need is at 44% (146), with much higher needs in certain regions such as Gambella (93% unmet needs) and Somali (89% unmet needs). For each of these cadres, there are several regions for which data is not available. The assessment also revealed a weak human resource management structure with sub-optimum staffing at all levels of the health system. Lack of systems and institutional standards for training and Human Resource Management (HRM) capacity development were cited as an impediment for effective human resources management and development in the country. Though the Federal Ministry of Health (FMOH) has successfully trained and deployed over 38,000 health extension workers (HEWs), there is an approximately 5% attrition rate, and there are some regions that still have high unmet needs for HEWs including Somali (34%) and Tigray (19%)3 . Though there are significant challenges related to HRH, there have been efforts made to address these challenges in Ethiopia. The Health Sector Transformation Plan (HSTP) of the GoE has resulted in an increased number of health care facilities within the three-tiered health care system, reaching from specialized urban hospitals to satellite rural health posts. Efforts to date have included a rapid scale-up of HEWs, increased attention to the production of midwives and anesthetists, and a master’s level program for Integrated Emergency Surgery and Obstetrics (IESO) for health officers. USAID’s evaluation policy encourages independent external evaluation to increase accountability to inform those who develop programs and strategies and to refine designs and introduce improvements into future efforts. In keeping with that aim, this evaluation will be conducted to review and evaluate the performance of the USAID- Strengthening Human Resources for health program activities implemented 14 Demographic and Health Survey (2011). Central Statistical Agency, Addis Ababa, Ethiopia and ICF International, Calverton, Maryland, USA 15 Ethiopia Mini Demographic and Health Survey 2014. Central Statistical Agency Addis Ababa, Ethiopia July 2014 16 Human Resources for Health Rapid Situational Assessment on Prioritized Geographic Areas, Jhpiego, 2012 48 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity by Jhpiego/Ethiopia. The evaluation will focus on assessing the program’s six years performance in achieving its program goal, objectives, and results. II. BACKGROUND The Strengthening Human Resources for Health (HRH) project is a six year (2012 – 2018) $55,000,000 budget bilateral cooperative agreement with an overall goal of improving the human resources for health status in Ethiopia. Funded by the United States Agency for International Development (USAID) under the “Investing in People” category of the foreign assistance framework, the project is contributing to the Government of Ethiopia’s (GOEs) efforts to build or strengthen local capacity for comprehensive and sustainable systems for developing and for managing human resources for health. The project is being implemented by a Jhpiego-led partnership that includes Management Sciences for Health (MSH), the Ethiopian Midwives Association (EMWA), the Ethiopian Association of Anesthetists (EAA), and the Open University (OU), in collaboration with the Federal Ministry of Health (FMOH). MSH is primarily responsible for Result One of the agreement which is improving Human Resources for Health. EAA and EMA are required to contribute to Result Three: Improved quality of training of HCW, particularly focusing on their respective cadres, i.e., anesthetists and midwives respectively. Open University is expected to contribute to Result Four of the project: Program Learning and Research Conducted The Activity is currently implemented in all eleven regions of the country and the support is mainly provided at National level (FMOH) and regional level (mainly RHB and Regional Health Science Colleges). Activity Name Strengthening Human Resources for health Award # AID-663-A-12-00008 Start and End Date 05/18/2012 – 09/30/2018 LOP budget $55,000,000 Funding Source Global Health (MCH, FP/RH, TB, Malaria, HIV) Implementing partner(s) Jhpiego/Ethiopia AOR Name Helina Worku III. PROJECT GOAL AND OBJECTIVES The goal of this SHRH project is to improve health outcomes for all Ethiopians (with emphasis on the reduction of infectious disease and gender-focused disparities in maternal and newborn mortality) through improved availability, distribution, and management of health work force throughout the 49 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity country. Accordingly, the project emphasizes achieving four major results related to human resources for health: Result 1: Improved Human Resources for Health Management To improve HRH management, the partner has been building the capacity of HR managers at all levels, reviewing existing retention strategies and proposing ways to improve it, reviewing existing HRH policies and practices and advocating for improved regulatory activities related to HRM. Result 2: Increased Availability of Midwives, Anesthetists, HEWs and other Essential Health Workers The partner’s main Activity to increase the availability of competent HCWs focused on building the capacity of local TIs, fostering the development of curricula and providing ongoing technical support during the implementation of training to support training of midwives, anesthetists, HEWs and other essential health workers. Result 3: Improved Quality of Training of Health Workers The activities implemented to improve quality of training of HCWs focused on improving the quality of pre-service education (PSE) and in-service training (IST) through improving clinical education sites and teaching facilities, the introduction of CPD courses, and the establishment of IST centers, etc. Result 4: Program Learning and Research Conducted The project’s focus was to generate evidence on key HRH issues through research and routine monitoring and evaluation of activities. IV. PURPOSE AND USE OF THE EVALUATION The conclusions and recommendations of this evaluation will be used to inform and guide the design of future similar Activity. Particularly, the development of the results framework for a future similar Activity will base on the gaps and bottlenecks identified through this evaluation. Therefore, a thorough assessment of the remaining challenges in the Human Resources for Health, particularly in improving quality of education is a required output of this evaluation. V. SCOPE OF THE EVALUATION The project is implemented at the national level as well as in all the eleven regions of the country where it supported the Federal Ministry of Health (FMOH); Federal Ministry of Education (FMOE); 11 RHBs; Food, Medicine and Healthcare Administration and Control Authority (FMHACA); Higher Education Relevance and Quality Assurance Agency (HERQA); Universities and Regional Health Science Colleges (RHSC). As it is not possible to conduct the evaluation in all regions, a convenient sampling will be done to identify at least two Agrarian regions, one City Administration and at least one Pastoralist region. VI. EVALUATION QUESTIONS The evaluation team will undertake the SHRH evaluation with a Collaborating, Learning, and Adapting (CLA) lens. The data collection tools, analysis, and report must identify the contributions and/or potential contributions of HRH to CLA. Specific evaluation questions are listed below: a. In what ways did the SHRH Activity contribute to improving HRH in Ethiopia? (Questions should examine the Activity’s contribution to improving HRH management, increasing supply and availability of midwives, anesthetists, health extension workers and other essential cadres, 50 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity improving quality of pre-service and in-service training of health workers and ensuring generation and utilization of the right type of evidence on HRH) b. What facilitated and/or hindered achievement of planned results? c. How effective were the approaches used to strengthen the regulatory environment related to HRH activities? d. To what extent did the SHRH Activity interventions facilitate local ownership, coordination and partnerships and sustainability? To what extent were gender issues addressed in pre-service and in-service trainings? VII. EVALUATION METHODS The evaluation team will be responsible for developing an evaluation design and methodology that include a mix of qualitative and quantitative data collection and analysis approaches. The methodology will be presented as part of the draft work plan as outlined in the deliverables below and included in the final report. The evaluation team will review a variety of Activity implementation documents and reports to inform the evaluation design, analysis and report writing. Methodology strengths and weaknesses as well as measures taken to address those weaknesses should be discussed in the evaluation report. All data collected and presented in the evaluation report must be disaggregated, as appropriate, by sex and geography. The following table provides a summary of evaluation design and methodology and supplements the narrative section above. 51 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity Questions Suggested Data Sources Suggested Data Collection Methods Data Analysis Methods Indicator (if any) 1. In what ways did the SHRH Activity contribute to improving HRH? Effectiveness?? Project documents like MEL Plan, annual reports, baseline evaluation, national FMOH annual report, FMOH staff and RHB staff, project staff, beneficiaries, etc. Structured questionnaire, Key informant interview, focus group discussions, observation, desk review To be defined by the evaluation team To be defined by the evaluation team 2. What facilitated and/or hindered achievement of results? Project documents like MEL Plan, annual reports), baseline evaluation, national FMOH annual report, FMHACA staff and RHB staff, project staff, etc. Structured questionnaire, Key informant interview, focus group discussions, observation, desk review To be defined by the evaluation team To be defined by the evaluation team 3. How effective were the approaches used to strengthen the regulatory environment related to HRH activities? annual reports, baseline evaluation, national FMOH annual report, FMOH & FMHACA staff and RHB staff, project staff, etc. Structured questionnaire, Key informant interview, focus group discussions, observation, desk review To be defined by the evaluation team To be defined by the evaluation team 4. To what extent did the HRH Project interventions facilitate local ownership by stakeholders, coordination and partnership by government organizations and institutions, and sustainability of the intervention? annual reports, baseline evaluation, national FMOH annual report, FMOH & FMHACA staff and RHB staff, project staff, etc. Structured questionnaire, Key informant interview, focus group discussions, observation, desk review To be defined by the evaluation team To be defined by the evaluation team 5. To what extent were gender issues addressed in pre-service and in-service trainings? annual reports, baseline evaluation, national FMOH annual report, FMOH & FMHACA staff and RHB staff, project staff, beneficiaries, etc. Structured questionnaire, Key informant interview, focus group discussions, observation, desk review To be defined by the evaluation team To be defined by the evaluation team 52 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity 5. EXISTING PERFORMANCE INFORMATION SOURCES The consultants will review the following documents: a) Program Descriptions and Modifications b) Annual Work Plan c) Quarterly Reports d) Annual Reports e) Activity M&E Plan f) Annual HRH profile data g) Project-generated researches and assessments h) National HRH Strategic Plan, 2016 – 2020 The following entities are expected to be included in the interview:  Federal Institutions: FMOH, FMOE, FMHACA, HERQA  Regional Institutions: RHBs, RHSC  Academic Institutions (Universities and RHSC)  Partners: MSH  Professional Associations: Ethiopian Midwives Association, Ethiopian Economists Association 6. TEAM COMPOSITION The evaluation team shall consist of a team leader who leads the team of four other local experts with high level experience in human resources for health. All experts must be fluent in English and have strong writing skills. The experts should have experience with human resources for health programming in Ethiopia and monitoring and evaluation. A statement of potential bias or conflict of interest (or lack thereof) is required of each team member. Team Member Qualification and Specific Responsibilities The evaluation team will be composed of International and Ethiopian staff who have solid knowledge and understanding of the Ethiopian HRH work and have expertise in Activity evaluation. 1. Team Leader (One): The Ethiopian team leader should have at least ten years of work experience in human resources for health in Ethiopia or other African countries and led at least three similar evaluations. The evaluation team leader must have strong team management skills, and sufficient experience with evaluation standards and practices to ensure a credible product. The team leader must also be fluent in English and have strong writing skills. The evaluation team leader, in consultation with other team members, will be responsible for team coordination and performance and for ensuring the timeliness and quality of deliverables. The team leader is also responsible to ensure that the evaluation report meets the USAID Evaluation Standards (see Annex A.) 2. Team Member 1: This expert should have at least seven years of work experience in human resources for health in East Africa, preferably in Ethiopia and has been part of at least one evaluation of similar nature. 53 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity 3. Team Member 2: This expert will have at least seven years of work experience in human resources for health in East Africa, preferably in Ethiopia and has been part of at least one evaluation of similar nature. 4. Team Member 3: This expert should have at least seven years of work experience in human resources for health in East Africa, preferably in Ethiopia and has been part of at least one evaluation of similar nature. 5. Team Member 4: This expert should have at least seven years of work experience in human resources for health in East Africa, preferably in Ethiopia and has been part of at least one evaluation of similar nature The contractor may also hire other low-level personnel as required and appropriate with defined responsibilities. 7. EVALUATION SCHEDULE The estimated time for undertaking this evaluation is 84 working days including time for review of documents and production of the final report. The ideal available time for the evaluation team is January 8, 2017 – April 24, 2018, however, the date will be finalized between USAID and the EPMES Contractor. The evaluation team is required to work six days a week, but with no premium payment for sixth day. The team is required to travel to selected provinces in each region where program activities are being implemented. At least 25% of the consultants’ time will be spent to conduct interviews with project staff, government partners, and project beneficiaries. The evaluation team will prepare an exit briefing and presentation of the findings, which it will deliver to USAID staff and other stakeholders as appropriate. Following from this briefing, the consultants will put together the draft report. USAID and other stakeholders such as implementing partner will have 2 weeks to provide comments and suggestions to the report. Comments from USAID and stakeholders will be incorporated before the submission of the final draft. The draft report of the evaluation will have to be available by mid-February. Level of Effort (LOE) in person days: Activity Team Leader [1] Abebe Ethiopian Expert [1] Ethiopian Expert [1] Ethiopian Expert [1] Ethiopian Experts [1] Desk review of documents and analysis of indicator and other secondary data 3 3 3 3 3 Introductory meeting with USAID 1 1 1 1 1 Introductory meeting with implementing partner (s) 1 1 1 1 1 Inception Report preparation (evaluation work plan and survey instruments; plan logistics) 14 14 14 14 14 Presentation to USAID on the Evaluation Design and Work Plan 1 1 1 1 1 Submit final inception report to USAID 3 3 3 3 3 54 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity Field Work (Data collection in Addis and other areas. This includes any data collection tool test) 20 20 20 20 20 Field work experience debrief to USAID 1 1 1 1 1 Data analysis, preliminary report and presentation preparation 8 8 8 8 8 Final exit presentation to USAID (with PowerPoint presentation report) 1 1 1 1 1 Draft Evaluation Report Preparation 15 15 15 15 15 Final Evaluation Report (including a Two-page Briefer) 15 15 5 5 5 Total 83 83 73 73 73 8. MANAGEMENT Social Impact, the Contractor managing the Ethiopia Monitoring and Evaluation Service (EPMES) Activity will identify and hire the evaluation team, pending the Contract Officer’s Representatives (COR’s) and Health office’s concurrence and CO approval for the rate; assists in facilitating the work plan, and arrange meetings with key stakeholders identified prior to the initiation of the fieldwork. The evaluation team will organize other meetings as identified during the course of the evaluation, in consultation with EPMES’s Contractor and USAID/Ethiopia. The EPMES Contractor is responsible for all logistical support required for the evaluation team, including arranging accommodation, security, office space, computers, Internet access, printing, communication, and transportation. The evaluation team will officially report to the Ethiopia Monitoring and Evaluation Service (EPMES) Contractor, Social Impact. The EPMES Contractor is responsible for all direct coordination with the USAID/Ethiopia Program Office through the EPMES COR. From a technical management perspective, the evaluation team will work closely with Helina Worku, the AOR of the Activity in the Health, AIDS, Population and Nutrition (HAPN) Office to maintain objectivity, all final decisions about the evaluation will be made by the Program Office. 9. LOGISTICS EPMES will be responsible for all travel and logistics associated with conducting the evaluation. 10. REPORTING REQUIREMENTS AND DELIVERABLES Description and Timeline of Deliverables 1. In-briefing: Within 48 hours of the availability of the evaluation team in the EPMES’s Contractor Office, the evaluation team will have an in-brief meeting with USAID/Ethiopia’s Program Office and HAPN for introductions; presentation of the team’s understanding of the assignment and initial assumptions. This pre-briefing meeting helps the evaluation team develop evaluation work plan/evaluation design and prepare for the subsequent presentation to USAID, which happens after two weeks of the initial introductory meeting. 55 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity 2. Evaluation Work Plan/Inception Report: Within fifteen working days following the in-brief presentation, the Contractor shall submit the evaluation work plan (evaluation inception report) to USAID/Ethiopia’s Program Office and HAPN. This work plan/inception report will include: (a) the overall evaluation design, including the proposed methodology, data collection and analysis plan, and data collection instruments; (b) a list of the team members and their primary contact details while in-country, including the e-mail address and mobile phone number for the team leader and (c) the team’s proposed schedule for the evaluation; and. USAID offices and relevant stakeholders are asked to take up to three working days to review and consolidate comments through the EPMES COR. Once the evaluation team receives the consolidated comments on the work plan/inception report, they are expected to return with a revised work plan/inception report within two working days. The revised work plan shall include the list of potential interviewees and sites to be visited. USAID Offices send their final comments/say on the Contractor’s re-submitted documents/work plan within two working days of receipts revised documents/work plan from the Contractor and the Contractor proceeds accordingly. 3. Fieldwork Debrief: The Team or the Contractor is expected to provide the COR for EPMES and Social Impact with periodic briefings (written, email, or using any other appropriate medium) and feedback on the evaluation progress. If desired or necessary, weekly briefings by phone can be arranged with the Program Office and the Health Office to provide updates on field progress and any problems encountered. Immediately after the team’s completion of the fieldwork, the team shall provide a debrief to USAID to discuss on and learn about field level data collection experiences as well as the evaluation team members’ preliminary impression on evaluation findings. 4. Final Exit Presentation (PowerPoint Presentation) to USAID and relevant partners that will include a summary of key findings and key conclusions as these relate to the evaluation’s questions and recommendations to USAID. To be scheduled as agreed upon during the in￾briefing. A copy of the PowerPoint file will be provided to the Program Office, at least three￾days before the final exit presentation day. The COR for EPMES shall compiles comments from participants in this presentation and submit it to the Contractor for consideration during the preparation of the report. 5. Draft Evaluation Report: The content of the draft evaluation report is outlined in Annex A below, and all formatting shall be consistent with the USAID branding guidelines. The focus of the report is to answer the evaluation questions and may include factors the team considers having a bearing on the objectives of the evaluation. Any such factors can be included in the report only after consultation with USAID. The Contractor will submit the draft evaluation to the Program Office within 15 working days after exist presentation and should incorporate comments made during the exit presentation. USAID’s Program Office, Health Office and other partners will have 10 working days to review and comment on the draft report and the Program Office shall submit consolidated comments to the Contractor. The Contractor will then have 10 working days to make appropriate edits and revisions to the draft and re-submit the revised final draft report to USAID. The Program Office, Health Office and other partners will have 10 working days after the submission of the second revised draft to again review and send any final comments. 6. Final Evaluation Report will incorporate final comments provided by the Program Office. The length of the final evaluation report should not be more than 45 pages, not including Annexes and Executive Summary. The Contractor should submit the final report to the Program Office within 10 days of receipt of comments. The Final Evaluation Report submission should also include a Two-pager briefer on key qualitative and quantitative findings and conclusions relative to the evaluation questions—to be given to the appropriate government counterpart(s) so that they can review evaluation findings and share them with the larger community. 56 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity 7. All project data and records will be submitted in full and shall be in electronic form in easily readable format; organized and fully document for use by those not fully familiar with the project or evaluation; and owned by USAID and made available to the public, barring rare exceptions, on the USAID Development Experience Clearinghouse (http://dec.usaid.gov). 57 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity ANNEX 4: LIST OF DOCUMENTS REVIEWED S/N Title Source 1 Activity Monitoring, Evaluation & Learning Plans JHPIEGO 2 Health and Health Relation Indicator 2017 FMOH 3 HRH Strategy -July 2015 JHPIEGO 4 Rapid Site-Level HRH Assessment Draft Report_8-23-2016 JHPIEGO 5 Activity Annual Reports (2012 – 2017) JHPIEGO 6 Activity Data Quality Assessment Reports JHPIEGO 7 Activity Annual Workplans (2013 – 2016) JHPIEGO 8 Summary of HRH FY17 High-Level Accomplishment JHPIEGO 9 TOR for HRH Coordination Mechanism- December 2013 JHPIEGO 10 Factors Affecting Turnover Intention among Nurses in Ethiopia JHPIEGO 11 Education, Practice, and Competency Gaps of Anesthetists in Ethiopia JHPIEGO 12 HRH Baseline Survey Report JHPIEGO 13 Task analysis study Report for 5 Cadres – December 2016 JHPIEGO 14 ETH85152 – Civil Service Proclamation JHPIEGO 15 05 July 2012 JHPIEGO Program Description JHPIEGO 16 Strengthening – HRH – Research-compendium JHPIEGO 17 SHRH Activity RFA - Instructions JHPIEGO 18 Global HRH strategy 2030 JHPIEGO 19 Health Professional Recruitment, Transfer & Training Guidelines JHPIEGO 20 Health Professional Career Structure Guideline FMOH 21 Health Sector Development Program IV 2010/11 – 2014/15 (2003- 2007EC). Final draft. Addis Ababa. MOH 2010a 22 Human Resource for Health Strategic Plan, Ethiopia 2009-2020. Addis Ababa MOH 2015a. 23 National Health Policy.2015, Addis Ababa. MOH 2015b 24 Health Sector Transformation Plan HSTP 2015/16 - 2019/20 (2008-2012 EFY). Addis Ababa MOH 2015c 58 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity ANNEX 5: LIST OF KEY INFORMANTS INTERVIEWED DURING DATA COLLECTION AT DIFFERENT LEVELS Government Development Partners Implementing Partners Federal Level Human Resource Management Directorate (FMOH) USAID (Cognizant Technical Officer) Project Mercy Human Resource Development Directorate (FMOH) United Nations Population Fund (UNFPA) Jhpiego management team National licensing examination for health professional directorate (FMOH) MSH Food, Medicine, and Healthcare Administration and Control Authority Ethiopian Association of Anesthetists Federal Ministry of Education (FMOE) Ethiopian Midwives Association Education Strategic Centre Higher Education Relevance and Quality Assurance Agency (HERQA) Teleconference with Open University Professional Associations (Ethiopian Medical Association [EMA], Ethiopian Public Health Association, nursing) Regional Level RHB (head, HRH process owner) Jhpiego branch offices Bureau of Finance and Economic Development (health sector coordinator) African Medical Research Foundation Civil Service Office Columbia University’s ICAP Training Institutes (in-service training and continuous professional development) Centre for International Reproductive Health Training (CIRHT) TIs (Universities and RHSCs) Managers Trainers Trainees HRH Alumni 59 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity ANNEX 6: DATA COLLECTION TOOLS Key Informant Interview Guides Introduction: This introduction page will be attached as an introduction preceding all the tools below. Let us first introduce ourselves. Our names are: …………………………………………. We are consultants working for Social Impact Inc. an international consulting firm, which is evaluating USAID support to the Government of Ethiopia (GoE) through the Strengthening Human Resources for Health (SHRH) Activity. This is an end of Activity evaluation and the purpose of the evaluation is to assess any progress made and any challenges encountered, by the Activity and to inform future design of similar interventions and support. The evaluation team has had the opportunity to review Activity documents to have a good understanding of the design and implementation of the Activity. However, review of the documents alone is not enough. Therefore, we would like to speak with you, as you have been a key stakeholder in the Activity, to hear about your experience, in your own words, to help us to better understand how this Activity is working. Confidentiality:  The information that we will collect will include individuals’ names, organizations, and positions. The Annex of the evaluation report will include a list of respondents, but the findings or statements in the report will not be associated to any particular name of the key informant in order to protect the confidentiality of the respondents.  Quotes from respondents will be included in the evaluation report, but there will be no link between the quotes and the names of the individuals who provided the quote. In the event that the team desires to use any personally identifiable information in the report (such as a photograph of the person), the evaluators will first contact the respondent(s) to seek permission to do so.  The information that we shall collect during this evaluation will be used for the sole purposes of this evaluation. This information will not be used for any other purpose.  Your participation in this interview is voluntary – you can withdraw from participating in the interview if you so wish. If you do not feel comfortable answering any particular question, please let us know and we will simply go on to the next question.  Do you give us permission to conduct the interview with you? i. Key Informant Interview Guides: FMOH-HRM Directorate & RHB Offices Interviewee’s Name and Title: ___________________________________ Designation: ___________________________________________ Location: _______________________________________ Date: ___________________________________ 60 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity EQ1: In what ways did the SHRH Activity contribute to improving HRH in Ethiopia? 1. What do you consider to be the main contributions/achievements of the SHRH Activity to the health sector in Ethiopia? Probe: Did the Activity make any improvements to the human resources for health? How? 2. What gaps have you observed in their contribution to HRH? 3. In your opinion, how has the SHRH Activity impacted on the following? Probe for: a. Use of Staff Needs Assessments to inform HR planning and forecast b. HRM process and procedure c. Partnership and collaboration d. HRH fora e. Attrition and improving motivation? f. Regulatory frameworks and practices g. Availability of health workers; HEW, Midwives and anesthetists as well as other essential and clinical cadres? h. Quality of education (IST and PSE, & Blended learning) i. Research (Probe for the contribution on policy and program learning) Further probe: If perception is that it met objectives – probe for drivers to meeting objectives; if perception is that it did not meet objectives – probe for gaps and challenges. Also probe for any innovations EQ2: To what extent have SHRH Activity result/sub-result areas and interventions continued to be relevant to HRH gaps and challenges in Ethiopia? 1. Which of the following SHRH initiatives do you consider relevant to address HRH gaps in Ethiopia? Probe for: why relevant and not relevant; meeting community needs and alignment with government priorities, policies & strategies a. Improving HRM (HR motivation and retention, development policies and practices, building management capacity through IST) b. Accreditation (policy & legal framework) c. Strengthening the accreditation processes, practices and their enforcement d. Improving the availability of midwives, HEWs and Anesthetists other essential non-clinical health workers and other clinical health workers? e. Improving the quality of education through PSE, IST, and other supports? f. Relevance of researches and program learning activities to uncover pertinent issues? Please describe any gaps in this regard and what could be done to improve similar activities in the future EQ3: To what extent did the SHRH Activity interventions facilitate local ownership, coordination and partnerships and sustainability? Local ownership and sustainability: 2. What things do you know of, that the SHRH Activity did to promote local ownership of the Activity strategies and results? a. Were these approaches effective in building local ownership? How? b. Considering the different initiatives supported by SHRH, can you please tell us which ones you feel can be sustained by the FMOH/RHBs when the SHRH ends? Why? Also probe for why the others may not be sustainable. Initiatives: HRM; increasing number of health workers; improving quality; research 61 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity Efficiency and coordination 3. What is your perception of the cost-effectiveness of investments of the SHRH Activity in terms of: a. Capacity building of TIs, RHB and FMOH in improving HRH HRM b. Production and distribution of health workforce c. Improving quality of education d. Production and use of research evidence and program learning 4. What innovations introduced by the SHRH Activity do you think are cost effective and needs to be scaled up? Which ones are costly need to be re-examined? 5. What organizations/institutions (local and international) did the SHRH Activity coordinate with during the Activity implementation? Probe: a. What did they do with each of these organizations/institutions? b. What resulted from that coordination? c. What challenges did they experience in their coordination efforts? 6. In your opinion which of the SHRH Activity intervention are fully integrated and owned by the government? Which ones do you think can continue to be implemented through government financing at federal and regional levels? Why? Also probe for why the others may not be sustainable. a. Improving HRM (HR motivation and retention, development policies and practices, building management capacity through IST) b. Accreditation (policy & legal framework) c. Strengthening the accreditation processes, practices and their enforcement d. Improving the availability of midwives, HEWs and Anesthetists other essential non-clinical health workers and other clinical health workers? e. Improving the quality of education through PSE, IST, and other supports? f. Relevance of researches and program learning activities to uncover pertinent issues? EQ4: To what extent were gender issues addressed in pre-service and in-service trainings? 7. Can you please describe the main gender issues that you have observed in the health sector? (probe: Equal opportunity within employment; opportunities for jobs; training/capacity building/education; leadership roles and responsibilities); 8. To the best of your knowledge, how did SHRH Activity support gender issues? Probe: Focus on the following:) a. Increasing the role of women in the new structures and positions that were created and filled b. Making the workplace hospitable for women? c. Improving gender balance in increasing the number of midwives, HEWs, anesthetists and other essential health workers in PSE and in-service trainings d. Supporting female students technically and financially e. Extent of the researches1 and program learning activities1 that incorporated gender as priority topic and considered building capacity of women to undertake the research? Please describe the successes, and success factors as well as gaps and challenges in this regard? 9. Are there any other approaches that the SHRH Activity could have addressed gender issues better? How? 62 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity Recommendations 10. What priorities would you recommend for future HRH activities? Probe for: a. What should be the major priorities of the support activities (Management, Workforce supply, Quality of education, accreditation practices and environment) b. Innovative mechanisms/ways of implementing the support Activity? c. What type of support should other development and implementing partners provide to strengthen human resource for health in Ethiopia? (probe for which partners could support HRH) d. What should the government do at all levels to support management, increase the number and quality of HRH production, as well as working environment? e. Any additional recommendation? ii. Key Informant Guide; FMHACA Interviewee’s Name and Title: ___________________________________ Designation: ___________________________________________ Location: _______________________________________ Date: ___________________________________ EQ1: In what ways did the SHRH Activity contribute to improving HRH in Ethiopia? 1. What do you consider to be the main contributions/achievements of the SHRH Activity to the health sector in Ethiopia? Probe: Did the Activity make any improvements to the human resources for health? How? 2. What gaps have you observed in their contribution to HRH? 3. What is your perception about the progress that the SHRH Activity made towards meeting its objectives on the following? a. Capacity building b. Improving performance of the agency c. Improving regulatory frameworks and professional practices d. Improving quality of education e. Generation of timely evidence through research and their effect on policy and program learning: Further probe: If perception is that it met objectives – probe for drivers to meeting objectives and innovations; if perception is that it did not meet objectives – probe for gaps and challenges. Also probe for any innovations and suggestions EQ2: To what extent have SHRH Activity result/sub-result areas and interventions continued to be relevant to HRH gaps and challenges in Ethiopia? 4. Which of the following SHRH initiatives do you consider relevant to addressing HRH gaps in Ethiopia? Probe for: why relevant and not relevant; meeting community needs and alignment with government priorities, policies & strategies a. Accreditation (policy & legal framework) b. Strengthening the accreditation processes, practices and their enforcement 63 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity c. Capacity building—probe for what types of capacity d. Partnership and collaboration e. Enforcement regulation and licensing f. Relevance of researches and program learning activities to uncover pertinent issues? Please describe any gaps in this regard and what could be done to improve similar activities in the future EQ3: To what extent did the SHRH Activity interventions facilitate local ownership, coordination and partnerships and sustainability? Local ownership and sustainability: 5. What things do you know of, that the SHRH Activity did to promote local ownership of the Activity strategies and results? a. Were these approaches effective in building local ownership? How? b. Considering the different initiatives supported by SHRH, can you please tell us which ones you feel can be sustained by the FMOH/RHBs when the SHRH ends? Why? Also probe for why the others may not be sustainable. Initiatives: HRM; increasing number of health workers; improving quality; research Efficiency and coordination 6. What is your perception of the cost-effectiveness of investments of the SHRH Activity in terms of: a. Capacity building of TIs, RHB and FMOH in improving HRH HRM b. Production and distribution of health workforce c. Improving quality of education d. Production and use of research evidence and program learning 7. What innovations introduced by the SHRH Activity do you think are cost effective and needs to be scaled up? Which ones are costly need to be re-examined? 8. What organizations/institutions (local and international) did the SHRH Activity coordinate with during the Activity implementation? Probe: a. What did they do with each of these organizations/institutions? b. What resulted from that coordination? c. What challenges did they experience in their coordination efforts? 9. In your opinion which of the SHRH Activity intervention are fully integrated and owned by the government? Which ones do you think can continue to be implemented through government financing at federal and regional levels? Why? Also probe for why the others may not be sustainable. a. Improving HRM (HR motivation and retention, development policies and practices, building management capacity through IST) b. Accreditation (policy & legal framework) c. Strengthening the accreditation processes, practices and their enforcement d. Improving the availability of midwives, HEWs and Anesthetists other essential non-clinical health workers and other clinical health workers? e. Improving the quality of education through PSE, IST, and other supports? f. Relevance of researches and program learning activities to uncover pertinent issues? 64 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity EQ4: To what extent were gender issues addressed in pre-service and in-service trainings? 10. Can you please describe the main gender issues/problems that you have observed in the health sector? (probe: Equal opportunity within employment; opportunities for jobs; training/capacity building/education; leadership roles and responsibilities); 11. To the best of your knowledge, how did SHRH Activity support/address gender issues? Probe: Focus on the following:) a. Increasing the role of women in the new structures and positions that were created and filled b. Making the workplace hospitable for women? c. Improving gender balance in increasing the number of midwives, HEWs, anesthetists and other essential health workers in PSE and in-service trainings d. Supporting female students technically and financially e. Extent of the researches1 and program learning activities1 that incorporated gender as priority topic and considered building capacity of women to undertake the research? Please describe the successes, and success factors as well as gaps and challenges in this regard? 12. Are there any other approaches that the SHRH Activity could have addressed gender issues better? How? Recommendations 13. What priorities would you recommend for future HRH activities? Probe for: a. What should be the major priorities of the support activities (Management, Workforce supply, Quality of education, accreditation practices and environment) b. Innovative mechanisms/ways of implementing the support Activity? c. What type of support should other development and implementing partners provide to strengthen human resource for health in Ethiopia? (probe for which partners could support HRH) d. What should the government do at all levels to support management, increase the number and quality of HRH production, as well as working environment? e. Any additional recommendation? iii. Key Informant Guide; Civil Service Commission Interviewee’s Name and Title: ___________________________________ Designation: ___________________________________________ Location: _______________________________________ Date: ___________________________________ EQ1: In what ways did the SHRH Activity contribute to improving HRH in Ethiopia? 1. What do you consider to be the main contributions/achievements of the SHRH Activity to the health sector in Ethiopia? Probe: Did the Activity make any improvements to the human resources for health? How? 2. What gaps have you observed in their contribution to HRH? 65 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity 3. Can you please describe how the Staff Needs Assessment supported by SHRH Activity helped to inform your decision for approving structures and positions in the health sector? 4. In your opinion, how effective were the HRM capacity building efforts of SHRH Activity to: a. Build HRH skills b. Develop tools to support HRH work c. Establish systems for HRH? 5. What efforts of the SHRH Activity helped with the motivation and retention of health workers? Further probe: If perception is that it met objectives – probe for drivers to meeting objectives and innovations; if perception is that it did not meet objectives – probe for gaps and challenges. Also probe for any innovations and suggestions EQ2: To what extent have SHRH Activity result/sub-result areas and interventions continued to be relevant to HRH gaps and challenges in Ethiopia? 6. In what ways has SHRH Activity contributed to addressing attrition rates of health sector staff? 7. Can you please provide an overview of how the following HRM initiatives have helped to meet the needs of the health sector? Probe: do you think the health sector perform better than other sector due to these supports? a. Motivation and retention of staff b. HRM policies and practices c. Continuous Professional Development 8. In what ways are the following SHRH Activity initiatives relevant to the HRH issues in Ethiopia? Please provide justification a. Continuous Professional Development b. Staff needs assessment projection (HR planning) c. HRM policies and practices d. In-service training for HRM personnel e. Motivation & Retention scheme EQ3: To what extent did the SHRH Activity interventions facilitate local ownership, coordination and partnerships and sustainability? Local ownership and sustainability: 9. What things do you know of, that the SHRH Activity did to promote local ownership of the Activity strategies and results? a. Were these approaches effective in building local ownership? How? b. Considering the different initiatives supported by SHRH, can you please tell us which ones you feel can be sustained by the FMOH/RHBs when the SHRH ends? Why? Also probe for 66 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity why the others may not be sustainable. Initiatives: HRM; increasing number of health workers; improving quality; research Efficiency and coordination 10. What is your perception of the cost-effectiveness of investments of the SHRH Activity in terms of: a. Capacity building of TIs, RHB and FMOH in improving HRH HRM b. Production and distribution of health workforce c. Improving quality of education d. Production and use of research evidence and program learning 11. What innovations introduced by the SHRH Activity do you think are cost effective and needs to be scaled up? Which ones are costly need to be re-examined? 12. What organizations/institutions (local and international) did the SHRH Activity coordinate with during the Activity implementation? Probe: a. What did they do with each of these organizations/institutions? b. What resulted from that coordination? c. What challenges did they experience in their coordination efforts? 13. In your opinion which of the SHRH Activity intervention are fully integrated and owned by the government? Which ones do you think can continue to be implemented through government financing at federal and regional levels? Why? Also probe for why the others may not be sustainable. a. Improving HRM (HR motivation and retention, development policies and practices, building management capacity through IST) b. Accreditation (policy & legal framework) c. Strengthening the accreditation processes, practices and their enforcement d. Improving the availability of midwives, HEWs and Anesthetists other essential non-clinical health workers and other clinical health workers? e. Improving the quality of education through PSE, IST, and other supports? f. Relevance of researches and program learning activities to uncover pertinent issues? EQ4: To what extent were gender issues addressed in pre-service and in-service trainings? 14. Can you please describe the main gender issues/problems that you have observed in the health sector? (probe: Equal opportunity within employment; opportunities for jobs; training/capacity building/education; leadership roles and responsibilities); 15. To the best of your knowledge, how did SHRH Activity support/address gender issues? Probe: Focus on the following:) a. Increasing the role of women in the new structures and positions that were created and filled b. Making the workplace hospitable for women? c. Improving gender balance in increasing the number of midwives, HEWs, anesthetists and other essential health workers in PSE and in-service trainings d. Supporting female students technically and financially 67 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity e. Extent of the researches1 and program learning activities1 that incorporated gender as priority topic and considered building capacity of women to undertake the research? Please describe the successes, and success factors as well as gaps and challenges in this regard? 16. What is your opinion regarding the structures and positions created and the proportion of women in health workforce and management? Probe: improved work environment for women 17. Are there any other approaches that the SHRH Activity could have addressed gender issues better? How? Recommendations 18. What priorities would you recommend for future HRH activities? Probe for: a. What should be the major priorities of the support activities (Management, Workforce supply, Quality of education, accreditation practices and environment) b. Innovative mechanisms/ways of implementing the support Activity? c. What type of support should other development and implementing partners provide to strengthen human resource for health in Ethiopia? (probe for which partners could support HRH) d. What should the government at all levels do to support management, increase the number and quality of HRH production, as well as working environment? e. Any additional recommendation? iv. Key Informant Interview Guide: Implementing/ Development Partners/EMA & EAA EQ1: In what ways did the SHRH Activity contribute to improving HRH in Ethiopia? 1. What do you consider to be the main contributions/achievements of the SHRH Activity to the health sector in Ethiopia? Probe: Did the Activity make any improvements to the human resources for health? How? 2. What gaps have you observed in their contribution to HRH? 3. What is your perception about the progress that the SHRH Activity made towards meeting its objectives on the following? a. Capacity building b. Improving performance of the agency c. Improving regulatory frameworks and professional practices d. Improving quality of education e. Generation of timely evidence through research and their effect on policy and program learning: Further probe: If perception is that it met objectives – probe for drivers to meeting objectives and innovations; if perception is that it did not meet objectives – probe for gaps and challenges. Also probe for any innovations and suggestions 4. In your opinion, how has the SHRH Activity impacted on the following? Probe for: examples a. Use of Staff Needs Assessments to inform HR planning and forecast b. HRM process and procedure c. Partnership and collaboration d. HRH fora e. Attrition and improving motivation? f. Regulatory frameworks and practices g. Availability of health workers; HEW, Midwives and anesthetists as well as other essential and clinical cadres? h. Quality of education (IST and PSE, & Blended learning) 68 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity i. Research (Probe for the contribution on policy and program learning) EQ2: To what extent have SHRH Activity result/sub-result areas and interventions continued to be relevant to HRH gaps and challenges in Ethiopia? 5. Which of the following SHRH initiatives do you consider relevant to address HRH gaps in Ethiopia? Probe for: why relevant and not relevant; meeting community needs and alignment with government priorities, policies & strategies a. Improving HRM (HR motivation and retention, development policies and practices, building management capacity through IST) b. Accreditation (policy & legal framework) c. Strengthening the accreditation processes, practices and their enforcement d. Improving the availability of midwives, HEWs and Anesthetists other essential non-clinical health workers and other clinical health workers? e. Improving the quality of education through PSE, IST, and other supports? f. Relevance of researches and program learning activities to uncover pertinent issues? Please describe any gaps in this regard and what could be done to improve similar activities in the future EQ3: To what extent did the SHRH Activity interventions facilitate local ownership, coordination and partnerships and sustainability? Local ownership and sustainability: 6. What things do you know of, that the SHRH Activity did to promote local ownership of the Activity strategies and results? a. Were these approaches effective in building local ownership? How? b. Considering the different initiatives supported by SHRH, can you please tell us which ones you feel can be sustained by the FMOH/RHBs when the SHRH ends? Why? Also probe for why the others may not be sustainable. Initiatives: HRM; increasing number of health workers; improving quality; research Efficiency and coordination 7. What is your perception of the cost-effectiveness of investments of the SHRH Activity in terms of: a. Capacity building of TIs, RHB and FMOH in improving HRH HRM b. Production and distribution of health workforce c. Improving quality of education d. Production and use of research evidence and program learning 8. What innovations introduced by the SHRH Activity do you think are cost effective and needs to be scaled up? Which ones are costly need to be re-examined? 9. What organizations/institutions (local and international) did the SHRH Activity coordinate with during the Activity implementation? Probe: a. What did they do with each of these organizations/institutions? b. What resulted from that coordination? c. What challenges did they experience in their coordination efforts? 10. In your opinion which of the SHRH Activity intervention are fully integrated and owned by the government? Which ones do you think can continue to be implemented through government 69 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity financing at federal and regional levels? Why? Also probe for why the others may not be sustainable. a. Improving HRM (HR motivation and retention, development policies and practices, building management capacity through IST) b. Accreditation (policy & legal framework) c. Strengthening the accreditation processes, practices and their enforcement d. Improving the availability of midwives, HEWs and Anesthetists other essential non-clinical health workers and other clinical health workers? e. Improving the quality of education through PSE, IST, and other supports? f. Relevance of researches and program learning activities to uncover pertinent issues? EQ4: To what extent were gender issues addressed in pre-service and in-service trainings? 11. Can you please describe the main gender issues that you have observed in the health sector? (probe: Equal opportunity within employment; opportunities for jobs; training/capacity building/education; leadership roles and responsibilities); 12. To the best of your knowledge, how did SHRH Activity support gender issues? Probe: Focus on the following:) a. Increasing the role of women in the new structures and positions that were created and filled b. Making the workplace hospitable for women? c. Improving gender balance in increasing the number of midwives, HEWs, anesthetists and other essential health workers in PSE and in-service trainings d. Supporting female students technically and financially e. Extent of the researches1 and program learning activities1 that incorporated gender as priority topic and considered building capacity of women to undertake the research? Please describe the successes, and success factors as well as gaps and challenges in this regard? 13. Are there any other approaches that the SHRH Activity could have addressed gender issues better? How? Recommendations 14. Please provide us your suggestions and recommendations regarding the: a. The priority of investment in management, supply, accreditation, quality of HRH in the medium to long term? b. Government roles (federal & region) in improving the leadership and management of human resource? c. Development partners roles in supporting the Ethiopian in HRH v. Key Informant Interview Guide: Health Institutions Interviewee’s Name and Title: ___________________________________ Designation: ___________________________________________ Location: _______________________________________ Date: ___________________________________ EQ1: In what ways did the SHRH Activity contribute to improving HRH in Ethiopia? 70 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity 1. What do you consider to be the main contributions/achievements of the SHRH Activity to this health institution? Probe: Did the Activity make any improvements to the human resources for health? How? 2. What gaps have you observed in their contribution/achievements at your institution? 3. What is your perception about the progress that the SHRH Activity made towards meeting its objectives on the following? a. Capacity building b. Improving performance of the agency c. Improving regulatory frameworks and professional practices d. Improving quality of education e. Generation of timely evidence through research and their effect on policy and program learning: Further probe: If perception is that it met objectives – probe for drivers to meeting objectives and innovations; if perception is that it did not meet objectives – probe for gaps and challenges. Also probe for any innovations and suggestions EQ2: To what extent have SHRH Activity result/sub-result areas and interventions continued to be relevant to HRH gaps and challenges in Ethiopia? 4. Thinking about this institution, how appropriate have the SHRH Activity interventions to your priorities for improving your capacity to train health workers? 5. What has been the effects of the SHRH Activity on the following: a. Faculty development b. Teaching material and infrastructure development c. Curriculum/module development d. Building the capacity of instructors, clinical preceptors and other teaching staffs e. Strengthening skill labs and practicum sites EQ3: To what extent did the SHRH Activity interventions facilitate local ownership, coordination and partnerships and sustainability? Local ownership and sustainability: 6. What things do you know of, that the SHRH Activity did to promote local ownership of the Activity strategies and results? a. Were these approaches effective in building local ownership? How? b. Considering the different initiatives supported by SHRH, can you please tell us which ones you feel can be sustained by the FMOH/RHBs when the SHRH ends? Why? Also probe for why the others may not be sustainable. Initiatives: HRM; increasing number of health workers; improving quality; research 7. Considering the different initiatives supported by SHRH Activity in your institution, can you please tell us which ones you feel can be sustained beyond the lifespan of the Activity? Also probe for why the others may not be sustainable. Efficiency and coordination 8. What is your perception of the cost-effectiveness of investments of the SHRH Activity in terms of: a. Capacity building of TIs, RHB and FMOH in improving HRH HRM b. Production and distribution of health workforce c. Improving quality of education d. Production and use of research evidence and program learning 71 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity 9. What innovations introduced by the SHRH Activity do you think are cost effective and needs to be scaled up? Which ones are costly need to be re-examined? 10. What organizations/institutions (local and international) did the SHRH Activity coordinate with during the Activity implementation? Probe: a. What did they do with each of these organizations/institutions? b. What resulted from that coordination? c. What challenges did they experience in their coordination efforts? EQ4: To what extent were gender issues addressed in pre-service and in-service trainings? 11. Can you please describe the main gender issues that you have observed in your institution? (probe: Equal opportunity within employment; opportunities for jobs; training/capacity building/education; leadership roles and responsibilities); 12. How did SHRH Activity support gender issues? Probe: Focus on the following:) a. Making the workplace hospitable for women? b. Improving gender balance in increasing the number of midwives, HEWs, anesthetists and other essential health workers in PSE and in-service trainings c. Supporting female students technically and financially d. Extent of the researches1 and program learning activities1 that incorporated gender as priority topic and considered building capacity of women to undertake the research? Drivers of success: Please describe the success factors any identified SHRH Activity success? Challenges: What were/are the challenges in the SHRH Activity? Recommendations If USAID considers supporting a future similar Activity, what would be your major recommendations? vi. Key Informant Interview Guide: FMOE, REB Interviewee’s Name and Title: ___________________________________ Designation: ___________________________________________ Location: _______________________________________ Date: ___________________________________ EQ1: In what ways did the SHRH Activity contribute to improving HRH in Ethiopia? 1. What do you consider to be the main contributions/achievements of the SHRH Activity to FMoE/REB? Probe: Did the Activity make any improvements to the human resources for health? How? 2. What gaps have you observed in the Activity’s contribution/achievements? 3. What is your perception about the progress that the SHRH Activity made towards meeting its objectives on the following? a. Capacity building b. Improving performance of the FMoE/REB c. Improving regulatory frameworks and professional practices d. Improving quality of education e. Generation of timely evidence through research and their effect on policy and program learning: 72 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity EQ2: To what extent have SHRH Activity result/sub-result areas and interventions continued to be relevant to HRH gaps and challenges in Ethiopia? 4. Which of the following SHRH initiatives do you consider relevant to address Health science training in Ethiopia? Probe for: why relevant and not relevant; a. Accreditation (policy & legal framework) b. Strengthening the accreditation processes, practices and their enforcement c. Improving the availability of midwives, HEWs and Anesthetists other essential non-clinical health workers and other clinical health workers? d. Improving the quality of education through PSE, IST, and other supports? e. Relevance of researches and program learning activities to uncover pertinent issues? Please describe any gaps in this regard and what could be done to improve relevance for the future Activity support 5. Please also describe for us the better approaches and innovation in improving the supply and quality of health workforce? What new approaches need to be considered? EQ3: To what extent did the SHRH Activity interventions facilitate local ownership, coordination and partnerships and sustainability? Local ownership and sustainability: 6. What things do you know of, that the SHRH Activity did to promote local ownership of the Activity strategies and results at the FMoE/REB? a. Were these approaches effective in building local ownership? How? b. Considering the different initiatives supported by SHRH, can you please tell us which ones you feel can be sustained by the FMOE/REBs when the SHRH ends? Why? Also probe for why the others may not be sustainable. Initiatives: HRM; increasing number of health workers; improving quality; research Efficiency and coordination 7. What is your perception of the cost-effectiveness of investments of the SHRH Activity in terms of: a. Capacity building of TIs, REB and FMOE b. Production of health workforce c. Improving quality of pre-service education d. Production and use of research evidence and program learning 8. What innovations introduced by the SHRH Activity do you think are cost effective and needs to be scaled up? Which ones are costly need to be re-examined? 9. What organizations/institutions (local and international) did the SHRH Activity coordinate with during the Activity implementation? Probe: a. What did they do with each of these organizations/institutions? b. What resulted from that coordination? c. What challenges did they experience in their coordination efforts? EQ4: To what extent were gender issues addressed in pre-service and in-service trainings? 10. Can you please describe the main gender issues that you have observed in the health sector? (probe: Equal opportunity within employment; opportunities for jobs; training/capacity building/education; leadership roles and responsibilities); 11. To the best of your knowledge, how did SHRH Activity support gender issues? Probe: Focus on the following:) 73 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity a. Increasing the role of women in the new structures and positions that were created and filled b. Making the workplace hospitable for women? c. Improving gender balance in increasing the number of midwives, HEWs, anesthetists and other essential health workers in PSE and in-service trainings d. Supporting female students technically and financially e. Extent of the researches1 and program learning activities1 that incorporated gender as priority topic and considered building capacity of women to undertake the research? Please describe the successes, and success factors as well as gaps and challenges in this regard? 12. Are there any other approaches that the SHRH Activity could have addressed gender issues better? How? Recommendations 13. Please provide us your suggestions and recommendations regarding the: a. The priority of investment in management, supply, accreditation, quality of HRH in the medium to long term? b. Government roles (federal & region) in improving the leadership and management of human resource? c. Development partners roles in supporting the Ethiopian in HRH d. Key Informant Interview Guide vii. Key Informant Interview guide: HERQA Interviewee’s Name and Title: ___________________________________ Designation: ___________________________________________ Location: _______________________________________ Date: ___________________________________ EQ1: In what ways did the SHRH Activity contribute to improving HRH in Ethiopia? 1. What do you consider to be the main contributions/achievements of the SHRH Activity to FMoE/REB? Probe: Did the Activity make any improvements to the human resources for health? How? 2. What gaps have you observed in the Activity’s contribution/achievements? 3. What is your perception about the progress that the SHRH Activity made towards meeting its objectives on the following? a. Capacity building b. Improving performance of the HERQA c. Improving regulatory frameworks and professional practices d. Improving quality of education 4. Generation of timely evidence through research and their effect on policy and program learning 5. In your opinion, what have been the effects of the SHRH Activity in ensuring high quality and relevant education in the supported TIs? Probe for: accreditation of TIs, quality standards for health programs, development of quality & relevance assurance policy, quality audit activities and SBM-R–related topics 6. What is your perception on the progress that the SHRH Activity made towards meeting its targets on the following? a. Improving regulatory frameworks and practices b. Increasing availability and reducing shortage of HEWs, Midwives, anesthetists as well as other essential and clinical cadres? c. Partnership and collaboration d. Improving quality of education through curriculum revision, accreditation as well as provision of PSE; effectiveness of blended learning 74 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity e. Generation of timely evidence through research and their effect on policy and program learning: Please also describe for us the better approaches and innovation in improving the supply and quality of health workforce? What new approaches need to be considered? EQ2: To what extent have SHRH Activity result/sub-result areas and interventions continued to be relevant to HRH gaps and challenges in Ethiopia? 7. What do you see the importance of the SHRH Activity on building the capacity of HERQA? Probe for: on accreditation of TIs, on improving quality standards for health programs, development of quality & relevance assurance policy and on quality audit activities. 8. How much are PSE interventions appropriate to improve the quality of education of health workers? EQ3: To what extent did the SHRH Activity interventions facilitate local ownership, coordination and partnerships and sustainability? Local ownership and sustainability: 9. What things do you know of, that the SHRH Activity did to promote local ownership of the Activity strategies and results at the FMoE/REB? a. Were these approaches effective in building local ownership? How? b. Considering the different initiatives supported by SHRH, can you please tell us which ones you feel can be sustained by the FMOE/REBs when the SHRH ends? Why? Also probe for why the others may not be sustainable. Initiatives: HRM; increasing number of health workers; improving quality; research 10. What activities supported by the SHRH Activity do you think could the government at all levels (FMOE, REBs, TIs) take over and implement without SHRH Activity support? Why and how? Which ones cannot be sustained and need further Activity support? Please provide evidence and justification Efficiency and coordination 11. What is your perception of the cost-effectiveness of investments of the SHRH Activity in terms of: a. Capacity building of HERQA Regional HERQA b. In enhancing the accreditation process, setting quality standards for health programs etc. c. Improving quality of education? d. Production and use of research evidence and program learning? 12. What innovations introduced by the SHRH Activity do you think are cost effective and needs to be scaled up? Which ones are costly need to be re-examined? 13. What organizations/institutions (local and international) did the SHRH Activity coordinate with during the Activity implementation? Probe: a. What did they do with each of these organizations/institutions? b. What resulted from that coordination? c. What challenges did they experience in their coordination efforts? EQ4: To what extent were gender issues addressed in pre-service and in-service trainings? 14. To the best of your knowledge, how did SHRH Activity support gender issues? Probe: Focus on the following:) a. Increasing the role of women in the new structures and positions that were created and filled 75 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity b. Making the workplace hospitable for women? c. Improving gender balance in increasing the number of midwives, HEWs, anesthetists and other essential health workers in PSE and in-service trainings d. Supporting female students technically and financially e. Extent of the researches1 and program learning activities1 that incorporated gender as priority topic and considered building capacity of women to undertake the research? Please describe the successes, and success factors as well as gaps and challenges in this regard? 15. Are there any other approaches that the SHRH Activity could have addressed gender issues better? How? Recommendations 16. Please provide us your suggestions and recommendations regarding the: a. The priority of investment in management, supply, accreditation, quality of HRH in the medium to long term? b. Government roles (federal & region) in improving the leadership and management of human resource? c. Development partners roles in supporting the Ethiopian in HRH 76 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity ANNEX 7: FOCUS GROUP DISCUSSION GUIDES Introduction will use the introduction for all tools: please refer to the beginning of this document 2.1 For Tutors/Instructors To what extent have SHRH Activity result/sub-result areas and interventions continued to be relevant to HRH gaps and challenges in Ethiopia? 1. Can you please describe all the activities that SHRH undertook at this institution? 2. What other activities do you know of that they carryout in this area (outside of the TI) 3. How would you assess the relevance of the support in facilitating your training activities? a. Increasing enrollment, infrastructure development, teaching material etc.? b. On quality of education in PSE c. Relevance of research and program learning activities to uncover pertinent HRH issues? Please describe any gaps in this regard and what could be done to improve relevance in future Activity? To what extent has the SHRH Activity progressed towards meeting its performance objectives? Effectiveness 4. In your opinion, which of the following activities did the SHRH perform very well? Why? And which did they not perform well? Why? a. Improving regulatory frameworks and practices b. Increasing availability and reducing shortage of HEWs, Midwives, anesthetists as well as other essential and clinical cadres? c. Improving quality of education through curriculum revision, accreditation as well as provision of PSE; effectiveness of blended learning d. Generation of timely evidence through research and their effect on policy and program learning: If they are effective, please describe for us the major drivers of success and innovation that need to be scaled up? If no, describe the major gaps and challenges? Please also describe for us the better approaches and innovation in improving the supply and quality of health workforce? What new approaches need to be considered? Efficiency 5. Which of the support you got do you think are less costly but valuable investments of the SHRH Activity in terms of Capacity building of TIs, REB and FMOE a. Production of health workforce b. Improving quality of pre-service education c. Production and use of research evidence and program learning 6. What innovation/s introduced by the SHRH Activity do you think is cost-effective and needs to be scaled up? Which one/s is not cost-effective and needs to be re-examined? Sustainability: 7. From the above activities that have been actively supported, which ones do you think could the government at all levels (FMOE, REBs, TIs) take over and implement without SHRH Activity support? 77 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity Why and how? Which ones cannot be sustained and need further Activity support? Please provide evidence and justification. To what extent were gender issues addressed in pre-service and in-service trainings? 8. Can you please describe the main gender issues that you have observed in the health sector? (probe: Equal opportunity within employment; opportunities for jobs; training/capacity building/education; leadership roles and responsibilities); 9. How far has the SHRH Activity responded to and supported gender issues in human resource development in terms of: a. Making the pre-service training workplace hospitable for women? b. Improving gender balance in increasing the number of midwives, HEWs, anesthetists and other essential health workers in PSE c. Supporting female students technically and financially d. Extent to which researches1 and program learning activities1 incorporated gender as a priority topic e. Building the capacity of women to undertake research? If positive, which? Elaborate. If no, why not? Please describe the successes, and success factors as well as gaps and challenges in this regard? Recommendations 10. If USAID considers supporting a future similar Activity, can you please propose the HRH priorities that they should consider? Probe for: a. What should be the major priorities of the support activities (Management, regulatory environment, Quality of service and licensing process) b. Innovative mechanisms/ways of implementing the support Activity? c. What should other development and implementing partners do to support the process? (probe for which partners could support HRH) d. Any additional recommendation? 2.2 For Students To what extent have SHRH Activity result/sub-result areas and interventions continued to be relevant to HRH gaps and challenges in Ethiopia? 1. Can you please describe all the activities that SHRH undertook at this institution? 2. What other activities do you know of that they carryout in this area (outside of the TI) 3. Which of the following SHRH initiatives do you consider relevant to meeting your training needs? a. Improving the availability of midwives, HEWs, Anesthetists and other essential non-clinical health workers and other clinical health workers? b. Improving the quality of education through PSE? c. Relevance of research and program learning activities to uncover pertinent HRH issues? Please describe any gaps in this regard and what could be done to improve relevance in future Activity? To what extent has the SHRH Activity progressed towards meeting its performance objectives? Effectiveness: 4. What is your perception on whether SHRH Activity met its objectives on the following? (provide evidence): 78 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity a. Tuition support for pre-service education b. Improving regulatory frameworks and practices c. Increasing availability and reducing shortage of HEWs, Midwives, Anesthetists as well as other essential and clinical cadres? d. Improving quality of education through curriculum revision, accreditation as well as provision of PSE; effectiveness of blended learning If perception is that it met objectives – probe for drivers to meeting objectives; if perception is that it did not meet objectives – probe for gaps and challenges. Also probe for any innovations Efficiency 4. How cost effective do you think the investments of the SHRH Activity are in terms of a. Production of health workforce b. Improving quality of pre-service education c. Production and use of research evidence and program learning What innovation/s introduced by the SHRH Activity do you think is cost-effective and needs to be scaled up? Which one/s is not cost-effective and needs to be re-examined? Sustainability: 5. From the above activities that have been actively supported, which ones do you think could the government at all levels (FMOE, REBs, TIs) take over and implement without SHRH Activity support? Why and how? Which ones cannot be sustained and need further Activity support? Please provide evidence and justification. To what extent were gender issues addressed in pre-service and in-service trainings? 6. How far has the SHRH Activity responded to and supported gender issues in human resource development in terms of: a. Making the pre-service training workplace hospitable for women? b. Improving gender balance in increasing the number of midwives, HEWs, anesthetists and other essential health workers in PSE c. Supporting female students technically and financially d. Building the capacity of women to undertake research? If positive, which? Elaborate. If no, why not? Please describe the successes, and success factors as well as gaps and challenges in this regard? Recommendations a. If USAID considers supporting a future similar Activity, please provide your major recommendations in terms of: b. What should be the major priorities of the support activities (Management, Workforce supply, Quality of education) c. What should the government do at all levels to support management, increase the number and quality of HRH production? Any additional recommendation/s? 2.3 For Alumni of institutions supported by SHRH Activity To what extent have SHRH Activity result/sub-result areas and interventions continued to be relevant to HRH gaps and challenges in Ethiopia? 1. Can you please describe all the activities that SHRH undertook at this institution? 2. What other activities do you know of that they carryout in this area (outside of the TI) 79 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity 3. Which of the following SHRH initiatives do you consider relevant to meeting your training needs? a. Improving the availability of midwives, HEWs, Anesthetists and other essential non-clinical health workers and other clinical health workers? b. Improving the quality of education through PSE? c. Relevance of research and program learning activities to uncover pertinent HRH issues? Please describe any gaps in this regard and what could be done to improve relevance in future Activity? To what extent has the SHRH Activity progressed towards meeting its performance objectives? Effectiveness: 4. What is your perception on whether SHRH Activity met its objectives on the following? (provide evidence): a. Tuition support for pre-service education b. Improving regulatory frameworks and practices c. Increasing availability and reducing shortage of HEWs, Midwives, Anesthetists as well as other essential and clinical cadres? d. Improving quality of education through curriculum revision, accreditation as well as provision of PSE; effectiveness of blended learning e. Supporting practical skills in PSE f. Supporting in-service training g. Retention and motivation during in-service If they are effective, please describe for us the major drivers of success and innovation that need to be scaled up? If no, describe the major gaps and challenges? Please also describe for us the better approaches and innovation in improving the supply and quality of health workforce? What new approaches need to be considered? Efficiency 5. How cost effective do you think the investments of the SHRH Activity are in terms of a. Production of health workforce b. Improving quality of pre-service education c. Production and use of research evidence and program learning What innovation/s introduced by the SHRH Activity do you think is cost-effective and needs to be scaled up? Which one/s is not cost-effective and needs to be re-examined? Sustainability: 14. From the above activities that have been actively supported, which ones do you think could the government at all levels (FMOE, REBs, TIs) take over and implement without SHRH Activity support? Why and how? Which ones cannot be sustained and need further Activity support? Please provide evidence and justification. To what extent were gender issues addressed in pre-service and in-service trainings? 15. How far has the SHRH Activity responded to and supported gender issues in human resource development in terms of: a. Making the pre-service training workplace hospitable for women? 80 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity b. Improving gender balance in increasing the number of midwives, HEWs, anesthetists and other essential health workers in PSE c. Supporting female students technically and financially d. Building the capacity of women to undertake research? If positive, which? Elaborate. If no, why not? Please describe the successes, and success factors as well as gaps and challenges in this regard? Recommendations 16. If USAID considers supporting a future similar Activity, please provide your major recommendations in terms of: a. What should be the major priorities of the support activities (Management, Workforce supply, Quality of education) b. What should the government at all levels do to support management, increase the number and quality of HRH production? Any additional recommendation/s? 81 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity ANNEX 8: OBSERVATIONS GUIDE Universities/health science colleges and clinical training/practicum sites Observation points Comments Library in the TI Staff available at time of visit: # Librarians: Male______ Female ___________ # Clerks: Male _______Female ___________ # Cleaners: Male ______Female___________ # Other staff (specify): Male ____________Female _____________ Students available at time of visit: # Male _____________Female ____________ Books relevant for training: (Consult Institution to verify key relevant books for that institution) Computers: # computers seen: _______________________ # computers in working condition: ___________ # students expected to use the computers: _____________________________ Furniture (Chairs, desks): Space for books: Space for quiet reading: Environment (quiet/noisy; dark/bright): Other (Specify): Teaching classrooms Staff seen at time of visit: # Male _____________Female ____________ Students seen at the time of visit: # Male _____________Female ____________ Furniture (Chairs, desks): Space in the classroom: Environment (quiet/noisy/dark/bright): Other (Specify): Skill laboratory Staff available at time of visit # Male _____________Female ____________ Students available at time of visit: # Male _____________Female ____________ 82 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity Observation points Comments Skills manuals: Equipment for skills training: (list equipment seen and number of equipment) Space for skills training: Environment (quiet/noisy/dark/bright): Any safety issues observed: Other (Specify): Science Laboratory Staff available at time of visit: # Male _____________Female ____________ Students available at time of visit: # Male _____________Female ____________ Equipment for training seen at time of visit: (list equipment seen and number of equipment) Space for the laboratory: Environment (quiet/noisy; dark/bright): Environment (quiet/noisy; dark/bright): Any safety issues observed: Other (Specify): General observation: To be used in institutions that are supported through the HRH activities -n such as infrastructure development, skills lab support, practicum/clinical training sites) Note to observer: Information collected should be in the presence of at least one key staff working in the TI and familiar with the training process. 83 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity ANNEX 9: SAMPLING FRAME BY THE THREE REGIONAL CONTEXTS Region Performance Levels RHBs Universities RHSC Total available institutions Number of institutions Total available institutions Number of ranked institutions Tigray High 3 3 2 2 Amhara High 7 7 6 5 Oromia High 8 8 5 4 SNNP Low 7 4 3 3 Addis Ababa High 3 0 1 0 Harari High 1 0 0 0 Dire Dawa High 1 0 1 0 Somali Low 1 0 2 1 Afar High 1 1 1 1 JUSTIFICATION FOR SAMPLE SELECTION Context Region Justification Agrarian context SNNP The only region that represents low performance of RHB in HRM among the agrarian regions. Amhara The evaluation team visited Debre Tabor, the university being used as a benchmark for training. This was also logistically easier to manage. Oromia The evaluation team visited the RHB, the best performer in HRM compared to other regions, to learn from Jimma University’s experience. Urban context Addis Ababa Has adequate number of TIs to be visited and is the only region where the private-sector training college has been supported by the Activity. Pastoral context Afar Has both TIs in the same area compared to the Somali region; fewer security risks. 84 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity ANNEX 10: QUANTITATIVE DATA COLLECTION FORMATS Data to be collected from Activity documents, FMOH documents and RHB documents as applicable Table 12: HRM Positions Created and Filled (by Year) RHBs/ZHDs/WoHOs 2013 2014 2015 2016 2017 Total available HR position10 regions (excluding Oromia) Oromia Available HR positions at RHBs (considered as best practice) Total available HR postions11 regions Number of Filled Positions 10 regions No of Oromia Filled positions in Oromia Filled Positions 11 % filled Table 13: Performance of HRM, 2012 - 2017 Identified Indicator Baseline Data Target (2012 - 2017) Plans by Year Performance by Year Total Performance % 2012 2013 2014 2015 2016 2017 2012 2013 2014 2015 2016 2017 % RHBs, ZHDs and WoHos that have staff requirement plan in place % human resource management post filled with staff who have qualification in health or management/social science with minimum of first degree Workforce Turnover rate 85 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity Identified Indicator Baseline Data Target (2012 - 2017) Plans by Year Performance by Year Total Performance % 2012 2013 2014 2015 2016 2017 2012 2013 2014 2015 2016 2017 from government health system # organizational capacity assessment conducted # Human Resource management capacity assessment conducted 15 100% # functional federal and regional HRH forums established # health workers who successfully completed an in￾service training program in HRM Table 14: Number of Midwives, Anesthetists, and HEWs Type Baseline 2012 2013 2014 2015 2016 2017 Total M F M F M F M F M F M F M F M F Midwives Anesthetists HEWs level III HEWs IV 86 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity Table 15: Performance on Quality Improvement (2012 – 2017) Identified Indicator Basel ine Data Tar get (201 2- 201 7) Plans by Year Performance by Year To tal Performa 20 nce % 12 20 13 20 14 20 15 20 16 20 17 20 12 20 13 20 14 20 15 20 16 20 17 # laws, policies, etc. Ratio academic staff/students # Institutions -internal QIP # Institutions - receive tuition Person-hours of training # regions-link CPD to career dev. # regions-in-service/CPD plan # workers completed SBM-R topics # In-service centers established Annual cost of in-service training supported per health worker Average cost per established skills lab per TI Average cost per curriculum revision % of Health Science TIs with the capacity to continue current level of pre-service education without further SHRH support What is the number of regions with the capacity to implement current levels of in￾service training without further support from SHRH Activity? # and % of female students financially and technically supported by the SHRH Activity 87 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity Table 16: Data on Program Learning Activities Undertaken Program Learning Titles Year Dissemination Expenditure In Activity Proposal Actual Undertaken Report Released Published Planned Actual Planned Actual Conduct a work climate assessment in sample regions, woredas and health facilities (IR1) Examine the joint appointments of health professionals in private & public health facilities in terms of job satisfaction, motivation and retention (IR1) Examine the impact of leadership and management training on human resource management (IR1) Examine/identify gender issues impacting women satisfaction in the work place(IR1) Conduct an assessment to examine the main reasons for students' attrition in midwifery and anesthesia (IR2) Examine whether HEWs' attrition has been effectively managed through use of HRIS data by RHB (IR2) Explore the impact of the problem based accelerated learning approach using a cohort of medical students (IR3) Assess whether students have adequate exposure to core competencies during clinical practice in a selected cadre (IR3) Examine whether performance recognition influences retention of technical and academic staff (IR3) Activity 4.1.1.8. Support Professional Associations to conduct manuscript writing workshops to draft manuscripts using data from a Task Analysis Study for 5 cadres (Medical doctors, Health Officers, Nurses, Pharmacists, and Medical Laboratory Scientists) Activity 4.1.1.9. Conduct a manuscript writing workshop to develop a draft manuscript using data from the Health Professional Regulation and Practice Study 88 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity Table 17: Data on Research Activities Undertaken Research Titles Year Dissemination Expenditure In Activity Proposal Actual Undertaken Report Released Published Planned Actual Planned Actual Pilot a retention strategy with one cadre (IR1) Conduct a comparative study to examine educational quality in the accelerated one-year midwifery program in comparison to the three-year diploma program (IR2) Conduct a study to examine whether establishing a community-based learning center effectively brings learning to the frontline to develop competent health workers (IR2) Conduct a study to examine the feasibility, implications and competency outcomes of implementing a blended learning approach for HEWs' integrated refresher training (IR2) Conduct a pilot study to assess the feasibility of shortening 3-week BEmONC course using a blended learning approach (IR3) 89 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity Table 18: Program Learning Activities by Number and Type of Participants Program Learning Titles # of participants by Institution JHPI EGO External (outside of Ethiopia) MOH/ Regions EMWA EAA HEI (Ethiopia) Other (specify) Total (Female) Conduct a work climate assessment in sample regions, woredas and health facilities (IR1) Examine the joint appointments of health professionals in private & public health facilities in terms of job satisfaction, motivation, and retention (IR1) Examine the impact of leadership and management training on human resource management (IR1) Examine/identify gender issues impacting women satisfaction in the workplace (IR1) Conduct an assessment to examine the main reasons for students' attrition in midwifery and anesthesia (IR2) Examine whether HEWs' attrition has been effectively managed through use of HRIS data by RHB (IR2) Explore the impact of the problem based accelerated learning approach using a cohort of medical students (IR3) Assess whether students have adequate exposure to core competencies during clinical practice in a selected cadre (IR3) Examine whether performance recognition influences retention of technical and academic staff (IR3) Activity 4.1.1.8. Support Professional Associations to conduct manuscript writing workshops to draft manuscripts using data from a Task Analysis Study for 5 cadres (Medical doctors, Health Officers, Nurses, Pharmacists, and Medical Laboratory Scientists). Activity 4.1.1.9. Conduct a manuscript writing workshop to develop a draft manuscript using data from the Health Professional Regulation and Practice Study 90 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity Table 19: Research Activities by Number and Type of Participants Research Titles # of participants by Institution JHPIEGO External (outside of Ethiopia) MOH EMWA EAA HEI (Ethiopia) Other (specify) Total (Female) Pilot a retention strategy with one cadre (IR1) Conduct a comparative study to examine educational quality in the accelerated one￾year midwifery program in comparison to the three-year diploma program (IR2) Conduct a study to examine whether establishing a community-based learning center effectively brings learning to the frontline to develop competent health workers (IR2) Conduct a study to examine the feasibility, implications and competency outcomes of implementing a blended learning approach for HEWs' integrated refresher training (IR2) Conduct a pilot study to assess the feasibility of shortening 3-week BEmONC course using a blended learning approach (IR3) 91 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity Table 20: Publications by Number and Type of Participants Publications Titles # of participants by Institution JHPIEGO External (outside of Ethiopia) FMOH EMWA EAA HEI (Ethiopia) Other (specify) Total (Female) Ayalew F Et al. Factors Affecting Turnover Intention among Nurses in Ethiopia. World Health and Population 20__; 16(2): 62-74. Desta FA et al. Identifying gaps in the practices of rural health extension workers in Ethiopia: a task analysis study BMC Health Services Research (2017) 17:839. DOI 10.1186/s12913-017-2804-0. Jhpiego. “NOW I CAN STAY FOCUSED ON MY DREAMS” Empowering Female Health Science Students in Ethiopia: A Case Study. 2017, Baltimore, Maryland. Kibwana S Et al. Education, Practice, and Competency Gaps of Anesthetists in Ethiopia: Task Analysis. Journal of PeriAnesthesia Nursing 2017; Kibwana S Et al. Generating Evidence for Strengthening the Health Workforce in Ethiopia. BULLETIN of the NETHERLANDS SOCIETY for TROPICAL MEDICINE and INTERNATIONAL HEALTH 2017; 55(3):3-9. Kibwana S Et al. Trainers' perception of the learning environment and student competency: A qualitative investigation of midwifery and anesthesia training programs in Ethiopia. Nurse Education Today 2017; 55: 5-10. Kibwana S, Woldemariam D, Misganaw A, Teshome M, Akalu L, Kols A, et al. Preparing the health workforce in Ethiopia: A Cross-sectional study of competence of anesthesia graduating students. Educ Health 2016; 29:3-9. Kols A Et al. Factors Predicting Ethiopian Anesthetists’ Intention to Leave Their Job. World J Surg 2017; (doi:10.1007/s00268-017-4318-7. 92 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity Yigzaw T et al. Using task analysis to generate evidence for strengthening midwifery education, practice, and regulation in Ethiopia. International Journal of Women’s Health 2016; 8: 181-190. Yigzaw T et al. How well does pre-service education prepare midwives for practice: competence assessment of midwifery students at the point of graduation in Ethiopia BMC Medical Education (2015) 15:130. DOI 10.1186/s12909-015- 0410-6. Yigzaw T Et al. Quality of Midwife-provided Intrapartum Care in Amhara Regional State, Ethiopia. BMC Pregnancy and Childbirth (2017) 17:261. 93 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity ANNEX 11: LIST OF KII PARTICIPANTS S/no Name Position and organization 1 Tilahun Bezabih Program Manager , SNNPR Jhpiego 2 Muradu Melis Education and training Officer, SNNPR Jhpiego 3 Helen Tiruneh Education and training Officer, SNNPR Jhpiego 4 Getaneh Girma Education and training Officer, SNNPR Jhpiego 5 Wondu Gideta M & E officer, SNNPR Jhpiego 6 Assfaw Demise HRM Officer, SNNPR Jhpiego 7 Tigist Fikadu Program Assistant, SNNPR Jhpiego 8 Getachew worku Office Manager, EAA 9 Desalegne Mekuria Technical Advisor, EAA 10 Jemal Mohamed HRM Directorate Director, Afar Regional Public Service and HR Bureau 11 Ato Tilahun Eshete- Deputy Bureau head , Amhara BoFED 12 Genet Kifle Felege Hiwot Hospital HRM coordinator 13 Abraham Yemanu Regional HRM Officer, Afar MSH team 14 Daniel Haile Selassie Education and training advisor, Jhpiego Tigray team 15 Abebe Tadele Education and training Officer, Jhpiego Tigray team 16 Osman Ahmed Quality Assurance Coordinator, Semera University 17 Kusse Urmale Nursing Department Head, Semera University 18 Ato Afkea Ali Teachers and Education development Director, Afar REB 19 Eyisha Abdella Gender Officer, Afar RHB 20 Ali Hussen TVET Agency Director, Afar regional state 21 Osman Ahmed Coordinator CMHS Semera University 22 Seraj Ahmed Instructor, Semera RHSC 23 Dawud Hussien Instructor, Semera RHSC 24 Gebru Getachew Instructor, Semera RHSC 25 Hussen Kaloyta Ahmed Instructor, Semera RHSC 26 Dutse Tamiru HRM support process owner, SNNPR RHB 27 Wossen Gizachew HRM officer, SNNPR RHB 28 Kirubel Bahailu Delegate Anesthesia Department Head, Araba Minch University 29 Tayiredin Nuriye Midwifery Department Head, Welkite University 30 Girum Yonas Anesthesia Department Head, Araba Minch HSC 31 Yitagesu Habte EDC coordinator, Hossana HSC 32 Andualem Girma HEW training department Head, Arba Minch HSC 33 Fikre Bojola Nursing department Head, Arba Minch HSC 34 Abera Feyissa Nursing Department Head, Hossana HSC 35 Zeritu Dawana Midwifery Department Head, Arba Minch HSC 36 Gudeta Berriso Midwifery Instructor, Welkite University 37 Mebiratu Demisse Midwifery Instructor, Welkite University 38 Seboka Abebe Midwifery Instructor, Welkite University 39 Shegaw Beze Midwifery Instructor, Welkite University 40 Mintewab Gelagay Education & Training Adviser, Jhpiego AA regional team 41 Ferew Tafese HRH Advisor, Jhpiego AA regional team 41 Abrehet FMHACA 42 Leulayehu Executive director of EAA 43 Dr Gabriel Instructor, Debere Tabor University 44 Dr Selam Instructor, Debere Tabor University 45 Dr Lealem Instructor, Debere Tabor University 46 Dr Getachew Tolera FMOH-HRD-HRM 47 Dr Tsedeke Asaminew EDC coordinator, Jimma University 48 Tesfamichael Adaro Faculty development coordinator, Jimma University 49 Waju Beyene Academic quality assurance director, Jimma University 50 Ben Tegegne Delegate HRM Head, AA RHB 51 Chale Hailu HR administration coordinator, AA RHB 52 Berhanu Fekadu HRIS office, AA RHB 53 Yewubdar Tilahun HRD coordinator, AA RHB 54 Muluken Asefa Dean, Bahir dar HSC 55 Tigabe Asmare EDC coordinator, Bahir dar HSC 56 Azezu Asires Gender office coordinator, ?? 57 Dr. Worku Belay Chief Executive Director (Delegate), ?? 58 Salamawit Leake Midwifery Department Representative, ?? 94 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity S/no Name Position and organization 59 Gabriel Hiwot Head of Anesthetist department, ?? 60 Mengistu Desalega Quality Assurance Coordinator, ?? 61 Ashebir Kidane Regional coordinator, Jhpiego Amhara regional team 62 Fantu Abebe Educational adviser, Jhpiego Amhara regional team 63 Yehilsew Asaya Educational & Training Adviser, Jhpiego Amhara regional team 64 Le’alem Menbere Educational & Training Adviser, Jhpiego Amhara regional team 65 Molla Belay HRH Officer, Jhpiego Amhara regional team 66 Desalegn Ademe M&E Officer, Jhpiego Amhara regional team 67 Addise Worku Programme Assistant, Jhpiego Amhara regional team 68 Bizualem Gashaw HRH management Officer, Jhpiego Amhara regional team 69 Tesfaye Director of Amhara COC Agency 70 Bazeze Gelaw Amhara COC Agency 71 Solomon Assefa HRH process owner, Amahara RHB 72 Mengistu Alelign Good governance and reform coordinator, Amahara RHB 73 Fantaye Awoke Chairman of health professional associations, Amahara RHB 74 Ato Tilahun Eshete- Deputy Bureau head, Amhara BOFED 75 Amsalu Molla Dean of Health science college, Bahir Dar HSC 76 Awoke Giletew EDC coordinator, Bahir Dar HSC 77 Eyerusalem Solomom Gender coordinator, Bahir Dar HSC 78 Alemayehu Nigussie Academic vice president, Bahir Dar HSC 79 Dessie Tegegne IST coordinator, Bahir Dar HSC 80 Seyoum Mekonne Delegate Dean and Nursing Department head, Semera HSC 81 Mohamed Abdulrahman Head of HEW department, Semera HSC 82 Mohammed Akber Head department of midwifery, Semera HSC 83 Najima Biza Dean College of Medical and Health Sciences, Semera University 84 Ibrahim Mohamed Head department of midwifery, Semera University 85 Ato Belete Akenie Job structure, Valuation and Remuneration Study Directorate, Amahara PSHRB 86 Tachawt Seleley Project Manager, EMA HRH Activity 87 Almaz Mamaru, Gender Officer, Jimma University 88 Ms. Eleni Admasu, Former IST Coordinator, Bahir Dar University 89 Yibeka Assefa HRM, PSHRB, SNNPR 90 Dr Getachew Tolera Directorate Director, FMOH 91 Zewde Asfaw Charities and Societies Projects Monitoring and Evaluation Officer, SNNPR BOFED 92 Tigist Kitambo Gender Coordinator CMHS Arba Minch University 93 Tigist Shiferaw Registrar and public health specialist, ACIPH 94 Dr. Mengistus Hailesele Country Representative/Director, CIRHT 95 Solomon Worku Director for Medical Education Training, CIRHT 96 Melaku HR Director, PSHRDB, AA city Administration 97 Dr Henok Zewdu CMD, Debre Tabor University 98 Belay Ambachew CEO, Debre Tabor University 99 Yibabe Tesfaye Dean, Debere Tabor HSC 100 Tadesse Wuletaw Academic Vice Dean, Debere Tabor HSC 101 Bantayehu Dejen IST Coordinator, Debere Tabor HSC 102 Saleamlak Adbaru Gender coordinator, Debere Tabor HSC 103 Mequanint Beze EDC focal person, Debere Tabor HSC 104 Demamu Mekonnen Admin & Development Vice Dean, Debere Tabor HSC 105 Ato Kuma Getahun Dean, Minilik HSC 106 Menelik Legesse EDC & Research Coordinator, Minilik HSC 107 Fikrete W/Selaise Gender focal Person, Minilik HSC 108 Dr Dereje Diguma Head, Oromia RHB 109 Habtamu Demissie Director, HRD and Administration, Oromia RHB 110 Tesfaye Dadi HRM and HRD directorate, Oromia RHB 111 Dereje HRM and HRD directorate, Oromia RHB 112 Geremew HRM and HRD directorate, Oromia RHB 113 Zerihune HRM and HRD directorate, Oromia RHB 114 Dr Alemayehu Haddis IST coordinator, Jimma University 115 Tekalign Berhane Deputy Bureau Head, Oromia PSHRDB 116 Akrem Abdusemed Regional HRH manager, Jhpiego team 117 Solomon Abebe Acting Program Manager, Jhpiego Oromia team 118 Mulugeta Debele HR Officer, Jhpiego Oromia team 119 Ayana Chimdessa Education & Training Officer, Jhpiego Oromia team 120 Gelmessa Bekana M&E officer, Jhpiego Oromia team 121 Worku Habte Program Assistant, Jhpiego Oromia team 122 Solomon Wolde Education & Training Officer, Jhpiego Oromia team 95 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity S/no Name Position and organization 123 Ezedin Aman Jhpiego Oromia team 124 Dr Shelemo Shawula MSH HRH Activity 125 Dr Tegibar Yigizaw Deputy Chief of Party, Jhpiego HRH Activity 126 Dr Damtew Dagoye Chief of Party, Jhpiego HRH Activity 127 Mihireteab Teshome Jhpiego HRH Activity 128 Alem Wakenie HR Manager, Zewuditu Memorial Hospital 129 Negat Director of directorate, St Paul MMC 130 Rahel Surgical department, St Paul MMC 131 Teshome EMT department, St Paul MMC 132 Tangut Research department, St Paul MMC 133 Teshale Operating theater department, St Paul MMC 134 Ermias Pediatric department, St Paul MMC 135 Dr Solomon Education Strategic center 136 Dr Daniel Jhpiego HRH team 137 Firew Jhpiego HRH team 138 Ephraim Jhpiego HRH team 96 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity ANNEX 12: POST EVALUATION ACTION PLAN This Post-evaluation Action Plan is developed to create an opportunity to openly discuss evaluation findings, conclusions, and recommendations within the Mission, in collaboration with MEL technical working group and Evaluation Contractor, and plan for use/ adoption of findings. The below two tables are adapted from the USAID Post-Evaluation Plan templates for this exercise, and the following are the principles (The P6) the Mission needs to follow to execute the Post-evaluation Action Plan. PRINCIPLE ONE: After an evaluation is completed, the COR or the Program Office at large and the Evaluation Contractor (as appropriate) shall seek the overall impression of the relevant Office/s within the Mission about the quality of the evaluation. PRINCIPLE TWO: After the evaluation report is approved, the evaluation COR or the Program Office at large and the Evaluation Contractor (as appropriate) shall sit together with the relevant office/s within the Mission to identify relevant/doable and non-doable recommendations (this is a completion to Table 1.) PRINCIPLE THREE: The evaluation COR or the Program Office at large and the Evaluation Contractor (as appropriate) shall discuss with the relevant Office/s within the Mission to plan how evaluation findings/recommendations shall be used, the responsible person to follow-up on that and also the timeline for adoption (this is a partial completion to Table 2.) PRINCIPLE FOUR: Actions completed under PRINCIPLES ONE, TWO and THREE shall be part of the Annexes of the Final Evaluation Report by the Evaluation Contractor. PRINCIPLE FIVE: Tables 1 and 2 shall be shared, for instances as google sheet, with the wider Mission community to promote transparency and learning. PRINCIPLE SIX: The evaluation COR or the Program Office at large and the Evaluation Contractor (as appropriate) shall keep track of the plan and update the status on a quarterly basis by following up on the actions with the technical team (this is a completion to Table 2.) 97 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity Table 21: Post evaluation Management Evaluation Title: SHRH Evaluation Completion Date: Overall Impression about the Evaluation: Evalua tion Recom menda tion USAID Response : Accepted / Partially Accepted / Rejected If not accepted or partially accepted, provide more information why it was not ACCEPTED or PARTIALLY Recommendations on Effectiveness ACCEPTED 1. The Government of Ethiopia (federal and regional) should review remuneration packages for HRM professionals to create interest from qualified candidates to apply for HRM positions and to improve retention of existing HRM staff. The experience of the Oromia region could be used as a best practice example for other regions to learn from and implement. 2. USAID should engage with the FMOH & RHB to prioritize, reconstitute, and reinvigorate the HRH forum, to strengthen its functioning as a platform for addressing cross-cutting HRH issues, especially strengthening it at the federal level to go beyond the technical to strategic policy discussions. 3. The Government of Ethiopia and USAID should set aside funds to promote the development of a clear IST implementation strategy, including infrastructure, technical capacity trainer motivation, generation and use of resources, etc.) 4. The Government of Ethiopia should review the effectiveness of IST expansion as a means for shifting hotel￾based training to IST centers. 5. USAID should not include increasing the availability of health workers as a result area for any future HRH interventions as this has been prioritized by the Government. 6. USAID should support the FMOE’s strategy to develop health science qualifications standards for colleges across the country, by providing technical and financial support. 7. USAID should focus investments for any future support on third- and fourth-generation universities that have proven themselves receptive to new ideas and can effect greater change. 8. USAID should consider designing the activity to support health science colleges to shift towards a competency￾based curriculum. 9. The Government of Ethiopia should develop and implement strategies to balance enrollment with available resources and infrastructure in TIs. 98 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity Evalua tion Recom menda tion USAID Response : Accepted / Partially Accepted / Rejected If not accepted or partially accepted, provide more information why it was not ACCEPTED or PARTIALLY 10. The Government of Ethiopia should develop and implement a continuing professional development strategy. ACCEPTED USAID should provide support for this strategy when finalized. 11. USAID’s future Activity designs should explore ways of strengthening the consultation processes with donors, government and other partners in developing priority topics/activities for program learning and research. 12. USAID should ensure ownership and use of PLAs and RA findings through greater participation in the selection of topics, the design of studies, analysis of data and dissemination Recommendations on Sustainability 13. USAID should focus on building leadership commitment among forum members in future Activity support. The case of Oromia region can be used as an example. 14. USAID should have the next SHRH Activity carry out organizational capacity assessments designed to engage stakeholders in not only assessing their organizational capacity but involve them in designing the action plan for the way forward. 15. USAID should focus on institutional capacity as equally or more important than the individual capacity to facilitate sustainability. USAID should continue to support the capacity building of TIs through technical support and ensuring the availability of adequate teaching materials. 16. USAID and the Activity should collaborate, learn and adapt by working with international TIs to implement best practices. 17. USAID should ensure that any future similar Activity should prioritize interventions that enhance sustainability in quality of IST by ensuring infrastructure development as a pre-requisite for providing future support. 18. USAID should direct the partner to work with and support FMOH, RHBs, and local research institutions need to build their research capacity rather than just having the implementing partner do the research. Recommendations on Gender 19. USAID should support future Activities to provide training and support the implementation of the sexual￾harassment reporting system. 20. The GoE or USAID should conduct an evaluation of the tutorial program to determine its effectiveness in increasing retention, grades, skills, and graduation rates of female students. 21. The Government of Ethiopia should consider providing financial assistance to needy female students and expand that assistance to IST. Recommendations on Relevance 22. The Government of Ethiopia should approach motivation and retention efforts differently. In line with government priorities, staff satisfaction and motivation to improve on-the-job performance should be emphasized over retention. 23. The Government of Ethiopia (supported by USAID) should strengthen the human resource information management systems in alignment with the implementation of District Health Information System 2 (DHIS2). 99 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Final Evaluation Report, Strengthening Human Resources for Health (SHRH) Activity Evalua tion Recom menda tion USAID Response : Accepted / Partially Accepted / Rejected If not accepted or partially accepted, provide more information why it was not ACCEPTED or PARTIALLY 24. USAID should support the strategic provision of technical assistance and material support to TIs for training ACCEPTED essential HWs should be continued as relevant in future HRH activities. 25. USAID should consider continuing to fund activities that build the institutional capacity at all levels of the government to provide quality pre-service and in-service training. 26. The Activity and USAID should place an increased emphasis on developing capacity through collaborative studies and research with all stakeholders. This research capacity development approach is especially important for new TIs with young and inexperienced staff. Recommendations on Facilitating/Hindering Factors 27. Ensure that a comprehensive stakeholders’ analysis is conducted prior to the introduction of the initiative that aims to increase the number of HRM positions. Table 22: Post-evaluation Action Plan Evaluation Title: SHRH Evaluation Completion Date: Recommendations fully or partially accepted Adoption plan Follow-up USAID Focal Office USAID Focal Person and Title How will the Office use/adopt the recommendation? When is the expected time for use/adoption? Status of Actions (as of date) Any remarks 1. 2. 3. 4. 5. 6. 7. 8. U.S. Agency for International Development Entoto Street PO Box 1014 Addis Ababa, Ethiopia