EVALUATION Final Report: Volume I (Main Report) Health Systems 20/20 Project in Egypt: End of Project Performance Evaluation April 2013 This publication was produced at the request of the United States Agency for International Development (USAID). It was prepared independently by Dayl Donaldson, Waleed El Feky, Mahinaz El Helw, Kate Fehlenberg, Mildred Howard, and Doaa Oraby on behalf of Social Impact, Inc. Cover image courtesy of USAID/Egypt. THE HEALTH SYSTEMS 20/20 PROJECT IN EGYPT: END-OF-PROJECT PERFORMANCE EVALUATION VOLUME I (MAIN REPORT) OPPORTUNITIES AND CHALLENGES OF WORKING TO STRENGTHEN HEALTH SYSTEMS IN A COUNTRY AND HEALTH SECTOR IN TRANSITION April 15, 2013 Task Order: RAN-1-00-09-00019/RFTOP# SOL-263-12-000007 This report will also be made available from the Development Experience Clearinghouse (DEC). DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. ACKNOWLEDGEMENTS The team wishes to acknowledge and thank the many individuals who helped us understand the government of Egypt’s evolving policies and strategies for reform of the health sector and issues related to the design and implementation of the Health Systems 20/20 Egypt (HS 20/20 Egypt) project. The team is indebted to Dr. Amr el-Shalakani, head of the Health Economics Unit in the Technical Office for the Minister in the Ministry of Health and Population, and Dr. Manal Mongy, head of the Quality Department in the Health Insurance Organization, for coordinating meetings between the team and representatives of their respective organizations. In addition, these individuals arranged site visits in Cairo, Luxor and Quena Governorates that included convening trainees from three project-sponsored courses to participate in evaluation-focus-group discussions. The team would also like to thank Dr. Nadwa Rafeh and other members of the Abt Associates HS20/20 Egypt project staff and consultants for agreeing to be interviewed, either in Cairo or Washington, D.C. Catherine Connor from Abt Associates headquarters also was instrumental in providing input on the design of HS20/20 Egypt and shared numerous important project documents with the team. Very special thanks goes to Laura Campbell and the staff of USAID/Egypt for facilitating all aspects of the team’s work in Egypt, as well as providing input on USAID’s views of the implementation of HS 20/20 Egypt. Many other individuals from the government of Egypt, the HS 20/20 Egypt project, USAID/Egypt and representatives of other donors in Cairo agreed to be interviewed by the team and are listed in Annex F of this report. Finally, the team would like to acknowledge the support of Luca Etter, Dr. Richard Blue and Maryam Hassan of Social Impact Inc. (SI) for their technical and administrative backup, and also Diego Velaquez of Management Systems International (MSI) for logistical support to the team. While the input of everyone is gratefully acknowledged, any errors of fact, omission or interpretation in this report are solely the responsibility of the evaluation team—team leader Mildred Howard and team members Dr. Dayl Donaldson, Dr. Waleed El Feky, Mahinaz El Helw, Kate Fehlenberg and Dr. Doaa Oraby. _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project iv TABLE OF CONTENTS ACKNOWLEDGEMENTS ......................................................................................................................... iii Table of Contents ......................................................................................................................................... iv ACRONYMS ............................................................................................................................................... vi EXECUTIVE SUMMARY ....................................................................................................................... viii Project Background ................................................................................................................ viii EVALUATION PURPOSE & AUDIENCE .......................................................................... viii Evaluation Methodology & Limitations ................................................................................... ix Findings and Conclusions ........................................................................................................ ix Moving Forward and Recommendations for Action ............................................................... xii I. PROJECT DESCRIPTION................................................................................................................... 1 II. EVALUATION PURPOSE AND QUESTIONS ................................................................................. 1 Key questions ............................................................................................................................ 1 Health sector development Challenges ..................................................................................... 3 Overview of HS 20/20 Egypt .................................................................................................... 3 HS 20/20 EGYPT’S UNDERLYING theory of change AND ASSUMPTIONS ..................... 4 An Illustrative HS 20/20 Egypt Results Framework ................................................................. 5 HS 20/20 EGYPT DESIGN CHANGES .................................................................................. 6 III. EVALUATION METHODS AND LIMITATIONS ............................................................................ 6 Methods ..................................................................................................................................... 6 Limitations ................................................................................................................................ 7 IV. FINDINGS AND CONCLUSIONS ..................................................................................................... 8 National Health Accounts (NHA) ............................................................................................. 8 Workforce Planning (WFP) ..................................................................................................... 12 Health Insurance Organization (HIO) ..................................................................................... 15 Training and Capacity Building (TCB) ................................................................................... 18 Other Strategic Interventions .................................................................................................. 21 Project Monitoring and Evaluation Systems ........................................................................... 24 V. MOVING FORWARD AND RECOMMENDATIONS FOR ACTION ........................................... 27 Annex A: Evaluation Statement of Work ................................................................................................... 30 Annex B: Modifications to the Evaluation Statement of Work .................................................................. 38 Annex C: Estimates of Donor Financing for Health Sector Reforms, 2008–2012 ..................................... 39 _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project v Annex D: Evaluation Methodology and Workplan .................................................................................... 40 Annex E: Persons Contacted ....................................................................................................................... 49 Annex F: Data Collection Instruments ........................................................................................................ 51 Annex G: Documents Reviewed and References ....................................................................................... 59 Annex H: How Project Outcome Performance Monitoring Indicators Might Have Looked ..................... 64 Annex I: Report on Three Focus Group Discussions with HS 20/20 Egypt Training Participants ............ 67 TABLES AND FIGURES Table 1: Illustrative HS 20/20 Egypt Results Framework ............................................................................ 5 Table 2 NHA Activities and Outcomes ........................................................................................................ 9 Table 3 Summary of HS 20/20 Egypt Training Activities .......................................................................... 19 Figure 1: Evaluation Key Questions ............................................................................................................. 2 Figure 2: HS 20/20 Illustrative Egypt Logic Model and Technical Approach ............................................. 4 _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project vi ACRONYMS ADS Automated Directives System CAOA Central Agency for Organization Administration CAPMAS Central Agency for Public Mobilization and Statistics CCT Conditional Cash Transfer CMO Case Management Office DCS Department of Compulsory Service DEC Development Experience Clearinghouse DOP Department of Planning EC European Commission ETS Expenditure Tracking System FGDs Focus Group Discussions FHC Family Health Center FP Family Planning GOE Government of Egypt GSAP Global Strategic Action Plan HEU Health Economics Unit HFG USAID Health Finance and Governance Project HIO Health Insurance Organization HISDP Health Insurance Systems Development Project HRTF Human Resources Task Force HS 20/20 Health Systems 20/20, the global project HS 20/20 Egypt Health Systems 20/20, the project in Egypt IC Infection Control KI Key Informant KPI Key Performance Indicators KQ Key Questions _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project vii M&E Monitoring and Evaluation MCH Maternal and Child Health MOHP Ministry of Health and Population MOIC Ministry of International Cooperation NHA National Health Accounts NICHP National Information Center for Health and Population OSI Other Strategic Interventions PHC Primary Health Care SHI Social Health Insurance SI Social Impact SKII Structured Key Informant Interview SOW Scope of Work TAP Technical Assistance Plans TCB Training and Capacity Building USG United States Government WB World Bank WFP Workforce Planning WHO World Health Organization WISN Workload Indicators of Staffing Needs _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project viii EXECUTIVE SUMMARY EVALUATION PURPOSE & AUDIENCE Social Impact, Inc. (SI) was engaged by USAID/Egypt to conduct an end-of-project performance evaluation of the health-systems-strengthening project, Health Systems 20/20, in Egypt (HS 20/20 Egypt). The evaluation had two overarching purposes: (1) assess to what extent the project’s objectives and goals contributed to the achievement of the intended results and (2) determine whether moving forward with a new health-systems-strengthening project is a good investment of USAID funds. In addition to performance and achievements, the evaluators were asked to assess the project’s working relationships with counterparts and stakeholders; to identify ways in which the monitoring and evaluation functions of future projects can be improved; and to provide guidance regarding the manner in which USAID might maximize lessons learned for future investments in health-systems strengthening. The primary audience for this evaluation report is the Office of Health and Population within USAID/Egypt. USAID intends to integrate report recommendations into consideration of future health￾systems-strengthening activities, and additionally plans to share the results of this evaluation, as appropriate, with the USAID Middle East Bureau, and the USAID Office of Health Systems. Furthermore, as HS 20/20 Egypt’s main collaborating partners, USAID/Egypt plans to share the results of this evaluation with the MOHP, the Health Insurance Organization (HIO) and interested donor partners. PROJECT BACKGROUND HS 20/20 Egypt, which began in February 2008 and ended in April 2012, was one of 51 country-specific projects of the global HS 20/20 project implemented by Abt Associates and its consortium through USAID/Washington’s flagship cooperative agreement for health-systems strengthening. Within the context of Egypt’s overarching health sector challenges, the design of HS 20/20 Egypt built upon USAID’s extensive experience and leadership in the health sector reform process in Egypt, complementing technical assistance from other international partners, such as the European Commission (EC) and the World Bank (WB). Five priority areas for HS 20/20 Egypt technical assistance were identified: (1) developing National Health Accounts (NHA) studies and institutionalizing NHA capacity within the government of Egypt (GOE); (2) developing a long-term strategic workforce plan and workforce planning capacity within the Ministry of Health and Population (MOHP); (3) supporting the Health Insurance Organization (HIO) in building financial administration and medical management capacity in anticipation of it assuming the role of payer under the new health insurance law; (4) providing training and capacity building (TCB); and (5) providing assistance in other strategic interventions (OSI), including serving as verifier for a bilateral, conditional cash-transfer program, conducting a major study on the sustainability of USAID health-sector assistance, and development of a battery of case studies to inform strategic health planning and management training. _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project ix EVALUATION METHODOLOGY & LIMITATIONS The evaluation team applied a variety of predominantly qualitative evaluation methods to address key evaluation questions and to formulate findings, conclusions and recommendations. Evaluation methods included: (1) extensive desk reviews; (2) structured key informant interviews (SKII); (3) site visits; (4) focus group discussions (FGDs); and (5) team information sharing, data synthesis, and triangulation. While the evaluation team was successful in interviewing most anticipated key informants (KIs) and stakeholders, it had to address a number of technical and logistical limitations. The nine-month lapse between the end of project and the start of the evaluation meant that many KIs had moved on and were difficult to locate. Nonetheless, the evaluation team managed to talk to 71 KIs from all relevant stakeholder groups during the four weeks of field work (in January and February 2013) in Egypt, or, in a very few instances, by phone or email with stakeholders no longer based in-country. While the evaluation team could draw from an extraordinary abundance of documentation available, the evaluators found that project file documentation related to the five different, and technically complex, interventions was interwoven into various funding documents, multiple, disjointed annual work plans and revisions, quarterly progress and work planning documents, technical assessments and reports, briefing documents and power point presentations, which at times made the review process cumbersome. Finally, the evaluation team’s field work coincided with the second anniversary of Egypt’s 2011 revolution, which posed a number of logistical challenges, such as cancelled or rescheduled meetings and onerous travel arrangements. FINDINGS AND CONCLUSIONS National Health Accounts (NHA): HS 20/20 Egypt performed well in assisting the MOHP to successfully complete two NHA estimates of acceptable quality by international standards. This effort profited from prolonged USAID experience and leadership in conducting NHAs in Egypt and elsewhere. HS 20/20 Egypt also performed well and was highly productive in conducting, or arranging the conduct of, several complex costing studies and tracking systems related to preparation of NHA estimates. While these technical tasks were carried out, to some extent, in collaboration with the MOHP, they remained mainly the product of external consultants, which may have reduced the extent to which HS 20/20 Egypt could leverage its technical capacity to assist the MOHP with the institutionalization of NHAs. The project failed, for the most part, to achieve the second main goal of the component, the institutionalization of NHAs. While HS 20/20 Egypt made a modest contribution to improving the MOHP’s capacity to manage NHAs by training a limited number of MOHP staff in costing and expenditure tracking, this did not translate into sustained institutional capacity to carry out future NHA exercises. This is particularly apparent in the lack of an organizational mandate for the Health Economics Unit (HEU), the dedicated entity within the MOHP to house NHA estimates and other studies and analytical functions to support sector strategic planning. Workforce Planning (WFP): The original goals regarding WFP in HS20/20 Egypt were to develop a long-term workforce plan for the MOHP and to establish a sustainable workforce-planning program. This translated into a host of activities aiming at (1) building the capacity of MOHP staff to produce WFP data _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project x and reports; and (2) institutionalizing WFP at the MOHP. While overall, HS 20/20 Egypt delivered a solid effort to produce WFP data and reports, it succeeded less on institutionalization. There were four main challenges to the institutionalization of the WFP effort. First, there was lack of clarity in HS 20/20 Egypt dialogue with MOHP policy makers and counterparts about the ultimate purposes and methodology of the planning system and the organizational capacities needed to sustain it. Second, HS 20/20 Egypt did not identify and engage the primary WFP stakeholder(s) within the MOHP and instead focused its engagement on offices that could generate and use workforce data for decentralized management purposes, but that did not ultimately have human resources policy or strategic decision making authorities. Third, human resource management functions inside MOHP were fragmented with different roles played by different divisions with no unified sector or departmental oversight. Finally, HS 20/20 Egypt did not consider the HR component of the sector as a whole, but focused narrowly on district/general MOHP-owned hospitals. PHC/FHC facilities, hospitals at tertiary levels and other government-owned facilities or private-sector facilities were not included. Health Insurance Organization (HIO): The HIO component focused on capacity building of the HIO for it to be able to better address its existing challenges related to quality improvement and financial management, and help prepare HIO staff to fill their anticipated new “payer” functions under the pending Social Health Insurance law. HS 20/20 Egypt’s objectives in assisting the HIO not only responded to the requests for assistance by HIO management but were highly relevant to the needs of the organization. However, in the areas of financial management training, HS 20/20 Egypt made insufficient effort during the design phase to: (1) obtain a detailed understanding of the HIO’s systems and training needs, (2) realistically assess the technical strengths of HS 20/20 Egypt’s staff and consultants and match these to the HIO’s expressed needs and (3) make a realistic assessment of what HS 20/20 Egypt could sustainably accomplish with the HIO within a three-year time horizon. HS 20/20 Egypt project outputs in the area of accounting, costing and financial management did not produce any sustained outcomes in terms of manuals, nor in the building of skills thought useful for the organization. On the other hand, HS 20/20 Egypt’s medical management outputs continue to be available to, and utilized by, the HIO. In fact, the trained staff has expanded medical management activities to new facilities and plans to conduct medical management training courses for HIO staff. Despite the fact that there are challenges to sustainability of the medical-management inputs, the HIO quality department expressed its commitment to sustaining the technical capacity and practice built during HS 20/20 Egypt. Thus, HS 20/20 Egypt’s assistance to the HIO can be considered partially sustainable, overall. Project Monitoring and Evaluation Systems (M&E): The absence of a full, strategic-planning process during the design phase—specifically, the lack of a collaborative development of a country-specific results framework and an outcome-based project monitoring and evaluation (M&E) plan—compromised HS 20/20 Egypt’s ability to successfully focus and direct its assistance. Neither the central HS 20/20 project, nor USAID project management, required HS 20/20 Egypt to develop a results framework and project M&E plan in collaboration and agreement with counterparts. An emphasis on strategic planning in the design phase would have helped the project and counterparts reach consensus about project objectives and expected results, as well as strategies for implementation. The project did not share work plans or progress reports with technical counterparts in the MOHP, nor did the project have access to senior MOHP decision makers for periodic review and guidance on the overall _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project xi direction of the project (particularly when changes were being made to the project scope). Additionally, the project’s design did not include the establishment of an M&E system within MOHP and HIO as a means of on-going measurement and monitoring of activities related to project inputs, outputs and outcomes. Initially, the project also lacked the in-house M&E expertise required to produce a robust M&E system and this deficiency was not recognized or acted upon by either HS 20/20 headquarters or USAID/Egypt. As the project clearly made an effort to be responsive and cooperative on all aspects of project management, it is likely that its performance could have been substantially improved with clearer and more assertive guidance from its headquarters and USAID/Egypt. Training and Capacity Building: Initially, training and capacity building constituted a discrete component of HS 20/20 Egypt as a focused program to establish the Leadership Academy, which was envisioned as a sustainable Egyptian institution designed to build leadership and management capacities to support the implementation of reforms in the Egyptian health sector. While the Academy’s initial year of operation appears to have been reasonably productive, the MOHP decided not to proceed with development of the Academy and the activity was dropped from the HS 20/20 Egypt SOW in August 2009. Subsequently, the project restructured its approach to provide training and capacity building within each of its technical intervention areas and also increased its efforts to strategically link activities with other related training programs supported by USAID. In particular, the project endeavored to maximize working linkages and coordination with GOE counterparts who were graduates of USAID-sponsored programs such as the FORECAST MBA Program and the Harvard Executive Program. Other Strategic Interventions: The Benchmark Conditional Cash Transfer (CCT) Program: At the time HS 20/20 Egypt was designed, USAID engaged in a dialogue with the MOHP about a new program of CCT financing that would entail approximately $110 million over a specified period, with additional adjunct technical assistance funded by USAID to facilitate documentation and verification of achievement of benchmarks. When HS 20/20 began operations in February 2008, responsibility for developing a verification plan and periodically conducting verification exercises was built into the HS 20/20 scope of work. In the SOW, the project was given responsibility for verification of benchmark completion, while the MOHP was tasked with the actual achievement of the benchmarks. It became clear early on that the MOHP would not be able to achieve the benchmarks. The MOHP proposed alternative parameters, leading to an impasse between the MOHP and USAID/Egypt. USAID brought in a respected external consultant (who also worked on the NHA studies and other project elements) to bring together USAID/Egypt, MOHP and the Ministry of International Cooperation (MOIC) to discuss the feasibility, content and timeline of the benchmarks. However, unable to resolve the main challenge—a hard requirement for verifiable evidence—a cash transfer program based on achieving benchmarks was abandoned. KIs to the evaluation confirmed their continuing belief that CCT as a financing mechanism is an effective form of support, and that one major value of this mechanism lies in the ability to focus commitment on specific targets, even during political turbulence. _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project xii MOVING FORWARD AND RECOMMENDATIONS FOR ACTION In looking ahead, USAID/Egypt needs to shift its focus, placing less emphasis on introducing innovative and progressive technology, methods, and tools, and more on creating an enabling environment to facilitate the institutionalization and sustainability of systems technologies and their use. This means substantially increased emphasis on human resource development and capacity building in development and use of evidenced-based strategic planning and management systems and tools (such as NHA, and medical management) -- areas where USAID has a long history of assistance and a well-established competitive advantage. There are a number of immediate management challenges that USAID/Egypt should address:  Limit the use of external consultants and require projects that use innovative strategies and approaches to work directly with Ministry staff to carry out work. This will require far more emphasis on training and capacity transfer. It may slow the pace of activities and results, but may improve engagement, ownership and institutionalization of interventions.  Any future technical assistance activities should focus on leadership development. This is particularly important in reaction to the high turnover of staff at the MOHP in the aftermath of the Egyptian revolution. Additional studies and assessments are necessary to move beyond the defunct Leadership Academy concept and the compartmentalized training and capacity building approaches of HS 20/20 Egypt to a more coordinated approach.  Any future health systems strengthening activity that would be based on the HS 20/20 Egypt experience needs to institute a robust theory of change, a logical framework and a monitoring and evaluation plan at the design stage. Moreover, based on the findings and conclusions presented in this report, the best performing interventions and those with most potential to show a return on future USAID investments in the medium to long term are: 1. National Health Accounts. USAID should continue to invest in “hands-on” production of NHA estimates with heavy emphasis on strategically increasing GOE/MOHP ownership and institutionalization. Emphasis should also be placed on training and capacity building to improve the human resources for production, demand-creation, and institutionalization of NHA estimates. USAID should revisit the possibility of further assistance in strengthening the HEU, but should not invest and become directly involved in attempting to help the MOHP resolve the underlying structural and organizational issues related to HEU. Any request for assistance from the MOHP to work with the HEU should be viewed through the prism of whether or not such assistance would serve the MOHP’s long term institutionalization objectives. As a further word of caution, the outlook for the HEU appears to be sufficiently complex and uncertain at this time, that institutional change in these areas would probably not represent the “low hanging fruit” that might be sought as a benchmark candidate for any future conditional cash transfer program. 2. Strategic studies and assessments. USAID should continue to invest in policy and program￾informing studies and assessments, such as (but not limited to) NHA-related costing research methods. USAID assistance might include support for production/utilization of studies as well as _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project xiii organizational/human resource development. As a word of caution, although USAID invested in good quality strategic studies and assessments under HS 20/20 Egypt, there appeared to be little demand or interests in these products. USAID will need to ensure an efficient way of identifying and effectively engaging Egyptian health policy makers and strategic planners in dissemination of results. 3. Medical management systems. Although HIO is in the midst of major organizational and functional changes, it has been a productive implementing partner and systems development “laboratory” for HS 20/20 Egypt interventions. USAID should continue to invest in HIO (along with HIO successor institutions) in the area of medical management quality improvement systems including human resource and systems prototype developments (in such areas as utilization management, case management, and Key Performance Indicator systems). It is important to note that any future USAID investment should be expanded to include both the curative care and preventive/primary care sectors. This must be considered in the interest of sustaining USAID’s past and current PHC investments, and in the interest of unifying and further strengthening medical management systems in the health sector. 4. Workforce Planning. The need for strategic workforce planning capacity (and a national strategic workforce plan) remains valid, and there is evidence of interest and demand at decentralized levels for WFP. However, assessment and design of any future workforce planning intervention needs to better understand approaches and technical support requirements. Such an assessment should include: a review of the organizational landscape; level of demand for decentralized and centralized workforce planning systems; a systems model and methodology that would best address MOHP strategic workforce planning needs; and identification of an approach to technical assistance that would be responsive to, and engage relevant actors within, the MOHP and GOE. The foregoing short term approaches should help facilitate the institutionalization of three major competencies/capabilities that USAID/Egypt should seek in the medium to long run: (1) Strategic planning is the last mile in achieving institutionalization of the Egypt HS 20/20 investments and in ensuring that quality data is produced, analyzed, brought to scale, and used to induce evidence-based decision making. USAID has a wealth of internal tools to draw from in this area and should include/ link these efforts closely to the Leadership and Management capacity building efforts, as the strategic planning skills taught there can be instrumental in shifting the organizational culture of the MOHP. In particular, these strategic planning skills can help fostering the enabling environment needed to reward transparency and efficiency and to allow such changes to thrive. Specific focus areas should include data utilization and evidence-based planning and coordination mechanisms to facilitate the collaboration that will be vital to realizing the HSRP agenda. These analytical skills will need to be coupled with a political atmosphere conducive to such leadership that will be only somewhat within the manageable interest of the Mission. Capturing progress through a solid M&E system, and potentially (in the long term) reverting back to a _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project xiv benchmark modality envisioned by USAID in 2007, may be important tools to manage the volatility of the political climate in Egypt. (2) Strategic communications. Although HS 20/20 Egypt’s technical assistance plans did not specifically identify communications, technical collaboration and strategic integration as a targeted activity within the project, there was evidence that technical cooperation and promotion of health systems strengthening technologies and tools were inherent to HS 20/20 Egypt approaches. Thus, the project provided several examples of effective technical information sharing and collaboration and which should be invested in further. Firstly, to continue to improve the role it plays in technical collaboration and strategic integration of USAID programs, USAID might consider building directly into future projects a requirement for the project to develop a technical information sharing or “strategic integration” plan. Secondly, any such plan should include activities that not only involve the client organization (such as the MOHP or HIO), but facilitate the client organization taking the lead (which was done quite effectively with HS 20/20 Egypt’s Sharm El Sheikh Conference on NHA). ____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 1 I. PROJECT DESCRIPTION HS 20/20 Egypt, which began in February 2008 and ended in April 2012, was one of fifty-one country￾specific projects of the global HS 20/20 flagship cooperative agreement for health-systems strengthening implemented by Abt Associates and its consortium. USAID/Egypt sought assistance from HS 20/20 to support the efforts of Egypt’s Ministry of Health and Population (MOHP) to build capacity in health sector strategic planning and financing, social health insurance (SHI), and quality of care. HS 20/20 Egypt intended to benefit client health institutions by strengthening health planning and management systems, and by strengthening the capacity of individual health professionals through transfer of health research, planning and management skills and capacity building. II. EVALUATION PURPOSE AND QUESTIONS Social Impact, Inc. (SI) was requested by USAID/Egypt to conduct an end-of-project performance evaluation of the health-systems-strengthening project, Health Systems 20/20, in Egypt (HS 20/20 Egypt). The evaluation had two overarching purposes: (1) assess to what extent the project’s objectives and goals contributed to the achievement of the intended results and (2) determine whether moving forward with a new health-systems-strengthening project is a good investment of USAID funds. In addition to performance and achievements, the evaluators were asked to assess the project’s working relationships with counterparts and stakeholders; to identify ways in which the monitoring and evaluation functions of future projects can be improved; and to provide guidance regarding the manner in which USAID might maximize lessons learned for future investments in health-systems strengthening. The Statement of Work (SOW) for this evaluation is shown in ANNEX A. The primary audience for this evaluation report is the Office of Health and Population within USAID/Egypt. USAID intends to integrate report recommendations into consideration of future health￾systems-strengthening activities, and additionally plans to share the results of this evaluation, as appropriate, with the USAID Middle East Bureau, and the USAID Office of Health Systems. Furthermore, as HS 20/20 Egypt’s main collaborating partners, USAID/Egypt plans to share the results of this evaluation with the MOHP, the Health Insurance Organization (HIO) and interested donor partners. English and Egyptian Arabic versions of an expanded executive summary of this final report are available for dissemination to relevant stakeholders and the general public via the USAID Development Experience Clearinghouse (DEC). KEY QUESTIONS Building from the illustrative Key Questions (KQs) contained in the original evaluation SOW, an initial task of the evaluators was to refine and/or expand KQs as necessary for clarity, measurability and comprehensiveness. A final set of KQs, derived with the approval of USAID/Egypt, was employed extensively for all aspects of this evaluation, including data collection, analysis, synthesis and reporting. Key Questions are shown below in Figure 1and data collection instruments are presented in Annex F. _____________________________________________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 2 Figure 1: Evaluation Key Questions LOOKING BACK 1. What were HS 20/20 Egypt’s rationale, objectives and planned activities? 2. What changes/improvements in health systems resulted from HS 20/20’s project design and implementation? 3. Compared to the project objectives and planned activities, what was actually achieved by HS 20/20, how was this done, and if objectives and targets were not reached, why not? 4. What evidence is there that the GOE/project’s strategies and achievements have led to actual improvements in sector systems? 5. What interventions (strategies/activities) have been adopted and are being implemented in a sustainable manner by the GOE? 6. To what extent was the project’s implementation strategy (of expert systems development combined with capacity transfer) perceived as effective by GOE counterparts? 7. To what extent did the project improve cooperation and strategic integration of program interventions among USAID implementing partners, donors, and public/private sector partners? MONITORING AND EVALUATION 8. Describe the formats and processes of HS 20/20’s M&E systems in Egypt. What results, indicators and targets were being regularly and routinely tracked? 9. How did the M&E system capture the systems strengthening changes? How could measurement of systems strengthening changes have been strengthened? 10. How was M&E information being used by the project and GOE counterparts for program management, decision making and improvements? 11. To what extent does the MOHP and the HIO have and sustain monitoring and evaluation capabilities in the project intervention areas? MOVING FORWARD 12. What HS 20/20 project interventions have the most potential to show a return on investment for health systems strengthening? 13. What should USAID do to sustain these achievements moving forward? 14. Considering the GOE’s direction, the commitment of other donors, and USAID’s broadly defined health sector support in Egypt, what health systems strengthening interventions are within USAID’s manageable capacity/interests? How can USAID better ensure program effectiveness in its future support for health systems strengthening interventions? 15. What enabling factors and critical assumptions would need to be in place? 16. In the future, what can USAID do to improve the role it plays in fostering strategic integration of program interventions in the sector among USAID implementing partners, donors, and public/private sector partners? _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 3 HEALTH SECTOR DEVELOPMENT CHALLENGES The Egyptian health sector has been in a state of constant reform over the past two decades. At the core of reforms put in place by the MOHP is the Family Health Model, which aims to improve service delivery by separating payer and provider functions, introducing performance-based contracts, and guaranteeing universal access to health care. While some aspects of the reform package have been quite successful— most notably, the new primary health-service-delivery model and improved access to care—others have lagged behind (mainly the financing and insurance components). As a result, Egypt spends a higher share of its GDP (6%) on health care than many countries in the region, and more than 70 % of the costs are direct, out-of-pocket expenditures by Egyptian households.1 Moreover, public health care provision in Egypt suffers from under-utilization and overstaffing of primary health-care facilities, low provider capacity, and a significant disconnect between the demand by the population for health services and the services offered by the MOHP.2 OVERVIEW OF HS 20/20 EGYPT The HS 20/20 Egypt project was funded for four years at a level of $10,788,902. Roughly $5.6 million of this total was allocated to development of strategic information and planning systems; $3 million to support development of HIO financing and medical management systems; and $2.1 million to support other strategic interventions (OSI), such as special studies.3 USAID support to health sector reforms and systems strengthening during the 2008–2012 period proved relatively modest when compared to inputs of other donors, namely the World Bank (WB) and the European Commission (EC). For more information on USAID and other donor financing for reforms, please see Annex C. Within the context of Egypt’s overarching health sector challenges, the design of HS 20/20 Egypt built upon USAID’s extensive experience and leadership in the health sector reform process in Egypt. HS 20/20 Egypt was designed over a 6-month period between February and September 2008, based on three consultative trips and a series of meetings between HS 20/20 headquarters and MOHP, HIO, and USAID/Egypt. Guided both by HS 20/20’s global framework and needs on the ground in Egypt, five related but distinct technical objectives emerged: (1) to develop national health accounts (NHA) studies and institutionalize NHA capacity within the government of Egypt (GOE); (2) to develop a long-term strategic workforce plan and workforce planning capacity within the MOHP; (3) to support the HIO in building financial administration and medical management capacity in anticipation of it assuming the role of payer under the new health insurance law; and (4) to undertake training and capacity building (TCB), including establishment of a sustainable “Leadership Academy” to build leadership and strategic 1 Farag, Mahmoud (2012). National Health Accounts in Egypt Overview and Key Findings. Presentation USAID/Egypt. 2 “Management and Service Quality in Primary Health Care Facilities in Alexandria and Menoufia Governorates”, World Bank, Washington D.C., 2010. 3 USAID/HS 20/20 Egypt funding documents: MAARD Amendments No. 1, 2 and 3. _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 4 management capacities in support of the implementation of health sector reforms.4 A fifth technical objective (which is referred to in this report as “Other Strategic Interventions”, or OSI) included the Benchmarks Conditional Cash Transfer (CCT) program, which was halted during the course of the project, a sustainability study, and the development of six case studies. HS 20/20 EGYPT’S UNDERLYING THEORY OF CHANGE AND ASSUMPTIONS As described in project documents, HS 20/20 Egypt used a technical approach in the design and implementation of its interventions suggestive of a generic theory of change that describes a pathway to systems change in complex institutions, such as Egypt’s health system. The “if-then” relationships between elements in HS 20/20 Egypt interventions involved: (1) generation of high quality data and evidence; (2) development of systems, structures and human resources (HR) to institutionalize data management capacities; and (3) use of data and evidence by appropriate decision makers to inform and strengthen policies, programs, or management systems. This theory of change driving HS 20/20 Egypt’s interventions—as interpreted by the evaluation team—is depicted in Figure 2. Figure 2: HS 20/20 Illustrative HS 20/20 Egypt Logic Model and Technical Approach HS 20/20 Egypt’s theory of change implies that a variety of conditions would need to be met to ensure effective systems change, including: management agreement, demand and political will for the systems changes targeted by the project; designated counterparts that have the capacity and authority to drive systems change; data and evidence that is demand-driven, widely disseminated and widely used; appropriate counterpart technical staff is partnered for effective skills transfer, training and capacity building; and effective, two-way communications are in place to ensure on-going monitoring and alignment of project activities. 4 The Leadership Academy was halted after one year of assistance at the request of the MOHP. Instead, HS 20/20 Egypt training and capacity-building activities were integrated into the project’s technical components. _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 5 AN ILLUSTRATIVE HS 20/20 EGYPT RESULTS FRAMEWORK HS 20/20 did not develop a logical framework or results framework for the country-specific HS 20/20 Egypt project. Instead, it produced many project design and strategy documents, as well as output-level indicators in response to the global HS 20/20 results framework. In the absence of a country-specific results framework (which is typically also the starting point for a performance evaluation)5 , the evaluation team developed an illustrative results framework, shown below in Table 1, depicting the logical structure of the country project and what it was supposed to accomplish. Table 1: Illustrative HS 20/20 Egypt Results Framework 5 USAID Performance Monitoring and Evaluation TIPS No. 17, Constructing an Evaluation report, states that, “If the team cannot find a description of …any model of the project’s cause-and-effect logic such as a results framework or a Logical Framework, this should be noted”, and goes on to suggest that the evaluation team will then have to summarize the project strategy …to show how the project designers envisioned the interventions as leading to desired results. Objective: To assist in strengthening selected health systems in support of the GOE/MOHP’s long￾term health sector reforms INTERMEDIATE RESULTS SUB-INTERMEDIATE RESULTS National Health Accounts (NHA): IR #1: Institutionalized GOE capacity to independently produce NHA estimates and strategic planning studies  NHA estimates and strategic planning studies regularly produced  Structures and processes for regular production of NHA estimates and strategic planning studies institutionalized  Number of persons trained and institutional capacity to regularly produce NHA estimates and strategic planning studies increased Workforce Planning (WFP) IR #2: Strengthened MOHP capacity to develop high￾quality workforce plans  Facility-based WISN planning system demonstrated  Governorate-level hospital workforce planning model demonstrated  National Strategic Workforce Plan Developed  Number of persons trained and institutional capacity to regularly plan and produce national strategic workforce plans increased Health Insurance Organization (HIO) IR# 3: Strengthened HIO financial and medical management capacity  Number of persons trained in financial management systems increased  Medical management systems, including utilization management and case management developed and implemented  Number of persons trained and institutional capacity to maintain medical management systems increased Institutional Training and Capacity Building (TCB) IR #4: Increased pool of health leaders and strategic planners  Leadership Academy for health managers and strategic planners established _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 6 HS 20/20 EGYPT DESIGN CHANGES Although the fundamental purpose and technical objectives of the project did not change substantially, a number of significant changes occurred. The Benchmarks Conditional Cash Transfer (CCT) program and the Leadership Academy, both developed in 2006/2007 and folded into the HS 20/20 Egypt SOW, were cancelled after a year during which HS 20/20 Egypt had expended substantial level of effort and resources on these components. Additionally, the claims management sub component of financial management training at HIO was cancelled at HIO’s request, and there were modifications to HS 20/20 Egypt’s technical approach to workforce assessment. These changes are discussed in the respective findings and conclusions sections of the report. III. EVALUATION METHODS AND LIMITATIONS The main goal of the evaluation was to assess the extent to which HS 20/20 Egypt’s project objectives and goals contributed to the achievement of the intended results and, based on these findings, make recommendations regarding USAID’s possible future investments in health-systems-strengthening projects in Egypt. The evaluation used primarily qualitative research methods, combined with a thorough review of project documentation from HS 20/20 Egypt as well as data and records from the MOHP and HIO. Being able to answer the evaluation’s KQs was of primary concern in formulation of the evaluation’s conceptual framework, methodology, and methods. As described in detail in Annex D, the basic approach used in this evaluation involved cross-referencing KQs to each of the five technical project components (i.e., results areas). A matrix format for this cross-referencing exercise then allowed the evaluation team to identify data collection and develop data gathering methods that focused on answering KQs. This approach ensured that all KQs were addressed in relation to all project components and activities. KIs included representatives of USAID/Egypt, HS 20/20, MOHP and HIO at the central level in Cairo and in two other governorates, and other partners such as international donors. A list of persons contacted and interviewed individually or in groups is included in this report in Annex E. METHODS A variety of evaluation methods were used, including: (1) extensive desk reviews; (2) KI interviews; (3) site visits; (4) focus group discussions (FGDs; and (5) team information sharing, data synthesis, and triangulation. Samples of data collection instruments used for SKII and FGDs are included in this report in Annex G. Each of these methods and relevant limitations are discussed further below. Desk Reviews: As this was an ex-post evaluation conducted well after project personnel had disbanded, review of project documentation became a key data collection method used by the evaluators. The volume Other Strategic Interventions (OSI) IR #5: Completed studies and assessments to support health strategic planning  Verification measures and procedures for the USAID Benchmarks CCT Program established  Results of strategic studies and assessments disseminated _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 7 of documents provided to the team beforehand was copious (over one-hundred documents from USAID/Egypt alone). Additional items, e.g., trip reports and other file data, were also provided to the team by HS 20/20 headquarters. Key Informant Interviews: The evaluation team conducted face-to-face interviews with key individuals and/or groups from all five technical areas. Group meetings, documented with meeting notes, and interviews (both group and individual) were facilitated by use of a Structured Key Informant Interview (SKII) technique tailored to the purposes of this evaluation. The SKII approach used predetermined, semi￾open questions to balance comparability against the need for interviewees to be able to speak freely about their experiences within the different project components. Focus Group Discussions (FGDs): In keeping with best practices for design of FGDs, evaluators developed and used a set of structured questions that are targeted to that specific group of FGD participants. In addition to discussions, in three FGDs with training beneficiaries, participants were asked to quantify their impressions of the training they received by completing a rating questionnaire. Site Visits: Due to the concentrations of intervention sites, the limited timeframe of the evaluation, and security considerations, most site visits were to offices and health facilities in the greater Cairo area. In addition, field trips were made to Luxor and Quena Governorates to review HS 20/20 Egypt Workforce Planning activities and the project’s M&E procedures. In both Luxor and Quena, the evaluation team also visited sites that had not received the Workforce Planning intervention in order to observe and compare potential differences between intervention and non-intervention sites. Team information sharing, data synthesis, and triangulation: As a further, non-structured approach to refining and processing evaluation data and information, team members were required to fully utilize the SKII data-gathering instrument to take notes and document interviews and discussions; resulting notes were shared among team members as part of the team’s data synthesis process. At every step during the evaluation, from the initial document review to synthesizing final recommendations, the team used a highly participatory approach to fully incorporate the knowledge and insights of all team members and KIs into the formulation of findings, conclusions and recommendations. In particular, the team worked closely with USAID/Egypt staff, who were consulted by the evaluation team throughout the evaluation and whose inputs were instrumental in ensuring that the evaluation team was responsive to the interests and needs of USAID, as stated in the evaluation SOW. LIMITATIONS Limitations in program documentation. In spite of the abundance of documentation available, the evaluation team noted one serious limitation in program documentation. As further discussed in Section IV, the evaluators found that project documentation related to five very different and technically complex interventions was interwoven into various funding documents; a performance monitoring plan and indicators with Egypt-specific activities (addressed to a HS 20/20 global results framework, not Egypt￾specific intermediate results); multiple, disjointed, annual work plans and revisions; quarterly progress and work planning documents; technical assessments and reports; briefing documents; and power point presentations. This approach to program documentation rendered it difficult for the evaluators to make connections between planned results and activities, what actually did or did not occur, and the reasons for certain implementation decisions. The technical rationale, inputs, outputs and outcomes for a specific intervention area were never found to be discussed together in a single document. Lack of coherency in _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 8 file documentation presented an unusual challenge and hampered the ability of this evaluation to accurately assess project achievements. The January 2011 Revolution. The political turmoil of Egypt’s 2011 revolution created two major challenges for this evaluation: first, change/loss of staff and program activity relocation led to both inconsistent (i.e. absence of quarterly reports for the last two quarters) and lost documentation. Second, the evaluation itself coincided with the second anniversary of the revolution, which posed logistical challenges for the evaluation team: meetings were delayed or cancelled; government buildings and certain field sites were sometimes inaccessible; and turnover in government staff created gaps in institutional knowledge among KIs. Overall, however, most primary data collection was completed as planned. Limitations on ability to identify gender issues. The pool of KIs included in this evaluation was specifically targeted to organizations. As such, opportunities to probe issues that might have a bearing on gender issues were limited. Where appropriate, however, data collected during the evaluation was disaggregated by sex, for instance in the FGDs. IV. FINDINGS AND CONCLUSIONS NATIONAL HEALTH ACCOUNTS (NHA) Background Over the past two decades, the management practice of periodically making NHA estimates has proven to be a powerful tool to inform health financing policy, as well as to monitor the outcomes of policy interventions. NHA is a rigorous, globally accepted approach to classifying the types and purposes of expenditures of actors in the health system. These estimates provide an integrated picture of mobilization, management and use of health funds in the health system. 6 NHAs historically have played an important role in generating the evidence needed for health policymakers in Egypt to move forward with health￾sector reforms and health-systems strengthening. Egypt was one of the first low and middle-income countries in the world to conduct NHAs. The first round of NHAs was conducted in 1991, 7 the second in 1994–1995, the third in 2001–2002, the fourth in 2007, and the fifth in 2008–2009. Over the years, USAID, the World Health Organization (WHO), and the World Bank have supported this effort in Egypt. The lack of institutionalization in Egypt has meant that NHAs have been conducted sporadically and often carried out by external consultants. 8 Between 2008 and 2011, HS 20/20 Egypt worked with the 6 “National Health Accounts and Public Expenditure Reviews: Redundant or Complementary Tools?” Health Systems 20/20, January 2009. 7 This first NHA estimation is frequently forgotten when listing prior NHAs in Egypt, but is frequently referred to in the literature as, “ENHA91”. (See Egypt National Health Accounts, 1994–1995). 8 Nandakumar, A.K. “National Health Accounts 2008-2009 Executive Summary”, USAID, 2010. _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 9 MOHP on the 2007 and 2008 NHA estimations and also renewed previous efforts to institutionalize NHA capacity in the MOHP.9 Findings Planned Activities and Outputs: All activities carried out under this component are summarized in Table 2. The NHA component had two sub-results: (1) to complete the fourth NHA estimates for 2007 and the fifth NHA estimates for 2008–2009; and (2) to institutionalize capacity within the MOHP to independently manage future NHAs. HS 20/20 Egypt came with strong credentials as HS 20/20 Egypt predecessor projects10 had assisted the MOHP with earlier NHA estimations, and more than one HS 20/20 Egypt consultant had provided NHAs for the MOHP, extending back to the early 1990s. In discussions with two donor partners who have supported MOHP health reform efforts since the 1990s, HS 20/20 Egypt was referred to as the presumed technical leader in support of NHAs and related capacity building in Egypt. Table 2 NHA Activities and Outcomes 9 “Implementing National Health Accounts in Egypt”, HS 20/20, 2012. 10 The Partnerships for Health Reform (1998–2001), and the Partnerships for Health Reform Plus (2003–2006), both also implemented by Abt Associates. PLANNED ACTIVITIES AND TASKS START END STATUS NHA 4 (2007–2008) 2008 2009 Completed NHA 5 (2008–2009) 2.1 Household Survey Oct. 1, 2009 Feb. 15, 2011 Completed 2.2 NGOs, Public, and Private Firms Surveys April 15, 2010 Nov. 15, 2010 Completed 2.3. Costing Studies Oct. 1, 2009 Oct. 31, 2010 Completed 2.4.1 Expenditure Tracking Methodology April 15, 2010 Feb. 28, 2011 Completed 2.4.2 Expenditure Tracking Piloting April 15, 2010 April 30, 2011 On hold 2.5 Collection of NHA 4 secondary data Sep. 1, 2010 Jan. 15, 2011 Completed 2.6 Data review to cover data gaps for NHA 4 data Oct. 1, 2010 Jan. 15, 2011 Completed 2.7 Develop t-accounts and matrices for NHA 4 Dec. 1, 2010 Jan. 15, 2011 Completed 2.8 Analysis: NHA final matrices for NHA 4 Dec. 16, 2010 April 25, 2011 Completed 2.9 Prepare report on NHA 4 results April 25, 2011 Nov. 18, 2011 Completed Institutionalization of NHA and establishment of an economic unit _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 10 Achievements: HS 20/20 Egypt produced two NHAs (2007 and 2008/2009) within a relatively tight timeframe. Both reports have been published and widely disseminated, and are now in the public domain. All KIs felt that the two NHA estimates were of acceptable quality and that the results will be important for future health-sector strategic thinking, planning and policy formulation in Egypt. The evaluation team shares the overall assessment of the quality of both NHA estimates, which has also been verified by an eminent NHA expert who was directly involved in the technical preparation of both estimates. The 2008– 2009 estimates appear to have been readily accepted and quoted in discussions and literature. This can be used as an indication that the technical processes used in developing the estimates were generally accepted. The results of the 2007 NHA were presented at the MOHP’s annual Sharm El Sheikh Health Sector Reform Conference in October 2010. This was the first time the MOHP was able to demonstrate technical leadership and ownership of NHA estimates and to disseminate results to a wide audience of policy-makers, donors, and senior GOE officials. Demand for NHA information and interest in the results and implications for policy appeared to increase as a result. In addition to producing two NHA estimates for consecutive years, HS 20/20 Egypt initiated and completed important studies and surveys related to sub-accounts for the 2007 and 2008/2009 NHAs. Implementation of these studies involved sub-contracting with local agencies and/or training of central MOHP and governorate staff to carry out the study protocols. Notable achievements included:  Costing study of seven hospitals: Completed cost analysis reports for seven hospitals (Suez General Hospital, Suez Fever Hospital, Um El Atabaa Hospital, Mamoura Psychiatry Hospital, Sohag General Hospital, Ras El Tin General Hospital and Al Salam Tumor Center); developed hospital-cost-analysis software to ensure sustainability and rolling out of the costing exercise; and conducted 14 on-the-job￾training sessions on costing analysis of hospitals for the MOHP Department of Planning (DOP) staff. The objective was (following this training) for the DOP costing team to be able to conduct hospital costing and train DOP staff at the governorate level.  Household Utilization Survey: Subcontracted with the Central Agency for Public Mobilization and Statistics (CAPMAS) to conduct the household utilization survey, covering 12,000 households in two seasons.  NGOs, public, and private firms surveys: Subcontracted with CAPMAS to conduct surveys for NGOs, public, and private firms to assess their health expenditures.  Expenditure Tracking System (ETS): assisted the MOHP DOP in updating the definition of the ETS, a financial tracking system that allows data to be collected on an ongoing basis to eventually feed into NHA estimates, rather than having data collected in a special study at the time of the NHA. The updated definitions were intended to make the ETS more responsive to MOHP strategic planning. In addition, HS 20/20 Egypt assisted the DOP in training personnel and piloting the ETS in three programs: family planning (FP), maternal and child health (MCH) and infection control (IC). MOHP Technical Capacity and Institutionalization of NHA : As defined by the project in documentation and interviews, and for purposes of this evaluation, “institutionalization” means clear 3.1 Collaborate with the WB to develop a startegic institutionalization plan July 1, 2011 Oct. 31, 2011 Completed 3.2 Capacity building for the MOHP HEU Oct. 1, 2011 Dec. 31 , 2011 Not yet started 3.3 Hospital Costing Software development Sep. 1, 2011 Nov. 18, 2011 In progress _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 11 evidence of technical capacity, clear evidence of organizational capacity (which includes structural elements like collaborative planning processes and mechanisms), clear evidence of an enabling environment (which includes both political support and institutional culture) and resources (which include both human and financial components). After two decades of USAID assistance to conduct NHAs, and with well-established GOE accounting and strategic planning standards, it would be reasonable to assume that the basic capacity to conduct NHAs would have long since been institutionalized within the MOHP. However, as described by HS 20/20 Egypt in project documentation and interviews, the reality in 2008 was quite different. Although a few MOHP technical staff with prior experience with NHAs remained in place, attrition, irregular practice of skills, and chronic conflicts within the MOHP regarding chain of command vis a vis coordination of NHAs hampered the ability of the MOHP to truly institutionalize its NHA capacity. Compared to the direct support related to the two NHA cohorts, HS 20/20 Egypt saw very limited success in its efforts to facilitate institutionalization of NHA at the MOHP. The main reason for this limited success was lack of strategic direction from the MOHP regarding the best institutional housing for NHA estimates and other research and costing functions to support sector strategic planning. Although this conflict clearly was an internal MOHP affair, HS 20/20 Egypt’s first approach in helping the MOHP to resolve the conflict was to take the matter to the court of public opinion. HS 20/20 Egypt conducted a telephone survey with stakeholders about their preferences for where the NHA functions should be housed in the MOHP. The results of that survey could not be clearly determined from the project’s records. HS 20/20 Egypt also collaborated with the World Bank in the development of a Global Strategic Action Plan (GSAP) for building consensus and a plan of action for the institutionalization of NHA, capitalizing upon the World Bank’s reputation as a global leader such endeavors.11 Results of the GSAP were presented by MOHP representatives at the international NHA Institutionalization Conference in Paris in October 2011, but the MOHP continued to struggle with effective implementation of the plan. In tandem with the GSAP plan, HS 20/20 Egypt assisted the DOP in the formal establishment of a Health Economics Unit (HEU) that would house NHA estimates and other studies and analytical functions to support strategic planning. According to an official in the DOP, HS 20/20 Egypt assisted the DOP in successfully arranging, through the Central Agency for Organization Administration, to house the HEU within the DOP. This would have represented a monumental step toward resolving the structural issues surrounding NHA management. However, this development apparently lacked the approval of the Technical Office of the Minister (of Health and Population) because, by the time of this evaluation, authority and mandates for the HEU had been transferred from the DOP to the Technical Office of the Minister (although the vetted health economics staff and positions remained in the DOP) and there are now effectively two HEUs. The outlook for resolution of this issue remains unclear at the time of this report, but, according to the Technical Office of the Minister, it is likely that Technical Office leadership will build strategic costing, research and planning capacity within all relevant central MOHP sectors, including the DOP, and coordinate all activities in the nascent HEU that lies under the authority of the Minister. 11 “Where is the money and what are we doing with it?” World Bank, 2011. _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 12 While HS 20/20 Egypt was unable to make significant progress on helping the MOHP with organizational/structural matters, the project did lay some groundwork for institutionalization by training DOP staff. Although, according to project documents, many DOP staff lacked some of the basic skills and qualifications needed for NHA work, HS 20/20 Egypt nevertheless responded to a MOHP request to train DOP staff in costing analysis and ETS. Project reports stated that DOP staff would still require considerably more skills training in the future. Conclusions HS 20/20 Egypt performed well in assisting the MOHP to successfully complete two NHA estimates of high quality. This effort profited from prolonged USAID experience and leadership in conducting NHAs in Egypt and elsewhere. HS 20/20 Egypt also performed well and was highly productive in conducting or arranging to conduct several complex costing studies and tracking systems related to preparation of NHA estimates. While these technical tasks were carried out, to some extent, in collaboration with the MOHP, they were mainly the product of external consultants, which may have reduced the extent to which HS 20/20 Egypt could leverage its technical capacity to assist the MOHP with the institutionalization of NHAs. The project failed, for the most part, with the institutionalization of NHA capacity, the second main goal of the component. Although the project put in considerable effort and made a modest contribution to improving the MOHP’s capacity to manage NHAs by training a limited number of MOHP staff in costing and expenditure tracking, this did not translate into sustained institutional capacity of the MOHP to carry out future NHA exercises. One of the main reasons for this shortcoming is the continued uncertainty about the manner in which the nascent HEU in the Technical Office of the Minister will function, develop and grow. The HEU was the designated entity within the MOHP to house NHA estimates and other studies and analytical functions to support strategic planning, however, mandates for the HEU had been transferred from the DOP to the Technical Office of the Minister (although the vetted health economics staff and positions remained in the DOP) and there are now effectively two HEUs. WORKFORCE PLANNING (WFP) Background Established in 1964, the MOHP is one of the largest public organizations in Egypt, with more than 500,000 employees. One of the key challenges facing the MOHP is ensuring that Egypt has the adequate human capacity and workforce to roll out and sustain its reform programs. The Human Resource (HR) functions inside MOHP are spread across several entities, with no centralized HR management program or department. The main challenges faced by the MOHP in its efforts to manage its workforce are:  Inadequate HR management capacity for a very large workforce  Challenges meeting workforce norms, both in terms of technical specialization and geographical distribution of staff  Sub-optimal quality of the graduating workforce at MOHP  Poor coordination between the MOHP and the Ministry of Higher Education to align the supply and demand of workforce needs  Low salaries and poor incentives that contribute to low motivation and productivity of the workforce _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 13 Findings Planned Activities and Outputs: According to the 2008 Technical Assistance Plan (TAP), HS 20/20 Egypt’s two main objectives were to (1) develop a long-term workforce plan for the MOHP and to (2) establish sustainable workforce planning capacity within the MOHP. The project’s SOW included eight major activities: (1) strategy validation, (2) MOHP task force selection and formalization, (3) collection and validation of workforce data, (4) development of a workforce model, (5) development and implementation of workforce standards, (6) assistance of MOHP on workforce surveys and report development, (7) development of a national strategic workforce plan, and (8) capacity building for carrying out the data collection analysis, data interpretation, and forecasting functions of workforce planning. HS 20/20 Egypt recruited a short-term international expert on Human Resources in 2008 to review the current situation within the MOHP and to propose a vision and implementation framework for Human Resources inside the Ministry. However, this proposal was never endorsed by MOHP or USAID. Later in the year, HS 20/20 Egypt proposed in its technical implementation plan, dated September 2008, to adopt the WHO’s WFP model—a proposal that was adopted and later rolled out throughout the life of the project. Three pilot governorates were to be included in the WFP activities: Asyut, Gharbia and Dakahlia (which was later replaced by Luxor). In addition to the pilot governorates, eight roll-out governorates were selected for the MOHP’s WFP assessment survey: Dakahlia, Sharkia, Damietta from Lower Egypt, New Valley, Red Sea as frontier governorates; Quena, Sohag from Upper Egypt and Cairo. Achievements: With support from HS 20/20 Egypt, the MOHP established and staffed a Human Resource Task Force (HRTF). The HRTF consisted of five volunteers from within the MOHP headquarters, led by the MOHP Undersecretary for Training and Research. Staffed by full-time MOHP employees with medical degrees and minimal, if any, human-resource-management experience, its function was intended to ensure buy-in, continuity and accountability. HS 20/20 Egypt worked with the HRTF to develop policies and procedures and to build capacity for conducting workforce planning over the longer term. The HRTF collected and validated the MOHP workforce data; developed the workforce model; developed health workforce standards; and conducted capacity-building activities, including training and technical assistance to develop a data collection and processing plan that addressed the indicators and variables required for workforce planning over the longer term. A software integration team was also established in the three pilot governorates, consisting of 12 statisticians and one software administrator acting as IT technical support and project manager. The team was tasked with providing technical support to the eight roll out governorates teams and conduct data entry, as well as data verification, whenever required. The National Information Center for Health and Population (NICHP) reported that it had both the access to the source code of the basic, MS Access software installed and the capacity to integrate it with MOHP tools or to upgrade this tool, if required. Unfortunately, this task was put on hold after the revolution and no progress was made in terms of roll￾out to remaining governorates, or in utilizing results derived from the assessment conducted in any of the 11 governorates. HS 20/20 Egypt helped establish a workforce-planning system designed according to the WHO’s Workload Indicators of Staffing Needs (WISN) planning tool. The standards established under this tool were used to develop Egypt’s activity standards for MOHP-district and general hospitals. Egypt’s standards were later verified and approved by the WHO for assessing workload and staffing needs inside _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 14 MOHP hospitals. A key point to note, however, is that the WISN methodology produced bottom-up Governorate-level data, and it will likely take some time before a critical quantity of data is available to attempt national strategic workforce projections. Challenges: During a series of field trips and hospital visits, the team found multiple facility-level databases within the same hospital/facility that track human-resource or human-resource-related data. None of these data sources were linked and staff either did not collect data or managed it ineffectively, thus increasing the work burden on staff, rather than trying to streamline redundant activities or data collected. In order to gain a better understanding of how WFP is conducted at the MOHP in the absence of HS 20/20 Egypt, the evaluation team visited a comparison governorate (Quena) that was not part of the project. This revealed two significant shortcomings of the project’s WFP efforts. First, on the organizational level, the project did not engage the agency that allocates clinicians to facilities across Egypt, the Department of Compulsory Service (DCS), nor any other GOE or MOHP offices concerned with supply and allocation of health human resources. Reasons for this oversight are unclear, but while a wide variety of stakeholders were engaged and consulted in the WFP efforts (including the HRTF), none of them had authority to actually assign staff to facilities and locations. Second, and perhaps even more importantly, it became clear that HS 20/20 Egypt had only engaged the supply side of the HR/WFP equation. The stakeholders involved in the project only deal with the ‘pull/supply’ side, trying to pull doctors to under-resourced areas, regardless of actual HR needs. This raises the question of how to integrate evidence-based demand and strategic WFP into the MOHP system, where only doctors have input into their placement. It became evident during the site visit to Quena that Egypt does not have too many doctors: rather, they are poorly allocated, both geographically and according to physicians’ specialty preferences after their initial, two-year obligatory services. With regards to PHC workload standards, HS 20/20 Egypt focused its engagement directly on MOHP district and general hospitals. The project suggested the establishment of a steering committee within the MOHP to oversee WFP activities and overall human resource services at the ministry. HS 20/20 Egypt reported that MOHP denied this request, for reasons not entirely clear to the evaluation team but likely because of a reflection of overall lack of engagement at central levels. HS 20/20 Egypt also reported that MOHP had requested support in developing a nationwide personnel database for all MOHP employees. After consultations with HS 20/20 headquarters and USAID/Egypt, HS 20/20 Egypt was able to offer limited technical assistance, but declined to finance this activity. Conclusion The original goals regarding WFP in the HS 20/20 Egypt project were to develop a long-term workforce plan for the MOHP and to establish a sustainable workforce planning program. This translated into several activities aiming at (1) building the capacity of MOHP staff to produce WFP data and reports; and (2) institutionalizing WFP at the MOHP. While HS 20/20 Egypt did successfully produce WFP data and reports, it made relatively little progress towards institutionalization. There were four main challenges to the institutionalization of the WFP effort. First, there was lack of clarity in HS 20/20 Egypt dialogue with MOHP policy makers and counterparts about the ultimate purposes and methodology of the planning system and the organizational capacities needed to sustain it. _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 15 Second, HS 20/20 Egypt did not identify and engage the primary WFP stakeholder(s) within the MOHP and instead focused its engagement on offices that could generate and use workforce data for decentralized management purposes, but that did not ultimately have human resources policy or strategic decision making authorities. Third, human resource management functions inside MOHP were fragmented with different roles played by different divisions with no unified sector or departmental oversight. It is possible that HS 20/20 Egypt needed additional expertise in this highly specialized area of workforce strategic planning to address these challenges. Finally, HS 20/20 Egypt did not consider the HR component of the sector as a whole, but focused narrowly on district/general MOHP-owned hospitals. PHC/FHC facilities, hospitals at tertiary levels and other government-owned facilities or private-sector facilities were not included. HEALTH INSURANCE ORGANIZATION (HIO) Background The HIO was established in 1964 as an independent government organization under the supervision of the MOHP to provide health insurance and services for formal sector employees. As of 2008, HIO provided insurance coverage to 55 % of all Egyptians identified for compulsory enrollment on a categorical basis.12 Enrolled groups include: GOE employees, other public and private sector employees, pensioners and widows, school children and newborns. All HIO beneficiaries are entitled to the services in the HIO’s basic benefit package provided by HIO’s own facilities and/or by HIO-contracted providers. Despite covering a large share of the population, only four percent of outpatient and eight percent of inpatient health service visits were reported to be to an HIO provider/facility, mainly due to location or a perception of low quality of care relative to private sector providers. HIO operations were also reported to run large deficits, due to expenditures exceeding revenues earmarked for each of the covered groups,13 which are financed out of general tax revenues.14 Thus, by the mid-2000s, the development/improvement of mechanisms to advance the quality of HIO services, as well as control the costs of providing those services, were priority challenges under the HIO’s existing missions. In 2005, the MOHP re-initiated its long-standing policy objective to expand health insurance coverage to all Egyptians. The GOE recognized that achievement of this objective would need to include passage of the SHI law, and a reform of the HIO to undertake a new role of “payer” in the reformed system, either by being reorganized into a pure insurance company or being folded into a newly formed, public-insurance entity. 12 National Health Accounts, 2007/08. 13 El Zanaty, F., El Adawy, M., ElGhazaly, N. “Health Insurance Household Survey”, MOHP/HSRP, 2006. 14 Antos, J., Hafez-Afifi, N. and S. el-Saharty. “Egypt’s Health Sector Reform and Financing Review”, World Bank, (unpublished), 2004. _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 16 Findings Planned Areas for Intervention: During the 2009–2012 period, HS 20/20 Egypt provided technical assistance and training to the HIO in the areas of: accounting, costing and financial management; claims management; and medical management. HS 20/20 Egypt’s inputs were intended to build HIO’s capacity to not only better address existing challenges related to quality improvement and financial control, but also to prepare HIO staff for anticipated new “payer” functions under the pending SHI law. Achievements and Challenges: HS 20/20 Egypt engaged an Egyptian academic to provide training and mentoring in the area of accounting, costing and financial management. A study of the operations of the financial department, including a training-needs assessment for financial department staff, was not conducted—instead, the terms of reference for the consultant listed areas in which courses were to be developed. Eventually, two courses were conducted for staff in HIO’s financing department: (1) financial forecasting, for junior and mid-level staff from all branch offices (80 trainees) that took place over FY 2009–2010 (Oct. 2009–Sep. 2010) and (2) cost accounting for hospitals, from second to fourth quarters of FY 2010-2011 (Jan.–Sep. 2011) for staff from the Alexandria and Cairo branch offices (60 trainees). Overall, financial department managers reported that the concepts presented in both courses were too theoretical and not linked to information that decision-makers could use in improving financial controls or pricing of services. HIO’s claims management processes are based on a manual system of review of claims submitted by contracted hospitals and not guided by any formal policies or procedures. The objective of HS 20/20 Egypt’s assistance was to help it to move towards a more formal system which could eventually be automated for the handling of a much larger number of claims when the HIO would become the single “payer/purchaser” organization. However, HS 20/20 Egypt did not adequately assess training needs for this sub-component and did not take into consideration the claims management systems and assistance that had been previously provided by other actors, such as the Humana Corporation and McKinsey and Company in the Suez pilot15, or GOE and World Bank under the Health Insurance Systems Development Project loan. HS 20/20 Egypt appeared to have missed the mark on the claims management sub￾component of its HIO assistance, and in consequence, this sub-component of HS 20/20 Egypt was halted in the third quarter of 2010 at the request of HIO management. The medical-management sub-component of HS 20/20 Egypt was intended to strengthen HIO’s ability to improve the quality of care provided to beneficiaries, at both its own and contracted facilities, and to do so with a more efficient use of resources. Throughout the project period, HS 20/20 Egypt was fortunate to be able to identify and utilize a senior consultant with: (1) expertise in development of quality improvement tools, provision of training and support of knowledge transfer through on-site observation and mentoring and (2) extensive experience applying this expertise with health organizations and systems in countries in the Near and Middle East, including Egypt. This consultant first assessed HIO’s 15 The HSRP pilot in Suez was intended as a Governorate-wide pilot demonstration of a comprehensive Family Health service delivery model that would include management structures and procedures, a social health insurance financing mechanism, and linkages between primary health care, curative care, and private sector providers. _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 17 requirements regarding improvement of medical management through discussions with HIO senior management, and then continued a dialogue with management as new annual work plans were developed. The medical management activities supported the development of a core of HIO staff with competencies in: (1) medical audit, (2) utilization management and (3) case management. The medical management consultant developed three documents used as guides for training of HIO staff which remain references for the HIO.16 The consultant trained three cohorts in medical audit for hospitals (84 trainees), one cohort in medical audit for PHC units (23 trainees), three cohorts in hospital utilization review and case management (79 trainees), one cohort in case management of referrals from PHC units and on how to apply utilization review and case management principals within a case management office (CMO) in an HIO-owned or HIO-contracted hospitals (30 trainees). A focus group of selected trainees reported that the medical management training materials were complementary to others they may been exposed to in previous certification or degree programs in quality improvement or health management and highly relevant to the task of improving the quality of care and achieving cost efficiencies. Furthermore, they reported that the training approach was very effective at building understanding and sustainable competencies. The medical management sub-component of HS 20/20 Egypt was implemented through the creation of audit guides and a utilization review and case-management manual to the HIO, and also in conducting training and on-the-job coaching in use of these materials. One HIO KI indicated that the utilization￾review skills lowered HIO’s administrative costs through a reduction in the number of partially paid claims contested, when those claims have been prepared by staff with utilization-review training as compared to staff without such training. Unfortunately, the key performance indicator (KPI) data provided to the evaluation team are inadequately documented to persuasively and conclusively show that either the quality of care has improved and/or costs reduced as a consequence of placement of CMOs in HIO hospitals.17 However, the HIO continues to collect and analyze KPI data through the CMOs and it is therefore possible to conclude that the project did indeed help to build HIO’s capacity for continuous quality improvement. During the final year of the project, HS 20/20 Egypt developed and implemented plans intended to strengthen the sustainability of medical-management inputs. Specifically, by June 2011 the project had helped the HIO establish CMOs in four hospitals, staffed by HIO personnel who had received all medical management trainings. The CMO staff carried out case-management activities with departments with high-risk/high-cost cases.18 During the final project year, HS 20/20 Egypt also modified and added to HIO’s Key Performance Indicators (KPIs) so that they could be used by CMO staff to monitor quality and/or cost aspects of care to high risk/high cost cases by department by month. Finally, HS 20/20 Egypt 16 Schwark, Rafeh and Farag, December 2010a and Schwark, Rafeh and Farag, December 2010b 17 Unfortunately, the data and their presentation are inadequate and thus it is not possible to draw any conclusions from them about project impact. For example, the data presented (Schwark, T October 2011) do not: i) provide any indication of how departmental costs were measured and whether this was consistent across departments and across hospitals. Further the document does not discuss alternate reasons why costs may have declined in August 2011, e.g. delay of high cost cases during Ramadan, higher unemployment post-revolution and loss of HIO coverage, higher household sensitivity to HIO co-payments post Revolution and substitution of lower cost self-care. 18 The HIO established another 7 CMOs (5 in HIO-owned hospitals and 2 in HIO-contracted hospitals) later in the final project year. _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 18 provided a “Training of Trainers” course so that HIO could continue to expand, beyond the end of the project, the number of staff with medical-management knowledge and skills could continue to be expanded into the future. During this evaluation, about one year after close of HS 20/20 Egypt, the medical management systems strengthened through project assistance continue and HIO management stated its intention to further expand and strengthen these activities. Conclusions HS 20/20 Egypt’s objectives in assisting the HIO not only responded to the requests for assistance by HIO management but were highly relevant to the needs of the organization. However, in the areas of financial management training, HS 20/20 Egypt made insufficient effort during the design phase to: (1) obtain a detailed understanding of the HIO’s systems and training needs; (2) realistically assess the technical strengths of HS 20/20 Egypt’s staff and consultants and match these to the HIO’s expressed needs; and (3) make a realistic assessment of what HS 20/20 Egypt could sustainably accomplish with the HIO within a three-year time horizon. HS 20/20 Egypt was only somewhat efficient in their implementation approaches, especially in the area of accounting, costing and financial management. For instance, the project only recognized after technical assistance had begun in the areas of accounting, costing and financial management that the consultant for this area would need very strong proficiency in spoken Arabic. In addition, the efficiency (and effectiveness) of cost-accounting training might have been improved if the cost-accounting training and studies for the HIO had been conducted as part of the hospital-costing studies performed for the NHA under the MOHP. However, the development of training materials and provision of training/mentoring related to medical management was reported to be efficient, in that the training complemented rather than duplicated what trainees had learned from other sources. HS 20/20 Egypt project outputs in the area of accounting, costing and financial management did not produce any sustained outcomes in terms of manuals. On the other hand, HS 20/20 Egypt’s medical management outputs continue to be available to, and utilized by, the HIO. In fact, the trained staff has expanded medical management activities to new facilities and plans to conduct medical management training courses for HIO staff. Despite the fact that there are challenges to the sustainability of the medical￾management inputs, the HIO quality department expressed its commitment to sustaining the technical capacity and practice built during HS 20/20 Egypt. Thus, HS 20/20 Egypt’s assistance to the HIO can be considered partially sustainable, overall. TRAINING AND CAPACITY BUILDING (TCB) Background The fourth technical objective/result of the HS 20/20 Egypt project concerned with training and capacity building began as a focused program to establish a Leadership Academy. Dialogue regarding formation of _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 19 the Academy actually predated the project, and was subsequently folded into the SOW of the HS 20/20 Egypt project. The MOHP envisioned that the academy would be “[a] sustainable Egyptian institution that builds leadership and management capacities to support the implementation of reforms.”19 HS 20/20 Egypt hired a specialist in early 2009 to oversee development of the institutional framework for the academy, and the academy ran its first Change Management Course in the summer of 2009. At the same time, however, the strategic approach to establishing and sustaining the academy came under review at MOHP policy levels. While the Leadership Academy’s initial year of operation appeared to have been reasonably productive, a decision was made by the MOHP—likely for a variety of reasons—to not proceed with development as originally envisioned, and the activity was dropped from the HS 20/20 Egypt SOW in August 2009. Table 3 Summary of HS 20/20 Egypt Training Activities 19 2008 Technical Assistance Plan, page 4. TECHNICAL INTERVENTION AREAS NUMBER OF SESSIONS / TRAINEES, YEAR 1 AND 2 NATIONAL HEALTH ACCOUNTS DOP staff – basic costing and NHA preparations 14 OJT sessions MOHP decision makers – orientation to NHA and how to interpret NHA findings 139 trainees OJT Sessions on Expenditure Tracking System (ETS) for central and peripheral implementers of ETS methodology 13 Governorate level sessions WORKFORCE PLANNING “Basics of Strategic Resource Management 59 trainees Workforce planning methods for Governorate teams 13 Governorate level sessions HEALTH INSURANCE ORGANIZATION Medical audit training for physicians, nurses, pharmacists and dentists in hospitals 84 trainees Medical audit training for PHC Units 23 trainees Utilization Management and Case Management in hospitals` 152 trainees Utilization Management and Case Management in PHC Units 32 trainees Establishment of Case Management Offices (CMOs) 9 hospitals Overview of Medical Audits and Utilization and Case Management for Hospital Mangers 40 trainees _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 20 There is evidence in HS 20/20 Egypt project documents that the project made a conscious and strategic effort to redirect its attention and approaches to training and capacity building. This was accomplished in two ways: (1) the project made renewed plans to emphasize training and capacity building within each of its technical intervention areas; and (2) the project increased its efforts to strategically link with other, related training programs supported by USAID. In terms of training and capacity building linkages, the project worked to maximize working linkages and coordination with those GOE counterparts that were graduates of USAID-sponsored programs, such as the FORECAST MBA Program and the Harvard Executive Program. For some period of time, the HS 20/20 Egypt’s designated counterpart in the MOHP was a graduate of the MBA program, and through this avenue, there were opportunities for informal linkages with the network of returning MBA graduates. In addition, HS 20/20 Egypt assisted the Harvard Program in organizing and scheduling executive business education courses delivered by the Harvard School of Public Health to MOHP emerging leaders. The program trained three different groups of approximately sixty leaders over two consecutive years. Findings Several of the training programs listed in Table 3 were notably successful, but others could have been improved. The training for WFP appears to have been well-received, and Key Informants at MOHP headquarters as well as in the Governorates expressed general satisfaction both with the training and the planning model and methods that was introduced. Similarly, in the area of medical management, feedback from Key Informants regarding the HS 20/20 Egypt training was positive. In both cases, managers stated their intention to continue and expand use of the systems in which they received training. Feedback from three focus group discussions (FGDs) on the WFP and medical management training activities was also positive. Participants in the FGDs generally felt that the training addressed actual needs and skills that could be used on the job. Participants felt that the training helped them improve their performance. However, there was one notable exception, the HIO financial management training. As discussed with Key Informants and reinforced by FGD participants, the cost accounting concepts presented in the didactic sessions were not very useful because they were (i) “divorced” from the actual business processes that the department followed; (ii) the training examples were paper-based and did not proceed past costing of a single hospital service (e.g. X-rays); and (iii) the training did not utilize any of the existing software programs developed for cost accounting (including of hospitals). Further, HIO staff reported that cost accounting studies started at Nasr City and 6th October Hospitals had not been completed (even by the date of this final evaluation) in part due to the lack of continued technical assistance. In the area of NHA training, it was felt by HS 20/20 Egypt staff that many of the DOP participants selected for training did not have the entry level skills and qualifications necessary for NHA. However, some progress was made as a result of the comprehensive training efforts of HS 20/20 Egypt. As part of the efforts to strengthen the DOP staff’s technical capacity and to institutionalize NHA, HS 20/20 Egypt conducted 14 on-the-job training sessions on hospital costing. Also, HS 20/20 Egypt conducted multiple on-the-job training sessions and is collaborating closely with DOP staff in all steps of methodology development, data collection and data analysis. The DOP staff is now better prepared to perform such tasks on its own. _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 21 Conclusion Consistently, in various KI interviews as well as in the FGDs, the need for human resource development in the areas of health leadership and strategic planning figured prominently. Although the institutional framework for this area of training and capacity building was evidently inappropriate for USAID support at the time, a critical need still exists. In the future, in addition to building specific training and capacity￾building activities into technical interventions, as was done ultimately under HS 20/20 Egypt, more global and coordinated approaches might be explored that would harness existing, local training resources and enrich the local capacities with high quality, external resources. OTHER STRATEGIC INTERVENTIONS The Benchmark CCT Program The cash transfer mechanism has previously been used by USAID in Egypt to promote health-sector policy change. During 1998–2002, USAID used a combination of approximately $25 million in technical assistance and $60 million in CCTs to fund systems developments as well as infrastructure (clinic rehabilitation) to assist the MOHP in developing and piloting the family health primary health care service delivery model. The cash transfer mechanism requires that a program or one or more policy and/or systems-change objectives are developed and agreed upon between USAID and the recipient line ministry —in this case, the MOHP—and bilateral agreement is then reached between USAID, MOHP and the Ministry of International Cooperation (MOIC) on the conditions—that is, achievement of specific benchmarks and timeframe—that will trigger payments in large tranches (usually $30 million dollars or more) to the Ministry of Finance for sector￾specific budget support. The funds are owned by the GOE and earmarked for the sector, but there is no guarantee that the Ministry of Finance will redirect these resources to the specific purposes of the program that “earned” them. However, in the earlier cash transfer experience during 1998–2002, the MOHP and the family health reform program did, in fact, directly benefit from the cash transfers.20 In 2007, USAID initiated a dialogue with the MOHP about a new program of cash transfer financing that would entail approximately $110 million over a specified period, with adjunct technical assistance funded 20 Purvis. G., and Reyes, P. “Egypt Health Policy Support Program (HPSP) Assessment”, USAID, 2001. Original Benchmarks for CCT in Egypt 1. Five hospitals accredited, based on quality indicators 2. The FHM operational and financially sustainable in three governorates 3. Performance-based budgeting system functional in 10 MOHP hospitals 4. Ten MOHP hospitals collate clinical and financial data in a standardized method to become centrally reported 5. The MOHP establishes an operational financing system in place that supports FP, including a protected line item for contraceptive procurement 6. Hiring standards established and published for primary healthcare service providers 7. Executive Order declares policy to cease guaranteed hiring of new medical school graduates into the MOHP system 8. The basic benefits package included in the MOHP’s new insurance system functioning in two governorates _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 22 by USAID to facilitate documentation and verification of achievement of benchmarks. When HS 20/20 Egypt began operations in February 2008, basic agreements had already been reached with the MOHP and MOIC on the terms and conditions, and responsibility for developing a verification plan and periodically conducting verification exercises was built into the HS 20/20 Egypt scope of work. It should be emphasized that HS 20/20 Egypt’s role was limited to (1) developing a verification plan to be used by an external examiner to periodically determine the extent to which the MOHP had met benchmarks; and (2) to prepare an assessment for USAID describing the capacity/ability of the MOHP to meet benchmarks. HS 20/20 Egypt was specifically not responsible for carrying out or assisting the MOHP in carrying out the work required to meeting benchmarks21 . Early on, as reflected in project reports, it became clear to the HS 20/20 Egypt team that capacity within the MOHP to reach the benchmarks, in addition to the political acceptability of the policy benchmarks, was much lower than USAID had anticipated. In late 2009, the MOHP (without the MOIC) countered the verification methods under discussion with HS 20/20 Egypt, and instead proposed alternative parameters, leading to an impasse between the MOHP and the Mission.22 USAID brought in a respected external consultant (who also worked on the NHA studies and other project elements) to bring together the Mission, MOHP and the MOIC to discuss the feasibility, content and timeline of the benchmarks. The main roadblock seemed to be a hard requirement for verifiable evidence. The Mission and MOIC argued that verifiable evidence was standard procedure for CCTs, but the MOHP disagreed. In the next quarter (April–June 2010), the external consultant brought in by USAID to help resolve the differences submitted a confidential status update on the benchmarks, and by the end of 2010, the MOHP rescinded the benchmarks as part of the HS20/20 Egypt project. KIs to this evaluation who had been involved in the defunct CCT confirmed that CCT as a financing mechanism is still an acceptable form of support, and that this mechanism has value in being able to focus commitment on specific targets, even during political turbulence. They also affirmed that getting disbursements from the Ministry of Finance (to the MOHP) after achievement of the benchmarks is not a problem. However, the major finding from the GOE side (as described by key informants) was that what was vaguely documented in project reports as a “failure of the GOE and the Mission to agree on validation measures” was actually a fundamental disagreement between the parties on the policy benchmarks themselves, with MOHP policy makers feeling that several of the policies being advocated through the CCT were politically unpalatable. Other Special Studies There were several activities added in to the HS 20/20 Egypt project over its four years; the special studies added to the list made their first appearance in in the first quarter of Year 3, where a report documented that in 2009 the Minister of Health requested the Project to develop a case study on Egypt’s health sector reform program; in 2009 a sustainability assessment of USAID’s investment in the Preventative Care Sector was also added. 21 USAID MAARD # 2, dated December, 2009. 22 As stated in the January–March 2010 Quarterly Report _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 23 Case Studies on the Egyptian Reform Experience: This was originally to be a single case study by an international consultant on Egypt’s health sector reform program, but six studies were eventually slated. The studies were to be modeled on the Harvard style of teaching cases, and Harvard’s director of the International Health Systems Program was brought in to work with project staff; it was after his initial visit with the Minister in February 2010 that multiple cases were identified: 1. Implementation of Health Reform in Egypt 2. Upgrading Ambulance Services in Egypt 3. Upgrading Nursing Education Programs in Egypt: Consolidation and Strengthening Existing Services 4. Piloting Insurance and Service Reform: The Suez Pilot in Egypt 5. Evolution of the Family Health Model 6. Capacity Building in Health Sector Management All six studies were intended to “emphasize the important issues of how to implement reforms—the technical and political issues and strategies that were adopted and the results that have so far been documented;23 publication in scientific journals was also considered. The cases themselves would serve two programs: the Harvard/WB Executive Course on health system strengthening and the Leadership Academy course slated for 2010 at the National Training Institute under HS 20/20 Egypt. In an effort to build local capacity, HS 20/20 Egypt’s external consultant was assigned a local counterpart and work began in August 2010. A subsequent visit did not take place due to the revolution in January 2011, and the activity was handed over to the local consultant for completion. Study reports were delivered to the MOHP in November 2011. Of note, no informants from either the Mission or MOHP staff were found who could point to use of the studies in strategic planning, policy formation or other purposes; neither was any specific use found in project documents. In addition, it was unclear if the case studies were used in either the Harvard/WB’s Executive Course or the Leadership Academy; there was also no evidence found that these cases were used in any MOHP strategic planning regarding the reform process. Sustainability Study: The Preventive Sector Assessment: In July 2009 (Year 2 of the project), this study was added in response to the then-impending conclusion of thirty years of USAID support to the Egyptian MOHP slated for September 20, 2009.24 The purpose of the assessment study was to review the sustainability and institutionalization of USAID’s investment in the health sector and provide any recommendations for future technical assistance to Egypt. In consultation with senior MOHP officials in September of 2009, the focus was narrowed to three sectors: preventive sector programs and FP, MCH, and IC in three governorates. The framework pillars of the study were: impact, effectiveness, efficiency, equity, and sustainability. The study had four objectives: 1. Understand how well the program structures and processes have worked to achieve program objectives (overall, not only attributable to USAID contributions); 2. Assess (Preventive Care service) program needs going forward; 23 HS20/20 Egypt March 2010 Quarterly Report 24 HS20/20 2009 Technical Assistance Plan _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 24 3. Use the analyses to make proper recommendations on how to assure long-term sustainability of these programs after the end (USAID assistance); and 4. Promote (MOHP) staff development by their involvement in conducting the assessment study. The study also intended to assess the impact of the family health model on these three preventive care programs. Data sources were a mix of two primary and two secondary sources: primary data came from KII with stakeholders both within and outside the MOHP and exit interviews with service clients across 63 facilities in six governorates; secondary data was pulled from the DHS on health outcomes and the ETS on spending patterns. The study was to be conducted over Years 2 and 3 (2009/10–2010/11) by an external consultant with principle counterparts from the three sectors and a team of researchers provided by the MOHP. Design and data collection occurred over the entire year of 2010; a preliminary draft was delivered in Q1 of 2011, and a full draft delivered to USAID in June. USAID requested a more comprehensive report, which was produced and published in June 2012. The project succeeded in engaging the MOHP at various levels in the assessment. Methodology, use of program logic models for the assessment, and input on instrument design and data collection approaches were finalized in consultation with the MOHP over several months in early 2010. According to project progress reports, most facility managers and governorate officials involved in the assessment pointed out that this was the first time such information (client perceptions and links to health outcome and financial information) was available to them, and said it would have a “tremendous value in informing their internal supervision and support policies.”25 The fact that this assessment and the way it was approached generated involvement and interest among health managers should be noted. However, this experience offers a suggestion that similar assessments could serve as part of a strategy for improving the evidenced￾based strategic planning culture of the health sector. Continuation of such assessments, properly organized and demand-driven, might be an appropriate function for the HEU to undertake. PROJECT MONITORING AND EVALUATION SYSTEMS In USAID’s Automated Directives System (ADS), Series 200, agency guidance is given to USAID/Washington and Missions on standards for performance management. In addition to development of PMPs, the guidance spells out approaches for developing project M&E plans with specifics on results, frameworks, project design and implementation, evaluation and monitoring, and learning and adapting.26 Guidance regarding development of project M&E plans is also normally passed on to centrally awarded USAID projects (such as the Global HS 20/20 Project) and specified in the award agreements. At the country level, in addition to the country-specific project (such as HS 20/20 Egypt) being responsive to requirements from its headquarters to feed into the projects overall project M&E plans and procedures, the country-specific project should also apply these best practices in performance management to its own in-country assistance program. 25 HS20/20 October 2010–December 2010 Quarterly Report 26 See ADS series 200.3.5.3 – 200.3.5.6. _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 25 Findings From the beginning of the project, the importance of developing a project M&E plan and framework was apparently not recognized as a high priority by HS 20/20 in Egypt or at its headquarters. Instead, a plethora of different types of planning and status reporting procedures were adopted. These included: TAPs, which were detailed strategic planning documents, probably prepared with input from the client organization (i.e., MOHP or HIO); annual work plans; project M&E plans that tracked local indicators at the input level against results specified for the global HS 20/20 project; and quarterly M&E reports. According to HS 20/20 Egypt staff, these plans and progress reports were used for internal management purposes, but were never shared with MOHP or HIO counterparts. The various monitoring and evaluation tools used in HS 20/20 Egypt may have facilitated internal project management, but did not involve the client organizations (MOHP and HIO), or USAID/Egypt. USAID/Egypt eventually requested HS 20/20 Egypt to submit a county-specific project M&E plan (complete with indicators), which was submitted late in the project in June 2010, two years into the project. The M&E indicators consisted only of output and progress indicators27 and were never fully implemented as an M&E system. In general, output and progress indicators such as those found in the Egypt M&E Plan cannot measure systems strengthening interventions. Both quantitative and qualitative outcome indicators as well as efficiency and effectiveness indicators are needed to capture health-systems strengthening programs. Of the 21 indicators of the M&E Indicator Report, only three indicators were outcome indicators for training and capacity building. In one of the work plans, it is stated that the set of M&E indicators would be complemented by a set of qualitative deliverables. In fact, the quarterly progress reports included statements about the completion of deliverables, which were also mostly outputs. In addition, the reports made statements about project activities and achievements, especially those performed by the counterpart as a result of increased capacity. Therefore, many more outcome indicators capturing systems strengthening could have been developed based on what was reported to have been achieved by the project. These achievements should have been envisioned from the outset of the project and developed into planned outcome indicators, with specific targets, and included in an initial project M&E plan, which should have been developed and submitted with the first work plan and revised subsequently to address changes in planned activities. Annex H of this report includes examples of outcome indicators that might have been devised at the outset of the project. Coordination with the MOHP on M&E was rather weak and there were no strategically planned capacity￾building activities in M&E. The project did not share the quarterly reports or the M&E Indicator Reports with counterparts as there was no guidance or directive to do so. However, counterparts at the middle￾management level reported that they already knew about the progress, since activities were implemented with their direct involvement. 27 The HS 20/20 Egypt M&E Plan / M&E Indicator Report shows the Egypt specific indicators and explains the activities that they measure. However, the report does not specify the type of indicator, whether it is measuring output, outcome, effectiveness or efficiency, and the indicators are presented within the framework of the global HS 20/20 Project (not a county-specific framework). _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 26 At the HIO, capacity building in M&E occurred in the area of utilization management and the use of KPIs. Before this project, HIO staff routinely collected KPI data at the hospital level and reported them to the central levels of HIO. However, this KPI data often was not presented or utilized and the data quality was questionable. HS 20/20 Egypt, together with counterparts, selected a few existing KPIs and added more to monitor utilization and case management. As part of the Utilization and Case Management Training activity in HIO, HS 20/20 Egypt increased the agency’s capacity in monitoring of utilization￾management KPIs. Staff of the CMOs of nine HIO-owned and contracted hospitals were trained in collecting, calculating, interpreting and comparing KPIs at departmental and hospital levels. Currently, these nine hospitals continue to monitor their KPIs at the hospital and department levels to make decisions to improve performance and to reduce waste. KPIs are also regularly presented to the chief executive officer of the HIO to monitor progress. Data collection for KPIs is more accurate now in these nine hospitals. Because staff knows that medical records and KPI data are being utilized and presented, they are more careful in filling medical records and in calculating indicators. Hospital staff members are conscious that they are being monitored, and thus are more likely to perform better. Furthermore, the HIO is satisfied with the level of competence of staff of the nine CMOs in calculating and interpreting KPIs as a management-monitoring tool. Conclusion The lack of a full strategic-planning process during the design phase, specifically the collaborative development of a country-specific results framework and an outcome-based project M&E plan, compromised HS 20/20’s ability to focus and direct its assistance. Neither the central HS 20/20 project nor USAID project management required Egypt HS 20/20 Egypt to state in its initial work plan and develop a results framework and project M&E plan in collaboration and agreement with counterparts, although this would have helped the project and counterparts reach consensus about project objectives and expected results, as well as to develop successful strategies for implementation. The project did not share work plans or progress reports at an appropriately senior level or hold periodic project review meetings that would include senior counterparts. The HS 20/20 Egypt project design did not include establishing an M&E system within MOH and HIO. HS 20/20 Egypt initially lacked the in￾house M&E expertise needed to produce a robust M&E system. This deficiency was an oversight on the part of both HS 20/20 headquarters and USAID/Egypt. Based on the project’s subsequent attempts in the project to be responsive to USAID guidance, it is possible that the project’s M&E systems could have been improved with additional prompting. _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 27 V. MOVING FORWARD AND RECOMMENDATIONS FOR ACTION USAID has assisted the MOHP and HIO in strengthening targeted health sector strategic planning and management systems through HS 20/20 Egypt and predecessor projects for nearly two decades. While some of these efforts have resulted in increased human resource capacity, laid groundwork for future development, and contributed to incrementally strengthening the strategic planning and management capacities of parts of the sector, HS 20/20 Egypt had limited success in strengthening the institutional framework for these systems. In looking ahead, USAID/Egypt needs to shift its focus, placing less emphasis on introducing innovative and progressive technology, methods, and tools, and more on creating an enabling environment to facilitate the institutionalization and sustainability of systems technologies and their use. This means substantially increased emphasis on human resource development, and capacity building in development and use of evidenced-based strategic planning and management systems and tools (such as NHA, and medical management) -- areas where USAID has a long history of assistance and a well-established competitive advantage. There are a number of immediate management challenges that USAID/Egypt should address:  Limit the use of external consultants and require projects to use innovative strategies and approaches to work directly with Ministry staff to carry out work. This will require far more emphasis on training and capacity transfer. It may slow the pace of activities and results, but may also improve engagement, ownership and institutionalization of interventions.  Any future technical assistance activities should focus on leadership development. This is particularly important in reaction to the high turnover of staff at the MOHP in the aftermath of the Egyptian revolution. Additional studies and assessments are necessary to move beyond the defunct Leadership Academy concept and the compartmentalized training and capacity building approaches of HS 20/20 Egypt to a more coordinated approach.  Any future health systems strengthening activity that would be based on the HS 20/20 Egypt experience needs to institute a robust theory of change, a logical framework and a monitoring and evaluation plan at the design stage. Particularly in complex health systems strengthening activities, the lack of a cohesive logical framework and use of performance management tools will not only hamper USAID/Egypt’s ability to manage this type of programs, but will also make it difficult for the implementing partner to communicate with stakeholders and counterparts, in particular the MOHP. Given the dynamic state of Egypt’s administration, the effects of not having clear project performance monitoring processes in place may be further amplified. Contextual issues such as a large and organizationally complex MOHP, high staff turnover, major power shifts, and general tension and unpredictability are also challenging environments in which to work, and difficult for project managers to predict and manage. Tools such as collaborative Results Frameworks, regular exchange of data and strategy documents and stronger communication mechanisms need to be in place to help mitigate these organizational dynamics. _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 28 Moreover, based on the findings and conclusions presented in this report, the best performing interventions and those with most potential to show a return on future USAID investments in the medium to long term are: 1. National Health Accounts. USAID should continue to invest in “hands-on” production of NHA estimates with heavy emphasis on strategically increasing GOE/MOHP ownership and institutionalization. Emphasis should also be placed on training and capacity building to improve the human resources for production, demand-creation, and institutionalization of NHA estimates. USAID should revisit the possibility of further assistance in strengthening the HEU, but should not invest and become directly involved in attempting to help the MOHP resolve the underlying structural and organizational issues related to HEU. Any request for assistance from the MOHP to work with the HEU should be viewed through the prism of whether or not such assistance would serve the MOHP’s long term institutionalization objectives. As a further word of caution, the outlook for the HEU appears to be sufficiently complex and uncertain at this time, that institutional change in these areas would probably not represent the “low hanging fruit” that might be sought as a benchmark candidate for any future conditional cash transfer program. 2. Strategic studies and assessments. USAID should continue to invest in policy and program￾informing studies and assessments, such as (but not limited to) NHA-related costing research methods. USAID assistance might include support for production/utilization of studies as well as organizational/human resource development. As a word of caution, although USAID invested in good quality strategic studies and assessments under HS 20/20 Egypt, there appeared to be little demand or interests in these products. USAID will need to ensure an efficient way of identifying and effectively engaging Egyptian health policy makers and strategic planners in dissemination of results. 3. Medical management systems. Although HIO is in the midst of major organizational and functional changes, it has been a productive implementing partner and systems development “laboratory” for HS 20/20 Egypt interventions. USAID should continue to invest in HIO (along with HIO successor institutions) in the area of medical management quality improvement systems including human resource and systems prototype developments (such areas as utilization management, case management, and Key Performance Indicator systems). It is important to note that any future USAID investment should be expanded to include both the curative care and preventive/primary care sectors. This must be considered in the interest of sustaining USAID’s past and current PHC investments, and in the interest of unifying and further strengthening medical management systems in the health sector. 4. Workforce Planning. The need for strategic workforce planning capacity (and a national strategic workforce plan) remains valid, and there is evidence of interest and demand at decentralized levels for WFP. However, assessment and design of any future workforce planning intervention needs to better understand approaches and technical support requirements. Such an assessment should include: a review of the organizational landscape; level of demand for decentralized and centralized workforce planning systems; a systems model and methodology that would best address MOHP strategic workforce planning needs; and identification of an approach to technical assistance that would be responsive to, and engage relevant actors within the MOHP and GOE. _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 29 The foregoing short term approaches should help facilitate the institutionalization of three major competencies/capabilities that USAID/Egypt should seek in the medium to long run: (1) Strategic planning is the last mile in achieving institutionalization of the Egypt HS 20/20 investments and in ensuring that quality data is produced, analyzed, brought to scale, and used to induce evidence-based decision making. USAID has a wealth of internal tools to draw from in this area and should include/ link these efforts closely to the Leadership and Management capacity building efforts, as the strategic planning skills taught there can be instrumental in shifting the organizational culture of the MOHP. In particular, these strategic planning skills can help fostering the enabling environment needed to reward transparency and efficiency and to allow such changes to thrive. Specific focus areas should include data utilization and evidence-based planning and coordination mechanisms to facilitate the collaboration that will be vital to realizing the HSRP agenda. These analytical skills will need to be coupled with a political atmosphere conducive to such leadership that will be only somewhat within the manageable interest of the Mission. Capturing progress through a solid M&E system, and potentially (in the long term) reverting back to a benchmark modality envisioned by USAID in 2007, may be important tools to manage the volatility of the political climate in Egypt. (2) Strategic communications. Although HS 20/20 Egypt’s technical assistance plans did not specifically identify communications, technical collaboration and strategic integration as a targeted activity within the project, there was evidence that technical cooperation and promotion of health systems strengthening technologies and tools were inherent to HS 20/20 Egypt approaches. Thus, the project provided several examples of effective technical information sharing and collaboration and which should be invested in further. Firstly, to continue to improve the role it plays in technical collaboration and strategic integration of USAID programs, USAID might consider building directly into future projects a requirement for the project to develop a technical information sharing or “strategic integration” plan. Secondly, any such plan should include activities that not only involve the client organization (such as the MOHP or HIO), but facilitate the client organization taking the lead (which was done quite effectively with HS 20/20 Egypt’s Sharm El Sheikh Conference on NHA). (3) Strengthening Relationships. While much time was given to validation processes and a wide array of stakeholders were engaged, often the correct stakeholders—key decision makers—were not fully engaged, compromising the utilization of project products and severely restricting institutionalization efforts. Contextual issues such as a large and organizationally complex MOHP, high staff turnover, major power shifts, and unpredictability are also challenges to working with the MOHP in Egypt, and difficult for project managers to predict and manage. Tools such as collaborative Results Frameworks, regular exchange of data and strategy documents and stronger communication mechanisms can be put in place to mitigate this volatility. _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 30 ANNEX A: EVALUATION STATEMENT OF WORK USAID/Egypt Statement of Work: “Evaluation of Health Systems 20/20” A. Purpose The USAID/Egypt Mission intends to conduct an evaluation of its health-systems-strengthening project assisting the Egyptian Ministry of Health and Population. This evaluation is meant to serve a dual purpose: (1) to learn to what extent the project’s objectives and goals contributed to the achievement of the intended results; and (2) to determine whether moving forward with a new health-systems￾strengthening project is a good investment of USAID funds and, if so, identify those areas that have the most investment potential to inform the design of a new health systems strengthening project. This USAID activity ended in April 2012. This evaluation will assist the Mission in reaching decisions related to: (1) defining what next steps need to be taken to maximize the investments made in health￾systems strengthening; (2) defining what areas of health-systems strengthening within the Ministry of Health and Population are within the manageable capacity of USAID to effect positive, lasting change; and, (3) defining the enabling factors that have to be in place to ensure that USAID’s future investment is effective. Technical recommendations will serve as the basis for the project description to be developed for ongoing health-systems-strengthening activities. B. Audience and Intended Uses The audience of the evaluation report will be the USAID/Egypt Mission, specifically the health team, the Middle East Bureau, and the future implementing partner of health-systems-strengthening activities. The executive summary, expanded executive summary final report, and recommendations (see IV. A. Deliverables) will be provided to the MOHP, the HIO, and other donors in Egypt working on healthy systems strengthening, and the general public via the Development Education Clearinghouse. USAID will integrate the report recommendations to future health-systems-strengthening activities and share lessons learned with other stakeholders; Abt Associates and its subcontractors will learn about their strengths and weaknesses; and the MOHP and HIO will learn more on how to better benefit from implementing partner technical assistance. It is expected that MOHP and HIO counterparts will have the opportunity to discuss how the HS 20/20 project assisted them and how this type of project could better assist them in the future to meet its goals. C. Evaluation Questions The evaluation questions have been organized into three areas: looking back, future activities and monitoring and evaluation. The organization of the questions was deliberate in that USAID is most interested in exploring what worked/didn’t work with the HS 20/20 project in terms of implementation and relationships with counterparts, how USAID can maximize the lessons learned for future investments _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 31 in health-systems strengthening and what USAID can do to design a project to that both captures and sustains monitoring and evaluation efforts. The evaluation will answer the following illustrative questions: Looking back 1) To what extent was HS 20/20 able to make changes/improvements in the health strengthening areas in the project design and supported by the project? a) To what extent did HS 20/20 achieve what was stipulated in the program description—if so, how, and to what extent; if not, why? b) What project interventions are sustainable and being successfully carried out by the government of Egypt as a result of HS 20/20? 2) What effect did HS 20/20’s implementation strategy have on relationships with government of Egypt counterparts? a) How is the project perceived/valued by Government of Egypt counterparts? 3) To what extent did the project achieve sector integration through partnerships with other cooperating agencies, international donors, the public sector, and the private sector? Future Activities 1) What HS 20/20 project interventions have the most potential to show a return on investment for health-systems strengthening? a. What are the next steps that USAID should take to maximize our investments in these? b. What health-systems-strengthening interventions are within the manageable capacity/interests of USAID that we can contribute to? c. What are the enabling factors and critical assumptions that have to be in place to make sure the programs are effective? 2) What can USAID do in the future to maximize sector integration through partnerships with other cooperating agencies, international donors, the public sector, and the private sector? Monitoring and Evaluation 1) How did the monitoring and evaluation system capture the systems-strengthening changes? a) How was the information used for program management and improvement? 2) To what extent does the MOHP and the HIO have and sustain monitoring and evaluation capabilities in the project intervention areas? I. Evaluation Design and Methodology A. Evaluation Design _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 32 This is a performance evaluation and is intended to focus on how HS 20/20 has been implemented, what it has achieved, whether expected results have occurred according to the project’s design, whether the project was cost effective, and how it is perceived, valued, and sustained. Evaluators will use a mix of quantitative and qualitative data collection and analysis methods to generate answers. B. Data Collection Methods The Evaluation Team should develop data collection tools that are consistent with the evaluation questions to ensure high quality analysis. The Evaluation Team is required to share data collection tools with the USAID Evaluation Program Manager for review, feedback and/or discussion with sufficient time for USAID’s review before they are applied in the field. The international evaluation team will start work on a paper review of all the documents cited in the “Existing Information” section above prior to arriving in Egypt. The local evaluation team members should complete the paper review prior to the international team’s arrival. The data collection methodology will be comprised of a mix of tools appropriate to the evaluation’s research questions. These tools may include a combination of the following:  Review HS 20/20 documentation (e.g., mid-term evaluation; quarterly reports, deliverables, output form the project monitoring system);  Cost-benefit or return on investment analysis, as appropriate;  One-on-one interviews, FGDs with HS 20/20, MOHP, HIO, training beneficiaries, and other counterparts and stakeholders; and,  Review health-systems-strengthening constraints identified by USAID/Egypt colleagues, MOHP, HIO and other sources (e.g., WB). Interviews Key Informant Interviews will include, but may not be limited to:  USAID/Egypt Health Team – including Activity Manager  MOHP staff  HIO staff  HS 20/20 staff, including prime partner, other partners and sub-contractors  Participants of HS 20/20 training/supervision programs  Staff of international donor partners involved in HS 20/20 health-systems-strengthening  activities  Others Other The evaluation team may implement direct observation of HS 20/20 activities that are ongoing, if feasible (Workforce Planning, HIO). C. Data Analysis Methods _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 33 Prior to the start of data collection, the evaluation team will develop and present, for USAID/Egypt review and approval, a data analysis plan that details how focus groups and key informant interviews will be transcribed and analyzed; what procedures will be used to analyze qualitative and quantitative data from key informant and other stakeholder interviews; and how the evaluation will weigh and integrate qualitative data from these sources with quantitative data from indicators and project performing monitoring records to reach conclusions about the effectiveness and efficiency of the health-systems￾strengthening activities conducted by HS 20/20. The Mission expects the evaluation team to present strong quantitative and qualitative analysis, within data limitations, that clearly addresses key issues found in the research questions. The Mission is looking for new, creative suggestions regarding this evaluation, and it is anticipated that the implementer will provide a more detailed explanation of the proposed methodology for carrying out the work. The evaluators should consider a range of possible methods and approaches for collecting and analyzing the information required to assess the evaluation objectives. The methodology will be discussed with and approved by USAID/Egypt Activity Manager and USAID/Washington Agreement Officer’s Technical Representative (AOTR) prior to implementation. D. Methodological Strengths and Limitations Key informant interviews and review of key deliverables are suggested as a primary data source for this evaluation. Given the short timeline for this study, the evaluation team may not be able to cross-check key informant characterizations of changes in MOHP systems and competencies through direct beneficiary interviews or observation. Further, in Egypt it is anticipated that some interviews may be conducted through translators by the international team required for this evaluation. As a result, some differences in language could enter the interview process and interview notes taken and analyzed by the evaluators in Egypt may not capture the full intent or meaning offered by the key informants. It is anticipated that some interviews may be conducted in the presence of at least one or more outside observers, including project and USAID staff, and that interview responses could be affected by the presence of these observers. USAID expects that all threats to validity be discussed and documented in the evaluation planning stage – including what will be done to minimize threats to validity, notified all team members and USAID team in the implementation phase and detailed in the final report. II. Evaluation Products A. Deliverables Work Plan: During the team planning meeting the evaluation team will prepare a detailed work plan, which will include the methodologies to be used in the evaluation, timeline, budget and detailed Gantt chart. The work plan will be submitted to both the HS 20/20 AOTR in Washington and the USAID Evaluation Program Manager for approval no later than the sixth day of work. _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 34 Methodology Plan: A written methodology and data analysis plan (evaluation design, data analysis steps and detail, operational work plan, see sections III. C and D) will be prepared during the team planning meeting and discussed with USAID prior to implementation. List of Interviewees and Schedule: USAID will provide the Evaluation Team with a stakeholder analysis that includes an initial list of interviewees, from which the Evaluation Team can work to create a more comprehensive list. Prior to starting data collection, the Evaluation Team will provide USAID with a list of interviewees and a schedule for conducting the interviews. The Evaluation Team will continue to share updated lists of interviewees and schedules as meetings/interviews take place and informants are added to/deleted from the schedule. Data collection tools: Prior to starting fieldwork, the Evaluation Team will share the data collection tools with the USAID Evaluation Program Manager for review, feedback and/or discussion and approval. In-briefing and Mid-term brief with USAID: The partner is expected to schedule and facilitate an in￾briefing and mid-term briefing with USAID. At the in-brief, the partner should have the list of interviewees and schedule prepared, along with the detailed Gantt chart that maps out the evaluation through the report drafting, feedback and final submission periods. At the mid-term brief, the partner should provide USAID with a comprehensive status update on progress, challenges, and changes in scheduling/timeline. Discussion of Preliminary Draft Evaluation Report: The team will submit a rough draft of the report to the USAID Evaluation Program Manager, who will provide preliminary comments prior to final Mission debriefing. This will facilitate preparation of a more final draft report that will be left with the Mission upon the evaluation team’s departure. Debriefing with USAID: The team will present the major findings of the evaluation to USAID/Egypt through a PowerPoint presentation after submission of the draft report and before the team’s departure from country. The debriefing will include a discussion of achievements and issues as well as any recommendations the team has for possible modifications to project design approaches, results, or activities. The team will consider USAID/Egypt comments and revise the draft report accordingly, as appropriate. Debriefing with Partners: The team will present the major finding of the evaluation to USAID partners (as appropriate and as defined by USAID) through a PowerPoint presentation prior to the team’s departure from country. The debriefing will include a discussion of achievements and activities only, with no recommendations for possible modifications to project approaches, results, or activities. The team will consider partner comments and revise the draft report accordingly, as appropriate. Draft Evaluation Report: A draft report of the findings and recommendations should be submitted to the USAID Evaluation Program Manager prior to the team leader’s departure from Egypt. The written report should clearly describe findings, conclusions, and recommendations. USAID will provide comment on the draft report within two weeks of submission. Final Report: The team will submit a final report that incorporates the team responses to Mission comments and suggestions no later than five days after USAID/Egypt provides written comments on the team’s draft evaluation report (see above). The format will include an executive summary, table of _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 35 contents, methodology, findings, and actionable recommendations. The report will be submitted in English, electronically. The report will be disseminated within USAID and to stakeholders according to the dissemination plan developed by USAID. Expanded Executive Summary: The team will submit an expanded executive summary to accompany the final report that will include a background summary on the evaluation purpose and methodology, and an overview of the main data points, findings, conclusions, and recommendations. The expanded executive summary should be easy to read for wide distribution to local audiences and the partner is encouraged to look for creative presentation styles, formatting and means of dissemination. The expanded executive summary will be submitted in English and Egyptian Arabic, in hard copy (50 copies) and electronically. The report will be disseminated within USAID and to stakeholders according to the dissemination plan. Data Sets: All data instruments, data sets, presentations, meeting notes and final report for this evaluation will be presented to USAID on three (3) flash drives to the Evaluation Program Manager. All data on the flash drive will be in an unlocked, editable format. A two-day team planning meeting will be held in Egypt before the evaluation begins. This meeting will allow USAID to present the team with the purpose, expectations, and agenda of the assignment. In addition, the team will:  Clarify team members' roles and responsibilities;  Establish a team atmosphere, share individual working styles, and agree on procedures for resolving differences of opinion;  Review and develop final evaluation questions (work out realistic expectations of the team within each of the topic areas during meetings with MOHP, HIO, and USAID);  Review and finalize the assignment timeline and share with USAID;  Present data collection methods, instruments, tools, and guidelines (materials should be developed prior to this meeting;  Review and clarify any logistical and administrative procedures for the assignment;  Develop a preliminary draft outline of the team's report; and,  Assign drafting responsibilities for the final report. B. Reporting Guidelines  The evaluation report should represent a thoughtful, well-researched and well organized effort to objectively evaluate what worked in the project, what did not and why.  Evaluation reports shall address all evaluation questions included in the scope of work.  The evaluation report should include the scope of work as an annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology, budget, or timeline need to be agreed upon in writing by the technical officer.  Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an Annex in the final report.  Evaluation findings will assess outcomes and impact on males and females.  Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 36 differences between comparator groups, etc.) and what is being done to mitigate the threats to validity.  Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings should be specific, concise and supported by strong quantitative or qualitative evidence.  Sources of information need to be properly identified and listed in an annex.  Recommendations need to be supported by a specific set of findings.  Recommendations should be action-oriented – organized according to whether recommendations are short-term or long-term, practical, and specific, with defined responsibility for the action. C. Evaluation report requirements The format for the evaluation report is as follows: 1. Executive Summary—concisely state the most salient findings and recommendations (2pp); 2. Table of Contents (1 pp); 3. Introduction—purpose, audience, and synopsis of task (1 pp); 4. Background—brief overview of HS 20/20 project in Egypt, USAID project strategy and activities implemented in response to the problem, brief description of HS 20/20, purpose of the evaluation (2–3 pp); 5. Methodology—describe evaluation methods, including threats to validity, constraints and gaps (1 pp); 6. Findings/Conclusions/Recommendations—for each evaluation question by health-systems￾strengthening component; also include data quality and reporting system that should present verification of spot checks, issues, and outcome (17–20 pp); 7. Challenges—provide a list of key technical and/or administrative, if any (1–2 pp); 8. Future Directions (2–3 pp); 9. References (including bibliographical documentation, meetings, interviews and focus group discussions); 10. Annexes—annexes that document the evaluation methods, schedules, interview lists andtables— should be succinct, pertinent and readable. The final report will be reviewed using the Checklist for Assessing USAID Evaluation Reports (http://www.usaid.gov/policy/evalweb/evaluation_resources.html). The final evaluation report will conform to the Criteria to Ensure the Quality of the Evaluation Report found in Appendix I of the USAID Evaluation Policy. The Evaluation Program Manager will determine if the criteria are met. This evaluation will not conclude until the Evaluation Program Manager has confirmed, in writing, that the report has met all of the quality criteria. The final version of the evaluation report will be submitted to USAID/Egypt electronically. The report format should be restricted to Microsoft products and 12-point type font should be used throughout the body of the report, with page margins 1” top/bottom and left/right. The report should not exceed 30 pages, excluding references and annexes. III. Team Composition USAID encourages the participation of local experts on evaluation teams, including in the roles of evaluation specialist and team leader. USAID staff are also encouraged to participate on evaluation teams, as are MOHP and HIO staff, implementing partners or other stakeholders when their participation would _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 37 be beneficial for skill development and not present a conflict of interest nor a threat to validity, or their engagement in the evaluation would help to ensure the use of evaluation results within USAID. All attempts should be made for the team to be comprised of an equal number of male and female members. Team Composition Team Leader – a senior consultant with extensive experience in leading and conducting USAID health program evaluations. Senior Technical Advisor - conducting data quality analysis and methodologies for health systems strengthening issues. Team Members: A mix of senior and mid-level consultants with a combination of expertise in: health-systems strengthening, implementing and evaluating USAID health-systems-strengthening programs, monitoring and evaluation, health economics, HR management, training/capacity building and/or with knowledge of and experience in the HS 20/20 approach. At least one team member should be fluent/professionally proficient in spoken Egyptian Arabic. USAID Staff – one or more USAID/Egypt and/or USAID/Washington staff. Should a USAID member not be able to participate, add one more external team member. Logistics Coordinator/Assistant - local consultant; fluent in written and spoken Egyptian. The Team Leader will:  Finalize and negotiate with USAID/Egypt the evaluation work plan;  Establish evaluation team roles, responsibilities, and tasks;  Facilitate the Team Planning Meeting (TPM);  Ensure that the logistics arrangements in the field are complete;  Manage team coordination meetings in-country and ensure that team members are working to schedule;  Coordinate the process of assembling individual input/findings for the evaluation report and finalizing the evaluation report;  Lead the preparation and presentation of key evaluation findings and recommendations to USAID/Egypt team prior to departing Egypt. _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 38 ANNEX B: MODIFICATIONS TO THE EVALUATION STATEMENT OF WORK The following minor modifications to the Statement of Work and technical proposal were made at preparatory stages with concurrence of USAID/Egypt: 1. In-country schedule: The original Gantt chart called for the full 6-person team to be in-country for 5 weeks. The international travelers were unable to obtain business visas (permitting for a stay of more than 30 days) in a timely fashion. In-country time therefore had to be limited to 30 days. 2. Draft Evaluation Report: The original SOW called for the Team Leader to deliver a preliminary draft to USAID/Egypt prior to departure from Egypt. Because of the need to limit the in-country time to 30 instead of 35 days, it was agreed that all team members would submit their draft contributions to the report to the Team Leader by February 23, 2013, and that the first draft submission to USAID/Egypt would be made by SI on March 15, 2013. 3. Data Sets: The original SOW called for compilation and submission of full transcripts of evaluation interviews and similar evaluation file documentation. It was agreed with USAID/Egypt that the Mission did not want to receive or retain such information. Instead, SI required team members to submit evaluation file notes which will be retained in an SI evaluation file for future reference if necessary. _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 39 ANNEX C: ESTIMATES OF DONOR FINANCING FOR HEALTH SECTOR REFORMS, 2008–2012 World Bank • HISDP $43m (increment of $76m) USAID • All of PHC • • HS 20/20 $84m* $10.7m Other donors that have contributed to health sector reforms over the years include: African Development Bank, Saudi Arabia, DANIDA, Italian Cooperation, and Japanese Development Fund, with smaller amounts for selected heatlh programs in selected geographic areas. *Sources: (1) Egypt Health and Population Legacy Review, 2003-2009, Volume II; (2) USAID/Egypt Strategic Plan Updates, FY 2003-2009 ; (3) USAID Dollars and Results, FY 2011; and (4) Vinard P. and Rhodes, G., “Final Evaluation: Egyptian Health Sector Policy Support Programme, 2006-2010, European Commission, June 2012. Contributors 2008 2009 2010 2011 2012 TOTAL European Commission • HSPSP-I • HSPSP-II €45m (increment of €$88m)* €44m (increment of €110m)* _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 40 ANNEX D: EVALUATION METHODOLOGY AND WORKPLAN INTRODUCTION Social Impact, Inc. (SI) has been requested by USAID/Egypt to conduct a performance evaluation of the Health Systems Strengthening Project in Egypt (HS 20/20 Egypt). HS 20/20 Egypt was a country-specific project implemented by Abt Associates through USAID/Washington’s flagship Cooperative Agreement for health-systems strengthening, HS 20/20. USAID/Egypt sought assistance from HS 20/20 to support the efforts of the Government of Egypt (GOE), Ministry of Health and Population (MOHP), to improve SHI coverage and quality of care, develop human capacity and help implement financing reforms. HS 20/20 Egypt, which began in February 2008, had four related but distinct technical areas, namely to (1) develop a long-term workforce plan for the MOHP and establish a sustainable workforce development plan; to (2) support the implementation of HIO’s new role as a payer and build its capacity in key payer functions and contract management; to (3) develop and institutionalize National Health Accounts (NHA) to monitor the impact of reforms while informing policy; and to (4) assist in the establishment of a “Leadership Academy” as a sustainable Egyptian institution that builds leadership and management capacities to support the implementation of reforms.28 HS 20/20 Egypt ended in April 2012. The life-of-project funding for HS 20/20 Egypt was $10,788,902. PURPOSE AND USES OF THE EVALUATION The purpose of this evaluation is for USAID and its partners to understand what has and has not worked in terms of the implementation of Egypt HS 20/20; how different aspects of the project affected relationships with counterparts; how USAID and the GOE can maximize the lessons learned for future investments in health-systems strengthening; and what USAID can do to design a project that both captures and sustains monitoring and evaluation (M&E) efforts. Further, this evolution is also intended to generate information that will be useful to policy-makers and program managers who are concerned with improving future programming in the area of health-systems strengthening. The audience for this evaluation report will be USAID/Egypt, Office of Health and Population. In addition, the report will be shared with the USAID/W Middle East Bureau, the USAID/W Office of Health Systems, and Abt Associates, the implementer for the new USAID/W Health Finance and Governance Project. As collaborating partners for implementation of HS 20/20 Egypt, key informant information will be sought, and the results of the evaluation shared, with the MOHP, the Health Insurance Organization 28 The “Leadership Academy” component was halted during the course of the project and can therefore not be evaluated separately. However, different activities originally planned under this component have been incorporated into the three other activities. The team will evaluate these activities as part of the three main activities, where appropriate. _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 41 (HIO), and other donors working to support health-systems strengthening in Egypt. To this end, the executive summary will be translated into Arabic and will be disbursed to MOHP, HIO and other stakeholders. It is expected that MOHP, HIO and other stakeholders will have opportunities to discuss how the HS 20/20 Egypt project assisted them and how USAID support could best assist them in meeting their future health-systems-strengthening goals. USAID will incorporate evaluation report recommendations into consideration of future health-systems￾strengthening activities. Abt Associates and its subcontractors will learn about the strengths and weaknesses of the HS 20/20 Egypt as viewed by the evaluation, and the MOHP and HIO will learn about improved ways of benefitting from implementing partner technical assistance. It is expected that USAID/Egypt will disseminate the evaluation report widely with relevant stakeholders and project beneficiaries, including in the Development Experience Clearinghouse (DEC). EVALUATION SCOPE AND FRAMEWORK The evaluators are charged with looking back to form judgments about the value of HS 20/20 Egypt and looking ahead to recommend approaches for the future and thus by definition this will be a summative evaluation. For each technical area, a variety of methods will be employed to assess the project’s performance—what was accomplished that can be directly associated with the project’s interventions, and to identify factors that may have contributed to the successes or failures. In assessing project performance, the evaluation will review the procedures used by the project to monitor and evaluate its progress and make mid-stream corrections to the project’s scope and implementation strategies. In looking forward, the evaluation will consider contextually the current environment for health-systems strengthening, the general direction of these health sector developments in Egypt, the current roles of other donor partners in the area, and the most appropriate and complementary support that might be provided by USAID/Egypt. Further, the evaluation will take advantage of available data to assess the effectiveness of one a selected systems strengthening intervention. A case study will focus on HS 20/20 Egypt’s work with the Utilization and Case Management (UCM) approach. This evaluation framework was built around HS 2020 Egypt’s 4 technical project areas as described in its project documents and two cross-cutting areas identified by USAID as important to examine. These are: Technical project areas:  Workforce Planning (WFP)  Health Insurance Organization (HIO)  National Health Accounts (NHA)  Training and Capacity Building (TCB) _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 42 Cross-cutting areas:  Other Strategic Interventions (SI)  Project Monitoring and Evaluation (M&E) Each of the technical and cross-cutting project areas will be examined against the evaluation’s Key Questions (KQs) as defined by USAID in its original RFTOP and further refined and expanded by the evaluation team during the preparatory phase of the evaluation and the Team Planning Meeting (TPM). This matrix approach to organizing the evaluation will ensure that all KQs are addressed in relation to all project components and activities. Each evaluation team member has been designated to take the focal lead in data gathering, analysis and final evaluation report writing for one technical or cross-cutting project areas. Findings, conclusions and recommendations in the final evaluation report will be consolidated and presented within the context of the KQs, ensuring that the evaluation is responsive to the interests and needs of USAID/Egypt’s Office of Health and Population and its stakeholders. KEY QUESTIONS Key Questions as originally stated in the Evaluation SOW Refined/Re-articulated Evaluation Key Questions Performance Assessment - Looking back What were HS 20/20’s rationale, objectives and planned activities? 1) To what extent was HS 20/20 able to make changes/improvements in the health strengthening areas in the project design and supported by the project? a) To what extent did HS 20/20 achieve what was stipulated in the program description – if so how, and to what extent; if not, why? b)What project interventions are sustainable and being successfully carried out by the Government of Egypt as a result of HS 20/20? 1) What changes/improvements in health systems resulted from HS 20/20’s project design and implementation? a) Compared to the project objectives and planned activities, what was actually achieved by HS 20/20, how was this done, and if objectives and targets were not reached, why not? b) What evidence is there that the GOE/project’s strategies and achievements have led to actual improvements in sector systems? 2) What interventions (strategies and activities) have been adopted and are being implemented in a sustainable manner (institutionalized) by the GoE? 2) What effect did HS 20/20’s implementation strategy have on relationships with Government of Egypt counterparts? 3) To what extent was the project’s implementation strategy (of expert systems development combined with capacity transfer) perceived as effective by GOE counterparts? a) How is the project perceived/valued by Government of Egypt counterparts? 3) To what extent did the project achieve sector integration through partnerships with other cooperating agencies, international donors, the public sector, and the private 4) To what extent did the project improve cooperation and strategic integration of program interventions among USAID implementing partners, donors, and public/private _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 43 Building from the illustrative key questions contained in the original evaluation SOW, a major responsibility of the evaluation team during the preparatory phase of the evaluation was to refine and/or expand KQs as necessary for clarity, measurability, and comprehensiveness. Annex D shows the relationship of the final list of evaluation’s KQs to questions found in the original evaluation SOW. KEY INFORMANTS Key informants will include representatives of USAID/Egypt, HS 20/20, MOHP, HIO, and other partners such as international donors. A comprehensive list of individual key informants has been developed and agreed upon with USAID. The team will also augment this list, as necessary, based on additional information from key informants as interviews progress. The comprehensive list of individual key informants identified up to now is further discussed in the work plan section of this document. sector? sector partners? Monitoring and Evaluation 5) Describe the formats and processes of HS 20/20’s M&E systems in Egypt. What results, indicators and targets were being regularly and routinely tracked? 4) How did the monitoring and evaluation system capture the systems strengthening changes? a) How was the information used for program management and improvement? b) To what extent does the MOHP and the HIO have and sustain monitoring and evaluation capabilities in the project intervention areas? 6) How did the M&E system capture the systems strengthening changes? How could measurement of systems strengthening changes have been strengthened? a) How was M&E information being used by the project and GOE counterparts for program management, decision making and improvements? b) To what extent does the MOHP and the HIO have and sustain monitoring and evaluation capabilities in the project intervention areas? Future Activities 5) What HS 20/20 project interventions have the most potential to show a return on investment for health￾systems strengthening? a) What are the next steps that USAID should take to maximize our investments in these? b) What health-systems strengthening interventions are within the manageable capacity/interests of USAID that we can contribute to? c) What are the enabling factors and critical assumptions that have to be in place to make sure the programs are effective? 7) What HS 20/20 project interventions have the most potential to show a return on investment for health￾systems strengthening? a) What should USAID do to sustain these achievements moving forward? b) Considering the GOE’s direction, the commitment of other donors, and USAID’s broadly defined health sector support in Egypt, what health-systems￾strengthening interventions are within USAID’s manageable capacity/interests? c) How can USAID better ensure program effectiveness in its future support for health-systems￾strengthening interventions? What enabling factors and critical assumptions would need to be in place? 6) What can USAID do in the future to maximize sector integration through partnerships with other cooperating agencies, international donors, the public sector, and the private sector? 8) In the future, what can USAID do to improve the role it plays in fostering strategic integration of program interventions in the sector among USAID implementing partners, donors, and public/private sector partners? _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 44 SOURCES AND METHODS OF DATA COLLECTION To answer the main evaluation questions and derive insights into possible future health system strengthening activities, the team will combine data from HS 20/20 documentation; file data and records from the MOHP and HIO; and data gathered during Structured Key Informant Interviews (SKIIs) and FGDs conducted by the evaluation team. If possible, methods of collecting qualitative information will be supplemented by methods that also lend themselves to quantification. For example, an anonymous rating scale questionnaire will be administered to the MOHP managers that interacted with the HS 20/20 project to gather their impressions about the effectiveness and utility of the project. In addition, a limited number of FGDs (FGDs) with training beneficiaries will be undertaken to gather (and rate) their impressions of the training they received in connection with HS 20/20 interventions. FGD participants will be identified by implementers (HIO and MOHP), and will include individuals who received training who are currently traceable and available. An analysis has been done to estimate what type of data gathering sources and methods of data are likely to be used in each technical area to answer key evaluation questions. This analysis is shown below as Appendix A of this document, “Sources and Methods for Collecting Quantitative and Qualitative Data.” At every step during the evaluation, from the initial document review to synthesizing final recommendations, the team will use highly participatory methods and approaches to fully incorporate the knowledge and insights of all team members and key informants. In particular, the team will work closely with USAID/Egypt staff, who will be consulted by the evaluation team throughout the evaluation and whose inputs are crucial to ensure responsiveness to the evaluative interests and needs of USAID as stated in the evaluation SOW. Additional steps have also been taken to ensure responsiveness of the evaluation. An analysis was completed by the team that specifies in detail all interview questions that will be covered in each of the technical areas. The team as a whole, including USAID input, participated in developing and finalizing interview questions for each technical and cross-cutting area of the evaluation. The detailed list of interview questions was further cross-checked to ensure alignment with key evaluation questions. The detailed interview questions are to be used as applicable in a Structured Key Informant Interview (SKII) instrument developed for purposes of this evaluation. The SKII is designed as a structured questionnaire with predetermined semi-open questions to balance comparability and the need for interviewees being able to speak freely about their experiences with the different project components. The SKII template will be continuously modified by evaluators to reflect specific KQs that will need to be covered with a particular category of key informants. Similarly, in keeping with best practices for design of FGDs, evaluators will always develop and use a set of structured questions that are targeted to that specific group of FGD participants. Team members will be required to fully utilize the SKII data gathering instrument to take notes and document interviews and discussions. Resulting notes will be shared among team members as part of the team’s data synthesis process. Although the pool of key informants to be included in this evaluation is specifically targeted to the organizations and is not, as such, expected to have a bearing on gender issues, when possible, data _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 45 collected during the evaluation will be disaggregated by gender. In particular, at least one gender-specific question will be interwoven into interview discussions such as the possible gender implications of a policy that may have been promoted through HS 20/20. In addition, and in accordance with USAID policy, the team will integrate into the analysis considerations of gender roles and inequalities as they may have affected HS 20/20 effectiveness, as well as to how the project may itself have influenced gender statuses and relationships. This is of particular importance for the MOHP Workforce Planning Component, since there seems to be a gender dimension to staff roles at the service delivery level. The composition of FGDs will ensure the representation of women at all levels in order to address gender specific issues. Our team members will ensure that at least one of the FGD facilitators is a woman and, should there be any sensitive issues where women may not be willing to speak freely in the presence of men, the team will consider asking additional questions separately to women. SITE VISITS Due to the concentrations of intervention sites, the limited timeframe of the evaluation, and security considerations, most site visits will be offices and health facilities in the greater Cairo area. In addition, field trips will be taken by team members to Luxor and Quena Governorates to review HS 20/20 Egypt interventions in the technical areas of Workforce Planning and HIO. Those two governorates are in geographic proximity to each other and offer contrasts in the health financing and governance structures and environments in which the HS 20/20 Egypt interventions were being implemented. The Workforce Planning site visits to Luxor and Quena will focus on reviewing sites where workforce plans have successfully been completed and some effort has been made to implement plans. Constraints to implementation, including systemic political, administrative, and human resource management challenges will be investigated. In both Luxor and Quena, the evaluation team will also visit sites that have not received the Workforce Planning intervention in order to observe potential differences between intervention and non-intervention sites. The HIO site visits to Luxor and Quena will focus on visits to facilities that have participated in prototype development and implementation of HIO’s payer/contracting model. The main purpose of those visits will be for the evaluators to better understand how the various management systems have been developed and implemented at the service delivery level. Finally, during the site visits to Luxor and Quena the team will also looking at the cross cutting questions related to capacity building as well as monitoring and evaluation. Additional Key Points from Notes The other technical area of the evaluation focused on National Health Accounts and was implemented at the national level and will therefore not require site visits to governorates. Similarly, evaluation inquiries focused on the project’s M&E system will not require visiting sites outside of Cairo. Separate from technical systems strengthening activities in Workforce Planning, HIO, and NHA, the Training and Capacity Building evaluative inquiries will focus on key informants at the national level who have been concerned with human resource development wherever HS 20/20 Egypt resources have been used for training and capacity building. Additionally, a limited number of FGDs will be organized _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 46 with persons trained in the HS 20/20 Workforce Planning, HIO, or NHA interventions to get direct input from training beneficiaries on their training/capacity building experiences. DATA SYNTHESIS AND TRIANGULATION The evaluation team will use a data analysis strategy before the actual data collection begins to encourage evaluators to begin thinking about trends in findings from different data sets to determine if findings are similar or divergent. Data will be compiled and analyzed for each of the components of the evaluation as the collection is completed. Data analysis will most likely employ a parallel, mixed-data approach in which quantitative data (for instance from HIO records) is reviewed independently by the evaluator(s). Quantitative data is expected to include secondary data available from measurements and assessments undertaken in connection with the HS 20/20 interventions such as workforce inventories in 11 governorates. Qualitative data obtained from the site observations and SKIIs will be analyzed separately. A parallel, mixed-data approach takes the findings and analysis from each data set and uses it to inform and explain findings from the other data set. As this evaluation methodology applies a mixed-methods approach to strengthen and validate findings for the same question through a triangulation process, the analysis involves comparing the findings of each data set to determine whether or not there is a convergence of findings. Toward the end of the data collection phase of the evaluation, the team will hold several structured synthesis sessions in which findings, conclusions and recommendations will be thoroughly vetted and finalized by group consensus. Any defensible dissenting views emerging from key informants or within the team will be noted in the final report. COMPARISON STUDY Because human resource development is likely to be a critical priority for future health sector reforms in Egypt, the area of human resource development has been identified as a candidate area to undertake a comparison study assessment. A case study will focus on human resource development in HS 20/20 Egypt’s HIO component concerned with the Utilization and Case Management (UCM) program. Subject to the availability of existing and available data, two different change theories/development hypotheses will be assessed:  UCM reduces costs. HIO has already conducted cost savings studies that will be additionally reviewed and analyzed for the evaluation case study. Useful comparisons between intervention and non-intervention sites may be possible.  UCM improves health care services and health outcomes. This hypothesis has greater potential than the cost hypothesis to inform possible long range policy in a health reform agenda. The MOHP’s human resources department has conducted pre and post on UCM training that will be reviewed and analyzed for the evaluation care study. While the comparison between intervention and non-intervention sites does not provide the robust evidence that would be needed to make any statements about whether or not the UCM intervention is causing some of the observed differences, they will inform the discussion about the effectiveness of utilization and case management practices. It can, for example, validate some of the findings from the _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 47 visit to the intervention site if the evaluation team does indeed find evidence of improved use of treatment standards and clinical guidelines in specific types of cases compared to similar case management in the non-intervention site. OUTLINE OF FINAL EVALUATION REPORT The final evaluation report outline has been developed based on USAID evaluation report guidance and requirements as specified by the USAID/Egypt Office of Health and Population, and in consideration of the team’s efforts to produce a high quality evaluation report for USAID and its stakeholders. The findings sections of the final report are organized in such a way that will be directly responsive to key evaluation questions, and will contain information dawn from any or all of the technical and cross-cutting evaluation areas. Separate and more detailed technical papers on Workforce Planning, HIO, and NHA, will be developed and presented separately as report appendices. WORK PLAN Gantt Chart: During the team planning week, the Gantt Chart showing major milestone events for the 11 week evaluation was reviewed and adjusted by the team to more accurately reflect the work schedule. Comprehensive Key Informant List: During the team planning week, a comprehensive list of persons to be contacted and interviewed (individually or in groups), was developed by the team with input from USAID. That list is now online utilizing a Google Drive application and is accessible to all evaluation team members. This information will be used for the duration of the evaluation to facilitate coordination of team scheduling and participation in scheduled appointments and events. Both USAID and the MOHP have expressed interest in participating in some of the scheduled appointments. Evaluation Appointments Scheduling and Events Calendar: During the team planning week, procedures were developed for the team itself to manage appointments scheduling utilizing the Google Drive application. Focal Leads are in charge of scheduling appointments for their designated key informants, and it is necessary for team members to coordinate among themselves when a team partner wants to attend one or more appointments during a work day. The Calendar has updating capability and is accessible to all evaluation team members. Overview of the Evaluation Team – Roles and Responsibilities: The six-person evaluation team is composed of the following members:  Dr. Dayl Donaldson, Health Economist with previous health sector evaluation experience in Egypt  Dr. Waleed El Feky, PublicHealth Expert with extensive health workforce planning experience in Egypt  Mahinaz El Helw, Monitoring and Evaluation Expert with extensive health sector program management experience in Egypt  Kate Fehlenbeerg, Public Health Expert with extensive health survey and evaluation experience  Mildred Howard, Team Leader, with an extensive background in USAID evaluation methods, as well as health sector reforms and health strengthening programs in Egypt _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 48  Doaa Oraby, Training and Capacity Building Expert with extensive health sector program management experience in Egypt The team members have been designated to take the lead responsibility for specific technical and/or cross-cutting components of the evaluation. _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 49 ANNEX E: PERSONS CONTACTED HEALTH INSURANCE ORGANIZATION Dr. Abu-Bakr el-Makawy, Director, HIO Branch Office Dr. Adel el Shaater, Senior Medical Officer, HIO Branch Office Dr. Rowia Hawash, Director, Contract Hospital Relations, HIO HQ and Cairo Branch Mona Abd el Menam, Director, Accounting and Finance Dr. Mohsen George, Chief Medical Officer Dr. Manal Mongy, Director, Quality Improvement Department Dr. Abdel Rahman Sakka, Chairman NASR CITY HOSPITAL Dr. Leila Ibrahim Desouki, Director, Quality and Technical Support Office, Nasr City Hospital Dr. Samir el-Assal, Medical Director, Nasr City Hospital Dr. Ramez Zaki Habib, Nasr City Hospital Dr. Emad Kathem, Executive Director Nasr City Hospital Dr. Hanan Nagar, Director, Case Management Office, Nasr City Hospital Dr. Hanaa Sobhy Tawfiq, Quality Coordinator, Nasr City Hospital NASSER INSTITUTE Dr. Iman el Zand, Director, Quality Office, Nasr Institute Dr. Fouad Ahmed, Director, Case Management Office, Nasr Institute MINISTRY OF HEALTH AND POPULATION, HEADQUARTERS Dr. Emad Ezzat, Undersecretary for Preventive and Primary Health Care Dr. Eng. Mohamed M. Abd El-Rahman, Head, Central Administration for Technical Support and Projects (CDTSP) Dr. Amr El Shalakani, Head, Health Economics Unit Dr. Mohga Mostafa, Undersecretary for Human Resources Development Dr. Medhat Refae, Undersecretary for Training and Research Eng. Marwa Shorbagy, Project Manager, National Information Center for Health and Population (NICHP) Dr. Mervat Taha, Head, Department of Planning Eng. Ismael Turk, Head, National Information Center for Health and Population (NICHP) Dr.Hala Zayed, Undersecretary for the Minister’s Cabinet LUXOR GOVERNORATE Okba Ahmed, Head , Information Center, Luxor Mohamed Amin, Head, Information Center, Gorna District Abdalla Ahmed, Head, Information Center, Gorna District Hospital Dr.Mahmoud Hegazy, Head, Secondary Care, Luxor Dr.Ahmed Hamza, Head, Medical Syndicate, Luxor Dr.Amgad Ibrahim, Head, Gorna District Hospital Dr.Mahmood Metwally, Head, Bayadea District Hospital Dr.Mona Mohamed, Head, Training and Research, Luxor Dr.Mohamed Rabea, Undersecretary and Head of Luxor Governorate Dr.Gamal Radwan, Head, Luxor General Hospital Mostafa Saad, Head, Information Center for Secondary Care Ali Saad, Head, Information Center, Luxor General Hospital _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 50 Doaa Zaki, Data Entry, Luxor General hospital QENA GOVERNORATE Mohamed Ahmed, Head, Administration and Finance, Qena Dr.Ashraf Mohamed, Head, Secondary Care, Qena Dr.Ahmed Saad, Head, Information Center, Qena Dr.Asaad Yassin, Undersecretary and Head of Qena Governorate HEALTH SYSTEMS 20/20 Catherine Connor, Deputy Director, Abt Associates, Inc., Health Financing and Governance Project Dr. Mahmoud Farag, National Health Accounts Consultant Dr. Mohammed Lotfi, Professor of Accounting, Faculty of Commerce, Ain Shams University Nagwan Hassan, National Health Accounts Coordinator Samir Mansour, Technical Lead - Workforce Planning Dr. A.K. Nandakumar, National Health Accounts Consultant Dr. Nadwa Rafae, Chief of Party, HS 20/20 Egypt Dr. Tom Schwark, Health Policy and Management Consultant Dr. Mahmoud Abdel Latif Salem, Hospital Costing Consultant USAID Seba Auda, Development Program Specialist, Program Office Shadia Attia, Senior Monitoring and Evaluation Advisor, Office of Health and Population Laura Campbell, Population, Health and Nutrition Officer, Office of Health and Population Holly Fluty Dempsey. (former) Chief, Office of Health and Population Akmal El Erian, Senior Project Management Specialist, Office of Health and Population Shahira Hussein, Project Management Specialist, Office of Health and Population Iman Abdel Halim, Senior Program Operations Specialist, Program Office Randy Kolstad, Chief, Office of Health and Population Vikki Stein, (former) HS 20/20 Activity Manager George Sanad, Senior Project Management Specialist, Office of Health and Population OTHER STAKEHOLDERS Dr. Naeema Al-Gasseer, Health Economist, World Health Organization, EMRO Dr. Magdy Bakr, National Program Officer, World Health Organization, Cairo Dr. Pierre Destexhe, Programme Manager, Health Sector Development, The Delegation of the European Union to Egypt Dr. Nasr El Sayed, Director, National Council for Childhood and Motherhood Dr. Alaa Hamed, Human Development Coordinator, World Bank, Cairo Dr. Awad Mataria, Health Economist, World Health Organization, EMRO Prof. Yehia Mekkia, Board Member, Egyptian Medical Syndicate and Professor, Ain Shams University Professor Ahmed Nada, Head of Foreign Affairs Comity, Egyptian Medical Syndicate, and Professor, Cairo University Dr. Hassan Salah. Technical Officer, Health Policy and Planning, EM _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 51 ANNEX F: DATA COLLECTION INSTRUMENTS SKII FOR HS 20/20 Egypt Evaluation Category of Respondent (Role during HS20/20; check all that apply) __ USAID/Egypt __ Abt Asso HQ staff __ Abt Asso HS20/20 project staff in Egypt __ Central MOHP staff __ Local MOHP staff __ HIO staff __ Donor partner (Name donor here) __ Private implementing partner rep (Name agency here) __ NGO staff (Name NGO here) __ Hospital Management (Name Hospital here) __ Participant of Sharm El Sheikh Health Sector Reform Conference Gender of Interviewee: __ Male __ Female Interviewer(s): Date of Interview: Please briefly describe your role in the HS20/20 in Egypt: QUESTIONS FOR KEY INFORMANT(S) Evaluator’s comments INTERNAL USE ONLY _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 52 FOCUS GROUP DISCUSSION GUIDE HIO ACCOUNTING/FINANCIAL MANAGEMENT TRAINEES Introduction: HS 20/20, team and final evaluation, purpose of focus group discussion Ethical consideration: voluntary participation/consent, confidentiality Questions: 1. Use flip chart to obtain information on posting of participants at time of project (2009-2011) 2. Use flip chart to obtain information on participation of trainees in courses of HS2020 1. Was the implementation of the WFP component implemented in a way that was participatory and inclusive of key stakeholders? How well did the WFP efforts meet the needs of MOHP staff to improve WFP (e.g., new skills; equitable workload)? 2. What changes have been made to the MOHP system due to the Workforce Planning activities? 3. In your opinion and based on your experience with the program, are there any aspects of the Workforce planning component that should or should not be applied in future health system strengthening efforts? [e.g., the Workload Indicators of Staffing Needs (WSIN) Model; the Workforce Planning Tool (WPT); etc.] ADDITIONAL KEY POINTS FROM NOTES: Junior Acct/Fin Staff (#) Senior Acct/Fin Staff (#) HIO HQ HIO Branch HIO Hospital/Other Junior Acct/Fin Staff (#) Senior Acct/Fin Staff (#) _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 53 3. Prior to the training of HS2020, did you have formal education (e.g. tech school/university) or certificate training in any of the areas related to the training? Which ones? 4. Do you feel that the technical content of the courses you participated was relevant to your your interests/needs for training related to your job at HIO? Explore why/why not, e.g. Course/curricula planning failed to adequately assess HIO’s overall financial/accounting training needs? Course/curricula development planning failed to include assessment of needs for knowledge/skill development and thus was redundant? Courses provided shallow breadth and should have aimed to provide depth in one key area – which one? Other? 5. Where the teaching methods (lectures, assignments to work with real HIO data, mentoring visits w/Professor Lofti) employed effective in: i. development of new knowledge ii. development of capacity to use that knowledge on the job Explore why/why not? i. Did any of the courses you participated in measure your acquisition and/or ability to utilize the course knowledge after the end of the course? If yes, how was this assessed? ii. Are you currently using any of the content presented in the courses in your current work with HIO? If yes, which course content? If no, why not? Probe as to whether impediment to use of new skills was lack of sufficient training, and/or lack of HIO management decisions to change systems, and/or lack of IT, and/or other factors. i. If the future, what content areas do you believe the training office of HIO should prioritize regarding addressing your needs for improvement of knowledge and skills in accounting and financial management current and future tasks? HIO Exp Analysis & Budget Forecasting Accrual Accounting (vs cost accounting) Step-down Cost Estimation of Costs Other (what?) _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 54 FOCUS GROUP DISCUSSION GUIDE HIO MEDICAL MANAGEMENT TRAINEES Introduction of HS20/20 Ethical consideration: voluntary participation, confidentiality, consent of participation Focus of Discussion: The following questions pertain to utilization review and case management training. Please respond according to your experience before, during and after that course. If you did not participate in that course, please indicate so now. Questions: 1. Prior to the course did you have any training or work experience related to utilization management/case review? Explore what prior training/work experience trainees had and their views on the quality and value of that training, e.g. training. 2. Did the content of the course fall short/meet/exceed your expectations? (Explore reasons for degree of satisfaction/dissatisfaction w/content). What are your views concerning the organization of the training course - including lectures and practicum - regarding developing your understanding and your retention of course materials? 3. How was your performance in the training measured (nb: methods were practicum verbal presentations/feedback and written exam)? Would one approach have been sufficient or were performance evaluation methods reinforcing? Do you think the trainer’s feedback and grade were impartial and fair? 4. Is the technical content of the course relevant to your work now? What parts of the curricula are most helpful in your current work and why? 5. Are there any structural (e.g. automatization), managerial (e.g. lack of managerial and/or medical staff interest and support) and/or other aspects that hinder applying what you learned in this course in meet your current position responsibilities? (note: collate responses by type of current position) 6. What are your priorities for future training in utilization review/case management or other quality improvement subjects? 7. Do you believe HIO should develop a comprehensive/on-going training program related to quality improvement? If yes, what form should such a program take (e.g. stipends for basic training at AUC and CU, support for basic training via HIO, development of training programs for quality teams)? Thank the participants for their time. _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 55 EGYPT HS20/20 END OF PROJECT EVALUATION MEDICAL MANAGEMENT (HOSPITAL/PHC AUDIT, UTILIZATION/CASE MANAGEMENT) FOCUS GROUP – QUESTIONNAIRE INSTRUCTIONS: Please respond to the following according to the instructions in each area of the questionnaire. CONFIDENTIALITY: Your responses on this form will be treated as confidential. Your name should not be written on the form. Further individual responses will not be reported nor shared with HIO nor USAID. The information from the questionnaires for all group members will be aggregated analyzed along with the information from the focus group discussions. CONSENT: You may choose to not answer any or all of the questions. 1. Educational Background 1.a Professional clinical degree (s) (check all that apply) MD _____ Nurse ____ Pharmacist _____ Dentist _____ 1.b Other professional degrees (check all that apply) Public Health (MPH/DrPH ____ MBA ____ Other (describe)____________ ______________ 1.c Health quality-related course(s): No _____ Yes _____ If yes, Certificate course _____; Degree course _____ When completed (mm/yy)?___________ 1.d Health Management and/or Finance-related course(s) No _____ Yes _____ If yes, Certificate course _____; Degree course _____ When completed (mm/yy)? __________ 2. Position Assignment within HIO 2.a At what Level of HIO did you work when you participated in your first HS2020 course? HIO HQ ____ HIO Branch Office ____ HIO Own Hospital ____ HIO Contracted Hospital ____ 2.b At what Level of HIO do you currently work? HIO HQ ____ HIO Branch Office ____ HIO Own Hospital ____ HIO Contracted Hospital ____ 2.c Please indicate if you have ever worked in any/all of the following positions and for how long: Quality Coordinator? No _____; Yes _____ (how long _____ months?) Infection Control Coordinator? No _____; Yes _____ (how long _____ months?) Case Management Office? No _____; Yes_____ (how long _____ months?) _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 56 3. Participation/Completion of HS2020 Medical Management Trainings Please indicate (X) all courses/workshops sponsored by HS2020 that you participated in and also indicate those that you completed/received a certificate (graduated) and the family name of the lead trainer. Participated Graduated Trainer Audit for PHC Units Audit for Hospitals Utilization/Case Management (UCM) Hospitals Training of Trainers for UCM - Hospitals Policies, Procedures, Outcome Indicators of CMO @ Hospital Level Please Respond to the Questions/sub-questions below if you received training on the subject matter addressed. Skip Questions/sub-questions when you did not receive the specific training in that specific area. 4. Training for Audit of Hospital 4.a In my present position, I am utilizing the skills learned in the audit course for hospitals at least: Once a Week ____ Once a Month ____ Once a Quarter ____ Once a Year _____ Never _____ Please indicate your agreement with the following statements 4.b The curricula and training approach (using classroom and practicum approaches) was very effective in conveying and retaining new information and building skills regarding audit of hospitals? Strongly Agree ____ Agree ____ Don’t Agree/Disagree ____ Disagree ____ Strongly Disagree 4.c The information in the audit course(s) for hospitals is highly relevant for improving the medical quality of services provided to HIO clients. Strongly Agree ____ Agree ____ Don’t Agree/Disagree ____ Disagree ____ Strongly Disagree ____ _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 57 5. Training for Utilization Review/Case Management in Hospitals 5.a In my present position, I am utilizing the skills learned in the utilization review/case management for hospitals training at least: Once a Week ____ Once a Month ____ Once a Quarter ____ Once a Year _____ Never _____ 5.b If you had other training in utilization review/case management, to what extent did you find the materials during the HS2020 presented: Highly Complementary _____ Complementary _____ Redundant _____ Highly Redundant _____ Please indicate your agreement with the following statements: 5.c The curricula and training approach (using classroom and practicum approaches) was very effective in conveying and retaining new information and building skills regarding utilization and case management of care provided in hospitals? Strongly Agree ____ Agree ____ Don’t Agree/Disagree ____ Disagree ____ Strongly Disagree 5.d The information in the utilization review/case management for hospitals training is highly relevant for improving the medical quality of services provided to HIO clients. Strongly Agree ____ Agree ____ Don’t Agree/Disagree ____ Disagree ____ Strongly Disagree ____ 6. Training for Trainers (TOT) for Utilization Review/Case Management in Hospitals 6.a Since completing the TOT course have you provided training to other HIO staff on how to use the utilization review/ case management guidelines? No _____ Yes _____ If yes, how may courses have you led_____? What was the average number of trainees per course that you led _____?) Please indicate your agreement with the following statements: 6.b I felt sufficiently prepared through the TOT course to be able to train others to be competent in conducting utilization review and case management in HIO hospitals. Strongly Agree ____ Agree ____ Don’t Agree/Disagree ____ Disagree ____ Strongly Disagree ____ 7. Key Performance Indicators Please indicate your agreement with the following statements: 7.a In comparison to the content and teaching methods for HS2020’s utilization review and case management course, the content and teaching methods concerning the collection, calculation and analysis of Key Performance Indicators (KPIs) was: Much Less Clear___ Less Clear ___ About the Same ___ More Clear ___ Much More Clear ___ Answer the following only if you have ever collected, calculated and/or analyzed KPI data. 7.b Based on the training you received on Key Performance Indicators (KPIs), you are able to do the following without further training or assistance (check all that you are able to do as a result of receiving training). _____ Collect data needed for Key Performance Indicators _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 58 _____ Calculate Key Performance Indicators _____ Analyze trends in Key Performance Indicators _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 59 ANNEX G: DOCUMENTS REVIEWED AND REFERENCES PROJECT DOCUMENTS AND REPORTS Activity Standard Booklet; Case Management Cover Letter; Collective Report January 2011; Costing Methodology; Cost Savings October 2011; Donors Statistics 2010 Summary; Draft GANTT Chart Egypt USAID Edits; Egypt Health Insurance Organization EOP-HIO Brief July 2012; Egypt Health Systems Brief HS 2011; Egypt HIO PRESS RELEASE March 2012; Egypt Infection Control HS 2020 September 2012 Egypt IC; Egypt NHA 2007-2008 Report; Egypt Press Release July 12; Egypt Support for Mgmt Capacity Building; Egypt UM Manual November 2011; Eight Gov Rolling Out Activity; ETS Arabic Manual; ETS Case Study; ETS Manual August 2010; Expenditure Tracking for NHA; Final Draft HS 2020 Q4 Year 2; Final Egypt Brief June 2010; Final Egypt MCH Report MCDV FOR; Final EOP HIO Brief July 2012; Final EOP NHA Brief July 2012; Final Q1 Year 3 Report; Final Q3 Year 3 July 2009; Final WISN Case Study Egypt NR April 2012; FP Report FOR 2012; _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 60 FW HS 2020 Quarterly Review December 2009; Gharbia Report Final November 2010; Health Expenditure module in DHS6 31Dec2009; Health Insurance Law June 2012; Health Systems 2020 at Work in Egypt; Health Systems Monitoring and Evaluation Indicators March 2011; HHEUS 2010 Report; HIO SOW June 2008; HIO Work Plan Revised; HIO Q3 Year 5 September 2011; HIO Q3 Year 5 Summary September 2011; Hosp-Cost & Efficiency Studies-Key Findings; HS 2020 Egypt Work Plan July 2009; HS2020 MAARD Number 3 FY 2011; HS2020 MAARD Amend 2009; HS2020 M&E Indicators October 2010; HS 2020 M&E January 2011; HS 2020 M&E Plan August 2010; HS 2020 M&E Indicators June 2010; HS 2020 M&E Plan March 2007; HS2020 Q1 October-December 2008; HS2020 Q2 & Q3 April-June 2009; HS2020 Q4 July-September 2009; HS 2020 Quarterly Review July 2009; HS 2020 MOH Debrief June 2010; HS2020 Q4 Year 5 October-March 2012; HS 2020 Q3 April-June 2010 August 2010; HS 2020 Egypt Q4 Year 5 October 2011; HS 2020 Q1 Year 4 March 2010; HS 2020 Q4 Year 4; HS 2020 Q1 October-December 2009; HS 2020 Q4 2010; HS 2020 Q1 Year 5 January 2011; HS2020 Q2 Year 3 M&E Indicators May 2011; _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 61 HS2020 Q2 Year 3 Egypt Year 5 Global May 2011; HS2020 Q2 Year 3 Egypt Year 5 Global May 2011; HS2020 Q4 Year 5 July-September 2011; HS 2020 Q1 Year 3 M&E January 2011; HS 2020 M&E Indicators February 2011; HS 2020 Egypt Year 5 Work Plan October 2010-September 2011; HS20 Egypt Work Plan September 2008; HS 2020 Egypt Work Plan July 2009; HS 2020 January-April 2012 Plan and Budget; HS 2020 October-December 2011 Plan No Budget; HS 2020 Work Plan August 2010; HSR Brief April 2009; HSR Brief September 2010; HSR Brief December 2009; HSR Brief for Minister June 2008; Human Resources for Health Egypt 2009; Human Resources for Health Egypt Year 3; Insurance for Health Assessment Team; Intro Course Utilization and Case Management; Key Findings NHA 2008-2009; Legacy 6 Quality Egypt; Legacy 9 Human Capacity; Legacy 10 Health Sector Reform; MAARD Number 2 July 2008; MAARD Amend HS2020 July 2008; MAARD Amend HS2020 October 2010; MAARD Amend HS2020 June 2011; Medical Audit Guide for PHC and Hospitals December 2010; Medical Audit Guidelines Cover Letter; M&E Q4 Year 4; National Health Accounts Overview; NHA 2008-2009 Report; NHA3 Executive Summary February 2010; NHA3 PPT DOP February; _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 62 NHA Implications on Health Reform; NHA Report Final; NHA Q3 Year 5 September 2011; NHA Q3 Year 5 Summary September 2011; NHA Q3 Y5 Summary September 2011; OHP HS 2020 Evaluation Stakeholder Analysis; Q3 Year 4 HS 2020 Egypt August 2010; Quena Workforce Analysis Results; Red Sea Workforce Analysis Results; Resource Tracking Egypt 2009; Risk pooling Egypt 2009; Report HR Framework September 2008; Report Standards Setting; Report Vision and Next Steps January 2009; Resource Tracking Egypt Year 3; Revised HS 2020 Work Plan August 2010; Revised HS 2020 Work Plan August 2010; Risk Pooling Egypt Year 3; Roll Out the WP Model Summary Report; Seven Hospitals Cost Analysis Report; Status Report of HIO Activities June 2010; Strategic Guide for the Institutionalization of NHA; Success Story Utilization and Case Management Egypt July 2011; Talking Points HIO Managers; Three Months WP; TO Evaluation; User's Manual for Development a Workload Based Staffing Model ; Workforce Activity Standards; Workforce Planning User's Guide; Workforce Survey Process Summary; WP Brief EOP July 2012; WP Q3 Year 5 September 2011; WP Q3 Year 5 Summary September 2011; WP Brief June 2012; _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 63 WP Model Rollout Summary Report February 2012; Year 3 HS 2020 M&E Indicators May 2011 WORKS CITED 2008 Technical Assistance Plan, page 4 Antos, J., Hafez-Afifi, N. and S. el-Saharty. “Egypt’s Health Sector Reform and Financing Review”, World Bank, (unpublished), 2004. El Shalakani, Amr, “The New Social Health Insurance Law”, Technical Office of the Minister, Ministry of Health and Population, June 2012. El Zanaty, F., El Adawy, M., ElGhazaly, N., “Health Insurance Household Survey:, MOHP/HSRP, 2006. Executive Summary of the “Preventive Health Sector Assessment - The Current Landscape Of Maternal And Child Health Services In Egypt”.June 2012 Farag, Mahmoud (2012). National Health Accounts in Egypt Overview and Key Findings. Presentation USAID/Egypt. “Implementing National Health Accounts in Egypt”, HS 20/20, 2012 Nandakumar, A.K., “National Health Accounts 2008-2009 Executive Summary”, USAID, 2010 “Where is the money and what are we doing with it?”, World Bank, 2011National Health Accounts, 2007/08. _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 64 ANNEX H: HOW PROJECT OUTCOME PERFORMANCE MONITORING INDICATORS MIGHT HAVE LOOKED HS 20/20 EGYPT OUTPUT INDICATORS SUGGESTED OUTCOME INDICATORS IR: Operations IR 3: Health Systems budget and implementing priority programs more effectively IR 3.2: Health Systems use effective human resource management  Number of MOHP general and district hospitals in the three pilot governorates (Gharbia, Assuit and Luxor) in which the hospital workforce assessment was performed. Comment: Could be outcome if performed by counterparts. Also put under this sub-IR it is only output as the achieved does not reflect the “use” of “effective human resource management”. As expressed in the report “The workforce assessment and gap analysis results will help MOHP and hospital team to set workforce plans based on actual needs.” It is the workforce plans and the reallocation of staff that is an outcome.  Workforce assessment institutionalized at the National Information Center for Health and Population at central level.  A functioning HR department that uses workforce assessment for planning HR needs is established.  Number of developed hospital workforce improvement plans developed by WFP taskforce.  Number of hospital departments (specialties) that HS 20/20 has set Egyptian workforce standards for (activity standards). Comment: Output indicator IR: Capacity Building IR 4: Skills, knowledge and tools in health finance, governance, and operations support disease control effort IR 4.2: Developing countries have local sources of ongoing support in heath financing, governance and operations  Number of workforce planning training courses conducted for MOHP taskforce and governorate teams.  Number of governorate staff trained on workforce data collection, entry and validation.  Number of MOHP staff trained on the “Basics of Strategic Human Resource Management.” Comment: Output indicators  Number of trained central MOHP staff that have conducted workforce assessment and reported their results.  Number of governorates where trained MOHP-NICHP conducted workforce analysis.  Number of governorate and hospital level Information Center staff trained by MOHP-NICHP trainers in workforce data collection and entry _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 65 IR 4: Skills, knowledge and tools in health finance, governance, and operations support disease control effort IR 4.2: Developing countries have local sources of ongoing support in heath financing, governance and operations.  Number of people trained in medical audits for hospitals.  Number of people trained in medical audit for PHC units.  Number of people trained in the principles and practices of utilization and case management for hospitals.  Number of people trained in utilization /case management for referral and PHC’s activities.  Number of people trained in selected pilot hospitals on the medical management activities including the implementation of the medical audit program and utilization and case management.  Number of HIO financial sector’s staff trained in budget and financial forecasting.  Number of HIO financial sector’s staff who got on-the-job training to develop financial reports.  The number of medical mock audits conducted in health facilities. Comment: Output indicators. Mock audits are part of the practical training. Medical Management  Number of certified HIO co-trainers on utilization/case management  Number of certified HIO auditors  Case Management Office institutionalized in HIO at central level  Number of HIO branches with functioning Case Management Offices  Percentage of HIO hospitals with functioning Case Management Offices  Number of Hospitals where performance monitoring systems established according to utilization and case management guidelines  Number of HIO targeted hospitals with a reduced waste of medical resources through reduction in re-admission cases for the same diagnosis within 30days by 5% and through reduction of the use of IV antibiotic by 30% guided by the medical necessity guidelines.  Number of audited health facilities that produced corrective action plans to address each identified issue.  Financial Management  Number of HIO branches that developed annual financial and costing reports using the principles of cost accounting and new methodology.  Comprehensive cost analysis study of all departments of an HIO hospital conducted by HIO Head Quarter Financial Department.  HIO planning, budgeting and control based on financial forecasts of future revenues and expenses is institutionalized in the financial department  Number of HIO financial departments that apply principles of cost-accounting to price services and control costs  Number of HIO branch level financial departments that use financial data to prepare cost analysis reports IR: Financing IR 1: Improved financing for PHN priority services IR: Operations IR 3: Health Systems budget and implementing priority programs more effectively  Number of hospital costing analyses conducted.  Comment: Output indicator. Could be an outcome indicator if stated as conducted by counterparts and placed under IR 4.  Costing team established in DOP conducts, analyzes and reports at least one costing study of health services provided in hospitals’ outpatient and inpatient departments. _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 66 IR: Capacity Building IR 4: Skills, knowledge and tools in health finance, governance, and operations to support the disease control effort  Number of staff from the MOHP’s Department of Planning, National Health Accounts taskforce, Ministry of Finance and Medical Syndicate trained on how to interpret NHA findings.  Number of MOHP’s Department of Planning staff trained on costing hospitals through workshops.  Number of on-the-job training sessions conducted to estimate costs of health services provided in hospitals’ outpatient and inpatient departments.  Number of people from the MOHP oriented on the importance and methodology of ETS.  Number of on-the-job training sessions conducted on the ETS for the MOHP’s Department of Planning staff.  Number of primary health care facilities implementing the ETS methodology. Comment: Output indicators. The last one could be an outcome indicator if performed by counterparts.  Expenditures of IC, MCH and FP health programs tracked by DOP using the HS 20/20 developed ETS methodology for tracking the flow of expenditure by program.  Performance based budgets developed for IC, FP and MCH.  NHA institutionalization action plan developed by MOHP  Hospital costing institutionalized in MOH Department of Planning (definition of institutionalized) IR: Governance IR 4: Skills, Knowledge and tolls in health finance, governance and operations support disease control efforts  Number of facilities that were assessed as part of HS 20/20’s assessment to evaluate the performance of three main MOHP Preventive Sector health programs (FP, MCH, and IC). This is a progress indicator. To conduct a MOHP Preventive Sector Assessment the project had to assess a specific number of facilities. The above indicator shows how many of these were done. The activity was requested by the Minister as an output for their use. _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 67 ANNEX I: REPORT ON THREE FOCUS GROUP DISCUSSIONS WITH HS 20/20 EGYPT TRAINING PARTICIPANTS Introduction Focus Group Discussions (FGDs) were conducted with a sample of the personnel trained in connection with HS20/20 Egypt interventions to gain insights into how participants perceived their training experiences; what they felt were critical gaps in their knowledge/skills that were addressed (or not addressed) by the training; and their suggestions of ways by which future training and efforts to build capacity in their organizations could be improved. Three FGDs were conducted, drawing purposive samples of personnel trained in different intervention areas of the HS 20/20 project. Individuals were identified by managers of the departments involved in the training, and included individuals who were: (1) participants in the training who were still employed with the department; (2) assigned to work in areas where they could use of the knowledge/skills for which they were trained; and (3) were available to participate in one of the FDGs (one scheduled on a Saturday, and two on a Sunday). The three FDGs involved:  Workforce planning (MOHP)  Medical management (HIO)  Financial management (HIO) The distribution of participants of FGDs is shown in the below graphs: Results - Workforce Planning The participants of workforce planning trainings were selected from both the information centers at the central and governorate levels in addition to supervisors at the district level. Training focused on the use of data collection forms which were continuously revised and updated based on feedback from the field while being piloted-tested in Assuit, Gharbia and Luxor Governorates. Participants from the Governorates received more days of training than did their counterparts who were trained in other Governorates after the pilot phase. Training focused on standardization of concepts/terminology, data collection, data verification and data entry using WHO-adapted software, and one participant was additionally trained in data analysis. _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 68 All 8 participants agreed that the training met their expectations. The content was easy to understand, there was no language barrier and the trainer was very competent. They all added that they are currently using the same data collection format, which they all believe to be comprehensive and user- friendly. They all said that they have been applying their acquired knowledge and skills to carry on their workforce assessments in the roll out Governorates even after phasing out of the HS 20/20 project. Based on the results of the exam conducted at the end of training and on-going observation throughout the training, some trainees were selected to act as trainers. They clarified that their professional achievement was assessed by interactions and group work throughout the training and by supervisory field visits after the training. All the participants confirmed absence of any hindering factors during the training and revealed that the harmony among selected trainees, the established rapport with the trainer and their appreciation of the validity of the training content to their work were motivating factors in their commitment to the training and subsequent implementation. When asked about their future training needs, the following were listed:  Communication skills  Management skills  Advanced computer courses  Data analysis  Data management  Data presentation  Developing data collection instruments Results - Medical management Selection criteria set for the attendees of Medical Management training included doctors and nurses familiar with the concept of quality. Training focused on medical management including length of stay, incurred costs and their relation with the diagnosis, medical audits and how to revise files of cases for completeness and accuracy of documentation. The theoretical training was complemented by practical training in hospital settings. Twelve out of 13 (92%) participants stated that the content of the training was completely new to them, one participants had previously attended a course focusing on the medical auditing at the American University in Cairo. All participants confirmed that content was easy to understand, there was no language barrier as the training team comprised both local and international consultants who were both assessed by the trainees to be very competent. Participants clarified that their professional achievement was assessed by interactions and group work throughout the training and by supervisory field visits after the training. Based on the results of the exam conducted at the end of the training and ongoing observations during the training, some trainees were certified as trainers and charged with the responsibility of sharing their acquired knowledge and skills in a cascade fashion with other co-workers in the workplace. The Gharbia team (46% of participants in the FDG) added that they are currently applying acquired knowledge and skills of medical management while other participants from Cairo’s 6 October Hospital, (54% of participants in the FDG) said that they were applying their acquired knowledge and skills only sporadically. All the participants agreed that one of the main benefits of the training received was the shift in their thinking and mind set about the concept of supervision. They better understood the _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 69 distinction between a routine surveillance visit compared to supervision for the purpose of mentoring and as a tool to improve the efficiency and effectiveness of work performed. All of the FDG participants said that they witnessed a drop in costs, length of stay, and use of intravenous antibiotics after applying acquired knowledge and skills with ultimate decrease of the expenditures in the units where utilization and case management were applied. All the participants confirmed absence of any hindering factors during the trainings and revealed that the harmony among selected trainees, the established rapport with the trainer and the practical application of the training content to their work were motivating them to commit to the trainings and subsequent implementation. There was close to unanimous agreement among participants that standardized clinical guidelines should be used at all HIO service providers to ensure smooth functioning of services in their facilities. When asked about the potential role of HIO under authority of the new (anticipated) health insurance law (where MOHP will play the role of payer and HIO will play role of contracted provider)29, Gharbia participants stated that they should be transferred to the payer side to best utilize the knowledge and skills they have acquired through the HS 20/20 Egypt training. When asked about their future training needs, the following were listed: • Referral system (6 October team) • Refresher training of medical management (Gharbia team) • Accreditation standards • Accounting for non-accountants • Management skills • Quality • Training of trainers Financial management Selection criteria set for the attendees of financial management training included those working in the financial department responsible for costing. Training focused on the costing of health services located on industrial premises, not public facilities. The theoretical training was supposed to be complemented by practical training in hospitals but this was interrupted by the 25 January Revolution and did not occur. All participants agreed that the content of the trainings was unrelated to their work because it focused on profit organizations not public facilities. Yet, they all stated that the content was easy to understand and there was no language barrier being presented by local consultant (Dr. Mohamed Al Lotfy) who was judged by the participants to be very competent. Participants added that the knowledge and skills acquired in the training were not related to their work but were beneficial on an individual basis. All felt that the training should have focused on their specific needs for costing in non-profit public service delivery facilities. They added that future training should involve, in addition to data collectors like themselves, 29 This interpretation of the impending law is not accurate. As proposed, the MOHP would be a regulator, and HIO would be divided between a newly authorized Curative Care Organization (as provider), and a newly formed National Health Insurance Organization (as payor). _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 70 their managers so that managers can better understand the role that costing can play in informing policy and budgetary decision making. Said one of the participants (an account), “I am like the flash light for the manager shedding light on the correct decision from financial point.” When asked about their future training needs, the following were listed: • Costing in hospitals • Financial analysis • Costing and decision making • Forecasting and budget planning CONCLUSIONS HS 20/20 Egypt made an adequate strategic shift to re-focus its approaches to training and capacity building away from an institutional framework once the Leadership Academy was dropped from the project. • HS 20/20 Egypt provided good quality training delivered by competent trainers – even in the instance where the training program on “costing” was not properly targeted. • HS 20/20 facilitated effective implementation of skills acquired upon completion of training (and likely enhanced the effectiveness of training), by establishing formal procedures for follow-up after training. Trainees are most likely to use their acquired knowledge and skills in the work setting when they have positive training experiences (“harmonious and without hindering factors”), and training content is specifically focused on their job assignments (such as with the utilization and case managers); theoretical and generic training programs may not be as effective. _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 71 FOCUS GROUP DISCUSSION GUIDE HIO ACCOUNTING/FINANCIAL MANAGEMENT TRAINEES Introduction: HS 20/20, team and final evaluation, purpose of focus group discussion Ethical consideration: voluntary participation/consent, confidentiality Questions: 1. Use flip chart to obtain information on posting of participants at time of project (2009-2011) Junior Acct/Fin Staff (#) Senior Acct/Fin Staff (#) HIO HQ HIO Branch HIO Hospital/Other 2. Use flip chart to obtain information on participation of trainees in courses of HS2020 Junior Acct/Fin Staff (#) Senior Acct/Fin Staff (#) HIO Exp Analysis & Budget Forecasting Accrual Accounting (vs cost accounting) Step-down Cost Estimation of Costs Other (what?) 3. Prior to the training of HS2020, did you have formal education (e.g. tech school/university) or certificate training in any of the areas related to the training? Which ones? 4. Do you feel that the technical content of the courses you participated was relevant to your your interests/needs for training related to your job at HIO? Explore why/why not, e.g. Course/curricula planning failed to adequately assess HIO’s overall financial/accounting training needs? Course/curricula development planning failed to include assessment of needs for knowledge/skill development and thus was redundant? Courses provided shallow breadth and should have aimed to provide depth in one key area – which one? Other? _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 72 5. Where the teaching methods (lectures, assignments to work with real HIO data, mentoring visits w/Professor Lofti) employed effective in: i) development of new knowledge ii) development of capacity to use that knowledge on the job Explore why/why not? 6. Did any of the courses you participated in measure your acquisition and/or ability to utilize the course knowledge after the end of the course? If yes, how was this assessed? 7. Are you currently using any of the content presented in the courses in your current work with HIO? If yes, which course content? If no, why not? Probe as to whether impediment to use of new skills was lack of sufficient training, and/or lack of HIO management decisions to change systems, and/or lack of IT, and/or other factors. 8. If the future, what content areas do you believe the training office of HIO should prioritize regarding addressing your needs for improvement of knowledge and skills in accounting and financial management current and future tasks? _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 73 FOCUS GROUP DISCUSSION GUIDE HIO MEDICAL MANAGEMENT TRAINEES Introduction of HS20/20 Ethical consideration: voluntary participation, confidentiality, consent of participation Focus of Discussion: The following questions pertain to utilization review and case management training. Please respond according to your experience before, during and after that course. If you did not participate in that course, please indicate so now. Questions: 1. Prior to the course did you have any training or work experience related to utilization management/case review? Explore what prior training/work experience trainees had and their views on the quality and value of that training, e.g. training. 2. Did the content of the course fall short/meet/exceed your expectations? (Explore reasons for degree of satisfaction/dissatisfaction w/content). What are your views concerning the organization of the training course - including lectures and practicum - regarding developing your understanding and your retention of course materials? 3. How was your performance in the training measured (nb: methods were practicum verbal presentations/feedback and written exam)? Would one approach have been sufficient or were performance evaluation methods reinforcing? Do you think the trainer’s feedback and grade were impartial and fair? 4. Is the technical content of the course relevant to your work now? What parts of the curricula are most helpful in your current work and why? 5. Are there any structural (e.g. automatization), managerial (e.g. lack of managerial and/or medical staff interest and support) and/or other aspects that hinder applying what you learned in this course in meet your current position responsibilities? (note: collate responses by type of current position) 6. What are your priorities for future training in utilization review/case management or other quality improvement subjects? 7. Do you believe HIO should develop a comprehensive/on-going training program related to quality improvement? If yes, what form should such a program take (e.g. stipends for basic training at AUC and CU, support for basic training via HIO, development of training programs for quality teams)? Thank the participants for their time. _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 74 EGYPT HS20/20 END OF PROJECT EVALUATION Medical Management (Hospital/PHC Audit, Utilization/Case Management) Focus Group – Questionnaire INSTRUCTIONS: Please respond to the following according to the instructions in each area of the questionnaire. CONFIDENTIALITY: Your responses on this form will be treated as confidential. Your name should not be written on the form. Further individual responses will not be reported nor shared with HIO nor USAID. The information from the questionnaires for all group members will be aggregated analyzed along with the information from the focus group discussions. CONSENT: You may choose to not answer any or all of the questions. 1. Educational Background 1.a Professional clinical degree (s) (check all that apply) MD _____ Nurse ____ Pharmacist _____ Dentist _____ 1.b Other professional degrees (check all that apply) Public Health (MPH/DrPH ____ MBA ____ Other (describe)____________ ______________ _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 75 1.c Health quality-related course(s): No _____ Yes _____ If yes, Certificate course _____; Degree course _____ When completed (mm/yy)?___________ 1.d Health Management and/or Finance-related course(s) No _____ Yes _____ If yes, Certificate course _____; Degree course _____ When completed (mm/yy)? __________ 2. Position Assignment within HIO 2.a At what Level of HIO did you work when you participated in your first HS2020 course? HIO HQ ____ HIO Branch Office ____ HIO Own Hospital ____ HIO Contracted Hospital ____ 2.b At what Level of HIO do you currently work? HIO HQ ____ HIO Branch Office ____ HIO Own Hospital ____ HIO Contracted Hospital ____ 2.c Please indicate if you have ever worked in any/all of the following positions and for how long: Quality Coordinator? No _____; Yes _____ (how long _____ months?) _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 76 Infection Control Coordinator? No _____; Yes _____(how long _____ months?) Case Management Office? No _____; Yes_____(how long _____ months?) 3. Participation/Completion of HS2020 Medical Management Trainings Please indicate (X) all courses/workshops sponsored by HS2020 that you participated in and also indicate those that you completed/received a certificate (graduated) and the family name of the lead trainer. Participated Graduated Trainer Audit for Hospitals _________ ________ __________ Audit for PHC Units _________ ________ __________ Utilization/Case Management (UCM) Hospitals _________ ________ __________ Training of Trainers for UCM - Hospitals _________ ________ __________ Policies, Procedures, Outcome Indicators of CMO @ Hospital Level _________ ________ __________ Key Performance Indicators (KPI) _________ ________ __________ Please Respond to the Questions/sub-questions below if you received training on the subject matter addressed. Skip Questions/sub-questions when you did not receive the specific training in that specific area. _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 77 4. Training for Audit of Hospital 4.a In my present position, I am utilizing the skills learned in the audit course for hospitals at least: Once a Week ____ Once a Month ____ Once a Quarter ____ Once a Year _____ Never _____ Please indicate your agreement with the following statements 4.b The curricula and training approach (using classroom and practicum approaches) was very effective in conveying and retaining new information and building skills regarding audit of hospitals? Strongly Agree ____ Agree ____ Don’t Agree/Disagree ____ Disagree ____ Strongly Disagree 4.c The information in the audit course (s) for hospitals is highly relevant for improving the medical quality of services provided to HIO clients. Strongly Agree ____ Agree ____ Don’t Agree/Disagree ____ Disagree ____ Strongly Disagree ____ 5. Training for Utilization Review/Case Management in Hospitals 5.a In my present position, I am utilizing the skills learned in the utilization review/case management for hospitals training at least: Once a Week ____ Once a Month ____ Once a Quarter ____ Once a Year _____ Never _____ _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 78 5.b If you had other training in utilization review/case management, to what extent did you find the materials during the HS2020 presented: Highly Complementary _____ Complementary _____ Redundant _____ Highly Redundant _____ Please indicate your agreement with the following statements: 5.c The curricula and training approach (using classroom and practicum approaches) was very effective in conveying and retaining new information and building skills regarding utilization and case management of care provided in hospitals? Strongly Agree ____ Agree ____ Don’t Agree/Disagree ____ Disagree ____ Strongly Disagree 5.d The information in the utilization review/case management for hospitals training is highly relevant for improving the medical quality of services provided to HIO clients. Strongly Agree ____ Agree ____ Don’t Agree/Disagree ____ Disagree ____ Strongly Disagree ____ 6. Training for Trainers (TOT) for Utilization Review/Case Management in Hospitals 6.a Since completing the TOT course have you provided training to other HIO staff on how to use the utilization review/ case management guidelines? No _____ Yes _____ If yes, how may courses have you led_____? _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 79 What was the average number of trainees per course that you led _____?) Please indicate your agreement with the following statements: 6.b I felt sufficiently prepared through the TOT course to be able to train others to be competent in conducting utilization review and case management in HIO hospitals. Strongly Agree ____ Agree ____ Don’t Agree/Disagree ____ Disagree ____ Strongly Disagree ____ 7. Key Performance Indicators Please indicate your agreement with the following statements: 7.a In comparison to the content and teaching methods for HS2020’s utilization review and case management course, the content and teaching methods concerning the collection, calculation and analysis of Key Performance Indicators (KPIs) was: Much Less Clear____ Less Clear _____ About the Same ____ More Clear ____ Much More Clear ____ Answer the following only if you have ever collected, calculated and/or analyzed KPI data. 7.b Based on the training you received on Key Performance Indicators (KPIs), you are able to do the following _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 80 without further training or assistance (check all that you are able to do as a result of receiving training). _____ Collect data needed for Key Performance Indicators _____ Calculate Key Performance Indicators _____ Analyze trends in Key Performance Indicators _____ Utilize KPI data to advise HIO/hospital managers on priority areas to improve quality _____ Utilize KPI data to advise HIO/hospital managers on priority areas to reduce cost _____ Other (please elaborate _____________________________________________________________) 7.c If you have ever had a responsibility for data collection, calculation or analysis of Key Performance Indicators which of the following would have been helpful to improve your performance with these tasks (please check all that apply)? _____ Additional Training _____ Technical Assistance _____ Additional Staff in the Quality Unit/CMO _____ Additional Computer/IT equipment _____ Other (please elaborate _____________________________________________________________) Focus Group Discussion Guide HIO Medical Management Trainees Introduction of HS20/20 Ethical consideration: voluntary participation, confidentiality, consent of participation _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 81 Focus of Discussion: The following questions pertain to utilization review and case management training. Please respond according to your experience before, during and after that course. If you did not participate in that course, please indicate so now. Questions: 8. Prior to the course did you have any training or work experience related to utilization management/case review? Explore what prior training/work experience trainees had and their views on the quality and value of that training, e.g. training. 9. Did the content of the course fall short/meet/exceed your expectations? (Explore reasons for degree of satisfaction/dissatisfaction w/content). What are your views concerning the organization of the training course - including lectures and practicum - regarding developing your understanding and your retention of course materials? 10. How was your performance in the training measured (nb: methods were practicum verbal presentations/feedback and written exam)? Would one approach have been sufficient or were performance evaluation methods reinforcing? Do you think the trainer’s feedback and grade were impartial and fair? 11. Is the technical content of the course relevant to your work now? What parts of the curricula are most helpful in your current work and why? 12. Are there any structural (e.g. automatization), managerial (e.g. lack of managerial and/or medical staff interest and support) and/or other aspects that hinder applying what you learned in this course in meet your current position responsibilities? (note: collate responses by type of current position) 13. What are your priorities for future training in utilization review/case management or other quality improvement subjects? 14. Do you believe HIO should develop a comprehensive/on-going training program related to quality improvement? If yes, what form should such a program take (e.g. stipends for basic training at AUC and CU, support for basic training via HIO, development of training programs for quality teams)? Thank the participants for their time. Focus Group Discussion Guide HIO Accounting/Financial Management Trainees Introduction: HS20/20, team and final evaluation, purpose of focus group discussion Ethical consideration: voluntary participation/consent, confidentiality Questions: 15. Use flip chart to obtain information on posting of participants at time of project (2009-2011) Junior Acct/Fin Staff (#) Senior Acct/Fin Staff (#) _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 82 HIO HQ HIO Branch HIO Hospital/Other 16. Use flip chart to obtain information on participation of trainees in courses of HS2020 Junior Acct/Fin Staff (#) Senior Acct/Fin Staff (#) HIO Expenditure Analysis & Budget Forecasting Accrual Accounting (as compared to cost accounting) Step-down Cost Estimation of Average Costs of Services Other (what?) 17. Prior to the training of HS2020, did you have formal education (e.g. tech school/university) or certificate training in any of the areas related to the training? Which ones? 18. Do you feel that the technical content of the courses you participated was relevant to your your interests/needs for training related to your job at HIO? Explore why/why not, (e.g. Course/curricula planning failed to adequately assess HIO’s overall financial/accounting training needs? Course/curricula development planning failed to include assessment of individuals needs for knowledge/skill development and thus was redundant? Courses provided shallow breadth and should have aimed to provide depth in one key area – which one? Other? 19. Where the teaching methods (lectures, assignments to work with real HIO data, mentoring visits w/Professor Lofti) employed effective in: o i) development of new knowledge o ii) development of capacity to use that knowledge on the job Explore why/why not? 20. Did any of the courses you participated in measure your acquisition and/or ability to utilize the course knowledge after the end of the course? If yes, how was this assessed? 21. Are you currently using any of the content presented in the courses in your current work with HIO? If yes, which course content? If no, why not? Probe as to whether impediment to use of new skills was lack of sufficient training, and/or lack of HIO management decisions to change systems, and/or lack of IT, and/or other factors. _____________________________________________________________________________________ Performance Evaluation of the HS 20/20 Egypt Project 83 22. If the future, what content areas do you believe the training office of HIO should prioritize regarding addressing your needs for improvement of knowledge and skills in accounting and financial management current and future tasks?