June 2017 This document was produced at the request of the United States Agency for International Development. It was prepared independently by Social Impact under the Liberia Strategic Analysis activity. USAID/LIBERIA COLLABORATIVE SUPPORT FOR HEALTH (CSH): MIDTERM PERFORMANCE EVALUATION REPORT 2 3 USAID/LIBERIA COLLABORATIVE SUPPORT FOR HEALTH (CSH): MIDTERM PERFORMANCE EVALUATION REPORT JUNE 2017 Liberia Strategic Analysis Contract No: AID-669-C-16-00002 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. 4 ACKNOWLEDGMENTS The Liberia Strategic Analysis evaluation team, Peter Bachrach, Endris Seid, Kelechi Udoh, Julius Lekpeh, and Anthony Tamba, would like to extend appreciation to the personnel of the Ministry of Health and the County Health Teams of Nimba, Bong, Margibi, and Grand Bassa counties; to the staff of the key regulatory agencies of LMDC, LBNM, and LPB; and to the collaborating partners. Without their willingness to speak candidly, the richness of the findings would have been greatly reduced. The evaluation team is also grateful for the time and information provided by the CSH team led by Chief of Party, Mr. Sjoerd Postma. Ms. Carla Trippe and staff of Liberia Strategic Analysis ably handled the administrative and logistical aspects of the field work as well as the editing and production of the final evaluation report. Our appreciation also goes to Ms. Sinu Kurian and the USAID/Liberia Health Team for their oversight, support and critical review of this report. 5 TABLE OF CONTENTS ACKNOWLEDGMENTS.................................................................................................................................................4 ACRONYMS.......................................................................................................................................................................7 EXECUTIVE SUMMARY................................................................................................................................................10 I. INTRODUCTION.......................................................................................................................................................15 EVALUATION PURPOSE.........................................................................................................................................15 EVALUATION QUESTIONS ...................................................................................................................................15 II. PROJECT BACKGROUND .....................................................................................................................................16 III. EVALUATION METHODS AND LIMITATIONS..............................................................................................17 IV. FINDINGS...................................................................................................................................................................19 V. CONCLUSIONS.........................................................................................................................................................32 VI. RECOMMENDATIONS...........................................................................................................................................35 ANNEX 1: EVALUATION STATEMENT OF WORK...........................................................................................37 ANNEX 2: LIST OF DOCUMENTS REVIEWED ....................................................................................................49 ANNEX 3: LIST OF PERSONS CONSULTED.........................................................................................................54 ANNEX 4: DATA COLLECTION INSTRUMENTS...............................................................................................58 ANNEX 5: CONFLICT OF INTEREST DISCLOSURE ..........................................................................................78 ANNEX 6: CSH CONSULTANTS DEPLOYED......................................................................................................82 ANNEX 7: CSH PROJECT MATERIALS PRODUCED..........................................................................................92 ANNEX 8: CSH TRAININGS PROVIDED ...............................................................................................................99 ANNEX 9: CSH INDICATOR STATUS................................................................................................................. 101 ANNEX 10: CSH CAPACITY ASSESSMENT SCORES BY COUNTY........................................................... 109 ANNEX 11: CSH SERVICE DELIVERY INDICATORS....................................................................................... 111 ANNEX 12: PRIORITY ACTIVITIES BY COMPONENT................................................................................... 114 ANNEX 13: NATIONAL RESILIENT PLAN PRIORITIES.................................................................................. 121 ANNEX 14: ANALYSIS OF CSH’S M&E FRAMEWORK................................................................................... 124 ANNEX 15: PROPOSED CSH INDICATOR REVISIONS................................................................................. 126 6 TABLE 1: Summary of Interviews Conducted and Interviewees Met.................................................................18 TABLE 2: Planned CSH Activities by Component...................................................................................................19 TABLE 3: Status of Deliverables for FY15 and FY16 ..............................................................................................20 TABLE 4: Achievement of Planned Targets by Component .................................................................................20 TABLE 5: Comparison of Capacity Development Scores (since 2012)..............................................................21 TABLE 6: Progress in Selected QI Indicators............................................................................................................22 TABLE 7: Patient Satisfaction in Selected Facilities..................................................................................................23 TABLE 8: Cumulative Number of Personnel Files Completed by County........................................................24 TABLE 9: LMIS Reporting Rates and Proportion of Facilities with Stock-outs ................................................25 TABLE 10: Table 10: Financial Management Assessment Scores.........................................................................25 TABLE 11: Progress in Mitigating PFM Risks.............................................................................................................26 TABLE 12: Proportion of Facilities Providing Timely and Complete HMIS Reports.......................................26 TABLE 13: Assessment of Component Success.......................................................................................................27 TABLE 14: Potential Short, Medium, and Long-term Contributions to the Nine Strategic Areas..............28 TABLE 15: Comparison of Component Success Rankings....................................................................................31 ACRONYMS ACCEL Academic Consortium to Combat Ebola in Liberia ANC Antenatal Care BCC Behavior Change Communication CDCS Country Development Cooperation Strategy CHAI Clinton Health Access Initiative CHB County Health Board CHC Community Health Committee CHDC Community Health Development Committee CHO County Health Officer CHT County Health Team CME Continuing Medical Education CPD Continuing Professional Development CSH Collaborative Support for Health DEN-L Development Education Network of Liberia DHIS2 District Health Information System DHO District Health Officer DHT District Health Team DIG Development Innovations Group eLMIS Electronic Logistics Management Information System EPHS Essential Package of Health Services ETU Ebola Treatment Unit EU European Union EUV End User Verification EVD Ebola Virus Disease FARA Fixed Amount Reimbursement Agreement FGD Focus Group Discussion FP Family Planning FY Fiscal Year GEMS Global Environmental Management Support GFATM The Global Fund to Fight AIDS, Tuberculosis and Malaria GOL Government of Liberia HCF Health Care Finance HFU Health Financing Unit HICD Human and Institutional Capacity Development HIS Health Information Systems HMER Health Management Information Systems, M&E and Research HMIS Health Management Information System HR Human Resources HRH Human Resources for Health HRM Human Resources Management HSS Health Systems Strengthening 8 IHI Institute for Healthcare Improvement IHP+ International Health Partnership iHRIS Integrated Human Resources Information System IP Implementing Partner IPC Infection Prevention and Control IRC International Rescue Committee JISS Joint Integrated Supervision and Support JSI John Snow, Inc. KII Key Informant Interview L&G Leadership and Governance LBNM Liberian Board of Nursing & Midwifery LHEF Liberia Health Equity Fund LMDC Liberia Medical and Dental Council LMG Leadership, Management, and Governance LMHRA Liberia Medicine and Health Regulatory Authority LMIS Logistics Management Information System LSA Liberia Strategic Analysis MCH Maternal and Child Health M&E Monitoring and Evaluation MCSP Maternal Child Survival Program MOH Ministry of Health MPW Ministry of Public Works MSH Management Sciences for Health NDS National Drug Store NHSWPP National Health and Social Welfare Policy and Plan 2011-2021 NGO Non-Governmental Organization OIC Officer-in-Charge PACS Partnership for Advancing Community-Based Services PBC Performance-Based Contract PBF Performance-Based Financing PFM Public Financial Management PFMRAF Public Financial Management Risk Assessment Framework PIDS Performance Indicator Database System PPE Personal Protective Equipment PRISM Performance of Routine Health Information System Management in Liberia QA/QI Quality Assurance and Quality Improvement QMU Quality Management Unit R4D Results for Development RBHS Rebuilding Basic Health Services RDF Revolving Drug Fund RH Reproductive Health SCM Supply Chain Management SCMU Supply Chain Management Unit SI Social Impact, Inc. 9 SMT Senior Management Team (MOH) SOP Standard Operating Procedure SOW Scope of Work SQS Safe and Quality Services STTA Short Term Technical Assistance TA Technical Assistance TOR Terms of Reference TOT Training of Trainers TTM Trained Traditional Midwife TWG Technical Working Group UNFPA United Nations Population Fund UNICEF United Nations Children’s Fund USAID United States Agency for International Development USG United States Government WASH Water, Sanitation, and Hygiene WB World Bank WHO World Health Organization 10 EXECUTIVE SUMMARY EVALUATION PURPOSE AND QUESTIONS USAID engaged the Liberia Strategic Analysis (LSA) activity to conduct a midterm performance evaluation of the Collaborative Support for Health (CSH) project in Liberia. The evaluation’s purpose was to provide an independent and in-depth examination of the overall progress of CSH. The evaluation was expected to: (i) document achievements, performance issues, and constraints related to activity implementation and effectiveness; (ii) analyze results and lessons learned from implementation; and (iii) provide succinct, actionable recommendations to determine which component(s) of CSH to scale up, modify, or re-design for option years and/or in other ongoing related activities or future procurements. The evaluation questions addressed the project’s outcomes and sustainability. With respect to the project’s outcomes, the evaluation was expected to analyze: • To what extent has progress been made in achieving the seven core systems strengthening objectives under CSH? • What are some of the value-added components of CSH (and the activity staff), if any, to the Ministry of Health, donor community or USAID that were not explicitly outlined in the award? • Are there new or emerging needs for health systems strengthening not within the current CSH scope? With respect to the project’s sustainability, the evaluation was expected to examine: • To what extent have CSH’s approaches supported the Ministry of Health (MOH) in fulfilling its core health systems functions? Comment on MOH’s improved capacity, stewardship and ownership of the core health systems functions, the quality of staff, internal systems, robustness of M&E system PROJECT BACKGROUND USAID and Management Sciences for Health (MSH) signed a contract for implementation of CSH on February 27, 2015. The contract was in the amount of $21.5 million and covered the period from February 27, 2015 to February 27, 2018. It provided for an additional two option years (through February 2020) in the amount of $10.9 million. CSH’s overall objective is to “improve the health status of Liberians” as expressed in USAID/Liberia’s Health Strategy and Country Development Cooperation Strategy (CDCS) Development Objective 3. Operationally, CSH’s was expected to: (i) enhance the capacity of the MOH and County Health Teams (CHTs) of Bong, Lofa, and Nimba counties; and (ii) ensure the necessary health systems and standards are in place to provide high quality services (implementation of the Essential Package of Health Services) for all Liberians. CSH’s anticipated results were closely aligned with the objectives of the National Health Policy and Plan (2011-2021) and the Investment Plan for Building a Resilient Health System (2016-2021). Project design built on the previous USAID-funded health project, Rebuilding Basic Health Services (RBHS) Project, which was implemented between 2008-2015. Project start-up was affected by the Ebola epidemic, which modified the context for and affected the pace of execution. CSH had both to: (i) address an altered partner landscape as many new actors had begun operations during the epidemic; and (ii) replace already recruited staff in the post-Ebola environment. Project implementation to date has had to adapt to: (i) new or revised national policies, plans, and strategies; and (ii) the shifting 11 financial and technical landscape which necessitates continuous coordination with other partners and frequent modification of the specific activities defined in the original Statement of Work (SOW). EVALUATION DESIGN, METHODS AND LIMITATIONS A team of five evaluators worked in Liberia from April 3 to May 5, 2017. In addition to interviews in Monrovia, the evaluation team visited four counties with the intention of comparing results in two CSH counties (Bong and Nimba) and two non-CSH counties (Margibi and Grand Bassa). Quantitative information was collected from: (i) a desk review of program and project documents; (ii) an analysis of complementary information from past evaluations and current databases; and (iii) questionnaires on capacity (at CHT level) and patient satisfaction (from hospital and clinic levels). Qualitative information was gathered through Key Informant Interviews (KII) and Focus Group Discussions (FGD) with project beneficiaries (at central, county, and district levels), collaborating organizations, and donor agencies. Throughout the evaluation, the team was concerned with three potential limitations: (i) the broad reach and relatively short duration of the evaluation; (ii) the criteria for measuring capacity; and (iii) the possibility of attributing any changes in capacity to CSH’s interventions. The team was able to manage the schedule but found only partial solutions for the issues of measurement and attribution. The lack of data for certain program indicators and the quality of some of the data (especially for the QA/QI dashboard) were unanticipated limitations. KEY FINDINGS AND CONCLUSIONS Outcome Question 1: To what extent has progress been made in achieving the seven core systems strengthening objectives under CSH? The evaluation presented an overview of CSH’s progress in producing its planned outputs, achieving its intended program results and capacity outcomes, and contributing to improved health status. • Despite the deployment of significant technical assistance, training, and materials, an assessment of CSH’s work plans for FY15, FY16, and FY17 shows that planned activities have only been partially achieved. Further, these activities were not accompanied by the expected results, as planned deliverables have only been partially achieved over the first two years of project implementation. • Assessment of CSH’s program targets indicates that they have been only partially achieved. • Assessment of management capacity shows that progress has been neither uniformly positive in the CSH counties nor markedly different from the non-CSH counties. • Though CSH’s mandate does not include health service delivery, indicators relying on MOH’s DHIS2 show slow improvement. Outcome Question 1.1: Which components have been most successful in meeting the overall objectives of CSH? At what level? Provide a prioritized list of components. Based on measurable, positive results, the Health Care Finance (HCF), Human Resources (HR), and Health Information System (HIS) components have been the most successful in meeting the overall objectives of CSH. Measurable progress was documented on process results for HR, HIS, and Supply Chain Management (SCM) and on outcome results for HCF and Quality Assurance/Quality Improvement (QA/QI); qualitative information was available for the Ministry of Public Works (MPW) and leadership and governance (L&G). The process results for HR (records and scholarship management) and HIS (timely reporting) were generally positive, while those for SCM (timely reporting) were mixed. The outcome results for HCF were positive (for both financial 12 management and risk mitigation) but mixed for QA/QI. Outcome results (stock-outs) for SCM were also available but predate the implementation of the revised Logistics Management Information System (LMIS) and are not directly related to this activity. Outcome Question 1.2: If CSH will receive option years, which areas should be strengthened and which areas could be discontinued? Consider their relevance against health system reforms/post-Ebola priorities. Given the interrelatedness of health systems strengthening, elimination of specific components is probably not advisable. However, analysis of the urgency and potential for specific components suggests that progress be considered in the short-, medium-, and long-term as follows: Outcome Question 1.3: What are recommendations for maintaining, revising, or removing indicators based on a revised health system strengthening activity? Include a proposed CSH results framework aligning recommendations with the overall USAID/Liberia results framework. The evaluation proposes a revised Results Framework comprising a restructuring of the current seven components into four broader components (stewardship, sector resource management, accountability, and health service delivery); (ii) an alignment of these components with the CDCS and the National Resilient Health Systems Plan; (iii) the selection from among a pool of indicators found in the different USAID, MOH, and CSH documents; and (iv) the negotiation of a final, operational list by the three parties. Outcome Question 2.0: What are some of the value-added components of CSH (and the activity staff), if any, to the Ministry of Health, donor community or USAID that were not explicitly outlined in the award? The original award’s SOW was extremely comprehensive and explicitly anticipated almost every conceivable contribution to MOH. Of those activities identified by the KI as priority activities not explicitly outlined in the award, the most important include: (i) enforcement of regulatory mandates; (ii) health insurance and drug revolving fund initiatives; and (iii) the interoperability road map for harmonizing various sources of health information. Outcome Question 2.1: Specifically, what has been the value-added components of CSH on the FARA (implementation, management)? USAID’s Government-to-Government relationship with MOH through the Fixed Amount Reimbursement Agreement (FARA) (and a similar, though delayed arrangement for WASH with MPW) has been the principal mechanism for providing financial resources to improve health service delivery in Short-term Medium-term Long-term Central L&G/Regulatory SCM HCF HIS QI/QA HR Preparedness & Response County L&G/CHB SCM Infrastructure HR QI/QA HIS Engagement HCF Potential short, medium, and long-term gains 13 the three targeted counties. With respect to the value-added components of CSH for FARA, the most important contributions have focused on strengthening fiduciary capabilities to ensure that CHTs: (i) utilize Government funds properly and transparently for their intended purpose; and (ii) account for Government funds in accordance with Government reporting requirements. Outcome Question 3.0: Are there new or emerging needs for health systems strengthening not within current CSH scope? Among the emerging needs, three related to quality improvement merit mention: (i) weaknesses in patient safety; (ii) laboratory and diagnostic capabilities; and (iii) inadequate use of the existing and extensive patient records in the facilities. Sustainability Question 4.0: To what extent have CSH’s approaches supported the MOH in fulfilling its core health systems functions? Comment on MOH’s improved capacity, stewardship and ownership of the core health systems functions, the quality of staff, internal systems, robustness of M&E system. CSH’s approaches have supported MOH in developing its core functions, but not in a systematic and efficient manner. CSH’s results framework and annual work plans have been process-driven, based not on implementing proven (or innovative) interventions but on responding to expressed needs for improving MOH’s core functions and to USAID demands. Without measures of the accrued benefits from or the estimated costs to completion of the selected core function strengthening activities, MOH lacks the information needed to select among (and therefore to own) these improvement measures. As measured by the capacity assessment tool applied previously by RBHS, CSH, and FARA, MOH’s management of the essential health sector functions has not appreciably improved at county level, perhaps due to systemic issues at central level which may have impeded and may continue to impede future developments. Though there have been many positive developments over the first two years of project implementation, the project has not yet demonstrated an ability to develop capacity by: (i) combining system reform measures, organizational strengthening of operational standards and procedures, and workforce development; and (ii) coordinating implementation of these measures at central and county levels. Sustainability Question 5.0: What interventions were most successful in improving the ability of the MOH to manage key health system efforts at the individual, unit, or county level and move toward a more decentralized health system? Assessment of the project’s success in improving the management of key health system efforts was addressed previously. KII mentioned 35 priority interventions where CSH was felt to have made improvements, but on balance, beneficiaries at county level ranked the project’s contributions higher than those at central level. Identifying whether these improvements were at individual, organizational (unit), or county level is not possible. However, CSH's capacity development efforts seem to have been more successful where organizational policies and procedures were already in place and the project’s tools and approaches focused on developing individual skills to implement these systems. CSH's capacity development efforts were less successful where policies and procedures were new and/or where potential gains relied on systemic changes beyond the project’s influence. Sustainability Question 6.0: What has been CSH’s contribution to achieving the objective of the National Resilient Health System Plan? 14 As noted above, CSH has contributed to many of the National Resilient Health System Plan’s nine strategic areas. Analysis of the activities proposed by the National Resilient Health System Plan indicate that of the 64 priority areas, CSH has addressed 15 and could potentially address an additional 16. Half of the Plan’s service delivery indicators are included in those reported on by CSH to USAID. RECOMMENDATIONS CSH should lessen its tendency to overpromise (based on its annual work plans) and underdeliver (based on its available resources). The project's ambitions should be reconsidered and the project re-designed to emphasize: more focused objectives and fewer activities (to account for limited HR and competing priorities); and better accountability for results (including clearer agreement on measures of project success). In future, CSH should select and plan activities based on progress to date and future potential for short, medium, and long-term gains. In this way, the project could demonstrate the cogency of its interventions through: (i) the logic and timely sequencing of the planned activities; and (ii) the anticipated endpoints of HSS capacity development. CSH must increase its engagement with MOH's Senior Management Team in approving the annual work plan and in communicating the results of the activities. To this end, a Memorandum of Understanding (MOU) between CSH and MOH (proposed earlier by CSH) could contribute to better collaboration and communication. The project should combine the current focus on demand-driven response with a more results-based approach. The project should not solely respond to MOH's expressed needs, but seek to collaboratively define expected results over the short, medium, and long term. To this end, the capacity assessment instrument could be: (i) improved (to make it more sensitive to changes on the ground); and (ii) considered as a potential tool for programming interventions to link proposed activities to defined indicators of progress in capacity development. 15 I. INTRODUCTION EVALUATION PURPOSE The evaluation’s purpose is to provide an independent and in-depth examination of the overall progress of the Collaborative Support for Health (CSH) project in Liberia. The evaluation is expected to: (i) document achievements, performance issues, and constraints related to activity implementation and effectiveness; (ii) analyze results and lessons learned from implementation; and (iii) provide succinct, actionable recommendations to determine which component(s) of CSH to scale up, modify, or re-design for option years and/or in other ongoing related activities or future procurements. The key intended audiences are USAID, MSH, and MOH. Secondary users include USAID/Washington, CSH subcontractors, donors, NGOs, and other organizations interested in the effectiveness and efficiency of different approaches and tools for developing capacity and strengthening health systems. EVALUATION QUESTIONS The evaluation questions included in the Statement of Work (Annex 1) addressed the project’s outcomes and sustainability. With respect to the outcomes, the evaluation is expected to analyze: • To what extent has progress been made in achieving the seven core systems strengthening objectives under CSH? o Which components have been most successful in meeting the overall objectives of CSH? At what level? Provide a prioritized list of components. o If CSH will receive option years, which areas should be strengthened and which areas could be discontinued? Consider their relevance against health system reforms/post￾Ebola priorities. o What are recommendations for maintaining, revising, or removing indicators based on a revised health system strengthening activity? Include a proposed CSH results framework aligning recommendations with the overall USAID/Liberia results framework. • What are some of the value-added components of CSH (and the activity staff), if any, to the Ministry of Health, donor community or USAID that were not explicitly outlined in the award? o Specifically, what has been the value-added components of CSH on the FARA (implementation, management)? • Are there new or emerging needs for health systems strengthening not within current CSH scope? With respect to the project’s sustainability, the evaluation is expected to examine: • To what extent have CSH’s approaches supported the MOH in fulfilling its core health systems functions? Comment on MOH’s improved capacity, stewardship and ownership of the core health systems functions1, the quality of staff, internal systems, robustness of M&E system. o What interventions were most successful in improving the ability of the MOH to manage key health system efforts at the individual, unit, or county level and move toward a more decentralized health system? o What has been CSH’s contribution to achieving the objective of the National Resilient Health Plan? 1 The U.S. Government defines country ownership as “the continuum of actions taken by political and institutional stakeholders in partner countries to plan, oversee, manage, deliver, and finance their health sector and achieve health goals. These actions advance sustainable, quality health programs that are locally owned and responsive to the needs of host country nationals.” 16 II. PROJECT BACKGROUND USAID and Management Sciences for Health (MSH)2 signed a contract to implement CSH on February 27, 2015. The contract was in the amount of $21.5 million and covered the period from February 27, 2015 to February 27, 2018. It provided for an additional two option years (through February 2020) in the amount of $10.9 million. CSH’s overall objective is to “improve the health status of Liberians” as expressed in USAID/Liberia’s Health Strategy, and Country Development Cooperation Strategy (CDCS) Development Objective 3. Operationally, CSH is expected to: (i) enhance the capacity of the MOH and County Health Teams (CHTs) of Bong, Lofa, and Nimba counties; and (ii) ensure the necessary health systems and standards are in place to provide high quality services (implementation of the Essential Package of Health Services) for all Liberians. CSH’s anticipated results are closely aligned with the objectives of the National Health Policy and Plan (2011-2021) and the Investment Plan for Building a Resilient Health System (2016-2021). The CSH project design is based in large measure on modifications incorporated into the implementation of the Rebuilding Basic Health Services (RBHS) Project. In support of the National Health Policy and Plan (2007-2011), USAID financed the (RBHS) Project over the period 2008-2013 (and subsequently to 2015). RBHS’s support to the then Ministry of Health and Social Welfare (MOHSW) focused initially on health service delivery. However, in keeping with the principles of the USAID Forward initiative, the mission opted in 2011 to provide Government-to-Government support for service delivery through a Fixed Amount Reimbursement Agreement (FARA). Consequently, and consistent with the 2011-2021 National Health and Social Welfare Policy and Plan (NHSWPP) and the National Capacity Development Strategy (2011), RBHS shifted its focus from health service delivery to country-led capacity-building and health system strengthening. As presented by USAID, the following graphic summarizes the relationship among the various funding instruments at the different levels: With FARA serving as the mechanism for improving health service delivery, RBHS (and subsequently CSH) addressed WHO’s six building blocks of health systems strengthening3:(i) delivering essential health services; (ii) the health workforce; (iii) health information systems; (iv) access to essential medicines; (v) health systems financing; and (vi) governance and leadership. By building capacities within each of these blocks at the central and county levels, the RBHS project planned to contribute to the National Decentralization Policy (2011) and Plan (2013-2017) and to the assumption of responsibility by 2 MSH’s subcontractors comprised: Development Innovations Group (DIG), Institute for Healthcare Improvement (IHI), JHPIEGO, and Results for Development (R4D). 3 In anticipation of USAID’s intention to enter into a direct financing agreement with MPW to finance rural water supply infrastructure improvements, a seventh component was added to CSH to implement the necessary risk mitigation efforts. 17 the County Health and Social Welfare Teams (CHSWTs) for managing all aspects of county health service delivery. It was recognized that success would also require strengthening the capacity of the central MOHSW. The Ebola epidemic and its aftermath greatly affected CSH’s implementation. First, project start-up was delayed. RBHS was extended from the planned end-date in October 2014 until February 2015 to continue support for the Ebola response, particularly in advance of the rapid increase in international assistance. Projected CSH staff changed considerably over the year between submission of the proposal and was not entirely in place at the time of project start-up on February 27, 2015. CSH was officially launched in May 2015. Second, the Ebola response brought significant changes in the health financing landscape, with multiple actors operating in similar technical areas as CSH and increasing the need for coordination of project activities with many projects, donors and stakeholders. Third, the health sector was restructured and its orientations modified. The Essential Package of Social Services (EPSS) was eliminated from CSH’s scope of work when the Ministry of Social Welfare moved to the Ministry of Gender (2015). In addition, the Ministry of Health (MOH) adopted the “Investment Plan for Building a Resilient Health System” (2015) with the objectives of: (i) restoring the gains lost due to the Ebola Virus Disease (EVD) crisis; (ii) providing health security for the people of Liberia; and (iii) accelerating progress towards universal health coverage. The six building blocks of the NHSWPP were modified to comprise nine strategic areas: leadership and governance (L&G), human resources (HR), health care finance (HCF), health information systems (HIS), Quality Assurance/Quality Improvement (QA/QI), supply chain management (SCM), preparedness and response, infrastructure, and community engagement. Fourth, the SOW evolved during the first year as the program was modified to correspond to USAID’s needs and the post-Ebola environment in Liberia. USAID provisionally approved CSH’s first year work plan on July 9, 2015, but it continued to evolve during the first year. Subsequently, project implementation had to adapt to: (i) new or revised national policies, plans, and strategies; and (ii) the shifting financial and technical landscape which necessitates continuous coordination with other partners and frequent modification of the specific activities defined in the original SOW. III. EVALUATION METHODS AND LIMITATIONS A team of five evaluators worked in Liberia from April 3 to May 5, 2017. In addition to interviews in Monrovia, the evaluation team visited four counties with the intention of comparing results in two CSH counties (Bong and Nimba) and two non-CSH counties (Margibi and Grand Bassa). The performance evaluation utilized a combination of quantitative and qualitative sources of information. Quantitative information was collected from: (i) a desk review of program and project documents; (ii) an analysis of complementary information from past evaluations and current databases (e.g., DHIS2, QA/QI dashboard, and LMIS); and (iii) questionnaires on capacity (at CHT level) and patient satisfaction (from hospital and clinic levels). Lists of the documents reviewed and persons consulted are presented in Annexes 2 and 3; and the semi-structured interview guides and questionnaires may be found in Annex 4. Qualitative information was gathered from Key Informant Interviews (KII) and Focus Group Discussions (FGD) with project beneficiaries (at central, county, and district levels), collaborating organizations, and donor agencies. The evaluation team also met with staff from the CSH project (at central and county levels), the FARA implementing agencies (International Rescue Committee and Africare), and USAID. The following table summarizes the number of interviews conducted, as well as the breakdown of interviewees by gender. 18 Table 1: Summary of Interviews Conducted and Interviewees Met The number of planned and conducted interviews differs primarily because the Evaluation Team was able to visit only one health clinic per county and not the two initially envisioned. Notes from the KII and FGD were entered into NVivo, a qualitative data analytical program, and the principal themes were analyzed using an inductive thematic analysis methodology. In other words, the data was reviewed and themes were developed to answer the evaluation question. A coding agenda was developed describing themes, difference between themes, and prototypical passages. Following the coding process, theme frequencies across all counties (CSH and non-CSH), per building block were identified. This served as the information that was then used to answer the evaluation questions. Implementers and stakeholders actively participated in the work of the evaluation and were frank during discussions. Though MOH staff at all levels were often busy with competing demands, they made time to meet with the evaluation team and responded to every question. The evaluation team was in constant contact with USAID (an inception report, an in-brief presentation, weekly update reports, and a final debrief presentation before departure). The team provided an in-brief to CSH and MOH at the beginning of the field work and met with the Chief Medical Officer at the end of the mission. Finally, the team met with CSH on several occasions to collect information and exchange ideas; a debrief was organized prior to departure. The evaluation team mitigated most but not every constraint identified in the inception report and discussed at the in-brief with USAID. Specifically, the evaluation team was concerned with three issues: (i) the broad reach and relatively short duration of the evaluation; (ii) the criteria for measuring capacity; and (iii) the possibility of attributing any changes in capacity to CSH’s interventions. With respect to the scope of the evaluation, the team managed the schedule and conducted assessments of the project’s components at the different levels, including visits to two (of three) CSH counties (Bong and Nimba) and two non-CSH counties (Margibi and Grand Bassa). Travel times were less than expected (since MOH requested that Lofa County be replaced by Nimba County), but preparation of the KII notes and NVivo coding proved to be more time-consuming than anticipated. With respect to the measurement of results in general and capacity in particular, the evaluation team found partial solutions. Though the CSH implementation data were limited to the period February 2015- December 2016, they were sufficiently detailed to provide an overview of the project’s accomplishments and difficulties over the time frame. The lack of data for certain program indicators and the quality of some of the data (especially for the QA/QI dashboard) were unanticipated limitations. The absence of key program indicators remains an issue. The data in the QA/QI dashboard were reanalyzed, but the team is not convinced that all of the problems have been resolved. Importantly, the evaluation team was able to adapt the capacity assessment used in previous evaluations (RBHS in 2012 and 2014, CSH in 2015, and FARA in 2016) to assess capacity development in 2017 and compare it with previous results. With respect to the issue of attribution, two issues were not resolved. First, CSH implementation has been limited to barely two years, which is a relatively short period for capacity development. Second, Planned Done Planned Done Planned Done Planned Done Planned Done Planned Done % Interviews KII/GII 20 18 12 15 0 1 12 14 0 0 44 53 120% FGD 6 3 6 4 12 7 58% Exit 40 32 40 25 80 57 71% Total 20 18 12 15 46 36 12 14 46 29 136 117 86% Interviewees Male NA 18 NA 24 NA 25 NA 16 NA 16 NA 106 54% Female NA 14 NA 0 NA 39 NA 4 NA 33 NA 91 46% Total 32 24 64 20 49 197 Total CSH Non-CSH MOH CHT Facilities CHT Facilities 19 efforts to attribute change are constrained by: (i) the previous results of the RBHS project in the same counties; and (ii) the selection of Margibi and Grand Bassa counties as comparison counties. Based on USAID’s recommendation, these counties were chosen primarily to assess their feasibility for future expansion of CSH (since they already have ongoing USAID-financed interventions) rather than to serve as “control” counties. Consequently, comparisons between Bong and Nimba counties and Margibi and Grand Bassa counties do not provide an adequate assessment of CSH’s impact in the CSH counties. IV. FINDINGS The Evaluation Questions focus both on the project’s current progress in strengthening the health sector (Outcomes) and on its potential contributions to a more decentralized health system and the achievement of the objectives of the National Resilient Health System Plan (Sustainability). OUTCOME QUESTION 1: TO WHAT EXTENT HAS PROGRESS BEEN MADE IN ACHIEVING THE SEVEN CORE SYSTEMS STRENGTHENING OBJECTIVES UNDER CSH? CSH’s theory of change posited that: (i) the project’s outputs would improve performance of the health system at the national and county level; and (ii) health system strengthening at national and county levels would improve the health status of the Liberian population. This section reviews CSH’s progress in producing its planned outputs, achieving its intended program results and capacity outcomes, and contributing to improved health status. The next section provides a detailed assessment of each of the seven components. Assessment of CSH’s work plans for FY15, FY16, and FY17 shows that planned activities have only been partially achieved. The original Statement of Work (SOW) listed a range of intervention. One hundred and twenty-six minimum required activities were identified for the seven objectives and 28 sub￾objectives. In accordance with CSH’s demand-driven response to human and institutional capacity building, however, these work plans were further expanded to comprise 182 activities planned for FY16 and 180 planned for FY17. Table 2: Planned CSH Activities by Component The planned activities were accompanied by a large volume of outputs. Project reporting data from March 2015 to March 2017 indicate that: (i) more than 80 consultants were deployed (of which about 30 have been long-term technical staff for various periods in Liberia); (ii) more than 100 kinds of materials were produced; and (iii) almost 1,300 persons participated in training activities. Annexes 6, 7, and 8 present details on these outputs. The planned activities were not, however, accompanied by the expected results. As shown in CSH’s SOW FY16 FY 17 1 Leadership and governance 32 38 23 2 Water supply management 3 4 27 3 QA/QI initiatives 26 40 44 4 HRH management 26 30 16 5 Supply chain management 12 13 19 6 Health care financing 13 31 39 7 Health management information 14 26 12 Total 126 182 180 Planned Activities Components 20 reports, planned deliverables have only been partially achieved over the first two years of project implementation: 40% have been completed, 19% are ongoing, and 41% have been postponed or lack information. The proportion of deliverables completed on schedule declined from FY15 to FY16. Project reports for the periods October-December 2016 and January-March 2017 were not made available and are not reflected in this evaluation. Table 3: Status of Deliverables for FY15 and FY16 A review of the reasons (“challenges”) identified by the project for the status of the deliverables includes the following kinds of issues: (i) institutional (e.g., determination of roles and responsibilities, ministerial priorities and planning, etc.); (ii) human resources (e.g., insufficient number or skills); (iii) financial (e.g., mobilization of funds from various partners); and (iv) physical (e.g., accessibility, internet, etc.). Assessment of CSH’s program targets indicates that they have been partially achieved; there have, however been many successful contributions to health system strengthening. As shown in Annex 9 and summarized in the following table, of 42 targets set for the FY16 program indicators, only a third were achieved, while more than half were either not achieved or not measured. Table 4: Achievement of Planned Targets by Component Assessment of management capacity shows that progress has been neither uniformly positive in the CSH counties nor markedly different from the non-CSH counties. Using an abbreviated version of the same assessment tool employed by previous evaluations (see Annex 4), the results are as follows: Deliverable Status No. % No. % No. % Completed 9 64% 14 32% 23 40% Ongoing 0 0% 11 25% 11 19% Postponed 0 0% 6 14% 6 10% No information 5 36% 13 30% 18 31% FY 2015 FY 2016 Total Achieved Partially Achieved Not Achieved No data % Achieved Performance indicators (P1-7) 0 0 0 6 0% 1 Leadership and governance 1 2 3 0 17% 2 Water supply management 2 1 0 2 40% 3 QA/QI initiatives 4 0 2 3 44% 4 HRH management 4 1 3 0 50% 5 Supply chain management 1 0 2 2 20% 6 Health care financing 0 1 3 2 0% 7 Health management information 1 1 1 0 33% Total 13 6 14 15 29% % 27% 13% 29% 31% Components Achievement of Planned Targets 21 Table 5: Comparison of Capacity Development Scores (since 2012) A detailed presentation of the scores by county for each of the criteria can be found in Annex 10. Assessment of health service delivery indicates slow improvement. Though CSH’s mandate does not include health service delivery, the project is required to present data on 14 indicators linked to improved health service delivery. For purposes of completing the data, CSH’s results are presented in Annex 11. OUTCOME QUESTION 1.1: WHICH COMPONENTS HAVE BEEN MOST SUCCESSFUL IN MEETING THE OVERALL OBJECTIVES OF CSH? AT WHAT LEVEL? PROVIDE A PRIORITIZED LIST OF COMPONENTS. Though information is not always complete and the results are often mixed, the project has achieved progress on each component at central and county levels. Given the incomplete information on program indicators, Annex 9 does not provide a basis for assessing component success. Annex 12 highlights the project’s principal achievements as summarized in the annual reports and cited in the KII results. Following is an overview of the results of the components and their contribution to meeting the project’s objectives. Component 1: Leadership and Governance. The component’s objectives are to: (i) measurably build MOH’s leadership and governance capacity at all levels; (ii) strengthen external regulatory frameworks for service delivery and pharmaceuticals; and (iii) improve management capacity through technical assistance to the CHTs in Bong, Lofa, and Nimba counties. CSH has financed a leadership development program (LDP+) at central level and contributed to the preparation of a County Health Board (CHB) operational manual and facilitator’s guide. For LDP+, consultations were held with MOH, copies of training materials shared, and facilitator training conducted. However, CSH was asked to postpone the roll out because MOH’s new Health Workforce Program (HWP) proposed a similar program. The CHB manual has been disseminated, but follow up has been irregular and the operations of the CHB have not been monitored. Of the three key regulatory bodies, the Liberian Board of Nurses and Midwives (LBNM) has received the most support with Board governance training, strategic planning, LDP+ training of school administrators, and finalization of pre-service standards for Nursing/Midwifery Education. CSH has supported LBNM and the Liberia Medical and Dental Council (LMDC) in two essential areas: accreditation and continuing professional development (CPD). Finally, along with LBNM and LMDC, the Liberia Pharmacy Board (LPB) have instituted iHRIS software to identify practitioners and track their licensure and certification status. Though not measured, progress was made in the basic regulatory steps of accreditation, CPD, and re-licensure tracking through iHRIS. In each instance, the KII felt that CSH had taken on too many activities and was unable to follow up sufficiently on them. 2012 2015 2016 2017 RBHS CSH FARA CSH/PE County Score Score Score Score CSH Counties Bong 15 40 40 37 Nimba 25 40 36 38 Non-CSH Counties Margibi 35 Gr. Bassa 37 22 At county level, the project embedded mentoring skills in the CHT to build capacity in the key technical areas: Quality Assurance/Quality Improvement (QA/QI), Human Resources (HR), Supply Chain Management (SCM), Financial Management (FM), and Health Information Systems (HIS). Component 2: Ministry of Public Works (MPW) Management. The component’s objectives are to: (i) support the implementation of MPW’s risk mitigation efforts; and (ii) strengthen MPW’s capacity to manage water supply infrastructure improvements. USAID intended to directly finance MPW’s mandate to provide rural water supply infrastructure to villages of less than 5,000 persons. CSH was expected to help reduce the fiduciary risks during the period when USAID would be negotiating the Government to Government agreement. These risks were to be identified by the Public Financial Management Risk Assessment (PFMRA), but the PFMRA was delayed and CSH’s intervention to reduce risks was minimal. With respect to the water supply management policies and practices, CSH made several contributions, including: (i) assessments of capacity building needs; and (ii) gender-related policies and practices. Contributions were also made to the waterpoint mapping exercise by: (i) developing mechanisms to enforce mandatory Implementing Partner (IP) reporting to the Waterpoint Atlas; (ii) training of MPW staff to manage the Water Point Atlas; and (iii) mapping of WASH facilities in 5,500 schools. CSH also organized workshops to develop County WASH plans. Component 3: Quality Assurance and Quality Improvement. The component’s objectives are to support the institutionalization of QA/QI practices as components of health service delivery by ensuring that: (i) EPHS and EPSS quality standards and service delivery protocols were in place and adhered to in the workplace; and (ii) workforce requirements and quality standards were met through linkages with continuing medical education (CME) and skills development. In addition, the component was expected to improve diagnostic services for support of clinical diagnosis and treatment, as well as for quality assurance functions. At central level, significant progress was made in establishing a Quality Management Unit, develop a National Health Quality Strategy, and update the EPHS and corresponding Joint Integrated Supervision and Support (JISS) tool. At facility level, initiation of collaborative improvement activities in 45 clinics and hospitals began in May 2016 with: antenatal care (ANC) first visit, partograph adherence, and FP new contraceptive acceptors. In December 2016, additional areas were introduced: HIV testing, postnatal care, malaria protocol adherence, and infection prevention and control (IPC) hand washing adherence. CSH has supported the monthly update of a dashboard comprising the results of these and other QA/QI indicators. Results in the year and months prior to commencement of the initiative, as well as results over the past year are shown in the following table: Table 6: Progress in Selected QI Indicators Jan-Dec 2015 Jan-Mar 2016 Apr 2016- Mar 2017 Introduced in May 2016 ANC 1 35.3% 24.7% 20.5% Partograph Adherence 85.6% 63.8% 83.4% New FP Acceptors 9.4% 12.2% 11.9% Introduced in December 2016 ANC Clients Tested for HIV 32.8% 57.1% 59.1% PNC 1 within 72 hours 54.1% 73.3% 75.7% Malaria Protocol Adherence 96.6% 91.3% 92.0% IPC Hand washing facilities 51.1% 53.7% 48.2% Quality Improvement Indicator 23 LSA’s Data Quality Assessment (DQA), as well as the Evaluation Team raised concerns about the quality of the data collected by CSH, particularly taken from month to month at the facility level. 4 Taken for all 45 facilities over extended periods, the data are indicative of some progress for certain elements of quality: (i) for those quality improvements introduced in May 2016, there have been slight increases in partograph (though not in comparison with 2015); and (ii) for those quality improvements introduced in December 2016, it is admittedly too soon to expect substantial change. The slow improvement in quality may be due to several factors: (i) poor results in Bong County (on ANC 1, FP, and IPC); (ii) inadequate laboratory capabilities and insufficient drugs which handicap diagnosis and treatment; (iii) inadequate space and poor organization which make patient records and QI data difficult to access; and (iv) inappropriate application of the JISS tool by large supervision teams which overwhelm available clinic staff. In addition to the QI data, summary information was collected during the field work from users of a small number of hospitals and clinics in the CSH and non-CSH counties. The results of the brief survey conducted during the visits is presented in the table below: Table 7: Patient Satisfaction in Selected Facilities As judged by the users, health facilities in CSH-supported counties (Bong and Nimba) perform better than those in non CSH-supported counties (Margibi and Grand Bassa). With QA/QI initiatives also being implemented in the non-CSH counties, these results would need further analysis before attributing them to CSH. Given the importance of citizen engagement in the Resiliency Plan, even a very brief patient satisfaction survey indicates the value of including their voice to provide useful information for improving services. The support for diagnostic services was expected to assess routine laboratory activities to support the implementation of the EPHS package and complement the quality assurance functions. CSH staffing problems and the ongoing Service Availability and Readiness Assessment (SARA), has postponed the survey which is now planned for later in FY17. Component 4: Human Resources for Health. The component’s objectives are to: (i) strengthen HRH management at all levels; and (ii) support pre-service institutions, MOH’s training division and service units, scholarships for pre-service training, and institutionalization of performance-based management strategies. At central level, CSH contributed to restructuring the HR Unit, preparation of an HR handbook, training in strategic HR planning and records management training, and development of iHRIS software. At county level, iHRIS training and software have contributed to updating and integrating personnel into 4 Liberia Strategic Analysis Fixed Amount Reimbursement Agreement (FARA) and Collaborative Support for Health (CSH) Data Quality Assessment Report. April 28, 2017. Nimba Margibi G.B. Total N=57 Acceptable waiting time (<2 hrs) 100% 93% 10% 87% 79% Condition/treatment explained (Yes) 71% 87% 50% 13% 56% Condition/treatment understood (Yes) 65% 87% 20% 13% 49% Use of sanitizer before & after (Yes) 18% 33% 40% 7% 23% Service rating (As expected) 82% 53% 60% 53% 63% Bong Questions N=17 N=15 N=10 N=15 CSH Non-CSH 24 iHRIS. Data from the counties show that the number of completed personnel files has increased as shown in the following table: Table 8: Cumulative Number of Personnel Files Completed by County While substantial progress has been made on completing the personnel files, internet connections have slowed progress on uploading the files into iHRIS and hindered the actual use of the personnel information. With respect to training, CSH has contributed at central level to: (i) assessments of long-term staff training needs and MOH's management of in-service training; (ii) establishment of an integrated MOH training unit and the revitalization of the MOH Scholarship Committee; and (iii) managed 200+ scholarships for pre-service training of laboratory technicians and midwives. At county level, CSH has provided support for the recruitment of 72 community health supervisors in Nimba County. Component 5: Supply Chain Management. The component’s objectives are to: (i) strengthen supply chain management by the National Drug Service (NDS), MOH’s Supply Chain Management Unit (SCMU), and the CHTs; and (ii) ensure continuous availability of essential drugs at service delivery points. At central level, CSH worked with: (i) NDS to review and disseminate standard pharmaceutical treatment guidelines; (ii) SCMU to review and revise the supply chain master plan; and (iii) the CHT to improve warehouse operations at the country drug depots. Visits to CSH and non-CSH depots revealed similar, poor conditions for drug storage; none of the CSH and non-CSH counties scored more than 1 (of 4) on the capacity to effectively store and distribute commodities. To address the availability of essential drugs, CSH organized an assessment of the Logistics Management Information System (LMIS) and, in collaboration with MOH and other partners rolled out a re-designed LMIS nationwide in early 2017. Though it is too soon to assess the revised LMIS and its potential impact on stock-outs, the following table indicates that reporting rates were already very high and that stock￾outs were a significant problem: No. % No. % No. % Bong Number of personnel 1 203 1 295 1 295 Personnel files completed 908 75% 1 043 81% 1 043 81% Personnel files uploaded in iHRIS 75 8.3% 100 9.6% 100 9.6% Lofa Number of personnel 983 1 124 1 124 Personnel files completed 700 71% 730 65% 730 65% Personnel files uploaded in iHRIS 100 14.3% 150 20.5% 150 20.5% Nimba Number of personnel 1 029 1 079 1 079 Personnel files completed 970 94% 970 90% 970 90% Personnel files uploaded in iHRIS 0 0.0% 116 12.0% 116 12.0% Total Number of personnel 3 215 3 498 3 498 Personnel files completed 2 578 80% 2 578 74% 2 578 74% Personnel files uploaded in iHRIS 175 7% 291 11% 291 11% 2016 2017 Total Counties 25 Table 9: LMIS Reporting Rates and Proportion of Facilities with Stock-outs The implementation of three rounds of End Use Verification (EUV) in the context of the President’s Malaria Initiative confirms an increasing number of health facilities with stock-outs of twelve malaria commodities between June and November 2016; the EUV for April 2017 is not yet available. Component 6: Health Financing. The component’s objectives are to: (i) strengthen financial oversight, planning, and management functions, at both central and county levels; (ii) support management approaches focusing on performance-based financing for contracting / purchasing of health care services; and (iii) support for planning and managing the transition from free health care to the introduction of affordable user-fees, including piloting of activities that explore financing options for enhancing revenue generation and collection, pooling, purchasing, and financial protection. With respect to financial management, a public financial management (PFM) manual and a procurement manual were finalized for county use, QuickBooks™ was introduced in the three counties, and peer learning sessions were organized for the counties to share experiences and identify best practices. Improvements across the components of PFM at the county level have been documented periodically over the past 18 months and indicate steady progress, as shown in the following table: Table 10: Table 10: Financial Management Assessment Scores Since October 2015, PFM scores have improved from a little better than 1 (defined as in place but not yet functional) to almost 3 (defined as functional but not yet up to best practice) of 4. In addition, based on the number of PFM risks identified by Deloitte (2012), the following table shows that steady progress is being made in mitigating these risks, particularly at the county level: Indicator Bong No. of facilities 46 46 46 46 46 % of facilities reporting 98% 100% 100% 100% 100% % of facilities reporting stockouts 31% 38% 30% 35% 35% Nimba No. of facilities 75 75 75 75 75 % of facilities reporting 84% 81% 88% 87% 87% % of facilities reporting stockouts 35% 42% 37% 29% 29% Indicator LMIS Reporting Jun-16 Aug-16 Dec-16 Feb-17 Total Indicator Baseline Oct 2015 Mar 2016 Jun 2016 Sep 2016 Dec 2016 Mar 2017 Indicator Baseline FY 16 Budgeting and spending 1.1 1.7 2.0 2.2 2.7 2.7 Audit steps 2.2 2.2 2.5 2.5 3.0 3.0 Fixed assets management 2.3 2.3 2.5 2.5 2.5 2.5 Warehouse management 0.7 1.0 1.2 1.4 3.0 3.0 Score 1.3 1.5 1.8 2.0 2.7 2.7 26 Table 11: Progress in Mitigating PFM Risks CSH has contributed to strengthening Performance-Based Financing (PBF) by: (i) building technical capacity in the newly-staffed MOH PBF Unit; and (ii) revising and rolling out the PBF Manual through TOT in Bong, Lofa, and Nimba counties. Finally, CSH has supported MOH in preparing the transition from free health care to affordable user￾fees by: (i) financing analytical studies to inform health care financing reforms (willingness to pay, costing of EPHS for clinics); (ii) presenting options to MOH Senior Management Staff; and (iii) organizing a workshop to consider the design of a Revolving Drug Fund (RDF) pilot to improve the financial sustainability of essential drugs. Component 7: Health Information Systems. The component’s objectives are to: (i) improve the accuracy and use of health information; and (ii) strengthen HMIS by integrating or linking it with other data collection systems at both central and county levels. At central level, CSH has contributed to finalization of the national HMIS Strategic Plan, facilitated mapping of partners involved in the HMIS strengthening process, and developed a five-year roadmap to establish inter-operable health information systems. At county and district levels, CSH has revised and rolled out revised HIS tools, trained data clerks and supervisors, and facilitated regular reviews and validations of health data. As shown in the following table, the number of facilities providing timely and complete reports is higher in CSH than in non-CSH counties. Table 12: Proportion of Facilities Providing Timely and Complete HMIS Reports Based on the project’s experience to date, assessment of current success can only be tentative. While project implementation is still early and no specific criteria for determining the success of a given component have been proposed by the project, the component results presented in Annex 12 and reviewed in the previous section have been reviewed based on whether there have been measurable results and whether these results can be considered generally positive. Applying these criteria yields the following table: Level Initial 2012 Oct 2015 Mar 2016 Sep 2016 Mar 2017 Central/MOH 78 58 55 45 36 Bong 46 33 29 22 10 Nimba 46 34 29 22 10 Lofa 45 34 29 23 10 Initial and remaining risks June 2016 August 2016 October 2016 December 2016 February 2017 Total CSH Bong 97.7% 95.3% 100.0% 97.7% 100.0% 98.1% Nimba 98.6% 98.6% 94.4% 98.6% 97.2% 97.5% Non-CSH Margibi 72.7% 72.7% 81.8% 88.6% 90.9% 81.4% Gr. Bassa 87.9% 66.7% 90.9% 100.0% 97.0% 88.5% Counties 27 Table 13: Assessment of Component Success As shown above, the evaluation found quantitative data on process for HR, HIS, and SCM and on outcomes for HCF and QA/QI; qualitative information was available for MPW and L&G. The process results for HR (records and scholarship management) and HIS (timely reporting) were generally positive, while those for SCM (timely reporting) were mixed in Bong and Nimba counties. The outcome results for HCF were positive (for both financial management and risk mitigation) but mixed for QA/QI for reasons indicated above. Outcome results (stock-outs) for SCM were also available but predate the implementation of the revised LMIS and are not directly related to this activity. OUTCOME QUESTION 1.2: IF CSH WILL RECEIVE OPTION YEARS, WHICH AREAS SHOULD BE STRENGTHENED AND WHICH AREAS COULD BE DISCONTINUED? CONSIDER THEIR RELEVANCE AGAINST HEALTH SYSTEM REFORMS/POST-EBOLA PRIORITIES. The consideration of future options will depend on assessments of the strengths of the current components and on their relevance for the National Resilient Health System Plan. Given the interrelatedness of health systems strengthening, the current project components are all potentially relevant for promoting health sector reforms and health systems strengthening. Assessment of future priorities relies on a determination of the urgency of and the potential for future developments. Three groupings are proposed: • HCF: Considerable progress has been achieved at county level with respect to financial management and reporting, and CSH has initiated essential work to promote health financing reforms, which are the basis for implementing other components. MOH and Parliament are both increasingly interested in piloting measures to secure adequate and predictable financial flows for the health sector. • HR, SCM, and QA/QI: Future progress on each of these components is linked to larger systemic issues which CSH only partially influences. For HR, information on health personnel is now available, however its use in decision-making is not sufficiently clear for the project to move forward on issues of deployment, motivation, etc. Similarly, for SCM, better information is available, but the larger issues of procurement and distribution are unresolved. For QA/QI, a framework for action has been established, but without major changes in HR and SCM, future progress in QA/QI will be difficult. • HIS, L&G, and MPW. In each case, progress has been made, but the next steps are not immediately apparent. For HIS, future progress will require difficult changes in attitudes towards the use of evidence-based results for future priority-setting, planning and budgeting. For L&G, the key areas of support (LDP+, CHB, and the regulatory bodies) would all seem to: (i) require additional commitment and substantial resources; and (ii) raise the issue of whether CSH is the appropriate vehicle to finance these costs. For MPW, the relevance of the activities, given the PFMRAF delays, is less than at the beginning of project implementation. Measured Not measured Generally positive HCF, HR, HIS MPW Mixed QA/QI, SCM L&G Results Results 28 Alternatively, the National Resilient Health System Plan provides a framework for considering future options and priorities. Furthermore, instead of selecting among HSS components, the following table indicates where CSH could make short, medium, and long-term gains on the nine strategic areas identified in the Plan: Table 14: Potential Short, Medium, and Long-term Contributions to the Nine Strategic Areas Annex 13 combines these two assessments by identifying in more detail those areas in the National Resilient Health System Plan where: (i) CSH has made progress and should continue; and (ii) related areas where CSH could logically intervene. OUTCOME QUESTION 1.3: WHAT ARE RECOMMENDATIONS FOR MAINTAINING, REVISING, OR REMOVING INDICATORS BASED ON A REVISED HEALTH SYSTEM STRENGTHENING ACTIVITY? INCLUDE A PROPOSED CSH RESULTS FRAMEWORK ALIGNING RECOMMENDATIONS WITH THE OVERALL USAID/LIBERIA RESULTS FRAMEWORK. Reporting on the achievement of indicators in the project’s Monitoring and Evaluation Plan has been handicapped by several problems. These problems have included: (i) technical shortcomings related to the definition of the indicators; (ii) problems with the measurement and presentation of the project’s results; and (iii) lack of formally defined relationships for reporting the project’s achievements. An analysis of CSH’s M&E arrangements to date is presented in Annex 14. Based on USAID's DO3 and IRs 3.1-3.42, CSH's activities could be re-organized to better reflect the National Resilient Health System Plan and indicators aligned with these different documents. Future modification of indicators would involve: (i) a revised presentation of USAID’s health system strengthening activity; and (ii) a realignment of indicators to be consistent with the CDCS and the National Resilient Health System Plan. The current seven components would be reorganized into four: Stewardship (L&G); Resource Management (HR, SCM, HCF, and infrastructure); Health Service Delivery (routine quality and emergency preparedness); and Accountability (HIS and citizen engagement). 5 In other words, the component building blocks would be replaced by the broader health sector functions. The CSH results framework would be realigned with the CDCS and the National Resilient Health System Plan. Annex 15 summarizes indicators found in the different USAID, MOH, and CSH documents, organizes them in terms of the reorganized project presentation, and aligns them with the CDCS. These indicators thus constitute a pool from which a finalized (and more operational) list can be negotiated among the three parties. 5 Assuming safe water and sanitation (WASH) remains in the project, this component would remain unchanged. Short-term Medium-term Long-term Central L&G/Regulatory SCM HCF HIS QI/QA HR Preparedness & Response County L&G/CHB SCM Infrastructure HR QI/QA HIS Community Engagement HCF 29 OUTCOME QUESTION 2.0: WHAT ARE SOME OF THE VALUE-ADDED COMPONENTS OF CSH (AND THE ACTIVITY STAFF), IF ANY, TO THE MINISTRY OF HEALTH, DONOR COMMUNITY OR USAID THAT WERE NOT EXPLICITLY OUTLINED IN THE AWARD? As noted earlier, the original award’s SOW was extremely comprehensive and explicitly anticipated almost every conceivable contribution to MOH. Of those activities identified by the KIIs as priority activities but not explicitly outlined in the award, the most important include: • Regulatory mandates (mentioned by the LMDC, LBNM, and LPB): The SOW identified several general activities, but CSH found numerous areas for development with the key regulatory agencies. What was neither mentioned in the SOW nor addressed at present was the issue of enforcement. • Health insurance and drug revolving fund initiatives (mentioned by the CMO, MOH/FARA & HFU, and WB): The SOW provided a wide window of possibilities for health financing reform but did not explicitly anticipate the Drug Revolving Fund, which MOH now views as an intermediate step toward health insurance and universal coverage. • The interoperability road map for harmonizing various sources of health information (mentioned by the MOH/M&E): The SOW mentioned the need for improved integration of the various information systems but did not specify how this might be done, which has now been identified by CSH. OUTCOME QUESTION 2.1: SPECIFICALLY, WHAT HAS BEEN THE VALUE￾ADDED COMPONENTS OF CSH ON THE FARA (IMPLEMENTATION, MANAGEMENT)? As indicated above, USAID’s strategy emphasized a Government-to-Government relationship with MOH (and anticipated a similar arrangement with MPW for WASH). Since 2011, FARA 1.0 and FARA 2.0 have provided the mechanisms and financial resources for improving health service delivery in the three targeted counties. CSH and PACS were: (i) designed as bilateral, complementary (“wrap￾around”), and integrated activities; and (ii) expected to address supply-related management issues (CSH) and demand-related communication issues (PACS). Overall, the wrap-around concept appears to have been more feasible in theory than in practice: (i) CSH and PACS have had very little interaction; and (ii) collaboration between CSH and FARA, and specifically with the implementing partners (Africare and IRC), has been hampered by operational issues. While CSH and the IP generally worked collaboratively, the KIIs noted the following constraints: (i) overlapping technical assistance expertise; (ii) occasional conflict/competition in determining the level of intervention (county, district, and facility), as well as the scheduling and responsibility for implementation (e.g., quarterly district reviews); and (iii) an imbalance in the resources available to the implementing partners as compared with CSH. Within this context, CSH has contributed to the transition in the health sector from contracting-out (to IPs) to contracting-in with the CHT by: (i) facilitating the development of tools to assess CHT readiness to contract-in; and (ii) strengthening CHT’s ability to fulfill the contractual conditions and account for the results. To this end, CSH focused on strengthening fiduciary capabilities to ensure that CHTs: (i) utilize Government funds properly and transparently for their intended purpose; and (ii) account for Government funds in accordance with the MOH’s Office of Financial Management (OFM) and Government of Liberia (GOL) reporting requirements. Significant progress in procurement, as well as financial management and reporting, were previously noted. In addition, CSH’s assistance in personnel 30 and asset management (e.g., LMIS, EUV) have contributed to better utilization of resources not directly financed at county level. The broader question going forward is whether USAID should reconsider the relationship between CSH and the FARA IPs: if the CHTs are ready to take over all FARA facilities, then strengthening CSH's presence may be appropriate. If the IPs will be required for the foreseeable future, then CSH's role may need to be reassessed. OUTCOME QUESTION 3.0: ARE THERE NEW OR EMERGING NEEDS FOR HEALTH SYSTEMS STRENGTHENING NOT WITHIN CURRENT CSH SCOPE? The original CSH SOW was based on the six building blocks of the National Health and Social Welfare Policy and Plan (2011-2021). Though the Investment Plan for Building a Resilient Health System (2016- 2021) expanded the six building blocks to nine strategic areas, two of the current strategic areas (infrastructure and surveillance and emergency response) were in the previous plan but not in the SOW. Infrastructure is certainly an issue (though beyond the scope of CSH), but surveillance and emergency response could be considered. What is new in the National Resilient Health System Plan but absent from the SOW is the emphasis on sustainable community engagement. As defined in the Plan, this strategic area would seem to be more within the purview of PACS, but consideration could be given to including the development of formal mechanisms for voicing satisfaction, expressing complaints, and seeking redress in capacity-building efforts at facility level. The original list of activities in the SOW anticipates virtually all of the health sector’s needs and remains an important source for identifying potential areas of support. However, three areas related to quality improvement merit mention. First, the weaknesses in patient safety were noted previously and by KII in the health facilities. Second, laboratory and diagnostic capabilities were mentioned in the award list and intermittently in the CSH work plans but have not received sufficient attention. Given the importance of diagnostic capabilities for quality health services and their almost total absence in the clinics visited, more attention should be accorded this area. Third, the clinics visited had extensive patient records, which do not seem to play a sufficiently important role in the current quality improvement efforts. SUSTAINABILITY QUESTION 4.0: TO WHAT EXTENT HAVE CSH’S APPROACHES SUPPORTED THE MOH IN FULFILLING ITS CORE HEALTH SYSTEMS FUNCTIONS? COMMENT ON MOH’S IMPROVED CAPACITY, STEWARDSHIP AND OWNERSHIP OF THE CORE HEALTH SYSTEMS FUNCTIONS, THE QUALITY OF STAFF, INTERNAL SYSTEMS, ROBUSTNESS OF M&E SYSTEM. CSH’s approaches have supported MOH in developing its core functions, but not in a systematic and efficient manner. CSH has relied on a package of technical assistance, training, and materials to improve capacity. As shown previously, CSH has addressed capacity through: (i) technical assistance (with an average of 30 long- and short-term consultants in country per month throughout 2016); (ii) the production of materials (100+ reports, tools, forms, etc.); and (iii) a combination of training, workshops, and mentoring which involved more than 1,300 persons. KII’s spoke of being “bombarded” with support and indicated that, while these inputs were necessary, they were not sufficient, and that accompanying financial and logistical support, as well as time would be necessary to actually carry out the work. All respondents at county and district level, and 4 (of 9) at central level requested financial and logistical support to accompany the technical support. CSH’s results framework and annual work plans have been demand-driven, based less on implementing proven (or innovative) interventions than on responding to needs expressed by project beneficiaries and/or USAID during the annual planning process. Without measures of the accrued benefits from or 31 the estimated costs to completion of the selected core function strengthening activities, MOH lacks the information needed to select among improvement measures. KII also noted that: (i) project planning was not sufficiently participatory; and (ii) project implementation was slow in its decision-making, overly bureaucratic in its budgeting, and lacked transparency. MOH’s capacity and ownership of essential health sector functions has not appreciably improved at county level, perhaps due to systemic issues at central level which may have impeded and may continue to impede future developments. Though Table 5 showed only minor differences between CSH and non￾CSH counties, previous analysis of the results demonstrated both quantitative and qualitative achievements over the first two years of project implementation. The project has not, however, demonstrated an ability to develop capacity by: (i) combining system reform measures, organizational strengthening of operational standards and procedures, and workforce development; and (ii) coordinating implementation of these measures at central and county levels. The HCF component has thus far been the most successful, but the QA/QI and SCM initiatives show how difficult implementing change can be. QA/QI attempted to develop a strategy at central level, provide ancillary training and other resources, and implement it in selected health facilities. Though early, the results to date have shown little progress, because the approach has been too burdensome for the beneficiary facilities. SCM has developed the LMIS tool to document commodity consumption and need but has not been able to resolve the broader issues of financing, procuring, and distributing commodities. SUSTAINABILITY QUESTION 5.0: WHAT INTERVENTIONS WERE MOST SUCCESSFUL IN IMPROVING THE ABILITY OF THE MOH TO MANAGE KEY HEALTH SYSTEM EFFORTS AT THE INDIVIDUAL, UNIT, OR COUNTY LEVEL AND MOVE TOWARD A MORE DECENTRALIZED HEALTH SYSTEM? Data were previously presented (Table 13) on the evaluation team’s assessment of the relative success of the different components. The KI were also asked to assess their satisfaction with the results of the components, and the comparison is presented in the following table: Table 15: Comparison of Component Success Rankings Comparing the evaluation team’s rankings with the project beneficiaries’ assessment indicates that: (i) there is a large degree of overlap in the assessments of project component success; and (ii) beneficiaries at county level ranked (on a scale of 1-10) the project’s contributions higher than those at central level. This was due in large measure to the presence of the mentors at the county level and to the absence of sustained assistance at central level. Within these six components (MPW was not included in the analysis), the KII mentioned 35 priority interventions where they felt that CSH had made improvements. It is not possible to identify precisely whether these improvements were at individual, organizational (unit), or county level. However, CSH's capacity development efforts were more successful where organizational policies and procedures were already in place (e.g. CHT/CHB, HR, FM, and MIS) and the 1 2 3 4 5 6 Total Evaluation Team ranking HCF HR HIS L&G QA/QI SCM Beneficiary ranking Central (N=11) L&G HCF HIS QA/QI SCM HR 6.5 CHT (N=15) HR HCF HIS L&G QA/QI SCM 7.8 Total 7.2 32 project’s tools and approaches (technical assistance, materials, training, etc.) focused on developing individual skills to implement these systems. CSH's capacity development efforts were less successful where policies and procedures were new (e.g., QI) and/or where potential gains relied on systemic changes beyond the project’s influence (e.g., SCM). Though CSH makes reference to USAID's Human and Institutional Capacity Development (HICD) Handbook, there is no evidence that the handbook was used to assess, plan, and implement capacity development initiatives. Project activities have had less success where its tools cannot effectively address systemic issues affecting capacity development. SUSTAINABILITY QUESTION 6.0: WHAT HAS BEEN CSH’S CONTRIBUTION TO ACHIEVING THE OBJECTIVE OF THE NATIONAL RESILIENT HEALTH PLAN? In addition to restoring as rapidly as possible the health care gains lost due to the EVD crisis, the Investment Plan for Building a Resilient Health System had more ambitious aims, such as: • Ensure health security for Liberians by creating a robust Health Emergency Risk Management System for prevention, preparedness, alertness, and response to disease outbreaks and other health threats; • Accelerate progress towards universal health coverage by ensuring access to safe and quality Essential Packages of Health Services improved capacity of the health service delivery network; • Establish an enabling environment that: (i) restores trust through community engagement in service delivery and utilization, improved leadership, governance and accountability at all levels; and (ii) narrows the equity gap for the most vulnerable populations. Analysis of the activities proposed by the National Resilient Health System Plan (Annex 13) indicate that of the 64 priority areas, CSH has addressed 15 and could potentially address an additional 16. Half of the service delivery indicators in the Plan are included in those reported on by CSH to USAID; half of the program results indicators in the Plan are included in the proposed results framework. V. CONCLUSIONS Progress has been made in achieving the seven core systems strengthening objectives. CHS’s results have been disappointing as measured by achievement of program targets (Table 4) and development of capacity (Table 5): (i) of 42 targets set for the FY16 program indicators, only a third were achieved; and (ii) capacity development scores have not been uniformly positive in the CSH counties or markedly different from the non-CSH counties. More detailed analysis by component shows that: (i) the results of three components (HCF, HR, and HIS) have been both measured and generally positive; and (ii) all of the components have had some positive results. Based on measurable, positive results, the HCF, HR, and HIS components have been the most successful in meeting the overall objectives of CSH. Measurable progress was documented on process results for HR, HIS, and SCM and on outcome results for HCF and QA/QI; qualitative information was available for MPW and L&G. The process results for HR (records and scholarship management) and HIS (timely reporting) were generally positive, while those for SCM (timely reporting) were mixed. The outcome results for HCF were positive (for both financial management and risk mitigation) but mixed for QA/QI. Outcome results (stock-outs) for SCM were also available but predate the implementation of the revised LMIS and are not directly related to this activity. Consideration of future options will depend on assessments of both the current components’ progress and their relevance for the National Resilient Health System Plan. Given the interrelatedness of health 33 systems strengthening components, elimination of specific components is probably not advisable. However, analysis of the urgency and potential for specific components suggests that progress be considered in the short-, medium-, and long-term as follows: CSH's activities could be re-organized and the indicators re-aligned to better reflect the CDCS and the National Resilient Health System Plan. A revised Results Framework (Annex 15) presents the reorganized project components, aligns these components with the CDCS, and proposes indicators which have already been identified in the different USAID, MOH, and CSH documents. Though the SOW was comprehensive, several value-added activities were not explicitly outlined in the award. The most important include: (i) enforcement of the regulatory mandates; (ii) the health insurance and drug revolving fund initiatives; and (iii) the interoperability road map for harmonizing various sources of health information. With respect to the value-added components of CSH for FARA, the most important contributions have focused on strengthening fiduciary capabilities to ensure that CHTs: (i) utilize Government funds properly and transparently for their intended purpose; and (ii) account for Government funds in accordance with the OFM and GOL reporting requirements. Among the emerging needs, three related to quality improvement merit mention. These include: (i) weaknesses in patient safety; (ii) laboratory and diagnostic capabilities; and (iii) inadequate use of the existing and extensive patient records in the facilities. CSH’s approaches have supported MOH in developing its core functions, but not in a systematic and efficient manner. CSH has relied on a package of technical assistance, training, and materials to improve capacity. As shown previously, CSH has addressed capacity through: (i) technical assistance (with an average of 30 long- and short-term consultants in country per month throughout 2016); (ii) the production of materials (100+ reports, tools, forms, etc.); and (iii) a combination of training, workshops, and mentoring which involved more than 1,300 persons. CSH’s results framework and annual work plans have been demand-driven, based less on implementing proven (or innovative) interventions than on responding to needs expressed by project beneficiaries and/or USAID during the annual planning process. Furthermore, while previous analysis has shown the project’s contribution to specific elements of capacity (e.g., regulatory strengthening, internal systems development, M&E improvement), the project has not demonstrated an ability to develop capacity by: (i) combining system reform measures, organizational strengthening of operational standards and procedures, and workforce development; and (ii) coordinating implementation of these measures at central and county levels. Analysis comparing the evaluation team’s assessment of component success with the KII’s satisfaction with the results of the components showed similar results. The comparison is presented below: 34 Analysis of the activities proposed by the National Resilient Health System Plan indicate that there is already some overlap which could be increased. Of the 64 priority areas identified in the National Resilient Health System Plan, CSH has addressed 15 and could potentially address an additional 16. In addition, half of the program results indicators in the Plan are included in the Results Framework proposed in Annex 15; and half of the service delivery indicators in the Plan are included in those already reported on by CSH to USAID. More generally, three conclusions related to the relevance, efficacy, and efficiency of the project should be mentioned. First, CSH’s objectives remain relevant but its arrangements for implementing activities and monitoring results have weakened its impact. The project’s objectives are consistent with: (i) GOL’s overall objectives for decentralizing authority and responsibility; (ii) MOH’s principal sectoral policies and programmatic strategies; (iii) USAID’s CDCS and HICB approach. As defined by the original SOW, CSH constitutes an important mechanism for contributing to the improved health status of Liberians. Furthermore, CSH contributes to the post-Ebola recovery through its support for: (i) national level initiatives in key substantive areas; (ii) county-level institutions to ensure appropriate Government leadership; and (iii) local use of local data and decision-making to improve the response to potential health needs and avoid delays in dealing with potentially major issues, such as Ebola. However, KII expressed concerns that CSH does not sufficiently: (i) support MOH’s existing approaches and often proposes new or different ways of doing things; and (ii) respect Government policies (including but not limited to DSA). Consequently, although the project’s substance remains pertinent, its execution often tends to create tensions. The KII noted that planning has been more ad hoc than systematic, focusing on a series of deliverables rather than the larger picture. CSH was also criticized for: (i) an overly bureaucratic approach which has slowed project implementation; (ii) a lack of transparency (and subsequent frustration) with CSH budgeting and decision-making; Second, CSH’s contributions have been important, but the project has not succeeded in convincingly documenting its effectiveness. From the perspective of CSH’s contractual obligations with USAID, the project’s outputs, though substantial during the period under review, have not fully achieved the originally planned or subsequently modified activities or results. From the perspective of CSH’s relationship with central and county-level beneficiaries, the view is more mixed, as there were opinions expressed by KII at both levels that: (i) CSH could have and should have done much more; and (ii) the volume of support provided by CSH to MOH, the regulatory bodies, and the CHT outpaced the ability of MOH/CHT to plan, implement, and absorb the HSS measures introduced. Project effectiveness was affected by a focus on building blocks rather than objectives and the post￾Ebola context, which led the project to focus on the details of what to do and how to fit into the evolving financial landscape rather than on the broader overall objectives of the project. M&E has essentially monitored outputs, which may be appropriate early in the project but requires better measurement of direction and speed to maintain momentum. These larger results are what need to be better communicated to MOH leadership. Specifically, though beneficiaries across the components and at the different levels have expressed their appreciation for the project’s interventions, the periodic reports and limited exchanges with senior MOH management have failed to adequately communicate these results in a meaningful way. In other words, where the partner landscape is complex, a project must not only document its results but also “market” them. 35 Third, CSH has thus far been a process-driven project and has not adequately addressed the efficiency of its operations. By focusing on “need” rather than on demonstrably successful interventions and without measures of the benefits to be expected or the costs to be incurred in achieving these benefits, the project has been unable to present options for choosing among activities to senior MOH management or CHT. As one KII put it, the project has at times "bombarded" MOH with support rather than focusing on activities that could be ultimately sustained. Project activities have had more success where capacity development can occur within existing organizational norms and procedures. Project activities have had less success where its tools (technical assistance, materials, training, etc.) cannot effectively address systemic issues affecting capacity development. Based on USAID’s ICB model, CSH’s capacity-building efforts have had success mostly at the organizational level and mostly where norms and procedures have already been developed. It has had less success with systemic issues. This may be due to CSH’s embedding of mentors at county level, since the RBHS evaluation found that embedding technical assistance at central level contributed to systems development. VI. RECOMMENDATIONS CSH should lessen its tendency to overpromise (based on its annual work plans) and underdeliver (based on its available resources). The project's ambitions should be reconsidered and the project re￾designed to emphasize: more focused objectives and fewer activities (to account for limited HR and competing priorities); and better accountability for results (including clearer agreement on measures of project success). The recommendation of the RBHS evaluation (2015) is pertinent: Planning new capacity-building interventions will require strong commitment, input and buy-in of MOH and county leaders and certain conditions to be in place, such as personnel with the capacity to absorb new knowledge and skills to carry out functions and taking a more comprehensive approach to capacity building and organizational development (i.e., ensuring that systems are in place for personnel to be managed, supervised and sufficiently equipped). Currently, the project appears to be a series of one-off exercises without the necessary follow-up. In future, CSH should select and plan activities based on progress to date and future potential for short, medium, and long-term gains. In this way, the project could demonstrate the cogency of its interventions through: (i) the logic and timely sequencing of the planned activities; and (ii) the anticipated endpoints of HSS capacity development. In addition, when reorienting its deliverables and programmatic outcomes, CSH should assess the effectiveness of its current mix of TA, materials, and training to determine whether other types and amounts of complementary inputs might be useful in achieving the project’s objectives. Again, the RBHS evaluation (2015) remains pertinent: All capacity-building activities require: (i) clear, specific and time-bound terms of reference or learning objectives; (ii) technical advisors, trainers or coaches with expertise and relevant experience and capability for knowledge transfer and teaming; (iii) clear benchmarks; and (d) careful monitoring of performance by the project and donor. CSH must increase its engagement with MOH's Senior Management Team in approving the annual work plan and in communicating the results of the activities. Earlier in project implementation, CSH proposed a Memorandum of Understanding (MOU) between CSH and MOH, and a similar mechanism could contribute to better collaboration and communication. 36 To this end, the evaluation team recommends: (i) a reorganization of the project’s components; and (ii) a modification of the existing indicators. In addition, the evaluation team found that including health service delivery indicators in the periodic reports was: (i) inappropriate, given CSH’s objectives; (ii) occasionally inaccurate, since MOH revises the data without necessarily informing CSH; and (iii) unappreciated by MOH, as CSH was simply reporting DHIS2 data without attribution. The project should combine the current focus on demand-driven response with a more results-based approach. This would include an attempt to estimate the cost to completion of achieving these results. While benefits would be difficult to estimate, the cost estimate would provide a criterion of choice for the selection of activities. The project should be modified with a view not to solely responding to MOH's expressed needs, but to collaboratively defining expected results over the short, medium, and long term. USAID should actively engage in these negotiations to avoid three way discussions between CSH and MOH on the one hand and CSH and USAID on the other. The capacity assessment tool, which provided useful measures of changes in capacity should be: (i) improved (to make it more sensitive to changes on the ground); and (ii) considered as a potential tool for programming interventions to link proposed activities to defined indicators of progress in capacity development. 37 ANNEX 1: EVALUATION STATEMENT OF WORK COLLABORATIVE SUPPORT FOR HEALTH (CSH) PERFORMANCE EVALUATION STATEMENT OF WORK 1. Background This Statement of Work (SOW) describes the conditions of work and terms of reference for an external evaluation of USAID’s Collaborative Support for Health (CSH) activity. The objective of CSH is to strengthen the Ministry of Health’s (MOH) overall capacity to consistently and effectively deliver quality health services through targeted health systems strengthening in priority areas. CSH capacity building activities are aimed at the central Ministry and three USAID priority counties: Bong, Lofa, and Nimba. The evaluation will measure results, examine the impact of health systems support on service delivery improvement and management efficiencies, and investigate the effectiveness of specific technical assistance and capacity building approaches. The Mission will use findings from the evaluation to decide whether to exercise two option years for the contract and to inform more strategic USAID investment in the sector. Overview of CSH Activity: Collaborative Support for Health Contract Number: AID-669-C-15-00001 Activity Dates: February 27, 2015 – February 27, 2018 Funding: $21.5 Million Implementing Partner: Management Sciences for Health Subcontractors: Development Innovations Group, the Institute for Healthcare Improvement, Jhpiego, and Results for Development COR: Sinu Kurian A/COR: Pamela Bernard Sawyer Logical Framework: See Figure 1 below. Geographical focus: Bong, Lofa, and Nimba Since 2008, USAID has supported the rebuilding of Liberia’s health system through its flagship Rebuilding Basic Health Services Activity (RBHS) that provided critical support to the Ministry of Health. To build on the successes of RBHS and to support the MOH in the implementation of the 2011-2021 National Health Policy and Plan, USAID signed a contract with Management Sciences for Health and its partners in February 2015 to implement the Collaborative Support for Health (CSH) activity. The overall purpose of the Collaborative Support for Health System Strengthening (CSHSS) contract is to “Improve the Health Status of Liberians”, as laid out in USAID/Liberia’s Health Strategy, and Country Development Cooperation Strategy (CDCS) Development Objective 3. The purpose will be achieved through: 1) Strengthened leadership and governance capacity of the Ministry of Health and Social Welfare, 2) Strengthened Ministry of Public Works’ capacity 38 to manage water supply infrastructure improvements, 3) Institutionalized Quality Assurance (QA) and Quality Improvement (QI) initiatives to improve health care service delivery, 4) Strengthened Human Resources for Health (HRH) Management, 5) Improved Supply Chain Management, 6) Increased financial sustainability of services, and 7) Strengthened Health Management Information System (HMIS). The main priority of CSH is to enhance the capacity of the MOH and County Health Teams (CHTs) to manage implementation of the Essential Package of Health Services (EPHS), while ensuring the necessary health systems and standards are in place to provide high quality services for all Liberians. The Essential Package of Social Services (EPSS) was initially part of CSH’s scope of work; however, as of 2015, the Ministry of Social Welfare moved to the Ministry of Gender. Therefore, CSH’s activities were realigned, to reflect this division of Ministries. The expected results of CSH are closely aligned with those of the National Health Policy and Plan (2011-2021) and the Investment Plan to Building a Resilient Health System (2016-2021), and implementation has been done in close collaboration with the MOH and the CHTs in the USAID priority counties of Bong, Lofa, and Nimba. The results framework below shows the expected results of this activity and their relationship with the four intermediate results of the Country Development and Cooperation Strategy (CDCS) for USAID/Liberia (Figure 1). Through health systems strengthening initiatives that target known gaps and emphasize governance and accountability, CSH also enhances simultaneous USAID investments in health service delivery and community health. Specifically, CSH provides: 1) Support for decentralization: CSH supports the MOH’s decentralization policy and process to ultimately accelerate improvements in health outcomes. CSH focuses on institutionalization and functional sustainability of systems that are not personnel dependent and foster continuous quality improvement at the most responsive levels of management through capacity building and institutional support at both central and county levels for each of the health system building blocks. This contract supports the MOH’s commitment to progressively decentralize responsibilities, to CHTs, District Health Teams (DHTs), and County Health Boards as part of an established wider County Administration. The support provided by CSH to the relevant units at the national level and to the CHTS aims to increase capacity within these units to carry out their essential public health functions more efficiently and effectively, specifically in the areas of leadership and governance, human resource management and planning, public financial management, and data. 2) Support to the Essential Package of Health Services (EPHS) and Essential Package of Social Services (EPSS): CSH works to improve the effectiveness of the MOH and CHT to fully implement the EPHS, while ensuring the necessary health systems and standards are in place for high quality service delivery in Bong, Lofa, and Nimba. Through this mechanism the EPHS will be adapted so that it prioritizes services that reflect the prevailing disease burden and health conditions affecting the population and provide those services that are necessary for social well-being, especially among the most vulnerable. 3) Support for the National Capacity Development Strategy: The GOL has adopted a national strategy to improve leadership, technical, and administrative capacities across all government agencies and ministries. Within the health sector, Liberia’s National Capacity Development Strategy focuses on building a competent workforce by ensuring that more health professionals are trained through pre-service and in-service development programs; certification standards are upheld and extended across the workforce; greater gender balance is achieved in the workforce, with additional support and empowerment being 39 extended to female professionals; and that health workers are deployed to serve all counties in Liberia instead of concentrating most human resources in urban areas while remote districts are under-served. The interventions implemented through CSH prioritize building the institutional capacity of the MOH, the CHT, and strengthening capacity of Liberian health workers at county and national levels for a Liberian-led health system. CSH applies USAID’s Human and Institutional Capacity Development model, which acknowledges that training is necessary but not sufficient. The broader environmental and individual context comprises six performance factors: information, resources, incentives, knowledge/ skills, capacity, and motives. The work of CSH is designed to go beyond training. Systemic issues are addressed taking into account recruitment, regulation, training, resources for supportive supervision, and skills to operationalize all health system components at the central and county levels. Challenging aspects of beginning implementation in the post-Ebola environment included significant changes in the donor landscape and greater need for coordination of project activities with many projects, donors and stakeholders, as multiple donors began to operate in similar technical areas as CSH. The development of a national health sector investment plan that focuses on health system strengthening and priority disease programming to make the Liberian health system more resilient post-Ebola, led to the development of a number of multiple-partner technical working groups. Both coordination and funding issues impacted the CSH scope of work, which evolved throughout the first year, with some activities being eliminated, others added, and with the program being generally tweaked to better fit USAID’s needs and the post-Ebola environment in Liberia. Figure 1. Results Framework for CSH 40 To link the CSH results framework to the overall USAID/Liberia objectives, refer to the USAID Liberia results framework in Annex 1. The development hypothesis of CSH is based on the theory that improvements in the health status of the Liberian population can be achieved through strengthening of the health system at the national and county levels. CSH outputs aim to improve performance of the health system at the national and county level by enhancing leadership and governance, human resource management, medicines and supply chain management, information (HMIS), financial management, and service delivery. It is hypothesized that improvements in the six HSS building blocks at the county and central MOH will lead to enhanced financial sustainability and the provision of contextually appropriate health services at local levels that are determined by local disease epidemiology and service data. While CSH does not provide health services, it is anticipated that improvements in the performance of key management and governance functions at the county and national levels will lead to better capacity to monitor and use the data to plan for the services. It is expected that this will create greater confidence in the services among the population leading to greater utilization of key public health services. Figure 2 below shows a graphical representation of how the activities interact with the system to induce longer term improvements in the performance of the health sector and ultimately improvements in the health status of Liberians. Figure 2: Theory of Change of CSH Short to medium term changes Long term changes 41 Trained staff in Leadership, supply chain management, financial management, human resource, service delivery & information Increased use of technology to support health service Better regulations to support HR, service delivery etc. Improved service delivery standards Community mobilization Better quality and timely data Qualified and motivated staff in the health sector Increased data use and evidenced based decision making Improved financial management and oversight Enhanced quality service including appropriate diagnostics Improved forecasting of drugs and services and reduced stock outs of essential medicines Regional systems that provide contextually appropriate services + Earlier uptake for essential services Improved health status of Liberians Higher ANC coverage and access to skilled support during deliveries Higher immunization rates + Safer water supply Reduced maternal mortality rates Reduced infant mortality rates 2. Purpose of CSH Evaluation The purpose of this evaluation is to provide an independent and in-depth examination of the overall progress of the CSH activity in Liberia. The evaluation will identify achievements, performance issues, and constraints related to activity implementation and effectiveness. The evaluation will also identify results and lessons learned from implementation and will provide succinct, actionable recommendations to determine which component(s) of CSH to scale up, modify, or re-design for option years and/or in other ongoing related activities or future procurements. To the extent possible, there will be an emphasis on quantitative results achieved as the result of the CSH. Evaluation findings and recommendations will be shared and discussed with USAID/Liberia, implementing partners, and relevant GOL partners. 3. Evaluation Questions Outcomes • To what extent has progress been made in achieving the seven core systems strengthening objectives under CSH? o Which components have been most successful in meeting the overall objectives of CSH? At what level? Provide a prioritized list of components. o If CSH will receive option years, which areas should be strengthened and which areas could be discontinued? Consider their relevance against health system reforms/post-Ebola priorities. o What are recommendations for maintaining, revising, or removing indicators based on a revised health system strengthening activity? Include a proposed CSH More control → Attribution Less control → contribution 42 results framework aligning recommendations with the overall USAID/Liberia results framework. • What are some of the value added components of CSH (and the activity staff), if any, to the Ministry of Health, donor community or USAID that were not explicitly outlined in the award? o Specifically, what has been the value added components of CSH on the FARA (implementation, management)? • Are there new or emerging needs for health systems strengthening not within current CSH scope? Sustainability • To what extent have CSH’s approaches supported the MOH in fulfilling its core health systems functions? Comment on MOH’s improved capacity, stewardship and ownership of the core health systems functions6 , the quality of staff, internal systems, robustness of M&E system. o What interventions were most successful in improving the ability of the MOH to manage key health system efforts at the individual, unit, or county level and move toward a more decentralized health system? o What has been CSH’s contribution to achieving the objective of the National Resilient Health Plan? 4. Evaluation Methods The evaluation is expected to apply both quantitative and qualitative methods for data collection and analysis. The evaluation team will conduct a desk review of available literature including project documents, quarterly and annual reports. The evaluation team will also look at RBHS, FARA, and CSH assessments that were used to determine interventions. Site visits in the field will provide qualitative data for analysis through methods such as in-depth and key informant interviews, focus groups, and direct observation. Sampling should include CSH counties – Lofa and Bong – and non-CSH counties – Margibi and Grand Bassa. CSH is planning to expand to Margibi and Grand Bassa so this could provide a level of baseline for future evaluations. The selection provides a balance of County Health Teams facing diverse challenges. The evaluation team is expected to meet with central and county-level MOH staff, health facility workers, health partners, community leaders, scholarship awardees, CSH staff and others in order to acquire the data needed to respond to the evaluation questions. Relevant donors include the World Bank and Japan International Coordination Agency (JICA) for their work in health system strengthening, specifically to explore what indicators they use to measure progress. Primary collection of quantitative data and large-scale structured surveys are beyond the scope of this evaluation. It is expected that the evaluation team will use data provided by the implementing partner in regular quarterly and annual reports, performance reporting, and special purpose publications for most of the quantitative data required. The team will draw on DHIS2 and iHRIS data from the MOH. 6 The U.S. Government defines country ownership as “the continuum of actions taken by political and institutional stakeholders in partner countries to plan, oversee, manage, deliver, and finance their health sector and achieve health goals. These actions advance sustainable, quality health programs that are locally owned and responsive to the needs of host country nationals.” 43 The USAID/Liberia Health team will provide documents for the desk review, as well as contact information for prospective interviewees. The evaluation team will be responsible for identifying and reviewing additional materials relevant to the evaluation, as well as additional contacts. Illustrative data sources include but are not limited to: 1. CSH Contract 2. CSH Annual Work Plans 3. CSH Activity Monitoring and Evaluation Plan 4. Performance Indicator Database System data 5. Quarterly and annual reports 6. Organizational capacity assessments and tools 7. FARA County Readiness assessments and tools 8. Other CSH assessments or studies 9. Data quality assessment reports 10. USAID/Liberia CDCS 11. Liberia Service Availability and Readiness Assessment 12. Liberia Demographic and Health Survey 13. Other related national data and reports 5. Deliverables and Timeline Evaluation deliverables include: a. Evaluation Team Planning Meetings b. Inbrief with USAID/Liberia, GOL and Implementing Partners c. Inception Report with work plan and data collection instruments d. Debrief with USAID/Liberia, GOL and Implementing Partners • Present the major findings from the evaluation to USAID/Liberia and partners through a PowerPoint (or similar) presentation. The debriefing will cover initial findings, conclusions and preliminary recommendations. The team, in consultation with USAID, should consider doing two to three presentations: one to USAID, one to partners, and one to the MOH. This will be determined by the presence of sensitive information, if any, in the report. e. Original data and data sets -- Copies of secondary quantitative data sets, transcripts of interviews and focus groups, and notes from direct observations. Quantitative data sets should be submitted to the DDL, per Agency policy. f. Draft Evaluation Report - A draft report should be submitted to Liberia Strategic Analysis (LSA) for review, and LSA must submit the draft report to USAID/Liberia within three weeks after the in-country work is conducted. The written report should clearly describe findings, conclusions, and recommendations and conform to USAID requirements outlined below. USAID will provide comments on the draft report within five to ten working days of submission. As agreed upon during planning meetings, LSA will also submit the draft report to the GOL for its comments and shared with the Implementing Partner for an opportunity to disclose a statement of difference, if applicable. 44 g. Final Report - The Team will submit a final report that incorporates the GOL’s and Mission’s comments and suggestions no later than ten days after final, written comments on the team’s draft report have been submitted by all parties. h. Learning Event – Liberia Strategic Analysis will facilitate a learning event one to two months after the report has been finalized to engage USAID/Liberia and stakeholders in the utilization of recommendations. This will be combined with an After￾Action Review, prescribed as good learning practice per ADS 201, to support USAID in the incorporation of recommendations into work plans, project or activity design, and the next Country Development Coordination Strategy. The draft evaluation report should meet the following criteria: i. The report should be in line with USAID Evaluation Policy (see Appendix I – Criteria to Ensure the Quality of the Evaluation Report) and USAID Secretariat Style guide. ii. The report should be no longer than 30 pages, excluding executive summary, table of contents, and annexes. iii. The report should include a 3-5 page Executive Summary highlighting findings and recommendations. iv. The report should represent a thoughtful and well organized effort to objectively respond to the evaluation questions. v. The report shall address all evaluation questions included in the SOW. vi. Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides shall be included in an Annex in the final report. vii. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, etc.). viii. Evaluation findings should be specific, concise and supported by strong quantitative or qualitative evidence. ix. Recommendations should be action-oriented, practical, specific, and evidence￾based. x. The final report should be edited and formatted. xi. Liberia Strategic Analysis must submit the final evaluation to the Development Experience Clearinghouse. The report will be submitted electronically. The final report will be edited/formatted by the contractor and provided to USAID/Liberia five working days after the Mission has reviewed the content and approved the final revised version of the report. The final evaluation report must be 508 compliant and comply with the USAID Evaluation Policy: http://www.usaid.gov/ sites/default/files/documents/1868/USAIDEvaluationPolicy.pdf Timeline The evaluation should follow the timeline for producing deliverables outlined in the Gantt chart below. 45 Performance Evaluation Gantt Chart Activity # of days Week 1-2 Week 3-5 Week 6 Week 7 Week 8 Week 9 Week 10 Week 11 Week 12 Week 13 Week 14 Week 15 USAID shares SOW with LSA 1 LSA publishes call for CVs 14 LSA recruits and onboards team members 21 Desk Review 5 Team travels to Liberia 1 Team planning meeting - internal 1 Submit draft Inception Report to USAID Planning Meetings with USAID and CSH/MOH 1.5 Inbrief with USAID and CSH/MOH 1 Prepare for field work 2.5 Submit final Inception Report to USAID Field Work 20 Preliminary analysis 1 Debrief with USAID and CSH/MOH 1 Travel from Liberia 1 Analysis and report drafting 9 LSA quality control and revisions 5 LSA submits Draft Report to USAID USAID, CSH, and MOH review the draft report 5 Evaluation team incorporates comments and feedback 5 LSA quality control 5 LSA submits Final Report to USAID LSA facilitates Learning Event with USAID, CSH, MOH, and stakeholders 1 1-2 month s after PE Prep work by Team Leader and national consultant 1 46 6. Team Composition All key staff should have methodological and/or technical expertise, regional or country experience, language skills, team lead experience and management skills, etc. Team leaders for evaluations must be an external expert with appropriate skills and experience. Additional team members can include research assistants, enumerators, translators, logisticians, etc. Teams should include a collective mix of appropriate methodological and subject matter expertise. Evaluations require an Evaluation Specialist, who should have evaluation methodological expertise needed for this activity. Similarly, other analytic activities should have a specialist with methodological expertise related to the evaluation. Note that all team members will be required to provide a signed statement attesting that they have no conflict of interest, or describing the conflict of interest if applicable. USAID/Liberia recommends the following staffing structure for the evaluation: 1. An international senior-level evaluation specialist Team Leader with extensive experience and expertise in health systems development, health management information systems, public health management, and/or institution building, who will be an international consultant with extensive USAID program implementation and evaluation experience, must possess proven skills in evaluation and analysis of transitioning development public health programs. S/he must have a proven track record supervising teams in the field and producing high quality and concise reports, as well as extensive experience working in Africa and similar fragile settings. At least ten (10) years of experience in evaluation management, and qualitative data collection and analysis; experience in conducting evaluations and designing performance evaluations, preferably of USAID projects. Ability to produce high quality evaluation reports in English is essential. Strong interpersonal skills are required. 2. An international public health evaluation specialist with experience in evaluating health system strengthening mechanisms in developing countries. At least five (5) years of public health experience focused on research using sector-specific assessment tools and guidelines. S/he must have experience managing research initiatives in West Africa, and have demonstrated grasp on incorporating qualitative analysis software in evaluations. Strong interpersonal skills and American English language speaking and writing skills also essential. This candidate will serve as the assistant team leader to support strong organization of the evaluation design, development of instruments, and oversight of analysis and report writing. 3. An international senior public health advisor with experience in designing and/or evaluating health system strengthening mechanisms in developing countries, preferably through USAID. Should have a background in implementing quality assurance strategies to improve health care service delivery. At least ten (10) years of public health experience, and some experience managing or implementing research programs is preferable. Strong interpersonal skills and American English language speaking and writing skills also essential. This advisory role will ensure that instruments and resulting findings and recommendations reflect technical sector best practice. 4. A national senior or mid-level Liberian Health Context Specialist with at least five (5) years’ experience and knowledge about the health context in Liberia, particularly on county and facility level health system administration. Must have strong gender and social analytical 47 skills, specifically in designing and evaluating health programs. The incumbent must be able to conduct interviews and focus group discussion and analyze the resulting data. Ability to conduct interviews and discussions in at least one local Liberian language. Strong American English language speaking and writing skills also essential. 5. A national evaluation specialist with at least five (5) years relevant experience in qualitative and quantitative data collection methods. The local consultant must have experience evaluating health programs, specifically analyzing HMIS data, and have demonstrated logistics and planning skills. He/she should broad knowledge of Liberian health issues and the ability to assist in key informant interviews, data collection, qualitative instrument preparation, and analysis of collected data. Combined qualifications should include: expertise in health systems strengthening (delivering essential health services, health workforce, health information system, access to essential commodities, health system financing, and governance and leadership), decentralization. Ability to communicate clearly in American English. Among these candidates, the team should have a background in EPHS clinical treatment guidelines, institutionalized quality assurance and quality improvement, regulatory strengthening of service provision and pharmaceuticals, and health financing and reform. USAID leaves to the offeror’s discretion other necessary team members/staff for the evaluation (e.g., Logistics, scheduling, translation, data analysis). Aside from the above mentioned key personnel, the offeror must decide how the evaluation team should be structured in order to successfully address the evaluation questions. All attempts should be made for the team to be gender balanced and to include local (Liberian) experts. A statement of potential bias or conflict of interest (or lack thereof) is required from each team member. USAID may propose internal staff from USAID/Liberia or from Washington to accompany the team in this evaluation as observers. As observers, their role will be to provide, when asked, background information and to reply to the external evaluators’ questions. They will review and comment on the report for accuracy, but evaluators may accept or reject comments. The final report should reflect the opinions of the external evaluators and is the sole responsibility of the selected evaluation team. The contractors will officially report to the Mission’s M&E Officer and technical guidance/leadership will be provided by the Government Agreement Technical Representative (GATR) and A/GATR. 7. Logistics and Level of Effort The contractor will be responsible for all international and in-country administrative and logistical support, including identifying and fielding appropriate consultants (international and local). The evaluation team should be able to make all logistical arrangements including vehicle rental for travel within and outside Monrovia and should not expect any logistical support from 48 the Mission. The team should also make their own arrangements for venues for team meetings, and equipment support for producing the report. Evaluation team members are authorized and expected to work a six-day week. Travel over weekends may be necessary. Work should commence as soon as practicable, but no later than mid-March 2017. For planning purposes, contractors should be aware of Liberian and US holidays during the evaluation time frame. The evaluation should follow the illustrative schedule and level of effort given below. Task/Deliverable Estimated time (Days) Review background documents & preparatory work (offshore): Draft work plan submitted to USAID/Liberia 5 int’l / 3 nat’l / 3 advisor Travel to Liberia-expatriate team members 2 Team Planning Meetings in Monrovia with implementing partners and USAID 2 In-brief with USAID/Liberia and prepare for field work 4 Field work: Data collection and On-Going Data Analysis 20 (including field travel) / 10 advisor Analysis and report drafting 10 int’l / 7 nat’l / 5 advisor Presentation and debrief with GOL, USAID/Liberia and IPs by Evaluation Team 1 LSA performs quality assurance check and edits draft report 4 LSA submits draft report to USAID/Liberia and GOL 1 USAID and MOH provide comments on draft report 5 Evaluation Team Leader revises draft report to incorporate comments and submit final report 5 TL / 1 int’l team member LSA performs quality assurance check and submits final report 5 LSA facilitates learning event 1 TL / 2 nat’l Total time required 65 Working days Team Leader (including international travel) 50 Working days international team member (including international travel) 45 Working days senior public health advisor 26 Working days local team members 39 49 ANNEX 2: LIST OF DOCUMENTS REVIEWED Government / MOH documents Policies • National Health Policy & Plan 2007-11 • National Health Promotion Policy, 2009 • Social Welfare Policy, 2009 • National Monitoring and Evaluation Policy and Strategy for the Health Sector, 2009 • National Management Information Systems Policy, 2009 • National Monitoring and Evaluation: Policy and Strategy for the Health Sector, 2009 • Liberia National Capacity Development Strategy, 2011 • National Health and Social Welfare Policy, 2011 • National Health and Social Welfare Financing Policy and Plan 2011-2021, 2011 • National Human Resources Policy and Plan for Health and Social Welfare 2011-2021, 2011 • National Policy on Decentralization and Local Governance, 2011 • Liberia NHQ Strategy, 2017 Plans • National Management Information Systems Strategy and Implementation Plan, 2009 • Supply Chain Strategy Plan 2011-2015, 2010 • HR Policy Plan, 2011-2021, 2011 • Financing Policy Plan MOH, 2011-2021, 2011 • National Decentralization Implementation Plan (NDIP) Phase I: 2013-2017, 2012 • Community Health Road Map, 2014 • Investment Plan for Building a Resilient Health System, 2015 • Leadership and Governance Development Plan: Liberian Ministry of Health Central Level, June 2015 • Supply Chain Master Plan 2010– 2020: Five Year Review and Update, 2015 • Liberian Health Information System Strategic Plan 2016-2021, January 2016 • RMNCAH Investment Case 2016-20, 2016 • Republic of Liberia, consolidated Operational Plan FY 2016/17 Reports • The National Census of Health Workers in Liberia, 2010 • MOHSW Situation Analysis, 2011 • Governance Commission Annual Governance Report- 2013: Delivering Education and Health Services to the People • Governance Report 2013 Conclusions • MOHSW Annual Report, 2013 • MOHSW Annual Report, 2014 • Republic of Liberia Ministry of Health and Social Welfare: Joint financial management assessment report, 2016 Project Documents RBHS documents • Health facility management training evaluation, 2010 50 • RBHS HSS Qualitative Capacity Assessment, 2012 • Capacity Assessment – Qualitative Interview: County Health and Social Welfare Team (CHSWT) level, 2012 • The Road To Recovery: RBHS, 2012 • Technical Brief: capacity development, an integrative approach to building capacity in Liberia • Recommendations for strengthening regulation of health professions education in Liberia, 2011 • Liberia WASH Compact Review, 2012 • Building capacity in health facility management: guiding principles for skills transfer in Liberia, 2010 • RBHS Community Health Services Technical Brief • Introducing health facility accreditation in Liberia, 2011 • EPHS Primary Care: The Community Health System, 2011 • EPHS Secondary and Tertiary Care: The District, County, and National Health Systems, 2011 • Results-Based Financing for Health (RBF): Identifying Indicators for Performance-Based Contracting (PBC) is Key: The Case of Liberia, 2010 • Evaluation of Bomi county performance based contracting-in pilot, February 2012 • Technical Brief: Building Sustainable Capacity for Performance-Based Financing in Liberia’s Health System, 2014 • Synthesis of health financing studies and alternative health financing policies in Liberia: discussion paper, 2010 • RBHS Improving HMIS performance: measurement and intervention, technical brief • RBHS Year 2 Assessment, 2011 • RBHS Final Evaluation, 2015 Documents provided by CSH Administration • Contract CSHSS AID-669-C-15-00001 CSHSS- MSH signed 27feb15 • CSH Statement of Work • CSH Workplan Apr 15-Mar 16 • CSH Workplan Oct 16-Sep 17 • CSH Work Plan 2015-16 budget clarification • CSH Program Activity M&E Plan, 2015-2018 • High level performance indicator tables for CSH - Performance Indicators, Baseline Values, calculation, and Year One Targets for the CSH Program • PIDS Report - Collaborative Support for Health Program, 2016 • Program & service delivery indicators Oct 15- Sep 16 • Quarterly Report Jul-Sep, 2015 • Quarterly Report Oct-Dec, 2015 • Quarterly Report Jan-Mar, 2016 • Quarterly Report Apr-Jun, 2016 • Quarterly Report Jul-Sep, 2016 • Quarterly Report Oct-Dec, 2016 Component 1: L&G Central level • MOH Liberia LDP+ Training of facilitators workshop report, February 2015 • Liberian Board of Nurses and Midwives: Governance Workshop June 27-30, 2016 • Liberian Board for Nursing and Midwifery: Mid-term review strategic plan 2012 to 2017 • CSH LBNM STTA Report, 2016 • Report on Training of Facilitators for Regulatory Bodies, 22 – 26 February 2016 51 County level • Leadership and Governance Development Plan, 2015 • Contracting in Guidelines: Summary of systems, procedures, and core competencies for the County Health Team (CHT), May 2013 rev. • CSH CHT Self-Assessment Tool, 2016 • CSH County Health Board Report, 2016 • CSH County Capacity Assessment Report, 2016 • CSH County HSS Capacity Assessment Tool, 2016 • CSH Self-Assessment, 2016 • County Health Board Operational Manual-STTA Report, 2016 • Rapid County Capacity Assessment: Bong, Lofa, and Nimba, December 2015-January 2016 • HSS capacity assessment – quantitative instrument county health team level, 2016 • CSH CHB Operational Manual, 2017 • County Health Board Orientation: Facilitators Guide, February 2017 Component 2: MPW • CSH Waterpoint Atlas Analysis, 2015 • WASH and Environmental Health Package in Health Facilities: Programme for early recovery and resilience building from EVD outbreak in Liberia, October 2015 • CSH DIG Rapid Assessment of MPW, 2015 • CSH MPW Capacity Building Plan 2015, rev. 2016 • Analysis for Accountability for WASH services Sustainability in Health in Liberia: Fact Sheet, 2016 • Analysis of Accountability for WASH services Sustainability within Health System in Liberia, 2016 Component 3: QA/QI • MOH, Essential Package of Health Services-Primary Care, 2010 • MOH, EPHS Accreditation Final Report, 2013 • CSH Joint Supervision Tool, 2015 • CSH Quality Improvement Indicators, 2016 • CSH/IHI, Quality Improvement Leadership Methods (QILM) Training, 2016 Component 4: HR • Terms of Reference for Human Resources for Health Technical Working Group, 2015 • MOH, Human Resources for Health Structure at Central/County Levels, December 2015 • iHRIS HR supervision tool, 2015 • CSH Nursing School Assessment, 2015 • CSH In-service training situation analysis, 2015 • CSH iHRIS installation: STTA Report, 2016 • CSH HR Mentorship Program Report, 2016 • CSH CPD SOP Review, 2016 • CSH CPD Activity Log, 2016 • Ministry of Health rapid training needs assessment, 2016 • CSH MOH Scholarship guidelines, 2016 • Scholarship Monitoring Report-March 22-23, 2016 Component 5: SCM • CSH eLMIS concept note, 2015 • CSH Rapid pharmaceutical supply chain analysis, 2015 52 • CSH LMIS technical design, 2016 • CSH LMIS Design Document Endorsement Letter, 2016 • Liberia EUV report, November 2016 Component 6: HCF • Contracting-in guidelines: Background and Context, 2013 • Ministry of Health Financial Management Policies and Procedures Manual, September 2015 • CSH FM County assessment report, 2015 • Technical Report: Rapid Participatory Assessment of Liberian PBF implementation, 2015 • Integrated counter-verification report: April-June, 2016, Joint PBF - EPI - Malaria Counter￾Verification and Data Harmonization • CSH PBF Operations Manual, 2016 • Health Financing in Liberia: Consumer Preference Market Research: Consultancy Scope of Work, 2016 • Costing of the Essential Package of Health Services: PHC level I and II, December 2016 • Concept Note on the Design of a Revolving Drug Fund Pilot Program in Liberia, August 2016 • Health Financing in Liberia: Consumer Preference Market Research, 2017 • CSH Financial Management County Update, 2017 Component 7: HIS • CSH HIS work plan Mapping, 2016 • CSH HMIS training manual, 2016 • CSH MIS interoperability roadmap, 2016 • CSH HIS Strategy and Plan, 2016-2021 USAID documents • USAID ICB Model Handbook, 2010 • USAID GHI Liberia Strategy, 2011 • An assessment of decentralization and local governance in Liberia: a strategic review with recommendations for USAID/Liberia’s cross-cutting decentralization strategy, 2012 • USAID CDCS 2013-17, 2013 • FARA Midterm evaluation, 2013 • Technical report: the FARA Mechanism Leveraging and Strengthening Local Capacity to Rebuild Basic Health Services • GEMS Annual Report, 2014 • President’s Malaria Initiative Liberia Malaria Operational Plan FY 2017 Miscellaneous Component 1: L&G • The Liberian National decentralisation and local development program (LDLD), 2007 • ICB policy summary tables, 2010 • An analysis of Liberia’s 2007 national health policy: lessons for health systems strengthening and chronic disease care in poor, post-conflict countries, 2011 • Decentralizing the State in Liberia: The Issues, Progress and Challenges, 2014 • Countries at the Crossroads, Countries at the Crossroads 2012: Liberia • Assessing the Implementation Barriers to the “Accelerated Action Plan to Reduce Maternal and Newborn Mortality” in Liberia, 2013 • Decentralizing the State in Liberia: The Issues, Progress and Challenges, 2014 • Improving health interventions in conflict affected Liberia: A community-based approach, 2014 • Database at a Glance: Licensing and Accreditation, 2014 • EPHS Country Snapshot: Liberia, 2015 53 • Governance and Health in Post-Conflict Countries: The Ebola Outbreak in Liberia and Sierra Leone, 2016 Component 3: QA/QI • The evolution of health service delivery in the Liberian health sector between 2003 and 2010: A policy analysis, 2012 • Can the health system deliver? Determinants of rural Liberians’ confidence in health care, 2015 Component 4: HR • Rebuilding human resources for health: a case study from Liberia, 2011 • Policy Options for Addressing Health System and Human Resources for Health Crisis in Liberia Post-Ebola Epidemic, 2015 • Cost of scaling up the health workforce in Liberia, Sierra Leone, and Guinea amid the Ebola epidemic: Frontline Health Workers Coalition Analysis - March 2015 Component 6: HCF • First Annual Audit of Pooled Fund, 2009 • Liberia’s second-round national health accounts, part I: institutional health spending 2009/10 • Innovative Financing in Early Recovery: The Liberia Health Sector Pool Fund, 2012 • Financing Liberia’s healthcare, 2012 • Measuring good pooled funds in fragile states, 2012 • Pooled Funding to Support Service Delivery Lessons of Experience from Fragile and Conflict￾Affected States, 2013 • National Health Insurance Design in Liberia: Key Considerations for Equitable, Efficient, and Sustainable Health Care Access November, 2014 • Quarterly Report of the Health Sector Pool Fund October 1, 2014, through December 31, 2014 • WHO Official Development Assistance for health to Liberia, 2011 • Public Expenditure and Financial Accountability Assessment (PEFA) 2016 on Liberia’s Public Financial Management Systems Final Report, 2016 Component 7: HIS • PRISM framework: a paradigm shift for designing, strengthening and evaluating routine health information systems, 2009 • Performance of Routine Information System Management (PRISM) framework: evidence from Uganda, 2010 • Liberia DHS, 2007 • Liberia DHS, 2013 • 2011 Lot Quality Assurance Sampling Survey in Liberia, 2011 54 ANNEX 3: LIST OF PERSONS CONSULTED CENTRAL LEVEL Name Component Position F M Data Collection Method Date Jarso Jallah-Saygbe Leadership/Governance Deputy Director General, Training and Development, LIPA 1 0 Key Informant Interview April 10, 2017 Torbandu Henry Kohar QA/QI Acting Laboratory Director (MOH) 0 1 Key Informant Interview April 10, 2017 John Harris SCM Director, SCM Unit (MOH) 0 1 Key Informant Interview April 11, 2017 Justin Korvayan Samuel Johnson Leadership/Governance Head, Decentralisation and Governance Unit (MOH) Capacity Building Officer, Decentralisation and Governance Unit (MOH) 0 2 Group Informant Interview April 11, 2017 Cecilia Morris Cecilia Gkpangola Mary Wah Darboi Korkoyah Leadership/Governance Chairperson, LBNM Registrar, LBNM Director, LBNM Director, M&E, LBNM 4 0 Group Informant Interview April 11, 2017 Louise Mapleh Topkah Garmetta Finance Manager, FARA Officer, FARA 2 0 Group Informant Interview April 11, 2017 Victor Kiatamba Marvin Davis Moibah Sherif HRH Officer, HR (MOH) Assistant Director, Training and Development (MOH) Assistant Director, HR and Planning (MOH) 0 3 Group Informant Interview April 11, 2017 Nowai Johnson Leadership/Governance Advisor, Education and Training (JHPIEGO) 1 0 Key Informant Interview April 11, 2017 Rustion Yarnkro QA/QI Advisor, In-Service Training (JHPIEGO) 0 1 Key Informant Interview April 11, 2017 Tehmeh Lekilay Leadership/Governance Medical Research Coordinator, LMDC 0 1 Key Informant Interview April 19, 2017 Munirat Ogunlayi Finance Health Specialist, World Bank 1 0 Key Informant Interview April 19, 2017 Mike Molbah HIS M&E Officer (MOH) 0 1 Key Informant Interview April 20, 2017 Vera Musa HRH Manager, PBF 2 1 Group Informant Interview April 20, 2017 Anne Fiedler Birhanu Sendek Getahon Mantue Reeves QA/QI Chief of Party, MCSP Technical Director, MCSP MER Officer, MCSP 1 2 Group Informant Interview April 20, 2017 55 Name Component Position F M Data Collection Method Date Phillip Benah Jammie Kekulah Garrison Kerwilliam QA/QI Director, QMU (MOH) Secretary to Director, QMU (MOH) IPC Coordinator 1 2 Group Informant Interview April 20, 2017 Cuallah Jabbe Leadership/Governance Director, Community Health (MOH) 1 0 Key Informant Interview April 21, 2017 Roland Kessullu Nuaker Kwenah Finance PBF Manager (MOH) Health Financing Officer (MOH) 0 2 Group Informant Interview April 21, 2017 Moses Massaquoi HRH CHAI 0 1 Key Informant Interview April 21, 2017 Jefferson Harris Nathaniel Woart, Sr. Leadership/Governance Inspector General (LPB) Deputy Registrar (LPB) 0 2 Group Informant Interview May 2, 2017 Wondwossen Teffera Water WASH Systems Advisor (USAID) 0 1 Key Informant Interview May 2, 2017 Francis Karteh Chief Medical Officer (Liberia) 0 1 Key Informant Interview May 3, 2017 Total 14 22 COUNTY LEVEL Name Component Position F M Data Collection Method DATE Prince Woloway Harris Nyakaryah HRH Assistant Officer, HR (Nimba CHT) Officer, HR (Nimba CHT) 0 2 Group Informant Interview April 12, 2017 Karntay Deemie QA/QI Clinical Supervior (Nimba CHT) 0 1 Key Informant Interview April 12, 2017 Paul Nyanzee Prince Sesay Leadership/Governance Community Health Department Dir. (Nimba CHT) County Health Services Administrator (Nimba CHT) 0 2 Group Informant Interview April 12, 2017 Fester Lee Bleh Vah Jackson Messan SCM Focal Person, Drugs Depot (Nimba CHT) Assistant, Drugs Depot (Nimba CHT) Acting County Pharmacist (Nimba CHT) 0 3 Group Informant Interview April 12, 2017 Gonleyen Dahn HIS M&E Officer (Nimba CHT) 0 1 Key Informant Interview April 13, 2017 Paul Zwuogbae Cyrus Quewon Sam Buoay Finance Accountant (Nimba CHT) Accountant (Nimba CHT) Procurement Officer 0 3 Group Informant Interview April 13, 2017 56 Name Component Position F M Data Collection Method DATE Jerries Limou Walker Resean Jomah Kplaiwru HRH HR Officer (Bong CHT) HR Assistant (Bong CHT) 0 2 Group Informant Interview April 17, 2017 Barsee Zogbaye QA/QI Clinical Supervisor (Bong CHT) 0 1 Key Informant Interview April 17, 2017 Samuel Gayflor SCM Chief Pharmacist (Bong CHT) 0 1 Key Informant Interview April 17, 2017 Korwan Flomo Finance Chief Accountant (Bong CHT) 0 1 Key Informant Interview April 17, 2017 Nyanquoi Urey HIS M&E Officer (Bong CHT) 0 1 Key Informant Interview April 17, 2017 Gorbee Logan Alphonso Kofa Leadership/Governance County Health Officer (Bong CHT) Community Health Department Dir. (Bong CHT) 0 2 Key Informant Interview April 17, 2017 Josiah Brownell Johannes Sackor SCM Pharmacist (Margibi CHT) Depot Assistant (Margibi CHT) 0 2 Group Informant Interview April 23, 2017 Karyou Johnson Finance Accountant (Margibi CHT) 0 1 Key Informant Interview April 23, 2017 Gregory Walker Leadership/Governance Health Services Administrator (Grand Bassa CHT) 0 1 Key Informant Interview April 23, 2017 Moses Logan Nathan Tartee SCM Logistics Supervisor (Margibi CHT) Depot Assistant (Margibi CHT) 0 2 Group Informant Interview April 24, 2017 Adolphus Yeiah Leadership/Governance County Health Officer (Margibi CHT) 0 1 Key Informant Interview April 24, 2017 Roland Reeves Kusie David Finance Financial Officer (Margibi CHT) Financial Assistant (Margibi CHT) 1 1 Key Informant Interview April 24, 2017 Larry Boahnda QA/QI Clinical Supervisor (Margibi CHT) 0 1 Key Informant Interview April 24, 2017 Benjamin Saygar HIS M&E Officer (Margibi CHT) 0 1 Key Informant Interview April 25, 2017 Monah Young HRH Human Resource Officer (Margibi CHT) 1 0 Key Informant Interview April 25, 2017 Erikson Napeh Mamie Kanty HRH Human Resource Officer (Grand Bassa CHT) Human Resource Assistant (Grand Bassa CHT) 1 1 Group Informant Interview April 26, 2017 John Gedeh QA/QI Clinical Supervisor (Grand Bassa CHT) 0 1 Key Informant Interview April 26, 2017 57 Name Component Position F M Data Collection Method DATE Gabriel Sesay HIS M&E Officer (Grand Bassa CHT) 0 1 Key Informant Interview April 27, 2017 Total 3 33 CSH MENTORS AT COUNTY LEVEL Name Role F M Data Collection Method DATE Mathias Kokpu Henry Howe John Nenwah James Tokpa HRH Mentor (Nimba CHT) SCM Mentor (Nimba CHT) M&E Mentor (Nimba CHT) QA/QI Mentor (Nimba CHT) 0 4 Group Informant Interview April 13, 2017 Henry Vincent Taywah Bombo M&E Mentor (Bong CHT) Quality Mentor (Bong CHT) 1 1 Group Informant Interview April 17, 2017 Total 1 5 DISTRICT LEVEL Name Component Position F M Data Collection Method Date Rancey Larkpor All District Health Officer (Sannequellie Mah DHT) 0 1 Key Informant Interview April 13, 2017 Marvin Tokpah HIS Data Clerk (Sannequellie Mah DHT) 0 1 Key Informant Interview April 14, 2017 Jimmy Kerkulah Joseph Garteh QA/QI Data Clerk (Suakoko DHT) District Health Officer (Suakoko DHT) 0 2 Group Informant Interview April 17, 2017 Joseph Saah Habakkuk Kollie Harweton Shilling QA/QI District Health Officer (Mamba Kaba DHT) District Health Officer (Firestone DHT) District Health Officer (Gibi DHT) 0 3 Group Informant Interview April 24, 2017 Yassah Sumo QA/QI District Health Officer (Buchanan DHT) 1 0 Key Informant Interview April 26, 2017 Total 1 7 58 ANNEX 4: DATA COLLECTION INSTRUMENTS Overview. In addition to secondary data from the project and other sources, the CSH evaluation has proposed several data collection instruments, including: 1. Coversheet for all protocols. This coversheet comprises of an introduction to the evaluation and a confidentiality protocol. In addition, the ages, numbers, and gender of the participants will provide information on the extent to which females have been included in the interviews. The verbal consent to be interviewed will be noted. 2. Interview schedules. Questionnaires have been developed for the various informants to be contacted by the evaluation team. These include the following categories: a. MOH and CSH CHT: The KII / Standard Questions tool will be administered to key informants identified among MOH Departments and CHTs in CHS-targeted counties (Bong and Nimba). b. Non-CSH CHT: A modified KII / Standard Questions tool will be administered to key informants in the non-CSH counties. c. CSH Collaborators and Partners: Separate questionnaires will be administered to these two groups. 3. Capacity Assessment schedule: For the CSH and non-CSH counties, questions from the capacity assessment tool (previously used by RBHS and CSH) have been selected (after discussion with CSH staff) and will be administered to the Key CHT Informants based on their responsibilities. For instance, the CHT officer responsible for M&E will be asked the relevant questions. 4. Quality Improvement schedule: Questions assessing quality improvement have been developed and comprise two complementary elements: a questionnaire and a summary data sheet. They will be administered to staff of the county hospital and in two clinics. a. CSH Facilities: The complete questionnaire and summary data sheet will be administered to quality improvement team members at the hospital and at two clinics in each county. b. Non-CSH CHT: A modified questionnaire and summary data sheet will be administered at the hospital and at two clinics in each county. 5. Client Focus Group schedule: Though CSH does not directly address service delivery, improvements in quality will be rapidly assessed by: (i) collecting some rapid data from clients using the services; and (ii) using on-the-spot summaries of this data to inform brief Focus Group Discussions with clients. Exit interviews with at least 10 clients in the hospitals and at least 5 in the clinics will probe the results of the QI initiative underway. 59 Coversheet for All Protocols Date of Interview: Interviewee Name, Title: Project Component: Time Start: Time End: Interviewer(s): Location Category of Interviewee: Age Group: No. of Interviewees ___________ M _____ F______ Introduction: Good morning/afternoon and thank you for taking the time to speak with us today. As mentioned during our interview request, we are working with USAID/Liberia as external evaluators to conduct an independent mid-term performance evaluation of the Collaborative Support for Health Project (CSH). The evaluation is expected to: (i) analyze the overall the project’s progress; (ii) identify lessons learned related to the project’s achievements and constraints; and (iii) make recommendations on the way forward. Our team has had the opportunity to review some background documents to get a better sense of the design and implementation of the project. However, these documents can only tell us so much. We would like to speak with you today to hear about your experience and in your own words; this will help us better understand how the project is viewed and felt from the perspective of its beneficiaries. If you agree to participate, this discussion will last approximately 1hour. Confidentiality Protocol • We will collect information on individuals’ names, organizations, and positions. A list of key informants will be made available as an annex to the final evaluation report, but those names and positions will not be associated to any specific finding or statement in the report. However, please keep in mind that while responses will be kept confidential, if you contribute something that only you or your office could comment on, that may then be recognizable. • We may include quotes from respondents in the evaluation report, but will not link individual names, organizations, or personally identifiable information to those quotes, unless express written consent is granted by the respondent. Should the team desire to use a particular quote, photograph, or identifiable information in the report, the evaluators will contact the respondent(s) for permission to do so. • All data gathered will be used for the sole purposes of this evaluation, and will not be shared with other audiences or used for any other purpose. However, if at any time, you do not feel comfortable answering a particular question, please let us know and we will simply go on to the next question. To ensure all useful information are captured for analysis; with your permission, I would be recording this interview. • Your participation in this interview is voluntary and you will not be receiving any compensation for your time. • Should you have additional questions about this interview, please contact the Ms. Carla Trippe of Social Impact on +231888 106151 or ctrippe@socialimpact.com. 60 Once again, thank you for taking the time to speak with us today. Do you have any questions for us before we get started? Verbal consent given? Yes/No__________________ 61 Key Informant Interview / MOH and CSH County Health Team Informants This questionnaire will be administered to key informants identified among MOH Departments and CHTs in CHS-targeted counties (Bong and Nimba) Knowledge of the project, its objectives, and its activities 1. Have you heard of the Collaborative Support for Health Project (CSH)? Have the project’s objectives been explained to you? By whom? When? How frequently do you work with CSH? 2. Did you also work with the RBHS project? If so, can you note any differences with the CSH approach? 3. Can you tell us how CSH has directly supported the MoH’s post-Ebola recovery priorities as outlined in the Resilient Health System Plan? Please provide specific examples. 4. How involved have you been with the project’s activities? (Probe: little, somewhat, very)? Did you participate in identifying priorities identified, scheduling activities, etc. Have there been any challenges in this prioritization and implementation process? 5. Can you please list the most important activities you have been involved with? (probe: try to identify three)? Appreciation of the project’s assistance 6. Why do you consider (Activity 1) to have been important? What was your involvement in the activity? (probe: design and/or delivery, participation only, etc.) How would you assess the content (probe: relevance/usefulness) and timeliness (probe: scheduling/integration) of the activity? 7. Why do you consider (Activity 2) to have been important? What was your involvement in the activity? (probe: design and/or delivery, participation only, etc.) How would you assess the content (probe: relevance/usefulness) and timeliness (probe: scheduling/integration) of the activity? 8. Why do you consider (Activity 3) to have been important? What was your involvement in the activity? (probe: design and/or delivery, participation only, etc.) How would you assess the content (probe: relevance/usefulness) and timeliness (probe: scheduling/integration) of the activity? Utilization of the project’s assistance 9. Did you receive any specific materials related to the activity? (probe: ask to see them and note titles) Have you been able to use your new knowledge and materials to improve your performance? How? 10. What are some of the noticeable impacts in operations/efficiency/effectiveness for your unit/team/staff? If so, please detail them. Assessment of the project’s influence 11. What have been the stronger and weaker aspects of the assistance you have received? Given your experience with the project, do you have suggestions for improving its performance? 12. On a scale from 1-10, one being not satisfied at all and 10 being extremely satisfied, overall, how satisfied with the CSH support to the Ministry (individual units, technical working groups, other coordination efforts, etc.? Please justify your response. 62 Future Priorities 13. Do you have any recommendations for future USAID HSS support to the Ministry of Health? Are there any new/emerging HSS priority areas that you believe the Ministry of Health would like/need support within in the next two years? Capacity Assessment Questions (for CHT only) 14. Capacity assessment question 1 / Delivery essential health services 15. Capacity assessment question 2 / Health workforce 16. Capacity assessment question 3 / Health information systems 17. Capacity assessment question 4 / Access to essential medicines 18. Capacity assessment question 5 / Health systems financing 19. Capacity assessment question 6 / Leadership and governance Key Informant Interview / Non-CSH County Health Team Informants This questionnaire will be administered to key informants identified in non-CHS-targeted counties (Margibi and Grand Bassa) Knowledge of the project, its objectives, and its activities 1. Have you heard of the Collaborative Support for Health Project (CSH)? Have the project’s objectives been explained to you? By whom? When? 2. In your opinion, what are the most important priority areas for strengthening your CHT’s ability to plan, manage, and deliver quality health services? Appreciation of the priority areas 3. Why do you consider (Priority Area 1) to be important? Do you already have partners helping you in this area? In your opinion, what kinds of support have been (if partner is helping) or would be (if no partner is helping) most useful to strengthen your abilities in this area? (probe: materials, training, TA, combination) 4. Why do you consider (Priority Area 2) to be important? Do you already have partners helping you in this area? In your opinion, what kinds of support have been (if partner is helping) or would be (if no partner is helping) most useful to strengthen your abilities in this area? (probe: materials, training, TA, combination) 5. Why do you consider (Priority Area 3) to be important? Do you already have partners helping you in this area? In your opinion, what kinds of support have been (if partner is helping) or would be (if no partner is helping) most useful to strengthen your abilities in this area? (probe: materials, training, TA, combination) Capacity Assessment Questions 6. Capacity assessment question 1 / Delivery essential health services 7. Capacity assessment question 2 / Health workforce 8. Capacity assessment question 3 / Health information systems 9. Capacity assessment question 4 / Access to essential medicines 10. Capacity assessment question 5 / Health systems financing 11. Capacity assessment question 6 / Leadership and governance 63 64 Key Informant and/or Focus Group Standard Questions / CSH Collaborators (e.g., LPIA, CHAI, JHPIEGO, Chemonics) Assessment of the project’s objectives and activities 1. After two years of implementation, do you think the project’s current objectives remain relevant? What changes in the environment may have or might affect their relevance? 2. Is the specific technical expertise that CSH/you are providing unique or are there other donors/partners supporting MOH (MPW)? If so, please describe how; if not, please explain how you coordinate with the others. 3. What are the three most important activities you have implemented since project start-up? Appreciation of the project’s assistance 4. Why do you consider (Activity 1) to have been important? What factors were involved in the selection, content, and timing of the activity? What was the involvement (probe: engagement) of your MOH/CHT partner in the design and implementation of the activity? Was the activity evaluated (probe: if so, ask for a copy of the evaluation)? 5. Why do you consider (Activity 2) to have been important? What factors were involved in the selection, content, and timing of the activity? What was the involvement (probe: engagement) of your MOH/CHT partner in the design and implementation of the activity? Was the activity evaluated (probe: if so, ask for a copy of the evaluation)? 6. Why do you consider (Activity 3) to have been important? What factors were involved in the selection, content, and timing of the activity? What was the involvement (probe: engagement) of your MOH/CHT partner in the design and implementation of the activity? Was the activity evaluated (probe: if so, ask for a copy of the evaluation)? Utilization of the project’s assistance 7. In your opinion, what has been the biggest benefit (i.e., technical area of most added value) of CSH to MOH (MPW) in fulfilling its HSS/Post-Ebola mandate for sector recovery and service delivery? 8. Have you used any specific tool or instrument to demonstrate the impact of your contribution (probe: ask to see the instrument)? 9. Were there any areas/approaches that were not well received/not adopted by MOH? If so, please describe why. Have there been any unintended benefits/consequences in providing HSS support to the Ministry under your project? If so, please describe. Assessment of the project’s influence 10. In your opinion, has the project met its objectives? Please give specific examples of why or why not? What have been the critical factors that allowed for successful accomplishment of objectives? What were some of the challenges (probe: CSH project management, Ministry￾donor relationships, etc.)? 11. Given your experience with the project, do you have suggestions for modifying the project’s priorities and/or improving its performance? Future Priorities 12. Do you have any recommendations for future USAID HSS support to MOH (probe: new/emerging HSS priority areas, requests for future TA, etc.)? 65 Key Informant and/or Focus Group Standard Questions / Other Partners (e.g., World Bank, WHO, UNICEF) Assessment of the project’s objectives and activities 1. Are you aware of the Collaborative Support for Health Project (CSH) and its objectives? Have CSH staff ever come to see you in your office? If so, how often? 2. In your opinion, what are CSH’s most useful areas of technical support to MOH? Do you think CSH provides a unique contribution to MOH? If so, can you explain? 3. Are there areas of overlap between your agency’s contributions and those of CSH? How do you coordinate the various interventions? Appreciation of the project’s assistance 4. In your opinion, what have been the three most important activities implemented by CSH since the start of the project? 5. Why do you consider (Activity 1) to have been important? Did you participate in the activity? In what way (probe: planning, implementation)? In your opinion, in what way(s) did the activity strengthen health system capacity? 6. Why do you consider (Activity 2) to have been important? Did you participate in the activity? In what way (probe: planning, implementation)? In your opinion, in what way(s) did the activity strengthen health system capacity? 7. Why do you consider (Activity 3) to have been important? Did you participate in the activity? In what way (probe: planning, implementation)? In your opinion, in what way(s) did the activity strengthen health system capacity? Assessment of the project’s current influence and future priorities 8. How would you assess the project’s overall usefulness in strengthening management capacity and improving service delivery in the health sector? 9. Given your knowledge of the project, do you have suggestions for modifying its priorities and/or improving its performance? 10. Do you have any recommendations for future USAID HSS support to the Ministry of Health (e.g., new/emerging HSS priority areas, requests for new/additional support)? CSH Performance Evaluation Capacity Assessment Tool– CHT level Building Block 1: Delivering Essential Health Services (CSH Objective 3) Indicator 1.2: Capacity of CHTs to Ensure Appropriate use of Policies and Standards Related to Health Service Delivery for the Essential Package of Health Services (EPHS) Areas and Subareas (Subareas refers to the components of the EPHS outlined on pages 50-52 of the 66 National Health and Social Work Plan, such as Antenatal care as a subarea within Maternal and newborn health services) Standard 1.2.1: Capacity of CHTs to supervise health facilities (HF) in the use of health service delivery standards 0 No existing systems in place to distribute or monitor adherence to policies, guidelines and standards 1 The CHT distributes policies, plans and standards to HF, but does not have a system in place to monitor adherence to these standards. If health partners are monitoring adherence to standards, it is using their own separate systems 2 The CHT distributes policies, plans and standards to HF. Some elements of a basic system are used for monitoring of adherence to standards. Specifically, the CHTs use guidelines for one or two of the following areas: lines of responsibility, supervision schedule, supervision guidelines/checklists for facilities and/or health workers 3 The CHT distributes policies, plans and standards to HF. Some elements of a basic system are used for monitoring of adherence to standards. Specifically, the CHTs use guidelines for more than two of the following areas: lines of responsibility, supervision schedule, supervision guidelines/checklists for facilities and/or health workers. No feedback is provided to central MOH regarding supervision of HF. 4 The CHT routinely distributes and monitors HF regarding adherence to standards as per the previous (#3) and provides feedback to the central MOH Comments: Standard 1.2.2: Number of operational public health facilities as compared to those projected in the National Health and Social Welfare Plan 0 CHT does not have a list of the number of public health facilities within the County 1 CHT has a list of the number of public health facilities, but there is no current documentation if these facilities are operational (An operational facility is one that provides EPHS at least 3 days a week, with at least two clinical staff members) 2 CHT has a current list of the number of operational public health facilities, but the number of operational facilities is basically unchanged (within 5%) from the levels reported in the National Health and Social Welfare Plan (page 61) 3 The number of GOL operational health facilities is at least 75%7 of the projected public network 20218 within the county 4 The total number of GOL operational health facilities is at least 80% of the projected public network 2021 within the county 7 The 2011 numbers (i.e., the baseline) are 73% of the 2021 numbers 8 Please see page 59 of the National Health and Social Welfare Plan, 2011-2021 67 Comments: A comprehensive medical products and bi-products disposal policy should include all of the following: waste minimization, segregation, handling, storage, transport, treatment and disposal. The policy should also include a pictorial representation of waste segregation and standard operating procedures for the different steps to handle various categories of waste Standard 1.2.3: Capacity of CHT to safely handle and dispose of medical products and bi-products 0 CHT has no policy or practices to support safe handling and disposal of medical products and bi￾products. 1 The CHT and health facilities have policies in place to support safe handling and disposal of medical products and bi-products, but the policies are not comprehensive (see definition above) and implementation of these policies is inconsistent. 2 The CHT and health facilities have comprehensive policies in place to support safe handling and disposal of medical products and bi-products, but the implementation of these policies is inconsistent (either across facilities, or across areas within the policy). 3 The CHT and health facilities have comprehensive policies in place to support safe handling and disposal of medical products and bi-products, and are routinely implementing the three minimum steps (waste segregation, waste storage, and waste treatment/ disposal). 4 The CHT and health facilities have comprehensive policies in place to support safe handling and disposal of medical products and bi-products, and are routinely implementing all seven steps (Waste minimization, segregation, handling, storage, transport, treatment, and disposal) Comments: Building Block 2: Health Workforce (CSH Objective 4) Indicator 2.2: Capacity of the CHT to strengthen existing health workforce Standard 2.2.1: Capacity of CHT to monitor staff performance 0 There are no policies or guidelines in place to review staff performance. 1 With minimal assistance from the CHT, health partners monitor staff performance based on health partner processes. Timing of supportive supervision/performance monitoring visits is less than yearly. 2 In partnership with the CHT, health partners assess staff performance using national guidelines and tools (e.g. standardized staff performance checklists). Timing of supportive supervision/performance monitoring visits is less than yearly. 3 The CHT takes the lead on assessing staff performance using MOH guidelines and tools (e.g. standardized staff performance checklists). Timing of supportive supervision/performance monitoring visits is annual. 4 The CHT coordinates and support staff performance process utilizing MOH developed guidelines. Staff performance is monitored at least annually. Staff performance is tied to annual reviews and promotions. 68 Comments: Standard 2.2.2: Capacity of CHTs to coordinate capacity development of health staff 0 No in-service training is provided for health staff, or training is completely ad hoc. 1 Health partners provide training on EPHS based on health partner curricula and processes. 2 In partnership with the CHT, health partners provide training on EPHS using curricula and processes developed by health partners. Training information is provided to MOH and CHT. 3 The CHT and health partners provide training on EPHS using MOH-approved curricula and processes. Training schedules are not fully coordinated/ communicated to all relevant stakeholders. 4 CHT coordinates in-service training, utilizing MOH approved curricula. Training follows a coordinated schedule that enables the appropriate staff to attend appropriate trainings, in a manner that prevents excessive loss of clinical service delivery time. Comments: Building Block 3: Health Information Systems (CSH Objective 7) Indicator 3.2: Capacity of CHT to promote evidence-based decisions and policy making Standard 3.2.1: Capacity of CHT to use collected data for planning and policy making 0 The CHT does not use data for strategic planning and decision making. 1 The CHT analyses available HIS data quarterly and distributes reports containing these analyses to key members of the CHT. Use of these data is unknown. 2 The CHT analyses available HIS data quarterly and distributes reports containing these analyses to key members of the CHT. In addition, presentation and discussion of data are part of CHT meetings. 3 The CHT analyses available HIS data quarterly and distributes reports containing these analyses to key members of the CHTs. The CHT can identify at least two examples of how data have been integrated into a decision-making process in the past year. 4 The CHT analyses available HIS data quarterly and distributes reports containing these analyses to key members of the CHTs. The CHT can identify at least four examples of how data have been integrated into a decision-making process in the past year. Comments: 69 Building Block 4: Access to Essential Medicines (CSH Objective 5) Indicator 4.1: Capacity of CHT to ensure access to essential medicines for the county Standard 4.1.1: Capacity of the CHT to estimate commodity needs 0 No capacity (partners or CHT) is available within the county to estimate commodity needs (supervise facilities in the reordering and report this information to the central MOH). 1 CHT is reliant on technical assistance from partners to estimate commodity needs. 2 CHT, with some technical assistance from partners, is able to estimate commodity needs 3 CHT is able to estimate commodity needs, however the timing of these estimates is haphazard. 4 CHT is able to estimate commodity needs according to schedule. Comments: Standard 4.1.2: The CHT’s capacity to use a Logistics Management Information System (LMIS) 0 No LMIS exists. 1 An LMIS exists, but the CHT and facilities do not have all of the LMIS forms, and use of these forms is inconsistent. Less than 60% of staff involved in the LMIS have been trained on the system. 2 The CHT and facilities have the LMIS forms, and at least 60% of staff involved in the LMIS have been trained in use of the forms/system. Reporting of LMIS data is below 50% for all facilities over the past reporting period. 3 The CHT and facilities have at all of the LMIS forms, and at least 75% of staff involved in the LMIS have been trained in use of the forms/system. Reporting of LMIS data is below 75% for all facilities over the past reporting period. 4 The CHT and facilities have at all of the LMIS forms, and at least 75% of staff involved in the LMIS have been trained in use of the forms/system. However, reporting of LMIS data is above 75% for all facilities over the past reporting period. Comments: Standard 4.1.3: CHT’s capacity to effectively store and distribute commodities 0 Storage and distribution of commodities, including essential medicines, is haphazard with no special storage requirements of pharmaceuticals and other items. 70 1 County warehouse(s) for commodity storage exists, with some accommodation for items requiring special storage. Maintenance, cleanliness and size of the county warehouse may be insufficient. 2 The county warehouse meets at least two of the following four criteria: warehouse size is adequate, storage space is well-maintained and clean (including pest control), has designated storage equipment for special storage needs, and distribution to health facilities follows a consistent schedule. 3 The county warehouse meets at least three of the following four criteria: warehouse size is adequate, storage space is well maintained and clean (including pest control), has designated storage equipment for special storage needs, and distribution to health facilities follows a consistent schedule. 4 County Warehouse(s) is adequate (i.e. large enough, regularly cleaned, dry, well organized) with all special needs storage areas clearly designated with correct signage, and established delivery intervals from county warehouse to health facilities exist and are routinely followed, including the use of protocols (such as first expired, first out). Comments: Block 5: Health Systems Financing (CSH Objective 6) Indicator 5.1 Capacity of CHT to formulate, distribute, and monitor financing for the health sector The four criteria necessary in a sustainable budget are as follows: Planning: CHT has a realistic and sustainable budget given past experience/expenses, donors and projections Input: All key stakeholders are involved (including MOH and CHTs, and as necessary, donors/partners) Allocation: CHT compiles an adequate budget to support facilities and services, with specific line items for key areas outlined in the National Health and Social Welfare Policy and Plan Initiative: Process for collection of budget information is led collectively by the MOH and CHTs and the system is standardized across all CHTs Standard 5.1.1: Capacity of the CHT to plan for, create and allocate a sustainable budget 0 No sustainable budget exists (Planning, input, allocation, and initiative) 1 Three of the budget sustainability criteria need improvement (Planning, input, allocation, and initiative) 2 Two of the budget sustainability criteria need improvement (Planning, input, allocation, and initiative) 71 3 One of the budget sustainability criteria needs improvement (Planning, input, allocation, and initiative) 4 All of the budget sustainability criteria are completed and sustainable, with the MOH/CHTs taking the lead on developing the budget (Planning, input, allocation, and initiative) Comments: The four factors necessary to effectively distribute finances are as follows: Financial System: A system exists within the CHT to distribute funds among its activities. This includes differentiating by funding recipient (e.g., by line item, and by facility or district). Tracking: CHT has a system to track its distributed funds against its total budget, the facility distributions against total budgets, manage cash flow and segregate expenses Policies: Guidelines for allowable expenses exist and are distributed among CHT. These policies are implemented on a regular basis. Responsibility: Monthly review of internal expenses versus revenue (both for the CHT budget and each facility’s budget) is designated to an employee(s) as a responsibility Standard 5.1.2: Capacity of CHT to effectively distribute finances 0 No system to distribute funds exists (Financial system, tracking, policies, and responsibility) 1 Three of the budget distribution factors need improvement (Financial system, tracking, policies, and responsibility) 2 Two of the budget distribution factors need improvement (Financial system, tracking, policies, and responsibility) 3 One of the budget distribution factors needs improvement (Financial system, tracking, policies, and responsibility) 4 All of the budget distribution factors are completed and sustainable (Financial system, tracking, policies, and responsibility) Comments: The four factors necessary to effectively monitor finances are as follows: Documentation: CHT keeps financial documentation in a secure place, has a policy for keeping receipts and requirements for documentation kept with each type of payment. These policies are flowed down to county health facilities and adherence is monitored. Review: CHT reviews expenses monthly to ensure applicability and allowability according to the budget and internal policies. Exceptions are documented. 72 Reporting: A reporting system exists both for the CHT to report to central MOH and for the county health facilities to report to the CHT. Reports are completed and submitted according to applicable deadlines. Audit: CHT either has an internal review of its and the county health facilities accounting systems or hires external auditors on an annual basis. Standard 5.1.3: Capacity of CHT to monitor finances at the County and facility levels9 0 No tracking/monitoring system exists. 1 Three of the factors necessary to effectively monitor finances within the CHT need improvement (Documentation, review, reporting, and audit); this includes monitoring of finances at facilities. 2 Two of the factors necessary to effectively monitor finances within the CHT need improvement (Documentation, review, reporting, and audit) this includes monitoring of finances at facilities. 3 One of the factors necessary to effectively monitor finances within the CHT need improvement (Documentation, review, reporting, and audit) this includes monitoring of finances at facilities. 4 All of the factors necessary to effectively monitor finances within the CHT are completed and sustainable (Documentation, review, reporting, and audit),including monitoring of finances at facilities. Comments: Block 6: Governance and Leadership (CSH Objective 1) Indicator 6.1: Capacity of CHT to implement activities aimed at improving the health of all people within the county Standard 6.1.1: Capacity of CHT to implement the National Health and Social Welfare Policy and Plan (NHSWPP) and County Health and Social Welfare Policy and Plan (CHSWPP) 0 CHT is not aware of the NHSWPP and CHSWPP. 1 CHT is aware of both plans; however, no systems of communication exist between CHT and the central MOH to implement the plans. 2 CHT is aware of the both plans and systems of communication exist between CHT and the central MOH to implement the plans. Central MOH takes the lead in implementation of both plans at CHT and HF level. 9 This applies only to facilities that have their own budget 73 3 CHT awareness and communication as per #2. MOH and CHT jointly implement activities listed in both plans, with CHT taking the lead. 4 CHT awareness and communication as per #3. CHT takes the lead in implementing the plans with input from HF and guidance from central MOH. Comments: 74 Focus Group Questions / Facility Health Staff in CSH counties-QI 1. Improvement collaborative • Do you have quality improvement team? (Probe: date of establishment, number of members, community member representation, presence of ToR and frequency of meeting) • Have you received any support for improving the quality of care? (probe: trainings given on quality improvement, presence tools, mentorship and/or onsite training on clinical skills and knowledge including provision of standards and protocols on EPHS) • How do you see the support provided by the QI mentors? (probe: frequency of support visits, time spent during each visit, quality of technical support to facilitate the implementation) • How do you monitor the improvement collaborative activities at your facility? (Probe: action plan, measurements tools, frequency of measurement, frequency and length of each meeting) • Could you please mention some major changes observed after starting implementing the improvement collaborative approach? (probe: ask them to highlight the changes observed in terms of percentage or numbers) • Any challenges faced during implementation? (probe: staffing, lack of adequate support, lack of adequate time to discuss the progress) • Could you please suggest ways to improve the implementation of improvement collaborative? (probe: additional training, learning sessions) 2. JISS (Joint Integrated Supportive Supervision): • Do you receive JISS visits? (If yes, probe, by whom and how frequently? Ask whether they are pre-informed about the visit) Do you also receive any other type of supervision visits in addition to the JISS? (probe: programme specific e.g. HIV and how frequently) • How long does it usually take? (probe: whether it does disrupt provision of services and how they handle it) • Do you receive feedback? (probe: whether they receive feedback during the day of supervision and copy of the tool, ask whether feedback is given to QIT or just to the OIC) • How do you utilize the findings of the supervision to improve the quality of service? (probe: analysing the findings, developing plan for improvement and any post supervision support/mentorship visits) • How do you see the contribution of the supervision visits in improving the quality of the services? (Probe: any major improvements recorded) • Could you please suggest ways to improve the supportive supervision activity? (probe: any need for improvement of tools, the way the supervisors conduct it, frequency of visit and feedback mechanism, and post-supervision mentorship/ support visits) 3. Clinical audit and Data management • Do you conduct clinical audits? (probe: presence of audit guideline, frequency of audit meetings, focus of audit-mortality or does it include morbidity) • Have you observed any significant changes in improved patient diagnosis and management and/or decline in un-wanted outcomes like mortality? (probe: give specific examples) • Have you observed any major change in your data management skills? (probe: timelines of reporting, completeness and analysis and data use for decision making-site specific examples) How important has the HMIS mentor been in this improvement (probe: frequency of support visit, focus of visit-reporting and/or data use for decision making through hands on training) 75 4. Supply chain management: • How do you see the support provided by the supply chain management mentor on drug supply chain management? (probe: frequency of visit, focus of visit-timely reporting of consumption, ordering and arrangement of medicines using FEFO/FIFO) • Have you observed any major changes on your drug supply management system? (probe: frequency of stock out, expiry of medicines, timely reporting using LMIS) 5. Empowering and engaging clients: • How do see the contribution of the community member to the quality improvement team? (probe: regular participation in the meeting, bringing community concerns to the team meeting) • Do you collect information on level of satisfaction from clients? (probe: presence of client satisfaction survey tool, frequency, suggestion box) • Do you have a mechanism to inform your clients about their rights? probe: presence of client charter) • Do you have a client complaint management system? (probe: presence of complaint officer, clear complain management SOP/protocol known by clients) Focus Group Questions / Facility Health Staff in Non CSH counties-QI 1. Quality Improvement • Do you have quality improvement team? (Probe: date of establishment, number of members, community member representation, presence of ToR and frequency of meeting) • Have you received any support for improving the quality of care? (probe: trainings given on quality improvement, tools, mentorship and/or onsite training to improve clinical skills and knowledge including provision of standards and protocols on EPHS) • Are you aware of improvement collaborative activities being implemented in health facilities at Bong, Lofa and Nimba Counties through CSH support? • Are you implementing any quality improvement activity at your facility? ( probe: give examples: to improve low performing indicators, waiting time with baseline data and target) • If you are implementing any quality improvement activity to improve performance of low performing indicators, o Have you received any technical support to facilitate the implementation of the quality improvement activity? ( probe: if they say yes, by whom? And how frequently?) o How do you monitor the quality improvement activities at your facility? (Probe: action plan, measurements tools, frequency of measurement, frequency and length of each meeting) o Could you please mention some major changes observed (probe: ask them to highlight the changes observed in terms of percentage or numbers) • What are the major challenges faced at your facility to improve the quality of service delivery? (probe: skill and knowledge, staffing, shortage of medicines, etc) • Could you please suggest ways to strengthen quality improvement activities at your facility? (probe: trainings, tools, support visits, peer learning) 76 2. JISS (Joint Integrated Supportive Supervision): • Do you receive JISS visits? (If yes, probe, by whom and how frequently? Ask whether they are pre-informed about the visit) Do you also receive any other type of supervision visits in addition to the JISS? (probe: programme specific e.g. HIV and how frequently) • How long does it usually take? (probe: whether it does disrupt provision of services and how they handle it) • Do you receive feedback? (probe: whether they receive feedback during the day of supervision and copy of the tool) • How do you utilize the findings of the supervision to improve the quality of service? (probe: analysing the findings, developing plan for improvement and any post supervision support/mentorship visits) • How do you see the contribution of the supervision visits in improving the quality of the services? (Probe: any major improvements recorded) • Could you please suggest ways to improve the supportive supervision activity? (probe: any need for improvement of tools, the way the supervisors conduct it, frequency of visit and feedback mechanism, and post-supervision mentorship/ support visits) 3. Clinical audit and Data management • Do you conduct clinical audits? (probe: presence of audit guideline, frequency of audit meetings, focus of audit-mortality or does it include morbidity) • Have you observed any significant changes in improvement patient diagnosis and management and/or decline in un-wanted outcomes like mortality? (probe: give specific examples) • Are you getting any form mentorship support to improve your data management skills? (probe: if the answer is yes, ask by how? How frequently? And any major changes observed due to the support?) • What are the major challenges you are facing in your data management skills and use of data for decision making? ( probe: timeliness, completeness of report, analysis, and skills/training, tools, mentorship/support) • Could you please suggest ways to improve your data management system? (probe: trainings, tools, support visits) 4. Supply chain management: • Do you get any support to improve your supply management system? (Probe: if the answer is yes, ask by how? How frequently? And any major changes observed due to the support (reduction of frequency of stock out, timely reporting)?) • What are the major challenges you are facing in your supply management system? (probe: timeliness, completeness of report, stock out of medicines, storage, FEFO/FIFO way of organizing medicines) • Could you please suggest ways to improve your supply management system? (probe: trainings, tools, support visits) 5. Empowering and engaging clients: • Do you collect information on level of satisfaction from clients? (probe: presence of client satisfaction survey tool, frequency, suggestion box) • Do you have a mechanism to inform your clients about their rights? (probe: presence of client charter) • Do you have a client complaint management system? (probe: presence of complaint officer, clear complain management SOP/protocol known by clients) 77 Exit Interview / FGD with Clients (SI staff will conduct exit interviews with at least 10 clients in the hospitals and at least 5 in the clinics to probe the results of the QI initiative underway. The results of these interviews will provide: (i) the basis for a FGD; and (ii) some initial data to compare the CSH and non-CSH facilities on this area.) General Information about the Client: A. Visit Status: NEW RETURNING B. Service Used: ANC PNC U5 children FP OPD Specific Information about the Services: 1. Were you treated with courtesy and respect by the staff? YES NO SOMEWHAT Problem: _______________________ 2. How many hours did you spent in the facility? Less than 2 hours 2-4 hours More than 4 hours How do you see the time you spent in the facility? ACCEPTABLE NOT ACCEPTABLE 3. Did the health worker/s who saw you explain your condition and treatment? YES NO SOMEWHAT Problem: _______________________ If yes, how well do you have you understood your condition and/or treatment? GREAT EXTENT SOMEWHAT VERY LITTLE NOT AT ALL 4. Did the health worker wash/use sanitizer before and after examining you? YES NO 5. How do you rate the service you received against your expectations? BETTER THAN EXPECTED AS EXPECTED WORSE THAN EXPECTED 6. For returning clients (only), have you observed any important changes in this facility in the past one year? (Probe: waiting time, treatment of clients with respect, availability of medicines, cleanliness of the environment) 78 ANNEX 5: CONFLICT OF INTEREST DISCLOSURE 79 80 81 82 ANNEX 6: CSH CONSULTANTS DEPLOYED Name Org C o m p Position /Subject 2015 2016 2017 M a r A p r M a y J u n J u l A u g S e p O c t N o v D e c J a n F e b M a r A p r M a y J u n J u l A u g S e p O c t N o v D e c J a n F e b M a r T ot al TOTAL (monthly TA presence) 3 8 8 1 6 1 9 2 0 2 9 2 2 2 1 2 3 2 7 2 8 3 1 3 4 3 1 3 2 3 3 3 1 3 4 3 2 2 5 2 4 61 5 Long-term consultants 1 4 5 7 1 6 1 8 1 9 2 0 2 0 2 1 2 1 2 0 2 1 2 5 2 7 2 6 2 7 2 6 2 5 2 6 2 3 2 3 48 9 MSH/CSH LT Management Sjoerd Postma MSH COP x x x x x x x x x x x x x x x x x x x x x 21 Floride Niyuhire MSH Acting COP x x x x 4 Floride Niyuhire MSH DCOP x x x x x x x x x x x x x x x x x x x x x 21 Garfee Williams MSH DCOP x x x x x x x x x x x x x x x x x x x x x x x x 24 Percy Ramirez MSH Fin & Admin x x x x x x x x x x x x x x x x x x x x x 21 Bushra Abbasi MSH HSS x x x 3 Edward Nyamba waro MSH HSS x x x x x x x x x x x x x x x x x x x 19 Gregg Pavitt MSH 6 HCF x x x x x x x x x x x x 12 Emmanue l Delamy MSH 6 HCF x x x x x x x x x x x 11 Alexande r Blidi MSH 7 HIS x x x x x x x x x x x x x x x x x x x x x 21 MOH advisors 83 Nee -Alah Varpila MSH 1 L&G x x x x x x x x x x x x x x x x x x x x x 21 Gladys Lavien MSH 1 L&G x x x x x x x x x x x x x x x x x x x x x 21 Henri Disselkoe n DIG 2 MPW x x x x x x x x x x x x x x 14 David Watak o MSH 2 MPW x x x 3 Markone e Knigthley MSH 3 QA/QI x x x x x x x x 8 John Dogba MSH 3 QA/QI x x 2 Arthur Brow n MSH 3 QA/QI x x 2 Felix Kelle y MSH 4 HR x x x x x x x x x x x x x x 14 Arthur Loryoun MSH 5 SCM x x x x x x x x x x x x x x x x x x x x x x x x 24 County mentors Bong Marie Padmore JH Pie go 3 QA/QI x x x x x x x x x x x x x x x x x 17 Taywah Bomb o JH Pie go 3 QA/QI x x x 3 Bernice Gardy MS H 4 HR x x x x x x x x x x 10 Joseph Kaiwood MS H 5 SCM x x x x x x x 7 84 Onesimus Davis MS H 5 SCM x x x x x x x 7 Vincent Henry MS H 7 HIS x x x x x x x x x x x x x x x x x x x x x 21 Lofa Michael Bondo MS H 3 QA/QI x x x x x x x x x x x x x x x x 16 Bill Togba MS H 4 HR x x x x x x x x x x x 11 Andrew Snorton MS H 5 SCM x x x x x x x x x x 10 Huckins Reeves MS H 7 HIS x x x x x x x x x x x x x x x x x x x x x 21 Nimba James Tokpa MS H 3 QA/QI x x x x x x x x x x x x x x x x x x x x x x 22 Matthias Korpu MS H 4 HR x x x x x x x x x x x x x x x x x x x x x x 22 Henry Howe MS H 5 SCM x x x x x x x x x x x x 12 John Nenwah MS H 7 HIS x x x x x x x x x x x x x x x x x x x x x 21 Floating Emmanuel Diabolo MS H 6 HCF x x x x x x x x x x x x x x x x x x x x x x x 23 Short -term consultants 2 4 3 9 3 2 10 2 1 2 6 8 10 9 4 6 6 5 9 6 2 1 126 CSH management - ST Christo pher Welch MSH Managem ent start - up x 1 85 Angela Lee MSH Managem ent/Tech start -up x x 2 Michele Teitelb aum MSH Managem ent start - up x 1 Ken Heise MSH Managem ent/techni cal x 1 Karina Noye s MSH Managem ent/techni cal x x 2 Peter Mater u MSH Fin & Admin (start -up) x 1 John Shin MSH Fin & Admin (start -up) x x x 3 Daniel Nelson MSH Fin & Admin (start -up) x 1 Kevin Fitzchar les MSH Fin & Admin (start -up) x 1 Joseph Kass e MSH Human Resource s start -up x 1 Ummur o Adan o MSH Technical oversight x x 2 86 Umar Arshe d MSH IT network installatio n x 1 CSH technical - ST Susan E. Pritch ett Post MSH 1 LDP Plan x x x 3 Susan E. Pritch ett Post MSH 1 LDP Training x 1 Lour des de la Peza MSH 1 LDP Plan x x 2 Lour des de la Peza MSH 1 LDP Facilitatio n x 1 Lour des de la Peza MSH 1 CHB orientatio n x 1 Sylvia Vrie s endo rp MSH 1 JFK Board Training x 1 Phelel o Marol e JHPIEG O 1 Service provider regulation x 1 87 Phelelo Marole JHPIE GO 1 Prof. board standards x 1 Phelelo Marole JHPIE GO 1 Accredita tion; CPD x 1 Benson Machari a Cons 1 LDP Training x 1 Sebalda Leshaba ri Cons 1 LBNM Mid -term review x 1 Henri Disselk oen DIG 2 WASH TA x x x x x x x x x x x 11 Brian Holst DIG 2 WASH Performa nce Review x 1 Marian ne Carliez Gillet DIG 2 MPW capacity assessme nt x 1 Marian ne Carliez Gillet DIG 2 MPW database mngt x 1 Delilah Khaled DIG 2 WASH gender review x 1 Julia Bluesto ne JHPIE GO 3 MOH Training x 1 88 Unit assistance Julia Bluesto ne JHPIE GO 3 National policy on QA/QI x 1 Julia Bluesto ne JHPIE GO 3 IS Training calendar/ Master x 1 Emman uel Otolori n JHPIE GO 3 Accredita tion standards x 1 Bill Briege r JHPIE GO 3 Malaria program review x 1 Faith Yuh IHI 3 TA for QA/QI activities x 1 Nneka Mobis s on -Etuk IHI 3 QA/QI National Policy x 1 Sodzi Sodzi - Tettey IHI 3 QA/QI National Policy x x x 3 Emman uel Aiyenig ba IHI 3 QA/QI National Policy x x x 3 Ernest Kanyok e IHI 3 TA for QA/QI activities x x x x x 5 Jolanda Steenwi jk IHI 3 TA for QA/QI activities x x x x x x 6 89 Salome y Der e IHI 3 TA for QA/QI activities x x x x x x 6 Catheri ne Ternes IHI 3 Institutio nal QA/QI initiatives x 1 Sophia Tesh o me IHI 3 Institutio nal QA/QI initiatives x 1 Sophia Tesh o me IHI 3 Validation of NHQS x 1 Sodzi Sodzi - Tettey IHI 3 NQS finalizatio n x 1 Jonas Akpakli IHI 3 QA/QI facilitatio n x 1 Jonas Akpakli IHI 3 Intra - county learning 2 x 1 Vidya Ramac h andran IHI 3 Validation of NHQS x 1 Elma Yabang IHI 3 Coach QMT/CS H mentors x x x 3 Salome y Dery IHI 3 Coach QMT/CS x 1 90 H mentors Salome y Akparib o IHI 3 QI Leadershi p training MOH x x x 3 Michele Youngl eson IHI 3 QI Leadershi p training MOH x 1 Jean - Paul Mutali MSH 4 iHRIS data base x x 2 William Kiarie MSH 4 Personnel Handboo k x 1 Gashaw Shifera w MSH 5 SCM Technical Rev. x x 2 Gashaw Shifera w MSH 5 LMIS Redesign x 1 Gashaw Shifera w MSH 5 LMIS Training x 1 Andwel e S. Mwans asu Cons 5 EUV support x 1 Grace Chee R4D 6 UHC financial strategy x x 2 Grace Chee R4D 6 LHEF support x x 2 91 Yoriko Nakam ura R4D 6 UHC financial strategy x x 2 Yoriko Nakam ura R4D 6 LHEF support x x 2 Ben Picillo R4D 6 UHC financial strategy x x 2 Ben Picillo R4D 6 LHEF support x x x 3 Amand a Folsom R4D 6 RDF pilot design x 1 Nathan Blanche t R4D 6 RDF pilot design x x x 3 Gary Gaume r R4D 6 RDF pilot design x 1 Navind ra Persaud MSH 7 M&E Plan/Basel ine Assess. x 1 Richard Gakuba Cons 7 HIS training x x x x x x x 7 Richard Gakuba Cons 7 HIS interoper ability x 1 92 ANNEX 7: CSH PROJECT MATERIALS PRODUCED As of April 2017: FY 2015 (Oct 14-Sep 15) FY 2016 (Oct 15-Sep 16) FY 2017 (Oct 16-Sep 17) Objective 1: Strengthen MOH Leadership & Governance Ministry management Central level Central level Central level Leadership and Governance STTA Report - Post TOR for Leadership & Governance TWG Leadership and Governance STTA Report - de la Peza LDP+ Training of facilitators workshop report Leadership and Governance Development Plan - MOH/Central level - Pritchett& de la Peza County level County level County level County Capacity Assessment Consolidated Report County Health Board Operational Manual County Health Board Orientation: Facilitators Guide Regulatory board management Hospital Hospital Hospital JFK Board Report Pharmacy Pharmacy Pharmacy Liberia Pharmacy Board (LPB) website (http://lpb.gov.lr/) Licensing Licensing Licensing Board Governance Training Curriculum for LBNM Global Reference Performance Standards and Continous Professional Development Course Evaluation Workshop Report Mid-term Review of the LBNM Strategic Plan Assessment Report LBNM Global Reference Performance Standards 93 Report on Health Regulatory Authorities’ Priorities for the World Bank Training Coordination Framework and Process SOPs for CPD with Health Regulatory Bodies Pre-Service training Pre-Service training Pre-Service training Updated National Pre-service Standards for Nursing/Midwifery Education Social Work Harmonized Curriculum, and Workshop Report Updated LMDC Accreditation Monitoring Tools (activity discontinued, WHO and LMDC changed strategy and developed a new tool ) Draft Social Work Curriculum￾Harmonization Harmonized National Social Work Curricula Breakthrough Series Learning collaborative/Session Report In-Service training / Continuing Professional Development In-Service training / Continuing Professional Development In-Service training / Continuing Professional Development Workshop Curriculum for Regulatory Bodies on Licensing and Regulation of Service Providers Draft Continuing Professional Development Candidate Booklet Results of Situational Analysis of Continuing Professional Development of LBNM and LMDC Draft Populated In Service Training Calendar Mapping of Local Institutions Offering In-Service Training and CPD TOR in Service Training TWG TOR for County HR Officer In Service Training Responsibilities TOR for Training Providers 94 TOR for Training Coordinators in Programs Liberia In Service Training Standards Objective 2: Strengthen MPW capacity to manage water supply infrastructure improvements Assessment Assessment Assessment WASH Sector Performance Report - Holst&Disselkoen Rapid Assessment of MPW Findings Report Gender Review of the Ministry of Public Works WASH Assessment STTA Report - Holst&Disselkoen Training Needs Assessment of MPW Staff Members Findings Report Preliminary Gender Analysis of MPW Findings Report Enabling Environment Analysis for Potable Water Findings Report WASH in Schools Survey Input Waterpoint Database Analysis and Revitalization Plan Solutions to Improve the Management of the Waterpoint Atlas Report Capacity-building Capacity-building Capacity-building County WASH Plans (Bong, Lofa, Nimba) Objective 3: Institutionalize QA and QI initiatives to improve health care service QA institutional framework QA institutional framework QA institutional framework QA/QI: National Health Quality Strategy scoping & development visit STTA Reports Mobisson-Etuk, et al. TOR for National Quality Advisory Committee Quality Improvement Leadership Methods Report TOR for Quality Management Teams at facilities Draft National Health Quality Strategy QMU Concept Paper, TORs and Workplan QMU Staff Organogram, TORs Guidelines Guidelines Guidelines 95 Revised Clinical Guidelines and Core Standards Technical Report: Costing of the Essential Package of health services: PHC Level I and II Updated Joint Integrated Supportive Supervision (JISS) Checklist Draft Malaria case management and training documents (4) County Plans for Diagnostic Service Provision Assessment (determined later to be replaced by SARA results) Recommendations for Malaria in Pregnancy Guide and Manuals Technical Guidelines for Malaria in Pregnancy, Training Manual for the Management and Prevention of Malaria in Pregnancy, and Technical Guidelines for Malaria Case Mngt Coaching Outreach Program Report Leading & Facilitating (L&F) session report Objective 4: Strengthen human resources for health management HR management HR management HR management Human Resources and Recruiting STTA Report TOR HRH TWG iHRIS Quality Training Curriculum, and Installation Report Situational Analysis/MOH Training Unit Report and Recommendations 96 Strategic Human Resources and Records Management Training Modules In-Service Training Needs Assessment, and Report iHRIS Component for MOH Two-Year Operational Plan iHRIS Supportive Supervision Tool HR development HR development HR development MOH Scholarship Guidelines Report on Key Findings from the Investigation of Discrepancies in the Administration of the MOH￾Funded Scholarships Program in Liberia Objective 5: Improve supply chain management SCM analysis SCM analysis SCM analysis Pharmaceutical Supply Chain Analysis STTA Report Categorization of Commodities by Facility Level Tracer Commodities listing Technical Review of Liberian Pharmaceutical Services and Supply Chain Systems for Medicines and Health Products Revised List of Tracer Commodities LMIS development LMIS development LMIS development LMIS Technical Redesign Document LMIS training materials: TOT Guide, Facilitator’s Guide, Trainer’s Guide, Health Facility Guide and job aids. LMIS Implementation Plan LMIS rollout plan (per county) HIS/LMIS Subsystem Implementation Plan/Two-Year Operational Plan (included a two year plan and an update) 3rd EUV Report 97 LMIS tools, including Bin Cards, Stock Registers, etc. LMIS Tools Field Testing Report EUV Findings Summary Report, and Presentation Objective 6: Increase financial sustainability of services PFM assessment and strengthening PFM assessment and strengthening PFM assessment and strengthening Updated PFM manual 2015 County PFM Capacity Building Assessment Tool Peer review learning session report County health team and hospital financial management guidelines County PFM Capacity Building Assessment Report Draft TOR for Audit committee PFM Training of trainers materials peer review learning session report (June 2016) Procurement SOPs (pending validation and final approval) Draft Fixed Asset and Warehouse Management SOPs PBF PBF PBF Rapid participatory assessment of the PBF scheme Revised PBF operational manual, with revised PBF Indicators ToT material on PBF for facility training SOW, detailed activities and budget Customized PBF Management Tool Training package for PBF county steering committee Guidelines to Develop PBF County Performance Indicator Targets ((incorporated in operational manual - see pages 41-42) Slide deck for PBF county orientation FARA FARA FARA 98 Contracting Readiness Tools Contracting In Readiness Guideline (FARA) Contracting In Readiness Assessment Tool (FARA) Health Financing Health Financing Health Financing Brief on LHEF Roadmap ToT stakeholders analysis Technical Report: Health Financing in Liberia Consumer Preference Market Research Revolving Drug Funds (RDF) consultation workshop materials Essential Package of Health Services (Primary Care) Costing Study report /Presentation of Preliminary Findings RDF Concept Note RDF financial and pricing model RDF roadmap Development conference / paper and presentation on effective and sustainable health financing in Liberia Health financing reform policy concept note for outline HF Memo to USAID Mission Objective 7: Strengthen health management information systems M&E Plan / Baseline Assessment STTA Report - Persaud HIS Interoperability Concept Note County Dashboard reports Liberia Standards-based Health Information Exchange (HE) Roadmap Data Collection Tool for Collaborative Indicators HMIS Training Manual HIS Operational Plan Costing Detail 14 72 37 123 99 ANNEX 8: CSH TRAININGS PROVIDED Beneficiary Subject matter Location Dates Participant Start End M F Component 1: L&G In-service MOH-Technical Directors LDP TOT Central Feb-16 Feb-16 25 2 Regulatory Agencies Feb-16 Feb-16 15 0 LMDC LDP TOT Central Feb-16 Feb-16 2 0 LBNM LDP TOT Central Feb-16 Feb-16 0 3 LBNM Leadership Training Central Jan-17 Mar-17 2 2 LPB LDP TOT Central Feb-16 Feb-16 9 0 Other LDP TOT Central Feb-16 Feb-16 7 4 Other Leadership Training Central Jan-17 Mar-17 1 6 Local Institutions DEN-L LDP TOT Central/County LIPA LDP TOT Central/County Feb-16 Feb-16 6 4 JFK Leadership Training Central Jan-17 Mar-17 0 6 Workshop Regulatory Agencies LMDC Governance Central Jun-16 Jun-16 3 17 County Health Boards Bong CHB Manual Orientation County Feb-16 Feb-16 15 3 Lofa CHB Manual Orientation County Feb-16 Feb-16 7 1 Nimba CHB Manual Orientation County Feb-16 Feb-16 10 1 Component 3: QA/QI Workshop QA/QI MOH & Partners Learning and Facilitation Central Mar￾16 Mar-16 15 12 CSH-Mentors Learning and Facilitation County May￾16 May-16 6 2 Counties Learning and Facilitation County-Lofa May￾16 May-16 34 23 Counties Intra-county Session 1 County-Bong Apr￾16 Apr-16 33 20 Counties Intra-county Session 1 County-Bong May￾16 May-16 38 27 100 Counties Intra-county Session 1 County-Nimba May￾16 May-16 13 17 Quality Standards Technical Review Central Apr￾16 Apr-16 9 14 Component 4 Pre-service Midwife Midwifery Sep-17 Sep-17 3 144 Lab Tech. Laboratory Sep-17 Sep-17 56 7 In-service training HR staff Strategic HR/Record Management County Jan-16 Jan-16 22 14 HR staff TOT iHRIS Management Central Feb-16 Feb-16 8 5 HR staff iHRIS Management Central Mar￾16 Mar-16 23 7 HR staff iHRIS/MHERO Super Users Central May￾16 May-16 9 5 Component 5 Workshop LMIS Roll-out LMIS National Mar￾16 Mar-16 222 133 Component 6 In-service training PFM TOT PFM National Apr￾16 Apr-16 12 7 Internal audit Internal audit National Apr￾16 Apr-16 9 1 Workshop Stakeholder analysis Stakeholder analysis National Feb-16 Feb-16 5 3 Stakehold consultation Stakehold consultation National May￾16 May-16 33 16 Peer learning Peer learning County Jun-16 Jun-16 26 0 Component 7 In-service training Data officers DHIS data entry/PBF Validation County Jun-16 Jun-16 36 10 M&E staff HMIS TOT County Feb-17 Feb-17 46 13 TOTAL 760 529 59% 41% 101 ANNEX 9: CSH INDICATOR STATUS Indicator FY 15 FY 16 FY 17 FY 18 FY 19 FY 20 Baseline FY 16 Target FY 16 Achievement Status FY 16 Target FY 16 Achievement Target Target Target P1.1 Percentage of facilities reaching two star levels in MOH accreditation survey including clinical standards On hold Bong 0% 15% 25% 35% 35% 45% Lofa 0% 15% 25% 35% 35% 45% Nimba 0% 15% 25% 35% 35% 45% P1.2 Number of skilled birth attendants (physicians, nurses, midwives, and physician assistants) per 10,000 population No data Bong 10 10 11 12 Lofa 9 9 10 11 Nimba 6 6 7 8 P1.3 Public expenditure on health as a percentage of total public expenditure 0.1239 na No data na na na na P1.4 Percentage of health facilities meeting 70% or more clinical standards in assessment No data Bong 43% 48% 53% 58% 63% 68% Lofa 42% 47% 52% 57% 62% 67% Nimba 49% 54% 59% 64% 69% 74% P1.6 Equity index TBE No data P1.7 Percentage of facilities meeting minimum staffing norm in USAID focus counties TBE No data 102 Objective 1: Strengthen leadership and governance capacity of MOHSW at all levels 1.1 No. of MOH staff participating in leadership development program Achieved MOH 0 10 15 10 0 10 10 10 Bong 0 15 0 Lofa 15 0 Nimba 15 0 1.2 % of key TWGs meeting on a quarterly basis 33% 90% 71% Partially achieved 95% 67% 95% 100% 100% HMER 3 QMU 1 SCMU 3 LMG 1 Health Finance 0 HRH 0 1.3 No. of CHB quarterly meetings conducted Partially achieved Bong 2 4 2 4 1 4 4 4 Lofa 1 4 2 4 1 4 4 4 Nimba 1 4 2 4 0 4 4 4 1.4 No. of HCWs with up to date license tracked in iHRIS, regulatory body databases 0 2 000 204 Not achieved 4 000 400 6 000 Male 20 Female 380 1.5 % of health care workers earning CPD credit 0% 10% 0% Not achieved 20% 0% 30% 40% 50% 1.6 Contracting-In Score using composite tool NA TBD 0 Not achieved TBD 0 Bong 50 0 Lofa 50 0 Nimba 50 0 Objective 2: 2. Strengthen MPW capacity to manage water supply infrastructure improvements 103 2.1 % of MPW PFMRAD RMP actions implemented NA TBD 0% No data TBD 0% TBD TBD TBD 3.1.8.3-1 No. of new policies, laws, agreements, regulations or investment agreements (public or private) that promote access to improved water supply and sanitation 0 1 3 Achieved 1 0 1 1 1 3.1.8.4-1 Public sector expenditures on drinking water and sanitation as a % of national budget 0.29% TBD 0.50% No data 0.75% 1.00% 1.20% 1.50% 2.4 % of planned water points completed (constructed or rehabilitated) by MPW NA TBD 27% new, 11% rehab. Achieved TBD TBD TBD Bong TBD Lofa TBD Nimba TBD 2.5 No. of water points managed by MPW NA 300 200 Partially achieved TBD TBD TBD Bong TBD Lofa TBD Nimba TBD Objective 3: Institutionalize QA and QI initiatives to improve health care service delivery 3.1 % of facilities implementing Improvement Collaborative (BTS) Achieved Bong 0% 35% 35% 47% 35% Lofa 0% 25% 25% 34% 25% Nimba 0% 20% 20% 26% 20% 3.2 % of health facilities receiving quarterly JISS supervisory visit from the CHT Achieved 104 Bong 71% 76% 78% 75% 73% Lofa 68% 73% 69% 75% 93% Nimba 53% 58% 73% 75% 70% 3.3 No. of maternal & newborn deaths audited in focus counties NA 90% Maternal No data Bong NA 90% 20% Lofa NA 90% 28% Nimba NA 90% 40% Newborn No data Bong NA 90% 52% Lofa NA 90% 68% Nimba NA 90% 30% 3.4 % of clients satisfied with service provision at the health facility NA TBD 0% No data Bong 30% 0% Lofa 30% 0% Nimba 30% 0% 3.5 % of health facilities in USAID - target counties that can conduct all of the following diagnostic tests (Hemoglobin , Urinalysis, RDT (HIV, malaria, syphilis) Not achieved Bong NA 30% 7% 30% Lofa NA 30% NA 30% Nimba NA 30% 8% 30% 3.6 No. of lab technicians receiving in -service training Not achieved Bong NA TBD 0 30 Lofa NA TBD 0 24 Nimba NA TBD 0 50 3.7 % of facilities in target counties that are in compliance with IPC protocols Achieved 105 Bong NA 30% 80% 85% 83% Lofa NA 30% 82% 89% 93% Nimba NA 30% 84% 87% 80% 3.8 % of facilities in target counties that are in compliance with all clinical malaria protocols Achieved Bong NA 30% 80% 85% 87% Lofa NA 30% 76% 87% 80% Nimba NA 30% 79% 89% 90% Objective 4: Strengthened HRH management 3.1.1 -84 No. of health workers successfully completing in -service training using USG funds NA 60 430 Achieved TBD 72 60 60 60 Male 356 TBD 61 Female 74 TBD 11 4.2 No. of scholarships awarded 0 210 210 Achieved 210 144 TBD TBD TBD 4.1 No. of health care workers graduating from pre -service training institutions with USG support 0 210 10 Partially achieved 210 24 TBD TBD TBD Male 9 TBD 1 Female 1 TBD 23 4.3 No. of pre -service programs accredited by regulatory bodies NA TBD 16 Achieved 19 19 TBD TBD TBD 3.1.3.1 -1 No. of health workers trained in case management using ACTs with USG funds 50 0 Not Achieved 50 50 50 50 Bong NA TBD 0 50 Lofa NA TBD 0 70 Nimba NA TBD 0 70 3.1.3.4 -1 No. of health workers trained in (IPTp) with USG funds 50 0 Not Achieved 50 50 50 50 Bong NA TBD 0 TBD 106 Lofa NA TBD 0 TBD Nimba NA TBD 0 TBD 4.4 No. of staff trained in the use of iHRIS 10 40 68 Achieved 15 15 15 15 MOH 10 7 Bong 1 0 Lofa 1 0 Nimba 1 0 4.5 No. of counties providing updates to iHRIS data 3 0 Not Achieved 15 15 15 15 Bong 0 1 0 1 0 Lofa 0 1 0 1 0 Nimba 0 1 0 1 0 Objective 5: Improved supply chain management 5.1 % of facilities that report no stock-out of tracer drugs during the quarter 0.7 75% Not achieved 0.8 0.9 0.95 1 Bong 73% 76% 25% 76% 87% Lofa 67% 71% 46% 71% 80% Nimba 70% 73% 62% 73% 90% 5.2 No. of County depots adhering to minimum standards of storage management 3 3 Not Achieved 3 Bong 0 1 1 1 1 1 1 1 Lofa 0 1 1 1 1 1 1 1 Nimba 0 1 1 1 1 1 1 1 5.3 % of health facilities submitting timely LMIS reports 0.7 75% 80% Achieved TBD TBD TBD Bong 85% 80% Lofa 75% 77% Nimba 87% 88% 3.1.7.1-3 % of SDP experiencing no stock out at any time during reporting period of contraceptive 0.7 75% No data No data 80% 90% 95% 1 107 method that the SDP is expected to provide. Bong 82% 87% Lofa 83% 85% Nimba 76% 78% 5.4 % of health facilities experiencing no stockout of essential IPC commodities during the quarter No data Bong NA TBD No data 79% 89% TBD TBD TBD Lofa NA TBD No data 75% 98% TBD TBD TBD Nimba NA TBD No data 72% 87% TBD TBD TBD Objective 6: Increase financial sustainability of services 6.1 % of actions in annual RMPs that are completed in the current fiscal year 0 50% 36% Not achieved 60% 70% 80% 90% Bong 78% Lofa 78% Nimba 78% Quarterly budget execution rate in counties ?? 0.005 2% 1% Partially Achieved 2% 1% 6.2 % improvement in quarterly budget execution rate in counties compared to same period/prior year 0 15% No data No data 20% 25% 30% 35% 6.3 % of timely submission of financial reports by counties within the current fiscal year 0 15% 0% Not Achieved 25% 35% 40% 55% Bong 80% 0% Lofa 80% 0% Nimba 80% 0% 108 6.4 % of facilities that receive PBF bonuses in a timely manner 0 25% 0% Not Achieved 35% 45% 55% 65% Bong 25% 0% Lofa 25% 0% Nimba 25% 0% 6.5 % execution of annual allocation of GOL health budget 12.5% 14% No data No data 14.0% No data Objective 7: Strengthen the Health Information System (HIS) 7.1 % of Health facilities with minimum data accuracy scores in the fiscal year Partially achieved Bong 93% 96% 91% 99% 100% 100% 100% Lofa 93% 96% 91% 99% 100% 100% 100% Nimba 73% 80% 81% 93% 100% 100% 100% 7.2 % of facilities that provide timely, and complete HMIS reports Achieved Bong 81% 86% 99% 100% 93% 96% 100% 100% Lofa 70% 75% 74% 90% 89% 85% 90% 95% Nimba 60% 65% 92% 97% 90% 75% 80% 90% 7.3 % of districts that conduct quarterly data review meetings Not Achieved Bong 0% 0% 0% 100% 100% 100% 100% 100% Lofa 0% 0% 0% 100% 32% 100% 100% 100% Nimba 0% 100% 44% 100% 100% 100% 100% 100% Overall Progress Achieved 13 15 Partially 6 8 Not achieved 14 12 No data 9 7 Total 42 42 109 ANNEX 10: CSH CAPACITY ASSESSMENT SCORES BY COUNTY Margibi G.B. 2012 2014 2015 2016 2017 2012 2014 2015 2016 2017 2012 2014 2015 2016 2017 2017 2017 RBHS RBHS CHS FARA CHS/PE RBHS RBHS CHS FARA CHS/ RBHS RBHS CHS FARA CHS/ CHS/PE CHS/PE Score Score Score Score Score Score Score Score Score Score Score Score Score Score Score Score Score Building Block 1: Delivering Essential Health Services 15 22 19 21.59 10 15 23 19 21 10 11 23 21 22 9 9 Extent of interaction between CHTs and central MOH 2 3 3.19 2 3 3.89 2 4 3.7 Extent of Interaction between CHTs and Health Facilities 3 2 3.38 3 2 3.47 3 4 3.83 Capacity of CHTs to Supervise Health Facilities in the Use of Health Service Delivery Standards 2 2 2.81 3 3 2 2.95 3 1 4 3 3 4 Number of operational public health facilities as compared to those projected in the National Health and Social Welfare Plan 4 4 3.07 4 2 4 2.61 4 2 4 3.56 3 3 Capacity of CHT to safely handle and dispose of medical products and bi￾products 0 2 3.33 3 2 2 2.63 3 0 3 3.1 3 2 Indicator 1.3: Capacity of CHTs to deliver health care in priority areas Capacity of CHT to implement programs developed by the Division of Health Services at the central MOH level 1 2.4 2.5 2 2.4 2.08 1 1.6 1.89 Standard 1.3.2: Capacity of CHT to address community health needs 3 4 3.31 1 4 3.19 2 0 3.2 Building Block 2: Health Workforce 10 14 12 12.75 5 9 10 12 12 6 8 14 11 13 5 5 Indicator 2.1: Capacity of CHT in workforce recruitment and deployment Ability to recruit human resources for health worker positions 3 4 3.45 3 4 3.35 4 4 3.6 Capacity of CHT to staff facilities as per National Staffing Guidelines 4 2 3.23 2 2 3.17 4 4 3.11 Capacity to review staff performance 0 3 3.15 2 1 3 2.44 2 0 0 3.11 2 2 Capacity of CHTs to coordinate capacity development of health staff 3 3 2.92 3 3 3 2.58 4 0 3 3.6 3 3 7 13 12 12.80 3 11 14 12 12 3 7 11 12 10 2 4 Capacity of the CHT roll out HIS policies and forms 0 2 3.17 3 2 2.76 2 4 2.22 Capacity of CHT to collect health data 4 3 3.36 3 3 3.37 3 4 3.11 Capacity of CHT to report health data to MOH and to health facilities 3 3 2.91 3 3 3.31 2 4 3.22 Capacity of CHT to use collected data for planning and policy making 0 4 3.36 3 2 4 2.94 3 0 0 1.89 2 4 Building Block 3: Health Information Systems Indicator 3.1: Capacity of the CHT to plan for and systematically collect health Indicator 3.2: Capacity of CHT to promote evidence-based decisions and policy Bong Nimba Lofa Capacity Building Assessment Quantitative Tool – CHT Indicator 1.1: Capacity of CHTs to engage with central MOH Health Facilities (HF) in delivering the health and social welfare services Indicator 1.2: Capacity of CHTs to ensure appropriate use of policies and standards related to health service delivery for EPHS areas/sub-areas Indicator 2.2: Capacity of the CHT to strengthen existing health workforce 110 Margibi G.B. 2012 2014 2015 2016 2017 2012 2014 2015 2016 2017 2012 2014 2015 2016 2017 2017 2017 RBHS RBHS CHS FARA CHS/PE RBHS RBHS CHS FARA CHS/ RBHS RBHS CHS FARA CHS/ CHS/PE CHS/PE Score Score Score Score Score Score Score Score Score Score Score Score Score Score Score Score Score 5 8 12 9 7 7 12 12 9 7 5 9 8 7 7 7 Capacity of the CHT to estimate commodity needs and request these commodities from the central MOH 3 4 2.73 3 2 4 3.47 3 4 3 3 3 3 CHT’s capacity to use Logistics Management Information System 1 4 3.27 3 2 4 2.93 3 0 4 2.56 3 3 CHT’s capacity to effectively store and distribute commodities 1 4 2.82 1 3 4 2.56 1 1 1 1.33 1 1 0 9 7 9 9 3 6 7 7 9 3 3 0 7 9 9 CHT capacity to plan for, create and allocate a sustainable budget 0 1 2.82 1 2 1 2.35 1 1 0 2.86 1 1 Capacity of CHT to effectively distribute finances 0 3 3.18 4 1 3 2.31 4 1 0 2 4 4 Capacity of CHT to monitor finances at the County and facility 0 3 3.08 4 0 3 2.47 4 1 0 2.29 4 4 6 15 13 14 3 7 15 13 13 4 14 10 10 3 3 Capacity of CHT to implement the NHSWPP 1 3 3.22 3 2 3 3.33 3 0 1 2.14 3 3 Capacity to communicate and share reports, documents and plans with the central MOH and other CHTs 0 3 3.56 2 3 3.42 1 3 2.75 Capacity of CHT to lead/engage with DHSWT and health facilities 3 3 3 1 3 2.95 1 4 2.88 Capacity of CHT to hold responsibility and ownership for the health system within their county 2 4 4 2 4 3 2 2 2.5 43 81 75 79 52 80 75 74 38 74 62 70 Selected scores for comparison (CHS and non-CHS Counties) 15 40 40 37 25 40 36 38 35 37 TOTAL SCORE Indicator 5.1 Capacity of CHT to formulate, distribute, and monitor financing for the health sector Building Block 6: Governance and Leadership Indicator 6.1: Capacity of CHT to implement activities aimed at Indicator 6.2: Capacity of CHT to communicate and share best Building Block 4: Access to Essential Medicines Building Block 5: Health Systems Financing Indicator 4.1: Capacity of CHT to ensure access to essential Bong Nimba Lofa Capacity Building Assessment Quantitative Tool – CHT 111 ANNEX 11: CSH SERVICE DELIVERY INDICATORS CSH Program Indicators County Baseline FY 2015 Target FY 2016 FY 2016 Quarter Achievement FY 2017 FY16 Q1 FY16 Q2 FY16 Q3 FY16 Q4 Total Target FY 2017 FY17 Q1 3.1.1-59 Number of pregnant women with known HIV status (includes women tested for HIV and receiving their results) Bong 8 774 9 213 2 428 3 194 3 279 3 317 12 218 9 673 3 385 Lofa 5 811 6 102 1 747 2 154 3 055 1 511 8 467 6 407 1 216 Nimba 11 058 11 611 7 109 6 703 7 995 2 578 24 385 12 191 3 237 3.1.1-10 Number of adults and children with advanced HIV infection receiving antiretrovira l therapy (ART) Bong 392 442 457 619 702 764 2 542 TBD 715 Lofa 310 350 335 412 587 788 2 122 TBD 451 Nimba 1 012 1 062 1 352 1 300 1 602 1 509 5 763 TBD 1 718 3.1.1-69 Number of eligible children and adult receiving care Bong 407 977 457 619 702 764 2 542 TBD 718 Lofa 630 662 335 412 587 788 2 122 TBD 484 Nimba 319 435 1 352 1 300 1 602 1 509 5 763 TBD 1 854 3.1.6-6 Number of cases of child diarrhea treated in USG Bong 1 629 1 710 819 879 916 1 153 3 767 1 796 904 Lofa 4 770 5 009 1 090 1 922 1 765 1 257 6 034 5 259 1 127 Nimba 6 469 6 792 1 364 2 488 2 469 2 024 8 345 7 132 1 615 112 assisted programs 3.1.6 -61 Number of children who received Penta3 (or measles) by 12 months of age in USG - assisted programs Bong 12 420 13 041 3 902 4 069 4 141 4 592 16 704 13 693 4 955 Lofa 7 373 7 742 2 602 3 025 3 156 3 452 12 235 8 129 2 617 Nimba 12 427 13 048 3 971 3 835 4 845 5 679 18 330 13 701 4 738 3.1.6 -63 Number of children under five years of age with suspected pneumonia receiving antibiotics by trained facility or community health workers in USG - assisted programs Bong 6 219 6 530 2 123 2 101 1 740 1 677 7 641 6 865 1 247 Lofa 17 437 18 309 5 240 5 567 7 363 7 381 25 551 19 224 7 134 Nimba 25 202 26 462 7 353 7 108 7 857 8 539 30 857 27 785 8 163 3.1.7.1 -1 Couple years of protection in USG supported programs Bong 7 362 NA 4 935 4 282 4 371 4 840 18 428 TBD TBD Lofa 7 672 NA 4 478 5 312 4 045 5 155 18 990 TBD TBD Nimba 7 577 NA 3 910 4 240 3 363 4 023 15 536 TBD TBD 3.1.7.1 -3 Percent of USG - assisted service delivery sites providing FP counseling and/or services Bong 97% TBD 95% 98% 98% 98% 97% 98% 98% Lofa 98% TBD 97% 98% 98% 98% 98% 76% 76% Nimba 90% TBD 83% 86% 87% 88% 86% 86% 86% Bong 58% 63% 70% 76% 86% 82% 79% 73% 77% 113 3.1.6 -48 Percentage of women with four ANC visits during their last pregnancy Lofa 53% 58% 66% 69% 76% 68% 70% 68% 56% Nimba 60% 65% 72% 69% 81% 79% 75% 75% 73% P1.6 Percentage of women with one ANC visits during their last pregnancy Bong 69% 72% 77% 98% 78% 82% 84% 75% 92% Lofa 67% 70% 62% 76% 63% 63% 66% 73% 59% Nimba 84% 87% 79% 91% 88% 80% 85% 90% 72% 3.1.3.4 -4 Percentage of pregnant women who received two doses of IPTp Bong 63% 68% 63% 78% 79% 80% 75% 73% 83% Lofa 55% 60% 45% 58% 60% 65% 57% 65% 53% Nimba 71% 76% 51% 60% 68% 73% 63% 81% 64% 3.1.6 -40 Percentage of deliveries that occur in a health facility and assisted by skilled staff Bong 63% 68% 84% 81% 90% 84% 85% 73% 93% Lofa 52% 57% 65% 61% 74% 65% 66% 63% 58% Nimba 59% 64% 77% 69% 85% 76% 77% 69% 79% P1.7 % of facilities meeting minimum staffing norm in USAID focus counties Bong 10% TBD NA NA 0 Lofa 25% TBD NA NA 0 Nimba 15% TBD NA NA 0 P1.8 % attrition among clinical workforce Bong NA TBD NA NA 0 Lofa NA TBD NA NA 0 Nimba NA TBD NA NA 0 114 ANNEX 12: PRIORITY ACTIVITIES BY COMPONENT The following table presents the key CSH achievements by SOW objectives identified through document review and KIIs. SOW objectives CSH MOH / Central level MOH / County level Component 1: L&G Measurably build MOH's leadership & governance capacity at all levels LDP+ Training of facilitators from MOH, counties, regulatory boards, & 2 local capacity￾building institutions (LIPA & DEN-L) to implement LDP+ Training of facilitators (MOH, counties, regulatory boards, & 2 local capacity￾building institutions-LIPA & DEN-L) to implement LDP+ County Health Boards Finalization/Roll out of CHB Operational Manual & Facilitator's Guide Finalization/Roll out of CHB Operational Manual & Facilitator Guide Finalization/Use of Contracting Readiness Tool to assess CHT capacities Strengthen external regulatory frameworks for service delivery & pharmaceuticals LMDC iHRIS training/software iHRIS training/software Development/Mapping of potential CPD possibilities SOPs & workplans for CPD/re-licensing Review/Updating of accreditation process for public/private hospitals & health centers Review/Updating of accreditation process for public/private hospitals & health centers LBNM Accreditation of 18 nursing/midwifery schools Accreditation of 18 nursing/midwifery schools LDP+ training for school adminstrators LDP+ training for school adminstrators Mid-term review of LBNM strategic plan with Board Mid-term review of LBNM strategic plan with Board iHRIS training/software iHRIS training/software Development/Mapping of potential CPD possibilities SOPs & workplans for CPD/re-licensing 115 Governance training for LBNM Board Leadership training for LBNM & JFK hospital staff Review/Finalization of pre￾service standards for Nursing/Midwifery Educ. LPB LPB website LPB website iHRIS training/software for for pharmaceutical establishments, pharmacists, & dispensers iHRIS training/software for for pharmaceutical establishments, pharmacists, & dispensers Improve management capacity through TA to the CHTs CHT mentors recruited for QA/QI, HR, SCM, HCF & HIS CHT mentors recruited for QA/QI, HR, SCM, HCF & HIS Component 2: MPW Strengthen capacity to manage water supply infrastructure improvements Survey/mapping of WASH facilities in 5,500 schools (WinS program) Survey/mapping of WASH facilities in 5,500 schools (WinS program) Assessments of: (i) capacity building needs at national & local levels; (ii) gender-related policies & practices; & (iii) enabling environment Development of 3 policies to enforce mandatory IP reporting to the Waterpoint Atlas Targeted training of MPW staff to manage the Water Point Atlas Strengthening of the water point data base system 116 Organization of workshops to develop County WASH plans Support implementation of risk mitigation efforts Assessment/Capacity development plan for PFMRAF Assessment/Capacity development plan for PFMRAF Component 3: QA/QI Ensure that EPHS & EPSS quality st&ards & service delivery protocols are in place Updating of EPHS Ensure standards are adhered to/continually reinforced in the workplace (CME, supervision, etc.) Revision of Joint Integrated Supervision & Support (JISS) tool Revision of Joint Integrated Supervision & Support (JISS) tool Revision of Joint Integrated Supervision & Support (JISS) tool Institutionalize QA & QI practices Establishment of a National Quality Management Unit QILM training for leadership in MoH QILM training for leadership in MoH Development of a National Health Quality Strategy Development of a National Health Quality Strategy Collaborative improvement activities in 45 health facilities Collaborative improvement activities in 45 health facilities QA/QI training and cross-county & intracounty learning sessions (2) QA/QI training and cross-county & intracounty learning sessions (2) Support for monthly dashboard of QA/QI progress Maternal mortality review Improve diagnostic services for support of clinical diagnosis/treatment & QA Support for baseline assessment/plan for priority EPHS lab services Component 4: HR Strengthen HRH management at all levels 117 Restructuring of MOH HR Unit Strategic HR & records management training for central & county staff Development of HR handbook Updating of personnel records for integration into iHRIS Updating of personnel records for integration into iHRIS Updating of personnel records for integration into iHRIS iHRIS training/software for counties Support pre-service institutions, MOH's training division & service units, scholarships for pre-service training & institutionalization of performance-based management strategies. Revitalization/Development of procedures for MOH scholarship committee Management of scholarships for pre-service training (lab tech & midwifery) Management of scholarships for pre-service training (lab tech & midwifery) Establishment of integrated MOH training unit National assessment of long￾term staff training needs in priority areas National assessment of long￾term staff training needs in priority areas Situational analysis of MOH's management of in-service training Situational analysis of MOH's management of in￾service training Workshop on in-service training calendar & master training plan Support for recruitment of 72 community health supervisors (Nimba County) Component 5: SCM Strengthen SCM by the National Drug Service (NDS), MOH's Supply Chain Management Unit (SCMU), & the CHTs 118 Review/Dissemination of pharmaceutical standard treatment guidelines Review of the supply chain master plan Improved warehouse operations in the county drugs depots Improved warehouse operations in the county drugs depots Ensure continuous availability of essential drugs at service delivery points Redesign/Roll out of LMIS (with MOH & partners) Redesign/Roll out of LMIS (with MOH & partners) Redesign/Roll out of LMIS (with MOH & partners) Implementation of 3 rounds of EUV for President’s Malaria Initiative Component 6: HCF Strengthen financial oversight, planning, & management functions, at both central & county levels Finalization/Roll out of PFM Manual (with OFM) for MOH & 15 counties Finalization/Roll out of PFM Manual (with OFM) for MOH & 15 counties Finalization/Roll out (with OFM) of PFM Manual for MOH/15 counties Development of SOPs for Procurement, Fixed Assets Management, & Warehouse Management Establishment of procurement manual and committee Establishment of procurement manual and committee Implementation of QuickBooks™ for FM at county level Implementation of QuickBooks™ for FM at county level PFM Peer Review Learning Session between counties PFM Peer Review Learning Session between counties Regular assessment of PFM capacity in Bong, Lofa, & Nimba 119 Review/mitigation of risks identified in the MOH Risk￾Mitigation Plan Support management approaches focusing on performance-based financing Support for building technical capacity in the newly-staffed MOH PBF Unit Revision/Roll out of PBF Manual through TOT in Bong, Lofa, & Nimba Revision/Roll out of PBF Manual through TOT in Bong, Lofa, & Nimba Support for planning & managing the transition from free health care Analysis/Presentation of options for HCF reform to MOH SMT Studies to inform HCF: willingness to pay study, costing of EPHS for clinics Studies to inform HCF: willingness to pay study, costing of EPHS for clinics Updating of LHEF road map Workshop to design/organize RDF pilot to improve sustainability of drugs Workshop to design/organize RDF pilot to improve sustainability of drugs Stakeholder analysis training for 8 MOH staff Component 7: HIS Strengthen HMIS at both central & county levels Finalization of the national HMIS Strategic Plan and partner mapping Finalization of the national HMIS Strategic Plan and partner mapping Development of 5-year, standards-based, HIS Roadmap to establish interoperable health information systems Development of 5-year, standards-based, HIS Roadmap to establish 120 interoperable health information systems Decentralization of M&E to county & district levels Decentralization of M&E to county & district levels Decentralization of M&E to county & district levels Improve the accuracy & use of health information Support for national HIS tools revision, validation, & roll out (TOT) Training of data clerks & supervisors on validation/use of HMIS data Training of data clerks & supervisors on validation/use of HMIS data Quarterly reviews & validations of health data at county & district levels Quarterly reviews & validations of health data at county & district levels Technical review of HMIS training manual completed 121 ANNEX 13: NATIONAL RESILIENT PLAN PRIORITIES The following table assesses current progress and potential assistance in conformity with the National Resilient Plan. Component Current progress Continuing support Potential assistance Component 1: L&G (Leadership and governance capacity) § Build the leadership, governance and management capacity of county and district health teams to ensure they are able to coordinate, and manage provision of health services. x x § Establish and ensure functionality of sector coordination mechanisms at community, health facility, district and county levels, in line with the experiences from the HSCC. x x § Make operational the governance and monitoring systems and structures at community, health facility, district, county and national levels that ensure citizen participation and involvement in health. x Component 3: QA/QI (Enhancement of quality service delivery systems) § Strengthen QA system to support and monitor adherence to clinical protocols for priority health conditions (e.g. malaria, pneumonia, labour and delivery management, care of the sick newborn, etc.). x x § Ensure functional QA and quality improvement processes practiced in facilities to enable continuous improvement in the health outcomes achieved; this approach will include clinicians and non-clinicians, including their managers and administrators in order to improve overall management and performance of the health system. x x § Integrate and monitor IPC standards and triage for all health care facilities are maintained. x x § Roll out initiatives to improve facility cleanliness and environment. x § Establish a system for eliciting and responding to client feedback in order to promote partnership in health service delivery. x Component 4: HR (Fit-for-purpose productive and motivated health workforce) § Strengthening and reforming the MOH Human Resources for Health structure and health worker information system. x x § Establishing scholarships to reduce student drop-outs, enrolments of students from underserved areas. x x § Coordinating appropriate recruitment of health workers for new institutions, infrastructure and reformed central, county and district level health team structures. x 122 § Designing and implementing a package of financial and non-financial incentives to recruit and retain priority health workers, particularly in underserved areas – including housing allowances and housing, mobile money platform for timely and transparent remuneration. x § Strengthening and aligning pre-service, post/basic graduate and continuing professional education to needs. x § Institutionalizing capacity for evidence-driven health workforce planning and establish a national health workforce account. x Component 5: SCM (Management capacity for medical supplies and diagnostics) § Improving information system to get accurate information about stocks and consumption at all levels; we will test new technologies aligned and interrelated with other data collection systems such as HMIS and LMIS. x x § Asses and implement, if the initiative is endorsed, a drug revolving fund to improve financial sustainability of essential medicines and supplies consistent with the Bamako Initiative. x x § Establishing and supporting overall distribution system; consideration will be given to the possibility of outsourcing distribution system throughout the entire supply chain but with the aim of building capacity to be handed over by the government before the end of this plan. x x § Develop HR capacities to enable proper functioning of supply management system, ensure last mile distribution from county depots to facilities through the availability of enough and particularly assigned vehicles for emergency supply distribution. x § Improving capacity of adequate medical waste management by centralizing waste destruction at the LMHRA, and installing appropriate technology for the destruction of all expired, counterfeit and damaged medicines. x Component 6: HCF (Efficient health financing systems) § Strengthen systems and capacities for planning and budgeting as well as financial accountability, to ensure that the resources made available are used for their intended purposes, as and when needed. This would involve needbased recruitment to strengthen and improve the skills of financial management teams at national, county, district and hospital levels. x x § Establish a health equity fund to ensure financial risk protection, cushion against financial risks that limit access to care and address systemic issues within existing provider payment mechanisms; specifically, addressing the sustainability of the provision of the EPHS as a services ‘free￾at-point-of-use’. The fund will be designed based on social insurance principles and processes. It will be a restructuring of the existing pooling mechanism towards one managed by x x 123 government, for financing provision of essential health services. § Institutionalize the planning and budgeting process as an annual and timely exercise to prepare government budget prior to MFDP’s announcement of the budget ceiling, starting with feedback on priorities from the county teams through a bottom-up approach. x § Advocate for improved allocative efficiency to counties through application of a resource allocation formula and monitor the trends in technical and allocative efficiency of health facilities and health districts. x § Institutionalize comprehensive financial monitoring system for all health resources to identify sources of funds, amounts spent, and activities financed; and address bottlenecks to efficient use of resources. Such an expenditure tracking and review process will be complementary to the existing expenditure review, and will enable the sector better identify unfunded priorities that require additional and targeted financing. x Component 7: HIS (Comprehensive information, research and communication management) § Strengthen and harmonize the different data collection systems; this will focus on harmonization of HMIS with LMIS, FMIS, iHRIS and CBIS systems. x x § Put in place capacities for information management including analysis and use of data at the local level. x x § Strengthen the review and use of robust data in improving health system performance at the health facility, county and central levels. x x § Establish adequate systems for sector information dissemination (UReport, Mhero, website, process diagrams). x § Establish a mechanism for tracking health research spending from all sources. x Re-engineered health infrastructure Epidemic preparedness, surveillance and response system Sustainable community engagement § Build health facility staff capacity to coordinate and manage community-based services to ensure they are receiving required oversight and guidance. x § Strengthen the functionality of systems for accountability of community-based services to the population, such as the community health committees. x Total 64 15 15 16 124 ANNEX 14: ANALYSIS OF CSH’S M&E FRAMEWORK Reporting on the achievement of indicators in the project’s Monitoring and Evaluation Plan has been handicapped by several problems, including: (i) technical shortcomings related to the definition of the indicators; (ii) problems with the measurement and presentation of the project’s results; and (iii) lack of formally defined relationships for reporting the project’s achievements. The M&E Plan’s technical shortcomings are linked to CSH’s results framework, theory of change, and selection of indicators. Firstly, the project’s initial results framework, which posited a relationship between “demand-driven human and institutional capacity building” and improved health status was neither clear nor consistent with USAID’s results framework. The most recent version of the M&E Plan has corrected this, but the theory of change has remained the same throughout and offers no clear guidance for the selection of indicators. In addition, the M&E Plan comprises indicators related to deliverables, component results, sector performance, and service indicators, but these are sometimes mixed (e.g., performance indicators are included among deliverables) and do not always indicate those which the project can reasonably be expected to influence. Finally, CSH does not adequately define capacity at the county level or the relationship between interventions at the central level and developments at the county level. However, the evaluation found that the measurement tool used to assess capacity over the period 2012-2017 could, with modification (to be more sensitive to changes on the ground), serve both as an indicator of and a programming tool for CSH interventions. Measurement and presentation of the results of the project’s selected indicators have been problematic throughout project implementation. First, a baseline assessment was expected to be completed early in the project but was never formally conducted; instead, CSH undertook a desk review of available documents (e.g., the RBHS report, DHIS2, the Accreditation 2013 report, etc.) to establish the baseline indicators. Second, as shown previously in Tables 3 and 4, no information is provided on 30% of the proposed indicators. Quarterly and Annual Reports were regularly prepared, and periodic project presentations organized, but these documents do not mention any formal review of the results by MOH. CSH’s M&E Plan does not sufficiently define the formal institutional relationships needed for the different parties to assess achievement. Overall, the plan does not distinguish among the different relationships which exist between CSH, MOH, and USAID, as shown in the following graphic: 125 The M&E Plan does not adequately (and consistently) define the link between the proposed indicators and the institutions which should assess progress and impact. More specifically, although the project drafted an MOU with MOH (and also with the CHT), it was never finalized10. Consequently, agreement on expected results and mechanisms for regular reporting was not adequately defined. 10 The lack of a formal agreement between CSH and MOH was due in part to USAID’s desire to establish an MOU with MOH for its entire portfolio, comprising: Partnership for Advancing Community Services (PACS), Health Communication Capacity Collaborative (HC3), Maternal and Child Survival Program (MCSP), Pre-service Training/Human Resources for Health, The Global Health Supply Chain/Procurement and Supply Management (GHSC-PSM), and President’s Malaria Initiative (PMI). This comprehensive MOU was initiated in 2015 but has never been finalized. CSH A: Measure the quantity and timeliness of deliverables C: Substantive and chronological fit of the activities (i.e., are the right things being done at the right time?) D: Quality of the activity and/or materials as judged by the recipients E: Measures of improved capacity (using the tools developed under RBSH and CSH) G: Measures of the effect of the portfolio (CSH, FARA, GEMS) on health indicators Contract (A) Link contract SOW - Plans of work - Deliverables MSH has committed to producing agreed-upon deliverables contributing to six priority areas. Link sector management capacity - Health service delivery - Health status Link deliverables - Component improvements - Sector management USAID assumes combining selected instruments and financing will contribute to better health MSH assumes combining inputs in six key priorities will strengthen health sector capacity and service delivery. USAID USAID has contracted MSH to provide assistance in six areas corresponding to key priorities of MOH's 2011-21 Health Policy and subsequently to those of the Plan for Building a Resilient Health System (2016-2021). Table 14: Overview of key project and program relationships MOH MOH assumes implementing sector reforms and capacity-building measures will improve health status. MOH assumes CSH interventions will measurably improve health capacity and service delivery. CDCS (G) Results Framework (C, D, and E) 126 ANNEX 15: PROPOSED CSH RESULTS FRAMEWORK AND INDICATOR REVISIONS The evaluation team recommends: (i) a reorganization of the project’s components; and (ii) a modification of the existing indicators. In addition, the evaluation team found that including health service delivery indicators in the periodic reports was: (i) inappropriate, given CSH’s objectives; (ii) occasionally inaccurate, since MOH revises the data without necessarily informing CSH; and (iii) unappreciated by MOH, as CSH was simply reporting DHIS2 data without attribution. Regarding (i), the evaluation team proposes that the current seven components be reorganized into four: Stewardship (L&G); Resource Management (HR, SCM, HCF, and infrastructure); Health Service Delivery (routine quality and emergency preparedness); and Accountability (HIS and citizen engagement). In other words, the component building blocks would be replaced by the broader health sector functions. This new proposed CSH Results Framework would be better aligned with the National Resilient Health System Plan and would support the results framework for USAID/Liberia’s DO 3 Improved Health Status of Liberians. Please find below a new proposed CSH Results Framework detailing the evaluation team’s revisions to CSH’s components and subcomponents. For comparison, this annex also includes the prior CSH Results Framework from its activity MELP v7 and the DO 3 Results Framework. Regarding (ii), the evaluation proposes a modification of CSH’s existing indicators to better reflect the activity’s objective. Further below is a table that summarizes indicators used by USAID, MOH, and CSH, and ones recommended and developed by the evaluation team. The table organizes them according to the proposed CSH Results Framework. These indicators originate in USAID’s PPR for FY 2017 and the list of new and updated F indicators from June 2016, as well as the National Resilient Health System Plan, other MOH documents, and previous CSH activity MELPs. Although the performance evaluation team recommends, removing health service delivery indicators, the table keeps several such indicators as USAID uses these to report in its Performance Plan and Report to Washington, DC. Without CSH collecting this data, USAID may face difficulty preparing its PPR in FY 2017. The evaluation team recommends USAID, CSH, and MOH select from between these 48 indicators to develop a new CSH activity MELP that better reflects the objectives of the activity going forward. As 48 indicators is likely beyond the manageable interest of CSH, collaboration and coordination between USAID, CSH, and MOH will help select the best ones to measure USAID’s support for strengthening health systems. 127 128 129 130 Proposed CSH Performance Indicator Summary Table USAID/Liberia Goal – Strengthened Liberian Institutions Positioned to Drive Inclusive Economist Growth and Poverty Reduction Development Objective 3 – Improved health status of Liberia USAID Intermedia Result (IR) Former CSH Objective Former CSH Strategic Objective (SO) New CSH Component New CSH Subcomponent Performance Indicator Indicator Type Proposed Data Source Origin IR 3.2 – Increased effectiveness of health systems at national and county level. Objective 1: Strengthen MOH Leadership & Governance SO 1.1; 1.3-1.4 Component 1: Strengthened stewardship at MOH at the central and county levels Sub-Component 1.1: Improved strategic leadership Database to track accreditation/CME established by professional councils (LMDC, LBNM, LPB) Output CSH Deliverable PE Plan to improve regulation & control of pharmaceuticals developed with stakeholders Output CSH Deliverable PE HL-1 Universal Health Coverage (UHC) implementation score Outcome CSH New F Indicator SO 1.2 Sub-Component 1.2: Enhanced operational capacity Counties with functional stakeholder forums (CHB) Output CSH M&E Plan & National Resilient Plan PE Capacity of CHT to hold responsibility and ownership for the health system within their county Output Capacity tool PE % of CHT fully established and functional Output National Resilient Plan PE HL-1 Number of Universal Health Coverage (UHC) areas supported by USG investment Output CSH PPR IR 3.3 – Improved Government of Liberia (GOL) capacity to control infectious diseases Objective 6: Increase financial sustainability of services SO 6.1- 6.3 Component 2: Improved resource management Sub-Component 2.1: Enhanced financial management, reporting and resource mobilization % of untied bilateral in the county Output National Resilient Plan PE Finalize design/Secure approval of NHI/HCF scheme Output CSH Deliverable PE Reduction of the number of Deloitte audit findings Outcome PFM Progress Tracker PE Annual planning and budget preparation Output M&E Plan, vers. 8 PE 131 % of execution of annual allocation of GoL budget for health Output CSH; National Resilient Plan PE Budget execution up to standard per PFM and SOPs Output PFM Progress Tracker PE Timely submission of monthly expenditure reports Output PFM Progress Tracker PE Annual internal/regional audit Output PFM Progress Tracker PE IR 3.1 – Increased utilizations of quality reproductive, maternal, newborn & child health services; IR 3.2 – Increased effectiveness of health systems at national and county level Objective 4: Strengthen human resources for health management SO 4.1- 4.5 Sub-Component 2.2: Institutionalized human resource management, training and licensure Skilled health workforce (physicians, nurses, midwives, physician assistants) per 10,000 persons Outcome CSH; National Resilient Plan PE % of facilities meeting minimum staffing norms Outcome CSH PE System tracks staffing and vacancies by unit (iHRIS) Output M&E Plan, vers. 8 PE Ability to recruit HR/Staff facilities per national norms Output Capacity tool PE No. of pre-service programs accredited by regulatory bodies Output CSH PE System tracks in-service training provided by recipient Output M&E Plan, vers. 8 PE Capacity to coordinate professional health staff development Output Capacity tool PE Capacity to review staff performance per national guidelines Output Capacity tool PE Strategic/Operational plan to strengthen performance and motivation; PBF Output CSH Deliverables PE 3.1.3.1-1 Number of health workers trained in case management with artemisinin￾based combination therapy (ACTs) with USG funds Output CSH PPR 132 HL-2 Presence of the Mission support to strengthen Human Resources for Health (HRH) Output CSH PPR IR 3.1 – Increased utilizations of quality reproductive, maternal, newborn & child health services Objective 5: Improve supply chain management SO 5.1- 5.3 Sub-Component 2.3: Increased non-consumable and consumable material resources Procurement SOPs implemented by committees Output PFM Progress Tracker PE % of facilities with no stock-outs of tracer drugs during a given period Outcome CSH; National Resilient Plan PE Warehouse management procedures followed Output PFM Progress Tracker PE Timely/Accurate estimates of commodity needs (LMIS) Output CSH M&E Plan & Capacity Tool PE HL.7.1-1 Couple Years protection in USG supported programs Outcome CSH PPR IR 3.1 – Increased utilizations of quality reproductive, maternal, newborn & child health services Objective 3: Institutionalize QA and QI initiatives to improve health care service SO 3.5 Component 3: Improved health service delivery Sub-Component 3.1: Improve availability and use of routine health services National laboratory policy/Implementation plan developed Output CSH PE % of health facilities that can conduct routine diagnostic tests Outcome CSH PE % of clients satisfied with service provision at the health facility Outcome CSH PE # and % of public sector facilities reaching two star level in accreditation survey incl. clinical standards Outcome CSH; National Resilient Plan; PE 3.1.1-59 Number of pregnant women with known HIV status (includes women who were tested for HIV and received their results) (PEPFAR output - #P1.1D) Outcome CSH PPR 3.1.1-69 Number of eligible adults and children provided with a minimum of one care service (PEPFAR output - #C.1.1D) Outcome CSH PPR IR 3.1 – Increased utilizations of quality reproductive, maternal, SO 3.1 Sub-Component 3.2: Institutionalize QI and QA initiatives NQS/IPC strategy developed Output M&E Plan, vers. 8 PE Clinical standards updated Output M&E Plan, vers. 9 PE QA/QI approaches being implemented Output M&E Plan, vers. 10 PE 133 newborn & child health services; IR 3.2 – Increased effectiveness of health systems at national and county level Guidelines adapted to national standards Output M&E Plan, vers. 11 PE Supportive supervision implemented Output M&E Plan, vers. 12 PE Sub-Component 3.3: Strengthen epidemic surveillance / IDSR at the county level % of counties with funded outbreak preparedness & response plans Output National Resilient Plan PE Objective 3: Institutionalize QA and QI initiatives to improve health care service; Objective 7: Strengthen health management information systems SO 3.3- 3.4 Component 4: Enhance accountability and transparency for improved health sector governance Sub-Component 4.1: Improve the frequency and quality of data collection to enhance monitoring and evaluation Timely collection/Submission MOH/Feedback to facilities Output Capacity tool PE # & % of timely, accurate and complete HIS reports submitted Output CSH; National Resilience Plan PE Quarterly reviews/Data use for planning Output Capacity tool PE HL-3 Presence of mission support for integration of health information systems (HIS) Output CSH PPR Objective 3: Institutionalize QA and QI initiatives to improve health care service SO 3.2 Sub-Component 4.2: Promote citizen engagement through feedback mechanisms Client feedback mechanisms designed/implemented Output CSH Deliverables PE 134 U.S. Agency for International Development - Liberia 502 Benson Street Monrovia, Liberia