MID-TERM EVALUATION OF THE USAID/LIBERIA HEALTH SECTOR FIXED AMOUNT REIMBURSEMENT AGREEMENT (FARA) 2013 CONTENTS Acronyms.......................................................................................................................................................................3 Executive Summary .......................................................................................................................................................5 I. Introduction............................................................................................................................................................6 II. Background.............................................................................................................................................................7 III. Methods and Data Sources...................................................................................................................................11 IV. Findings.................................................................................................................................................................11 A. Clinical Indicators and Service Delivery Performance ...................................................................................11 B. Capacity Building and Health Systems Strengthening...................................................................................16 C. Financial Management & Cost Savings .........................................................................................................23 V. Conclusions and Key Findings...............................................................................................................................28 VI. Recommendations................................................................................................................................................30 Mid-Term Evaluation - USAID/Liberia Health Sector FARA 2 Mid-Term Evaluation - USAID/Liberia Health Sector FARA 3 ACRONYMS ACT Artemisinin-based combination therapy AIDS Acquired immune deficiency syndrome ANC Ante-natal care BCC Behavior change communication BPHS Basic Package of Health Services CHSWT County Health and Social Welfare Team (or CHT) CHV Community health volunteer (also gCHV) CHAI Clinton Health Access Initiative CM Certified midwife DHIS District Health Information System ENA Essential Nutrition Actions EPI Expanded Program on Immunization EPHS Essential Package of Health Services (replaced BPHS) FARA Fixed Amount Reimbursement Agreement FP Family planning gCHV (General) community health volunteer HMIS Health Management Information System HFAS Health Facility Accreditation Survey HIV Human immunodeficiency virus HMIS Health management information system (HIS2) IEC Information, Education and Communication IPT Intermittent preventive treatment of malaria (in pregnancy) IPT2 Intermittent preventive treatment of malaria (in pregnancy), 2nd dose IR Intermediate result IRC International Rescue Committee ITN Insecticide-treated net JHU/CCP Johns Hopkins University Center for Communication Programs JSI John Snow Research & Training, Inc. LBNM Liberia Board of Nursing and Midwifery M&E Monitoring and evaluation MOF MOHSW Ministry of Finance Ministry of Health and Social Welfare MSH Management Sciences for Health NDS National Drug Service NGO Non-governmental organization NHP National Health Plan (2007-2011) NHSWPP National Health and Social Welfare Policy & Plan 2011-2021 NMCP National Malaria Control Program PBC Performance-based contract PBF Performance-based financing PMI President’s Malaria Initiative PMTCT Prevention of mother-to-child transmission Mid-Term Evaluation - USAID/Liberia Health Sector FARA 4 QA Quality assurance RBHS Rebuilding Basic Health Services RFP Request for proposal SP Sulfadoxine-pyrimethamine (Fansidar) TB Tuberculosis TTM Trained traditional midwife USAID United States Agency for International Development Mid-Term Evaluation - USAID/Liberia Health Sector FARA 5 EXECUTIVE SUMMARY In 2008, USAID established the Rebuilding Health Services (RBHS) Project in Liberia to rebuild the capacity of the Ministry of Health and Social Welfare (MOHSW), and to contract out the delivery of the government’s BPHS1 for one third of the Liberian population across seven Counties. In 2011, USAID made the strategic decision to consolidate programming in three of the largest Counties and transition responsibility for service delivery from RBHS to the MOHSW directly, citing that the new approach would accelerate MOHSW stewardship and focus capacity building efforts. The transition was made possible through a four-year, $42 million Fixed Amount Reimbursement Agreement (FARA) that USAID signed with the MOHSW and Ministry of Finance (MOF). The FARA was pre-financed by the GOL through the National Budget with USAID reimbursing the GOL based on its achievement of pre￾determined and costed deliverables The RBHS Project was redirected to focus on capacity building and providing technical support to the MOH in its management of the FARA. The Liberian Health Sector FARA was an innovative adaptation of the FAR mechanism, traditionally used within USAID to finance construction projects. In Liberia, USAID used the “output” orientation of the FAR approach to design a program with the MOH that would reimburse the GOL for achieving quarterly process deliverables. The deliverables were structured in such a way to not only incentivize timely “contracting out” of NGOs but to ensure the services delivered were of high quality and accessible through strong MOHSW oversight and support. Given that the FARA was a novel approach for delivering development assistance in the health sector, in 2013, USAID contracted out a mid-term evaluation of the FARA, asking the evaluators to consider: 1. If the transition of NGO contract management from RBHS to the MOHSW had resulted in a decline in clinical indicators, which was a concern of many stakeholders when the design was introduced; 2. If the “learning by doing” approach utilized in the FARA had strengthened MOHSW capacity and health systems, as intended by the designers of the FARA; and 3. If the transition to a government-to-government approach from a USAID implementing partner had generated cost savings for USAID, which was the perspective of the MOHSW. The evaluators concluded that, in general, the reach of service delivery, as indicated by MNCH indicators in Bong, Lofa, and Nimba Counties, did not recede in the first year of FARA implementation. Since it was too early to assess progress against a Capacity Building Plan that measured the MOHSW’s capacity to perform specific functions deemed critical for extending the coverage of the EPHS, the mid￾term evaluation used feedback and opinions of stakeholders to determine whether the MOHSW and CHSWT capacity to oversee and manage service delivery improved. The mid-term evaluation identified several examples of enhanced stewardship and effective management, especially at the central MOH, in the implementation of the FARA. 1 The BPHS was replaced by the EPHS in 2011. Mid-Term Evaluation - USAID/Liberia Health Sector FARA 6 The mid-term evaluation was unable to assess whether the shift to the FARA approach generated cost savings; however, the evaluators did find evidence of cost-savings in some specific areas, such as use of MOHSW financial systems and fixed procurement costs. I. INTRODUCTION The purpose of this evaluation was to review the performance and the progress of the USAID Liberia Health Sector Fixed Amount Reimbursement Agreement (FARA) (2011 – 2015). The evaluation served as a mid-term program assessment and review of changes made since the outset of the project, as well as an opportunity to provide recommendations for future direction. Preliminary results and program data were reviewed to assess how the FARA is affecting health service delivery, capacity building, and systems strengthening, and whether the project is on track to achieve its objectives. Where appropriate, the evaluation also considered the preliminary results of the FARA alongside complementary USAID investments, particularly for capacity building. The evaluation was focused on three key hypotheses, which were articulated by various stakeholders during the FARA design process. Stakeholders identified key questions against which to measure performance, and hypothesized that the FARA may: 1. Result in a decline in clinical indicators, at least during and immediately following the transition of NGO contract management from RBHS to the MOHSW. Did clinical indicators decline, maintain or improve in facilities that transitioned contract management from RBHS to MOHSW? 2. Facilitate a “learn by doing” approach to strengthen GOL capacity and health systems. Has the FARA built the capacity of the MOHSW and CHSWTs and health facilities in FARA counties? Were the underlying health systems in the health sector strengthened as a result of the FARA? 3. Generate cost savings. What were the service delivery and technical assistance costs associated with the program before and after the introduction of the FARA? What are the approaches and critical next steps that the mission should take with the MOHSW to determine if the FARA is producing good value for money? The evaluation team reviewed project data and interviewed a broad range of stakeholders to address these hypotheses and assess whether the FARA approach was contributing to both USAID’s Development Objective2 and the primary objectives of Liberia’s National Health and Social Welfare Policy.3 In addition, the evaluation identified lessons learned and provided recommendations to strengthen the FARA as well as guide future activities. The complete SOW for the evaluation is located in Annex A. 2 The three Intermediate Objectives (IR) under DO3 (Improved Health Status of Liberians) are: IR1: Increased Utilization of Quality Health Services, IR2: More responsive services through effective health systems decentralization, and IR3: Increased financial sustainability of services 3 The National Health and Social Welfare Policy objectives are laid out on page 8. Mid-Term Evaluation - USAID/Liberia Health Sector FARA 7 II. BACKGROUND Health in Liberia Liberia emerged from its protracted civil war as a deeply divided country, its social fabric torn by ethnicity, religion, geography and history. Basic infrastructure, including roads, water and electricity supply, schools, health clinics, was destroyed. Social, political, and economic governance systems at all levels dissolved as government functions were disrupted and skilled individuals left the country. Overcoming a severely damaged economy, a nearly dysfunctional state and a war-weary, fragmented society has been a major challenge facing the GOL over the last decade. Ensuring that ordinary Liberians see real benefit from their government has been critical for solidifying peace and bringing stability and prosperity to Liberia and the region. 4 Bringing tangible health benefits to Liberian citizens is emblematic of the GOL’s role and responsibility in attending to the care of its citizens. Since 2006, Liberia has sought to move away from a post‐conflict context—where health care delivery was fragmented, uneven, and dependent on donor‐funded vertical programs and humanitarian relief organizations supplying emergency services—toward a development context focused on building systems for more sustainable health care delivery and strengthening GOL stewardship of the health sector. A key step in making this transition was the launch in 2007 of the Government of Liberia’s National Health Plan, which set out a policy for transitioning from post-conflict recovery to sustainable development under GOL leadership. The cornerstone of the National Health Plan was the Basic Package of Health Services (BPHS), a set of high‐impact interventions designed to address the leading causes of mortality and morbidity. At the time the National Health Plan was developed, Liberia had some of the worst health indicators in the world: maternal mortality was 994 per 100,000, under‐five mortality rate was 110 deaths per 1,000 live births; and population’s geographic access to health services was estimated at 41 percent. Three-fourths of government-owned health facilities were managed by NGOs with emergency funding from international donors, while facilities run by the government lacked staff, drugs and equipment.5 By 2008, total health and social welfare expenditure had reached over US$ 100 million (or US$ 29 per person), or 15 percent of GDP. This was higher than Liberia’s pre-war level of spending, and brought Liberia more in line with the West and Central Africa Region. Yet external donors and household spending largely accounted for the high levels of expenditure (47 and 35 percent respectively); government spending accounted for just 15 percent. In addition, 85 percent of out-of-pocket (or household) spending went to private providers, despite a policy of free services. This high level of household spending had a detrimental effect on poor households, and supported the need for sustained 4 USAID Liberia CDCS – Public Version – From Final Approved 4-4-13. Mid-Term Evaluation - USAID/Liberia Health Sector FARA 8 government health spending and more efficient use of health resources to maximize its potential impact.6 In 2011, the Ministry of Health and Social Welfare (MOHSW) developed its second National Health and Social Welfare Plan and Policy (NHSWPP) and introduced the Essential Package of Health Services (EPHS), building on the experience of implementing the 2007 NHP. The NHSWPP intended to establish an evidence-based policy framework to guide decision-makers over the next ten years (2011-2021). The stated goal of the policy is to improve the health and social welfare status of the population of Liberia on an equitable basis. To accomplish this, the NHSWPP lays out three broad objectives: 1) Increasing access to and utilization of a comprehensive package of quality health and social welfare services of proven effectiveness, delivered close to the community, endowed with the necessary resources and supported by effective systems; 2) Making health and social welfare services more responsive to people’s needs, demands and expectations by transferring management and decision-making to lower administration levels; and 3) Making health care and social protection available to all people in Liberia, regardless of their position in society, and at a cost that is affordable to the Country. To support the NHSWPP, government spending has nearly doubled in absolute terms yet remained stable as a percentage of the national budget (between 7 and 8 percent). 7 Donor funds predominantly are used to support primary health care, while referral hospitals consume the largest portion of government expenditure. While donor funding (in absolute terms) has remained fairly consistent over the last five years, as the economy continues to grow and the national budget increases, foreign assistance should decline in both relative and absolute terms, requiring that GOL spending on health and social welfare increase in proportion. USAID/Liberia Country Strategy and Health Sector Assistance USAID is the largest bilateral donor in Liberia, with the goal to support “Strengthened Liberian Institutions Positioned to Drive Inclusive Economic Growth and Poverty Reduction”. The results framework leading to this goal is built around four Development Objectives (DO); DO-3, “Improved health status of Liberians”, is directed towards support of health services that expand all Liberian’s opportunities to contribute to and benefit from development progress.8 Implementation of this objective is grounded in the principle of county-led development and strategically aligned with GOL priorities. In keeping with the principles of the USAID Forward initiative, planning is country-led with a major focus on capacity building. The Intermediate Results (IRs) of DO-3 are directly aligned with the three key objectives of the 2011-21 NHSWPP: 6 USAID Liberia CDCS – Public Version – From Final Approved 4-4-13. 6 MOHSW Country Situational Analysis Report, July 2011. 7 MOHSW Country Situational Analysis Report, July 2011. 8 USAID Liberia CDCS – Public Version – From Final Approved 4-4-13. Mid-Term Evaluation - USAID/Liberia Health Sector FARA 9 MOHSW NHSWPP 2011-21 Goal: Improved health status of the population USAID/Liberia CDCS DO-3: Improved health status of Liberians Key Objectives: 1. Increasing access to and utilization of high quality services; 2. Making services more responsive to the population, with attention to equity; and 3. Providing services that are affordable to the country. Intermediate Results: 3.1. Increased utilization of quality services; 3.2. More responsive services through effective health system decentralization; and 3.3. Increased financial sustainability of services. Overview of the FARA The Liberian Health Sector FARA, entitled Support to the Ministry of Health and Social Welfare for Implementation of Liberia’s 2011-2021 National Health and Social Welfare Policy and Plan and signed in September 2011, was designed to both build MOHSW capacity to manage the systems required to successfully implement the agreement, as well as directly support health service delivery through performance based contracts (PBCs) to NGOs in Bong, Nimba and Lofa counties (North Central region). The FARA was developed in the context of the gradual transition away from a post-conflict setting to a GOL-managed health sector; building on incremental improvements in health indicators and systems strengthening while maintaining provisions for service delivery and support to national systems. The groundwork for the FARA built on the experience and implementation of the five-year $65 million USAID/Liberia health project, Rebuilding Basic Health Services (RBHS), which began in November 2008. Under RBHS, the PBCs for health service delivery by NGOs began in July 2009. The USAID funds and primary responsibility for award and management of PBCs with NGOs rested with international NGOs. The MOHSW had only limited involvement in the management of RBHS, with a minority vote on the technical evaluation committee for NGO selection and a secondary role in technical oversight through joint monitoring with RBHS and RBHS-run partner meetings. The development of a Government-to-Government (G2G) approach by the Mission and the GOL transitioned responsibility for PBC management from RBHS to the MOHSW. Under the FARA, responsibility for awarding and managing PBCs was formally transferred to the MOHSW. From 2011 to 2012 the PBCs with NGOs were phased out, and new contracts were introduced by the MOHSW, beginning with Lofa County in 2011, and followed by Nimba and Bong Counties in 2012. The role of the RBHS Project was revised to focus on technical support, capacity building of the MOHSW (at county and central levels), and infrastructure improvements. Program Areas Supported through the FARA The GOL agreed to finance the FARA through the National Budget with USAID reimbursing the GOL based on its achievement of pre-determined and costed deliverables. Through this arrangement, the FARA makes available up to $42 million of USAID assistance to the MOHSW in support of the NHP and in particular, delivery of the EPHS at the primary care level in 3 counties. The program description outlines the three main FARA activities as follows: Mid-Term Evaluation - USAID/Liberia Health Sector FARA 10 1) MOHSW Technical and Management Support. The FARA provides support for 20 technical and management consultants, assigned to key units within the MOHSW, including the Program Management Unit (PMU), the M&E Unit, the Office of Financial Management (OFM), the Procurement Unit, and the Quality Assurance (QA) and Performance-based Finance (PBF) teams. 2) Supervision Activities by the National Malaria Control Program (NMCP) and the Expanded Program for Immunization (EPI). The FARA outlines supervision of facilities by both the NMCP and EPI unit to ensure quality delivery of high priority services. 3) Procurement of Preventative and Curative Health Services and Commodities. The FARA supports PBCs with NGOs to implement the EPHS in Lofa, Bong, and Nimba Counties. The MOHSW is expected to manage these contracts with technical guidance, supportive supervision, provision of drugs and supplies, and payments for services. This activity also included the management of contracts with NGOs to distribute Insecticide-treated Nets (ITNs), and procure pharmaceuticals for supported facilities, though these were later removed from the FARA. For each activity, a comprehensive set of time-bound deliverables was developed, and quarterly reimbursements were made conditional upon successful completion and USAID verification of the deliverables. Program Monitoring and Evaluation Reporting and monitoring of the FARA is primarily the responsibility of the MOHSW, with support from County Health and Social Welfare Teams (CHSWT or CHT), which submits quarterly progress reports to USAID. USAID staff also conduct quarterly field visits to monitor implementation and assess progress against baseline assessment results. Quarterly reports provide an overview of outputs achieved, problems encountered, and other relevant information related to producing the deliverables outlined in the FARA. Specific deliverables outline additional reporting and other documentation to be provided by the MOHSW, including validation of administrative and service delivery indicators, procurement documents, work plans, management of technical consultants, quality assurance reports, and meeting reports. Deliverables build progressively with each year of implementation of the FARA, which are intended to simultaneously monitor progress while building the capacity of the MOHSW to do so independently. A selection of FARA deliverables, which are linked directly to reimbursement amounts, are shown below:  Monthly partner meeting report  RFP and selection memo for EPHS implementation in Lofa County  Annual work objectives for Technical and Management Consultants  Annual supervision/monitoring work plans  Quarterly supervision/monitoring report countersigned by NGO and CHSWT  Quarterly data quality harmonization review report  Quarterly newsletter  RFP for implementation of EPHS in Nimba and Bong Counties Mid-Term Evaluation - USAID/Liberia Health Sector FARA 11  Procurement selection memo for NGOs to implement EPHS in Nimba and Bong Counties  Annual Accreditation and QA report  Quarterly QA assessment report  Quarterly RBHS data validation report  Annual performance appraisals for Technical and Management Consultants  Post ITN campaign survey report (Year 2) As shown above, FARA deliverables contain a mix of process indicators, procurement functions, and specific activities, such as ITN distribution and the annual accreditation. Through the combination of direct program monitoring and evaluation and other MOHSW reporting, the FARA generates a significant volume of data describing performance of the project as well as progress in the health sector. III. METHODS AND DATA SOURCES The evaluation took place in country between July 3 and August 9, 2013. The evaluation team used both quantitative and qualitative methods during the evaluation. Quantitative information was derived from program data collected by RBHS using MOHSW systems, and an analysis was performed by both RBHS and the evaluation team. Qualitative information was generated through interviews and observations, informed by the use of a semi-structured questionnaire, which was modified to illicit views and feedback on each individual’s area of expertise. The evaluation report is based upon a number of sources of information: 1) a desk review of Mission, RBHS, other donor and MOHSW reports, 2) an analysis of health facility statistics reported by the MOHSW and verified by RBHS and USAID, 3) interviews with key stakeholders, and 4) survey data from the Liberia DHS (2007 and preliminary data for 2013). Limitations A national strike of nurses (and some other workers) in the MOHSW occurred on the first day of a scheduled field visit (July 22) to Bong County. Two members of the evaluation team managed to meet for several hours with the Bong county health team and later with Africare, the NGO contracted to manage health facilities under FARA. The strike precluded facility visits – other than one brief stop at Salala health center, and thus greatly constricted the time available to interview health facility workers and assess conditions in FARA-assisted clinics. The evidence collected by the evaluation team from the field – anecdotal and first-hand, was therefore severely limited by this unexpected turn of events. IV. FINDINGS A. Clinical Indicators and Service Delivery Performance The first key hypothesis to be addressed by this evaluation is whether the introduction of the FARA led to a decline in access to quality health services. The Evaluation Team explored the following question: Did clinical indicators decline, maintain or improve in facilities that transitioned contract management from RBHS to MOHSW under the FARA? Mid-Term Evaluation - USAID/Liberia Health Sector FARA 12 To address this question, the evaluation reviewed seven clinical indicators that routinely are collected by the MOHSW via its Health Management Information System (HMIS). These indicators are monitored to assess and track health facility performance and are tied to the delivery of high impact interventions prioritized in the EPHS. Two of the six indicators are performance-based indicators, incentivized in the current PBCs with NGO implementing partners. 9 Six Clinical Indicators % Penta 3 coverage % ANC 4 coverage % facility delivery by skilled birth attendants % Pregnant women receiving IPT2 % <5 malaria treated with ACT Couple-years protection Each month, health facilities report service delivery data via an integrated HMIS form to the County Health Team, where the form is reviewed for errors and entered into the DHIS2 – an online database that is the repository for the MOHSW service delivery statistics, and available to users across the MOHSW. The HMIS provides a standardized reporting system for some 480 MOHSW facilities (including some private facilities) in all 15 counties of the country, and covers delivery of the EPHS. DHIS2 data is generated by the health facilities, which compile summary statistics from their hard copy registers and submit a written monthly report to the CHSWT. The CHSWT in turn, consolidates reporting by district, records the data in DHIS2, and submits a county-level report to the Central MOHSW. Staff at the MOHSW including the M&E, PBF and other divisions review and validate (after verifying accuracy of reported data and making necessary adjustments) county reporting and produce a consolidated, national report. These reports are reviewed on a monthly basis, and guide planning and management functions across the various programs. On a quarterly basis, the MOHSW uses DHIS2 to compile health indicators, performance trends and activities in a report that is shared with partners and stakeholders. In addition, the MOHSW verifies a sample of the 112 health facilities’ supported under the FARA each quarter to ensure accuracy of reporting and validity of the results. Results of the verification are made available in the FARA quarterly report on “Verification and Data Harmonization”, which is co-signed by RBHS. Using the Verification and Data Harmonization Report, the Evaluation Team reviewed the quarterly results for the six clinical indicators in two ways: 1) Comparison of quarterly indicators, by county, for the quarter ending March 2012 (the first quarter of FARA in Lofa and prior to FARA introduction in Bong and Nimba) with indicators for the quarter ending March 2013 (following 15 months of implementation in Lofa, and 9 months in Bong and Nimba). 9 For the most part, the specific indicators tied to performance incentives have changed very little since 2009 (when PBCs were first introduced under the RBHS project). Mid-Term Evaluation - USAID/Liberia Health Sector FARA 13 2) Multi-year trend analysis and comparison of FARA facilities, by county, for each quarter from July 2009 (the start of RBHS) to September 2013. For each indicator, the Evaluation Team compared the program data from health facilities (described above) with health survey data from the 2007 and 2013 Demographic Health Surveys (DHS) as well as the 2011 Malaria Indicator Survey, which provide region and county level coverage estimates based on household responses. The 2007 and 2013 DHS provide the most comprehensive overview of the health status of Liberians, focusing on health indicators targeted by the EPHS and objectives of the NHPP. Direct comparison with health facility data is challenging in some cases, but taken together provide a good overview of progress made and areas that need additional attention. Survey results are included as appropriate to interpret trends and may provide a counter-check for facility reports in some cases. Individual Indicator Findings and Discussion Child Health (Immunization Coverage): Reported immunization coverage of children under 1 with the third (and final) dose of pentavalent vaccine (also referred to as DPT3 or penta-3) increased since the start of the FARA, continuing a multi-year upward trend. Comparing the first quarter of 2012 with the first quarter of 2013, coverage increased from 104% to 112% in Bong, from 94% to 122% in Lofa, and from 110% to 113% in Nimba. While the number of children vaccinated has increased, coverage over 100 percent may also indicate data quality issues, and may require additional verification. Higher than expected coverage estimates may be due to inclusion of children over one year who were vaccinated during immunization outreach, underestimating the population denominator (i.e. the percent of children in the population that are children under one year), or in some cases poor quality data. Ongoing review and verification is necessary to understand what accounts for these figures and identify potential data quality issues, to ensure that figures remain relevant for measuring progress and setting future targets. Comparing routine facility data with DHS results is difficult. For the reasons outlined above facilities typically report higher coverage, whereas in the case of DHS data, coverage may be underestimated due to recall bias and parents not having vaccination cards at the time of an interview. Nonetheless, survey data also show a general improvement in DPT3 coverage in the North Central region since 2007 and no statistically significant difference in coverage between the 2012 EPI Cluster Survey and the 2013 DHS. The preliminary results of the 2013 DHS estimated that 68 percent of children under 1 year received three doses of pentavalent vaccine, up from just 46 percent in 2007. Coverage of penta-3 as estimated by the 2013 DHS was 81 percent in Lofa, 62 percent in Bong, and 69 percent in Nimba. Maternal Health (ANC4+): The ANC4+ indicator measures the percentage of pregnant women who receive the recommended four or more antenatal care visits during pregnancy. ANC4+ was a performance indicator under the RBHS project, but is not incentivized under the FARA (though it is routinely monitored). In the three quarters since the launch of the FARA, ANC4+ has increased slightly in Bong and Nimba, but shown no improvement in Lofa. A comparison of Jan-Mar 2012 with Jan-Mar 2013 showed that over the one-year period ANC4+ visits had increased in Bong (from 72 to 88 percent) and Nimba (from 69 to 91 percent), and decreased slightly in Lofa (from 69 to 65 percent). In Lofa, ANC4+ Mid-Term Evaluation - USAID/Liberia Health Sector FARA 14 declined considerably in the final year of RBHS, from around 125% throughout 2010 and the first two quarters of 2011, to less than 75% in the final two quarters of 2011, and has remained at these lower levels under the FARA. Given the unusually high estimates for 2010-2011, it’s unclear whether the lower estimates are actually an improvement in data quality, or a real decline in the percent of women receiving 4 or more ANC visits. The preliminary 2013 DHS results did not assess the percentage of women receiving four or more ANC visits during pregnancy. However the DHS estimated that the majority (96 percent) of women in the North Central region received at least one prenatal visit during their most recent pregnancy in the last five years, up from just 63 percent in 2007. Skilled Birth Attendant at Delivery: The percent of deliveries that are facility based with a skilled birth attendant (SBA) has increased in all three FARA-supported counties, continuing an upward trend that began in 2010 during the RBHS project. Comparing the quarter ending in March 2012 with the same quarter in 2013, progress on SBA has continued under the FARA, from 46% to 61% in Bong, 45% to 62% in Lofa, and 64% to 68% in Nimba. Progress on this indicator has been remarkable in the four years included in this evaluation; in 2009 Bong and Nimba County reported a quarterly average of 11% and 13% SBA, respectively, and in 2012 the quarterly average was 54% in Bong and 62% in Nimba. In Lofa the results were slightly less dramatic, but impressive nonetheless, from an average of 31% in 2009 to 47% in 2012. The 2013 DHS results reported similar gains for the North Central region; in 2013 51.4 percent of women reported that their last live birth in the five years preceding the survey was delivered by a skilled provider. Fifty-seven percent of births were delivered in a health facility, a level much greater than that reported in the 2007 LDHS (31 percent). Prevention of malaria in pregnant women (IPT2): The rate of administration of IPT2, which records the second of at least two presumptive (and preventive) doses of anti-malarial drugs (usually SP/Fansidar), has varied considerably over the past four years in all three FARA/RBHS supported counties. In Bong County, IPT2 has increased since the introduction of the FARA, from 70% to 79% in the first three quarters. In Nimba and Lofa, performance is more difficult to assess, increasing from 45% to 58% in Lofa, and dropping from 78% to 61% in Nimba, but varying considerably from quarter to quarter. While IPT2 coverage has been erratic, there has been an overall decrease in the North Central region between the 2011 Malaria Indicator Survey and the 2013 DHS and coverage appears lower than may be expected, particularly when considering the high coverage of ANC4+. This may partly be due to challenges with recording which dose is being given in instances where women may attend different facilities for ANC visits, or do not attend regularly during their third trimester (when the 2nd dose should be given). In addition, WHO guidelines for IPT state that each dose should be given at least 1 month apart, beginning in the second trimester; women attending ANC later in their pregnancy may not have enough time to receive the second dose. Mid-Term Evaluation - USAID/Liberia Health Sector FARA 15 Anecdotally, some of those interviewed attributed a lack of improvement to widespread and periodic stock-outs of malaria drugs, especially SP/Fansidar which was procured by GOL and not GFATM or PMI. Given the widely variable performance, even in the same facility from one quarter to the next, supply chain issues are likely to be a contributing factor. Partners and supervisors also report that SP/Fansidar is often used for presumptive treatment for malaria, particularly when ACT is not available, which also affects availability of the drug for IPT. In non-FARA facilities in Bong and Nimba counties, which often have less logistical support and experience more frequent stock-outs, IPT2 coverage during the first two quarters of the FARA was much lower (26 and 22 percent, respectively), despite comparable ANC visit rates. Strengthening the supply chain is not a direct activity under the FARA and receives dedicated investment through the USAID DELIVER project, however these issues may continue to limit performance in FARA supported facilities, and may require additional attention. Treatment of Malaria in Children Under 5: ACT is the national standard for treatment of malaria in children under 5, and the ACT indicator tracks adherence to the treatment protocol. The percentage of children under 5 with malaria treated with ACT, among those who sought care, has been reported consistently at or above 90% since 2011 in all three counties, and a high level of adherence has continued since the transition to the FARA. The percentage of malaria cases treated with ACT in the first quarter of 2012 was 100% in Bong, 92% in Lofa, and 93% in Nimba, compared to 99%, 93%, and 97% in the first quarter of 2013. The 2013 DHS reports a desired decline, compared to the 2011 MIS, in the percentage of children under five with fever who took ACT from 46.1% to 23.6% in the North Central region. This change is attributable to the convergence of two important factors during this period, namely the increase in care seeking behavior and the revised treatment protocols requiring confirmatory diagnosis prior to provision of ACT. Family Planning (CYP): The CYP indicator - a proxy for access to modern contraceptive methods - estimates the total contraceptive protection provided by family planning commodities distributed over a given period. Two quarters of facility data (2012 Q3 and 2012 Q4) were available for review from Bong and Nimba following the introduction of the FARA. During that time total CYP per 1,000 women of child￾bearing age (WCBA) remained largely in line with the two quarters prior to the transition to the FARA (from 31 to 30.5 in Bong, and from 17.5 to 18 in Nimba). In Lofa, rates were slightly higher; in the four quarters under the FARA, CYP averaged 26 per 1,000 WCBA, although this was below the rates reported in 2011 (prior to the FARA), when Lofa averaged 34 CYP per 1,000 WCBA. Overall, access to modern methods of contraception appears to have changed relatively little since the introduction of the FARA, and remains low. The CYP data show that only some 3% percent of WCBA – or one out of 20 eligible women - are obtaining a FP method from the MOHSW facilities in the three FARA-assisted counties. These results are consistent with the recent DHS, which estimated a modern contraceptive prevalence rate (CPR) of 13.6 percent in 2013 for the North Central region, though the rate was below 10 percent in Lofa and Nimba. While this is an improvement from the 8.5 percent CPR estimated in 2007 for the region, these results may indicate a need to redouble efforts – both within FARA and through other Mid-Term Evaluation - USAID/Liberia Health Sector FARA 16 mechanisms - in demand creation and supply and distribution networks, alongside other FP donors, such as UNFPA. Conclusions about Service Delivery Performance Performance data from the three counties on maternal and child health, malaria and family planning indicators generally show some improvement from the onset of FARA to 2013. Although it is premature to draw definitive conclusions about trends, available data suggests that for ANC visits, IPT2 coverage, CYP and SBA – performance since the onset of the FARA has slightly increased or at least maintained previous trends. The evidence suggests that in general, the reach of service delivery (as defined by these six indicators) in the three FARA counties did not recede in the first year of FARA implementation. Nonetheless, the evaluation has noted the following critical challenges that may limit service delivery performance and require greater support: Logistics and Transportation: CHSWTs, NGOs and MOHSW/Central officials were quick to mention the challenges of logistics and transport to perform outreach and supervision with the aging and small fleet of vehicles available on the one hand, and the poor, rainy season impassable roads on the other. A quick review of transport available to CHSWTs suggest that most – including the three FARA counties, have only one 4WD vehicle (purchased under RBHS four years ago) and one ambulance available to support their efforts to serve communities which may be as far as 8 or more hours away. Expanding logistical support will be necessary to reach targets for more integrated supportive supervision (one of FARA’s innovations), along with financial encouragements to outreach staff to walk long distances to serve distant and remote communities. USAID should engage with the MOHSW to further increase logistics and transport capacity at the county level. Supply Chain Management: Privately, health donors and some MOHSW officials admit that “drug leakage” is a problem, particularly with regard to malaria drugs, and technical challenges with procurement, supply, warehousing, and rational use of drugs are frequently reported. Dedicated supply chain strengthening is outside of the scope of the FARA, and primarily supported through complementary investments via the USAID DELIVER project. In addition, FARA does not purchase or distribute drugs; procurement of pharmaceuticals has been provided by RBHS. Nonetheless, supply chain issues have a profound impact on implementation of the EPHS under the FARA, and remedial action may be required. In May 2013 USAID and Global Fund placed a moratorium on all distribution of anti-malarials and reproductive health commodities, and took corrective actions to better monitor usage and quantify consumption. A provisional approach to quantification and ordering was introduced while a longer-term strategy is developed to reform and improve performance of the National Drug Service and the MOHSW in procurement and distribution of vital medicines and malaria products. This topic is addressed separately in the recommendations section. B. Capacity Building and Health Systems Strengthening The second key hypothesis to be addressed by this evaluation is whether or not the FARA facilitates a “learn by doing” approach to strengthen GOL capacity and health systems. Key questions reviewed by the Evaluation Team included: Mid-Term Evaluation - USAID/Liberia Health Sector FARA 17 1) Has the FARA built the capacity of the MOHSW and CHSWTs and health facilities in FARA counties? 2) Were the underlying health systems in the health sector strengthened as a result of the FARA? The FARA, along with the RBHS10 and DELIVER11 projects, represent the principal means by which USAID is building the capacity of the MOHSW and strengthening health sector systems. The FARA promotes ‘learn by doing’ approaches, which emphasize successfully achieving quarterly deliverables rather than explicit ‘capacity building’ or ‘system strengthening’ activities. A key assumption is that by supporting the MOHSW to “do” the work, USAID and its implementing partners are strengthening the MOHSW and the underlying systems within the health sector. A Capacity Building Plan was developed by RBHS with the MOHSW during the first year of the FARA. The plan outlines specific objectives and performance targets that address the gaps identified in baseline capacity assessments conducted by RBHS prior to the start of the FARA, specifically related to the MOHSW’s capacity to perform specific functions deemed critical for extending the coverage of the EPHS. While it is too early to assess progress, development of the Capacity Building Plan was cited by stakeholders as a major step towards defining and better articulating capacity needs and activities. Progress in implementing the plan, and performance of the MOHSW related to each of the gaps identified at baseline should be reviewed at the end of the current FARA agreement in 2015. While a review of performance relative to the Capacity Building Plan was not possible at this time, the mid-term evaluation did examine whether MOHSW and CHSWT capacity to oversee and manage service delivery has improved, based on feedback and opinions of stakeholders. Oversight of Service Delivery Integrated Supportive Supervision of Service Delivery by the MOHSW The capacity of the MOHSW to conduct supportive supervision of service delivery was identified as a critical function to monitor and improve coverage of the EPHS and implementation of the FARA in supported counties. Supportive supervision, which is supported through multiple mechanisms within the Ministry, has been revised and prioritized, and represents a major achievement of the FARA to improve coordination between programs and donors while strengthening technical supervision of services. Prior to the FARA, supervision in most counties was carried out with narrowly defined funding and focus specific to individual donors and vertical programs; integrated supervision required a major shift in approach, as well as sufficient funding. Taking advantage of the flexibility afforded by the FARA which focuses on the output/deliverables instead of dictating specific processes, the Chief Medical Officer was able to champion the introduction 10 A review of the capacity building activities of RBHS and NGOs at the county level was outside the scope of this evaluation, but complement FARA objectives and support for service delivery; progress in these areas (particularly related to BCC and communication strategies) may be important to consider at the end of the FARA. 11 The FARA is complemented by the USAID/Liberia investment in the DELIVER project, to purchase medical supplies and for capacity building specific to supply chain management. The evaluation did not examine DELIVER performance, except as an externality affecting service delivery under FARA. Mid-Term Evaluation - USAID/Liberia Health Sector FARA 18 of a joint integrated supervision system. Through extensive discussion, FARA, the Pool Fund, and Global Fund agreed to support joint supervision that met the requirements of each respective donor while simplifying the working requirements for CHSWT and central MOHSW supervisors (at least in the three FARA-supported counties). This approach consists of a team of monitors from the central MOHSW, selecting and visiting a sample of facilities and catchment communities within each of the supported FARA counties, and employs mixed methods (checklists, interviews, observation, and document review) to assess performance against PBCs and the NHPP. The CHSWT and MOHSW program managers generally expressed positive view on the new ‘integrated supportive supervision’, while acknowledging that logistical challenges remain. To ensure coverage of a sufficient number of facilities, supervisory teams are often expected to travel continuously for two weeks or longer, and finding time in the schedules of qualified supervisors is difficult. The pool of qualified supervisors, who have been cross-trained to supervise multiple technical areas as required for integrated supervision, is still limited but gradually increasing through FARA-supported trainings. Routine Supervision of Facilities and Health Volunteers by CHSWTs and NGOs In addition to the introduction of integrated supportive supervision, the evaluation also reviewed performance towards routine supervision targets, designed to improve the capacity of health workers as well as perform quality assurance. Clinical supervision at the county level (conducted by the CHSWT and supporting NGOs) fluctuated from quarter to quarter; the percentage of facilities receiving three supervision visits during the quarter (the standard target for clinical supervision) ranged from around 50% to 70% during the review period. Informants note that seasonal fluctuations, including the number of non-working holidays in December and January, impact the number of facilities supervised. County Health Teams also consistently reported a need for more vehicles and logistics support for field visits, which they identified as a key constraint hindering more frequent supervisory visits. Transport and vehicle shortages were reported as a constraint for multiple program outputs, and may require further assessment by USAID to identify workable solutions with GOL and other donor partners. Community health services also play a significant role in access to key health interventions in rural areas, and the FARA supports management and supervision of the general Community Health Volunteers (gCHVs) who provide these services in their communities. Supervision of gCHVs is primarily a responsibility of the host health facility – and does not generally include CHSWT or MOHSW members. At the start of the FARA, facilities generally met the target of 1 supervision of each gCHV per quarter (99% in April-June 2012), and supervision has continued to cover the vast majority - around 90% - of gCHVs each quarter. While only a modest indicator of how effectively the MOHSW is supervising gCHVs, the results demonstrate that it is within their capacity to offer regular supervision at least once a quarter. Nonetheless, this is a low standard and may be insufficient to have much effect. Informants indicated that gCHVs have a high rate of attrition post training. More frequent supervision and other approaches to motivate gCHVs may be needed to improve the return on significant investments in training and Mid-Term Evaluation - USAID/Liberia Health Sector FARA 19 orientation. The actual level of performance should be re-examined after a more thorough review and discussions with the MOHSW, especially county and facility level managers. Facility Accreditation As to quality and structural issues in health services, the MOHSW’s annual Health Facility Accreditation Report provides the best information available on the conformity with standards for services, equipment and processes (proxies for quality) at public facilities and also covers a fair measure of private health facilities as well. In May 2013, the MOHSW conducted its second EPHS accreditation assessment. The introduction of the EPHS in 2011 marked a significant shift from the input-driven accreditation of the BPHS to a more quality-focused survey of implementation of the EPHS. This transition evolved from several years of experience implementing an accreditation survey, and the advocacy and support of RBHS to pilot and integrate measure for quality of services. The revised approach (and the need to place a greater emphasis on quality) is reflected in the assessment scores of both 2012 and 2013, where almost every facility received much higher scores on health systems compared to health services; the national averages were 38 percent for health services compared to 73 percent for health systems. In its 2013 Accreditation Report, the MOHSW acknowledged the need to continue to emphasize the provision of quality care, and adopted the 2013 results as a baseline for measuring future performance. Due to changes made to the assessment tools, it was not possible for this evaluation to directly assess performance based on changes between the two years’ scores. However, in 2013 the MOHSW established a new target of a 60 percent score in both health services and health systems for facilities to be considered as fully implementing the EPHS. Only Nimba County met the target overall, with a total overall score of 63 percent, though none of the three counties scored above 60 percent for health service indicators. The 2014 accreditation should provide an opportunity to better evaluate FARA facilities’ improvements against the MOHSW standards set by the EPHS. Table. MOHSW Facility Accreditation Scores, by County, 2012 vs. 2013 2012 2013 Bong County System Indicators 87% 70% Service Indicators 42% 43% Total score 64% 57% Lofa County System Indicators 81% 74% Service Indicators 39% 42% Total score 60% 58% Nimba County System Indicators 84% 79% Service Indicators 51% 49% Total score 65% 63% Mid-Term Evaluation - USAID/Liberia Health Sector FARA 20 Technical Management Support Technical support for management functions and service delivery simultaneously build the capacity of the MOHSW through mentorship while ensuring the ability of the MOHSW to perform essential functions to implement the FARA. Technical assistance for both health service delivery and health systems are provided through RBHS through a team of advisors within and external to the MOHSW, focused on specific technical aspects of implementation of the NHPP and EPHS. In addition to the support from RBHS, the FARA currently provides annually a $500,000 budget to contract high and mid￾level advisors within the MOHSW to manage and assist with implementation of the FARA and related health systems. Specifically RBHS support focuses on training of clinical and management employees of the MOHSW, and takes a specific role in systems development – including a major emphasis on the county health teams and health facility staff. The FARA advisors on the other hand, are teaching by example and application of management tools to the everyday operation of the central Ministry. Sustained support for these mutually reinforcing approaches should yield transformational improvements within the healthcare system in the long run. At present, FARA supports 17 high-skilled positions embedded within the MOHSW, including the FARA Project manager, PBF manager, financial manager, and M&E advisor. The majority of the FARA-funded positions work to strengthen systems in use throughout the Ministry and not just in “FARA counties.” This flexibility was deliberately built into the FARA to ensure it did not specify or limit the positions supported by the FARA, and allow the MOHSW to identify and address systems in need of additional support. Stakeholders at the MOHSW stressed that these FARA positions are very important to the Ministry. Both USAID and the MOHSW identified the technical advisors within the MOHSW and at RBHS as a significant investment and critical component in the operation of the FARA. USAID design and pre￾award documentation on FARA tends to highlight the risk-management benefits of placing high-level staff working in the MOHSW whereas reports from RBHS and the FARA emphasize service delivery and capacity building activities. When asked how the capacity is being built, several respondents cited the “modeling effect” of skilled and dedicated officers in the MOHSW, which is instructive to the workforce. This may prove particularly true given the fact that 47% of the Ministry’s workforce is less than 40 years old and 53% have less than five years of experience. Program Management Adoption of an Integrated Program Design There is widespread agreement that integrated approaches adopted by FARA have stretched the MOHSW capacity to better coordinate activities, communicate more effectively, and implement across programs. The weekly FARA management meetings and the monthly partner meetings have succeeded in bringing at least some of the “vertical” programs to the table and involved them in FARA program activities. A major challenge to meeting FARA requirements was the large degree of operational independence and administrative separation of programs receiving donor funds from a variety of sources, including Global Fund, GAVI, the Pool Fund, and other USAID programs. This challenge was largely overcome due to the insistence of the Minister of Health and Chief Medical Officer on broad and Mid-Term Evaluation - USAID/Liberia Health Sector FARA 21 active involvement of these program heads in FARA meetings and management. A number of respondents, including the Minister of Health, noted that the MOHSW struggled to meet all of the first quarter deliverables under FARA and did not receive the full reimbursement amount. This served as a wakeup call for MOHSW managers, and since then the MOHSW used the weekly FARA and monthly partner meetings to closely track the status of deliverables and identify issues requiring Ministry or sector leadership policy or program decisions. In addition, and with a few exceptions, FARA’s operative management and administrative structures are working in an integrated manner to more effectively plan, fund, and manage MOHSW priority health activities in the three counties, which have previously only received direction and support from vertical programs with little coordination. A few key observations related to the adoption of more integrated activities to meet FARA program deliverables are noted below:  The HMIS has undergone further integration and standardization to facilitate reporting of FARA results. It had previously been a challenge to integrate M&E from vertical programs, where supervision and indicators were specific to a single project or donor. The technical support of RBHS and other donor programs has been instrumental in providing the MOHSW with the means to meet FARA’s integrated reporting requirements, through negotiation with donors and programs to consolidate and review required indicators and produce new tools, guidelines, and training. Integrated HMIS reports lighten the burden of front-line workers and provide decision makers with more simplified reporting on activities of various programs. More integrated reporting literally ensures that MOHSW decision makers are ‘on the same page’, where all programs review the same monthly report, and opportunities for linkages as well as shared constraints are more visible.  FARA relies on GOL and MOHSW’s own systems of accounting and financial management, rather than USAID project-specific accounting. This was cited by many as an area where FARA promoted integration while strengthening the GOLs own systems in a more direct and cost￾effective way. Strengthened financial systems mitigate the risks of financial mismanagement of GOL resources for the FARA, and in the future should incorporate other sources of funding, allowing more efficient allocation of costs and identification of gaps.  Program integration and coordination also occurs through the MOHSW administration and oversight of the PBF contracts with NGOs. In addition to integrated program management (described above), administration of the PBF contracts required strengthening of the MOHSW procurement mechanisms, which now apply to all MOHSW contracts and procurement. Secondly, support to the PBF unit through FARA has led to continual improvements in the ability of the MOHSW to implement and monitor PBF, which is premised upon targeted, timely, and transparent processes to effectively incentivize performance. The recently updated PBF operations manual is an example of continued progress to better utilize incentives for emerging program priorities. PBF approaches piloted through RBHS and supported through the FARA have been extended to multiple counties, with promising results and positive feedback. Mid-Term Evaluation - USAID/Liberia Health Sector FARA 22 Management Mechanisms Routine mechanisms to manage the implementation of the FARA were established as deliverables for payment, and were also seen as part of capacity building efforts for “learning by doing”. Meetings and required reports are explicitly outlined to ensure managers are informed of progress and coordinate activities, and place responsibility for implementation squarely within the MOHSW. Once a month the FARA partners, including MOHSW (central and county), RBHS, FARA advisors (embedded within the MOHSW), contracted NGOs, and USAID managers meet to review progress and discuss issues related to implementation. The monthly partners’ meetings provide an opportunity to review performance against indicators and targets, and share findings of ongoing monitoring of project implementation. Holding face-to-face discussions on a monthly basis was cited by the majority of stakeholders as critical for the strong performance of the FARA to date. The meetings reflect a “best practice” as they review actual versus expected performance on 17 PBF indicators and findings from supervisory visits, and often lead to recommendations on how to address problems and overcome difficulties. Technical discussions and exchanges of views and ideas are routine at these monthly meetings. In addition, weekly meetings between USAID and MOHSW that were begun during the final FARA negotiations have continued, though the nature of discussions and participants have changed over time. The FARA Project manager embedded at the MOHSW leads the meeting attended by EPHS program leads, deputy ministers and the CMO, and external partners such as RBHS and USAID, among others. Interestingly, several other donors (not participating in PBF) complained that the FARA meetings were exclusive, took up too much of the senior management time, and made FARA just another vertical program. Senior MOHSW and mid-level managers, however, insisted that the weekly meetings were important, productive and essential. They credited the weekly meetings with immediate benefit to execution of FARA actions – which include extensive pre-contract and contract administration, development of policies, tools and guidelines, and promotion of inter-department communication and collaboration which was far from the norm. In all, the weekly meetings seem to be one effective way of tracking results, achieving coordination, sharing information and as one respondent put it, “helping to create a culture of performance in the MOHSW.” The MOHSW management structure has changed very little since the start of the program, and the MOHSW and FARA did not indicate that consensus was yet settled on any future changes such as the creation of an integrated PMU to oversee all donor-funded programs such as GFATM, GAVI, Pool Fund, and FARA. The MOHSW managers appear to believe that current arrangements for managing FARA have been discussed but the changes are not yet clear and unlikely to change in the very near future. Conclusions about Capacity Building and Health System Strengthening FARA management: The MOHSW has introduced and promoted new integrated management strategies to meet FARA project requirements for reporting program and tracking financial data. Strong leadership at the MOHSW has also contributed to improved coordination and involvement of largely vertical programs, an anticipated challenge to successful implementation of the FARA. In addition, the increased frequency of coordination meetings and the focus on reviewing program results (expected verses actual) Mid-Term Evaluation - USAID/Liberia Health Sector FARA 23 have reportedly strengthened accountability and communication between the many different stakeholders implementing the FARA. More specific and measurable capacity building plans were recently introduced, and performance should be reviewed more fully at the end of the current FARA agreement in 2015. FARA technical advisors: FARA advisors embedded in the MOHSW represent a small share of the program’s operational cost, but contribute a great deal to USAID development objectives – including both capacity building of the MOHSW and systems development (e.g., PBF contract mechanisms, M&E, etc.) but also with risk management of G2G assistance. One positive externality to the FARA skilled advisors – and for many of the RBHS advisors (COP, DCOP, Senior Training Advisor, M&E Director, PBF Advisor) is that their work is not limited to capacity or systems building in the three “FARA counties,” but rather, extends on national systems with recognized contributions in all 15 counties. Supportive supervision: From the central level, supervisory visits at the start of the FARA were conducted by vertical programs, and the introduction of integrated supervision has been well received by most of those interviewed. However, logistical challenges persist, and the quality of supervision is not well known. At the county level, CHSWT members reiterated logistical challenges with vehicles and seasonal fluctuations illustrated the relatively small number of staff available for supervision. At the facility level, while PBF targets are largely being met for supervising gCHVs, the target of supervision once per quarter is unlikely to be effective, and jeopardizes the substantial investment in training for community-based activities. C. Financial Management & Cost Savings The third and final key hypothesis to be addressed by this evaluation is whether or not the FARA would generate cost savings. Within the SOW, several key questions were asked: 1) What were the service delivery and technical assistance costs associated with the program before and after the introduction of the FARA? 2) What are the approaches and critical next steps that the mission should take with the MOHSW to determine if the FARA is producing good value for money? The mid-term evaluation was unable to address these questions directly though it does provide some insights into potential cost efficiencies. Related to these issues, the evaluation discusses aspects of the financial management of the FARA, particularly with regards to pre-financing by the MOHSW and its impact on program activities, and provides input on financial sustainability. Costs and Cost Efficiency Costing the services provided. USAID, using data from RBHS and MOHSW, carried out a detailed cost study prior to the introduction of the FARA. This provided a certain degree of confidence that cost estimates for supervision, PBCs to NGOs for service delivery, and other benchmark activities were realistic. To date, the payment amounts received for reimbursement have been generally accepted. However, when interviewed, a number of informants asked as to whether the cost uses didn’t allow for Mid-Term Evaluation - USAID/Liberia Health Sector FARA 24 some type of institutional savings. Others, including a few donors, noted their financial inputs were also contributing to producing the FARA outputs, but were not included in the cost calculations and reimbursement. Other than raising questions about the actual costs of implementation, it appears that the payments received from USAID reimbursing the GOL are not controversial and are generally accepted as fair. FARA management costs. In addition to the usual indirect (and non-quantified) cost of USAID (associated with all USAID activities), MOHSW and MOF officials are also expending time and effort on the implementation and oversight of FARA. Direct and dedicated management responsibilities are attributable to the embedded FARA advisors. Our inquiry into the long-term plan for these advisors reveals that senior MOHSW officials expressed a desire to put FARA and other donor-supported advisors on the ministry payroll and “normalize” their positions, but also admitted it is currently not feasible. This is due to budget limitations of the GOL and the limited budget for health, and also to the widely appreciated fact that existing GOL pay scales are not competitive with prevailing alternative employment, including private sector, NGOs and international levels of remuneration. To put the USAID investment in the Ministry’s HR in perspective, the FARA managers represent about 4.5% of FY12/13 total funds in the FARA. Moreover, both USAID and MOHSW officials cited the need for high-skill level positions to continue above GOL pay grades, and expressed the opinion that for some time to come, they would be necessary for FARA to function effectively and to succeed in capacity building of the MOHSW. Cost efficiencies. An in-depth analysis of cost efficiencies was outside the scope of this evaluation, but according to informed observers, the potential savings and cost efficiencies are likely from the following: a) use of GOL accounting and financial reporting procedures (and local staff) rather than external (USAID or USAID contractor) systems run by expatriate accountants and managers, b) integrated supervision saving fuel and vehicle costs, and c) fixed cost on provision of services and management, encouraging lower cost means of providing the necessary – and not incentives to run up the costs. These sound plausible and may indeed be in effect. This evaluation is unfortunately unable to confirm these with any hard evidence. Financial Management & Experience with Pre-Financing An assessment and review of USAID’s experience developing the FARA and the implications of the financing arrangement on Liberia’s country systems were performed prior to this evaluation. The assessment identified several key benefits of the FARA, as well as conditions and operational strategies to manage the programmatic risk inherent in pre-financing a program with known capacity gaps for implementation. Building on that assessment, this evaluation reviewed the experience with the financial aspects of the FARA to date. For USAID, pre-financing from the GOL continues to be advantageous. This was confirmed in the recent PFMRAF assessment of the MOHSW. While the MOHSW received better marks in the public financial management assessment than other ministries in Liberia—in great part due to the efforts by other donors to strengthen the MOHSW procurement and financial management systems through contracted Mid-Term Evaluation - USAID/Liberia Health Sector FARA 25 efforts with PWC, Ernst and Young, and the Pool Fund Manager—several critical issues were identified. The reimbursement aspect of the FARA design allowed USAID to shift fiduciary risk to the GOL, thus making use of country systems to implement the program possible. Not only is the use of host country systems for channeling donor assistance the expressed preference by the GOL, it also creates several opportunities to improve aid effectiveness. However, now underway, the GOL presents a different view. For the MOF and the GOL macro￾perspective, pre-financing has some disadvantages. From the MOF’s point of view, the FARA put a financial squeeze on the MOF (to meet all of its planned expenditures). MOF officials noted the difficulty of pre-financing millions of dollars for the health budget in a severely resource-constrained country when FARA reimbursements might be several months later. [The cash flow situation in the MOF is of course, magnified by the cash management basis on which the MOF operates. The MOF has very limited (and restricted) ability to take loans for its operations]12. At the extreme, this could be perceived as a distortion in country ownership and Liberian priorities whereby USAID is asking the MOF to take resources from domestic budget priorities—namely, energy, ports and transportation--in order to finance a donor priority--health. Managing relations among the MOHSW, MOF and USAID. A second issue raised by several respondents of the MOF is that they (MOF) do not receive adequate information on the results obtained from the GOL’s overall investments in health (FARA plus the GOL regular budget financing of the MOHSW). Senior and knowledgeable MOHSW officials are quick to dismiss this criticism as inaccurate and unfair. The MOHSW published not only an annual report – and presents it at an annual 3-day conference to which all GOL ministries, donors and health providers are invited, but also makes available technical reports and assessments on its web-site. It is also fair to say that the MOHSW is by far doing the most of any GOL entity to publicize and regularly report on its activities. Despite these challenges, there was a general acknowledgement that the interactions between the MOF and MOHSW may improve communication between the two line ministries for both FARA and non-FARA related budgetary issues. A USAID official involved with FARA summarized as follows: It is not clear to outsiders how day to day relationships have fared at various levels between the MOHSW and the MOF, but it is clear the FARA is forcing the MOHSW to deal with the larger GOL systems. The MOHSW has expressed strong interest in having an advance – liquidation process set up with USAID instead of MOF. This has not yet happened. The MOHSW does appear to have improved their ability to make sound fiscal justifications to the MOF, and the FARA has allowed them to get a better understanding of the costs associated with service delivery. In sum, FARA’s existing arrangements have and should continue to improve the communication between the MOHSW and the MOF. Timeliness and Financing Impact on Program Activities. With the exception of a delay of several months for the MOF to provide the MOHSW with an initial tranche of Year 2 funds for FARA activities, 12 A MOHSW finance official cautioned that the HIPC (Highly Indebted Poor Country) agreement for Liberia had ended in 2010, but that similar limitations on GOL borrowing were still in effect. Mid-Term Evaluation - USAID/Liberia Health Sector FARA 26 the MOHSW has received payments from the MOF which are adequate to carry out its management and monitoring role, fund FARA – specific activities, including the PBF contracts and staff payments. The table below summarizes payments (disbursements and reimbursements) to date under FARA. FARA Disbursements from MOF to MOHSW and USAID to MOF, for Year 1 and Year 2 as of July 24, 2013 Source: FARA Accounting Office, MOHSW ($US Dollars) Year 1 Year 2 Total Receipt* 4,499,000 14,000,000 18,499,000 Payment** 1,630,262 9,820,849 11,451,111 Accrued - 1,941,681 1,941,681 Balance*** 2,868,737 5,106,206 5,106,206 * Receipts are MOHSW receipts from MOF **Payments are from USAID to the MOF ***Balance is cash in the MOHSW account As shown in the table above, as of early July, the MOF reports they had pre-financed $18.5 million for the FARA, and been reimbursed for $11.5 million. A lag in reimbursement (between MOF pre-financing to the MOHSW, and receipt of USAID reimbursements to the MOF) is due to several factors. Upon receipt of funds from the MOF, MOHSW conducts the FARA activities (at least one quarter), then must gather and submit the deliverables (up to 60 days), and verify the deliverables (up to 15 days). Upon receipt and review of deliverable and the quarterly report, it can time for USAID to process the reimbursement (in principle up to 45 days)13. As the summary shows, the process can result in a six￾month lag for the MOF to receive FARA reimbursement payments. Conclusions about Financing and Cost Savings We recognize that this is an important issue with sharply differing perspectives, and the willingness of the GOL to accept the pre-financing arrangement is at the core of the current FARA. Based upon the evidence and reporting to date, while individual stakeholders, primarily within the MOF, expressed concerns, it seems that for now at least, the pre-financing arrangement is working. We recommend that USAID and the MOHSW continue to confer regularly and review their experience with the FARA over the longer term, while continuing to work towards GOL support for pre-financing without threatening or politicizing funding in the GOL budget. When/if support is withdrawn, MOHSW and USAID may consider a discussion with other donors or lending institutions (e.g., World Bank). Continued and increasing GOL support to the health budget is necessary for Liberia to meet its current commitments to improved health of Liberians. Secondly, while the evaluation was unable to assess whether the FARA generates cost savings, there was evidence that costs are in line with expectations, and cost-savings in some specific areas, such as use of MOHSW financial systems and fixed procurement costs, may indeed exist. Given the critical importance 13 In fact, the OFM at USAID has endeavored to “quickly turn around FARA requests for reimbursement,” and gave 15 days as the average time for USAID to process FARA reimbursement requests (to which must be added several days for check clearance as well). Mid-Term Evaluation - USAID/Liberia Health Sector FARA 27 of the financing arrangement to the success of this project, cost efficiencies and a more detailed exploration of the ‘costs’ of the FARA mechanism on program outcomes may be worthwhile. Discussion on Financial Sustainability The major source of funding for the MOHSW – and for primary health care (the EPHS) is from donors. Up to 70% of the funds available to the Ministry come from external sources. Health Financing from the Treasury of Liberia (MOF) and general revenue account comprise the 2nd largest source of funds for the EPHS, while the MOF is also directly providing about 20% of its total health budget to major public (and mixed) hospitals. Based upon recent experience, the MOF anticipates annual general revenue increases on the order of 5-7% per year. While increasing, revenue inflows are very low. The MOF overestimated revenues for the current FY12/13 GOL fiscal year with a $642 million budget when revenue inflow was closer to $435 million. Recently the MOF issued a revised budget of $500 million. Many competing priorities vie for that $500 million, and 60 – 70% of the budget is employee compensation (benefits and salaries), which is prioritized. The most important result of this analysis is that the non-incremental increase in GOL funding for health – expected to climb from an 8% share of total GOL expenditure by 2014, is unlikely to increase beyond its current share. Further, the GOL in its national development plan assigns greatest priority to Transport, Education and Energy sectors. Thus, health is not likely to obtain substantial increases in funding from general GOL sources. Strong pressure from health workers for higher pay envelopes and benefits, and the intensifying pressure from donors to absorb health workers in the MOHSW budget have severely strained existing and anticipated funding for health. A huge challenge facing the MOHSW is the demand for higher compensation by the health workers. A nursing strike occurred the first week of July (during the evaluation) and it affected all of the Ministry’s facilities. The current system involves regular pay plus a small incentive. The current payroll is some $30 million – of which $21 million (FY13/14 expected level) is provided from the GOL Treasury (via the MOF). Absorbing workers from the current donor programs (Global Fund, EC, former pooled fund areas) would raise the personnel envelope to $54 million – quite a bit more than the entire MOHSW budget (consultant’s estimate $28 million). For the MOHSW, getting workers off of donor incentives and on to regular payroll is an immediate priority. It is likely that the Ministry is going to have to phase in any salary increase over a number of years, and not try to do it all in one or even two years. The two-fold pressure on the MOHSW to absorb into the regular budget health workers currently funded by donors (such as GFATM, USAID, EU) will only intensify along with pressure on wages from the workforce. The MOHSW and its partners will need to investigate alternative financing schemes to address these pressures. Increases in the wages for health workers must be considered in line with overall trends in the labor force and national economy. Projected costs of the MOHSW’s national health plan to improve Liberians’ access to quality health services will continue to increase faster than the GOL budget and associated general revenues. The high levels of out-of-pocket spending (accounting for 35% of total health expenditure), suggest that Liberians Mid-Term Evaluation - USAID/Liberia Health Sector FARA 28 are spending a fair amount of their disposable income on health services and products (e.g., at pharmacies), and that these expenditures tend to grow as income increases (some elasticity). While not specific to the FARA, the MOHSW and USAID should vigorously explore options for new insurance mechanisms, which will attract investments to the health sector and fund the private as well as potentially public expansion of services. During the period the evaluation team was in Liberia, a team of social insurance experts was in Monrovia and leading discussions with the MOHSW, other GOL ministries and technical advisors on possible paths to creating and expanding social insurance. V. CONCLUSIONS AND KEY FINDINGS  For the (EPHS) health services monitored by Performance Indicators (PI) in Bong, Lofa, and Nimba Counties, the evidence suggests that in general, the reach of service delivery did not recede in the first year of FARA implementation.  The review of performance data from the three supported counties (albeit only three quarters reported since FARA began) and the results of more than twenty interviews suggest that the FARA is on track to contribute toward the first IR of the Mission’s Development Objective 3 for Health (IR 3.1: Increased utilization of quality services. On the other hand, we have no definitive finding or are aware of much evidence to comment on FARA contributions and progress to date on IR 3.2 (More responsive services through effective health system decentralization) and IR 3.3 (Increased financial sustainability of services).  FARA managers embedded in the MOHSW represent a small share of the program’s operational cost, but contribute a great deal to USAID development objectives in health and to implementation and oversight of the FARA– including both capacity building of the MOHSW and systems development (e.g., PBF contract mechanisms, M&E, etc.) but also with risk management of G2G assistance.  One positive externality to the FARA skilled advisors – and to RBHS advisors (COP, DCOP, Senior Training Advisor, M&E Director, and PBF Advisor) as well, is that their work is not limited to capacity or systems building in the three “FARA counties,” but rather, extends on national systems with recognized contributions in all 15 counties. This may also be considered a “best practice” as USAID investments in health are leading to improved national capacity and systems, and go well beyond the three focus counties.  A consensus view is that the FARA would not be successful and leading to positive changes at all levels of the MOHSW if strong technical support were not in place. The RBHS project, implemented by JSI and partners from 2008 until 2014, provides an essential counterpart to FARA. This evaluation recognizes that the direct G2G funding of FARA stretches MOHSW capacity and only increases the requirements for credible health statistics and supervision. Capacity building in M&E, supervision and planning have seen the greatest improvement to date. Mid-Term Evaluation - USAID/Liberia Health Sector FARA 29  The limited timeframe and information available for this evaluation did not allow for an opinion on whether cost efficiencies have been achieved.  The two-fold pressure on the MOHSW to absorb into the regular budget health workers currently funded by donors (such as GFATM, USAID, European Union) will only intensify along with pressure on wages from the workforce. The MOHSW and its partners will need to investigate alternative financing schemes to address these pressures.  Regarding the demands for higher compensation for health workers, it is important to reaffirm that the health sector does not function in vacuum but is a dynamic and integral part of the Liberian economy. A revision of the wage structure of the public sector labor force needs to be in lockstep with overall trends in the labor force and the economy. Lessons Learned  The pre-financing aspect of the FARA design allowed USAID to shift fiduciary risk to the GOL, thus making use of country systems to implement the program possible. Not only is the use of host country systems for channeling donor assistance the expressed preference by the GOL, it also creates several opportunities to improve aid effectiveness. The commitment by the GOL to pre￾finance the FARA made use of GOL systems possible.  One of the FARA advisors at the MOHSW noted it would have been advantageous if USAID could have started discussions earlier with the MOF. Of course, the sudden change in Finance Ministers, which occurred precipitously in the summer of 2012, and the extensive changes (personnel, policy) made by the incoming Minister, were external shocks that could not have been anticipated by USAID.  A key lesson noted in the Mission’s CDCS, has been that, because of Liberia’s weak institutional capacity, and particularly weak inter-ministerial mechanisms for information sharing and coordination, progress on FARA requires much more extensive direct engagement between USAID staff and Liberian counterparts, particularly in the initial stages of the process. The evaluation findings confirm this lesson.  One positive and commendable example of donor cooperation is in the joint funding of integrated supportive supervision. As noted earlier, the vertical programs had been funding supervision of facilities in a non-coordinated way with “silo funding.” To move ahead with integrated supervision, sufficient funding support needed to be arranged. After some extensive discussions, FARA, the Pool Fund and Global Fund agreed to support joint supervision (at least in the three counties). This sets an important precedent and is an example of a “best practice” in donor coordination. Mid-Term Evaluation - USAID/Liberia Health Sector FARA 30  An example of successful skill building is evident in the Ministry’s management of PBCs. From 2009 until the end of 2012, the RBHS project managed PBCs receiving USAID funds. Over that period, the Ministry was an active partner in solicitation, review and award of PBF grants. The tools and methodology were documented by RBHS, and the MOHSW was able to build upon this in-country shared experience to adopt and adapt PBF tools and procedures. VI. RECOMMENDATIONS The evaluation has identified areas where the FARA is functioning well as well as areas that need attention. Listed below are the evaluation’s summary recommendations, aimed at helping strengthen the FARA and any follow-on activity. 1. The FARA “experiment” to use G2G financing measures as a component of a major USG health assistance program, has met its objectives, both explicit and implicit, and should be continued. The benefits not only include greater GOL ownership and understanding of USAID assistance, but a greater appreciation for USG willingness to put confidence and resources in the MOHSW. 2. USAID and the MOHSW should continue to confer regularly and see over the longer term, whether the GOL can eventually provide pre-financing without prejudice in funding its GOL budget allocations or some other arrangement can be made – perhaps in discussion with other donors or lending institutions (e.g. World Bank). 3. Go slow on decentralization to County Health Teams. In the past twelve months, RBHS carried out capacity building assessments of three CHSWTs. In general, the assessments show that capacity of county health teams has not kept pace with the Central MOHSW, and the teams are generally weak. Many county staff do not meet minimum required qualifications, while absenteeism and attrition are also concerns. From a wider governance point of view, it is also true that progress in achieving Civil Service reforms and government-wide plans for decentralization are going slowly. Thus, the evaluation team recommends further decentralization of FARA management be paced with evidence of demonstrable progress in enhancing the civil service and rigor of operational and financial systems. 4. Advocacy for Family Planning. FARA and USAID, as well as other FP donors such as UNFPA, should meet together with the MOHSW and review their investments in demand creation, as well as supply and distribution networks for FP services (since stock-outs of some essential commodities are in evidence). The lackluster results in FP to date suggest approaches are needed that build upon effective communication and outreach to improve the “image and information” of FP services, as well as access to affordable, accessible and quality services. Private sector involvement and focus on youth programs should be given special consideration. As the Mission’s CDCS states, over the medium- to long-term, a game-changer for development prospects in Liberia is the very high population growth rate. The population growth rate in Mid-Term Evaluation - USAID/Liberia Health Sector FARA 31 Liberia is currently 3.1%, and there has been little progress in reducing fertility in the last thirty years. 5. The MOHSW and USAID should vigorously explore options for new insurance mechanisms, which will attract investments to the health sector and fund the private as well as potentially public expansion of services. As noted in the report, the donor community covers approximately 40% of total health spending, and private out-of-pocket expenditures account for almost 35% of total health expenditures. The GOL accounts for just 25%. 6. County health and social welfare teams (CHSWT) are in need of help with logistics – particularly transport. Without recommending per se that USAID purchase replacement vehicles for the ambulances and 4WD SUV vehicles provided to all 7 RBHS counties in early 2009, it is clear that the fleet of SUVs and motorcycles has been greatly depleted in all 3 FARA counties, and if not replaced soon, will lead to lower performance on outreach and management targets. Plans for new vehicles or other transport means (e.g., allowances) need to be developed and applied in the package of resources for the counties. 7. Put priority on Reform of the National Drug Service. All MOHSW health facilities and the major public hospitals count on the NDS for procurement and distribution of drugs and supplies. For more than three years, the MOHSW has been on the cusp of a major organizational revision of the NDS, which has not gone forward. The NDS currently is in political and organization limbo, and has not had a clear legal status since the NGO NDS was dissolved in 2010. USAID under the DELIVER project has reserved funds that with a comparable contribution of the GFATM to build a new central warehouse, to be managed by the NDS. USAID and the GFATM should use the warehouse as leverage to encourage a full reorganization of the NDS. Further, USAID should support an expert analysis on privatization and contracting out as one option for the GOL to consider in reorganizing the NDS. 8. USAID, and possibly other donors, should give high priority to support the development of an effective HR and personnel management capacity within the MOHSW, including accurate database covering all employees, and regular survey and reporting on employee attitudes and morale, attrition rates, and individual training files. This was foreseen in the NHP of 2007-2011, but has sadly not made much progress. It is a key constraint now to improving quality of care as well as management of the Ministry at all levels (central, county, facility). 9. USAID technical support should assist the MOHSW HR department with periodic surveys of worker morale, attitudes and reactions to new policies. The PBF system, for example, was developed under a hypothesis that greater compensation for greater output should be rewarded. While FARA and the MOHSW (with help from RBHS) are monitoring the performance outputs, it is not monitoring the intervening variable – the mindset and motivation of the affected workforce. Mid-Term Evaluation - USAID/Liberia Health Sector FARA 32 10. USAID should distribute meeting notes and final reports from RBHS relating to technical programmatic issues to other donors and international agencies, beyond the current distribution list (CHAI and Global Fund). 11. A new technical assistance project – follow on to RBHS – should be developed and brought on line by September 2014. The new project would focus on developing systems at the MOHSW, with initial rollout to USAID focus counties, and continued investment in capacity building in skill areas in greatest need. It should be understood that the level of investment required is likely to be significant, but potentially less than the current LOE under RBHS. Savings from the more focused technical support effort could be used to expand the number of health facilities receiving assistance under FARA or a follow on to FARA. 12. If possible, USAID should endeavor to ensure all health facilities in a focus county are receiving adequate support. Financing inequities within a country will only serve to aggravate the labor management issues faced by the MOHSW, and create a greater risk of theft and malfeasance (particularly in those areas with lower levels of support).