May 2017 This document was produced at the request of the United States Agency for International Development. It was prepared independently by Social Impact under the Liberia Strategic Analysis activity. Liberia Fixed Amount Reimbursement Agreement Final Evaluation: Health Outputs Report 2 3 LIBERIA FIXED AMOUNT REIBURSEMENT AGREEMENT FINAL EVALUATION: HEALTH OUTPUTS REPORT A USAID/LIBERIA GOVERNMENT-TO-GOVERNMENT FUNDING MECHANISM FOR SUPPORTING IMPLEMENTATION OF THE NATIONAL HEALTH PLAN MAY 2017 Liberia Strategic Analysis Contract No: AID-669-C-16-00002 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. 4 5 TABLE OF CONTENTS 1. EXECUTIVE SUMMARY.........................................................................................9 2. PROJECT BACKGROUND...................................................................................14 3. PURPOSE & QUESTIONS....................................................................................16 3.1 Purpose of the Study................................................................................................................................... 16 3.2 Evaluation Questions.................................................................................................................................. 16 4. EVALUATION METHODS & LIMITATION ......................................................18 4.1 Methods......................................................................................................................................................... 18 4.2 Limitations..................................................................................................................................................... 20 5. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS ..........................21 5.1 Findings and Conclusions for Question 1............................................................................................... 21 5.2 Findings and Conclusions for Question 2............................................................................................... 29 5.3 Findings and Conclusions for Question 3............................................................................................... 30 5.4 Recommendations for FARA 2................................................................................................................. 39 ANNEXES ...................................................................................................................41 Annex I: Evaluation Statement of Work ....................................................................................................... 41 Annex II: Interview Respondent List ............................................................................................................... 48 Annex III: Data Collection Instruments.......................................................................................................... 51 Annex IV: Analysis of Evaluation Question 2: ADS Compliance.............................................................. 54 Annex V: Disclosure of any Conflicts of Interest.......................................................................................... 56 6 TABLE OF FIGURES Figure 1: Average Yearly Change in Health Stream Indicators, 2011-2015, FARA vs. Non￾FARA Counties........................................................................................................................................ 22 Figure 2: Average Yearly Change in Health Stream Indicators, 2011-2015, FARA vs. Non￾FARA Facilities within Bong, Lofa, & Nimba Counties.................................................................... 23 Figure 3: Average Yearly Change in Health Stream Indicators, 2011-2015, Among FARA Facilities in Bong, Lofa, and Nimba Counties.................................................................................... 25 Figure 4: Key Infrastructure and Equipment Indicators of Facilities Visited (n=18) ................. 27 Figure 5: FARA vs. Non-FARA Facilities WASH Comparison...................................................... 28 Table 1: Recommendations for GOL.................................................................................................. 12 Table 2: Recommendations for USAID .............................................................................................. 13 Table 3: KIIs Conducted at the National Level................................................................................. 19 Table 4: KIIs and FGDs Conducted at the County Level............................................................... 19 Table 5: Evaluation Limitations and Risk Mitigation Strategies...................................................... 20 Table 6: Key FARA Health Stream Indicators................................................................................... 21 Table 7: Average yearly change in health stream indicators, 2011-2015, among FARA and non-FARA facilities in Bong, Lofa, and Nimba counties.................................................................. 26 Table 8: FARA County Stewardship Progress towards Targets, 2015........................................ 30 7 ACRONYMS AAP ACT ADS ANC CBA CHDC CHDD CHO Accelerated Action Plan Artemisinin-Based Combination Therapy Automated Directives System Antenatal Care Cost-Benefit Analysis Community Health Development Committee County Health Department Director County Health Officer CHT CHS CSH CYP County Health Team County Health Supervisor Collaborative Support for Health Couple-Years of Protection DHIS-2 DHO EPHS EPI District Health Information System - 2 District Health Officer Essential Package of Health Services Expanded Program for Immunization FARA Fixed Amount Reimbursement Agreement, 2011 FARA 2 FP Fixed Amount Reimbursement Agreement, 2016 Family Planning FY Fiscal Year G2G GFATM HMIS HMM HSS ICT IEC IGCE IP Government-to-Government Global Fund to Fight AIDS, Tuberculosis and Malaria Health Management Information System Hybrid Management Model Health System Strengthening Intermittent Preventive Therapy Information, Education, and Communication Independent Government Cost Estimate Implementing Partner GOL Government of Liberia JSI LDHS John Snow Inc. Liberia Demographic Health Survey 8 LLITN LLM LMIS LSA M&E MCH Long-Lasting Insecticide-Treated Nets Long-Lasting Methods Logistics Management Information System Liberia Strategic Analysis Monitoring & Evaluation Maternal and Child Health MFDP MOH Ministry of Finance and Development Planning Ministry of Health NGO NDS OIC PBC Non-Governmental Organization National Drug Service Officer In-Charge Performance-Based Contract PBF PMI Performance-Based Financing Presidential Malaria Initiative PPCC PPE RBHS RDT Public Procurement and Concession Commission Personal Protection Equipment Rebuilding Basic Health Services Rapid Diagnostic Test SARA SI Service Availability and Readiness Assessment Social Impact SOW Statement of Work TTM Trained Traditional Midwives USAID U.S. Agency for International Development VFM WASH Value for Money Water, Sanitation, and Hygiene 9 1. EXECUTIVE SUMMARY USAID/Liberia engaged the Liberia Strategic Analysis (LSA) activity to conduct a performance evaluation of USAID’s health sector Fixed Amount Reimbursement Agreement (FARA) with the Government of Liberia (GOL). The FARA supported the provision of health services in Bong, Lofa, and Nimba counties from September 2011 through December 2015. Through the FARA mechanism, USAID made available US $42 million to the GOL in support of the National Health and Social Welfare Policy and Plan and, in particular, the delivery of the Essential Package of Health Services (EPHS) at the primary healthcare level. The FARA was developed in the context of a gradual transition away from the post-conflict setting, with numerous international health actors, to a government-managed health sector, building on the incremental improvements made in health indicators and health systems strengthening since the end of the Liberian civil war. The FARA was designed to build the Ministry of Health’s (MOH) capacity to manage the health systems necessary to successfully implement the FARA, as well as to directly support health service delivery through awarding and managing performance-based contracts with NGOs. The theory of change underpinning the mechanism was that, if USAID health funds were provided directly to the GOL for implementation, it would result in more effective and efficient long-term development by reducing transaction costs and strengthening local stewardship and capacity through a learning-by-doing approach. The purpose of the FARA Performance Evaluation was to validate some of the assumptions on which the FARA theory of change was based and to inform decisions related to the follow-on phase of FARA, known as FARA 2, which was signed in January 2016. While the overall FARA Performance Evaluation will assess the achievement of results at the end of the implementation period, this Health Output report is intended to answer a narrower set of questions that will inform the overall evaluation as well as to serve as a resource for USAID and the MOH. Specifically, the Health Output component focused on the following questions: 1. What were key health outputs for the following health streams, comparing FARA to non￾FARA counties? a. President’s Malaria Initiative (PMI) b. Expanded Program for Immunization (EPI) c. Family Planning (FP) d. Maternal and Child Health (MCH) e. Water, Sanitation, and Hygiene (WASH) 2. Were the results achieved commensurate with the level of these streams of funding? 3. What MOH and CHT capacity gaps pose the greatest barriers to health outputs? A final summary report entitled, “The FARA Performance Evaluation Report,” accompanies this report and provides comprehensive answers to the overall evaluation questions. Methodology To answer each of the Health Output questions described above, LSA conceived a tailored methodology (see section IV. Evaluation Methods and Limitations) that draws upon primary and secondary data sources and appropriate analysis techniques, including a desk review of relevant documents, interviews of key informants, analysis of health program data, and field site visits. For quantitative data linked to health stream outputs, we reviewed Health Management Information System (HMIS) and District Health Information System (DHIS-2) data sources. For qualitative data collection, the evaluation team carried out KIIs and FGDs with national, county, and facility-level actors including Ministry of Health (MOH), County Health Teams (CHT), Officer in Charge (OIC), and facility staff and clients. The team conducted site visits to 18 clinics in 4 counties, 10 FARA and 8 10 non-FARA clinics. Facilities were chosen for location convenience and stratified by the two highest and two lowest performing facilities according to national facility scorecards. Consistent with USAID policies, the team applied a gender perspective to its work, involving, among many aspects, a balance of female and male participants in its data collection activities. Over the course of the evaluation, we conducted 19 KIIs at the national level, and 53 KIIs and 24 FGDs at the county level. Data Limitations This evaluation is constrained by several important data limitations. For quantitative indicators, LSA based its analysis almost exclusively on HMIS and DHIS-2 data. This makes a large and important assumption that the data are accurate. To mitigate the risk of inaccurate data, the evaluation team carried out a data audit in four randomly selected facilities which concluded that data accuracy was high. Other limitations include the fact that small sample sizes, myriad observable and unobservable biases, and contamination effects limit our ability to make confident assertions about the effectiveness of FARA. Findings for question 1 on health stream outputs were based mostly on reviews of HMIS data at the county and facility level which may have been influenced by the health-related effects of the Ebola virus epidemic, the influx of county-specific funding and infrastructure post-Ebola, or similar other USAID funded health systems strengthening projects in FARA and non-FARA supported counties. Given that there was no recent quantitative data available for WASH indicators, findings on WASH were based wholly on site visits with a small sample of 18 facilities across 4 counties. We worked to mitigate these data limitations by relying heavily on qualitative data collection as a means to supplement, triangulate, and clarify our quantitative findings. Findings and Conclusions Evaluation question 1: What were key health outputs for the following health streams, comparing FARA to non-FARA counties? Quantitatively, at the national level, FARA counties across the board do not perform better than non-FARA counties. Non-FARA counties performed better than FARA counties in 6 of 7 indicators, with the exception being the second IPT dose given to pregnant women. Within FARA counties, FARA facilities performed nearly the opposite as they did at the national level, doing better over time compared to non-FARA facilities in 6 of 7 indicators tracked, with the exception of couple￾years of protection (CYP). Though there are many limitations to this analytical approach as well, quantitatively comparing FARA and non-FARA facilities within the same counties is a more accurate measure given that those served by the two facility types are assumed to be more or less similar in health burden and population demographics, and might be affected equally by any confounding factors. Among health stream indicators, FARA facilities saw the largest average increases over time in women receiving second IPT doses (7% average increase per year), cases of malaria diagnosed by RDT or microscopy (7% per year), CYP (27% per year), and normal deliveries carried out at a facility by a skilled attendant (16% per year). Conversely, FARA facilities decreased on average each year in malaria cases treated by ACT (1% decrease per year) and 3 doses of pentavalent given to children under one years old (1% decrease per year). Further, FARA facilities saw relatively small increases (3% per year) in pregnant women attending 4 ANC visits accompanied by a trained traditional midwives (TTM). For WASH indicators, site visits among 18 clinics showed FARA facilities to outperform or perform equally to non-FARA facilities among all WASH indicators. Among all key FARA health stream indicators, FARA facilities perform substantially worse than non￾FARA facilities in FP (utilizing the CYP measurement); non-FARA facilities registered almost twice as much gain over time in CYP compared to FARA facilities. In general, FARA does not track a sufficient number of clinical level indicators to make general statements about effectiveness of FARA on specific health stream outputs. 11 For a contracting model, it remains unclear whether the contracting-in model employed by 7 facilities in Bong results in better health outcomes than a contracting-out model. Bong County FARA facilities outperformed Lofa and Nimba FARA facilities in 4 of 7 indicators and performed better than Bong non-FARA facilities in 5 of 7 indicators. However, they did not perform better on more indicators when compared to analyses conducted between FARA and non-FARA facilities in Lofa and Nimba counties. Evaluation Question 2: Were the results achieved commensurate with the level of these streams of funding? After several attempts, the evaluation team was ultimately unsuccessful in obtaining expenditure data for specific health outputs. Though the MOH provided financial expenditures, these were tracked at a level higher than health outputs, which were rolled up into more general cross-cutting line items, such as supervision or technical and management support. This financial reporting did not allow for us to measure the expenditures by FARA health output specifically. Evaluation Question 3: What MOH and CHT capacity gaps pose the greatest barriers to health outputs? Facilities in FARA counties are most lacking in capacity in quality assurance (facilities achieving at least 60% in Quality Assurance Assessment and score on client satisfaction surveys) and stock-outs of tracer drugs and HMIS quarterly reporting. Facilities receive the best scores on supervision indicators, (% of CHT/NGO submitting timely and complete quarterly report to MOH and % of facilities that received at least 1 joint Integrated supportive supervision visit in last quarter) as well as community ownership measures (three CHDC meetings held during the quarter and % of CHVs who received at least 1 supervision visit in last quarter). According to facility scorecards, when comparing contracting models within Bong County, the contracting-out model carried out in Africare facilities slightly outperformed the contracting-in model in CHT facilities. Furthermore, Africare’s contracting-out model scored substantially better than the CHT’s contracting-in model for one supervision indicator (% of CHT/NGO submitting timely and complete quarterly report to MOH) and for one medicines and supply chain indicator (% of facilities submitting timely LMIS report to MOH during the quarter). Results for the contracting model are limited due to the small sample size. Facilities lacked access to consistent running water, staff lacked knowledge around waste management, and there was a general lack of maintenance of medical equipment. At the MOH and CHT levels, data were not effectively disseminated or used. While data reporting procedures were generally followed and data accuracy for HMIS was high overall, the MOH admitted that it lacked strategies to operationalize data, and that data reporting at the district level was especially weak. There was a lack of defined roles and responsibilities for MOH/CHT supervision. Roles and responsibilities for supervision between the MOH and IPs were not well defined resulting in little to no involvement from District Health Officers (DHOs) and a weak third-party verification system for performance of contracting-in facilities. The FARA performance-based finance (PBF) incentive scheme appeared to have a highly positive effect on staff motivation, performance, and attendance. Non-FARA clinics that do not benefit from these schemes had worse staff and performance related indicators. This could signal that staff in general were not motivated to achieve high performance at clinics, perhaps due to a lack of communal purpose/goals and/or low salaries. As a result of PBF incentives, staff at FARA clinics have increased leadership and governance over their resources, and new forums have been created for community stakeholders to come together and discuss the health issues in their community. Non￾FARA facilities and staff that do not benefit from incentives and may have an arguably weaker sense of pride and ownership and trust of the community, limiting performance and service utilization. 12 Non-FARA facilities did not benefit from integrated, joint supervision by MOH/CHT and IPs, resulting in subpar administrative capacity and performance. While non-FARA clinics received periodic visits from MOH and CHT, they could benefit from increased, targeted external support, particularly in terms of integrated supervision, mentoring and coaching, and management/administrative training. Lack of funds and a weak supply chain might have affected the quality of service provision at facilities. Non-FARA clinics were observed in site visits to have experienced more frequent stock-outs than FARA facilities, possibly due to weaker commodity logistics systems than IP-supported FARA facilities. Recommendations Table 1: Recommendations for GOL Theme Recommendation Health Services/ Outputs As stock-outs continue to be a problem for non-FARA facilities, consider integrating other USAID interventions that focus on supply chain strengthening and commodity financing for non-FARA clinics. Study and/or replicate best practices of IPs and NGOs in terms of commodity logistics systems, and transfer these skills to CHT. Consider using a review unit with the MFDP to provide independent third￾party verification, and potentially in a joint manger with the MOH, of health facilities. Assess the difference in FP interventions and supply and demand of FP methods between FARA and non-FARA facilities and catchment areas to help explain the large imbalances in CYPs. WASH and Infrastructure For facility level improvements and capacity building: ensure running water and operational separate delivery and post-partum rooms at all facilities. Improve staff awareness about incinerators. Supervision Support effective and regular supervision in non-FARA counties for improved service delivery. Study and/or replicate best practices of IPs and NGOs in terms of supportive supervisory and mentorship systems, and transfer these skills to CHT. Ownership Encourage staff to take ownership of maintenance of equipment as their responsibility. This could be achieved, for example, by incorporating benchmarks/targets for medical equipment maintenance into PBF performance standards. Data Devise and integrate a strategy that links data collection, analysis, and dissemination with program management and implementation into the National Health Policy and Plan. Health Stream Funding Allocation Continually reinforce injection safety precautions as part of systemized Universal Precautions in all healthcare facilities. Provide additional training and annual refresher training to facility staff on Family Planning counseling and technical training for longer lasting methods. Reporting Prioritize and develop a district-level capacity building program, including mentoring, to strengthen district level data collection and reporting (including expenditures). Consider development of a FARA management dashboard providing more detailed analysis of performance (current, comparison with same period 12 months ago, against targets, etc.) Link to financial system for more detailed analysis by USAID. Track health stream output spending at the line-item level to allow for easier tracking of spending, increased transparency, and better programmatic management decisions. 13 Capacity Gaps Expedite decentralization of the healthcare delivery system in the areas of planning and implementation. These efforts could directly improve the health system by providing increased training and management skills to the DHOs. The DHOs’ roles and responsibilities should be strengthened and supported by the CHT and MOH Central. By empowering the DHOs, OICs, facility staff, and the community, they will feel a greater sense of control and take on more responsibility to ensure that they are getting the best care from their local healthcare system. Table 2: Recommendations for USAID Theme Recommendation Health Services Current FARA clinical indicators do not cover a wide enough range of health outputs. Revise FARA indicators to reflect more clinical versus administrative and stewardship indicators. Given that quantitative and qualitative evidence suggests that FARA facilities outperform non-FARA facilities in the majority of health outputs among comparable populations, collaborate and introduce the FARA mechanism to other partners for a holistic health system strengthening approach. Supervision Continue support for supervision-related activities such as training and capacity building to increase and maintain effectiveness in service delivery. Consider funding IP-MOH joint supervisory activities. Health Stream Funding Allocation / Reporting Ensure all USAID and MOH staff working with the FARA program understand the current USAID Assistance Automated Directives System (ADS) 200 policies, guidance, and compliance requirements for all aspects that FARA supports. Consider increased participation from the MFDP to provide independent verification of FARA performance. This, or an alternative body, preferably within the GOL system, would be required if the contracting-in mechanism for service delivery is expanded. Establish a performance framework/logframe for FARA 2 and include WASH indicators, VFM indicators and any additional indicators, if any, recommended and agreed by the performance evaluation. Support development of a management dashboard including provision of technical assistance (the USAID GMS project has extensive expertise) to support the MOH. Require disaggregated financial reporting tying inputs to outputs and outcomes for better analysis of economy, efficiency and effectiveness. Require tracking of health stream output expenditures at a line-item level. Capacity Gaps Decentralize health system strengthening and support more supervision capacity building at the district and facility levels for sustainability. Continue to train new, and build capacity of existing, FARA staff to ensure local capacity and sustainability of the FARA mechanism programming. 14 2. PROJECT BACKGROUND After the post-civil war transition period, successful elections were conducted in 2006 and a newly elected Liberian Government led the development of a five-year National Health Policy and Plan in 2007, USAID’s flagship health sector activity was the Rebuilding Basic Health Services (RBHS) activity. RBHS was based on a six-year (2008-2014) cooperative agreement between USAID and a consortium led by John Snow Research and Training Institute, Inc. The activity began with a focus on health service delivery in five counties (Bong, Nimba, Lofa, River Gee, and Grand Cape Mount) by awarding and managing performance-based contracts to INGOs. In 2011 RBHS shifted its focus to capacity-building, while the service delivery component became the core activity of a new, government-to-government health sector activity for USAID. The Liberia Health Sector Fixed Amount Reimbursement Agreement (FARA), formally titled Support to the Ministry of Health and Social Welfare for Implementation of Liberia’s 2011-2021 National Health and Social Welfare Policy and Plan, was signed in September 2011. The FARA was developed in the context of a gradual transition away from the post-conflict setting, with numerous international health actors, to a government-managed health sector, building on the incremental improvements made in health indicators and health systems strengthening since the end of the Liberian civil war in 2003. The FARA was designed to build the Ministry of Health and Social Welfare’s – now the Ministry of Health (MOH) – capacity to manage the health systems necessary to successfully implement the FARA, as well as to directly support health service delivery through awarding and managing performance-based contracts with NGOs. The theory of change underpinning the mechanism was that, if USAID health funds were provided directly to the GOL for implementation, it would result in more effective and efficient long-term development by reducing transaction costs and strengthening local stewardship and capacity through a learning-by-doing approach. From 2011 to 2015, USAID made available $42 million through the FARA mechanism. The GOL agreed to pre-finance the FARA on a quarterly basis through the national budget with USAID quarterly reimbursing fixed amounts to the GOL for the achievement of predetermined deliverables, verified by USAID. These were largely based on the provision of health services at health facilities in the target counties. The expected cost of the predetermined deliverables was estimated by USAID before the start of implementation and agreed with the GOL in the FARA. The quarterly US dollar reimbursement amounts were fixed in the agreement for the duration of the activity. Directly linked to the deliverables, the FARA included three main areas of activity designed to improve the capacity of the MOH to manage, oversee, and monitor the delivery of services, including: i. Increasing technical and management support by funding up to 20 locally contracted management consultants assigned to key units and departments within the MOH; ii. Quarterly MOH supervision visits at the facility level and quarterly data harmonization reviews; iii. Procurement of preventive and curative health services and commodities through performance-based contracts [health commodities were later removed and funded through an alternative USAID mechanism]. FARA implementation went ahead as scheduled, with RBHS providing technical support to the MOH until that activity ended in 2014. Beginning in early 2012, the MOH issued short-term (3 to 10 month) contracts to INGOs that were formerly funded under RBHS in Bong, Lofa, and Nimba counties, while procurement processes were conducted to award multi-year contracts in these counties. During this same period, the MOH was exploring contracting-in arrangements in other counties, with funding from other donors, and assessing County Health Team (CHT) readiness to participate in contracting-in. As the Bong CHT scored well in the contracting readiness assessment exercise, it was selected by the MOH to participate in contracting-in under the FARA mechanism, 15 with the consent of USAID. Initially, the Bong CHT was funded by the MOH under FARA to support seven health facilities in Bong, while Africare continued to support 29 facilities. This mixture of contracting-in to a CHT and contracting-out to an NGO in the same county is referred to as the “Hybrid Model” in FARA as well as by health sector stakeholders. The Hybrid Model of contracting-in and contracting-out officially began on July 1, 2014, which was the start of the GOL Fiscal Year (FY) 2014-15. This timing coincided with a rapid worsening of the Ebola outbreak, from an isolated area in northwest Liberia to a national epidemic. The Ebola epidemic had many unexpected effects, including a deep public concern about the safety of receiving health services at public health facilities that resulted in a steep decline in the utilization of health services across the country. As a result of the rapid decline in utilization, restoration of health services became a top priority for the GOL, its development partners, and the local population. By the end of FY 2014 (June 30, 2015), health system data indicated that utilization had returned to near normal levels. Like all other counties in Liberia, FARA-supported counties were the location of a major international humanitarian response to the Ebola epidemic. The Ebola response brought with it significant emergency funding and assets, but it also proved to be a significant challenge to coordinate and manage for the CHTs, including the Bong CHT that was newly implementing the contracting-in approach. FARA was regularly amended by USAID and the GOL to increase the amount of obligated funds and to address any unforeseen programmatic aspects of the agreement, such as the Ebola epidemic. In January 2015, FARA was also amended to include several additional activities that were directly linked to implementation of Liberia’s “Accelerated Action Plan to Reduce Maternal and Newborn Mortality.” However, these additional activities represented only one percent of total FARA expenditure. Despite experiencing delays with the release of the pre-financing by the Ministry of Finance and Development Planning (MFDP), due to fiscal constraints experienced by the GOL, implementation continued until FARA’s conclusion on December 31, 2015. However, delays in the release of pre￾financing did impact the implementation of some planned activities, especially health worker training, which contracted NGOs were unable to pre-finance with their own organizational funds. Based on a new design and costing process conducted in 2015 by USAID, a subsequent five-year activity, referred to as FARA 2, was signed in January 2016 and implementation of that activity is currently on-going. 16 3. PURPOSE & QUESTIONS 3.1 Purpose of the Study The overall USAID FARA Performance Evaluation was undertaken to evaluate the status of results at the end of the implementation period (December 31, 2015) and investigate the validity of some of the assumptions underlying the FARA. LSA designed an approach to answer USAID’s FARA Performance Evaluation questions iteratively through a separate capacity assessment, costing study, and this final piece on health outputs. The FARA Performance Evaluation is comprised of these three distinct components. However, they draw on each other’s evidence to complement and triangulate findings across performance areas. A final summary report entitled, “The FARA Performance Evaluation Report,” will follow this report and provide comprehensive answers to the overall evaluation questions. The specific purpose of exploring health output performance is to determine whether transition to the FARA mechanism resulted in an improvement in clinical indicators; if the “learn by doing” approach strengthened MOH health systems; and whether the FARA was effective in incentivizing organizational transformation in Liberia. 3.2 Evaluation Questions This report is intended to answer a narrower set of questions that will contribute to the overall FARA Performance Evaluation and to serve as a resource for USAID and the MOH, including for potential future modifications to FARA 2. The Health Output component, which comprised Evaluation Question 2, focused on the following three specific questions in the design of instruments, analysis tools, and in the overall findings, conclusions, and recommendations of the report. • What were key health outputs for the following health streams, comparing FARA to non-FARA counties? o President’s Malaria Initiative (PMI) o Expanded Program for Immunization (EPI) o Family Planning (FP) o Maternal and Child Health (MCH) o Water, Sanitation, and Hygiene (WASH) • Were the results achieved commensurate with the level of these streams of funding? • What MOH and CHT capacity gaps pose the greatest barriers to health outputs? The Health Output component contributes to providing a holistic picture of the overall FARA Performance Evaluation questions listed below. The Health Output component answers the overall Evaluation Question 2 of the FARA Performance Evaluation. Question 1 was answered by the capacity assessment. Question 3 was answered by the costing study. The three overarching FARA Evaluation Questions were: Question 1: Institutional Capacity, Ownership, and Sustainability • Did the FARA mechanism strengthen government capacity, stewardship and ownership1 ? 1 The U.S. Government defines country ownership as “the continuum of actions taken by political and institutional stakeholders in partner countries to plan, oversee, manage, deliver, and finance their health sector and achieve health goals. These actions advance sustainable, quality health programs that are locally owned and responsive to the needs of host country nationals.” 17 o Which activity components were most effective at building MOH and CHT capacity? o For FARA 2.0, where should further capacity, ownership and stewardship efforts concentrate? o Are capacity advances to date sustainable? Question 2: Health Outputs • What were key health outputs for the following health streams, comparing FARA to non-FARA counties? o President’s Malaria Initiative (PMI) o Expanded Program for Immunization (EPI) o Family Planning (FP) o Maternal and Child Health (MCH) o Water, Sanitation, and Hygiene (WASH) • Were the results achieved commensurate with the level of these streams of funding? • What MOH and CHT capacity gaps pose the greatest barriers to health outputs? Question 3: Costing Study of FARA Mechanism • Were the results achieved through the FARA commensurate with the costs and funding that USAID put into the mechanism? In other words, what was the value for money of the FARA G2G funding/implementation mechanism? • Is the FARA mechanism of contracting in – as reflected in the Bong Hybrid model -- more cost￾effective than contracting out to NGOs such as IRC and Africare? • What are recommendations going forward for FARA 2.0? What tool can USAID use to measure cost-effectiveness that balances achieving results, building capacity, and mitigating risk? 18 4. EVALUATION METHODS & LIMITATION 4.1 Methods To answer each of the specific Health Output questions described above, LSA conceived a tailored methodology that draws upon varying quantitative and qualitative information sources and appropriate analysis techniques. The evaluation employed the following methodologies: • Document and data reviews. The team reviewed HMIS data, health facility monitoring data and activity reports, accreditation reports, MOH annual reports, capacity assessments, and DHIS-2 data, among others. • Key informant interviews. These were carried out with a wide range of stakeholders including Liberian public institutions (MOH, etc.), implementing partners, and bilateral and multilateral organizations. County-level civil servants and facility managers were also interviewed. • Focus group discussions. FGDs were carried out with county-level civil servants as well as staff and clients of public health facilities. • Site visits. The evaluation team visited 18 clinics (10 FARA, 8 non-FARA) at which observations of key clinical infrastructure, medical equipment, commodities, and staffing were recorded. Geographic sampling: A total of four counties were selected for field work and detailed analysis of qualitative and quantitative data. Within FARA counties, Bong has selected to provide a detailed comparison of how two service delivery modalities (contracting in vs contracting out) have performed within the same county. The selection of Lofa as the second site for field work is based on its relative remoteness, in order to assess if this in any way affected overall performance in terms of health outputs. In addition, the Implementing Partner (IP) for FARA is different from the IP in Bong County. Non-FARA counties selected were Bomi and Gbarpolu. The main criteria for selection was the time limitations of the field work, and therefore, both counties had to be within relatively easy travel time from Monrovia. Both counties represent varying degrees of health care access, different catchment populations, and NGO support. For the sampling of facilities that deliver primary health care services in the community, four to five facilities were visited per county, resulting in 18 facilities. These sample points were selected based on the average scoring of facilities based on MOH scorecards, with the highest scoring and lowest scoring facilities selected. This provided a range of performance for analysis. A final determination of accessibility was used to make the final selection of two high scoring facilities and two low scoring facilities for field visits. Stakeholders interviewed in KIIs and FGDs were categorized based on the operational levels: national, county, and facility. This ensured a range of opinions and experiences were captured to provide quantifiable and expert opinions regarding the performance of FARA and some of the broader health system challenges that FARA 2.0 may be able to address. Included below (Table 3 19 and Table 4) is an overview of the type of stakeholders targeted, the methods for consultation, and a final number of individuals and groups reached. Table 3: KIIs Conducted at the National Level National Level Data Collection Level KIIs Target Achieved National Ministry of Health MFDP USAID Implementing Partner Others (Gavi, GFATM, RBHS, CSH) 8 4 2 2 4 11 4 2 2 2 TOTAL 18 19 Table 4: KIIs and FGDs Conducted at the County Level County Level Data Collection Level KIIs Target Achieved FGDs Target Achieved County County Health Teams (CHO, CHDD, CHS) 12 25 County Health Teams (MCH, FP, EPI, M&E) 16 16 Facility Officer-in-Charge 16 28 Clients and staff 16 8 TOTAL 28 53 32 24 A critical change to the qualitative data sampling took place in the field. The team quickly identified that by doing focus group discussions with facility staff, time was diverted away from serving clients. In order to be sensitive to the clients, the facility staff were interviewed individually. Hence, the number of facility interviews was much higher than the target, and the number of facility focus groups was lower. Gender-sensitive Perspective Consistent with USAID’s evaluation policy, and recognizing that effects of integration and the success of the activity might vary across gender, the evaluation team applied a gender perspective to the entire evaluation process. Starting with the desk review, data related to gender-based variances in outcomes were examined and documents reviewed with a gender-sensitive lens to inform a better understanding of gender dynamics in the implementation environment. The evaluation team included both female and male researchers, and a gender balance among respondents was sought during the field work data collection phase, especially among CHT staff and facility workers, per respondent availability. Social-sensitive Perspective In addition to issues of gender, the evaluation team considered issues of social demographics in the sampling and data analysis processes, and how they may affect the outcomes of the activity and/or the evaluation findings. For the qualitative data, KII and FGD instruments collected and recorded these data, which were coded into the analysis tool and disaggregated in analyzing the qualitative findings. For the quantitative data, the evaluation team worked with the data systems developed by the MOH and disaggregated data on each demographic point for which data was available and relevant to the analysis process. 20 4.2 Limitations The evaluation team was able to review the necessary data to answer the related FARA Performance Evaluation questions; however, limitations exist that are potentially relevant to the findings, conclusions, and recommendations. These are laid out in Table 5. Table 5: Evaluation Limitations and Risk Mitigation Strategies Constraint/Limitation Risk Mitigation For the evaluation’s results to be accurate, the data collected and reported by the MOH through the HMIS must be reliable. Sample indicator data from collection points at the facility was compared to HMIS. The sampled facilities data analyzed indicated that reported data was accurate. However, the method is not as rigorous as a Data Quality Assessment. When arriving in the afternoons at facilities, a reduced number of clients were available for FGDs. Reaching a planned number of interviewees was not as important as getting the qualitative data from as many informants as possible. Facility visits provided a real-time “snapshot” of the actual operating situation on that given day at that specific facility. There was sufficient qualitative data to provide comprehensive analysis. It was originally anticipated that time allocations and distances for field work could be insufficient if any delays were experienced. The team was able to exceed the planned number of facilities visited. Other health systems strengthening projects are ongoing in FARA counties simultaneously alongside FARA interventions. Spillover from these interventions could heavily contaminate quantitative findings. We list caveats in the quantitative data analysis sections. We rely heavily on qualitative data in the form of KIIs and FGDs and site visit observations to differentiate results from FARA and non-FARA interventions. FARA and non-FARA counties may not be directly comparable as health burdens and other demographics of the county populations could differ significantly. Further, FARA counties consist of both FARA and non-FARA facilities making it difficult to distinguish the specific effect of FARA on health outcomes While the three counties of Bong, Lofa, and Nimba are the only counties with FARA facilities, comparisons of quantitative indicators that use these counties should be interpreted with caution given that there are both FARA and non-FARA supported facilities located there. We rely heavily on qualitative data at the facility level in the form of KIIs and FGDs and site visit observations as a more accurate means of teasing out the FARA contribution to health outputs. The influx of post-Ebola funding and infrastructure for certain counties could confound findings. Qualitative data seeks to parse out the effect of FARA versus post-Ebola response. We note in the findings where we believe post-Ebola activities have influenced health outputs. There were different IPs in the FARA counties of Lofa and Bong. Implementation of FARA may not be standardized across FARA counties. We note this discrepancy and attempt to differentiate between findings in Lofa, Bong, and Nimba where possible. Small sample sizes and lack of data on contracting-in vs contracting-out facilities limits our ability to make definitive statements about some FARA programming. We note in the report that both statements on the effectiveness of contracting models and quantitative indicators based on site visits (e.g. WASH) should be viewed cautiously. Longitudinal data for contracting-in facilities does not yet exist given that the model was only begun in 2014. 21 5. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS 5.1 Findings and Conclusions for Question 1 Question 1: What were key health outputs for the following health streams, comparing FARA to non-FARA counties? • President’s Malaria Initiative (PMI) • Expanded Program for Immunization (EPI) • Family Planning (FP) • Maternal and Child Health (MCH) • Water, Sanitation, and Hygiene (WASH) The evaluation determined outputs by examining quantitative and qualitative health outcome data now and over time (since 2011) at the county level, facility level, and by contracting model. The following indicators are examined to determine progress per health stream (Table 6). Table 6: Key FARA Health Stream Indicators Health stream Indicator(s) measured President’s Malaria Initiative 2nd IPT dose LLITN distributed for <5 years children* Malaria cases diagnosed by RDT & Microscopy Malaria cases treated with ACT Expanded Program for Immunization Pentavalent coverage (3 doses given) Family Planning CYP Total Maternal and Child Health Normal deliveries by skilled health personnel 4th+ ANC visit accompanied by TTM Postpartum women attended post-natal clinic within 2 days of delivery** Water, Sanitation and Hygiene Existence of functional infrastructure and amenities at a facility (See WASH section) *After reviewing HMIS data, yearly variation for the indicator “LLITN distributed for <5 years children” was deemed too high for meaningful quantitative analysis. **The indicator “Postpartum women attended post-natal clinic within 2 days of delivery” was not tracked in HMIS data. FARA currently tracks 18 stewardship and health stream output indicators, including those in the table above. While the following analyses in this report show findings with regard to the tracked indicators above, the evaluation team believed the list to be insufficient to make larger generalizations about the status of health stream outputs. For instance, tracking one indicator for FP—CYP total—does not necessarily give us a full picture of the deficiencies of different FP methods among countries or facilities. Further, the current MCH indicators above do not cover child survival measures, such as diarrhea or TB. FARA 2 may do well to expand the indicator selection to better measure potential impact on health stream outputs. 22 HEALTH INDICATOR RESULTS - FARA AND NON-FARA COUNTIES Comparison of health indicators at the county level was completed via review of HMIS data. Using this data, we examine the change in key FARA health output indicators among FARA counties since 2011 and compare these data to changes in health outputs in non-FARA counties since 2011, looking at the average yearly change. We hypothesize that FARA counties should have higher average annual changes in health output indicators when compared to non-FARA counties. Figure 1 displays these findings. Note that “FARA counties” refer to the aggregation of only FARA facility data in Bong, Lofa, and Nimba counties, and does not take into account non-FARA facility data in those counties. Figure 1: Average Yearly Change in Health Stream Indicators, 2011-2015, FARA vs. Non-FARA Counties • PMI. FARA counties outperform non-FARA counties in terms of second dose of IPT for malaria (7% increase on average per year versus 2%). However, non-FARA counties outperformed FARA counties in terms of malaria cases diagnosed by RDT or microscopy (4% average increase per year versus 1% average decrease per year in FARA counties) and for ACT treatment for under 5 children (20% increase on average per year compared to 13% in FARA counties). • EPI. Non-FARA counties outperformed FARA counties in terms of coverage rates of children under 1 year who received DPT3/pentavalent-3 vaccination, increasing on average 2% per year compared to a 1% decrease per year among FARA counties. • FP: Non-FARA counties outperformed FARA counties in terms of Couple-Year Protection (CYP), increasing an average of 35% per year compared to 27% in FARA counties. 2% 6% 35% 11% 4% 20% 2% 7% 3% 27% 7% -1% 16% -1% -10% 0% 10% 20% 30% 40% 2nd IPT dose 4th+ ANC visit CYP Total Normal deliveries conducted at health facilities by skilled health personnel Malaria cases diagnosed by RDT & Microscopy Malaria cases treated with ACT Pentavalent 3 doses given Non-FARA FARA 23 • MCH. Non-FARA counties outperformed FARA counties in normal deliveries conducted at health facilities by skilled health personnel, increasing an average of 11% per year compared to 7% in FARA counties. Non-FARA counties also outperformed FARA counties in terms of proportion of women attending at least 4 ANC visits accompanied by a trained traditional midwife (TTM), increasing on average 6% per year compared to 3% in FARA counties. • WASH: Data for WASH indicators at the national level were not available for analysis. HEALTH INDICATOR RESULTS - FARA AND NON-FARA FACILITIES Comparing data between FARA and non-FARA counties has many limitations, as the two groups of counties may differ significantly in terms of disease burden, population demographics, or other important characteristics. To account for this, we also looked at health outcomes within FARA counties. We looked to the health data of those served by FARA facilities within FARA counties and compared it to those served by non-FARA facilities within the same counties with the assumption that the population served by these facilities should be more or less similar and thus more comparable than at the national level. As in the section above, we look at the average yearly change of key FARA health output indicators as a means of judging impact of FARA. Figure 2 displays these findings. Figure 2: Average Yearly Change in Health Stream Indicators, 2011-2015, FARA vs. Non-FARA Facilities within Bong, Lofa, & Nimba Counties • PMI. FARA facilities outperform non-FARA facilities in terms of second dose of IPT for malaria (7% increase on average per year versus 1%), malaria cases diagnosed by RDT or microscopy (7% versus 4% average increase per year for non-FARA facilities), and ACT treatment for under 5 children (1% decrease on average per year compared to 3% decrease in non-FARA facilities). • EPI. FARA facilities outperformed non-FARA counties in terms of coverage rates of children under 1 year who received DPT3/pentavalent-3 vaccination, decreasing on average 1% per year compared to a 5% decrease per year among non-FARA facilities. 1% -6% 45% 4% -3% 7% -5% 7% 3% 27% 7% -1% 16% -1% -20% -10% 0% 10% 20% 30% 40% 50% 2nd IPT dose 4th+ ANC visit CYP Total Malaria cases diagnosed by RDT and microscopy Malaria cases treated with ACT Normal deliveries conducted at health facilities by skilled health personnel Pentavalent 3 doses given Non-FARA FARA 24 • FP: As at the national level, non-FARA counties outperformed FARA counties in terms of Couple-Year Protection (CYP), increasing an average of 45% per year compared to 27% in FARA counties. • MCH. FARA counties outperform non-FARA counties in normal deliveries conducted at health facilities by skilled health personnel, increasing an average of 16% per year compared to 11% in non-FARA counties. FARA facilities also outperformed non-FARA facilities in terms of proportion of women attending at least 4 ANC visits, increasing on average 3% per year compared to an annual average decrease of 6% in non-FARA facilities. • WASH: Data for WASH indicators at the county level were not present in documents reviewed. Results for the WASH indicator were calculated using observation data collected during field visits and presented in the Infrastructure and WASH section of this report. Site visit findings: Quantification of observational data collected at site visits helps paint a picture of the differences between FARA and non-FARA supported facilities, particularly as it pertains to key medical equipment, infrastructure, commodities, and staff. KIIs and FGDs carried out during site visits also add valuable qualitative data to supplement and triangulate quantitative data. FARA facilities had generally positive findings (compared to non-FARA facilities) with regard to the following: • Stocking of essential drugs, including malaria treatment and LLITNs. FARA appears to have increased the availability and distribution of malaria treatment and long￾lasting insecticide-treated bed nets (LLITNs). Availability of needed medications, drugs, and supplies at FARA-supported facilities has also improved (9 of 10 FARA facilities reported infrequent stock outs). In non-FARA facilities, drug and supply stock-outs were expressed as a major concern for clients because it forces them to go to a drug store or market where the medications were prohibitively expensive, of unknown origin, or unreliable. Stocking drugs and commodities is especially important in the Liberian context as patients often face long, arduous walks to access a health center and must be confident that drugs and medications will be available when they need it. • Referral systems. The FARA program provides non-cash incentives to motivate TTMs to refer pregnant patients to local facilities. This has increased the number of in-facility deliveries with skilled attendants, and significantly reduced home deliveries. FARA clinics have developed systems for referrals to hospitals for complicated obstetrical and newborn cases. Where an ambulance is available, FARA clinics are not only using the vehicles but also sending a qualified RN or Midwife to care for the patient. • Construction and renovation of key infrastructure. The FARA program has worked towards the construction or renovation of clinic facilities, in particular the labor, delivery, recovery, and postpartum rooms and related medical equipment. Maternal waiting rooms and houses have been built with funds from a variety of sources including FARA’s administrative bonuses. The majority of facilities (9/10 FARA, 6/8 non-FARA) had separate delivery and post-partum rooms, most of which were newly constructed post-partum rooms. Though separate rooms exist, it does not mean that the necessary equipment for deliveries and post-partum care was available and/or operational. • IEC promotional materials, and posted treatment guidelines and health goals. Both FARA-supported and non-FARA supported facilities displayed a wide range of appropriate IEC posters and health promotion materials. Staff in many facilities supplemented these posters by offering “waiting room health talks” to patients, explaining in detail prevention, detection, and treatment of common illnesses. Nearly all FARA and non￾FARA facilities visited displayed IEC materials, treatment guidelines, and health targets/goals. 25 • Human resources for health: With regards to staff, FARA was credited with increasing the number of qualified staff at FARA-supported facilities (compared to non-FARA facilities) and provided a range of support to ensure deliverables were met. The PBF mechanism was mentioned as being responsive, timely and regular, and these FARA-supported incentives are noted under Question 3 in the report as being key to the staffing improvements seen. FARA has built the capacity of staff through training and refresher workshops and collaborated with the MOH to have qualified staff in the proper place to improve quality service delivery. It is important to note that non-FARA facilities had a more equal gender balance of staff than FARA facilities (52/48% female-to-male versus 59/41%, respectively). CONTRACTING-IN VERSUS CONTRACTING-OUT To compare the potential association between contracting models and health outcomes, we looked at different indicators at the county level, splitting them out among the three FARA counties. Bong County utilizes a hybrid model of contracting-in (7 facilities) to the CHT and contracting-out (29 facilities) to an INGO, among a total of 36 FARA-supported facilities. We make the hypothesis that Bong County, the only county of the three to have any facilities with a contracting-in model, will have substantially different health outcomes than Lofa and Nimba counties, a fact which might indicate the effectiveness of the contracting-in approach. We first compare the data among only FARA facilities in the three counties. Given that 7/36 FARA￾supported facilities in Bong County are a contracting-in model, we hypothesize that there may be some difference in health output data among only FARA facilities in these counties. Figure 3 below shows that Bong County FARA facilities outperformed FARA facilities in Lofa and Nimba counties for 4 of 7 indicators: second IPT dose, malaria cases treated with ACT, normal deliveries conducted at health facilities by skilled health personnel, and pentavalent doses given. Nimba County outperformed the others in four ANC visits for pregnant women, while Lofa County outperformed the other counties in CYP total and malaria cases diagnosed by RDT or microscopy. While these findings do not differentiate between contracting-in and contracting-out FARA facilities in Bong itself, they may indicate that the contracting-in model performs better than the contracting-out model among some key indicators. Figure 3: Average Yearly Change in Health Stream Indicators, 2011-2015, Among FARA Facilities in Bong, Lofa, and Nimba Counties 14% 1% 24% 8% 6% 22% 3% 11% 1% 51% 13% -3% 13% -1% 1% 9% 15% 10% -3% 12% -10% -4% 10% 30% 50% 2nd IPT dose 4th+ ANC visit CYP Total Malaria cases diagnosed by RDT and microscopy Malaria cases treated with ACT Normal deliveries conducted at health facilities by skilled health personnel Pentavalent 3 doses given Bong FARA Lofa FARA Nimba FARA 26 Next, we compare health output data within the three counties between FARA and non-FARA facilities. For this argument, we posit that Bong County FARA facilities will outperform non-FARA Bong facilities on more indicators than do FARA facilities in Lofa and Nimba counties compared to non-FARA facilities there. For example, if the contracting-in model is indeed more effective than a contracting-out model, we might expect Bong FARA facilities to outperform Bong non-FARA facilities in 7 of 7 indicators, whereas Lofa and Nimba FARA facilities would outperform Lofa and Nimba non-FARA facilities in perhaps only 4 of 7 indicators. Table 7 shows that this is not necessarily the case. Indicators in green are where FARA facilities outperformed non-FARA facilities and indicators in red are where the opposite was true. While Bong FARA facilities outperformed Bong non-FARA facilities in 5 of 7 indicators, Nimba FARA facilities also outperformed Nimba non￾FARA facilities in 5 indicators, and Lofa FARA facilities outperformed Lofa non-FARA facilities in 6 of 7 indicators. This approach does have several limitations, including different implementing partners in Lofa and Bong counties. Table 7: Average yearly change in health stream indicators, 2011-2015, among FARA and non-FARA facilities in Bong, Lofa, and Nimba counties Data Bong Lofa Nimba Non￾FARA Facilities FARA Facilities Non￾FARA Facilities FARA Facilities Non￾FARA Facilities FARA Facilities PMI 2nd IPT dose -19% 14% 1% 11% 5% 1% Malaria cases diagnosed by RDT & Microscopy 2% 8% 6% 13% 3% 10% Malaria cases treated with ACT 3% 6% -5% -3% -2% -3% EPI Pentavalent 3 doses given -1% 3% -3% -1% -7% -4% FP CYP Total 31% 24% 57% 51% 41% 15% MCH Normal deliveries conducted at health facilities by skilled health personnel 3% 22% 11% 13% 6% 12% 4th+ ANC visit accompanied by TTM 6% 1% -9% 1% -3% 9% INFRASTRUCTURE AND WASH Comprehensive quantitative data on facility-level WASH and infrastructure indicators were not readily available in document and data reviews conducted by the evaluation team. To fill this gap, we conducted a comparison of key indicators in FARA and non-FARA facilities during field visits to the 18 sample clinics. The evaluation team used a checklist at each facility to note the existence and quality of certain infrastructure and WASH-related components. Analysis of these data shows that when it comes to infrastructure, FARA facilities slightly outperform 27 non-FARA facilities in some areas (especially in the family planning services area and having a separate postpartum room) but did not surpass non-FARA facilities in other areas (presence of a solar power/generator and a microscope). All FARA and non-FARA facilities had an ambu-bag, a handheld resuscitator, and nearly an equivalent number stocked rapid diagnostic tests for malaria (Figure 4). It is important to point out the apparent discrepancy between Figures 2 and 4, which show non￾FARA facilities outperforming FARA facilities in CYPs (Figure 2) but show FARA facilities outperforming non-FARA facilities in FP methods (Figure 4). This discrepancy can be explained through a better understanding of the calculation of CYPs. Couple Years Protection (CYP) is a measurement (in years) by which a specific FP method delays a couple from starting a family. The number of years of delay varies widely by FP method. For example, oral contraceptives justifiably receive a CYP score far lower than a long acting reproductive method such as an implant or IUD. In Figure 2, we report CYP counts, which do not necessarily indicate the frequency or volume of FP method delivery, but rather simply the sum of all CYPs. The resulting displayed measurement could be artificially inflated among non-FARA clinics because they may, for example, successfully promote long-acting methods that garner higher CYP scores more than FARA clinics promote and distribute such methods. However, as Figure 4 shows, FARA clinics are more successful in stocking and distributing FP methods, though these methods could overwhelmingly be those that receive lower CYP measurements. This would account for the difference in CYPs in Figure 2, yet high marks on general FP indicators in Figure 4. Delving deeper into the types of FP methods promoted and distributed by different clinic types will ultimately shed better light on the situation around FP availability and access. Figure 4: Key Infrastructure and Equipment Indicators of Facilities Visited (n=18) In terms of WASH indicators, FARA facilities scored at or above non-FARA facilities on all indicators except having separate latrines for males and females (the difference was slight: 8/10 FARA facilities had separate latrines versus 7/8 non-FARA facilities). FARA facilities had better performance in terms of the presence of an autoclave, water storage tank with water stored, incinerator in use, separate latrine for staff and clients and sharps boxes in each of the clinical rooms/areas. All FARA and non-FARA facilities had placenta pits and waste disposal areas ( 100% 88% 63% 75% 88% 63% 100% 90% 90% 90% 80% 40% 0% 20% 40% 60% 80% 100% 120% Ambu-bag Rapid diagnostics for malaria FP methods Separate postpartum room Solar/generator Microscope FARA non-FARA 28 Figure 5). Figure 5: FARA vs. Non-FARA Facilities WASH Comparison The generally positive findings around infrastructure and WASH for FARA and non-FARA facilities can probably be explained, in part, by the recent Ebola virus epidemic. Hand washing stations inside and outside the facilities were present, operational, and universally accepted. Written and graphic hand washing instructions were posted at the entrances of the initial screening areas and main clinic buildings, examination and service rooms, as well as latrines and bathrooms. Personal Protection Equipment (PPE) was stored in a convenient place and readily available if needed. Posters depicting the signs and screening protocols for potential Ebola patients were present in the screening and treatment areas. CONCLUSIONS FOR QUESTION 1 FARA versus non-FARA counties: Quantitatively, At the national level, FARA counties across the board do not perform better than non-FARA counties. Non-FARA counties performed better than FARA counties in 6 of 7 indicators, with the exception being second IPT dose given to pregnant women. Data for these findings were extricated from a simple review of the DHIS2 data and do not consider innumerable confounding factors and biases, such as differing health burdens and population demographics between counties, nor other potentially overlapping health systems strengthening projects that could contaminate results. FARA versus non-FARA facilities: Within FARA counties, FARA facilities performed nearly the opposite as they did at the national level, doing better over time compared to non-FARA facilities in 6 of 7 indicators tracked, with the exception of CYP. Though there are many limitations to this 100% 100% 88% 63% 38% 88% 38% 25% 100% 100% 100% 90% 90% 80% 70% 70% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Waste disposal area Is there a placenta pit Sharps boxes in each of the clinical rooms/areas Separate latrine for staff and clients Incinerator in use Separate latrine for male and female Water storage tank and confirm if water is stored Does the facility have an autoclave FARA non-FARA “During the Liberian Ebola Virus Epidemic, the community people did not trust the Healthcare workers or the Healthcare system, [feeling that] it may have been the source of the illness. Since 2015, the community people's trust in the Healthcare workers has returned." - Lofa Clinic Staff FGD 29 analytical approach as well, quantitatively comparing FARA and non-FARA facilities within the same counties is a more accurate measure given that those served by the two facility types are assumed to be more or less similar in health burden and population demographics, and might be affected equally by any confounding factors. Health stream indicators: Among health stream indicators, FARA facilities saw the largest average increases over time in women receiving second IPT doses (7% average increase per year), cases of malaria diagnosed by RDT or microscopy (7% per year), CYP (27% per year), and normal deliveries carried out at a facility by a skilled attendant (16% per year). Conversely, FARA facilities decreased on average each year in malaria cases treated by ACT (1% decrease per year) and 3 doses of pentavalent given to children under one years old (1% decrease per year). Further, FARA facilities saw relatively small increases (3% per year) in pregnant women attending 4 ANC visits accompanied by a TTM. For WASH indicators, site visits among 18 clinics showed FARA facilities to outperform or perform equally to non-FARA facilities among all WASH indicators. WASH findings were limited methodologically, however: the analysis did not benefit statistically from the availability of large scale quantitative data and results among all facility types are probably confounded by post-Ebola interventions. Among all key FARA health stream indicators, FARA facilities perform substantially worse than non-FARA facilities in FP (utilizing the CYP measurement); non-FARA facilities registered almost twice as much gain over time in CYP compared to FARA facilities. Understanding the differences in FP interventions as well as demand and supply of FP methods among FARA and non￾FARA facilities and catchment areas may help to better explain these findings. Contracting model: For contracting model, it remains unclear whether the contracting-in model employed by 7 facilities in Bong results in better health outcomes than a contracting-out model. While Bong County FARA facilities outperformed Lofa and Nimba FARA facilities in 4 of 7 indicators and performed better than Bong non-FARA facilities in 5 of 7 indicators, they did not perform better on more indicators when compared to analyses conducted between FARA and non-FARA facilities in Lofa and Nimba counties. Limitations abound for this quantitative comparison, including our inability to distinguish the contracting-in versus contracting-out facilities in Bong County and the fact that the FARA IP is different between Bong and Lofa counties. Further, a straight comparison of changes in health outputs among only FARA facilities in the three counties may be susceptible to inter-county biases such as differences in health burdens or population demographics. Larger sample sizes for contracting-in facilities combined with supplemental qualitative data to triangulate and strengthen findings would go a long way to clarifying the effect of contracting model on health outputs. 5.2 Findings and Conclusions for Question 2 Question 2: Were the results achieved commensurate with the level of these streams of funding? After several attempts, the evaluation team was ultimately unsuccessful in obtaining expenditure data for specific health outputs. Though the MOH provided financial expenditures, these were tracked at a level higher than health outputs, which were rolled up into more general cross-cutting line items, such as supervision or technical and management support. This financial reporting did not allow for us to measure the expenditures by FARA health output specifically. The evaluation analyzed FARA 1 compliance with USAID Assistance Automated Directives System (ADS) 200 policies and guidance in areas such as Programming Policy (rev: 2014). Analysis and conclusions thereof can be found in Annex IV. 30 CONCLUSIONS FOR QUESTION 2 The evaluation team was unable to link MOH FARA expenditures and specific health stream outputs due to the fact that the MOH does not track line-item spending at the health stream output level. Rather, health stream output indicators are rolled up into more overarching management level indicators. A change in financial reporting whereby future expenditure line items are disaggregated further into health streams would benefit not only donor auditing processes but simultaneously increase financial transparency and better inform programmatic management. 5.3 Findings and Conclusions for Question 3 Question 3: What MOH and CHT capacity gaps pose the greatest barriers to health outputs? The methodology for this approach included a review of MOH documentation, policies and guideline procedures, facilities’ scorecards, key informant interviews, focus group discussions, and observations at MOH Central Office, County Health Offices, and healthcare facilities. According to facility scorecards, facilities in FARA counties are most lacking in capacity in the area of quality assurance and, to a lesser extent, stocking of tracer drugs and HMIS quarterly reporting. Table 8 illustrates the performance of each FARA County, with Bong County disaggregated by contracting in and contracting out, during the last year of FARA 1 implementation. The color scoring reflects the ability to meet indicator targets, with green representing excellent, yellow representing medium, and red representing very poor. In proportion of indicators receiving excellent scores, Bong County outperformed Lofa and Nimba counties, scoring excellent on 11 of 18 scorecards versus 2 of 9 and 4 of 9, respectively. Lofa county, which saw five stewardship indicators scoring very poor, two scoring medium, and two scoring excellent performed arguably the worst of the three counties. When it came to stewardship-specific indicators, the three counties collectively performed best on supervision indicators (% of CHT/NGO submitting timely and complete quarterly report to MOH and % of facilities that received at least 1 joint Integrated supportive supervision visit in last quarter) as well as community ownership measures (three CHDC meetings held during the quarter and % of CHVs who received at least 1 supervision visit in last quarter), with excellent scores on 12 out of 16 possible scorecards. The three counties collectively performed the worst on quality assurance indicators (facilities achieving at least 60% in Quality Assurance Assessment and score on client satisfaction surveys), where a full 8 of 8 scorecards received a very poor score. When comparing contracting models within Bong county, the contracting-out model carried out in Africare facilities appears to have outperformed the contracting-in model in CHT facilities, with an excellent score in 6 of 9 scorecards (Africare) compared to 5 of 9 (CHT). Furthermore, Africare’s contracting-out model scored substantially better than the CHT’s contracting-in model for one supervision indicator (% of CHT/NGO submitting timely and complete quarterly report to MOH), with a score of 100% versus 25%, and for one medicines and supply chain indicator (% of facilities submitting timely LMIS report to MOH during the quarter), with a score of 95% versus 75%. Table 8: FARA County Stewardship Progress towards Targets, 2015 Stewardship Indicators Bong CHT Bong Africare Lofa IRC Nimba Africare 31 Medicines & Supply Chain % of facilities with no stock-out of tracer drugs during the quarter (amoxicillin, cotrimoxazole, paracetamol, ORS, Iron folate, ACT) 100% 24% 82% 17% % of facilities submitting timely LMIS report to MOH during the quarter 75% 95% 75% 96% HMIS % of facilities submitting timely and complete HMIS reports to the MOH during the quarter 100% 97% 70% 64% Quality Assurance Facilities achieving at least 60% in Quality Assurance Assessment 0% 29% 0% 0% Score on client satisfaction surveys 0% 0% 0% 0% Supervision % of CHT/NGO submitting timely and complete quarterly report to MOH (financial, narrative, and performance) 25% 100% 100% 100% % of facilities that received at least 1 joint Integrated supportive supervision visit in last quarter 100% 100% 24% 98% Community Ownership Three CHDC meetings held during the quarter 100% 100% 73% 83% % of CHVs who received at least 1 supervision visit in last quarter 100% 100% 94% 93% During qualitative data collection, the evaluation team found several elements at FARA and non￾FARA facilities and among health sector actors that constituted barriers for achieving health outputs: • Lack of access to consistent and running water • Lack of knowledge and proper procedures around medical waste incineration • Inoperable and unmaintained medical equipment • Weaknesses around dissemination and use of data • Lack of defined roles for supervision To strengthen the answers to this evaluation question, we also looked at why some facilities consistently perform at a higher level than others. In looking at positive factors, we made the assumption that components of a successful FARA clinic or FARA CHT may act as key capacity gaps in non-FARA clinics or CHTs. The evaluation team found five common elements in all FARA high performing facilities: • PBF incentive schemes • External support • Integrated, supportive supervision • Robust management standards and logistic systems • New sense of ownership and accountability KEY CAPACITY GAPS Access to water. The evaluation team was repeatedly told that running water was needed in each facility visited, but that this basic request had not been always fulfilled. Seventy percent (7/10) of FARA facilities visited had their own water source on the compound, compared to just 38% (3/8) of non-FARA facilities, mostly in the form of recently installed water tanks used to recover rain water. While some facilities had water stored at the time, the evaluation team found others to be not fully 32 operational and would be of little value during the dry season from December through May. Medical waste incineration. Evaluation findings indicated a lack of awareness of the essential nature of medical waste incinerators. While 9/10 FARA and 3/8 non-FARA facilities were using medical waste incinerators, not all were operational. Some had been damaged since being installed, and others were never put into operation. Nearly all FARA and non-FARA facilities visited had sharps boxes, but when it came to compliance around medical safety, staff were witnessed following incorrect procedures to mitigate risk around needle-stick injuries. Maintenance of equipment. A flaw in the new facility ownership approach is that maintenance of equipment has yet to be taken up by staff as their responsibility. There were two specific problems: one was that staff believe preventive maintenance, repairs, and replacement of needed parts for donor-donated equipment were the donor’s responsibility, and the second was that when facility personnel report a maintenance need to an IP or CHT, they assume that it will be taken care of and do not ensure that the repair or replacement is completed. Essential equipment at all facilities such as generators, motorbikes, laboratory items, and incinerators were found to be non-operational. While construction and renovation of key infrastructure was observed, new post-partum and/or delivery rooms often lacked equipment or had non-operational equipment rendering them effectively useless. Consequences for non-maintenance over time can be high; for example, when a generator is not operational for many months, the microscope cannot be used for laboratory tests, and medical problems are more likely to go undetected and therefore untreated. Dissemination and use of data. At the MOH and CHT levels, weaknesses remain in understanding, disseminating, and using data effectively. Key health sector actors interviewed identified these weaknesses, including a lack of strategies to improve the use and understanding of data by different stakeholders, internal and external to the MOH, and at different levels of health management from the Central MOH to primary health clinic OICs and staff. An additional weakness, confirmed at all levels within the MOH, was the capacity at the district level to accurately report and compile district level reports. Lack of defined roles for supervision. We found that that roles and responsibilities as far as supervision were not clearly articulated and defined across all levels of MOH and between MOH and IPs. While supervision activities under FARA followed and generally supported the gradual decentralization process happening across Liberia, the evaluation team noted that district health offices and officers play a minimal role in supervision and could benefit from capacity building to strengthen their abilities in this area. An outstanding issue in accountability that was identified in FARA I is that there was no independent third-party verification of any contracted-in facility performance. Currently, the central MOH functions as third-party verification for all FARA IPs, but efficient and cost effective independent approaches do not currently exist to ensure programmatic integrity. COMMON ELEMENTS OF SUCCESS PBF incentives schemes. The single most talked about success factor for FARA was the system of performance incentives/bonuses. Per the MOH, Performance Based Financing Operational Manual, dated March 2012: “Performance Based Financing is an implementing strategy of the National Health Policy and Plan.” The FARA mechanism delivers upon this implementing strategy. 33 From the facility staff perspective, we found that FARA facilities consistently perform at a higher level than others because the FARA performance incentives/bonuses system was widely regarded as responsive, timely and regular. Although the clinic staff appeared to have little to no knowledge about the calculation of the PBF bonus, they saw and therefore believed that meeting the targets created a bonus for them. Timely payment of incentives and regular and effective supervision encouraged staff to demonstrate consistently high performance. The fact that the majority of staff personnel who should have been working in the facility were present during site visits, and all absences explained and recorded, may be linked to the FARA bonus system indicators around staff availability and absence tracking. Further, direct cash payments of the incentives appeared to keep staff on site, as they did not have to travel to a bank to take out money. However, it was the evaluation team’s understanding that due to a recent decision by GOL and MOH to make compensation payments to civil servants through direct deposit into personal bank accounts, the practice of cash payments may not continue. Although viewed as critical for high performance and achievement of targets, a lack of understanding about how bonuses are calculated was described by most facility staff members. Some workers believed that the bonuses were an expected part of their compensation rather than an added incentive and reward for superior performance. It was unclear the extent to which this understanding (or lack thereof) contributed to the motivation of staff to perform. The community also felt well-served by the FARA performance incentives/bonuses. TTMs were acknowledged and rewarded through the Health Facility Improvement and Community Initiatives’ non-financial bonus allocation scheme which rewarded them for bringing and referring pregnant women to the healthcare facilities with lappas (traditional textiles/clothing), maternal care supplies, and snacks for their monthly cluster meetings. The community further benefited from the FARA performance incentives/bonuses with improved healthcare infrastructure activities, such as new water wells, maternal waiting houses, facility improvements, staff housing, and other projects. The community development activities were coordinated and controlled through Community Health Development Committees known as CHDCs. The CHDC concept is used throughout Liberia, but FARA County CHDCs had the FARA Health Facility Improvement and Community Initiatives funding source to assist in financing of community projects. An additional community benefit was that the CHDCs provided a discussion platform for community members and leaders to actively participate in healthcare planning and development for their village. FARA-supported facilities with qualified staff, regular and timely payment of incentives and bonuses and supportive supervision have proven to consistently perform better than non-FARA facilities as well as facilities with vertical programs. FARA-supported facilities reported higher mean performance both over time and compared to non-FARA facilities. Findings showed that well￾incentivized FARA facilities were providing better services than non-FARA facilities in maternal health (TTM, AAP and RED and REP support), Family Planning intake and EPI services through collaborative integrated outreach. All FARA facilities receiving regular incentives remained open and provided services during the Ebola period, which helped them to restore community trust, as compared to non-FARA facilities both in FARA-supported counties and elsewhere. “The staff expects and appreciates the FARA bonuses. It has helped the staff stay engaged. In Bolahun…we get paid by our IP for the HMIS bonus and it is on time now.” - Bolahun HC Acting OIC 34 External support. FARA facilities received direct support from an IP and more focused attention from their CHT. This external support targeted both policies and procedures as well as the quality of care provided. On the administrative side, facility staff received support for learning new skill sets, instructions on how to use specific equipment, and support in computing and filing monthly facility statistics reports, including monthly reminder messages and cellphone calls from IPs about data report deadlines. Although the reminders may appear to be minor, the OICs expressed appreciation for the prompts because the end-of-the-month activities are extremely busy at the facility level and data reporting is linked to incentives. On the technical side, mentoring and coaching during supervision visits have improved the skills of service providers and the quality of care over time. In Bong County, the FARA Hybrid Management Model (HMM) was piloted in some facilities. The CHT and District Offices appear to be benefiting from the support of the IP, who has management and delivery responsibilities for the non-HMM county facilities. This spillover effect of the IP’s general oversight and techniques of service administration has been absorbed by the CHT leadership, and replicated to share training events and management tools with the HMM facilities. This unexpected and very welcome benefit has given Bong CHT officials a sense of preparedness for the anticipated MOH movement toward decentralization and use of the contracting-in model throughout the country. Our visits to non-FARA facilities indicated these facilities received periodic supportive visits from their CHTs and MOH Central Office personnel, but without the same regularity or intensity enjoyed by the FARA facilities. Integrated, supportive supervision. Integrated supportive supervision by MOH, CHT, and IPs, including regular reminders of key agreed actions, was reported by those in the field as another key FARA success factor. These Joint Supportive Supervision visits were welcomed by CHT and facilities, with facility staff noting that it was helpful to have all key stakeholders engaged and jointly addressing facility issues. These visits included verifying monthly data, identifying achievements, and solving problems and challenges facing facility staff. Supervision activities under FARA followed and generally supported the gradual decentralization process happening across Liberia. Robust management standards, logistics systems and a quality improvement approach. One of the largest benefits of the current relationship with the NGOs implementing FARA was that they bring to the table and put into practice methods and tools to improve not only management and logistic systems but, most importantly, procedures and processes to improve the quality and safety of healthcare. That said, a repeated theme heard from the CHTs, DHOs, and facility OICs was how they could manage their own healthcare delivery system if only they had the funds that were available to the NGOs. The importance of this issue was more than simply a drive to gain control: it was an integral part of the pride of ownership. CHTs, facility management, and staff stated that medications and supplies are routinely supplied and shipped by IPs to meet the needs of the facilities. This systematic approach may be the direct result of a well-organized supply chain and logistic system, but it may also relate to the IP’s access to funds to facilitate the procurement, delivery, and distribution of the supplies. To be sure, the IPs’ systems “Decentralization is now occurring much more often; the District Health Officers (DHOs) now have a staff of four people to assist in mentoring and monitoring the OICs and clinic staff members.” - MOH Central Office, PBF Manager “When I came here, I did not know when you treat malaria with Artemether, you also need to back it up with ACT; even though I have not gone for the training [this omission] got corrected during the supervision” - Lofa CHT 35 and standards have contributed to reductions in stock-outs, resulting in benefits for the patient as ultimate user, but also staff. When it comes to management systems, staff in public sector facilities did not seem to consistently adhere to and rigorously follow local procedures. Despite this, CHT, District Health Office, and facility staff and management have shown a great appreciation for and compliance with the IP and NGO’s own systems and procedures. This may be due to differences in communication style: among IPs and NGOs, communication comes in written and graphic format, as opposed to orally which is the norm for many things in Liberia. New sense of ownership and accountability. Our findings indicate a new sense of pride and ownership within the CHTs and facilities but also community healthcare volunteers and surrounding communities that has not been present in the past. This sense of ownership was born out of FARA’s funding and accountability methods, in particular the performance targets and incentives to reach them as part of the PBF schemes. Our findings indicate that the most important success factor in enhancing feelings of ownership over a facility were the FARA-supported incentives, provided through the project’s responsive, timely and regular PBF mechanism. The incentives fostered facility accountability by posting targets, making them visible to staff and patients alike. A high level of staff knowledge of the targets, and monthly compliance with these goals, was observed during site visits. Facility checklist records reflected 100% (10/10) of the FARA facilities targets were being displayed while 75% (6/8) of non￾FARA facilities did the same. Offering well-planned rewards for the attainment of these goals fits well within Liberian cultural traditions, and has both motivated staff and served as a source of pride. Additionally, FARA has strengthened the system of accountability and reporting at clinics and health centers by improving leadership and governance. An example was that some of the FARA administrative bonuses were used in accordance with what local facilities collectively deemed to be priority areas, such as building staff housing and maternal waiting homes. Senior CHT staff in Bong consider themselves a model County for contracting-in, and were eager to take on more responsibilities and budgetary management of their programs. This partial transfer of authority was part of the government’s push towards decentralization, placing decision making closer to the people. Although TTMs received no direct financial compensation from FARA, the program provided an opportunity to encourage TTMs to refer their patients to the clinics or hospitals in exchange for token rewards of lappas and other maternal supplies. In addition, the FARA administrative bonuses also provide funding for food during the monthly TTM meetings at the clinics. These acknowledgements of appreciation to community and traditional healthcare volunteers and workers appear to be building a closer relationship and trust between the government healthcare facilities and the communities they serve. “We now manage over 40% of the County budget through FARA at the CHT level, and in less than five years we will be able to manage 100% of the County’s health budget.” - Bong County CHDD “FARA has efficiently enabled the five health streams by just designing simple strategies, i.e. making the facilities achieve targets and giving bonuses to the staff once the targets are met.” - Lofa CHT staff 36 Under FARA, the Community Health Development Committees (CHDCs) monthly meetings, attended by village, quarter, and clan chiefs, community leaders, patients and spouses, TTMs, and clinic staff, have become the setting for discussions involving the community in healthcare planning and development, since the FARA administrative bonus funds were not distributed until the CHDC identified and approved projects. Community members were also given a forum for airing opinions and complaints about the facility’s services and operations while allowing clinic staff to share health education and promotion strategies planned to address healthcare issues, prevention and warnings. Another key finding around sense of ownership was noted during the FGD with facility clients. Despite the negative impact of the Ebola crisis, the rapid post-Ebola recovery was due, in part, to the recognition of the failures prior to the crisis, in particular the need for sustaining trust between facility and community. A HOLISTIC, HEALTH SYSTEMS STRENGTHENING INTEGRATED APPROACH The FARA mechanism was viewed by the MOH Central Office, Bong and Lofa CHTs, and health facilities as a highly welcome holistic approach to strengthen their healthcare system. FARA’s integration and alignment with current MOH systems and structures allowed it to provide harmonized delivery of primary health services in clinics and health centers, and it was a key success factor that differentiated the project from vertical funding utilized by the majority of development partners for specific interventions. The project directly supported the Liberia National Health and Social Welfare Policy and Plan (2011-2021), which emphasizes the need to increase access to a comprehensive package of health services, improve responsiveness of services by transferring management to lower administrative levels, and make services more affordable at a cost that provides medical services to all. Healthcare services delivered at the facility level in FARA facilities were in line with health sector priorities, and our visits, inspections and interviews found that the strategies now in place for planning and collaboration have led to enhanced provision of integrated services, especially in the areas of MCH, EPI, and FP. This theme of cooperation, teamwork, and service integration was repeated in all healthcare facilities visited. Staff at both FARA-supported CHTs and facilities said FARA allowed them to focus on the whole health status of the patients, versus a more limiting vertical approach. The focus at the CHT level also reflected a distinct, holistic approach to guiding and supervising the delivery of county healthcare services. These findings underscore the importance of an integrated approach and its effects on performance and health systems strengthening. “Yes, I am the secretary for the CHDC which meets every month…I am very proud of my role with the community.” - Bong & Gbarpolu County OICs “We all do our work together, MCH, EPI, and FP [and] do weekly outreach support by our implementing partner. Outreach is done together. We take all of the services provided at the clinic out to the people, especially in hard to reach areas, to help us meet our targets.” - Bolahun Health Center Staff FGD “The best thing FARA has improved is the integration of all our planning and delivery of services.” - Lofa CHT FGD 37 OTHER FINDINGS Human resources for health: FARA has increased the number of qualified staff at FARA facilities and contributed to capacity building of staff. All professional staff at FARA facilities were found to be capable of preparing reports and carrying out other administrative functions. Skilled birth attendants (Certified Midwives) and professionals (Physician Assistants and Nurses) were being put into strategic leadership positions at the FARA facilities. FARA-supported facilities proved to have well managed staff and better management as compared to non-FARA facilities. Capable and qualified staff were almost always present and any absences were well documented. During fieldwork visits, facilities in Lofa or Bong had nearly all staff on duty, professionally dressed in uniforms, with qualified staff sitting in for any staff member that was absent. In the non-FARA supported county of Gbarpolu, we observed a shortage of manpower at the facilities visited with unprofessional staff on duty trained through apprenticeship only. Respondents expressed that FARA capacity building efforts have made CHTs in FARA counties confident to assume contracting-in responsibilities. Data: Other improvements were noted in the area of data collection and management. With regard to data, support from Implementing Partners on data collection and report development, and assistance provided by supportive supervision teams and internal facility checks and balances (with more than one staff member responsible) appear to have contributed to improved data management and a stronger HMIS. At three FARA facilities, we found a high degree of accuracy when cross-checking with CHT records. One CHO in a FARA county reported he had a 75% confidence level in his county data; based on a small sample; our estimate was closer to 90% accuracy. When we reviewed the last two months of the clinics’ HMIS reports, 100% (8 of 8) of the OICs had monthly reports readily available. The team found that there was regular HMIS/DHIS-2 reporting as well as separate reporting by MOH of FARA contributions to the USAID performance indictor database system (FP, HIV, MCH, EPI and child health). A review of quarterly data reports, PBF assessments and ad-hoc information provided on request revealed consistent EPI and PBF reporting, along with extensive HSS reporting. Other health indicators for MCH (including malaria) and FP were also correctly reported. A data accuracy (recording and reporting) test conducted in six FARA facilities in two counties and two non-FARA facilities in one county found a higher level of accuracy in FARA records compared with the non-FARA county. CONCLUSIONS FOR QUESTION 3 Facilities lacked access to consistent, running water, staff lacked knowledge around waste incineration, and there was a general lack of maintenance of medical equipment. Most clinics managed their own water tanks but these are insufficient for year-round support. While almost all clinics had sharps boxes and incinerators, staff did not know how to use them or poorly followed safety protocols to avoid needle-stick injuries. Medical equipment was present in most clinics but oftentimes not maintained and inoperable and staff had little motivation to enact repairs. At the MOH and CHT levels, data were not effectively disseminated or used. The FARA M&E and HMIS unit at the MOH is strong and has in-depth institutional memory, the capacity to “The OIC went to Monrovia for the past weeks for her pay, no midwife since a year ago.” -Staff, Gokala Clinic “We have leadership in place for contracting in because we have a good set up for management, young, trained and energetic staff. We have a ready management available to sit in the seat of IRC.” – Lofa CHT 38 produce a range of data reports rapidly and a commitment to further capacity building of MOH in M&E and data quality, evidenced by the amount of time spent in the field and confirmed by CHTs and facility OICs during fieldwork. However, data reporting procedures are generally followed and data accuracy for HMIS appears adequate. The MOH admitted that it lacked strategies to operationalize data, and that data reporting at the district level is especially weak. There was a lack of defined roles and responsibilities for MOH/CHT supervision. Roles and responsibilities for supervision between the MOH and IPs is not well defined resulting in little to no involvement from DHOs and a weak third-party verification system for performance of contracting-in facilities. Scaling-up of the contracting-in mechanism in all FARA counties, then, may present potential conflicts of interest within the MOH regarding the independent verification of performance linked to PBF. A lack of incentives for facility staff may be stunting performance at clinics. The FARA PBF incentive scheme appears to have a highly positive effect on staff motivation, performance, and attendance. Non-FARA clinics that do not benefit from these schemes had worse staff and performance related indicators. This could signal that staff in general are not motivated to achieve high performance at clinics, perhaps due to a lack of communal purpose/goals and/or low salaries or patronage. Non-FARA facilities did not benefit from integrated, joint supervision by MOH/CHT and IPs, resulting in subpar administrative capacity and performance. In FARA clinics, external support (both monetary and skills training) from NGOs and IPs strengthens staff capacity, service delivery quality, and facility oversight and management. FARA clinics that receive external funds and administrative and technical training saw better performance over time than non-FARA facilities. Further, CHTs benefitted from spillover effects of IP oversight and management. Stronger performance indicators were also seen in FARA clinics that benefit from integrated supervision (MOH, CHT, and IPs participating). While non-FARA clinics receive periodic visits from MOH and CHT, they could benefit from increased, targeted external support, particularly in terms of integrated supervision, mentoring and coaching, and management/administrative training. Lack of funds and a weak supply chain may be affecting the quality of service provision at facilities. FARA-supported facilities benefit from consistent IP funding and strong logistics and commodity management supply chains for drugs. These new systems have positively contributed to reductions in stock-outs. FARA programming was separate from the USAID-funded commodity strengthening program, and the evaluation team was unable to assess FARA’s overall impact on the drug supply chain. However, non-FARA clinics were observed in site visits to have experienced more frequent stock-outs than FARA facilities. A lack of sense of ownership of a facility acts as a barrier to increased performance. Bonuses from the PBF incentive scheme and the accountability and reporting systems that it has tangentially strengthened have resulted in increased sense of ownership and accountability of FARA clinics. FARA clinics have increased leadership and governance over their resources, cooperatively deciding how to best use the administrative bonuses they receive for priority projects that benefit the community at large. CHDC meetings supported under FARA provide a new and useful forum for all community stakeholders to come together and discuss the health issues in their community. These FARA-related byproducts further increase trust in the community for public health services, contributing positively to health services utilization. Non-FARA facilities and staff that do not benefit from incentives and their byproducts may have an arguably weaker sense of pride and ownership and trust of the community, limiting performance and service utilization. 39 5.4 Recommendations for FARA 2 Recommendations for GOL Theme Recommendation Health services/ outputs As stock-outs continue to be a problem for non-FARA facilities, consider integrating other USAID interventions that focus on supply chain strengthening and commodity financing for non-FARA clinics. Study and/or replicate best practices of IPs and NGOs in terms of commodity logistics systems, and transfer these skills to CHT. Consider using a review unit with the MFDP to provide independent third￾party verification, and potentially in a joint manger with the MOH, of health facilities. Assess the difference in FP interventions and supply and demand of FP methods between FARA and non-FARA facilities and catchment areas to help explain the large imbalances in CYPs. WASH and infrastructure For facility level improvements and capacity building: ensure running water and operational separate delivery and post-partum rooms at all facilities. Improve staff awareness about incinerators. Supervision Support effective and regular supervision in non-FARA counties for improved service delivery. Study and/or replicate best practices of IPs and NGOs in terms of supportive supervisory and mentorship systems, and transfer these skills to CHT. Ownership Encourage staff to take ownership of maintenance of equipment as their responsibility. This could be achieved, for example, by incorporating benchmarks/targets for medical equipment maintenance into PBF performance standards. Data Devise and integrate a strategy that links data collection, analysis, and dissemination with program management and implementation into the National Health Policy and Plan. Health stream funding allocation Continually reinforce injection safety precautions as part of systemized Universal Precautions in all healthcare facilities. Provide additional training and annual refresher training to facility staff on Family Planning counseling and technical training for longer lasting methods. Reporting Prioritize and develop a district-level capacity building program, including mentoring, to strengthen district level data collection and reporting (including expenditures). Consider development of a FARA management dashboard providing more detailed analysis of performance (current, comparison with same period 12 months ago, against targets, etc.) Link to financial system for more detailed analysis by USAID. Track health stream output spending at the line-item level to allow for easier tracking of spending, increased transparency, and better programmatic management decisions. Capacity gaps Expedite decentralization of the healthcare delivery system in the areas of planning and implementation. These efforts could directly improve the health system by providing increased training and management skills to the District Health Officers. The DHOs’ roles and responsibilities should be strengthened and supported by the CHT and MOH Central. By empowering the DHOs, OICs, facility staff, and the community, they will feel a greater sense of control and take on more responsibility to ensure that they are getting the best care from their local healthcare system. 40 5.3.2 Recommendations for USAID Theme Recommendation Health Services Current FARA clinical indicators do not cover a wide enough range of health outputs. Revise FARA indicators to reflect more clinical versus administrative and stewardship indicators. Given that quantitative and qualitative evidence suggests that FARA facilities outperform non-FARA facilities in the majority of health outputs among comparable populations, collaborate and introduce the FARA mechanism to other partners for a holistic health system strengthening approach. Supervision Continue support for supervision-related activities such as training and capacity building to increase and maintain effectiveness in service delivery. Consider funding IP-MOH joint supervisory activities. Health Stream Funding Allocation / Reporting Ensure all USAID and MOH staff working with the FARA program understand the current USAID Assistance Automated Directives System (ADS) 200 policies, guidance, and compliance requirements for all aspects that FARA supports. Consider increased participation from the MFDP to provide independent verification of FARA performance. This, or an alternative body, preferably within the GOL system, would be required if the contracting-in mechanism for service delivery is expanded. Establish a performance framework/logframe for FARA 2 and include WASH indicators, VFM indicators and any additional indicators, if any, recommended and agreed by the performance evaluation. Support development of a management dashboard including provision of technical assistance (the USAID GMS project has extensive expertise) to support the MOH. Require disaggregated financial reporting tying inputs to outputs and outcomes for better analysis of economy, efficiency and effectiveness. Require tracking of health stream output expenditures at a line-item level. Capacity Gaps Decentralize health system strengthening and support more supervision capacity building at the district and facility levels for sustainability. Continue to train new, and build capacity of existing, FARA staff to ensure local capacity and sustainability of the FARA mechanism programming. 41 ANNEXES Annex I: Evaluation Statement of Work FINAL EVALUATION STATEMENT OF WORK The end-of-project evaluation for the Fixed Amount Reimbursement Agreement (FARA), 2011-2015 1. Background The purpose of this Statement of Work (SOW) is to describe the conditions of work and terms of reference for an external evaluation firm to conduct an evaluation of USAID’s Fixed Amount Reimbursement Agreement (FARA) with the Government of Liberia (GOL) which supported the provision of high quality facility- and community-based services in three counties in Liberia from September 16, 2011 - December 31, 2015. The evaluation will evaluate the status of results at the end of the FARA, examine the impact of Ebola on service delivery and utilization under the FARA, and investigate the validity of some of the assumptions underlying the FARA. The Mission will use findings from the evaluation to inform strategies and interventions that will be implemented by the FARA 2.0 follow-on project (2016-2020). Overview of the FARA Communities/geographical focus of FARA: Primary FARA counties: Bong, Lofa, Nimba; Accelerated Action Plan (AAP) counties: Montserrado, Grand Bassa, Margibi Activity Number: 669-FARA-A11-11-01 Activity Dates: 2011- 2015 Funding: $ 42,065,256 The Liberian Health Sector FARA, entitled Support to the Ministry of Health and Social Welfare for Implementation of Liberia’s 2011-2021 National Health and Social Welfare Policy and Plan signed in September 2011, was designed to both build the Ministry of Health and Social Welfare’s (MOHSW, now the Ministry of Health) capacity to manage the systems required to successfully implement the agreement, as well as directly support health service delivery through performance-based contracts (PBCs) to NGOs in Bong, Nimba and Lofa counties. The FARA was developed in the context of the gradual transition away from a post-conflict setting to a GOL-managed health sector; building on incremental improvements in health indicators and systems strengthening while maintaining provisions for service delivery and support to national systems. The groundwork for the FARA built on the experience and implementation of the five-year, $65 million USAID/Liberia health project, Rebuilding Basic Health Services (RBHS), which began in November 2008. Under RBHS, the PBCs for health service delivery by NGOs began in July 2009. The USAID funds and primary responsibility for award and management of PBCs with NGOs rested with international NGOs. The MOHSW had only limited involvement in the management of RBHS, with a minority vote on the technical evaluation committee for NGO selection and a secondary role in technical oversight through joint monitoring with RBHS and RBHS-run partner meetings. The development of a Government-to-Government (G2G) approach by the Mission and the GOL transitioned responsibility for PBC management from RBHS to the MOHSW. Under the FARA, responsibility for awarding and managing PBCs was formally transferred to the MOHSW. From 2011 to 2012 the RBHS PBCs with NGOs were phased out, and new contracts were introduced by the MOHSW, beginning with Lofa County in 2011, and followed by Nimba and Bong Counties in 2012. 42 The role of the RBHS Project was revised to focus on technical support, capacity building of the MOHSW (at county and central levels), and infrastructure improvements. In 2014, the GOL and USAID jointly agreed to a programmatic modification to FARA that would allow a specified amount to be set-aside within the FARA budget to focus on and support an ambitious plan specifically concentrated on reducing maternal and neonatal deaths by fifty percent over a three year period. The Accelerated Action Plan (AAP) focuses, in part, on what is needed to achieve the “last mile” – that is, strategies and actions to ensure that women and children in the hardest-to-reach communities have access to quality health care. The AAP closely monitors progress in health outcomes through use of proxy indicators to measure reductions in maternal and neonatal mortality. Six County Health Teams (Nimba, Lofa, Bong, Grand Bassa, Margibi, Montserrado) were engaged to identify and select facilities with critical MNH needs to benefit from the grant. A total of 200 facilities were selected in which the AAP has been implemented. Program Areas Supported through the FARA The GOL agreed to finance the FARA through the National Budget with USAID reimbursing the GOL based on its achievement of predetermined and costed deliverables. Through this arrangement, the FARA made available up to $42 million of USAID assistance to the MOHSW in support of the National Health Plan (NHP) (2007-2011) and in particular, delivery of an Essential Package of Health Services (EPHS) at the primary care level in 3 counties. The three main FARA activities were: 1) MOHSW Technical and Management Support. The FARA provided support for 20 technical and management consultants, assigned to key units within the MOHSW, including the Program Management Unit (PMU), the M&E Unit, the Office of Financial Management (OFM), the Procurement Unit, and the Quality Assurance (QA) and Performance-based Finance (PBF) teams. 2) Supervision Activities by the National Malaria Control Program (NMCP) and the Expanded Program for Immunization (EPI). The FARA outlined supervision of facilities by both the NMCP and EPI unit to ensure quality delivery of high priority services. 3) Procurement of Preventive and Curative Health Services and Commodities. The FARA supported PBCs with NGOs to implement the EPHS in Lofa, Bong, and Nimba Counties. The MOHSW was expected to manage these contracts with technical guidance, supportive supervision, provision of drugs and supplies, and payments for services. This activity also included the management of contracts with NGOs to distribute Insecticide￾treated Nets (ITNs), and procure pharmaceuticals for supported facilities, though these were later removed from the FARA. For each activity, a comprehensive set of time-bound deliverables was developed, and quarterly reimbursements were made conditional upon successful completion and USAID verification of the deliverables. Program Monitoring and Evaluation Reporting and monitoring of the FARA was primarily the responsibility of the MOHSW, with support from County Health and Social Welfare Teams (CHSWT or CHT), which submitted quarterly progress reports to USAID. USAID staff also conducted quarterly field visits to monitor implementation and assess progress against baseline assessment results. Quarterly reports provided an overview of outputs achieved, problems encountered, and other relevant information related to producing the deliverables outlined in the FARA. Specific deliverables outlined additional reporting and other documentation to be provided by the MOHSW, including validation of administrative and service delivery indicators, procurement documents, work plans, management of technical consultants, quality assurance reports, and meeting reports. Deliverables built progressively with each year of implementation of the FARA, and were intended to simultaneously monitor progress 43 while building the capacity of the MOHSW to do so independently. Examples of FARA deliverables, which were linked directly to reimbursement amounts, are shown below: ● Monthly partner meeting reports ● RFP and selection memo for EPHS implementation in Lofa County ● Annual work objectives for Technical and Management Consultants ● Annual supervision/monitoring work plans ● Quarterly supervision/monitoring report countersigned by NGO and CHSWT ● Quarterly data quality harmonization review report ● Quarterly newsletter ● RFP for implementation of EPHS in Nimba and Bong Counties ● Procurement selection memo for NGOs to implement EPHS in Nimba and Bong Counties ● Annual Accreditation and QA report ● Quarterly QA assessment report ● Quarterly RBHS data validation report ● Annual performance appraisals for Technical and Management Consultants ● Post ITN campaign survey report (Year 2) As shown above, FARA deliverables contained a mix of process indicators, procurement functions, and specific activities, such as ITN distribution and annual accreditation. Through the combination of direct program monitoring and evaluation and other MOHSW reporting, the FARA generated a significant volume of data describing performance of the project as well as progress in the health sector. 2. Purpose of FARA Evaluation This performance evaluation is intended to provide an independent and in-depth examination of the overall progress and achievements of the FARA activity in Liberia. The evaluation will identify achievements, performance issues, and constraints related to activity implementation and effectiveness. The evaluation will also identify results and lessons learned from implementation and will provide succinct, actionable recommendations to determine which component(s) of FARA to scale up, modify, or re-design in other ongoing related programs or future procurements in order to improve overall activity performance. To the extent possible, there will be an emphasis on quantitative results achieved as the result of the FARA. Evaluation findings and recommendations will be shared and discussed with USAID/Liberia, USAID/Washington, implementing partners, and relevant GOL partners. Primary objectives of the FARA final evaluation 1. Determine whether transition to the FARA mechanism resulted in an improvement in clinical indicators;2 2. Determine if the “learn by doing” approach utilized in the FARA strengthened MOHSW capacity and health systems, as intended by the designers of the FARA; and 3. Determine whether the FARA was effective in incentivizing organizational transformation in Liberia. a) Identifying prospects for shifting health service delivery to the MOHSW under the FARA motivated improvements in specific functions, notably in procurement and supportive supervision, b) If the pressure to meet deadlines on FARA deliverables has led to improved cross‐unit coordination and timeliness in Ministry operations, which are early signs of the potential for larger‐scale organizational transformation. 4) Determine if the FARA effectively reduced transaction costs. a) The harmonization across service delivery models and related administrative processes (financing arrangements and reporting requirements) produced a significant reduction in 2 The first three objectives are variations of the original objectives of the FARA Mid-term Evaluation, 2013. 44 transactions costs, especially at the county level where the demands of multiple donors are a serious and persistent drain of scarce staff time. b) The fixed cost character of reimbursements provided incentives for the Ministry to identify and implement more cost‐efficient integration of supportive supervision. c) Determine if the transition to a government-to-government approach from a USAID implementing partner generated cost savings for USAID, which was the perspective of the MOHSW. 3. Evaluation Design and Methodology Evaluation Questions This draft matrix has been developed by USAID/Liberia but we kindly request review and feedback on the methodology, per the implementing partner’s technical expertise. Question Level of Effort Data Collection Method Data Source Selection Criteria Data Analysis Method (See attached list of questions) Key informant interviews, document reviews Key Informant Interviews, project reports Project reports, notes from meetings with MOH, MOH county￾level health data/ statistics, key informant interviews Interviews, health assessments Disaggregate by gender County/ Communit y location Key informant interviews, FGD’s, document review, quarterly reports Interviews, NGO budget reports, other reports/records from the project, NGO location Key informant interviews, FGD’s GOL MOH, MOF, NGO leaders, implementing partners Key partners, organization type Key informant interviews, FGD’s GOL MOH, other key govt. ministries, implementing partners Organization type, key partners Key informant interviews, FGD’s Implementing partners, key GOL ministries Organization type, key partners Key informant interviews, FGD’s/ hot wash (after action review) Implementing partners, GOL, MOH, NGO’s, civil society organizations, Organization type, key partners 45 The analysis and findings regarding each of the evaluation questions should be followed by specific, tailored, realistic and actionable recommendations aimed at improving outcomes for USG investments in Liberia’s health sector. 4. Evaluation Deliverables (Data Sources) Evaluation deliverables include: a. Evaluation Team Planning Meeting (s) b. Inception Plan. This will include: • Work Plan • Data Collection Instruments c. Debriefings with USAID/Liberia, GOL and Implementing Partners – The evaluation team will: • Do a mid-evaluation “touch base” with USAID highlighting initial impressions, challenges, questions. • Present the major findings from the evaluation to USAID/Liberia and partners through a PowerPoint (or similar) presentation. The debriefing will cover initial findings, conclusions and preliminary recommendations. The team, in consultation with USAID, should consider doing two to three presentations: one to USAID, one to partners, and one to the MOH. This will be determined by the presence of sensitive information, if any, in the report. d. Original data and data sets -- Copies of secondary quantitative data sets, transcripts of interviews and focus groups, and notes from direct observations. Quantitative data sets should be submitted to the DDL, per Agency policy. e. Draft Evaluation Report - A draft report on the findings and recommendations should be submitted to USAID/Liberia within two weeks after the in country work is conducted. The written report should clearly describe findings, conclusions, and recommendations, and include an Executive Summary. USAID will provide comments on the draft report within fifteen working days of submission. The draft should also be submitted to the GOL for its comments; f. Final Report - The Team will submit a final report that incorporates the GOL’s and Mission’s comments and suggestions no later than five days after final, written comments on the team’s draft report have been submitted by all parties. The final evaluation report should meet the following criteria: i. The report should be in line with USAID Evaluation Policy (see Appendix I – Criteria to Ensure the Quality of the Evaluation Report) and USAID Secretariat Style guide ii. The report should be no longer than 30 pages, excluding one page summary, executive summary, table of contents, and annexes. iii. The report should include a one page summary of high level findings in a visual and easily digestible format iv. The report should include a 3-5 page Executive Summary highlighting findings and recommendations. v. The report should represent a thoughtful and well organized effort to objectively respond to the evaluation questions. vi. The report shall address all evaluation questions included in the SOW. vii. Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides shall be included in an Annex in the final report. 46 viii. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, etc.). ix. Evaluation findings should be specific, concise and supported by strong quantitative or qualitative evidence. x. Recommendations should be action-oriented, practical, specific, and evidence-based. xi. The final report should be edited and formatted. xii. The final report should be submitted electronically along with 2 printed and bound copies. xiii.The evaluation team must submit the final evaluation to the Development Experience Clearinghouse. The following content should be included in the final report: Executive Summary - concisely state the evaluation purpose, methodology, key evaluation questions, key findings and recommendations; Introduction – Evaluation context, including a summary of any relevant history of FARA program, demography, socio-economic status etc.; FARA Program description - brief overview of the FARA program including the development hypothesis, key intervention areas and implementation arrangement/approach (may rely heavily on existing documents); Purpose of the Evaluation - purpose, audience, and synopsis of task; Evaluation design and Methodology - describe evaluation design and methods, including sampling procedure; Findings/Conclusions - describe and analyze findings for the project using graphs, figures and tables, as applicable supported with concise narratives; Lessons Learned - provide a brief of key technical and/or administrative lessons on what has worked, not worked, and why, for immediate corrective measures and future project or relevant program designs; Recommendations – prioritized for each key question; should be separate from conclusions and be supported by clearly defined set of findings and conclusions. Include recommendations for future project implementation or relevant program designs and synergies with other USAID/Liberia health projects and other donor interventions as appropriate; Annexes – to include statement of work, list of documents reviewed, tools used, interview lists, meetings, and data tables. Annexes should be pertinent and readable. The report will be submitted electronically. The final report will be edited/formatted by the contractor and provided to USAID/Liberia seven working days after the Mission has reviewed the content and approved the final revised version of the report. The final evaluation report must be 508 compliant and comply with the USAID Evaluation Policy as it relates to performance evaluations, and should use the criteria for quality evaluation reports listed in Appendix I of the Evaluation Policy http://www.usaid.gov/ sites/default/files/documents/1868/USAIDEvaluationPolicy.pdf 5. Team Composition/Technical Qualifications and Experience USAID/Liberia recommends the following staffing structure for the evaluation: i. An international mid- to senior-level evaluation specialist Team Leader3 with extensive experience in evaluating primary health care and community-based service delivery activities in developing countries. At least ten (10) years of experience in evaluation management, and qualitative data collection and analysis; experience in conducting evaluations and designing 3 We can be flexible regarding which staff member is the Team Leader. 47 performance evaluations, preferably of USAID projects. Ability to produce high quality evaluation reports in English is essential. Strong interpersonal skills are required. ii. A mid- to senior- level public health expert (international), with experience in designing and/or evaluating government-to-government health mechanisms in developing countries, preferably through USAID; at least ten (10) years of public health experience, and some experience managing or implementing research programs is preferable. Strong interpersonal skills and American English language speaking and writing skills also essential. iii. A host country or international senior or mid-level social scientist with strong analytical skills and gender expertise, specifically in designing and evaluating health programs with a strong emphasis on gender. The incumbent must be able to conduct interviews and focus group discussion, analyze the resulting data, and be familiar with qualitative and quantitative data collection methods. Ability to conduct interviews and discussions in at least one local Liberian language. Strong American English language speaking and writing skills also essential. iv. A host country or international senior or mid-level social scientist with strong financial or costing expertise. Strong American English language speaking and writing skills are essential. v. A national evaluation specialist with at least five years relevant experience and strong logistics and planning skills. Ability to communicate clearly in American English. USAID leaves to the offeror’s discretion other necessary team members/staff for the evaluation (e.g., Logistics, scheduling, translation, data analysis). Aside from the above mentioned key personnel, the offeror must decide how the evaluation team should be structured in order to successfully address the evaluation questions. All attempts should be made for the team to be gender balanced and to include local (Liberian) experts. A statement of potential bias or conflict of interest (or lack thereof) is required from each team member. USAID may propose internal staff from USAID/Liberia or from Washington to accompany the team in this evaluation as observers. As observers, their role will be to provide, when asked, background information and to reply to the external evaluators’ questions. They will review and comment on the report for accuracy, but evaluators may accept or reject comments. The final report should reflect the opinions of the external evaluators and is the sole responsibility of the selected evaluation team. The contractors will officially report to the Mission’s M&E Officer and technical guidance/leadership will be provided by the Government Agreement Technical Representative (GATR) and A/GATR. 6. Timeline: Scheduling and Logistics Funding and Logistical Support The contractor will be responsible for all international and in-country administrative and logistical support, including identifying and fielding appropriate consultants (International and local). The evaluation team should be able to make all logistical arrangements including vehicle rental for travel within and outside Monrovia and should not expect any logistical support from the Mission. The team should also make their own arrangements for venues for team meetings, and equipment support for producing the report. Schedule Evaluation team members are authorized and expected to work a six-day week. Travel over weekends may be necessary. Work should commence as soon as practicable, but no later than ---. For planning purposes, contractors should be aware of Liberian and US holidays during the evaluation time frame. 48 Annex II: Interview Respondent List NAME TITLE CATEGORY NATIONAL Francis N. Kateh Chief Medical Officer Ministry of Health Sophie Parwon Manager, Global Fund Ministry of Health Vera Mussah Manager, PBF Ministry of Health Louise Mapleh Manager, FARA Ministry of Health Toagoe T. Karzon Comptroller, OFM Ministry of Health Shelford Somwarbi Financial Officer, FARA Ministry of Health Jacob L. N. Wapoe Director, Procurement Ministry of Health J. Mike Mulbah FARA M&E Officer Ministry of Health Rev. Tijli Tyee Chief Pharmacist Ministry of Health Stephen Gbaryan Director, HMIS Ministry of Health Adolphus D. Forkpa Deputy Minister for Fiscal Affairs Ministry of Finance, Development, and Planning Julius H. Thompson Assistant Director, Health & Social Development Sector Ministry of Finance, Development, and Planning Hanson S. Kiazolu Comptroller & Accountant General Ministry of Finance, Development, and Planning Ernest Kromah Gaie Country Director, Africare Implementing Partner Garfee Williams Deputy Country Director, MSH Implementing Partner Anjuli Shivshanker Country Director, IRC Implementing Partner Rose Macauley Country Director, JSI & former RBHS Chief Implementing Partner LOFA COUNTY George S. Dunor Superintendent Ministry of Internal Affairs D. Mark S. Tengbeh PBF Focal person County Health Team George S. Dunor Superintendent, Lofa County County Health Team Dr. Ballah, MD County Health Officer County Health Team James Gizzie Clinical Supervisor/Acting CHDD County Health Team Wolobah Y. Moore County Pharmacist County Health Team Blima R. Sirleaf IPC&WASH focal person County Health Team Timothy Y. Topkah TB/HIV focal person County Health Team Varlee M. Kamara Cold Chian Supervisor County Health Team William K. Sherman Health Promoter focal person County Health Team Dormowah Samuka MCH staff County Health Team Esther Y. Argba RH Supervisor County Health Team Kpana Sevelee RN, MCH Bolahun Health Center Rebecca J. Kollie RN, Screener Bolahun Health Center Blama B. Kollie Vaccinator Bolahun Health Center Kessellie William Ambulance Driver Bolahun Health Center Miatta M.Johnson RN, MCH Bolahun Health Center Alphonso M. Kerkula RN, Screener Bolahun Health Center Maima Dawolo RN, MCH Bolahun Health Center Ballah Karkay Dispenser Bolahun Health Center James M. Kordor Laboratory Assistant Bolahun Health Center Hawa Kanneh Cleaner Bolahun Health Center O-You-You S. Akoi Registrar Bolahun Health Center Margret Jusu TTM Bolahun Health Center 49 Kebbeh Zubah RN, Acting OIC Bolahun Health Center Lorpu Gbolumah RN Borkeza Clinic Lorpu Sumo CM Borkeza Clinic Edward Beyan LPN/Screener/Surveillance focal person Borkeza Clinic James Kanneh Lab. Assistant/IPC focal person Borkeza Clinic Roland Toto Massah Vaccinator Borkeza Clinic Roland M. Dolo PA/OIC Borkeza Clinic Yassah K. David OIC Shello Clinic Kabeh S. Paye RN Shello Clinic William T.Tatapaa Screener Shello Clinic Anderson K. Tatapaa Screener Shello Clinic Benson F. Kamara Vaccinator Shello Clinic Albert Musa Dispenser Shello Clinic Tellie Tumbay Registrar Shello Clinic Prince S. Taylor Security Shello Clinic Hawa Jowoe Cleaner Shello Clinic Wolobah Z. Kowoyan Dispenser Kpakamai Clinic Ma-fata K. Dunor RN Kpakamai Clinic James O. Mulbah Vaccinator Kpakamai Clinic Wolobah Johnson OIC Kpakamai Clinic BONG COUNTY Gormah Gertrude Cole RH Supervisor County Health Team Bornor Korlewala RH/MCH Supervisor County Health Team Peter yarkpawolo HIV/TB Coordinator County Health Team Melepalay K. Sumo Acting CHDD County Health Team Nadu Paasewe RN Salala Clinic Florence Franklin Student Nurse Salala Clinic Winifred Williams RN Salala Clinic Josephine Ballah Vaccinator Salala Clinic George Walters Vaccinator Salala Clinic Samueal Turmi Dispenser Salala Clinic Yalafai Thomas HIV Counsellor Salala Clinic Augustine Seh Registrar Salala Clinic Miatta Yekee OIC Salala Clinic Wilma Dormea RN/Acting OIC Naama Clinic James T. Nuta RN/Screener Naama Clinic Abraham Flomo Vaccinator Naama Clinic GBARPOLU COUNTY Anthony Tucker, MD County Health Officer County Health Team Bennie T Clerk Clinical Supervisor/ Acting CHDD County Health Team Jusu Kpanah Child Survival focal person County Health Team Maima N. Kollie RH Supervisor County Health Team Augustine Saye EHT-Coordinator County Health Team O. Wanga Wanley OIC Totoquelleh Clinic Sylvester M. Gbarla Aide Nurse Totoquelleh Clinic Rennie S. Norbor Dispenser Totoquelleh Clinic Sekou S. Gbessay Registrar/Security Totoquelleh Clinic Maude K. Kerkulah CM/Acting OIC Bambuta Clinic Korto L. Togbah RN Bambuta Clinic James G. Scott Vaccinator Bambuta Clinic 50 Clarence Sirleaf Vaccinator Gokala Clinic Marcus O. paye Registrar Gokala Clinic A. Momo Kamagie Dispenser Gokala Clinic Elizabeth R. Sameli OIC Gbaryama Clinic Catherine B. Freeman Registered midwife Gbaryama Clinic Esther B. Cartor J/S Gbaryama Clinic Boima T. Kamara Lab. Aide Gbaryama Clinic Bendu Kamara Registrar Gbaryama Clinic FACILTIES VISITED Lofa Bong Gbarpolu Bomi Bolahun Health Center Salala Clinic Gokala Clinic Bonjeh Clinic Borkeza clinic Naama Clinic Gbaryama Clinic Dagweh Clinic Shello Clinic Gbartala Clinic Totoquelleh clinic Jenneh Clinic Bazagizia Clinic Zebay Clinic Bambuta Clinic Sass Town Clinic Kpakamai Palala Clinic 51 Annex III: Data Collection Instruments Key Informant Interview Standard Questions Thank you for taking time from your day to meet with us. Liberia Strategic Analysis (LSA) has been engaged by USAID/Liberia to conduct an external performance evaluation of the Fixed Amount Reimbursement Agreement (FARA) implemented by the Ministry of Health. The purpose of this performance evaluation is to provide an independent and in-depth examination of the overall progress of the FARA activity in Liberia and identify results and lessons learned from implementation. To be clear we are not evaluating you or your organization in any way. Your answers will not have any impact on your involvement with the FARA. Your opinions are important to us and we will not include your name in our records or in our report; we hope that you will feel free to share your information and opinions with us. Our conversation will last about 45-60 minutes. We will take notes of our discussion (or record the conversation with your permission). 1. Kindly describe your role and responsibility regarding health service provision as a part of the CHT. 2. Looking at these five areas of health service provision (PMI, EPI, FP, MCH and WASH), to what extent has the MOH, other system or program enabled the efficiency and resilience of the health sector? 3. How much can you attribute to the MOH, other program or system in regards to improvements in the Health Information System Management? 4. To what extent has governance and leadership improved through the MOH, that program or system? 5. Are local authorities and/or community representatives involved in project planning or oversight? If yes, please explain. Thank you very much for sharing your thoughts and experiences. Do you have any questions for us? Are there other comments that you wish to make? 52 Focus Group Discussion Standard Questions Facility Staff Note: 1. Respondents to fill out an attendance sheet that will include demographic questions Thank you for taking time from your day to meet with us. Liberia Strategic Analysis (LSA) has been engaged by USAID/Liberia to conduct an external performance evaluation of the Fixed Amount Reimbursement Agreement (FARA) implemented by the Ministry of Health. The purpose of this performance evaluation is to provide an independent and in-depth examination of the overall progress of the FARA activity in Liberia and identify results and lessons learned from implementation. To be clear we are not evaluating you or your organization in any way. Your answers will not have any impact on your involvement with the FARA. Your opinions are important to us and we will not include your name in our records or in our report; we hope that you will feel free to share your information and opinions with us. Our conversation will last about 60+ minutes. We will take notes of our discussion (or record the conversation with your permission). 1. What changes has been achieved since FARA was introduced in 2011 to the health system particularly with these health streams? a. PMI b. EPI c. FP d. MCH e. WASH 2. Please describe the Performance Based Financing (PBF) under the FARA project and its impact on specific services and overall facility performance 3. If you would make changes or modify the FARA program, what are the things you will like to change or modify? Thank you very much for sharing your thoughts and experiences. Do you have any questions for us? Are there other comments that you wish to make? 53 Focus Group Discussion Standard Questions Facility Clients Note: 1. depending upon availability of clients, these may be a series of individual interviews 2. Respondents to fill out an attendance sheet that will include demographic questions Thank you for taking time from your day to meet with us. Liberia Strategic Analysis (LSA) has been engaged by USAID/Liberia to conduct an external performance evaluation of the Fixed Amount Reimbursement Agreement (FARA) implemented by the Ministry of Health. The purpose of this performance evaluation is to provide an independent and in-depth examination of the overall progress of the FARA activity in Liberia and identify results and lessons learned from implementation. To be clear we are not evaluating you or your organization in any way. Your answers will not have any impact on your involvement with the FARA. Your opinions are important to us and we will not include your name in our records or in our report; we hope that you will feel free to share your information and opinions with us. Our conversation will last about 60+ minutes. We will take notes of our discussion (or record the conversation with your permission). 1. Why are you visiting the facility today? 2. Is this the closest facility to where you live? 3. Do you visit other health facilities and if so why? 4. Have you or your family received any of the following services in the past year for: a. Malaria testing and/or treatment b. Child Immunisation c. Family Planning d. MCH e. WASH 5. What is your overall impression of the facility? 6. Are you treated well and with respect by the staff? Do they explain your treatment/diagnosis? Thank you very much for sharing your thoughts and experiences. Do you have any questions for us? Are there other comments that you wish to make? 54 Annex IV: Analysis of Evaluation Question 2: ADS Compliance In determining FARA I compliance with USAID Assistance Automated Directives System (ADS) 200 policies and guidance in areas such as Programming Policy (rev: 2014), this review considered MOH documentation, policies and guidelines, key informant interviews, focus group discussions, and observations at MOH Central Office, County Health Offices, and healthcare facilities. The Automated Directives System (ADS) 200 Guidance clearly lays out and defines the list of approved uses for Global Health Programs Account funding. Funding is broken down into all major sectoral priorities (MCH, WASH, nutrition, FP, malaria, HIV/AIDS, etc.). It also lays out guidance for programming with vulnerable children and cross-cutting program areas such as HSS, gender, salary payments, and integration, among others. Per USAID/Liberia guidance and in order to answer the evaluation question, we attempted to distinguish the line-item spending at the MOH to ensure that USAID funds under FARA aimed at specific sectoral or cross-cutting areas were indeed spent on programming for those specific areas. In this vein, the team sought to distinguish MOH spending by specific FARA health output (MCH, malaria, EPI, FP, WASH), irrespective of IP or county, in order to determine the percentage expended of total FARA budget over 5 years by each health output. To obtain this information, the evaluation team requested financial expenditure data from the MOH. After several attempts, the team was ultimately unsuccessful in obtaining expenditures for specific health outputs. While the MOH provided financial expenditures, these were tracked at a level higher than health outputs, which were rolled up into more general cross-cutting line items. For example, one line item around supervision - MOH Supervision Activities (EPI Unit and NMCP) – encompassed both EPI and malaria activities. Another line item, Accelerated Action Plan (AAP), aggregated, among many things, FP and MCH activities. While this financial reporting did not allow us to measure the expenditures by FARA health output specifically, we were however able to make general conclusions about compliance. In compliance: The following findings pertain to activities that were in compliance with applicable guidelines. • In terms of the “direct impact” directives of ADS 200 guidelines (rev: 2014), this review confirms that FARA I was compliant. • Review of ADS 200 guidelines confirms compliance with the allowable and priority activities for MCH, MCH integration (malaria, FP, HIV), EPI and WASH. • FARA was fully compliant with all salary payment guidelines, including the payment of “bonuses or incentives to personnel (public or private sector) who meet performance-based criteria that were directly linked to achieving program goals.” • FARA was compliant with Helms and Siljander Amendments regarding abortion and forced sterilization. • There was full compliance with cross-cutting areas including i) integration, ii) HSS and iii) HSS in fragile/conflict states. Uncertainties/weak compliance: The following findings pertain to activities where compliance was uncertain or weak with regard to applicable guidelines. 55 • There was some evidence of non-compliance with injection safety and potential needle-stick injury. For example, from direct observation in four facilities visited, staff were exposing themselves to needle-stick injury due to inappropriate footwear and unattended sharps within easy reach/risk of falling. • Family Planning compliance had some weaknesses regarding potential gaps in counselling for longer lasting methods. This was evidenced in the DHIS-2 and the low LLM CYP contribution to total CYP provided. CONCLUSIONS FOR QUESTION 2: ADS COMPLIANCE The FARA mechanism is highly compliant with most USAID ADS policies and guidance, though some areas of uncertainty remain. FARA is compliant with USAID Assistance ADS 200 policies and guidance in areas such as Programming Policy (rev: 2014), specifically with regards to i) integration, ii) HSS and iii) HSS in fragile/conflict states. There is also full compliance with guidelines for MCH, Malaria, EPI, integration, HSS, salary payment guidelines, and Helms and Siljander Amendments. Compliance was unclear or weak with malaria investments, informed choice principles for new FP acceptors, injection safety protocols, and family planning, particularly around low LLM CYP contribution. 56 Annex V: Disclosure of any Conflicts of Interest 57 58 59 60 61 U.S. Agency for International Development - Liberia 502 Benson Street Monrovia, Liberia