1 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Social Impact under the Liberia Strategic Analysis activity. MIDTERM EVALUATION REPORT USAID/Liberia Partnership for Advancing Community￾Based Services Midterm Performance Evaluation Partnership for Advancing Community-based Services Midterm Performance Evaluation Cover photo: Community members in Nimba County; hand-dug well and pump in Lofa County; Community Health Assistants in Bong County. Photo credits: Alfredmy Chessor, Ravi M. Ram, Albert J. Thompson, 2017 USAID/LIBERIA PARTNERSHIP FOR ADVANCING COMMUNITY-BASED SERVICES MIDTERM PERFORMANCE EVALUATION: MIDTERM EVALUATION REPORT September 2017 Liberia Strategic Analysis Contract No: AID-669-C-16-00002 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. CONTENTS Acronyms......................................................................................................................................................................... i Executive Summary...................................................................................................................................................... iv Activity Background......................................................................................................................................................1 Evaluation Purpose and Evaluation Questions.......................................................................................................3 Evaluation Methods and Limitations.........................................................................................................................5 Findings, Conclusions, and Recommendations.....................................................................................................10 Findings......................................................................................................................................................................10 Recommendations..................................................................................................................................................30 Utilization Plan.............................................................................................................................................................35 Annexes.........................................................................................................................................................................37 Annex I: Evaluation Statement of Work, .........................................................................................................37 Annex II: Evaluation Methods and Limitations.................................................................................................46 Annex III: Data Collection Instruments............................................................................................................53 Annex IV: Sources of Information......................................................................................................................59 Annex V: Respondents for Key Informant Interviews...................................................................................61 Annex VI: Pairwise Correlation Coefficients Table .......................................................................................63 Annex VII: Difference in Differences Analysis Results Table.......................................................................64 Annex VIII: Difference in Differences Analysis Figures .................................................................................65 Annex IX: Segmented Regression Analysis Results Table ............................................................................68 Annex X: Segmented Regression Analysis Figures.........................................................................................69 Annex XI: CBA Parameters and Data Sources...............................................................................................71 Annex XII: Cost-Effectiveness Literature Review ..........................................................................................73 Annex XIII: Comparison of NCHA Program Implementation: PACS and Other Models....................75 Annex XIV: PACS Hand-Dug Wells in Relation to GOL and USAID Standards for Water Quantity (Yield)........................................................................................................................................................................76 Annex XV: List of Field Sites Visited for Data Collection............................................................................80 Annex XVI: Disclosure of Conflicts of Interest..............................................................................................81 FIGURES AND TABLES Table 1: Number of Key Informant Interviews and Focus Group Discussions Targeted and Reached.......6 Table 2: CBA Intervention Packages, Primary Activities, and Beneficiary Groups by PACS Result Area ...7 Table 3: PACS Performance Summary (Bong, Lofa and Nimba Counties)........................................................10 Table 4: Summary of PACS Component Performance ..........................................................................................13 Figure 1: Difference in Differences: A Statistical Comparison of the Measles Coverage in PACS and Non￾PACS Districts before and after PACS Implementation ........................................................................................16 Figure 2: Segmented Regression: A Statistical Comparison of Severe Malaria Cases before and after PACs Implementation in Nine Districts of Bong, Lofa, and Nimba Counties..................................................17 Table 5: Cost-Benefit Analysis of PACS Interventions, Ranked by BCR............................................................19 Figure 3: Comparison of Three PACS CHA Scenarios: (1) Stipends Ending in December 2017; (2) Stipends Paid Over 5 Years, No Drugs or Commodities; (3) Stipends Paid, Drugs and Commodities Provided Over 5 Years...................................................................................................................................................20 Figure 4: Map of PACS CLTS Sites and Wells in Bong, Lofa and Nimba Counties, July 2017......................26 i ACRONYMS ACT Artemisinin-Based Combination Therapy ALMEP Activity-Level Monitoring and Evaluation Plan ARI Acute Respiratory Infection BCC Behavior Change Communication BCR Benefit-to-Cost Ratio CBA Cost-Benefit Analysis CEA Cost-Effectiveness Analysis CDCS Country Development Cooperation Strategy CHA Community Health Assistant CHC Community Health Committee CHDD Community Health Department Director CHFP Community Health Focal Person CHSS Community Health Services Supervisor CHT County Health Team CHV Community Health Volunteer CHW Community Health Worker CLTS Community-Led Total Sanitation CSO Civil Society Organization DALY Disability Adjusted Life Year DHIS-2 District Health Information System-2 DHS Demographic and Health Survey DID Difference-in-Differences EEF Ebola Emergency Fund EHT Environmental Health Team ETL Education Through Listening EQ Evaluation Question EVD Ebola Virus Disease FGD Focus Group Discussion FP Family Planning FY Fiscal Year GC Global Communities ii gCHV General Community Health Volunteer GOL Government of Liberia HMIS Health Management Information System iCCM Integrated Community Case Management IPTp2 Intermittent Preventive Treatment in Pregnancy- Second Dose IRC International Rescue Committee IRR Internal Rate of Return KII Key Informant Interview LiST Lives Saved Tool LOE Level of Effort LSA Liberia Strategic Analysis M&E Monitoring and Evaluation MCH Maternal and Child Health MOGCSP Ministry of Gender, Child, and Social Protection MOH Ministry of Health MPW Ministry of Public Works NCHA National Community Health Assistant NGO Non-Governmental Organization NHPD National Health Promotion Division NPV Net Present Value NWSHP National Water, Sanitation & Hygiene Promotion ODF Open Defecation Free OPV3 Oral Polio Vaccine – Third Dose OVC Orphans and Vulnerable Children p P-value PACS Partnership for Advancing Community-Based Services PIDS Performance Indicator Database System PIP Performance Improvement Plan PPAL Planned Parenthood Association of Liberia PPP Purchasing Power Parity PSI Population Services International PSM Propensity Score Matching PV Present Value QALY Quality Adjusted Life Year iii RBHS Rebuilding Basic Health Services RHFP Reproductive Health Focal Person SI Social Impact, Inc. SOW Statement of Work TA Technical Assistance TOT Training of Trainers TT Tetanus Toxoid TTM Trained Traditional Midwives TWG Technical Working Group USAID United States Agency for International Development WASH Water, Sanitation and Hygiene WHO World Health Organization WQ Water Quality WTP Willingness-to-Pay YMCA Young Men's Christian Association iv EXECUTIVE SUMMARY EVALUATION PURPOSE AND EVALUATION QUESTIONS This midterm performance evaluation provides an independent and in-depth examination of the progress of the Partnership for Advancing Community-Based Services (PACS) activity in Liberia. The evaluation measured results, examined the impact of health systems support on service delivery improvement and management efficiencies, and investigated the effectiveness of specific technical assistance and capacity building approaches. USAID will use findings from the evaluation to inform more strategic investment in the sector and continue support to the delivery of community health services. The key intended users are USAID, PACS implementing partners, and Government of Liberia (GOL) partners including the Ministry of Health (MOH), Ministry of Public Works (MPW) and Ministry of Gender, Children and Social Protection (MOGCSP). Secondary users include donors, GOL health facilities and local authorities, non-governmental organizations (NGOs), and other Civil Society Organizations (CSOs) working in similar settings. Four evaluation questions (EQ) framed and guided the PACS midterm performance evaluation, as below. EQ1: Performance: To what extent has progress been made in achieving the objectives under PACS? • Which components have been most successful in meeting the overall objectives of PACS and at what level of engagement, with results disaggregated by gender? • How have the community health service interventions contributed to improving the health of women and men in all PACS districts? Please discuss in terms of the highest-level outcome indicators available, as well as qualitative information. EQ2: Cost-Benefit Analysis: Should there be budget constraints, what interventions should be kept and which should be discontinued? Please discuss and provide a prioritized list using cost effectiveness or cost-benefit analysis, as appropriate. EQ3: MOH Policy Changes and Community Health Assistants (CHA): How has the recent MOH policy change and transition from Community Health Volunteers (CHV) to CHAs affected PACS community health and WASH implementation, with attention to sustainability of the CHA program? • How does the transition from CHVs to CHAs, particularly the revised remuneration package, exert different effects on each of the four result areas of the PACS implementation model (1- capacity building, 2-health service delivery, 3-community behaviors and demand, and 4-water, sanitation and hygiene)? • Based on the design of the CHA program and its incorporation into PACS’ model of community health, describe factors that may affect the sustainability of the CHA program as well as any opportunity costs in relation to diversion of resources from planned activities. Include gender and social factors in this analysis, as relevant. • What are the recommendations for redirecting PACS’ focus to align more with MOH priorities? EQ4: Integration of Community Health and Water, Sanitation and Hygiene (WASH): How do the results from the implementation of the WASH component impact the community health outcomes, including gender as a factor? • To what extent has the capacity of WASH personnel been built to effectively manage water and sanitation infrastructures? • To what degree are PACS supported WASH activities likely to be sustainable (technically, socially, financially and environmentally), considering gender and relevant social factors? v • To what extent does the quantity of water supplied by PACS supported facilities (boreholes, dug wells, etc.) meet World Health Organization (WHO) or national standards? • To what extent has the use of a water quality assurance plan, if any, contributed thus far to meeting and ensuring the standards for quality of water supplied to communities? ACTIVITY BACKGROUND The overall purpose of PACS is to advance USAID Liberia’s Country Development and Cooperation Strategy (CDCS), Development Objective 3, “Improved Health Status of Liberians." PACS is designed to contribute to the CDCS through four results: 1. Broadened capacity of central MOH, County Health Teams (CHT), local CSOs and NGOs to implement and manage community services; 2. Increased service quality and availability of community-based health and social welfare services; 3. Improved health-seeking behavior and practices; 4. Access to safe WASH services. The expected results of PACS are closely aligned with those of the National Health Policy and Plan (2011- 2021), the Investment Plan to Building a Resilient Health System (2016-2021), and the Revised Community Health Services Policy and Strategic Plan (2016-2021). Implementation has been done in close collaboration with the MOH and the CHTs in the USAID priority counties of Bong, Lofa, and Nimba, with three additional counties (Margibi, Montserrado, and Grand Bassa) added to the PACS scope for a limited time under the Ebola Emergency Funding. In addition to being aligned with specific policies and strategies of the MOH, MOGCSP, and MPW, PACS also addresses the GOL’s Poverty Reduction Strategy Paper Pillar 3 and post-Ebola objective: “To rehabilitate infrastructure and rebuild systems to deliver basic service in order to create the conditions and linkages needed to achieve broad-based growth and poverty reduction.”1 EVALUATION DESIGN, METHODS, AND LIMITATIONS The PACS midterm performance evaluation was designed to answer the four EQs and to focus on areas of implementation that were ongoing or would continue, so that findings and recommendations could be applied to improve implementation and results. An emphasis was placed on obtaining the highest level of outcome data for health, prioritizing county, district, and community-level outcomes over national-level outputs. Retrospective longitudinal data on health service utilization was drawn from the Liberia Health Management Information System (HMIS) and PACS’ databases, including information reported to the USAID Performance Indicator Database System (PIDS). Original qualitative data was collected from a sample of key stakeholders at the central level, comprising the MOH, MPW, and MOGCSP; county and district level counterparts, and community level representatives including CHTs, Community Health Services Supervisors (CHSSs), CHAs, Natural Leaders, Trained Traditional Midwives (TTMs), WASH Entrepreneurs, shopkeepers, orphanage staff, clan chiefs, town chiefs and lay community members. Data was collected from the three original PACS counties (Bong, Lofa and Nimba) and from two comparison counties (Grand Gedeh and Grand Cape Mount) to assess the PACS model of integrated community health and WASH interventions. This evaluation did not include the three additional ‘PACS extension’ counties of Montserrado, Margibi and Grand Bassa, where support was scheduled to end in December 2017. 1 PACS ALMEP, July 2016. vi Field work included 51 semi-structured Key Informant Interviews (KIIs), 33 Focus Group Discussions (FGDs), and 15 benefit valuation sessions and other consultations, with 565 total respondents, of whom 369 (65%) were male and 196 (35%) female. Data collected during KIIs and FGDs were captured and processed systematically using a coding system for each topic area. These data were analyzed using NVivo qualitative analysis software; this process informed development of key findings, conclusions and recommendations that make up the bulk of this report. A mixed-method cost-benefit analysis (CBA) examined nine PACS intervention packages within the four PACS result areas and included three PACS-specific Community Health Assistant (CHA) scenarios. Cost data was primarily sourced from PACS financial data. Benefits were identified and valuated using a mixed￾method approach including Willingness-to-Pay (WTP), and morbidity and mortality estimation using Spectrum v5.51 Lives Saved Tool (LiST), as well as statistical and direct modeling. A total of eight CBA sessions were conducted with beneficiary groups including 63 PACS-supported CHAs, 53 community members, and 14 CSO representatives. These sessions focused on the identification and valuation of benefits as well as the validation of CBA assumptions. CBA Builder v1.4, an Excel based cost benefit analysis tool, was used to compile and compute economic metrics presented in this evaluation. The results of the CBA are presented as benefit-to-cost ratios (BCR), with values above 1.0 indicating a positive economic return and values below 1.0 showing a negative return. Challenges encountered included a strong male bias among interview and discussion respondents (65.6% male and 34.4% female), reflecting the structural gender bias in health workers at all levels of the health system. Special efforts were made to obtain perspectives and information from female health workers and community members. Quantitative data sources were generally considered to be prone to data quality problems, including incomplete or inconsistent data in the District Health Information System-2 (DHIS-2) and Akvo Flow. PACS databases were reported to be of limited quality, with recent efforts to improve those already underway. No systematic bias in quantitative data was found, and as a result, some outcome￾level data may be undetected because of additional random variance in the datasets. Several efforts were made to establish interviews with key representatives at MOH and MPW, but some of those representatives were unable to meet with the evaluation team for interviews. A final debrief with these representatives did not take place. Consequently, the recommendations are made without the benefit of feedback from some GOL representatives. FINDINGS AND CONCLUSIONS EQ1: Performance PACS is largely delivering on its commitments, given that it is a moderately complex community health and WASH activity. Despite unexpected challenges such as the Ebola Virus Disease (EVD) outbreak and a mid-activity change in MOH community health strategies, PACS has aligned itself with national priorities and is already demonstrating statistically significant outcome effects in health services utilization. The most successful components include CHSS and CHA capacity building in alignment with the new National Community Health Assistant (NCHA) strategy and the Community-Led Total Sanitation (CLTS) approach leading to over 1,200 communities becoming open-defecation-free (ODF). Under CLTS, PACS has trained “Natural Leaders”, a cadre of community sanitation workers, some of whom are further trained in maintenance of pumps and wells. CLTS has had a remarkable effect on community empowerment, and may be a contributing driver to health outcomes, both technically in reducing fecal-oral transmission and water-borne diseases as well as socially in motivating communities and households to manage their own health and hygiene. There is much more work to be done in integrating the community health and WASH components, but the PACS model demonstrates the synergy that can be achieved in promoting community health. vii This evaluation also identified areas for improvement. Foremost among these are the risks to the effectiveness and sustainability that have emerged in the NCHA program implementation (IR2), around the lack of drugs and health commodities, a strong gender bias and the lack of sustainable financing; see EQ3 below for an expanded discussion on these issues. PACS components in capacity building (IR1) for the MOH and for CSOs, and technical assistance for the MOGCSP policies and guidelines, have largely completed their output potential, but require further attention and rollout at county and community levels to achieve health outcomes. None of the county, district or community respondents were aware of any of these national-level outputs, such as the health communication strategy or the new policy and guidelines for orphans and vulnerable children (OVCs) and orphanages. Behavior Change/Communication (BCC) activities in communities (IR3), intended as mentoring for CHAs, instead often substitute for work by the CHAs and appear to duplicate efforts that CHAs and perhaps Natural Leaders are now trained to conduct. This may be an unintended consequence of PACS BCC staff having stronger skills than CHAs, but not always restraining themselves from conducting BCC sessions during mentoring sessions with CHAs and communities. While CLTS (IR4) is successful in motivating communities toward ODF status, a few ODF communities have relapsed to open defecation, representing a limit to the sustainability of the CLTS model. After some delays, support for hand-dug wells (IR4) is close to achieving its target of 90 wells (82 currently completed, and 8 to be completed in late 2017), but PACS has not conducted routine testing of water quantity or quality as per national standards. Sales of WaterGuard, the social marketed point-of-use water treatment, is an area of substantial underperformance in the face of strongly expressed demand for the product. Fifty-five percent of vendors received zero, one, or two supplies over two and one-half years, and no wholesale distribution has taken place in 12 of the 29 months since PACS’ launch. EQ2: Cost-Benefit Analysis CSO capacity building yielded a positive yet very modest net economic benefit, with a benefit-to-cost ration (BCR) of 1.16, relatively low compared to PACS’ other intervention packages. BCC activities were not estimated to produce a net economic benefit over the next five years, with a BCR of 0.66, indicating a negative return on investment. The high incidence of diarrhea in the three PACS counties drives the high BCRs related to the water and sanitation interventions. Every $1 of present value (PV) costs yields $4.02, $3.45, and $2.43 in PV benefits for hand-dug wells, WaterGuard, and CLTS, respectively. Three PACS-specific CHA program scenarios were modeled. Scenario 1 reflects the current plan to discontinue payment of CHA remuneration at the end of December 2017. Scenario 2 assumes that CHA remuneration costs continue to be paid over a five-year time horizon. Scenario 3 models two assumptions over a five-year horizon: first, that NCHA costs continue to be paid as in scenario 2, and second, that relevant drugs and health commodities are provided to CHAs outside of PACS. Scenarios 1 and 2 yielded negative returns on investment: $0.63 and $0.53 in PV benefits for every $1 in PV costs, respectively. By contrast, under the third scenario modeled, CHA services with drugs, commodities and CHA remuneration produces a 2:1 return on investment over a five-year horizon. It is important to note that there may be other benefits related to these interventions that were not captured in the cost-benefit analysis. Factors such as higher-level strategic objectives, quality, and equity should also be considered when prioritizing interventions that should be continued. EQ3: MOH Policy Changes and CHAs As described under findings for EQ1 above, PACS has aligned itself to the NCHA strategy, but the evaluation identified serious, emerging risks in the implementation of the NCHA program. First, the potential effectiveness of newly trained CHSSs and CHAs is severely impeded unless they are provided with appropriate drugs and health commodities to meet community health needs. Such supplies are viii outside of PACS’ mandate. Without those supplies, the newly trained staff offer essentially the same services as the previous gCHVs, representing a potential waste of investment in capacity building. A few community members and CHAs are already voicing concerns that providing health advice, counseling, and referrals is not sufficient for CHAs to retain a reputation as agents for community health if they are not able to provide treatment. Secondly, a structural male gender bias in the composition of CHSSs (68% male) and CHAs (83% male) may limit or undermine community health service delivery for women and possibly children, both of whom are the primary intended beneficiaries of the program. PACS did not prevent this gender bias as it was emerging, and must now address it during implementation so that women are not deterred from seeking maternal and reproductive health services from male service providers. Finally, the MOH has no clear plan for financing the large and increasing costs of the NCHA program, including remuneration of CHAs and CHSSs, transport costs, and supply of drugs and commodities. Even if PACS pays some costs, a sustainable NCHA program requires some financial buy-in from the MOH. It is likely that PACS’ payment of just the remuneration and supervision costs will result in the activity consuming its available budget prior to the scheduled activity end date. These risks to NCHA effectiveness and sustainability are linked to the design of the national program and are not specific to PACS. Nevertheless, PACS must address these risks if it is to succeed in delivering on community health, and moreover, PACS is in a position to test and implement corrective actions alongside development partners. EQ4: WASH and Integration with Community Health PACS has completed the construction of 82 of 90 target wells, with the remaining eight to be completed by late 2017. However, 68 (83%) out of 82 wells do not meet the GOL water quantity standards and run the risk of running dry during the peak of the dry season. Though two water quality tests were carried out on each well before completion, PACS has not put in place an ongoing water quality assurance plan for the wells it has supported, and water quality status is unknown after the wells were capped. PACS has trained WASH Entrepreneurs with the capacity to maintain and repair hand pumps, and to complete the construction of hand-dug wells. In some communities, water committee members confirmed the establishment of a cashbox system for the collection of fees for the maintenance of the hand pump over time. These user fees vary from community to community, and the system remains at a small scale but is proving to be effective. PACS has not assessed the outcomes of its WASH outputs (wells with hand pumps, water quantity and quality). As a result, the translation of PACS outputs to sustained, community￾level outcomes of safe water sources that last for years could be jeopardized. Under the CLTS approach, PACS has trained Natural Leaders in community engagement for sanitation. These Natural Leaders have triggered 1,489 communities to undertake CLTS, exceeding the indicator target of 1,396 communities to become ODF over the life of the activity. Community ODF status may indicate a higher level of community leadership, organization capacity and/or empowerment. This suggests that the CLTS approach may be linked to improvements in health service seeking behavior. Nevertheless, a few ODF communities have relapsed to open defecation, and a substantial number of the CLTS latrines were observed not to have functioning hand-washing stations. Many of the hand-washing stations verified were set up in a fashion that does not effectively break the chain of contamination. Though CLTS is community driven, PACS has not provided technical guidance relative to durable latrine construction and the setting up of hand washing stations with specific consideration for breaking the fecal-oral transmission routes of sanitation-related diseases. With WASH interventions split among PACS members, the dissemination of hygiene promotion messages, especially on hand washing at critical times, appeared to be weak or non-existent at the community level. ix RECOMMENDATIONS A summary of key recommendations from this evaluation is provided below. EQ1: Performance PACS must develop an outcome-oriented theory of change, not only to document its current model of community health and WASH, but also to ensure that it raises its performance to the outcome level and incorporates an explicit focus on gender. The PACS theory of change must emphasize integration across components, sustainability through regular GOL involvement and engagement at national, county and community levels, and gender as a cross-cutting link across PACS components. Starting with the next annual work plan (Fiscal Year 2017-18), PACS should demonstrate steps toward integration of its components, for instance by linking CHAs and Natural Leaders in community-level structures so that health communication and sanitation efforts directly reinforce each other. At a management level, PACS teams must collaborate in their interventions rather than merely update each other on different activities. Similarly, the PACS Monitoring and Evaluation (M&E) staff must orient other PACS and GOL staff on the expected outcomes to be measured and achieved, and streamline internal reporting, so that all information flows to a common set of databases, from which the partners draw data for their home organizations. USAID should support PACS in a shift toward focusing on outcomes by reducing the administrative burden of meetings and reporting more than once per quarter. Concurrently, USAID should demand outcome￾level data on a scheduled basis, with semiannual and annual reports being a suggested mechanism for higher level results reporting. Additional, specific recommendations for PACS programming include the following: transition capacity building from training and workshops to mentoring, by including GOL and stakeholders in PACS ongoing work; continue support to the NCHA program only if drugs and health commodities are provided to CHAs (see EQ 3 for more details); consolidate community-level BCC efforts with CHAs and Natural Leaders; complete the eight remaining wells, and train WASH Entrepreneurs to conduct routine testing of water quantity and quality; work with communities to resolve the emerging issue of relapsed ODF communities, and involve GOL and international CLTS partners in refining the model to address the factors that lead to relapse to open defecation; and substantially raise targets for WaterGuard sales to match market demand, and prepare WaterGuard for a self-sustaining operation before PACS’ closure. EQ2: Cost-Benefit Analysis Paying for NCHA services with CHA remuneration but no drugs or commodities is the least advantageous scenario, and therefore not recommended. PACS’ investment in CHAs can only be recovered within a reasonable time frame with the iCCM services which requires provision of drugs and other health commodities from non-PACS sources. If budget constraints limit PACS’ continued payment of CHA stipends and other NCHA costs for the nine health districts over the remaining activity period, PACS should consider the following steps, contingent on supply of drugs and commodities to CHAs: discontinuing the capacity building under IR1, including MOH, MOGCSP and CSOs; accelerate the transition of BCC activities to CHAs taking on health messaging, and/or a reduction in the geographic scope for NCHA program implementation. It is noted that an acceleration of the transition of BCC activities to local stakeholders reflects the original activity design, and thus, would not represent a significant cost savings. To maximize benefits and impact of water and sanitation interventions, PACS should consider additional hand-dug wells in ODF communities, by reducing the number of CLTS communities to be triggered in the remaining period of the activity (Fiscal Year [FY] 4 and 5). PACS can determine how many additional wells to construct based on GOL discussions and/or an allocative efficiency analysis to assess different water/sanitation scenarios and optimize the proportional programming of the remaining water and x sanitation budget resources. WaterGuard should be more aggressively promoted as it is low cost with a high positive net economic benefit. EQ3: MOH Policy Changes and CHAs Recommendations for PACS regarding its alignment with the MOH policy changes and the CHA program are based on evidence from the interviews and discussions during the field visits in the counties, as well as the cost-benefit analysis. These two independent streams of analysis both indicate that the NCHA program is not value for money unless drugs and health commodities are supplied to the CHAs, and that unless CHAs can provide treatment in communities, they will be ineffective. Accordingly, recommendations for PACS, USAID and the MOH are based on three scenarios, as follows: • Temporary extension of NCHA payments: PACS should continue to pay the NCHA program costs, including CHA remuneration, only if drugs and commodities are supplied to CHAs. o USAID should authorize those payments up to 31 March 2018, three months beyond the scheduled cut-off date of December 2017, to allow time for the MOH and PACS to discuss and agree on a means of supplying drugs and commodities to CHAs. o MOH and the CHTs should ensure that CHAs are supplied with drugs and commodities so that questions of PACS support to other NCHA costs do not arise again. • Reduction or discontinuation of NCHA payments: If there is no plan for supplying drugs and commodities to CHAs, PACS can negotiate with MOH and CHTs for a reduction in geographic coverage, a phasing-in of MOH/CHTs contributions to NCHA costs, including a portion of the CHA remuneration, o If no agreement can be reached regarding supply of drugs and commodities to CHAs, PACS should reprogram its NCHA support toward expanding its support to CSOs engaged in community health, as a substitute for the NCHA program. PACS should also consider additional wells and CLTS in new communities, invest in refinements to the CLTS model to resolve the question of relapsed ODF communities, and explore other alternatives. o PACS should openly and proactively communicate its changes in support to the NCHA program costs to all stakeholders, including affected CHAs and CHSSs. • Agreed continuation of NCHA payments: Provided that CHAs are supplied with drugs and health commodities, PACS should continue paying NCHA costs so long as its budget will allow, after reprogramming other components based an internal review and recommendations elsewhere in this evaluation. o Given its limited budget, PACS should ascertain and communicate with the MOH and CHTs regarding the projected end date for paying NCHA costs. o To extend the time frame for PACS support, PACS and the MOH may consider alternative arrangements such as a reduced geographic scale limited to fewer districts, and modest but rising contributions from GOL toward the NCHA costs over the remaining life of the activity. EQ4: Integration of Community Health and WASH PACS should enhance the technical capacity of WASH Entrepreneurs to include the excavation of wells from start to finish, beyond maintenance of the PACS-supported wells. Such technical capacity enhancement should also include the monitoring of residual chlorine level of wells as well as the chlorination of wells as per national and international standards. PACS should provide technical guidance to community members to ensure that new latrine construction meets basic technical standards that balance material durability and latrine capacity with a maintenance effort acceptable to households. PACS should expand its BCC approach including community organization and empowerment, alongside specific health messages. PACS should aggressively expand the sales of WaterGuard for treatment of water at the household level, with targets that reflect a goal of market expansion and that drive higher sales. 1 ACTIVITY BACKGROUND The overall purpose of Partnership for Advancing Community-Based Services (PACS) activity is to advance USAID Liberia’s Country Development and Cooperation Strategy (CDCS), Development Objective 3, “Improved Health Status of Liberians." PACS was specifically designed to “support sustainable country ownership for community-based health, social welfare, and Water, Sanitation and Hygiene (WASH) services [by] ensuring [Government of Liberia] leadership on policy and planning down to service delivery.” 2 Key partners for PACS include the Ministry of Health (MOH), Ministry of Public Works (MPW), the National Water, Sanitation and Hygiene Promotion (NWSHP) Secretariat, and the Ministry of Gender, Children and Social Protection (MOGCSP). PACS is designed to contribute to the CDCS through the following results: • Result 1: Broadened capacity of central MOH, County Health Teams (CHT), local Civil Society Organizations (CSO), and non-governmental organizations (NGO) to implement and manage community services. This result area contributes to obtaining and maintaining stakeholder agreement at national and county levels; establishing partnership agreements with MOH units and CHTs; and facilitating self-assessments and performance improvement plans that will lead to ensuring that the MOH have the systems and skills necessary to effectively plan manage and monitor community health services. These activities will enable the Government of Liberia (GOL) to successfully increase availability of community-based health and social welfare services. • Result 2: Increased service quality and availability of community-based health and social welfare services. This result supports the MOH to review, develop, and roll out an updated, standardized, and integrated package of community health and social welfare services to be delivered by community health cadres. PACS also engages CSOs to strengthen linkages between communities and health facilities and improve accountability within the health system. To better serve the needs of orphans and vulnerable children, PACS works with the MOGCSP to conduct accreditation assessments of orphanages, develop or revise guidelines for accreditation of orphanages and for foster care, and train county level staff according to the guidelines. • Result 3: Improved health-seeking behavior and practices. Ensuring the availability of health services at the community will only be effective if those communities are informed about the services and adopt more positive health behaviors. Result 3 activities will generate information and behavior change strategies at the community level to increase demand for quality health services. • Result 4: Improved access to safe WASH services. This result supports increasing access for improved WASH services and products as well as building the capacity building of water and sanitation personnel to effectively manage WASH infrastructure; supports GOL in deploying trained and equipped water and sanitation staff, establish a pump fund to prepare MPW for future management of WASH infrastructure improvement programs. 2 PACS Cooperative Agreement AID-669-A-15-00001, page 20. 2 During the second year of implementation, PACS received $7m additional funding from USAID Global Health Bureau under the EEF to expand the program to three additional counties affected by the Ebola Virus Disease (EVD) outbreak. With this modification, PACS quadrupled its performance targets originally established in the original Activity-Level Monitoring and Evaluation Plan (ALMEP) regarding the number of people trained, equipped and supervised. The midterm evaluation focused only on the three original PACS counties because the EEF funding was scheduled to close in December 2017. The expected results of PACS are closely aligned with those of the National Health Policy and Plan (2011- 2021), the Investment Plan to Building a Resilient Health System (2016-2021), and the Revised Community Health Services Policy and Strategic Plan (2016-2021). Implementation has been done in close collaboration with the MOH and the CHTs in the USAID priority counties of Bong, Lofa, and Nimba, as well as the three EEF counties. In addition to being aligned with specific policies and strategies of the MOH, MOGCSP, and MPW, PACS also addresses the GOL’s Poverty Reduction Strategy Paper Pillar 3 and post Ebola objective: “To rehabilitate infrastructure and rebuild systems to deliver basic service in order to create the conditions and linkages needed to achieve broad-based growth and poverty reduction.” In the post-Ebola environment, the MOH’s rapid restoration of community based health services and the remuneration of community health workers through a more robust community health assistant (CHA) program led to the modification of the award. This shift in the GOL’s priorities for community health service implementation has altered the scope of programming for PACS since many of the service requirements were not included in the original design of the award. Despite the articulation of the four results and the development of the ALMEP, PACS was designed and implemented without an explicit theory of change. The lack of a theory of change or any other documentation of PACS’ original and revised design is reported by senior management to hinder coordination, implementation, and reporting. 3 EVALUATION PURPOSE AND EVALUATION QUESTIONS EVALUATION PURPOSE The purpose of this mid-term performance evaluation is to provide an independent and in-depth examination of the progress of the PACS activity in Liberia to date. The evaluation identified achievements, performance issues, and constraints related to activity implementation and effectiveness. The evaluation was designed to answer the four evaluation questions (EQs) and to focus on areas of implementation that were ongoing or would continue, so that findings and recommendations could be applied to improve implementation and results. An emphasis was placed on obtaining the highest level of outcome data for health, prioritizing county, district, and community-level outcomes such as changes in practices or behaviors over national-level outputs, including policies, strategies, and guidelines. In addition to measuring progress, the evaluation makes recommendations to USAID regarding components of PACS to scale up, modify, or re-design for the remaining life of the activity. Recommendations can also impact other ongoing related USAID activities or future procurements in the region and elsewhere. To achieve this, the evaluation identified lessons learned and best practices from the first years of PACS implementation and compared them against similar implementers, including Plan International, Last Mile Health, GOL CHTs leading community health service delivery, and others. The key intended users of the evaluation are USAID, GOL, and the PACS implementing partners. The GOL includes MOH, MPW, and its NWSHP Secretariat. PACS implementing partners include International Rescue Committee (IRC), Population Services International (PSI), Global Communities (GC), Young Men's Christian Association (YMCA)-Liberia, and the Planned Parenthood Association of Liberia (PPAL)3. Secondary users include donors, GOL health facilities and local authorities, NGOs, and other CSOs working in similar settings. EVALUATION QUESTIONS The mid-term evaluation focused on four primary evaluation questions with additional sub-questions. LSA and USAID/Liberia held a planning meeting on June 20, 2017, during which they finalized the evaluation questions.4 The original evaluation questions, as framed in the PACS Midterm Performance Evaluation Statement of Work, are provided in Annex I. The final evaluation questions are provided below. EQ1: Performance: To what extent has progress been made in achieving the objectives under PACS? • Which components have been most successful in meeting the overall objectives of PACS and at what level of engagement, with results disaggregated by gender? • How have the community health service interventions contributed to improving the health of women and men in all PACS districts? Please discuss in terms of the highest-level outcome indicators available, as well as qualitative information. 3 At the time of the evaluation, PACS and PPAL had announced that PPAL would exit the PACS consortium by August 2017, in accordance with the International Planned Parenthood Federation’s objection to the Mexico City policy. 4 The final evaluation questions are modified from the evaluation questions as originally proposed in the SOW in order to improve the relevance and utility of the information generated for USAID's planning and decision-making. These changes were mutually agreed upon by LSA and USAID/Liberia during the June 20, 2017 planning meeting and subsequent email correspondence. 4 EQ2: Cost-Benefit Analysis: Should there be budget constraints, what interventions should be kept and which should be discontinued? Please discuss and provide a prioritized list using cost effectiveness or cost-benefit analysis, as appropriate. EQ3: MOH Policy Changes and CHAs: How has the recent MOH policy change and transition from CHVs to CHAs affected PACS community health and WASH implementation, with attention to sustainability of the CHA program? • How does the transition from CHVs to CHAs, particularly the revised remuneration package, exert different effects on each of the four result areas of the PACS implementation model (1- capacity building, 2-health service delivery, 3-community behaviors and demand, and 4-WASH)? • Based on the design of the CHA program and its incorporation into PACS’ model of community health, describe factors that may affect the sustainability of the CHA program as well as any opportunity costs in relation to diversion of resources from planned activities. Include gender and social factors in this analysis, as relevant. • What are the recommendations for redirecting PACS’ focus to align more with MOH priorities? EQ4: Integration of Community Health and WASH: How do the results from the implementation of the WASH component impact the community health outcomes, including gender as a factor? • To what extent has the capacity of WASH personnel been built to effectively manage water and sanitation infrastructures? • To what degree are PACS supported WASH activities likely to be sustainable (technically, socially, financially and environmentally), considering gender and relevant social factors? • To what extent does the quantity of water supplied by PACS supported facilities (boreholes, dug wells, etc.) meet World Health Organization (WHO) or national standards? • To what extent has the use of a water quality assurance plan, if any, contributed thus far to meeting and ensuring the standards for quality of water supplied to communities? 5 EVALUATION METHODS AND LIMITATIONS EVALUATION METHODOLOGY To answer the evaluation questions, the tailored methodology consisted primarily of quantitative data sources, supplemented by qualitative information where relevant, and backed by appropriate analysis techniques. The methodology was guided by intended use by key users, including extensive discussions at the planning stage and a trial use of mock evaluation findings with GOL representatives and the PACS senior management team. Quantitative data sources included USAID's Performance Indicator Database System (PIDS), the MOH’s national HMIS database, PACS databases, facility records, and NWSHP data including Akvo Flow. Sources of qualitative data for this evaluation are summarized in Tables B and C in Annex II: Evaluation Methods and Limitations. Sampling The evaluation was conducted in three PACS intervention counties (Nimba, Bong, and Lofa) and two non￾PACS intervention counties (Grand Gedeh and Grand Cape Mount). The latter counties included one county with a non-USAID supported community health intervention (Grand Gedeh) and one county without any donor-supported community health intervention (Grand Cape Mount). Within counties, the selection of communities, health facilities, and hand-dug wells for site visits was determined first by random selection. From the list of randomly selected sites, a sub-set of final locations was selected based on availability of respondents (particularly GOL staff), feasibility of travel during the rainy season, and purposive sampling to capture a range of health facilities, CLTS sites, and active CHAs in order to inform the qualitative data. For PACS-supported counties, districts for site visits were selected based on support from USAID, PACS, and Plan International to test the PACS model of integrated community health and WASH interventions against an alternative of stand-alone community health programming. This purposive sampling approach did not bias the quantitative data, which was largely drawn from centrally administered databases at USAID, MOH, and MPW/NWSHP. To test the PACS model for improving community health, non-intervention counties (Grand Cape Mount and Grand Gedeh) were compared to PACS-led intervention counties (Nimba, Bong, and Lofa). See Annex XV for a list of field sites visited during data collection. Key Informant Interviews and Focus Group Discussions The evaluation conducted 51 semi-structured key informant interviews (KIIs) and 33 focus group discussions (FGDs) to address PACS’ response to emerging health needs and MOH priorities, including the new national CHA program, and supplement the quantitative data for all result areas, as well as 15 benefit valuation sessions and other consultations. In total, 565 total respondents contributed data, of whom 369 (65%) were male and 196 (35%) female. Data collected during KIIs and FGDs were captured and processed systematically using a coding system for each topic area. These data were analyzed using NVivo qualitative analysis software; this process informed development of key findings, conclusions and recommendations that make up the bulk of this report. The US National Institutes of Health’s procedures for confidentiality, informed consent and ethical treatment of respondents were employed. Table 1 below lists the respondent groups that participated in KIIs and FGDs, disaggregated by community level (lay community members, CHAs, CHSSs, CSO staff, TTMs, Natural Leaders, WASH Entrepreneurs) and representatives of GOL or NGOs. The evaluation conducted just over the expected number of KIIs and double the number of expected FGDs. This overachievement in the FGDs often stemmed from keen 6 interest by CHAs, CHSSs, and community members to be heard regarding their experiences with community health and WASH. A third category included benefit valuation sessions for defining the value of PACS interventions from the perspectives of users and beneficiaries. 5 In a few cases, this category also included impromptu community walk-throughs, which were undertaken both as a means of extricating men from a concurrent FGD that was intended for women participants, as well as a means of gathering additional information. A more detailed breakdown of these categories is included in Annex II: Evaluation Methods and Limitations. Table 1: Number of Key Informant Interviews and Focus Group Discussions Targeted and Reached Method Community￾level GOL and NGO Total Meetings Total Respondents Planned Actual Planned Actual Planned Actual Male Female All KII 20 21 27 30 47 51 34 19 53 FGD 10 25 5 8 15 33 205 142 347 Other 6 9 5 6 11 15 130 35 165 Total 36 55 37 44 73 99 369 196 565 As seen in the gender breakdown, respondents were largely male, reflecting the composition of the CHT, CHA and CHSS health cadres. Special efforts were made to include female respondents from these cadres as well as communities, by seeking out female CHSSs, CHAs, TTMs and lay women in communities in order to ensure perspectives from both genders were adequately captured in the data. QUANTITATIVE METHODS Three quantitative approaches to assess the contribution of PACS’ community health service interventions to health service use at the facility were employed. These analyses are based on monthly data (from January 2014 to May 2017) at the district level extracted from the Liberia Health Management Information System database called the District Health Information System-2 (DHIS-2). Fifteen key health facility service delivery indicators were selected based on their relevance to PACS activities and USAID/Liberia’s health funding elements. More broadly, indicators correspond to four health areas: immunization, childhood illness and treatment, family planning, maternal and neonatal health. This data was merged with other sources including the Liberia Demographic and Health Survey (2013), Global Communities’ master WASH database, and the WHO Ebola situational reports. First, bivariate analysis was used to compare the relationship between key health service indicators and PACS outputs. Second, Difference in Differences (DID) mimicked an experimental research design by examining the differential effect of an intervention on a 'treatment group' (i.e. PACS districts) versus a ‘control group’ (i.e. non-PACS districts) in a natural experiment. 6 The 15 health facility indicators in the nine PACS health districts were compared with: (1) non-PACS districts before/after November 2015, (2) non-PACS districts from March-May 2015 with March-May 2017, (3) non-PACS districts in Bong, Lofa, and Nimba counties before/after November 2015, (4) non-PACS districts in Bong, Lofa, and Nimba counties from March-May 2015 with March-May 2017, and (5) similar districts using Propensity Score 5 Further information on this method is provided in Annex II: Evaluation Methods and Limitations. 6 Lechner, 2011. 7 Matching (PSM) on wealth and education. 7 This analysis controls for factors expected to have a direct effect on health service seeking at the facility: rainy season, population (at district level), and total cases of Ebola (at county level). Third, segmented regression analysis—a quasi-experimental econometric approach— was applied to evaluate immediate and trend changes in health service utilization following PACS implementation. 8 The analysis controlled for population (at district level) as well as seasonality. COST-BENEFIT ANALYSIS (CBA) METHODS CBA was used to address Evaluation Question 2. CBA values intervention outcomes in monetary terms to identify whether the benefits of an intervention exceed its costs. This enables a direct comparison of incremental costs and benefits in monetary terms which can help to inform decisions about allocative efficiency when resources are scarce.9 To enable a comparison of PACS interventions, a mixed-method CBA was applied. In addition, this analysis was complemented with a high-level summary of the relevant cost-effectiveness literature. See Annex X: Cost-Effectiveness Literature Review for more detail. This CBA was informed and guided by relevant literature and guidelines. More detail is provided in Annex IV: Sources of Information. CBA intervention packages corresponding to each of the four PACS result areas were defined. Primary beneficiary groups were identified for each intervention package to value the benefits using a modified Willingness-to-Pay (WTP) approach and/or establish benefit and consequence areas to inform modeling. A summary of the CBA intervention packages, primary activities and beneficiary groups is presented in Table 2. Table 2: CBA Intervention Packages, Primary Activities, and Beneficiary Groups by PACS Result Area PACS Result Area Intervention Package Primary Activities Respondent Group Result 1: Broadened capacity of central MOH, CHTs, local CSOs and NGOs to implement and manage community services. 10 Capacity development Technical assistance, training, equipment, CHA support CSO stakeholders CHSS CHAs Result 2: Increased service quality and availability of community-based health and social welfare services Community health services iCCM/pre-referral treatment, routine household visits, health center referral and treatment Community members 7 Heckman et al, 2008. 8 Lagarde, 2012. 9 By contrast, cost-effectiveness and cost-utility analyses address questions of production efficiency. Because these latter approaches require a common measure of effectiveness (i.e. Disability Adjusted Life Years (DALYs), Quality Adjusted Life Years (QALYs)), they do not enable a direct comparison of interventions that produce different or multiple outcomes (Drummond et al, 2005). 10 The evaluation could not conduct cost-benefit analysis of additional PACS outputs under Result 1, such as national policies and strategies, due to the non-availability of central-level GOL respondents for their perspective and valuation of those outputs (noted in the limitations) as well as the lack of awareness of these outputs among respondents at the county and district levels. 8 Result 3: Improved health￾seeking behavior and practices BCC Education Through Learning (ETL) training, BCC tools, on-site coaching, community engagement CHVs Result 4: Improved access to safe WASH services Water, sanitation and hygiene Hand-dug wells CLTS WaterGuard Community members The analysis used CBA Builder v1.4, an Excel-based CBA tool, to compile all cost and global WTP estimates and compute economic metrics by intervention or intervention group. Estimates account for annual economic growth at 6.11% based the 10-year average (2005 to 2015) in Liberia. Two discount rates were applied: 20% for water and sanitation interventions11, and 12% for all other intervention packages. The 12% rate was adopted as it was recommended by USAID for another recently conducted CBA.12 A five￾year time horizon was used, unless otherwise noted. Additional detail about the CBA method is provided in Annex II: Evaluation Methods and Limitations. Direct Observation To measure the water and sanitation situation of the communities, direct observation was conducted alongside FGDs and KIIs. Direct observation targeted functional and non-functional hand pumps, latrines, and hand washing stations, as well as general sanitary conditions of the community. LIMITATIONS AND CONSTRAINTS First, PACS was designed and implemented without an explicit theory of change that would have made explicit the assumed logic between the activity’s outputs and expected outcomes, as well as the underlying assumptions. The evaluation team developed a working model of a PACS theory of change for the mid￾term evaluation, but this has not been validated with PACS and its stakeholders. It is possible that some areas of integration are not adequately described in this evaluation, though this is considered unlikely because of the feedback from PACS staff over the course of data collection and in two debrief sessions. Quantitative data available for this evaluation depended primarily on DHIS-2 and PACS monitoring databases, especially the CLTS data. There are a number of limitations and potential biases related to this routine data. DHIS-2 data is known to contain inaccuracies and missing data, as reported by the MOH M&E staff, CHTs, and District Health staff. PACS’ own monitoring data is of uneven quality and is not integrated (as attested by PACS’ lead M&E manager, who had begun addressing some of those gaps). In some instances, PACS reported on a few activities or outputs in its quarterly reports but had not routinely tracked that information, such as community engagements and some trainings, so that it is difficult to analyze those intervention components. In summary, performance data that is based on the existing data in PACS databases, data reported to the USAID Liberia PIDS, external databases including DHIS-2, is of uneven quality. None of the data quality problems appear to be systematic or indicative of bias. Rather, the quality problems with the quantitative data limit the robustness of this analysis, meaning that effects may be broader or greater than the existing analysis can demonstrate. No systematic bias in quantitative data was found, and as a result, some outcomes may be undetected because of additional random variance in the datasets, and the statistical significance of the outcomes found in this analysis may be underestimated. The findings from this analysis can therefore be taken as conservative rather than speculative. 11 A higher discount rate was applied to water and sanitation interventions to account for identified relapse and sustainability issues. 12 International Development Group, 2016. 9 The evaluation worked to mitigate the quantitative data quality problems with several approaches. To minimize the influence of outliers and irregular data in the quantitative analysis, data was winsorized by truncating extreme values. In relation to the quantitative methods, causal attribution is limited in the absence of randomization. To account for this issue, applied statistical methods that are recognized approaches to assess outcomes of natural experiments (Difference-in-Differences and segmented regression) were applied. Control variables to account for known confounding factors were employed. A comparison group using Propensity Score Matching was also constructed. Original qualitative data was collected and used to triangulate data from several methods, such as examination of the effectiveness of the National Community Health Assistance (NCHA) program through key informant interviews, focus group discussions and cost benefit analysis. Similarly, the lack of water quality data from PACS hand-dug wells was verified with PACS databases and from KIIs with Natural Leaders, Environmental Health Teams (EHT), and PACS staff. Accordingly, the evaluation focuses on PACS quality assurance process for water. Qualitative methods are susceptible to recall and selection biases. To address these potential sources of bias the evaluation only considered data that was reported by multiple classes of respondents from different sites. For instance, information about CHA performance was considered reliable when voiced not only by CHAs, but by CHSSs, CHT members and lay community representatives, from multiple sites. A strong selection bias for males was both expected and observed among interview and discussion respondents (65.6% male and 34.4% female), reflecting the structural gender bias in health workers at all levels of the health system. Special efforts were made to obtain perspectives and information from female health workers and community members. In addition, to avoid male dominance in focus group discussions and community-level CBA sessions, these were held as single gender events; in cases when men tried to join a female-only group, they were invited to lead a community transect so as not to influence women’s discussions. Repeated efforts were made to establish interviews with key representatives at MOH and MPW, but those representatives were unable to schedule time with the evaluation team for interviews. A final debrief with these representatives did not take place. Consequently, the findings, conclusions, and recommendations are made without the benefit of feedback from some central-level GOL representatives. Inaccessibility of remote sites, particularly during the rainy season was mitigated by real time monitoring of weather and road conditions; as well as the prudent selection of districts and community sites, based on feasibility. In the original site planning, Nimba and Lofa were the PACS counties selected for visits. One community in Nimba county was inaccessible by road, and that visit was necessarily dropped and substituted with another community visit. Midway through data collection, it became apparent that two of the three PACS districts in Lofa County had limited to no accessibility due to heavy rains. With concurrence from USAID, the site visits were changed to Bong County. From Bong County, a visit to the third PACS district in Lofa was conducted. A limitation of the findings is that they may not fully reflect the additional challenges of PACS’ community health and WASH interventions in these remote communities. Finally, there are a number of limitations relating to the CBA. Costs are limited to direct costs as full budget information was not available for this evaluation. In addition, the abstract nature of willingness-to￾pay/valuation of benefits among respondent groups was a challenge. Responses were validated at the end of each session, and benefit modeling was completed using information specific to Bong, Lofa, and Nimba to the greatest degree possible. Also, time constraints precluded a representative sample, therefore some results may have limited generalizability. Further detail on methodological and logistical constraints and potential biases related to this evaluation are provided in Annex II: Evaluation Methods and Limitations. 10 FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS FINDINGS EQ1: Performance: To what extent has progress been made in achieving the objectives under PACS? The questions of PACS performance are first addressed by assessment of PACS outputs and level of engagement with GOL or other relevant stakeholders (question 1a), and secondly by examination of outcome-level effects in utilization of health services (question 1b, below). 1a. Which components have been most successful in meeting the overall objectives of PACS and at what level of engagement, with results disaggregated by gender? At midterm, PACS most successful components have been the NCHA implementation, for is alignment with the MOH, and CLTS with its strong community empowerment. Other components, including capacity building for CSOs, technical assistance for MOGCSP, BCC activities, the health communications strategy, and hand-dug well constructions, are meeting their outputs but not directly contributing to an integrated set of outcomes. Two components, the MOH central capacity building and the WaterGuard sales, are classified as underperforming, for lack of ownership in the MOH capacity building and for insufficient distribution and sales relative to vendor expectations in the case of WaterGuard. PACS’ performance is summarized in Table 3 below, with each intermediate result unpacked into its component interventions. Where output targets were available, the table notes achievements against those targets. In many cases, especially for IR2 and IR3, either PACS did not establish any targets or PACS altered its intervention in order to align with the MOH policy change. In many cases, gender disaggregation is not appropriate (such as for policy support) or not available (such as for radio audiences, for which no systematic assessment has been done nor is any planned by any implementer). Table 3: PACS Performance Summary (Bong, Lofa and Nimba Counties) Intervention Performance Engagement & Notes IR1: Capacity building 1. MOH units: CHS, EOH, HPD 2. Support to National Community Health Assistant Strategy and curriculum 3. MOGCSP 4. CSOs 1. Completed MOH units self-assessments, PIPs drafted but not implemented 2. Contributed to NCHA strategy and curriculum 3. New policy and guidelines for OVCs, foster care and orphanages developed 4. Capacity building for 25 CSOs, funded 9 • PIP actions by MOH not completed • PACS aligned with NCHA, variable engagement with County/District teams • Child policies not visible at county level • CSOs value for own functions, not linked to community needs • Strong engagement by PACS but limited ownership by GOL IR2: Comm. Health Services 1. CHV/gCHV training 2. CHSS/CHA training 1. 57 CHSSs trained (Male: 39, Female: 18) and 662 CHAs trained (Male: 551, Female:111) • iCCM trainings for CHVs & CHAs • Bias toward male CHAs • Strong alignment and engagement with GOL in human capacity development • Drugs, commodities, motorcycles remain a gap in implementation 11 Intervention Performance Engagement & Notes IR3: BCC 1. Training and mentoring of CHV/gCHVs in BCC 2. Developed National Health Communications Strategy and BCC materials/tools 3. Radio-based health shows and call-in discussions 1. 378 CHVs trained (Male: 259, Female: 119) 2. Despite training for CHT members, no mention or reference to the health communications strategy or BCC materials 3. Radio-based shows for health messaging ongoing 4. BCC activities are described as additional work outside of the PACS commitment, and were reported quarterly but not systematically tracked until requested by the evaluation team. • Outputs delivered, but outcomes not seen • BCC materials are completed as a national-level output, but not seen or mentioned at the county, district or local levels. • Limited sustained ownership by GOL and engagement at national, county and community levels IR4: WASH 1. Wells + hand pumps 2. CLTS – ODF status 3. WaterGuard sales 1. 82/90 wells completed; 8 wells on-going 2. 1,225/1,396 CLTS communities triggered. 3. Slow WaterGuard sales: 33% of total sales since inception took place in the most recent quarter (Q3/Y3). • Wells: 91% outputs; 27% outcomes • Strong community empowerment in CLTS, Natural Leaders network • Some relapsed ODF communities • Strong community engagement, little to no engagement with GOL Engagement with GOL counterparts has varied over time and by type of intervention. The PACS components oriented to the national level include the MOH capacity self-assessment and performance improvement plans, the health communication strategy and BCC materials, and the orphans and vulnerable children (OVC), foster care and orphanage policy and guidelines. At the county or local levels, however, there is no evidence of awareness or an effect of these activities, including for those that have been completed such as the health communication strategy and the orphanage guidelines; i.e., the national-level results have not cascaded to counties, districts and communities. County-level interventions, with involvement at the district and health facility levels, include the trainings for gCHVs, followed by the CHSSs and CHAs, as well as initial discussions regarding CLTS and selection of communities for hand-dug wells. PACS’ community level engagements included guidance on selection of the CHAs, some BCC activities, triggering of communities for CLTS, and supporting the construction of hand-dug wells. Representatives of district and county health offices and some PACS field staff all state that there had been greater involvement of GOL staff in community-level activities at the beginning of PACS than at present. Some GOL staff said they believed the change in engagement with GOL arose because PACS became directly absorbed in training CHAs and “neglected” the county and district health teams. Under CLTS, PACS trained “Natural Leaders”, a cadre of community sanitation workers, some of whom were further trained by PACS in maintenance of pumps and wells. In addition to their usual activities of triggering communities and leading them from OD to ODF, Natural Leaders are engaged into agricultural activities for the purpose of generating income to support their works at the community level and to reduce the syndrome of dependency13. This practice including other initiatives by Natural Leaders should be shared across the Natural Leader network. Across all four result areas, PACS has provided substantial human capacity development, particularly through training and mentoring. Respondents from interviews and discussions have expressed appreciation for the training, but a few senior CHT members have said that there is no need for further training. These comments refer to the frequency of training and workshops by donor-funded interventions, including but not limited to PACS. During visits to county offices, there was no occasion in 13 FGD with Natural Leaders in Lormata, Salayea district, Lofa County. 12 which a full CHT was present. At least one member was away for a training or workshop (not conducted by PACS in these instances). Several county and district health staff respondents said that they needed to learn more from PACS about how to conduct community engagement. These staff said that PACS had previously been proactive in involving them in its work, but that in the past year PACS had simply informed them after interventions had taken place, or the staff learned of PACS work in the course of their own duties but not from PACS. Under IR2, PACS has met its output-level ambitions for training community health workers. The duplication of training the same people, first as gCHVs and later as CHAs, in at least one curriculum component (i.e. iCCM), is due to PACS’ alignment with the new NCHA strategy. Design of the NCHA strategy began in early 2015, with the MOH supported by another USAID partner, Last Mile Health. By the third quarter of 2015, IRC and other partners were included in the MOH working group, and elements of the NCHA program were incorporated into other health sector plans. The NCHA strategy document was published in December 2015, citing contributions by IRC, Last Mile Health, USAID and others. Based on the President’s availability for a formal event, the NCHA launch was held in July 2016, but the strategy was in place in late 2015, eight months earlier, at the same time that PACS was training gCHVs. The evaluation was not able to determine exactly when PACS and USAID became aware of the NCHA strategy in designing and implementing the community health training. The absence of a strong relationship with the MOH, for dialogue and consultation beyond specific tasks, may be a root cause for this inefficiency in duplicating that training. In the rollout of the NCHA program, PACS field staff guided communities in selecting CHAs, but did not follow the NCHA stipulation that female candidates be given preference. As a result, the communities selected CHAs with a 5:1 male-to-female ratio, despite the community health program providing greater services to women. When interviewed about this issue, PACS staff at all levels did not express recognition or concern about the gender bias, and said they were merely implementing the communities’ choices. During follow-up discussions, senior staff did note a concern that literacy level requirements were a barrier to female participation. While gender is one of twelve criteria for CHA selection, there is some evidence that the gender composition of CHAs has a greater male bias than seen in the predecessor cadre of gCHVs. PACS data, albeit limited in gender disaggregation, show that a subset of CHVs (within 5 km of a health center) enrolled in training were 32% female, against the 16% of CHAs who are female, suggesting that progress on gender was reversed in PACS’ implementation of the NCHA program. Data related to the entire gCHV cohort prior to the NCHA program was not available for analysis. That may provide broader evidence regarding gender bias in CHA selection. Additional issues related to gender-related risks in the NCHA program are explored further under EQ3. In other respects, PACS has not incorporated gender into its programming. Notably, the annual reports for the first two years make no reference at all to gender; the term is used only as part of the name of the Ministry of Gender, Children and Social Protection, which was receiving orphans-related technical assistance from PACS. Despite the multiple trainings and community engagements over the first two years of implementation, there is only one instance of gender-disaggregated data in the two annual reports - that too, in reference to an estimated target population, not PACS actual achievement data. In the most recent quarterly report available to the evaluation (Year 3, Quarter 2), PACS made strides in presenting higher level data on results, but none of the figures or tables were disaggregated by gender. There is some progress in this quarterly report, as PACS reported five instances of gender-disaggregated training participant data in the narrative, and also specified a minimum 50% female composition for Community WASH Committees. More significantly, however, no consideration of gender power dynamics appears in any PACS documentation, nor is gender an explicit factor in implementation across the four result areas. In discussions, PACS senior management expressed an interest in gender, but that interest has not been translated into implementation nor shared with PACS staff, GOL counterparts, or other stakeholders. 13 In the WASH component (IR4), PACS is modestly delayed in completing hand-dug wells (82 out of 90). The eight remaining hand-dug wells are currently under construction, and PACS reports that those will be completed by the end of 2017. CLTS has shown strong community engagement and empowerment. Discussions with community members as well as Natural Leaders and WASH Entrepreneurs all demonstrate a common theme of motivation and ownership regarding sanitation and toilet use. For example, several randomly selected CLTS communities showed their recently constructed latrines, shower stalls, and handwashing stations. The near universal presence of this new infrastructure was underlined by the expression by community members of toilet use and open-defection-free (ODF) status as a community norm. According to WASH Entrepreneurs and Natural Leaders, use of the ‘bush’ for defecation was widely frowned upon or scorned by community members, and in one instance a new inhabitant was repeatedly admonished for not having yet constructed his own toilet. The effect of community empowerment, with CLTS as both a cause and a proxy for further community actions, was tested statistically (see section EQ1b below). Despite the strong effect of CLTS on community behaviors, a few communities that had been ODF have since relapsed. According to Natural Leaders especially in Bong and Lofa counties, this occurred largely because their infrastructure, locally constructed in accordance with the CLTS approach for community ownership, deteriorated too soon, and households did not make repairs due to migration of labor, demands of farming, or an expectation that another NGO might build latrines for them. These factors that led to relapsed ODF communities represent a limit to the sustainability of the CLTS model. Based on PACS data under IR4, WaterGuard sales have been extremely low and sporadic over the 29- month period assessed (February 2015 – June 2017), with 48% of total sales taking place in the two quarters preceding the evaluation, 33% in the most recent three months alone. Sixty-eight percent of vendors have been supplied with stock just once in 2.5 years, and over half of the period shows just zero, one, or two vendor sales per month. Despite this uneven and low sales performance, pharmacy staff and drug store proprietors all strongly said that they would like to have more stocks, including several pharmacies that were last supplied with WaterGuard before PACS was launched (i.e., vendors not on the PACS list of outlets but within PACS districts). A few pharmacy staff and drug store proprietors mentioned the radio marketing and community demonstrations of WaterGuard that PACS has conducted, which those vendors regarded positively. PACS has stated that it has met its targets for WaterGuard as agreed with USAID but also acknowledged the gaps in sales performance, noting the frequent unavailability of their sales agents in the field and a three-month production halt due to quality problems with the product. Taken together, the success of PACS components is summarized in Table 4. Components that are ranked as ‘most successful’ are those that are on track to achieve targets by the end of the activity and have a high level of stakeholder engagement. These components show evidence of contributing to outcomes in community ownership and effects on health services utilization (see EQ 1b below for a discussion of the ‘PACS effect’ detected in DHIS-2 data). For example, CLTS is included due to its exceptionally strong community engagement and the potential spill-over effects of its community empowerment with health services utilization. ‘Satisfactory performance’ includes components that are delivering expected outputs but can be improved to address outcome-level changes. This includes national-level interventions that have not been rolled out to the county and community levels. ‘Underperformance’ covers the components that have not met expectation, either for lack of engagement and ownership or for other reasons as noted in the table below. Table 4: Summary of PACS Component Performance Level of success PACS component Notes Most successful • IR2: NCHA implementation • Strong alignment with MOH, at extra cost to PACS. 14 • IR4: CLTS and Natural Leader training • Strong empowerment model, with effect on health. Satisfactory performance • IR1: CSO institutional capacity building • IR1: MOGCSP technical support • IR3: Health communication strategy and BCC materials • IR4: Hand-dug wells, WASH Entrepreneurs • Outputs achieved but high risk of no outcomes on health. • Wells meeting outputs, but no routine testing of quantity and quality. • Gender considered in water committees, but not in other components. Underperforming • IR1: MOH capacity building • IR4: WaterGuard • Incomplete, not clearly valued or owned by central MOH units. • Sales not meeting expressed demand by vendors; PACS targets set too low. All of the national-level deliverables, including the guidelines for orphanages and foster care, the National Health Communications Strategy, are ranked as satisfactory. Despite delays, these important products carry a potential to provide outcome benefits across Liberia. At PACS’ midterm, however, there is more evidence that these high-level documents will remain as mere output deliverables rather than lasting contributions to Liberia’s health, with the exception of integration of ETL and BCC tools into the NCHA curriculum. Specifically, development of these national deliverables did not substantially involve stakeholders outside of Monrovia, and there is weak capacity to put these deliverables into use. As a result, there is little to no awareness of these deliverables at the county, district, and community levels, including among the CHTs. Further, plans for dissemination, rollout, and use of these deliverables at county, district, and community levels are underdeveloped. Some senior PACS managers have said that PACS does not have responsibility for rollout and use of these products, underlining the risk that these products will not affect community health. In examining performance, the evaluation also noted factors related to PACS management and M&E, as well as USAID’s demands on the activity. PACS is very task-oriented, with staff focused on specific actions or outputs as opposed to higher outcomes. For instance, when asked about capacity building, PACS staff spoke about the self-assessments but not about how any change in MOH capacity might facilitate community health. As another example, discussions with PACS staff about hand-dug wells focused on the mechanics of well construction, rather than the effect of safe water sources on health. PACS senior management openly said that they are extremely occupied in responding to frequent requests and reports from USAID, as well as preparing and attending regular biweekly meetings with USAID. In preparing items for discussion on a biweekly basis, PACS management necessarily reports on new and ongoing activities, because outcomes and outputs do not occur or change on such a frequent basis. The demands on field staff to inform those biweekly reports then draws the entire PACS team – management and field staff – to focus on activities rather than outputs, outcomes, or even risks and opportunities for PACS. Servicing those meetings, and any action items that arise, has distracted PACS from its own outcomes. PACS staff were unprepared to answer questions regarding gender in community health and WASH implementation, often attributing male bias in CHAs to community choice and not expressing awareness of gender aspects of water sources or sanitation. Further, PACS behaves in many ways as a collection of NGOs rather than an integrated consortium of partners. For instance, PACS has only recently consolidated into shared field offices. In Monrovia, the partners continue to operate from separate offices. PACS staff admit joint planning is not the norm; rather, partners meet to inform and update each other on plans and activities, but do not plan and implement together. M&E is handled by each implementing partner, with partner-specific monitoring forms, databases, reports and schedules. 15 1b. How have the community health service interventions contributed to improving the health of women and men in all PACS districts? Please discuss in terms of the highest￾level outcome indicators available, as well as qualitative information. PACS has not made any assessment of its outcomes, and some staff said that they did not expect outcomes at this stage of PACS implementation. In part, that uncertainty arises from the lack of an explicit PACS theory of change that would connect activity component and outputs to their logical outcomes. Statistical analysis of DHIS-2 data, however, shows a statistically significant difference in several indicators of health services utilization between PACS community health districts and the rest of the country. Because these analyses controlled for secular trends and known confounders, this difference can reasonably be attributed to PACS and termed a ‘PACS effect’. This analysis first identifies positive correlations between the PACS community health interventions and selected outcome variables. Next, comparisons of PACS community health districts against all other districts in Liberia shows differences in the rate of change of health services utilization (difference-in-difference analysis) as well as a significant pre/post outcome effect at the start of PACS interventions (segmented regression analysis). It was not possible to complete a sub-analysis by gender as the DHIS-2 dataset does not report service provided by patient gender. Very limited statements of the gender effects can be inferred based on the inclusion of pregnancy and reproductive health outcomes in the analysis, to the extent that this limited data does not show appreciable differences in the ‘PACS effect’ between services oriented to women and those for the general population. BI-VARIATE ANALYSIS In PACS districts, the achievement of ODF status is significantly correlated with 10 of 15 key health facility service indicators with a mean correlation coefficient r equal to 0.38 [range 0.24-0.50]. This means there is a significant, positive association between ODF and 10 of the 15 key health indicators analyzed. For example, an increase in the cumulative number of PACS ODF communities in a district is associated with increased vaccination coverage, treatment of childhood illnesses, antenatal care, short-acting family planning use, and post-partum health facility visits. PACS hand-dug wells are significantly associated with these same indicators and negatively associated with the severe malaria treatment. Likewise, the number of CHVs (relative to the population size) is also statistically correlated with decreased treatment for severe malaria, increased treatment of childhood illnesses, short-acting family planning use, and post￾partum health facility visits, but not immunization coverage. The decreased severe malaria treatment may be the result of fewer cases due to improved prevention and/or prompt treatment of simple malaria. Therefore, ODF status in PACS districts may serve as a proxy indicator for community leadership, organization and/or empowerment. These community attributes in turn may be explanatory factors for improved health service seeking behavior. The correlation coefficient matrix for the 15 indicators, based on DHIS-2 data from the three PACS counties, along with variables for ODF status (cumulative number at the district level), the presence of hand-dug wells (cumulative number at the district level), CHWs (at the county level) and the percent of CHVs who are female (at the county level) is shown in Annex VI: Pairwise Correlation Coefficients Table. DIFFERENCE IN DIFFERENCES The quantitative Difference in Differences (DID) analysis compared PACS and non-PACS districts on 15 key health facility service indicators. This analysis detected a statistically significant effect in 9 of the 15 indicators: increases for measles coverage, Penta3, Oral Polio Vaccination third dose (OPV3), malaria cases treated with Artemisinin-based Combination Therapy (ACTs), Depo-Provera users, Antenatal care (ANC) visits, Intermittent Preventive Treatment for malaria for pregnant women second dose (IPTp2) coverage, and Tetanus Toxoid second dose (TT2) coverage; and a decrease in Acute Respiratory Infection (ARI) treated with antibiotics. This means that performance for 60% of the indicators assessed is better 16 than one would expect in districts where PACS is implementing community health activities, compared with non-PACS districts. For example, in relation to measles immunization, the DID analysis revealed that the average coverage in PACS and non-PACS districts was statistically the same for the baseline period (January 2014-October 2015). As seen in Figure 1, during the PACS implementation period (from November 2015 to May 2017) average coverage increased in PACS districts from 64.7% to 81.7%. This is 6.49% points higher (p<0.000) than expected compared to the increase in non-PACS districts. Figure 1: Difference in Differences: A Statistical Comparison of the Measles Coverage in PACS and Non-PACS Districts before and after PACS Implementation The DID summary results table is included as Annex V: DID Analysis Results Table. Figures for all statistically significant results are included in Annex VI: DID Analysis Figures. SEGMENTED REGRESSION ANALYSIS The quantitative longitudinal analysis compared PACS districts in Bong, Lofa and Nimba during the pre/post intervention period on the 15 key health facility service indicators. This analysis detected a statistically significant trend change associated with PACS for six of the 15 indicators: severe malaria, IPTp2, ARI treated with antibiotics, delivery outside of a health facility, and oral and injectable contraceptive users. This means that the average monthly change in these indicators during the implementation period is statistically greater than the average monthly change during the baseline period. These differences are reasonably attributable to the PACS activity. Figure 2 depicts the mean change by month (solid blue line) and the predicted values (red dotted line) after controlling for time and seasonality (i.e. rainy and dry seasons). In relation to severe malaria, the results show a highly statistically significant (p<0.000) increase of 41.18 severe malaria cases presenting at the health facility at the beginning of PACS implementation (November 2015), and a highly statistically significant (p<0.000) monthly trend decrease of 3.6 severe cases after 64.7% 81.7% 75.2% 62.9% 73.5% 55% 60% 65% 70% 75% 80% 85% Jan. 2014- Oct. 2015 Nov. 2015- May 2017 Difference in Differences: Measles Coverage PACS Projected Control 6.49%*** 8.26%*** 1.8% 17 November 2015. A sharp drop in severe malaria cases can be observed in November 2016. This may be attributable to reporting issues. However, the number of severe malaria cases following that point remains low which suggests a real decrease in cases. As expected, both severe malaria and malaria cases treated with ACT increased significantly during rainy season months (June-October14): 14.71 (p<0.009) severe malaria cases, and 103.78 malaria cases (p<0.084). Figure 2: Segmented Regression: A Statistical Comparison of Severe Malaria Cases before and after PACs Implementation in Nine Districts of Bong, Lofa, and Nimba Counties Segmented regression results using the other comparators were similar in direction (i.e. positive or negative association) and magnitude (i.e. the coefficient or numerical value of the association between PACS during the implementation period and the key indicator), but less significant. This is likely the result of the reduced sample size either from limiting the time period or counties. Results from Propensity Score Matching were also very similar (and in some cases the same) compared to the national comparison of PACS vs. non-PACS districts. The segmented regression results table is provided in Annex VII: Segmented Regression Results Table. Figures for all statistically significant results are included in Annex VIII: Segmented Regression Figures. EQ1 Conclusions PACS is largely delivering on its outputs, and is in a position to correct areas of delayed performance, such as the eight hand-dug wells due to be completed in 2017. Many of the delays in implementation have 14 Liberia Demographic and Health Survey, 2013. 0 20 40 60 80 2014m1 2015m1 2016m1 2017m1 date in year/month Mean change Predicted values Severe malaria cases Start of Implementation 18 been driven by external factors, such as the introduction of the new national CHA strategy and program, which necessitated a restart of much of the training PACS had already completed with community-level health workers. Other factors that affected PACS implementation included the EVD outbreak that was ongoing at PACS’ start-up and the related short-term expansion of PACS work to three additional counties (Margibi, Montserrado, and Grand Bassa; not assessed in this midterm evaluation), lack of ownership by MOH for the performance improvement plans, as well as internal disruptions due to relatively high staff turnover among senior staff, particularly in IRC, the lead partner for the consortium. The quantitative analyses provide evidence of a ‘PACS effect’ relating to increased health facility care seeking for several key MCH, FP, and malaria indicators. This suggests that PACS is measurably increasing demand for facility health services. It is notable that both multi-variate analyses found statistically significant changes for three indicators: ARI treated with antibiotics15, Depo-Provera users, and IPTp2. As the PACS effect has been observed in the time period prior to the start of the NCHA program and the rollout of central strategies such as the National Health Communications Strategy, the effect cannot be attributed to those components, though they may amplify or sustain the effect in the future. Community engagement through CLTS may be a factor in the PACS effect, as the level of empowerment described by respondents in many CLTS communities may have had a cascade result on household agency in accessing health care services. As noted in the limitations, the absence of a PACS theory of change underlies the lack of an outcomes focus, the insufficient attention to gender, and the low level of integration. The reporting and meeting with USAID at greater than quarterly frequency also diverts attention from higher-level outcomes that take time to emerge to activities that can be reported more easily. The artificially low targets for point-of-use water treatment (WaterGuard sales) were easily reached despite two years of patchy and irregular sales, particularly in relation to expressed market demand. The low targets may have become a disincentive for PACS to develop the market for this useful product. A low level of integration across partners and interventions means that potential collaboration for reinforcing outputs and outcomes is being missed, such as building on the community empowerment from CLTS in enhancing BCC and the NCHA program. Ignoring gender as a factor in community health and WASH implementation creates risks to PACS’ own effectiveness, and perpetuates rather than changes existing social factors that undermine health. It is not clear why neither PACS nor USAID were aware of some of the upcoming external changes in MOH, particularly the NCHA program, or why knowledge of such changes was not factored into the design and/or timing of the PACS intervention. PACS and USAID were involved in the 2015 development of the NCHA strategy, but did not alter the PACS intervention in a timely manner based on the emerging strategy. For instance, PACS training of gCHVs appears to have been a poor use of time and resources, given that it was superseded by the more expanded training for the new CHA cadre, the vast majority of whom had just recently completed the PACS training as gCHVs. That duplication may have been avoided through closer consultation and coordination with the MOH and other implementing partners in community health, beyond the transactional discussions related to specific tasks. Additionally, infrequent engagements between USAID and MOH may have contributed to this case of inefficient use of resources. EQ2: Cost-Benefit Analysis: Should there be budget constraints, what interventions should be kept and which should be discontinued? Please discuss and provide a prioritized list using cost effectiveness or cost-benefit analysis, as appropriate. 15 It is unclear as to why ARI treated with antibiotics is decreasing over time, especially as the DID analysis also revealed that ARI treatment in non-PACS districts is decreasing in both the DID and segmented regression analyses. This trend may be explored in a future analysis. 19 Given the CBA methods and assumptions applied, the interventions found to yield a positive net economic benefit over the time horizon were hand-dug wells, WaterGuard, CLTS, CHA services (with stipends and drugs), CHA training, and CSO capacity building. Interventions with negative net economic benefit were BCC and CHA services without provision of drugs. It is important to note that other factors including higher-level strategic objectives, quality, and equity which are not captured by CBA should also be considered when prioritizing interventions. Table 5 presents the cost-benefit metrics for nine PACS interventions and packages including three CHA scenarios rank-ordered by the Benefit-to-Cost Ratio (BCR). Present Value (PV) costs and benefits refers to the total present value after adjusting for economic growth and discounting over the time horizon. The Net Present Value (NPV) is the difference between PV benefits and costs. Positive NPVs indicate that the intervention will “pay for itself” in benefits within the time horizon. Interventions estimated with a BCR less than 1.0 are estimated to yield a benefit less than the investment over the period. The Internal Rate of Return (IRR) is presented as a discount rate sensitivity analysis, indicating the discount rate that would yield a NPV of zero, at which point the intervention produces an economic benefit. The break￾even period is the estimated number of years to fully recover all PV costs. Table 5: Cost-Benefit Analysis of PACS Interventions, Ranked by BCR Findings related to each intervention are described below, beginning with the interventions that yielded the highest BCR. Every $1 spent by PACS for hand-dug wells is estimated to yield a return of $4.02 over the five-year time horizon. In addition, the direct NPV costs associated with wells (construction supplies, tools, training, and technical assistance) are relatively low ($531,016). Direct PV costs are construction materials, tools and supplies, training, and technical assistance. PV benefits correspond to health care savings, productivity gained, time saved from proximal sources and mortality averted. The 90 wells are estimated to increase overall access to safe water to 1.8% of the population in the three counties. Likewise, WaterGuard has a high BCR (3.45) and a low PV cost ($295,233). Direct NPV costs include promotional items, marketing and sales, and staff costs. Procurement costs are off-set as the product is sold at full cost recovery. The monetized benefits relate to increased productivity and reduced travel costs resulting from diarrhea cases averted. CLTS was estimated to have a high BCR (2.43), but with notably higher VP costs ($3,694,120). CLTS cost and benefit categories are similar to the hand-dug wells. The diarrhea burden is substantial in the three counties. Spectrum LiST estimates project 323,507 simple and 7,065 severe diarrhea cases in children under five years in sub-national estimates for the three counties in 2017. This is driving the high BCRs related to the hand-dug wells, WaterGuard, and CLTS. Three PACS-specific CHA program scenarios were modeled. Scenario 1 reflects the current PACS plan to discontinue CHA stipends in December 2017, as authorized by USAID. Scenario 2 assumes that stipends continue over a five-year period but no drugs or commodities are supplied. Scenario 3 assumes Intervention PV Costs PV Benefits NPV BCR IRR Break-even priod Horizon Hand-dug wells $ 531,016 $ 2,136,321 $ 1,605,305 4.02 91.8% 1.1 5-years WaterGuard $ 295,233 $ 1,018,439 $ 723,206 3.45 >1000% 5-years 0.3 CLTS $ 3,614,920 $ 8,781,010 $ 5,166,090 2.43 65.3% 1.1 5-years CHA scenario 3: Stipends & Drugs $ 5,775,953 $ 12,164,236 $ 6,388,283 2.11 94.6% 1.6 5-years CHA training $ 1,063,840 $ 1,978,755 $ 914,915 1.86 86.0% NA one-off CSO capacity building $ 1,569,537 $ 1,822,329 $ 252,792 1.16 16.1% NA one-off BCC $ 1,755,890 $ 1,165,655 $ (590,235) 0.66 -52.5% 5-years 8.4 CHA scenario 1: Current $ 3,230,240 $ 2,043,297 $ (1,186,944) 0.63 -13.5% 5-years 7.1 CHA scenario 2: Stipends continue $ 5,204,245 $ 2,783,773 $ (2,420,472) 0.53 -48.0% 5-years 9.5 20 that stipends continue and that drugs and commodities are provided to CHAs for treatment of simple diarrhea, malaria, and pneumonia. For cost-benefit modeling, Scenarios 2 and 3 include an assumption that stipends continue for five years,16 i.e., through 2021, beyond the scheduled closure of PACS. This assumption expects another organization, such as the MOH or an NGO, to continue payment of the NCHA costs for benefits to be realized within a five-year horizon. Figure 3 illustrates the three CHA scenarios: scenario 1 is the current or base case, the assumptions used for this scenario are carried forward to scenarios 2 and 3, except as noted above. Direct investment costs include CHA training and tool kits as well as CHSS training. Other costs that are assumed to continue include CHSS salaries and fuel, monthly meeting costs, and beneficiary time costs related to care-seeking at the health facility for antenatal care and delivery. PV cost categories are shown as negative values in the figures below. Benefits related to access to information and referral from the CHA as well as neonatal and maternal mortality averted related to increased ANC and delivery. This scenario is estimated to yield a five-year return of $0.63 for every dollar invested, resulting in a negative return of 37% on investment. Figure 3: Comparison of Three PACS CHA Scenarios: (1) Stipends Ending in December 2017; (2) Stipends Paid Over 5 Years, No Drugs or Commodities; (3) Stipends Paid, Drugs and Commodities Provided Over 5 Years Scenario 2 assumes that CHA stipends continue beyond 2017 for at least a five-year period. This is expected to increase CHA health promotion activities, which may yield additional mortality reductions related to increases in health center care-seeking. Expected changes in service seeking were estimated using statistical testing of 15 key DHIS-2 health service indicators described in EQ1 above. Only statistically 16 CBA requires an extended time horizon, typically five years or longer. In part, that is to ensure that initial costs such as training do not distort the benefit-to-cost ratio, as would be the case for shorter time periods. In the present case, in which PACS is working toward long-term outcomes, a five-year time horizon, extending beyond PACS’ activity period, is appropriate for the cost-benefit analysis. -$2,500,000 -$1,500,000 -$500,000 $500,000 $1,500,000 $2,500,000 $3,500,000 Scenario 1 Scenario 2 Scenario 3 Scenario 1 Scenario 2 Scenario 3 Scenario 1 Scenario 2 Scenario 3 Scenario 1 Scenario 2 Scenario 3 Scenario 1 Scenario 2 Scenario 3 2017 2018 2019 2020 2021 Drug costs CHSS salaries, fuel, monthly meetings Time at HF for ANC and delivery CHA stipends Investment costs Resources saved/productivity increased Mortality averted Access to information and referral 21 significant ‘premium’ increases were retained to estimate mortality averted using Spectrum LiST. The additional PV benefit is less than the PV cost: scenario 2 yields a similar, but lower return of just $0.53 for every dollar invested over a five-year time horizon. The negative return is due to the higher costs and disproportionately lower benefits. By contrast, scenario 3 assumes that stipend payments continue and drugs are provided. The total five￾year PV iCCM drug cost (inclusive of international shipping, wastage, and in-country distribution) is estimated at $742,152. As a point of comparison, this is approximately one-third of the total PV cost for CHA stipends which amounts to $2,245,632, a sizable investment in a service delivery platform which could potentially enable an additional $12,589,811 in PV benefits over the five-year time horizon if drugs are also provided to PACS-supported CHAs. The benefits are associated with additional mortality averted (primarily due to prompt treatment) and resources saved (including time saved from not having to go to the health facility as well as productively gained related to a reduction in the duration of the illness episode due to prompt treatment). Reduced mortality accounts for nearly 20% of benefits that would be gained under this scenario. Additionally, resources saved accounts for 59.5% of benefits accrued. Scenario 3 is estimated to yield a $2.11 return on every dollar invested over the five-year time horizon. After adjusting for PPP, PACS CSO capacity building was valued slightly under the actual costs of providing the support. Every $1 invested yielded $0.85 in benefits specific to the CSOs themselves. Several PACS CSOs explained that the application of policy changes within their organization is a challenge. PACS CSOs cited the need for general institutional capacity building. Factoring in the benefits at the community level, the return becomes $1.16 for every dollar invested in PACS CSO capacity building. Qualitative findings indicate that most PACS CSOs believed that capacity building support has empowered the community; PACS CSO representatives stated, “the community now has a voice and is a force.” Another CSO noted: “Capacity building at the community level gives local people the sense of ownership and belonging” and “it also increases their say in terms of decisions affecting their lives.” In relation to the comprehensive CHA training, CHAs highly valued the recently completed four-month training, at nearly double the direct cost (BCR=1.86). By contrast, the BCC training yielded a relatively low BCR of 0.66 over the five-year time horizon. Direct PV costs amount to $1,755,890 which include training, CHSS fuel and stipends, as well as technical assistance and support. NVP benefits are estimated at $1,165,655 and include access to information and referral by the community as well as mortality averted. Mortality averted was estimated following the statistical modeling method based on the 15 key DHIS-2 health service indicators (described above). EQ2: Cost-Benefit Analysis Conclusions The PACS water and sanitation interventions yield the highest BCRs: Every $1 of present value (PV) costs yields $4.02, $3.45, and $2.43 in PV benefits for hand-dug wells, WaterGuard, and CLTS, respectively. The daily time spent collecting surface water and going to the bush for defecation is substantial, time￾savings from proximal wells and latrines accounts for 61.7% of benefits for hand-dug wells, and 54.9% of benefits for CLTS. Direct costs for hand-dug wells represent about 5% of the PACS IR4 budget. By contrast, direct costs for CLTS represent about 66% of the PACS IR4 budget. Three PACS-specific CHA program scenarios were modeled. Scenario 1 reflects the current plan to discontinue CHA stipends in December 2017. Scenario 2 assumes that stipends continue; and scenario 3 assumes that stipends continue and drugs are provided to CHAs for treatment of simple diarrhea, malaria, and pneumonia. Scenarios 1 and 2 yielded negative returns on investment: $0.63 and $0.53 in PV benefits for every $1 in PV costs, respectively. The substantial CHA investment costs (training and initial stipend payments) can only be recovered after more than seven years if stipends are discontinued as planned. 22 Although CHA scenario 1 assumes that basic health promotion and referral would continue, it is highly questionable if that is a realistic assumption. This is because, given the high CHA expectation of continued stipend payments, as expressed in all CHA benefit valuation sessions and focus group discussions, a discontinuation is likely to have a serious and direct negative effect on CHA performance. Regardless, paying stipends to CHAs without provision of iCCM drugs (scenario 2) does not yield an economic return. By contrast, CHA services with drugs, commodities and stipends produce a 2:1 return on investment over a five-year horizon. Interestingly, this modeled BCR is very similar to the CHA training BCR (1.86). The latter is based on the valuation by the CHAs themselves who assume that drugs and stipends will be provided. This provides a level of internal validity to the modeled estimates. CSO capacity building yielded a positive net economic benefit, however with a lower BCR compared to the other intervention packages. Qualitative feedback suggests that the CSO capacity building to the CSOs themselves is constrained by the limited scope of support.17 In addition, based on interviews with gCHVs to inform the benefits quantification, the gCHVs indicated that their activities are focused on cleaning roads, removing animals from the village, and hanging clothes on a line. Although these may be important community tasks, the health impact is not documented in the literature. Therefore, these outcomes were not quantified as benefits for this analysis. Additionally, a discount rate sensitivity analysis was conducted. The results are presented as the Internal Rate of Return (IRR) which indicate the discount rate at which the activity costs are equal to the activity benefits. A higher IRR indicates that the result is more robust in the sense that the rate by which future values are discounted would decrease the BCR below 1, at which point the intervention would no longer be considered to yield an economic benefit. Five interventions yielded IRRs higher than 20%: hand-dug wells (91.8%), WaterGuard (>1000%), CLTS (65.3%), CHA scenario 3 (94.6%), and CHA training (86%). The net positive benefit of these interventions is highly robust in relation to the discount rate. EQ3: MOH Policy Changes and CHAs: How have the recent MOH policy change and transition from CHVs to CHAs affected PACS community health and WASH implementation, with attention to sustainability of the CHA program? 3a. How does the transition from CHVs to CHAs, particularly the revised remuneration package, exert different effects on each of the four result areas of the PACS implementation model? The transition from gCHVs to CHAs, as part of PACS’ alignment with the MOH new community health policy, significantly affects PACS overall program in technical and financial aspects. CHA stipends are the largest portion of PACS’ recurrent expenditures for the NCHA program, and other costs include CHSS remuneration and fuel for supervisory visits. The training costs are not considered recurrent, although future trainings for new CHAs and CHSSs will likely be necessary on a multiyear basis, to replace those CHAs and CHSSs who stop working. Together, the costs that PACS has been paying are termed as ‘NCHA costs’, which excludes the costs that PACS is not already covering. Those additional costs of the national program include motorcycles, drugs and health commodities, and stationery for the community￾based (health) information system (CBIS) that is yet to commence. None of those are present in the nine PACS-support community health districts. 17 Please note that CSO capacity building refers to TA, training, and equipment (see Table 2). It does not refer to CSO sub-grant activities. 23 Financially, PACS does not have sufficient budget to pay NCHA costs through its scheduled end date of February 2020. PACS’ payment of these costs will likely exhaust its budget by mid-2019, or earlier if PACS were to adopt the newly increased CHSS remuneration scale announced by the MOH. PACS has conducted an internal review and identified areas that would need to be reprogrammed to continue paying the NCHA costs, which are currently authorized through the end of 2017. 3b. Based on the design of the CHA program and its incorporation into PACS’ model of community health, describe factors that may affect the sustainability of the CHA program as well as any opportunity costs in relation to diversion of resources from planned activities. Include gender and social factors in this analysis, as relevant. CHA activities were observed in six of the nine PACS-supported community health districts, spanning all three counties. Some of the remaining 12 districts in the 3 counties are supported by other partners for community health, notably Plan International in Nimba and Lofa counties. In Bong County, there is no NGO support for the NCHA program in the five districts not covered by PACS, nor is there any observable cascade effect from PACS support via the CHTs to these districts. Over 90% of PACS’ CHAs had been gCHVs, whom PACS had previously trained in iCCM in 2016. Their 2017 CHA training repeated the iCCM component of that earlier training, along with three additional components as part of PACS alignment with the MOH’s NCHA program. The selection of CHAs was made by their home communities, with guidance from PACS and CHTs. Eighty-three percent (551 of 662 total) of CHAs in the PACS community health districts are male. The male bias in CHA selection appears to be a common feature of the CHA program nationwide, despite the NCHA policy selection criteria that “females should be given preference” (p. 9 of Revised Community Health Services Policy, 2016-21). It was too early to define actual effects of this gender distribution on program effectiveness, but in several interviews and FGDs with female community members, TTMs and CHAs, respondents said that women were uncomfortable discussing female health issues with a male present. PACS had recently completed the training of CHAs in all four modules of the national CHA curriculum at the time of data collection. The CHAs had already begun working in communities through their training period. All CHAs interviewed expressed satisfaction with the training itself, describing it as providing them with a greater range of knowledge and skills. However, none of the CHAs were provided with drugs or health commodities, and this was universally cited as an impediment to using the knowledge and skills that they had acquired in training. PACS was not designed to provide drugs and commodities, and USAID has not authorized PACS to supply those on a routine basis. Across all respondents – spanning community members of both genders, health workers at facilities and CHSSs, District and CHT members, and CHAs themselves – there is agreement that the lack of provision of drugs and health commodities for CHAs hinders the work that they do. Several CHSSs and members of the District and CHTs remarked that the credibility of CHAs and the national program is being undone as a consequence. Some CHAs said that they are already receiving negative feedback from households, who want actual treatment for illnesses rather than advice and “a piece of paper” (referral slip for the health facility). These observations, taken four months after PACS’s alignment with the NCHA program, spanned all three PACS counties. To provide a basis for comparison of the PACS implementation of the community health component, site visits were conducted in non-PACS districts and counties. Annex XIII (Comparison of NCHA Program Implementation: PACS and Other Models) summarizes the findings from this comparison across PACS and non-PACS counties. Key points from this comparison are that PACS is currently fielding the largest NCHA-aligned community health program in Liberia, covering twenty districts with WASH interventions 24 of which nine districts include the integrated community health and WASH approach. The NGO models, including PACS, all outperform the CHT-led interventions, but all models are prone to a male bias in CHA and CHSS selection. PACS is aligned with the NCHA program in its human capacity elements, but is not designed to operate a supply chain for drugs and health commodities, nor has USAID authorized such supplies. The NGO model that outperforms PACS in some respects, Last Mile Health in Grand Gedeh18, adds significantly more staffing and infrastructure than the NCHA strategy envisions and draws on private external funding to do so. On the other hand, PACS is the only model that provides WASH as a co￾intervention, though with limited direct interaction between health and WASH components. 3c. What are the recommendations for redirecting PACS’ focus to align more with MOH priorities? The recommendations regarding PACS alignment with MOH priorities are addressed in full in the recommendations section of this report, on pages 40-42. EQ3 Conclusions PACS has aligned itself to the new NCHA strategy to the extent allowed by its mandate (i.e., in human capacity strengthening for CHAs and CHSSs, but excluding provision of CHSS motorcycles, and CHA drugs and commodities). CHAs and CHSSs trained by PACS have a broader, deeper skills set and understanding of community health than the preceding gCHV cadre, which had been trained in only one of the four modules in the CHA curriculum. Among comparators in other counties, CHAs and CHSSs trained by PACS are on par or more advanced in skills than peers in comparator counties. This alignment to the MOH policy change has come at some operational cost to the activity. For instance, after training gCHVs in iCCM, PACS then re-trained many of the same people as CHAs using the new MOH curriculum, including another iCCM training. PACS cannot be held fully accountable for that inefficiency in its outputs, as it is not clear how much USAID and PACS factored in (or were aware of) the forthcoming 2016 NCHA program when designing and implementing the 2015 training for gCHVs. More significantly, PACS has had to ‘restart’ its community health intervention in order to align with the new policy, and that shift has significantly affected the nature, timing and extent of PACS deliverables. Several critical risks to the NCHA program were identified, in both effectiveness and sustainability. These risks are not specific to PACS, and the comparison of PACS implementation of the NCHA program against other models in Liberia shows that the risks extend nationwide to all implementing partners in community health. Nevertheless, PACS must respond to these issues, in collaboration with the MOH, other GOL bodies, other implementing partners and USAID. The risks to effectiveness and sustainability are described below. Effectiveness CHAs must be supplied with drugs and health commodities in line with their training; supporting and paying stipends for CHAs and CHSSs is insufficient. Evidence from community discussions, interviews with all cadres of health staff, and the cost-benefit analysis (see EQ2) all point to a common conclusion: CHAs are effective only if provided with drugs and commodities to serve their communities. Otherwise, CHAs are perceived as no more than gCHVs with a higher stipend. From an economic perspective, paying stipends to keep CHAs active without drugs and commodities provides the lowest benefit to cost ratio (BCR) of any scenario for the NCHA. On the other hand, supplying drugs and health commodities to 18 Last Mile Health also operates in all eight districts of Rivercess County, with substantial USAID support for community health. 25 CHAs (and paying stipends for their work) yields a 2:1 positive return in health benefits, apart from also meeting demands from communities and being in line with the NCHA program. The bottom line is that the NCHA program presents an effective, cost-beneficial design. PACS is currently restricted from implementing that design in full because it is not designed to supply drugs and commodities to CHAs, nor has USAID authorized a supply chain component for PACS. A solution to the lack of drugs and commodities is necessary for PACS to field an effective community health program aligned to the MOH policy; otherwise, PACS will be neither an effective nor economic intervention to fund. Another significant risk in the NCHA program is the widespread gender bias in selection of CHAs and CHSSs. The NCHA policy requires CHA selection by the communities with a preference for female candidates. Much of community health is geared to address women’s unmet reproductive, maternal and other health needs, and women are also considered more responsive to child health needs. In a strongly patriarchal society such as rural Liberia, the pool of eligible females meeting CHA qualification criteria may be smaller than that of males. All evidence is that PACS, in common with other implementing partners across the country, made little to no effort to proactively recruit women as CHAs and simply allowed male bias to reproduce in the CHA program. It is not possible at this stage to adjust the current gender composition of CHAs. The risks to program effectiveness, if women do not engage fully with male CHAs, will need to be addressed by PACS, the MOH, MOGCSP and other stakeholders in community health. Sustainability There are two issues related to sustainability of PACS financing of CHAs and CHSSs under the NCHA. This discussion assumes that the question of effectiveness is resolved, meaning that CHAs are supplied with drugs and health commodities so that it makes sense for PACS to continue to pay their remuneration. First, there is a wide difference between PACS and GOL staff in their interpretations of sustainability. PACS sees sustainability as handing a set of activities and results over to the MOH. MOH staff see sustainability as finding another donor-partner pair to substitute with similar activities and funding after PACS closes. To date, the two parties have not had a discussion on the practical elements of sustaining PACS achievements. Secondly, there is no plan at the national or county levels to undertake payment of CHA and CHSS remuneration, even as demands for increases in the stipends are being considered. PACS has a limited budget and time frame with which to work with the MOH and CHTs on securing future funding for this remuneration beyond PACS. Without a plan for sustained financing of these NCHA costs, there is a strong risk that current investments in the community health system may collapse as soon as PACS completes its funding. EQ4: WASH and the Integration of Community Health: How do the results from the implementation of the WASH component impact the community health outcomes, including gender as a factor? The distribution of PACS’ CLTS sites and hand-dug wells for the three counties, as of July 2017, is shown in Figure 4. As per design, the wells, shaded in white, light green, or dark green, are situated in CLTS communities that have been certified as open defecation free (ODF). The shading of the hand-dug wells relates to the population served by each well, as shown in the map legend. It is expected that each well serves a population of no more than 250 people, shaded in white. For this analysis, data on additional, non-PACS wells was not available. Therefore, the wells shaded in light green (possibly serving up to 500 population) and those in dark green (linked to communities of 500-5000 people) must be examined further, to ascertain the number of other wells that serve those communities. 26 Figure 4: Map of PACS CLTS Sites and Wells in Bong, Lofa and Nimba Counties, July 2017 The PACS CLTS sites, shown in red on the map, largely cluster around rural population centers, as expected. It is notable, however, that some districts in each of the three counties have zero or only one CLTS site under PACS. Further, as is apparent from the siting of CLTS communities in Lofa County, there is a tendency for CLTS communities to be close to the major roads. 4a. To what extent has the capacity of WASH personnel been built to effectively manage water and sanitation infrastructures? WASH activities in Liberia operate through the 2012-17 WASH sector capacity development plan, led by the NWSHP Secretariat. PACS support under this plan comprises technical assistance to the NWSHP Secretariat to conduct quarterly review meetings, provision of basic materials including furniture and water quality analysis kits to county MOH and MPW offices, and training of 2,817 Natural Leaders at the community level, of whom 2,186 (78%) are males and 631 (22%) are females. Natural Leaders were trained in community mobilization skills to enable them to trigger their communities and catchment areas in undertaking CLTS and become open-defecation free (ODF). PACS further trained 226 of these Natural Leaders (67 female and 159 male) to become WASH entrepreneurs through advanced training in well digging, culvert lining, and installation and repair of hand pumps. PACS also provided the WASH Entrepreneurs with tools for the repair of hand pumps in their communities as well as soap-making materials as a means of generating income to sustain themselves. PACS trained 75 Community Water Committees in the communities where wells have been constructed and hand pumps installed, though it has not applied a gender lens or criteria to these committees. In the communities visited, decision-making and management regarding water and sanitation infrastructure was primarily led by men, from the Natural 27 Leaders and Communities Water Committees through to the county-level EHTs. Women are the primary users of community wells, but are not consulted about water standards or included in the planning and siting of wells at the level of the communities. In CLTS communities, including those that have become ODF, the sanitation infrastructure comprises locally constructed latrines, showers, and hand-washing stations. This infrastructure is to be maintained by the households that built and own them, in accordance with the CLTS approach of local ownership. In most communities visited, households did maintain that infrastructure. In a few communities, however, latrines and shower stalls had collapsed, reportedly due to poor construction with weak materials, such as weak sticks covering latrine pits. Compounding the infrastructure decay, many households in these communities resided away from home to be close to their farms during planting and harvesting seasons. While residing near their farms, household members said that they do not construct a second set of latrines and revert to open defecation in the bush. When their household latrines become unusable from lack of maintenance, or when the latrine pits become full because they are too small for the household size, community members simply continue defecation in the bush as they had practiced on their farms. These factors need further exploration, and there may be other mechanisms and behaviors that lead to ODF communities relapsing to open defecation. 4b. To what degree are PACS supported WASH activities likely to be sustainable (technically, socially, financially and environmentally), considering gender and relevant social factors? Among PACS’ WASH activities, hand pump maintenance and repair appear to be sustainable, particularly in communities with a cash-box savings system and in areas where WASH Entrepreneurs are active. Latrine maintenance and repair face challenges to sustainability due to the CLTS emphasis on use of local materials and lack of technical guidance. Most CLTS communities show a strong motivation to remain ODF, and social pressure is strong to construct and use latrines in those sites. A few communities have reverted to open defecation when their latrines became full or broken, and the CLTS approach may have a sustainability limit in the Liberian context. Construction of new wells is not likely to be a sustainable feature of PACS’ interventions, both because of insufficient knowledge and experience among WASH Entrepreneurs, the need for professional materials, and relatively high costs. Technical factors: Many WASH Entrepreneurs were already engaged in using their skills for maintenance and repair of hand pumps, including those not constructed by PACS. A few had established small businesses, with knowledge of spare parts suppliers and marketing of their services across PACS and non-PACS communities. WASH Entrepreneurs were able to demonstrate the technical processes and procedures involved in the fabrication of culverts (e.g., concrete mix ratio and curing time). None of the WASH Entrepreneurs who participated in FGDs, however, were able to describe in detail the number of culverts to be submersed under water, so as to meet the required water column in the construction of a new well. Natural Leaders have led the CLTS approach in their own and other communities. They have motivated communities to develop strong norms against open defecation, and used those norms to guide communities in developing action plans to become ODF with minimal external guidance. CLTS latrines observed during field visits were constructed with local material with little or no technical advice. The majority of latrines were functioning and in use, but some latrines had become full and required decommissioning 1-2 years after construction. As Natural Leaders focus on community motivation for sanitation, they do not appear to have or use knowledge about latrine construction guidelines. 28 Social factors: Women are the primary users of PACS’ hand-dug wells and pumps, but are not adequately involved in WASH activities. 19 Most WASH Entrepreneurs and Community Water Committee members are male, and despite recognition that women use water more often than men, there is no structured mechanism to involve women or their perspectives in water management. The wells have rules to regulate the opening time and the closing of the hand pump during the day, which supports long-term sustainability. CLTS has been well accepted by many communities, leading to transformative ODF status and other aspects of local empowerment in many cases. However, when locally constructed latrines become ‘spoiled’ (broken, filled, etc.) within 12-24 months, communities have relapsed to use of the bush for defecation, despite their own rules and MOUs. Financial factors: The ‘cash box’ system appears to promote collections and/or savings for hand pump maintenance and repairs. The system has some safeguards against loss, but those have not been tested in actual situations of loss, theft or manipulation. All WASH committees were trained in the cash box system, but many have not been provided with actual cash boxes. Other informal community savings systems are operational in some communities due to the lack of cash boxes, but those are less common. The use of locally available materials of the construction of CLTS latrines may have supported the rapid adoption of community sanitation, though it may also lead to early deterioration of the latrines. Environmental factors: All 82 wells constructed by PACS considered the management of waste water from the hand pumps. All wells are constructed with a domed shape apron and a properly structured drainage system with a soak away pit, which allows wastewater to drain from the slab to the soak away pit where it is properly drained into the ground. A few wells were observed to be built in close proximity to latrines, within 30 meters, creating a potential fecal contamination risk. 4c. To what extent does the quantity of water supplied by PACS supported facilities (boreholes, dug wells, etc.) meet WHO or national standards? As previously noted, 82 out of 90 targeted hand-dug wells have been completed to date, with the remaining eight wells due for completion by late 2017. However, only 19.5% of existing PACS-supported wells (16 out of 82) meet GOL water quantity standards as specified in the guidelines for water and sanitation service in Libera (October 2010). The guidelines require that "any well done in Liberia during the rainy season May 31 to October 31, the water column should be 6 meters and well done during the dry season November 1 to April 30, the water column should be 4.5 meters.” Fully, 80.5% (66 out of 82) of these hand-dug wells do not meet GOL water quantity standards for adequate water column in the dry and rainy seasons, with a high risk of either no water during the peak of the dry season (January to March) or insufficient water in the wells relative to the population size in the catchment areas. To assess PACS wells against the USAID water quantity standards of 20 liters of water per person per day, there are few factors that need to be considered. These include, but are not limited to, the yield of the well, recharge rate, and water fetching time. As it is, PACS has not tested its wells to determine the yields and the recharge rates. In relation to the USAID water quantity standard (yield of 20 liters per person per day per pump, over eight productive hours of use per day), 46% (38 out of 82) of PACS￾completed wells do not meet the USAID water quantity standard, based on the GOL-stipulated maximum 19 Community FGDs in all three counties. 29 population of 250 people served by each well. On the other hand, the 54% (44 of 82) of wells that meet USAID standard might fall short of the standard during the peak of the dry season, especially in counties like Nimba where the average maximum yield per well per hour is 640 liters. 20 Based on discussions with the Environmental Health Teams (EHT) both at the county and district levels, the county WASH office and EHT have not been adequately involved in the construction of wells in the different communities, which has affected monitoring of the water quantity during the construction stage. PACS has limited the County WASH Coordinator, county, and district EHTs to site selection and not involved them in the construction of wells until after the wells are sealed and pump installed. At that time, assessing the water column is difficult because it would require removal of the well cap, calculating the water column based on measurement of the static water level and the well depth, and replacement of the cover, all of which would contaminate the water and require post-assessment chlorination. In sum, there is no procedure for either PACS or GOL to assess sufficiency of water quantity from the PACS-supported wells on a routine basis. The only information available on water quantity is from the original well completion reports, which are not only dated but also indicate that over three-quarters of PACS wells are below national standards and may not deliver sufficient quantity of water in all seasons. 4d. To what extent has the use of a water quality assurance plan, if any, contributed thus far to meeting and ensuring the standards for quality of water supplied to communities? PACS has conducted initial (pre-handover and use) water quality testing at two milestones for each of its wells: on reaching ground water and after fitting the hand pump. After well completion, PACS has not conducted routine water quality assessments of the wells it has supported in its communities, which is required by GOL standards every six months (though apparently not widely practiced by either GOL or other NGOs). County and District EHTs said that they face capacity constraints based on the volume of water samples to be collected and tested, as well as limits on their ability to travel across the districts. The County EHTs further asserted that PACS was developing a water quality assurance plan, but copies of such a plan were not available with either the EHTs or PACS staff during the field visits. There is no evidence that a water quality assurance plan has been drafted, finalized or implemented, by either PACS or GOL. During field visits, PACS as well as MPW and MOH teams at the county levels were unable to present a copy of a water quality assurance plan. The absence of such a plan contravenes GOL guidelines for water and sanitation services in Liberia and does not satisfy one of the four Initial Environmental Examination criterion of ensuring the availability of a water quality assurance plan. EQ4: Conclusions PACS is progressing to meet the output target for hand-dug well completion, but has not taken steps to ensure that the completed wells meet GOL water standards for water quantity or quality, nor applied an adequate gender lens to its WASH interventions. Although women are expected to comprise at least 50% of WASH committees, WASH Entrepreneurs and community women are not adequately engaged in assessment of water quantity or quality from the new hand-dug wells. Though the 82 wells constructed by PACS meet GOL technical specifications and design including the installation of hand pumps, a vast 20 ACF WASH. Evaluation of hand dug wells and boreholes case study of Nimba County 2006, page 6). 30 majority of those wells (78%) do not meet GOL water quantity standards21, and no information is available about current well water quality, except that no water quality assurance plan is evident in the field. With the transfer of knowledge and skills to community members through the training and equipping of Natural Leaders, WASH Entrepreneurs, and Community Water Committees, the WASH supported activities of PACS have a high potential to meet sustainability objectives. However, with the majority of wells not meeting GOL minimum standards and a few communities relapsing from ODF to OD status, there is a need to work with the structures already organized to ensure that the quantity of water in the wells is monitored especially during the peak of the dry seasons, water quality is monitored through a well-developed water quality monitoring plan, and communities be provided with technical guidance as it relates to materials to be used for CLTS latrines. The cash box system is an appropriate means of sustaining both WASH infrastructure, activities as well as community ownership, but has not been fully rolled out due to the lack of the cash boxes. Based on interviews in PACS counties, some outcome-level changes have arisen due to PACS capacity building in WASH. In some instances, Natural Leaders and WASH Entrepreneurs have formed community￾based organizations for the purpose of working with NGOs and other organizations, to be hired for WASH infrastructure construction, maintenance and repair. A few WASH Entrepreneurs have begun small businesses of retailing pump parts as replacements for communities that have pumps requiring repair. These advances in WASH personnel capacity have not been widely documented or disseminated across the network of Natural Leaders. Similarly, the relapse of some ODF communities has neither been documented nor acted on, by PACS, GOL or other actors. CLTS has been widely accepted in many PACS communities. Exceptions to CLTS adoption and cases where ODF communities have relapsed to open defecation are not incorporated into the PACS model or a Liberian refinement of the CLTS approach. While communities are motivated to construct their own latrines, there appears to be no guidance to communities on latrine construction, e.g., weak stick coverings that collapse over pit latrines, or pour-flush latrine compost pits with inadequate space for the household size and expected usage. This lack of guidance leads to wasted effort by community members, frustration, and sometimes relapse of ODF communities to open defecation. Further, the clustering of CLTS sites in close proximity to roads may also reflect settlement patterns, but the geographic gaps in CLTS coverage, shown in Figure 4 above, require further analysis. In particular, the population not covered by CLTS should be categorized based on equity factors, including ethnic or linguistic minorities or socio-economic isolation. RECOMMENDATIONS EQ1: Performance Recommendations based on findings and conclusions related to PACS performance include the following: 1. Theory of Change: PACS must develop an outcome-oriented theory of change, not only to document its current model of community health and WASH but also to ensure that it raises its performance to the outcome level and incorporates an explicit focus on gender. The PACS theory of change can be simple, and must emphasize integration across components and sustainability through regular GOL involvement and engagement at national, county and community levels. a. USAID should support PACS in a shift toward an outcome focus by reducing the reporting and meetings to no more than once per quarter. Concurrently, USAID should demand 21 Water and Sanitation Services in Liberia, October 2012. 31 outcome-level data on a scheduled basis, with semiannual and annual reports being a suggested mechanism for higher level results reporting. 2. Gender: PACS should focus on gender as a cross-cutting link across components interventions. PACS should seek technical assistance from a gender expert to update its senior and field staff on the value and appropriateness of a gender approach in its interventions. 3. Sustainability: PACS should immediately design and implement a hand-over process for sustainability of its outcomes, with full involvement of GOL stakeholders in the planning and execution of the hand-over process. As a first step, PACS and GOL must agree on a shared definition of sustainability for PACS components. PACS should communicate to all stakeholders the potential for an earlier close-out of its activity if it pays the NCHA costs as outlined in EQ3, below. a. To ensure the sustainability of PACS contributions to community health, GOL stakeholders should define their own commitments to sustainability of PACS interventions. b. Specifically, MOH and CHTs, and MPW and EHTs, must set their own targets for learning from PACS about community engagement and WASH infrastructure management, and incorporate actions toward those targets into their workplans. Those bodies must decide whether DSA is appropriate for routine work with PACS, as opposed to DSA payments for a special event such as a workshop or planning retreat. 4. Integration: Starting with the next annual work plan (FY 2017-18), PACS should integrate its interventions, by linking CHAs and Natural Leaders in community-level structures so that health communication and sanitation efforts directly reinforce each other. At a management level, PACS teams must collaborate in their interventions rather than merely update each other on different activities. Similarly, the PACS M&E staff must orient other PACS and GOL staff on the expected outcomes to be measured and achieved, and streamline internal reporting, so that all information flows to a common set of databases, from which the partners draw data for their home organizations. 5. Engagement with GOL: PACS should involve GOL stakeholders in implementation, particularly in community engagement and management of new WASH infrastructure. Additional, specific recommendations for PACS programming components include the following: • IR1: Transition capacity building from trainings and workshops to mentoring, by including GOL and stakeholders in PACS ongoing work; • IR2: Continue support to the NCHA program only if drugs and health commodities are provided to CHAs (see EQ 3 for more details); • IR3: Consolidate community-level BCC efforts with CHAs and Natural Leaders; • IR4: Complete the eight remaining wells, and train WASH Entrepreneurs to conduct routing testing of water quantity and quality; • IR4: Work with communities to resolve the emerging issue of relapsed ODF communities, and involve GOL and international CLTS partners in refining the model to address the factors that lead to relapse to open defecation; and • IR4: Revise WaterGuard sales targets upward to support aggressive expansion of WaterGuard sales to meet current market demand and generate further demand; and develop a mechanism to continue WaterGuard sales on a self-sustaining basis including most indirect costs before PACS’ closure. The revised targets for the number of liters of drinking water disinfected with point-of￾use treatment products as a result of USG assistance should be linked to the targets for the corresponding F Indicator (Outcome HL 8.2-6) for the percentage of households in target areas practicing correct use of recommended household water treatment technologies. EQ2: Cost-Benefit Analysis 32 Recommendations based on cost-benefit analysis are to be considered along with recommendations based on findings related to the other evaluation questions. 1. NCHA Costs, Including CHA Remuneration: PACS should continue to pay the NCHA costs it has undertaken only if MOH and the CHTs find a means to supply drugs and health commodities to the CHAs. Payment of NCHA costs without supply of drugs or commodities provides negative economic returns, and therefore not recommended. Detailed recommendations related to each scenario examined for the NCHA program are presented under EQ3, below. 2. WASH: PACS should consider allocating any additional resources to hand-dug wells, which provide the highest economic return of all its components. 3. PACS should expand WaterGuard sales as it is low cost with a positive net economic benefit. 4. Capacity Building: PACS should limit further capacity building efforts, as those provide very limited economic returns. EQ3: MOH Policy Changes and CHAs Based on the findings and cost-benefit analysis regarding the NCHA program, recommendations are presented for three scenarios: payment of NCHA costs for a limited period, discontinuation of NCHA payments, and payment of NCHA costs through the PACS remaining time frame. These recommendations are tailored for PACS, USAID and the MOH, as follows: Scenario 1: payment of NCHA costs for a limited period PACS should continue to pay NCHA costs (remuneration for CHAs and CHSSs, and other related expenses) only if CHAs will be supplied with drugs and health commodities before the end of Year 4 (i.e., before September 30, 2018). • A six-month period for Year 4, October 2017-March 2018, is recommended for continuing with the current payments to allow time for PACS to reach an agreement with the MOH and CHTs on regular supply of drugs and commodities for CHAs in the three counties, in line with USAID requirements. This is an extension of the current 3-month authorization for payments in Year 3, up to December 31, 2017. • PACS should initiate discussions with MOH and the CHTs regarding supply of drugs and commodities to CHAs; • USAID should authorize payment of NCHA costs by PACS through March 31, 2018; • USAID should participate with PACS and MOH in the negotiations regarding supply of drugs and commodities; • MOH and the CHTs should commit to the supply of drugs and commodities to CHAs, including use of the NDS or alternative mechanisms if the NDS is not in a position to deliver, with a clear schedule, procurement process and financing. Scenario 2: reduction or discontinuation of NCHA payments If an agreement on supplying drugs and commodities is not reached, or if any other determination is made that PACS will not continue to pay NCHA costs, then PACS should redirect its community health funding as follows: • Hand-over responsibility for CHA stipend payments to MOH (or another partner), with the following options: o Discuss with MOH a plan for absorption of the PACS NCHA costs into the MOH and county health budgets, including provision of drugs and commodities as well as remuneration; o Reduction in the geographic scale for PACS support to the NCHA program, for instance supporting one or two counties rather than the three counties, or focusing on a reduced number of districts overall; 33 o Establish a co-financing mechanism for NCHA costs, with either the MOH/CHTs or another partner, so that CHAs remain active and supported while PACS reduces its payments; o Discontinue payments of the NCHA costs, with open communication about the decision￾making and the scheduled time-frame for cessation of payments, including notice to the affected CHSSs and CHAs of at least 30 days prior to cessation or in accordance with Liberian law. • Use funds to enhance other PACS components: o Expand support to CSOs for community health; o Allocate additional funds to hand-dug wells, prioritizing CLTS communities with the greatest need based on population per existing safe water source (GOL standard is 250 population per well); o Invest in extending the sustainability limits of CLTS/ODF that were discovered, by working with communities that have relapsed from ODF status as well as drawing on international lessons from CLTS practice; o Incorporate Natural Leaders in health messaging for CLTS communities; o Explore alternative means of enhancing systems that support and sustain community health. Scenario 3: Continue payment of NCHA costs • Provided that CHAs are supplied with drugs and health commodities, PACS should continue paying NCHA costs so long as its budget will allow, after reprogramming other components based an internal review and recommendations in this evaluation. o Given its limited budget, PACS should ascertain and communicate with the MOH and CHTs regarding the projected end date for paying NCHA costs. o To extend the time frame for PACS support, PACS and the MOH may consider alternative arrangements such as a reduced geographic scale limited to fewer districts, and modest but rising contributions from GOL toward the NCHA costs over the remaining life of the activity. Regardless of whether PACS will continue to cover NCHA costs, it is critical that PACS prioritize sustainability of the program starting with the Year 4 (2017-18) operating plan to ensure a smooth hand￾over to the county health teams and MOH at an earlier date. Given the projected depletion of the PACS budget by mid-2019 if PACS were to continue to pay NCHA costs, the issue of sustainability becomes even more important under scenario 3. PACS’ role in sustainability should include the following: • PACS senior leadership, in consultation with USAID, negotiate with the Director of Community Health Services and the CHOs from Bong, Lofa and Nimba counties and develop an agreement on what components can be sustained and what cannot, without assuming that another NGO will be funded to do similar community health work after PACS. That agreement should be reflected in the annual workplans for PACS, the CHTs in each county, and the MOH/DCHS; it may also inform the design of future USAID support to community health. • Delegate much of the community engagement and BCC work to CHSSs and CHAs, and involve Natural Leaders and WASH Entrepreneurs as well; • Mentor the CHAs and CHSSs, so that they practice and gain experience in the skills acquired from the NCHA training, and avoid giving the impression to CHAs and communities that PACS staff are providing the BCC messages; 34 • Communicate with the MOH, including CHTS, and other stakeholders that PACS will plan to close at an earlier date because its budget is being consumed by the CHA stipend payments and other NCHA costs; • Work with the MOH, CHTs and other key stakeholders to develop and finalize a sustainability plan within Year 4, and to start the hand-over of PACS components no later than 12 months prior to the expected closure of PACS. This plan must define sustainability as the MOH taking on the financing and operation of PACS activities, including the NCHA program, rather expecting another donor-financed activity to substitute for PACS. The MOH role, including CHTs, includes the following: • The MOH should convene a national review of the NCHA program, to address and resolve the identified risks to effectiveness and sustainability of the program from lack of drug and commodity supplies to CHAs, gender imbalance among CHAs and CHSSs, and financial uncertainty, as well as challenges identified by other partners. This review should involve MOH as the lead, with MOGCSP, PACS and other implementing partners, and USAID and other donors as key partners. USAID’s role in sustainability should include the following: • Discontinue the biweekly/monthly meetings and reporting requirements for PACS, in recognition that sustainability is a higher-level outcome that cannot be monitored on a weekly-to-monthly basis (such monitoring diverts attention from outcomes to activities); • Require a sustainability plan by the first quarter of Year 4 (December 2017), including outcome measures for PACS mentoring and GOL performance, with reporting against outcomes in PACS annual reports. EQ4: Integration of Community Health and WASH PACS should address the gaps in its water component as follows: • Conduct routine pump testing in accordance with GOL requirements for all of the wells to determine their yield and recharge rates; • Monitor the wells during the peak of the dry season (January-March) and take appropriate actions for all wells that run dry during the period; • Work with the EHT and the Community Water Committees to decommission the latrines that are close to the hand pumps; • Ensure the cash box system is functional in all areas with WASH committees, including provision of cash boxes at the conclusion of trainings; • Work with MOH and MPW to increase the capacity of WASH Entrepreneurs to include digging of wells from start to finish and to monitor residual chlorine level of wells as per GOL standards; • Work with County WASH Coordinators to conduct test pumping of the wells to determine and monitor the yield and recharge rates, and modify those wells with severe water quantity deficiencies to ensure that these wells provide water year-round. • Reinforce the network of WASH Entrepreneurs and link them to both spare parts suppliers as well as to communities and NGOs that build WASH infrastructure, as a means of expanding local WASH infrastructure management; • Facilitate a link between the County EHT/WASH Coordinator and the WASH Entrepreneurs, who may assist in routine monitoring of the hand pumps and water quality assurance, including well chlorination; • Document initiatives by Natural Leaders (for instance, income generation through agriculture) and share across the Natural Leader networks; • Enhance the technical support to community members, in ensuring that new latrine construction meets basic standards over a length of time acceptable to households; 35 • Work with Natural Leaders and ODF communities to establish simple pit latrines in farm areas outside of settled communities; and • Explore other exceptions and relapses from ODF status, to refine the CLTS model; • Examine the clustering of CLTS sites, identify districts and communities that are outside the CLTS coverage, and work with CHTs and DHTs to plan for CLTS triggering in those areas; and • Expand WaterGuard sales targets and performance, as noted under recommendations for EQ1. PACS Activity Management and M&E PACS should complete the following steps by end-2017, to serve the activity through its remaining period of operation: • Develop a PACS theory of change, with clear end-of-activity outcomes agreed with USAID and a focus on gender and sustainability; • Drive implementation for the remaining activity period based on three key principles: sustainability, integration, and outcomes; • Reinforce involvement of GOL counterparts and other stakeholders in implementation, to reinforce the positive relationships, and negotiate with MOH/CHSD, MPW, and the CHTs the occasions when DSA should and should not be provided to GOL staff, based on routine operational tasks (such as community engagements, CLTS triggering, monitoring of wells – no DSA provision) and planning/strategic events (such as annual work planning, discussions on the NCHA program – DSA may be provided); • Integrate PACS’ own internal operations through key steps such as sharing of offices and communication of a PACS’ identity; joint design, planning, and management of field interventions, both within and across result areas; pooled budgeting to support integrated field implementation (e.g. budget by activity components, rather than by partner); and integrated M&E and internal reporting for all components; • Improve data quality, at a minimum with gender-disaggregation of all population data in PACS databases and reports. USAID should undertake the following steps to support PACS in its activity management: • Negotiate a revised schedule of senior management meetings, potentially quarterly in relation to the reporting schedule, to support management attention to outcomes over tasks; • Leverage PACS’ relationships with GOL (MOH, MPW, and MOGSCP) to deepen USAID’s own engagement with GOL, for instance by accompanying PACS senior management on at least two meetings with GOL directors per quarter (these joint meetings with GOL may substitute for the existing USAID-PACS biweekly meetings); • Emphasize outcomes and gender in PACS reporting, with an agreed schedule; • Support PACS in negotiations with GOL regarding payment of DSA. UTILIZATION PLAN The evaluation strongly recommends that a joint review meeting of PACS, USAID, and GOL representatives from MOH, MOGCSP, and MPW be held to agree on next steps based on the recommendations put forward in this evaluation report. A shared meeting would enhance dialogue among PACS, USAID, and the GOL partners and be useful in building relationships. Particularly with regards to the risks identified in the NCHA program, it is important that all parties work together toward a common approach. The method for determining actions that was employed in the joint planning session with PACS and MOH (June 2017) can be drawn on to reinforce use of the evaluation. 36 LSA plans to facilitate a learning event between one to three months after submission of the final evaluation report with USAID, PACS, GOL, and other stakeholders. This will provide an opportunity to apply findings and lessons learned directly to relevant decision making, including policy development, and larger sectoral assessment needs. The workshop will be a one-day session facilitated by LSA’s Collaborating, Learning, and Adapting team with the aim of ensuring that the findings of the performance evaluation are being used. One or more additional learning events should be considered by USAID and MOH, to explore and identify mitigations and solutions to the risks to the effectiveness and sustainability of the NCHA program, particularly focused on the supply of drugs and health commodities to CHAs. Along with MOH, USAID, and PACS, participants in such an NCHA-focused event should include other NGOs, implementing partners, and stakeholders in community health, such as Plan International, Last Mile Health, UNICEF, and Conseil Sante, as well as partners in health sector supply chain management. Evidence and experience from NCHA program implementation elsewhere in Liberia, as well as other cost-benefit analyses of the NCHA program, should be discussed alongside the findings, conclusions, and recommendation in this evaluation. An outcome of those discussions should be a resolution of the identified risks to effectiveness and sustainability, so that the MOH and partners can plan an effective use of resources for community health over the coming five to ten years. 37 ANNEXES ANNEX I: EVALUATION STATEMENT OF WORK22, 23 PARTNERSHIP FOR ADVANCING COMMUNITY-BASED SERVICES (PACS) MIDTERM PERFORMANCE EVALUATION STATEMENT OF WORK 1. Background This Statement of Work (SOW) describes the conditions of work and terms of reference for a Performance evaluation of USAID’s Partnership for Advancing Community-based Services (PACS) activity. The objective of PACS is to strengthen the delivery of high quality community health, water, sanitation and hygiene, and social welfare services by providing technical and operational support to the Ministries of Health (MOH), Public Works (MPW), and Gender, Child, and Social Protection (MoGCSP), as well as to civil society organizations (CSO). The evaluation will measure results, examine the impact of health systems support on service delivery improvement and management efficiencies, and investigate the effectiveness of specific technical assistance and capacity building approaches. The Mission will use findings from the evaluation to inform more strategic USAID investment in the sector and continue support to the delivery of community health services. Overview of PACS Activity: Partnership for Advancing Community-based Services (PACS) Contract Number: AID-669-A-15-00001 Activity Dates: February 23, 2015 – February 22, 2020 Funding: $31.8 Million (original 24.8 Million + EEF 7 Million) Implementing Partner: International Rescue Committee Subs: Global Communities, Population Services International, Young Men Christian Association of Liberia and Planned Parenthood Association of Liberia AOR: Jannie M. Horace 22 In this original Statement of Work, the selection of counties for field visits was two original PACS counties (Lofa and Nimba), one EEF county (rural Montserrado) and one non-PACS county (Grand Gedeh), as described in Section 4 below. Based on discussions with USAID, this selection was modified to the three original PACS counties (Bong, Lofa and Nimba) and two comparison counties (Grand Gedeh and Grand Cape Mount). PACS’ EEF counties were not included in the midterm evaluation as support was scheduled to end within five months, by December 2017. 23 The evaluation questions in this original statement of work were revised based on discussions with USAID in the inception phase. The revised evaluation questions are provided in the executive summary (pages 4-5) and main body of the report (pages 13-14). 38 A/AOR: Wondwossen Teffera Geographical focus: Bong, Lofa, Nimba EEF Counties: Margibi, Montserrado, and Grand Bassa Since 2008, USAID has supported the rebuilding of Liberia’s health system through its flagship Rebuilding Basic Health Services Activity (RBHS) that provided critical support to the Ministry of Health. To build on the successes of RBHS and to support the MOH in the implementation of the 2011-2021 National Health Policy and Plan, USAID signed a Cooperative Agreement with the International Rescue Committee and its partners in February 2015 to implement the Partnership for Advancing Community-based Services (PACS) activity. The overall purpose of PACS is to advance USAID Liberia’s Country Development and Cooperation strategy (CDCS) Development Objective 3 (DO3), of “Improved Health Status of Liberians." The entire CDCS Results Framework is available in Annex 1. PACS is designed to contribute to DO3 through the following objectives: • Broadened capacity of MOH, County Health Teams (CHTs), Non-Governmental Organizations and CSOs to implement and manage community services, • Increased the availability of sustainable and quality community-based health services, • Increased demand for and provision of community health and social (child) welfare services; and • Improved access to safe water supply and sanitation at the community level. During the second year of implementation, PACS received additional funding from USAID Global Health Bureau under the Ebola Emergency Fund (EEF) which enabled the activity to expand community-based service delivery activities to three additional counties: Montserrado, Margibi and Grand Bassa. The award was modified to increase the original total estimated cost by $7 million from $24.8 million to $31.8 million. With this modification, community-based health services were restored at a more rapid pace of implementation since the entire system was broken due to the Ebola Virus Disease outbreak. Expanding support to these additional counties more than quadrupled the number of people that were trained, equipped and supervised versus the performance targets established in the original M&E indicator table. Likewise, water, sanitation and hygiene (WASH) activities were expanded to the additional counties. The main priority of PACS is to enhance the capacity of the MOH, MPW, MOGCSP and CHTs to support community-based delivery of health and social welfare services and increase utilization through health communications and support of community structures. The expected results of PACS are closely aligned with those of the National Health Policy and Plan (2011- 2021) and the Investment Plan to Building a Resilient Health System (2016-2021), the Revised Community Health Services Policy and Strategic Plan (2016-2020) and implementation has been done in close collaboration with the MOH and the CHTs in the USAID priority counties of Bong, Lofa, and Nimba. Results Framework: As well as being aligned with specific policies and strategies of the MOH, MOGCSP and MPW, PACS also addresses the GOL’s Poverty Reduction Strategy Paper (PRSP) Pillar 3 and post Ebola objective: “To rehabilitate infrastructure and rebuild systems to deliver basic service in order to create the conditions and linkages needed to achieve broad-based growth and poverty reduction.” In addition to indicators to measure progress toward achievement of targets set for the four result areas and the overall development objective, the Activity Level Monitoring and Evaluation Plan (ALMEP) includes results and sub-results for tracking lower-level indicators of progress and outcome indicators of achievements. 39 The PACS Results Framework contributes to the four IRs of USAID’s Development Objective 3, “Improved Health Status of Liberians.” Result 1: Broadened capacity of central MOH, CHTs, local CSOs and NGOs to implement and manage community services. This result area contributes to obtaining and maintaining stakeholder agreement at national and county levels; establishing partnership agreements with MOH units and CHTs; and facilitating self￾assessments and performance improvement plans that will lead to ensuring that the MOH have the systems and skills necessary to effectively plan manage and monitor community health services. These activities will enable the GOL to successfully increase availability of community-based health and social welfare services. Result 2: Increased service quality and availability of community-based health and social welfare services. This result supports the MOH to review, develop, and roll out an updated, standardized, and integrated package of community health and social welfare services to be delivered by community health cadres. PACS also engages CSOs to strengthen linkages between communities and health facilities and improve accountability within the health system. To better serve the needs of orphans and vulnerable children (OVC), PACS works with the MOGCSP to conduct accreditation assessments of orphanages, develop or revise guidelines for accreditation of orphanages and for foster care, and train county level staff according to the guidelines. Result 3: Improved health-seeking behavior and practices. Ensuring the availability of health services at the community will only be effective if those communities are informed about the services and adopt more positive health behaviors. Result 3 activities will generate information and behavior change strategies at the community level to increase demand for quality health services. Result 4: Improved access to safe WASH services. This result supports increasing access for improved WASH services and products as well as building the capacity building of water and sanitation personnel to effectively manage WASH infrastructure; supports GOL in deploying trained and equipped water and sanitation staff, establish a pump fund to prepare MPW for future management of WASH infrastructure improvement programs. One major challenge has been that much of what the MOH requires now for the implementation of community health services in the post-Ebola environment was not anticipated in the design of the award, and GOL priorities for community health service implementation have shifted from the original scope of the award. The rapid restoration of community based health services and the remuneration of community health workers through a more robust community health assistant program led to the modification of the award. PACS Results Framework Aligned with USAID DO 3 DO 3: Improved Health Status of Liberians IR 3.1 Increased Utilization of Quality RMNCH Services IR 3.2 Increased Effectiveness of the Health System at IR 3.3: Improved GOL Capacity to Control Infectious Diseases IR 3.4: Increased Access to Safe Water and Sanitation 40 National and County Levels R 1: Broadened capacity of central MOH, CHTs and local CSOs to implement and manage community services (Contributes to IR 3.1; 3.2; 3.3 and 3.4) R 1: Broadened capacity of central MOH, CHTs and local CSOs to implement and manage community services (Contributes to IR 3.1; 3.2; 3.3 and 3.4) R 1: Broadened capacity of central MOH, CHTs and local CSOs to implement and manage community services (Contributes to IR 3.1; 3.2; 3.3 and 3.4) R 1: Broadened capacity of central MOH, CHTs and local CSOs to implement and manage community services (Contributes to IR 3.1; 3.2; 3.3 and 3.4) IR 1.1: Management and technical capacity of MOH and CHT strengthened (Supports DO IR 3.1.1; 3.2.2; 3.2.3; 3.2.4) IR 1.1: Management and technical capacity of MOH and CHT strengthened (Supports DO IR 3.1.1; 3.2.2; 3.2.3; 3.2.4) R 2: Increased availability of community-based health and social welfare services (Contributes to IR 3.1; 3.2 and 3.3) R 3: Improved health￾seeking behavior and practices (Contributes to IR 3.1; 3.3; and 3.4) R 2: Increased availability of community-based health and social welfare services (Contributes to IR 3.1; 3.2 and 3.3) IR 1.2: Local CSO capacity to implement, manage and oversee community services strengthened (Supports IR 3.2.2; 3.2.3 and 3.2.4 R 3: Improved health￾seeking behavior and practices (Contributes to IR 3.1; 3.3; and 3.4) IR 3.2 High quality health communication designed and implemented at national and county level (Supports DO 3.1.3; 3.3.1; 3.4.3 IR 2.1: Sub-awards to local CSOs designed and managed (Supports DO IR 3.1.1) R 2: Increased availability of community-based health and social welfare services (Contributes to IR 3.1; 3.2 and 3.3) IR 3.1 National Communication Strategy operationalized (Supports DO IR 3.1.3; 3.3.3) R 4: Improved access to safe WASH services (Contributes to IR 3.3 and 3.4) IR 2.2: Comprehensive TA for community health & social welfare services provided to MOH and CHTs (Supports DO IR 3.1.2; 3.2.2; 3.2.3; 3.2.4) IR 2.2: Comprehensive TA for community health & social welfare services provided to MOH and CHTs (Supports DO IR 3.1.2; 3.2.2; 3.2.3; 3.2.4) IR 3.2 High quality health communication designed and implemented at national and county level (Supports DO 3.1.3; 3.3.1; 3.4.3 IR 4.1: WASH infrastructure management improved to create an enabling environment (Supports DO IR 3.3.1; 3.4.1; 3.4.2) IR 2.3: Quality Assurance (QA) strategies for community-based services designed and implemented. (Supports DO IR 3.1.2) IR 3.3 Local CSOs mobilized to engage community and create enabling environment (Supports DO 3.1.1; 3.1.3; 3.3.3) IR 4.2: WASH infrastructure and products expanded (Supports 3.3.1; 3.4.1; 3.4.3) R 3: Improved health￾seeking behavior and practices (Contributes to IR 3.1; 3.3; and 3.4) R 4: Improved access to safe WASH services (Contributes to IR 3.3 and 3.4) IR 3.1 National Communication Strategy operationalized (Supports DO IR 3.1.3; 3.3.3) IR 4.1: WASH infrastructure management improved to create an enabling environment (Supports DO IR 3.3.1; 3.4.1; 3.4.2) 41 IR 3.2 High quality health communication designed and implemented at national and county level (Supports DO 3.1.3; 3.3.1; 3.4.3 IR 4.2: WASH infrastructure and products expanded (Supports 3.3.1; 3.4.1; 3.4.3) 2. Purpose of PACS Evaluation The purpose of this evaluation is to provide an independent and in-depth examination of the overall progress of the PACS activity in Liberia. The evaluation will identify achievements, performance issues, and constraints related to activity implementation and effectiveness. On the other hand, the evaluation will focus on the implementation of the robust community Health Service program in relation to the needs and expectations of the Ministry of Health. To the extent possible, there will be an emphasis on quantitative results achieved as the result of the activity. Evaluation findings and recommendations will be shared and discussed with USAID/Liberia, implementing partner, and relevant GOL units. 3. Evaluation Questions 1. To what extent has progress been made in achieving the objectives under PACS? o Which components have been most successful in meeting the overall objectives of PACS and at what level of engagement? o To what extent has PACS changed from its original scope? How has the activity adapted to the MOH’s changing needs? Are there recommendations for redirecting PACS’ focus to align more with MOH priorities? o Should there be budget constraints, what interventions should be kept and which should be discontinued? Provide a prioritized list using cost benefit analysis. 2. What are the new or emerging needs for community health service delivery not within the current PACS scope, both at the central and community level? o How successful have the CHAs been? Given the costs associated with the CHAs, do we want to continue funding them? Is this model sustainable? Please provide a cost benefit analysis of this component if not already included. o How have policy changes (CHV vs CHA) affected community health program implementation (moving from volunteerism to remuneration) under PACS? 3. What are the best results from the implementation of the WASH component and how do they impact the health outcomes of the community in terms of improving household sanitation and hygiene behaviors? o Which components of the WASH Intermediate Result have been most successful in contributing to results? Provide a prioritized list using cost benefit analysis. o To what extent has the capacity of WASH personnel been built to effectively manage water and sanitation infrastructures? o To what degree are PACS supported WASH activities sustainable (technically, socially, financially and environmentally)? o To what extent do the quantity and quality of the water supplied by PACS supported 42 facilities (boreholes, dug wells etc.) meet standards? 4. Evaluation Methods The evaluation is expected to apply both quantitative and qualitative methods for data collection and analysis. The evaluation team will conduct a desk review of available literature including project documents, quarterly and annual reports. The evaluation team will also look at the baseline assessment report of the activity and other (RBHS, FARA) assessments that were used to determine interventions. Site visits in the field will provide qualitative data for analysis through methods such as in-depth and key informant interviews, and focus groups discussions. The qualitative data collection will be supplemented through quantification of qualitative information, especially through participatory ranking exercises that will provide numeric insight on respondent priorities. Sampling should include two of PACS original focus counties including Bong and Nimba and one of the three EEF Counties and one non-USAID supported county. The selection provides a balance of diverse challenges faced by PACS in supporting CHTs. Bong was excluded due to its frequency in other USAID research. Comparing original PACS counties with an EEF County allows insight into originally designed interventions coupled with the rapid restoration of health services following the EVD outbreak. Rural Montserrado is a preferred EEF sample to better understand the emerging CHT needs and the role and lessons from JICA support. Lastly, Grand Gedeh was chosen as the non-USAID County to glean best practices from similar UNICEF and Last Mile Health support of community based health service delivery through the CHT. The evaluation team is expected to meet with central and county-level MOH staff, health facility staff, community leaders, Community Cadres (Community Health Volunteers/Assistants, Natural Leaders), WASH Entrepreneurs and CSOs, PACS staff and other health partners in order to acquire the data needed to respond to the evaluation questions. Relevant partners include MOH/FARA, MPW, Last Mile Health, Plan International, Partners in Health, Oxfam as the head of the WASH Consortium, and UNICEF for their work in community based service delivery, specifically to explore what implementation situations they are experiencing. It is expected that the evaluation team will use data provided by the implementing partner in regular quarterly and annual reports, performance reporting, and assessments for most of the quantitative data required. The team will also draw on data from the County Health Teams. The USAID/Liberia Health team will provide additional documents when necessary for the desk review, as well as contact information for prospective interviewees. The evaluation team will be responsible for identifying and reviewing additional materials relevant to the evaluation, as well as additional contacts. Illustrative data sources include but are not limited to: 1. PACS Award and modifications 2. PACS Annual Work Plans 3. PACS Activity Monitoring and Evaluation Plan 4. PACS Baseline assessment reports (Pop based and the Malaria KAP) 5. Performance Indicator Database System data 6. Quarterly and annual reports 7. Organizational capacity assessments reports of CSOs 8. Data quality assessment report 9. Revised National Community Health Services Policy and Strategic Plan 10. Other related national data and reports 43 5. Deliverables and Timeline Evaluation deliverables include: a. Evaluation Team Planning Meetings b. In brief with USAID/Liberia, GOL and Implementing Partner c. Inception Report with work plan and data collection instruments d. Debrief with USAID/Liberia, GOL and Implementing Partners • Present the major findings from the evaluation to USAID/Liberia and partners through a PowerPoint (or similar) presentation. The debriefing will cover initial findings, conclusions and preliminary recommendations. The team, in consultation with USAID, should consider doing two to three presentations: one to USAID, one to partners, and one to the MOH. This will be determined by the presence of sensitive information, if any, in the report. e. Draft Evaluation Report - A draft report should be submitted to Liberia Strategic Analysis (LSA) for review, and LSA must submit the draft report to USAID/Liberia within three weeks after the in-country work is conducted. The written report should clearly describe findings, conclusions, and recommendations and conform to USAID requirements outlined below. USAID will provide comments on the draft report within five working days of submission. LSA will also submit the draft report to the Implementing Partner and GOL for comments and an opportunity to disclose a statement of difference, if applicable. f. Final Report - The Team will submit a final report that incorporates the GOL’s and Mission’s comments and suggestions no later than ten days after final, written comments on the team’s draft report have been submitted by all parties. g. Findings Workshop – LSA will facilitate a Findings Workshop one to three months after submission of the final evaluation report with USAID, PACS, and relevant stakeholders. This will provide an opportunity to engage USAID and stakeholders in identifying opportunities for findings application. This engagement will lead to a more tailored plan on how to collaboratively apply findings and lessons learned directly to relevant decision making, including policy development, and larger sectoral assessment needs. The workshop will be a one-day session facilitated by LSA’s Collaborating, Learning, and Adapting team with the aim of ensuring that the findings of the performance evaluation are being used. 6. Team Composition The evaluation team will be composed of six individuals: team leader, three technical experts, a health economist, and an evaluation specialist. USAID/Liberia recommends the following staffing structure for the evaluation: i. An international senior-level evaluation specialist Team Leader with experience and expertise in community health systems development and/or institution building with extensive USAID program implementation and evaluation experience. The candidate must have a background in facilitating and measuring progress of self-assessments, performance improvement plans, and capacity strengthening plans. S/he must have a proven track record supervising teams in the field and producing high quality and concise reports, as well as extensive experience working in Africa and similar fragile settings. At least ten (10) years of experience in evaluation management, and qualitative data collection and analysis; experience in conducting evaluations and designing performance evaluations, preferably of USAID projects. Ability to produce high quality evaluation reports in English is essential. Strong interpersonal skills are required. 44 ii. An international mid- to senior- level community health specialist with experience in designing and/or evaluating community-based health and social welfare services through technical assistance and capacity building of Government institutions and CSOs. Should have a background in behavior change strategies and working in partnership with CSOs to improve community-based health service delivery. It is an added benefit if the candidate has experience in conducting cost-effectiveness studies. At least eight (8) years of community health experience, and some experience managing or implementing research programs is preferable. Strong interpersonal skills and American English language speaking and writing skills also essential. iii. An international mid- to senior- level water and sanitation expert with experience in designing and/or evaluating community-led WASH activities in developing countries, preferably through USAID. Should have a background in capacity building of water and sanitation personnel and implementing infrastructure development and maintenance in close coordination with government institutions. Ideally, the candidate will have experience in conducting cost￾effectiveness studies. At least eight (8) years of water and sanitation experience, and some experience managing or implementing research programs is preferable. Strong interpersonal skills and American English language speaking and writing skills also essential. iv. An international mid- to senior- level health economist with strong financial or costing expertise related to community health. S/he should have at least eight (8) years of relevant work experience in evaluating health interventions in developing countries, preferably through USAID. S/he must have demonstrated analytical skills in estimating costs and modeling of a variety of health care-related factors. Analysis will include budgets and costing by outputs. The candidate must have exceptional organizational, analytical, writing and presentation skills. Knowledge of USAID rules, regulations, and procedures in this sector is highly desirable. S/he will contribute to the overall drafting of the evaluation framework and participate in the desk study, interviews, and other data collection; and analyze the data with input from team members and USAID/Liberia to draft the evaluation report. v. A national senior or mid-level social scientist with at least five (5) years’ experience and knowledge about the health context in Liberia, particularly on county and community level health system administration. Must have strong gender and social analytical skills, specifically in designing and evaluating health programs. The incumbent must be able to conduct interviews and focus group discussions and analyze the resulting data. Ability to conduct interviews and discussions in at least one local Liberian language. Strong American English language speaking and writing skills also essential. vi. A national evaluation specialist with at least five (5) years relevant experience in qualitative and quantitative data collection methods. The local consultant must have experience evaluating health programs, specifically analyzing DHIS-2 data, and have demonstrated logistics and planning skills. He/she should have broad knowledge of Liberian health issues and the ability to assist in key informant interviews, data collection, qualitative instrument preparation, and analysis of collected data. Strong American English language speaking and writing skills also essential. The Team Leader is required to have demonstrated expertise in evaluation methodology. Collectively the team members must have experience in conducting both quantitative and qualitative data collection and analysis. Prior to their arrival in Liberia, all team members are required to familiarize themselves with USAID’s Evaluation Policy, with USAID’s publication outlining a good evaluation report, and with USAID’s checklist for assessing an evaluation report. Additionally, all team members should possess a strong familiarity with the political, economic, policy and educational context in Liberia. 45 USAID leaves to the offeror’s discretion other necessary team members/staff for the evaluation (e.g., Logistics, scheduling, translation, data analysis). Aside from the above mentioned key personnel, the offeror must decide how the evaluation team should be structured in order to successfully address the evaluation questions. All attempts should be made for the team to be gender balanced and to include local (Liberian) experts. A statement of potential bias or conflict of interest (or lack thereof) is required from each team member. 7. Logistics and Level of Effort The contractor will be responsible for all international and in-country administrative and logistical support, including identifying and fielding appropriate consultants (international and local). The evaluation team should be able to make all logistical arrangements including vehicle rental for travel within and outside Monrovia and should not expect any logistical support from the Mission. The team should also make their own arrangements for venues for team meetings, and equipment support for producing the report. Evaluation team members are authorized and expected to work a six-day week. Travel over weekends may be necessary. Work should commence as soon as practicable, but no later than June 2017. For planning purposes, contractors should be aware of Liberian and US holidays during the evaluation time frame. The evaluation should follow the illustrative schedule and level of effort given below. Task/Deliverable Estimated time (Days) Review background documents & preparatory work (offshore): Draft work plan submitted to USAID/Liberia 5 int’l / 3 nat’l Travel to Liberia-expatriate team members 2 Team Planning Meetings in Monrovia with implementing partners, MOH M&E Working Group, and USAID 2 In-brief with USAID/Liberia and prepare for field work 5 Field work: Data collection and On-Going Data Analysis 25 (including field travel) Analysis and report drafting 10 int’l / 7 nat’l Debrief presentation with GOL, USAID/Liberia and IPs by Evaluation Team 2 LSA performs quality assurance check and edits draft report 4 LSA submits draft report to USAID/Liberia and GOL 1 USAID and GOL provide comments on draft report 5 Evaluation Team Leader revises draft report to incorporate comments and submit final report 5 TL / 1 int’l team members LSA performs quality assurance check and submits final report 5 LSA facilitates learning event 1 TL / 2 nat’l Total time required 72 Working days Team Leader (including international travel) 57 Working days international team member (including international travel) 52 Working days local team members 46 46 ANNEX II: EVALUATION METHODS AND LIMITATIONS Methods for EQ 2: Economic evaluation Cost data was primarily sourced from PACS financial data. This information was used to develop unit costs for each intervention or intervention package. Cost estimates were limited to direct costs (i.e. no overhead, administrative, or management costs) and include PACS costs as well as beneficiary time expended for travel to the health facility. Community contributions of time and materials were quantified; those values were used to estimate the WTP for the direct benefits corresponding to CHA services and CLTS. Benefits also include time saved due to proximal services. For example, beneficiaries spend less time collecting water, going to the bush, or seeking health facility services when they have a hand-dug well, latrine, or CHA who can provide treatment for simple diarrhea, malaria, or pneumonia. Benefits were identified and valuated using a mixed-method approach including WTP, Spectrum Lives Saved Tool (LiST), and modeling. A summary of the benefits valuated and methods used by intervention package is shown in Table A. Table A. PACS Economic Evaluation: Benefits valued, methods and intervention packages Benefits Valuated Methods Intervention Packages Access to information & referral WTP / shadow pricing using in-kind community contribution CHA services Direct benefits WTP / shadow pricing using PACS cost-share estimates CLTS Morbidity reduced (health care savings & productive days gained) Spectrum/ LiST CHA services, hand-dug wells, CLTS Mortality averted Spectrum/LiST CHA services, hand-dug wells, CLTS Time-saving from proximal services* Direct modeling CHA services, hand-dug wells, CLTS Capacity strengthened WTP CSO capacity; CHA services; BCC Capacity strengthened Statistical modeling BCC *Proximal services refer to having closer access to services such as a hand-dug well, a latrine, or treatment for simple illness. Respondents in the cost-benefit analysis are determined based on the primary beneficiary group of the intervention component; the groups involved in designing or delivering the intervention component are excluded from the valuation of that component due to the inherent bias in requiring a valuation of their own work. Accordingly, CHSSs and CHAs were identified as respondents for some of the work under Result 1, which enhanced their capacity to provide information, referrals, and treatment services. Other aspects of Result 1, including national policy and strategies, could not be valued because respondents from the primary beneficiary groups were either unavailable (MOH directors) or unaware of those outputs (CHTs). Likewise, the CHT, DHT, CSO, CHSS, and CHAs could not value their own services, and respondents were drawn from the primary beneficiary group of lay community members. CBA session guides were developed for each beneficiary group to facilitate discussions along three themes: (1) review the PACS project activities related to the intervention package; (2) identify and discuss benefits and consequences related to the intervention package; and, (3) valuate the related benefits and consequences. Each session was validated by asking participants what they were thinking about when assigning a monetary value to the intervention or intervention package. The identification of benefits was 47 also used to inform and validate model assumptions. Methodological constraints related to WTP exercises with CSOs and CHAs necessitated that capacity building and training valuations be directly linked to the intervention. Therefore, cost-benefit metrics associated with those interventions are not calculated over a time horizon. The CBA small group session guides are included in Annex III: Data Collection Instruments. To quantify the primary health benefits directly attributed to each intervention, changes in morbidity and mortality were estimated using the Spectrum v5.51 Lives Saved Tool (LiST) - software that estimates survival of mothers and children. Time-saving from proximal sources was estimated using direct modeling whereas the average walk-time in rural areas to the health facility (75 minutes) or the bush (15 minutes per day) was multiplied by the corresponding hourly rate ($0.39) based on the Liberia GDP per capita Purchasing Power Parity (PPP). It is important to note that CHSS fuel and stipend costs were included for the three CHA scenarios and BCC. This was done to be consistent with the CHW costing algorithm which includes costs related to direct supervision.24 In the case of BCC, the team met with various key informants to identify the primary interventions: ETL training, didactic materials, and technical assistance (to gCHVs/CHAs), community engagement at the community level. One issue in attempting to isolate the benefits/effect of BCC training is the integration of the training into the technical modules of the CHA training. To address this issue, the team met with gCHVs who had not benefited from CHA training (as they were not eligible to become CHAs due to proximity to the health facility). These gCHVs testified to the general benefits of the four primary BCC interventions. They reported that this support improved their overall effectiveness at facilitating improved health practices and prompt care-seeking at the health facility. Based on these discussions, analysis counted “access to information and referral” and “mortality averted” as benefits associated with gCHV activities enhanced by the four BCC interventions. The value of “access to information and referral” was calculated by monetizing the in-kind contribution of farm labor provided by communities to CHWs which was revealed in CBA sessions with community members. “Mortality averted” was quantified using statistical modeling. Stata v14.2 was used to identify statistically significant changes on 15 key DHIS-2 health service indicators in PACS districts. Health facility service statistics from September 2014 to February 2015 (before BCC training) were compared with September 2016 to February 2017 (after BCC training, but prior to CHA training). The statistically significant ‘premium’ increases were then modeled in Spectrum LiST to estimate the effect on morbidity and mortality. All modeling was specified using data for Bong, Lofa and Nimba. When county specific information was not available, national data was used. This information was extracted from the PACS Baseline Evaluation report, the Liberia Demographic and Health Survey (DHS), and the DHIS-2. Liberia specific economic data was taken from the World Bank. For example, productivity gains and loss (both direct and indirect) and lives lost were valued using the current GDP per capita (PPP). A detailed list of parameters, sources, and assumptions used for these calculations is included as Annex IX: CBA Parameters and data sources. Methods for Evaluation Question 3 Understanding the effects of the MOH policy change and the new CHA program on PACS scope and performance relied on qualitative data. KIIs were valuable for gathering information for several groups of stakeholders involved in PACS. The team conducted KIIs with stakeholders such as CHAs, CHFPs, RHFPs, and respective PACS staff within each county. FGDs were facilitated with members of MOH, MPW, CHTs, 24 Columbia University. Earth Institute. Community Health Workers Technical Task Force; One Million Community Health Workers, Technical Task Force Report. Costing Algorithm for Community Health Workers. 2013 New York City, p. 58. 48 water and sanitation and committees, CHAs and community members. The FGDs included separate groups of women and men to gauge views on the PACS that may differ by gender. Tables B and C below describe the sampling of respondents for qualitative data collection. Table B: Stakeholder Groups for KIIs Stakeholder Groups for Key Informant Interviews Total # of KIIs Targeted Location COMMUNITY LEVEL Community Health Assistants (CHAs) 4 + (4 CBA) 2 PACS (Lofa or Nimba) ; 2 Non-PACS (Maryland or Cape Mount) Community Health Services Supervisors 2 1 PACS; 1 Non-PACS Community Health Volunteers (CHVs) 4 2 PACS; 2 Non-PACS Trained Traditional Midwives (TTMs) 2 1 PACS ; 1 Non-PACS Natural Leaders 2 1 PACS; 1 Non-PACS COUNTY HEALTH TEAM STAFF County Health Officer 2 1 PACS ; 1 Non-PACS County Surveillance Officer 2 1 PACS ; 1 Non-PACS Community Health Focal Person 2 1 PACS ; 1 Non-PACS Reproductive Health Focal Person 2 1 PACS; 1 Non-PACS PACS Capacity Building Advisor (County Level) 2 1 PACS ; 1 Non-PACS Environmental Health Technician 1 1 PACS County WASH Coordinator (MPW) 1 1 PACS COMMUNITY BASED ORGANIZATIONS & IMPLEMENTING PARTNERS PACS, Chief of Party (Barbara Jones) 1 MONROVIA PACS, Deputy Chief of Party (Dianah B. Majekodunmi) 1 MONROVIA PACS, Technical Advisor (Justin DeNormandie) 1 MONROVIA PACS M&E (Isaac Mwase) 1 MONROVIA PACS WASH Advisor (Alex Kiembe) 1 MONROVIA YMCA (Program Manager) 1 MONROVIA Planned Parenthood Association of Liberia 1 MONROVIA CBO Sub-grantee(s) 2 Lofa & Nimba (PACS) MINISTRY OF HEALTH & MINISTRY OF PUBLIC WORKS Director of Community Health Services Division (Bioma Tamba or his deputy) 1 MONROVIA Director of National health Promotion Division (Rev. John Smoo 10-12) 1 MONROVIA 49 Director of Department of Environmental & Occupational Health(D. Omarley Yeabah) 1 MONROVIA Coordinator of National Water Sanitation and Hygiene Promotion Secretariat (H. Watara Sackor or his deputy) 1 MONROVIA MINISTRY OF GENDER, CHILDREN, & SOCIAL PROTECTION PACS Capacity Building Advisor (Tupee Freeman) 0900 @ IRC offices 1 MONROVIA Deputy Minister of Gender (Lydia Sherman or Victoria Zarway- director) 1 MONROVIA TOTAL number of KIIs 41 (+4 CBA) Table C: Stakeholder Groups for FGDs and CBA Sessions Stakeholder Groups for Focus Group Discussions Total # of FGDs Targeted Total # CBA sessions Location Ministry of Health (Technical Working Group) 1 1 MONROVIA Ministry of Public Works 1 MONROVIA CSO Representatives 1 WASH Committees (Community Level) 2 1 PACS ; 1 Non￾PACS Community Health Committees (CHCs Community Level ) 2 1 PACS ; 1 Non￾PACS CHDC 2 1 PACS ; 1 Non￾PACS CHT Staff: Group 1: PACS led intervention 1 Lofa or Nimba Group 2: Non-USAID Intervention 1 Grand Gedeh or Maryland Group 3: CHT (zero community health intervention) 1 Cape Mount Community Members (including youth): 4 Men 2 1 PACS ; 1 Non￾PACS Women 2 6 (3 Lofa; 3 Nimba) 1 PACS ; 1 Non￾PACS TOTAL 15 5 Methods for Evaluation Question 4 To measure change through WASH support, the evaluation compared Akvo Flow data for PACS counties against non-PACS counties. The data is percent change in WASH indicators captured by PACS (reported in PIDS) and the Akvo Flow database at MPW, from 2015-17. Indicators range across capacity building, behavior change, and access to services and infrastructure (PACS RF IR 3.4). This data can be compared at a county level because PACS provides full coverage across these counties for WASH. LIMITATIONS 50 The evaluation faced numerous methodological and logistical constraints as well as potential biases. Table D summarizes the limitations which are described in further detail below. Table D. Limitations, Constraints, Biases, and Mitigation Strategies Constraint/Bias Evaluation Question Mitigation Strategy Inaccessibility of remote sites Questions 1-4 • Real time monitoring of weather and road conditions • Prudent selection of districts and community sites, based on feasibility • Remaining risk of not capturing information from remote areas Water quality analysis data dependent on IP Question 4 • Triangulation of different data sources and data collection methods • Assessing water QA process, rather than direct analysis of water quality Male dominance in focus group discussions Questions 1-4 • FGDs were held as single gender events Limitations and potential bias of routine data collection sources Questions 1,3-4 • Winsorizing and truncation to minimize the influence of outliers and irregular data. • Control variables to account for known confounding factors • Caution should be used with the interpretation of results Limited causal attribution in a non-randomized evaluation Question 1 • Application of statistical methods that are recognized approaches to assess outcomes of natural experiments (Difference-in￾Differences and segmented regression) • Construction of a comparison group using Propensity Score Matching Convenience sample for CBA may limit generalizability of findings Question 2 • Comparison of responses within and between group to check for consistency • Point estimates based on medians (not means) Abstract nature of willingness-to-pay/valuation of benefits Question 2 • Responses validated at the end of each session (to ensure estimates reflect what is being measured) • Use of secondary data sources (i.e. PACS baseline survey and DHS data) to inform model parameters. QUANTITATIVE ANALYSIS LIMITATIONS It is important to note that there are a number of limitations relating the use of routine health management information data for the evaluation of a health intervention. 25 In the absence of a systematic bias, these limitations are likely to reduce the power to detect a significant difference or reduce the magnitude of a measured difference. No such bias was seen or postulated for the DHIS-2 data analyzed. For the present 25 Kane, 2000. 51 analysis that concluded that there is a ‘PACS effect’, the magnitude of the effect is likely to be underestimated, rather than overestimated. First, data may be inaccurate or incomplete from recording and data entry errors. Furthermore, these data include administrative statistics which rely on incidence and population estimates, not a population based survey, so the data is not necessarily representative of the population. In addition, data available for this analysis, both from DHIS-2 and from PACS internal database, do not distinguish patients by gender. In addition, administrative data can be irregular for a variety of reasons. For example, administrative coverage statistics are calculated using population estimates for the denominator; and no adjustments are made when service provision (the numerator) exceeds the population estimate. This can produce coverage rates in excess of 100%. In addition, immunization coverage can be particularly problematic as it is commonly calculated using the number of vaccine doses per vial opened as the denominator, regardless of the number of people actually vaccinated. It is important to note that these issues are not Liberia specific. To partially address these problems, outliers were corrected for by Winsorizing (at the 0.10 level) for indictors reported as a count; and, truncated at 100% for coverage indicators. There are also limitations with applying econometric methods to DHIS-2 data including multiple potential biases, such as selection bias, mean regression, reverse causality, and omitted variable bias. Particularly noteworthy, the purposeful selection of PACS intervention areas inherently introduces a selection bias. For example, as geographic areas were selected based on accessibility, those areas are likely to be fundamentally different in terms of health services provision, nutrition levels, education and poverty, compared to areas that were not included. If such factors are highly correlated with the presence or absence of PACS, it may confound the analysis, whereas trend changes are not comparable between PACS and non-PACS districts as a change could be attributed to another factor or factors. We attempted to limit this issue by employing Propensity Score Matching (PSM) on poverty and education. Finally, although it is unfeasible to control for all possible confounders given the time constraints of this analysis and reliance on existing and available data sources, we did employ control variables that with a direct impact on health facility service utilization (population, Ebola, and rainy season). We also introduced a time variable to account for a general trend increase in health facility utilization, independent of other factors. Finally, it is important to note that the gold standard for causal inference is a randomized controlled experiment. Given the data limitations and context, we applied well-documented quantitative methods that are recognized approaches to assess outcomes of natural experiments. COST-BENEFIT ANALYSIS LIMITATIONS There are several limitations related to Cost-benefit analysis (CBA). First, cost estimates were limited to direct costs; indirect costs including activity and government overhead, administration, and management were not captured. As such, the cost estimates presented in this report do not fully represent the cost of service delivery and are not appropriate to use as estimates for budgeting purposes. In addition, cost analyses of CHW programs can pose methodological challenges because they do not capture the full benefits of enhanced equity, increased self-reliance by communities, and contributions to other social benefits and community norms. 26 We used a mixed-method approach which applied modeling informed by the literature and benefits identified by beneficiaries. In addition, we undertook a modified WTP methodology with CSOs and CHAs. The method proved to be conceptually abstract. We validated the results by asking participants about what they were thinking when assigning a value to the intervention package. We also note that beneficiary valuation is highly sensitive to personal income. To partially 26 Lehmann and Sanders, 2007. 52 correct for this issue, we applied a PPP conversion to all beneficiary valuations. Also related, we used GDP per capita (PPP) to value time and lives lost. This approach highly undervalues time and life, particularly in low-income countries. Therefore, the monetization of these benefits is considered to be extremely conservative. As noted above, there may be other costs and benefits related to the interventions that were not captured in this analysis. Due to time constraints, the costing of interventions did not include possible government contributions. Furthermore, the valuation of benefits do not incorporate longer term potential benefits which may include important policy or strategic objectives including quality and equity. PV estimates apply a discount rate to value benefits occurring in the more immediate term higher. The high discount rates (12% and 20%) quickly erode the value of benefits over time. Regardless, the short five-year time horizon limits the accrual of benefits: applying a longer time horizon would yield a higher return on investments. The team spoke with multiple key informants to explore the possibility of separately assessing CSO sub￾grant activities as part of the CBA. However, the team was unable to identify and quantify benefits that have a plausible direct and independent attribution to these activities. Due to these methodological constraints, the CBA did not assess the benefits related to CSO sub-grant activities. It is also important to note that CBA is based on welfare economics theory which is focused on determining economic efficiency or achieving the ‘greatest good for the greatest number.’ There is no regard to equity. Finally, due to time constraints feedback was gather in group sessions and participant invitation was based on geographic proximity. Sixty-three PACS-supported CHAs participated in four CBA sessions. Also, 14 PACS-supported CSO representatives participated in a separate CBA session. Therefore, the results for training and capacity building, may have limited generalizability. The recently published Strengthening Primary Health Care through Community Health Workers: Investment Case and Financing Recommendations report cites a return on investment for Community Health Workers at 10:1. By contrast, the PACS midterm evaluation CBA found a return on investment of 2:1 for the most optimistic scenario (#3). There are several important differences relating to the assumptions for each exercise. First, the costs used for the Investment Case report are sourced from the One Million CHW report. The later report does include Liberia specific CHW costs. However, they are substantially lower than the actual direct expenditure from PACS. It is also important to note that the report was published in 2015 – prior to the NCHA program. So, arguably the 10:1 return is somewhat representative of the Liberia CHW program before the NCHA. For example, that report estimates the annual training cost per CHW to be $0.13. However, PACS direct expenditure for CHA training per CHA amounted to $1583. Also, the CHW report estimates the annual salary of CHWs in Liberia to be $1.97. With the introduction of stipend payments, the annual “salary” is now $840. Second, the Investment Case report monetized three categories of benefits: increased productivity; avoided costs of global health crises; and increased formal employment and economic activity. By contrast, the PACS midterm evaluation CBA includes the three following benefit categories: access to information and referral, mortality averted, resources saved/productivity gained. Benefits quantification was limited to what one may consider immediate or proximate. The analysis did not attempt to quantify possible benefits of CHWs relating to major health crisis or increased formal employment and economic activity as those assumptions are less direct. Estimates may not have included all benefits. 53 ANNEX III: DATA COLLECTION INSTRUMENTS CBA small group (12-15 participants) Session guide No. 1 Local CSO stakeholders (valuation of IR1 capacity building) I. Introduction Give an explanation Good afternoon. My name is _______and I’m an evaluation specialist working to evaluate the USAID PACS project. The evaluation will be used by stakeholders to help make the project more effective. Thank you for coming. Present the purpose We are here today to talk about the benefits and consequences relating to capacity building work supported by PACS. The purpose is to get your perceptions relating to the capacity building support provided as part of the PACS project. I am not here to share information, or to give you my opinions. Your perceptions are what matter. There are no right or wrong or desirable or undesirable answers. You can disagree with each other, and you can change your mind. I would like you to feel comfortable saying what you really think and how you really feel. Discuss procedure ______ (colleague) will be taking notes during the discussion so that I do not miss anything you have to say. I would like to highlight that everything we discuss is confidential. No one will know who said what. We will be going through some exercises with group discussion. Participant introduction I understand that you all have been involved in some way with the PACS capacity building activity. To get started could we by everyone sharing your name, and what you do in your organization, and how long you have been working on public health. Rapport building I understand that each of you have been involved in different ways such as planning or participating in the PACS capacity building activities. I would like to go around the room again and ask if each of you could describe how you have been involved. Probes: Get participants to elaborate on their involvement. Discussion 1. I would like to define the capacity building support provided by PACS. Can you all describe what it has entailed (write each activity as a header on newsprint) 54 Probes: What else was involved? 2. Now, I would like to identify and discuss the benefits and any other consequences related to each activity. To get started, who can tell me about the benefits relating to activity A (categorize summary comments under each defined activity. Probes: Can you tell me more about that? What else? 3. Now, I would like to try to quantify each benefit and consequence. For example, you mentioned that one benefit of the activity is that you are better organized which saves you time. How much time would it save you? Daily, weekly? Probes: Ask participants how they would quantify the benefit. All benefits and consequences should be quantified if possible. 4. Next, we are going to do a small role play. Are you ready? I have very good news for you. You have received an official communique from USAID. USAID would like to estimate the value of the support you have received. Please think about everything we have discussed during this session relating to the benefits and trade-offs. Pass out paper and take questions as need. Collect responses and write summary results. Note the range and average of assigned values. 5. What do you think about the results? I would like to understand what you were thinking about when assigning a value. Probes: Encourage various opinions. II. Closure Though there were many different opinions about capacity building support, it appears unanimous that _______. Does anyone see it differently? It seems most of you agree ______, but some think that _____. Does anyone want to add or clarify an opinion on this? Is there any other information regarding your experience with PACS capacity building support that you think would be useful for me to know? Thank you very much for coming this afternoon. Your time is very much appreciated and your comments have been very helpful. 55 CBA small group (12-15 participants) Session guide No. 2 Community Health Agents (valuation of IR1 capacity building) Name of Moderator____________________ Date_______________________________ Attendees___________________________ Evaluation Questions: III. Introduction Give an explanation Good afternoon. My name is _______ and I’m an evaluation specialist working to evaluate the USAID PACS project. The evaluation will be used by stakeholders to help make the project more effective. Thank you for coming. Present the purpose We are here today to talk about the benefits and consequences relating to CHA training supported by PACS. The purpose is to get your perceptions relating to this support provided as part of the PACS project. I would like you to feel comfortable saying what you really think and how you really feel. Discuss procedure ______ (colleague) will be taking notes during the discussion so that I do not miss anything you have to say. I would like to highlight that everything we discuss is confidential. No one will know who said what. We will be going through some exercises with group discussion. Participant introduction To get started could everyone share your name, how long you have been working on community health in your community. Rapport building Great! Can you tell me what do you do to promote health in your community? Probes: Use reflective listening. 56 Discussion 6. I would like to talk in more detail about the important training that you recently completed. Can you tell me about that? Probes: What else was involved? Can you tell me more about that? 7. Now, I would like to identify and discuss the benefits and any other consequences related to the training. To get started, who can tell me about the benefits? (categorize summary comments) Probes: Can you tell me more about that? What else? 8. Let’s explore that, how many people have been able to apply any new skills that you gained in the training? Probes: Can you give an example? What else? 9. Now, I would like to try to quantify each benefit and consequence. For example, you mentioned that one benefit of the training is ____ [that you are now better prepared to convince pregnant women to go to the health facility for early ANC]. Probes: Ask participants how they would quantify the benefit. All benefits and consequences should be quantified if possible. 10. Next, we are going to do a small exercise. Imagine that you receive a letter from the Minister of Health which explains that you have been appointed to be the OIC. Because you have been promoted you will need to plan training for the new CHA who will replace you to continue the community work. The Minister wants you to propose an amount that reflects the value of the training. Because you have been through the training, and you have first-hand knowledge of the benefits and consequences of the training, you are uniquely qualified to prepare this estimate for the Minister. 11. Pass out paper and take questions as need. Collect responses and write summary results. Note the range and average of assigned values. 12. What do you think about the results? I would like to understand what you were thinking about when assigning a value. Probes: Encourage various opinions. IV. Closure Though there were many different opinions about ETL training, BCC tools, on-site coaching, and community engagement activities supported by PACS. It appears unanimous that _______. Does anyone see it differently? It seems most of you agree ______, but some think that _____. Does anyone want to add or clarify an opinion on this? Is there any other information regarding your experience that you think would be useful for me to know? Thank you very much for coming this afternoon. Your time is very much appreciated and your comments have been very helpful. 57 CBA small group (12-15 participants) Session guide No. 3 Community Health Volunteers (valuation of IR3 intervention package) Name of Moderator____________________ Date_______________________________ Attendees___________________________ Evaluation Questions: V. Introduction Give an explanation Good afternoon. My name is _______ and I’m an evaluation specialist working to evaluate the USAID PACS project. The evaluation will be used by stakeholders to help make the project more effective. Thank you for coming. Present the purpose We are here today to talk about the benefits and consequences relating to Education, Through Learning (ETL) training, Behavior Change Communication (BCC) tools, on-site coaching, and community engagement activities supported by PACS. The purpose is to get your perceptions relating to this support provided as part of the PACS project. Discuss procedure ______ (colleague) will be taking notes during the discussion so that I do not miss anything you have to say. I would like to highlight that everything we discuss is confidential. No one will know who said what. We will be going through some exercises with group discussion. Participant introduction To get started could everyone share your name, how long you have been working on community health in your community. Rapport building Great! Can you tell me what do you do to promote health in your community? Probes: Use reflective listening. 58 Discussion 13. I would like to talk in more detail about Education, Through Learning (ETL) training, Behavior Change Communication (BCC) tools, on-site coaching, and community engagement activities supported by PACS. Can you tell me about that? Probes: What else was involved? Can you tell me more about that? 14. Now, I would like to identify and discuss the benefits and any other consequences related to the training. To get started, who can tell me about the benefits? (categorize summary comments) Probes: Can you tell me more about that? What else? 15. Let’s explore that, how many people have been able to apply any new skills that you gained in the training? Probes: Can you give an example? What else? 16. Now, I would like to try to quantify each benefit and consequence. For example, you mentioned that one benefit of the training is ____ [that you are now better prepared to convince pregnant women to go to the health facility for early ANC]. Probes: Ask participants how they would quantify the benefit. All benefits and consequences should be quantified if possible. 17. Next, we are going to do a small exercise. Imagine that you receive a letter from the Minister of Health which explains that you have been appointed to be the OIC. Because you have been promoted you will need to plan training for the new CHV who will replace you to continue the community work. The Minister wants you to propose an amount that reflects the value of the support you have received from PACS. Because you have been through the training, and you have first-hand knowledge of the benefits and consequences of the training, you are uniquely qualified to prepare this estimate for the Minister. 18. Pass out paper and take questions as need. Collect responses and write summary results. Note the range and average of assigned values. 19. What do you think about the results? I would like to understand what you were thinking about when assigning a value. Probes: Encourage various opinions. VI. Closure Though there were many different opinions about ETL training, BCC tools, on-site coaching, and community engagement activities supported by PACS. It appears unanimous that _______. Does anyone see it differently? It seems most of you agree ______, but some think that _____. Does anyone want to add or clarify an opinion on this? Is there any other information regarding your experience that you think would be useful for me to know? Thank you very much for coming this afternoon. Your time is very much appreciated and your comments have been very helpful. 59 ANNEX IV: SOURCES OF INFORMATION Angrist, J. Pischike, JS. 2009. Mostly Harmless Econometrics: An Empiricist’s Companion: Princeton University Press Dahn, B. Woldemariam, AT. Perry, H. Maeda, A. von Glahn, D. Panjabi, R. Merchant, N. Vosburg, K. Palazuelos, D. Lu, C. Simon, J. Pfaffmann, J. Brown, D. Hearst, A. Heydt, P. Qureshi, C. 2015. Strenghtening Primary Health Care through Community Health Workers: Investment Case and Financing Recommendations. Heckman, J., Ichimura, H., Todd, P. Matching as an Econometric Evaluation Estimator. 2008. The Review of Economic Studies, Vol. 65, No. 2: 261-294. Liberia Institute of Statistics and Geo-Information Services (LISGIS), Ministry of Health and Social Welfare [Liberia], National AIDS Control Program [Liberia], and ICF International. 2014. Liberia Demographic and Health Survey 2013. Monrovia, Liberia: Liberia Institute of Statistics and GeoInformation Services (LISGIS) and ICF International. Kane, Roslyn. Wellings, Kaye. Free, Caroline. Goodrich, Joanna. 2010. Uses of routine data sets in the evaluation of health promotion interventions: opportunities and limitations. Health Education Volume 100:1, pp. 33-41 Kennedy, Peter. A Guide to Econometrics, Edition 6E. 2008. Blackwell Publishing, Victoria, Australia. Lagarde, Mylene. How to do (or not to do)… Assessing the impact of a policy change with routine longitudinal data. 2012. Health Policy and Planning. 27: 76-83 Lechner Michael. 2011. The Estimation of Causal Effects by Difference-in-Difference Methods, Foundations and Trends in Econometrics: Vol. 4: No. 3, pp 165 Archer C. L. Bollinger, W. DeCormier Plosky, C. Pretorius, R. Sanders, E. Sonneveldt, J. Stover, B. Winfrey, J. Ross, I. Friger, N. Walker, R. McKinnon, T. Podkowiak, M. Roman and J. Schein, Spectrum Manual: Spectrum System of Policy Models v. 5.51, 2017. Glastonbury: Futures Institute Boardman, Anthony E; Greenberg, David, Wining Aidan, Weimer, David. 2001. Cost-Benefit Analysis: Concepts and Practice. Second Edition. Prentice Hall, Inc. Brent, RJ. Cost-benefit Analysis for Developing Countries. 1998. Edward Elgar Publishing Company Columbia University. Earth Institute. Community Health Workers Technical Task Force; One Million Community Health Workers, Technical Task Force Report. Costing Algorithm for Community Health Workers. 2013 New York City, p. 58. Darmstadt, Gary, Bhutta Zulfiqar, Cousens Simon, Taghreed Adam, Walker Neff, Bernis Luc de. 2005. Evidence￾based, cost-effective interventions: how many newborn babies can we save? Neonatal Survival 2, The Lancet Drummond, Michael, Sculpher Mark, Torrance George, O’Brien, Bernie, Stoddart, Greg. 2005. Methods for the Economic Evaluation of Health Care Programmes, 3rd Edition. Oxford University Press. New York. Horton S, Levin C. Cost-effectiveness of interventions for reproductive, maternal, neonatal, and child health. In: Jamison D, Nugent R, Gelband H et al. (ed). Disease Control Priorities: Improving Health & Reducing Poverty. Hutton Guy, Laurence Halter, and Bartram Jamie. Global cost-benefit analysis of water and sanitation interventions. 2007. Journal of Water and Health: WHO 5.4 International Development Group (IDG). 2016. Cost-Benefit Analysis of USAID/Liberia’s Rice and Goat Value Chain Interventions Final Report. Arlington, Virginia Jones Gareth, Steketee Richard, Black Robert, Zulfiqar Bhutta, Morris Saulant The Bellagio Child Survival Study Group. 2003. How many child deaths can we prevent this year? The Lancet, Vol 362. Layard Richard, Glaister Stephen. Cost-benefit Analysis Second Edition. 1999. Cambridge University Press: Cambridge MA. Lehmann U, Sanders D. Community health workers - what do we know about them? The state of the evidence on programmes, activities, costs and impact on health outcomes of using community health workers. 2007. Department of Human Resources for Health, Evidence and Information for Policy, World Health Organization. Geneva. Luckow, P. Kenny, A. White, E. Ballard, M. Dorr, L. Erlandson, K. Grant, B. Johnson, A. Lorenzen, B. Mukherjee, S. Ly, EJ. McDaniel, A. Nowine, N. Sathananthan, V. Sechler, GA. Kraemer, JD. Siedner, MJ. Panjabi, R. Implementation research on community health workers’ provision of maternal and child health services in rural Libera. 2017. Bull World Health Organization. 95:113-120. Management Sciences for Health, Inc. International Drug Price Indictor Guide 2014. 2015. Medford, MA: MSH and WHO. Mishan, E.J. and Quah, E. 2007. Cost Benefit Analysis (5th Ed.). Oxon: Routledge Sarley, David, Linda Allain, and Anup Akkihal. 2009. Estimating the Global In-Country Supply Chain Costs of Meeting the MDGs by 2015. Arlington, Va.: USAID | DELIVER PROJECT, Task Order 1. 60 Stevens Kaye, Rogers Patricia, Boymal Jonathan Boymal, Humble Roslyn. 2008. Evaluation of the Stronger Families and Communities Strategy: Qualitative Cost Benefit Analysis 2000-2004 Taghreed Adam, Stephen S Lim, Sumi Mehta,Zulfiqar A Bhutta, Helga Fogstad, Matthews Mathai, Jelka Zupan, Gary L Darmstadt. 2005. Cost effectiveness analysis of strategies for maternal and neonatal health in developing countries. BMJ 331:1107 Wheatly D. CBA Builder Simple and Advanced Instruction Manual v1.2. 2010. Nottingham Trent University: Nottingham, England. World Health Organization. Guidelines for conducting cost-benefit analysis of household energy and health interventions. 2006. Guy Hutton with Eva Rehfuess. Ministry of Health and Social Welfare, Liberia. 2011. National Health and Social Welfare Policy. Monrovia: MoHSW. Ministry of Health and Social Welfare, Liberia. 2015. Revised National Community Health Services Policy, 2016- 2021. Monrovia: MoH. Nimba County Health Team, Liberia. 2017. Operational Plan, Fiscal Year 2017-2018. Saniquellie: NCHT. Nimba County Health Team, Liberia. 2017. Projected Catchment Population by Facility, Fiscal Year 2017-2018. Saniquellie:NCHT. Nimba County Health Team(NCHT) 2017. Activities Workplan Fiscal Year 2017-2018. Saniquellie: NCHT. Liberia. 2014. Liberia Demographic and Health Survey, 2013. Monrovia: Geo-Information Services (LISGIS) and Ministry of Health and Social Welfare Ministry of Health, Liberia. 2016 Health Sector Performance Report, 2016. Monrovia: MoHSW. Liberia Community Mapping Report Ministry of Public Works. 2017. Nationwide Water Point Mapping Update – Vol 4, Monrovia-Liberia: MPW Investment -plan for building a resilient Health system in Liberia 2015 Ministry of Health. 2017. National Health Communication Plan. Monrovia. MOH National WASH data Revised Community Health Service Policy 2016-2021 Ministry of Health, Liberia. 2011-2017. District Health Information System(DHIS-2). Monrovia: MOH. Ministry of Public Works. 2017. Water and Sanitation database. Akvo Flow. Ministry of Public Works. 2010. Guidelines for Water and Sanitation Services in Liberia. Ministry of Health, 2017. Water Quality Analysis Results, Monrovia-Liberia: Ministry of Health 2017. Community Health Services supervisor (CHSS) Listing with email address. Monrovia: MOH Ministry of health 2016. Community Health Services Supervisor and Community Health Assistance Gap Matrix 2017: Monrovia. MOH Grand Gedeh County CHA Funds Program Partnership for Advancing Community-Based Services (PACS).2015. Malaria in Pregnancy Knowledge, Attitudes, and Practices (KAP) Survey. Monrovia. PACS Partnership for Advancing Community-Based Services (PACS) 2017. Performance Indicators. Monrovia. PACS Partnership for Advancing Community-Based Services (PACS). 2015. Annual Report FY 15 Y1. Monrovia. PACS Partnership for Advancing Community-Based Services (PACS). 2016. Data Quality Assessment Report. Monrovia. Liberia Strategy Analysis (LSA). Partnership for Advancing Community-Based Services (PACS) Performance Indicator Database System (PIDS) 2017. Monrovia: PACS Partnership for Advancing Community-Based Services (PACS) Y2. 2017. Annual Report. Monrovia: PACS Partnership for Advancing Community-Based Services (PACS) Annual Implementation Plan Year-2. 2016. Monrovia. PACS Partnership for Advancing Community-Based Services (PACS). 2017. Annual Implementation Plan Year-2, Year 3. Monrovia. PACS. Partnership for Advancing Community-Based Services (PACS). 2017. PACS Activity Level Monitoring and Evaluation Plan- May 2017. Monrovia. PACS Partnership for Advancing Community-Based Services (PACS) year 3. Quarter-1 Narrative Report Mar 2017. Monrovia: PACS Water Quality Analysis Results February 2017 Ministry of Health National 2012. Community Led Total Sanitation(CLTS) Guidelines. Monrovia. MOH. 61 ANNEX V: RESPONDENTS FOR KEY INFORMANT INTERVIEWS No Name Sex Title County PACS Head Offices 1. Barbara Jones F PACS, Chief of Party Montserrado 2. Dianah B. Majekodunmi F PACS, Deputy Chief of Party Montserrado 3. Justin DeNormandie M PACS, BCC Technical Advisor Montserrado 4. Ernree Bee-Neeplo F PPAL (Program Manager) Montserrado 5. Mabel M.K. Kear F YMCA Program Manager Montserrado 6. Tupee Freeman F PACS Capacity Building Advisor Montserrado 7. Alex B. Keimbe M PACS ST. WASH Advisor Montserrado 8. Isaac Mwase M M&E Advisor Montserrado 9. Delielie Kollie M PSI-PACS Program Manager Montserrado 10. Jacob Zahye M SEARCH Prog. Manager Montserrado Other Partners or Stakeholders 11. Jenny Rabinowich F Deputy Country Director-Last Mile Health Montserrado 12. Benjamin Grant M Acting Country Director -Last Mile Health Montserrado County, Community and District Level KII 13. Mamayan Kromah F WASH Coordinator, MPW-Nimba Nimba 14 Bartoa Bartuah Sr. M District Superintendent Nimba 15 Precious D. Davis F WaterGuard Vendor Nimba 16 Mercy Richards F WaterGuard Vendor Nimba County 17 Aaron S. Glay M District Health Officer Nimba County 18 Winston Farngalo M EPI Supervisor Nimba County 19 Process K. Tokpah F Community Health Services Supervisor(CHSS) Nimba County 20 Samuel B. Koi M PACS- Finance Officer Nimba County 21 Josiah Gbusseh M PACS-Marketing Coordinator Nimba County 22 Mark Cooper M PACS Community Development Officer Nimba County 23 Dakagar Daniel M OIC-Karnplay Health Center Nimba County 24 Saye Z. Glaymein M OIC-Ganta Community Clinic Nimba County 25 Willimina Saye F Community Health Services Supervisor(CHSS) -Ganta Nimba County 26 Betty Wehdah F Community Health Services Supervisor(CHSS) -Ganta Nimba County 27 Kula Wesseh F Community Health Services Supervisor(CHSS) -Flumpa Nimba County 28 Jerry Tuah M OIC-Flumpa Clinic Nimba County 29 Esther Gatei F Gbahn Community Member Nimba County 30 Dr. Collins Bowah M County Health Officer(CHO) Nimba County 31 C. Paul Nyanzee M Community Health Department Director(CHDD) Nimba County 32 Rufus G. Saye M Clinical Health Services Director(CHSD) Nimba County 33 Lee Danno M Community Health Promotion Focal Person Nimba County 34 Julius Menwan M Community Health Focus Person(CHFP) Nimba County 35 Isaac Cole M County Surveillance Officer(CSO) Nimba County 36 Austin G. Mehn M Environmental Health Technician Nimba County 37 John Dawalee M Coordinator, Christ Children Home Bong County 38 Young A. Paegar Sr. M County Environmental Health Technician/IPC Officer Bong County 39 Korpo Akoi F Community Health Services Supervisor(CHSS)-Gbonota Clinic Bong County 62 40 Youngor Wilson F Community Health Services Supervisor(CHSS)-Totota Clinic Bong County 41 Ciazue B. Melvin F RH- Supervisor -BCHT Bong County 42 Joseph Carter M Community Health Promotion Focal Person(CHPFP) Bong County 43 Parker Paye M Child Survival Focal Person Bong County 44 Jerome Toto M PACS -Community Mobilization Officer Bong County 45 Ranger Martin M PACS Finance Officer Bong County 46 Dr. G. Gorbee Logan M County Health Officer Bong County 47 George Arthur M PACS County Coordinator Bong County 48 Joe K. Tonorlah M Officer In-charge-Gbonota clinic Bong County 49 Prince Kollie M Project Officer -DEB-L/PACS Bong County 50 J. Kessilee M M&E Officer-BCHT Bong County 51 Juebah V. Mattews M PACS/PPAL-Project Coordinator Bong County 52 Mark Momoh M PACS County Supervisor Bong County 53 John Quetela M Dispenser-Panta Medical Center Bong County 54 Mary J. Loweal M Dispenser-Panta Medical Center Bong County 55 David Forkpah M Pharmacy assistant/Dispenser-WaterGuard Vendor Bong County 56 Emmanuel Dennis M PACS Monitoring and Evaluation Officer Lofa County 57 Ansamana Fayia Jr. M PACS Marketing Coordinator Lofa County 58 Momoh M EHT/EPI Supervisor -LCHT Lofa County 59 Adbullah Andu M PACS County Coordinator Lofa County 60 Youngor D. Mulbah F Community Health Focal Person -LCHT Lofa County 61 Gorpu George F District Health Officer-Salayea District. Lofa County 62 Massayan K. Jallah M County Health Department Director G. Cape Mount Co. 63 Lawrence V. Moore M. Environmental Health Coordinator G. Cape Mount Co. 64 Bintu Sheriff F Screener-Mambo Clinic G. Cape Mount Co. 65 Maima Darblo F General Community Health Volunteer G. Cape Mount Co. 66 Sando kiadii M Spare Parts Dealer-Gbeleh Town G. Cape Mount Co 67 Ivan Jarbeh Gray M Project Field Officer-Conseil Sante International G. Cape Mount Co 68 Varney Kaizulu M Owner- VFK Medicine Store-Sinje Town G. Cape Mount Co 69 Newongbay Geh M Shopkeeper, Kenson’s Medicine Store -Sinje Town G. Cape Mount Co 70 Otis Zarzar M County WASH Coordinator Grand Gedeh Co. 71 Netus N. Nowine M Community Health Department Director Grand Gedeh Co. 72 William Mensah M Community Health Focal Person Grand Gedeh Co. 73 Matilda Billy F Reproductive Health Supervisor Grand Gedeh Co. 74 Alfred Collins M Environmental Health Supervisor Grand Gedeh Co. 75 Momodu Kromah M Program Coordinator-Last Mile Health Grand Gedeh Co. 63 ANNEX VI: PAIRWISE CORRELATION COEFFICIENTS TABLE *Indicates Bonferroni-adjusted significance levels of 0.05 or less Variable ODF Hand-dug well CHVs % female CHVs U5 diarrhea Hand-dug well 0.57 * 1.00 CHVs -0.07 0.21 * 1.00 % female CHVs -0.06 0.21 * 0.99 * 1.00 U5 diarrhea cases 0.24 * 0.26 * 0.52 * 0.47 * 1.00 Measles coverage 0.45 * 0.27 * -0.08 -0.05 0.17 Penta coverage 0.39 * 0.26 * 0.01 0.02 0.29 * OPV coverage 0.40 * 0.26 * -0.01 0.00 0.28 * Severe malaria 0.03 -0.28 * -0.22 * -0.22 * -0.15 U5 treated with ACTs 0.29 * 0.39 * 0.68 * 0.67 * 0.74 * U5 ARI treated with anitbiotics 0.09 0.13 0.50 * 0.44 * 0.75 * U5 diarrhea treated with ORS/zinc 0.24 * 0.27 * 0.52 * 0.48 * 0.99 * Short-acting FP methods 0.41 * 0.34 * 0.44 * 0.44 * 0.71 * Total ANC visits 0.33 * 0.34 * 0.69 * 0.70 * 0.76 * IPTp2 0.50 * 0.33 * 0.11 0.15 0.36 * TT2 0.15 0.16 0.26 * 0.29 * 0.27 * 4th ANC visit 0.40 * 0.34 * 0.64 * 0.64 * 0.76 * Delivery outside health facility 0.07 -0.01 0.12 0.16 0.12 Post-partum HF visit 0.36 * 0.29 * 0.49 * 0.50 * 0.73 * 64 ANNEX VII: DIFFERENCE IN DIFFERENCES ANALYSIS RESULTS TABLE DHIS2 Indicators and comparators PACS v. Non￾PACS districts 1/2014-10/2015 to 11/2015-5/2017 PACS v. Non￾PACS districts 3-5/2015 to 3-5/2017 PACS v. non￾PACS districts in Lofa, Nimba, Bong counties 1/2014-10/2015 to 11/2015-5/2017 PACS v. similar districts using Propensity Score Matching (PSM) 1/2014-10/2015 to 11/2015-5/2017 PACS v. similar districts using PSM 3-5/2015 to 3-5/2017 Child immunization Measles coverage Difference in Differences (6.488)*** (4.919) (5.827)* (6.470)*** (4.919) p-value 0.005 0.397 0.063 0.005 0.397 Penta coverage Difference in Differences (3.505)* (1.472) (1.335) (3.495)* (1.472) p-value 0.103 0.780 0.620 0.103 0.780 OPV3 coverage Difference in Differences (5.501)* (1.709) (1.035) (3.492)* (1.709) p-value 0.103 0.746 0.707 0.103 0.746 Childhood Illness & treatment Severe malaria Difference in Differences (1.068) (-3.830) (-6.475) (0.666) (-3.830) p-value 0.801 0.704 0.301 0.874 0.704 Malaria cases treated with ACT Difference in Differences (61.429)** (131.982)* (74.522)* (61.429)** (131.982)* p-value 0.019 0.066 0.103 0.019 0.066 Diarrhea cases treated with ORS Difference in Differences (0.618) (-8.291) (-0.392) (0.651) (-8.291) p-value 0.761 0.129 0.914 0.748 0.129 ARI cases treated with antibiotics Difference in Differences (-13.090)* (-20.481) (-8.862) (-13.089)* (-20.481) p-value 0.104 0.327 0.547 0.104 0.327 Family Planning Injectable contraceptive users Difference in Differences (33.452)*** (29.061) (-16.816) (33.462)*** (29.061) p-value 0.000 0.208 0.223 0.000 0.208 Oral contraceptive users Difference in Differences (0.028) (3.176) (0.823) (0.028) (3.176) p-value 0.995 0.783 0.930 0.995 0.783 Maternal and neonatal health ANC visits Difference in Differences (43.591)** (22.097) (0.579) (43.591)** (22.097) p-value 0.013 0.630 0.980 0.013 0.630 IPTp2 coverage Difference in Differences (3.733)* (2.804) (-0.467) (3.733)* (2.804) p-value 0.083 0.622 0.881 0.082 0.662 TT2 coverage Difference in Differences (6.630)*** (8.461) (11.493)*** (6.630)*** (8.461) p-value 0.010 0.202 0.001 0.010 0.202 4th ANC visit Difference in Differences (4.613) (3.668) (-8.599) (4.614) (3.668) p-value 0.333 0.778 0.195 0.333 0.778 Deliveries outside HF Difference in Differences (0.673) (0.085) (1.642)** (0.676) (0.085) p-value 0.130 0.940 0.046 0.128 0.940 Post-partum Vitamin A Difference in Differences (4.629) (-0.915) (-1.395) (4.629) (-0.915) p-value 0.139 0.914 0.738 0.139 0.914 *p<0.10; **p<0.05; ***p<0.01 65 ANNEX VIII: DIFFERENCE IN DIFFERENCES ANALYSIS FIGURES 64.7% 81.7% 75.2% 62.9% 73.5% 55% 60% 65% 70% 75% 80% 85% Jan. 2014- Oct. 2015 Nov. 2015- May 2017 Difference in Differences: Measles Coverage PACS Projected Control 6.49%*** 8.26%*** 1.8% 73.6% 91.2% 87.7% 69.1% 83.2% 60% 65% 70% 75% 80% 85% 90% 95% Jan. 2014- Oct. 2015 Nov. 2015- May 2017 Difference in Differences: Penta 3 Coverage PACS Projected Control 3.5%* 8.0%*** 4.5%*** 73.6% 91.3% 87.8% 69.2% 83.4% 60% 65% 70% 75% 80% 85% 90% 95% Jan. 2014- Oct. 2015 Nov. 2015- May 2017 Difference in Differences: Oral Polio Vaccine 3 Coverage PACS Projected Control 3.5%* 7.9%*** 4.4%*** 66 199 349 288 115 205 50 100 150 200 250 300 350 400 Jan. 2014- Oct. 2015 Nov. 2015- May 2017 Difference in Differences: Malaria Cases Treated with ACTs PACS Projected Control 83*** 145*** 61** 92 87 100 35 43 00 20 40 60 80 100 120 Jan. 2014- Oct. 2015 Nov. 2015- May 2017 Difference in Differences: Children with ARI Treated with Antibiotics PACS Projected Control 57*** -13* 144 207 174 57 87 00 50 100 150 200 250 Jan. 2014- Oct. 2015 Nov. 2015- May 2017 Difference in Differences: DMPA Users PACS Projected Control 87*** 33*** 120*** 67 232 315 271 104 144 50 100 150 200 250 300 350 Jan. 2014- Oct. 2015 Nov. 2015- May 2017 Difference in Differences: ANC visits PACS Projected Control 127** 44** 171*** 46.5% 61.5% 57.8% 39.1% 50.4% 30% 35% 40% 45% 50% 55% 60% 65% Jan. 2014- Oct. 2015 Nov. 2015- May 2017 Difference in Differences: IPTp2 Coverage PACS Projected Control 11.1%*** 3.7%* 7.4%*** 50.0% 62.7% 56.1% 48.8% 54.9% 40% 45% 50% 55% 60% 65% Jan. 2014- Oct. 2015 Nov. 2015- May 2017 Difference in Differences: TT2 Coverage PACS Projected Control 1.2% 6.6%*** 7.9%*** 68 ANNEX IX: SEGMENTED REGRESSION ANALYSIS RESULTS TABLE VARIABLES Measles Penta3 OPV3 Severe Malaria Malaria treated Diarrhea treated ARI treated PACS start coefficient (-0.115) (2.882) (2.351) (41.181)*** (18.741) (7.261) (0.067) p-value 0.990 0.757 0.801 0.000 0.890 0.339 0.997 Trend change after PACS start coefficient (0.613) (0.458) (0.458) (-3.597)*** (15.884) (1.213) (-3.441)** p-value 0.495 0.609 0.614 0.000 0.352 0.270 0.034 Time coefficient (0.629) (0.344) (0.363) (-0.573) (2.567) (-0.748) (1.634) p-value 0.256 0.533 0.514 0.275 0.794 0.230 0.106 Rainy season coefficient (-2.601) (-2.047) (-2.405) (14.714)*** (103.778)* (-1.072) (-1.358) p-value 0.558 0.646 0.587 0.009 0.084 0.736 0.886 Dry season coefficient (2.566) (1.290) (1.005) (-2.137) (5.588) (0.179) (5.608) p-value 0.559 0.770 0.817 0.747 0.915 0.949 0.595 Constant coefficient (58.320)*** (72.434)*** (72.215)*** (57.082)*** (474.869)*** (57.704)*** (159.700)*** p-value 0.000 0.000 0.000 0.000 0.001 0.000 0.000 Observations 41 41 41 41 41 41 41 R-squared 0.384 0.361 0.363 0.625 0.225 0.418 0.221 VARIABLES DMPA Pills ANC ANC4+ Delivery Post￾partum Vit A IPTp2 TT2 PACS start coefficient (53.446)* (17.663) (14.390) (-8.167) (2.502)*** (7.549) (-3.107) (3.249) p-value 0.054 0.215 0.341 0.648 0.002 0.461 0.716 0.661 Trend change after PACS start coefficient (4.855)** (2.071)* (1.671) (0.772) (-0.155)** (-0.823) (1.744)* (0.227) p-value 0.044 0.097 0.187 0.709 0.020 0.391 0.065 0.738 Time coefficient (-0.526) (-0.789) (-0.327) (1.178) (-0.025) (0.564) (0.005) (0.203) p-value 0.757 0.374 0.683 0.335 0.540 0.342 0.993 0.631 Rainy season coefficient (15.298) (10.415) (11.657) (-1.026) (1.146)*** (-0.228) (-2.936) (-6.182)* p-value 0.322 0.198 0.166 0.897 0.010 0.963 0.448 0.096 Dry season coefficient (26.700) (11.279) (11.600) (2.786) (0.399) (3.961) (-1.868) (-2.128) p-value 0.169 0.263 0.259 0.698 0.444 0.432 0.602 0.572 Constant coefficient (222.773)***(107.051)***(103.133)***(127.491)*** (3.000)*** (85.810)*** (56.819)*** (54.362)*** p-value 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 Observations 41 41 41 41 41 41 41 41 R-squared 0.635 0.321 0.296 0.395 0.390 0.295 0.451 0.324 *** p<0.01, ** p<0.05, * p<0.10 69 ANNEX X: SEGMENTED REGRESSION ANALYSIS FIGURES 0 20 40 60 80 2014m1 2015m1 2016m1 2017m1 date in year/month Mean change Predicted values Severe malaria cases 50 100 150 200 2014m1 2015m1 2016m1 2017m1 date by year/month Mean change Predicted values ARI cases treated with antibiotics 20 40 60 80 100 2014m1 2015m1 2016m1 2017m1 date in year/month Mean change Predicted values Proportion of pregnant women receiving IPTp2 70 1 2 3 4 5 6 2014m1 2015m1 2016m1 2017m1 date by year/month Mean change Predicted values Deliveries outside of the health facility 100 200 300 400 500 2014m1 2015m1 2016m1 2017m1 date by year/month Mean change Predicted values Depo-provera users 50 100 150 200 2014m1 2015m1 2016m1 2017m1 date by month/year Mean change Predicted values Oral contraceptive users 71 ANNEX XI: CBA PARAMETERS AND DATA SOURCES Parameters Value Source Interpolated change Notes Population (2015) 1,261,507 Census data (2008) adjusted for annual population growth For Bong, Nimba and Lofa Population growth rate 2.44% World Bank - - GDP per capita (PPP) $812.90 World Bank - - Economic growth rate 6.1% World Bank annual 10-year average from 2005-2015 Discount rates 12% 20% (water and sanitation) Cost-Benefit Analysis of USAID/Liberia’s Rice and Goat Value Chain Interventions Final Report - Higher discount rate applied to water and sanitation interventions to account for identified relapse and sustainability issues Contraceptive Prevalence Rate (CPR) 12.3% DHS - Weighted average for 3 counties Improved water source 82.7% KPC Baseline report 83.7% Increase based on 90 wells*250 beneficiaries as a proportion of the total population of the 3 counties Improved sanitation 34.6% KPC Baseline report 70% - Handwashing with soap (after handling feces and before food prep.) 60.7% KPC Baseline report 70% - Diarrhea prevalence 20.8% DHS Modeled Weighted average for 3 counties Severe diarrhea prevalence 4.2% DHS Modeled Weighted average for 3 counties ORS for diarrhea 28.6% KPC Baseline report 80% - Zinc for diarrhea 12.7% KPC Baseline report 80% - Antibiotics for pneumonia 46.15% DHS 80% No data for Lofa; weighted average for Bong and Nimba ACTs for malaria 51.8% KPC Baseline report 80% - ANC 4+ visits 72% DHS 85% Rural areas ANC 2-3 visits 17.6% DHS Rural areas Facility delivery /skilled delivery 50.8% /53.7% DHS 72% /75% Weighted average for 3 counties 72 Parameters Value Source Interpolated change Notes Cost per ORS & zinc treatment $0.88 International Drug Indicator Guide, MSH/WHO - Inclusive of ITSH, in-country distribution and wastage Cost per antibiotic treatment $0.12 International Drug Indicator Guide, MSH/WHO - Inclusive of ITSH, in-country distribution and wastage Cost per ACT $2.75 International Drug Indicator Guide, MSH/WHO - Inclusive of ITSH, in-country distribution and wastage ITSH 10% Industry standard - - Wastage 10% Industry standard - - In-country transport costs 31% Estimating the Global In-Country Supply Chain Costs of Meeting the MDGs by 2015 Technical Brief, DELIVER - - Mean walk time to health facility (one￾way) 75 minutes DHS Rural areas Weighted average Rainy season June-Oct DHS Control variable 73 ANNEX XII: COST-EFFECTIVENESS LITERATURE REVIEW The results from a systematic search of the cost-effectiveness literature on reproductive, maternal, newborn, and child health (RMNCH) are shown in Figure A. PACS interventions are highlighted in yellow; relevant vaccinations are highlighted in blue. Cost-effectiveness in health is commonly expressed as a cost (in 2012 USD) per Disability Adjusted Life Year (DALY) which represents one lost year of "healthy" life27. Rural water supply and sanitation has been shown to yield a cost $2,000 per DALY averted. By contrast, zinc supplementation ($50 per DALY averted) and ORS ($150 per DALY averted) and averted rotavirus vaccine ($100 per DALY at the Gavi price in LICs). And point-of-use water treatment in rural areas ($180-$200 per DALY averted). And, handwashing promotion has been found to cost just $88 per DALY averted. Access to modern contraceptives cost about $250 per DALY averted. Facility – based treatment of pneumonia illness is estimated to cost $600 per DALY averted. In relation to community-based malaria treatment, the cost per DALY averted is $93 compared to standard care. As a guideline for low-resource countries, WHO considers an intervention to be cost-effective when the cost per DALY averted is less than three times the Gross Domestic Product (GDP) per capita; interventions that cost less than GDP per capita are classified as highly cost-effective. Currently, GDP per capita in Liberia is $455.40 (USD) and $812.90 (PPP) (World Bank, 2017). 27 http://www.who.int/healthinfo/global_burden_disease/metrics_daly/en/ 74 Figure A. Cost-Effectiveness of Interventions for RMNCH, in 2012 USD per Disability Adjusted Life Year (DALY) Averted Source: Horton S, Levin C. Cost-effectiveness of interventions for reproductive, maternal, neonatal, and child health. In: Jamison D, Nugent R, Gelband H et al. (ed). Disease Control Priorities: Improving Health & Reducing Poverty 75 ANNEX XIII: COMPARISON OF NCHA PROGRAM IMPLEMENTATION: PACS AND OTHER MODELS NCHA feature Implementing partner/agency PACS Plan International Last Mile Health CHT with UNICEF support Unsupported CHT Location(s) Bong, Lofa & Nimba (9 districts) Bong, Lofa (7 districts) Grand Gedeh (2 districts) Grand Gedeh (3 districts) Grand Cape Mount (5 districts) Community health cadre CHA and CHSS CHA and CHSS CHA and CHSS CHA and CHSS gCHV Selection process Community-driven Gender bias Community-driven Gender bias Community-driven Gender bias Community-driven Gender bias Gender bias CHA training Completed 4 modules Completed 4 modules Completed 4 modules Module 1 only, other training is delayed Not started Remuneration 2016 rates, regularly paid 2016 rates, regularly paid 2017 rates, regularly paid 2016 rates, 3-4 month delays in payment n/a Drug & health commodities Not authorized Limited supply Provided in full, outside NDS Authorized but long delays in NDS provision n/a CBIS/data collection Paper-based registers No formal link to MOH Unclear Electronic data collection tools/mobile phone app None None Integration with WASH Awareness by CHA, NL, others, unstructured interactions None None None None Status July 2017 Community engagement Health messaging/advice Referrals to health facility Community engagement Health messaging/advice Referrals to health facility Community engagement Health messaging/advice Referrals to health facility Limited and irregular engagement No regular community engagement Alignment with NCHA Training CHA/CHSS, fuel support, Aligned with CHA, CHSS, training, motorcycles, some drugs Full package, with significant additional support Limited package, no motorcycles, delays in stipends and supplies Not implemented Engagement with CHT Communication, limited support, no coordination Communication, limited support, no coordination Parallel system, some resentments CHT-led CHT-led Differences from NCHA Male bias in selection, no drugs/commodities None Community engagement officers, electronic app, None None Financing 5-year USAID cooperative agreement to 2020 USAID (Possible additional private funding, unverified) Private funding (USAID support for core office and Rivercess) Annual contract with UNICEF via MOH, renewable MOH (World Bank funded NGO from Aug 2017) 76 ANNEX XIV: PACS HAND-DUG WELLS IN RELATION TO GOL AND USAID STANDARDS FOR WATER QUANTITY (YIELD) # County Community District Population near well Well depth (m) Static water level (m) Water column (m) Req’d water (ltr/day) Calc’d water vol. (m3) Meets standards Male Female Total GOL USAID 1 Bong Bellekpala #2 Jorquelleh 138 204 342 10 7.6 3 6840 3.39 NO NO 2 Bong Bilitania Suakoko 41 44 85 10 4.4 5.6 1700 6.33 YES YES 3 Bong Bucket-ta Jorquelleh 161 137 298 9.6 5.6 4 5960 4.52 NO NO 4 Bong Featuah Zota 288 216 504 12.5 7.5 5 10080 5.65 YES NO 5 Bong Gbarsue Jorquelleh 70 80 150 10 5 5 3000 5.65 YES YES 6 Bong Gobonokole Suakoko 35 41 76 9 5 4 1520 4.52 NO YES 7 Bong Gokai Suakoko 48 53 101 9.9 5 4.9 2020 5.54 YES YES 8 Bong Gorniyeleyah Salala 43 36 79 7.4 3.4 4 1580 4.52 NO YES 9 Bong Gwetita Jorquelleh 60 65 125 10 6 4 2500 4.52 NO YES 10 Bong Jamieyea Jorquelleh 83 100 183 9 5 4 3660 4.52 NO YES 11 Bong Kar farm Salala 100 110 210 9 6 3 4200 3.39 NO NO 12 Bong Kelekpai Jorquelleh 293 317 610 10 5.5 4.5 12200 5.09 YES NO 13 Bong Keliker-ta Suakoko 12 13 25 14.37 10 4.37 500 4.94 NO YES 14 Bong Kokolo-ta Jorquelleh 45 73 118 14.6 11.6 3 2360 3.39 NO YES 15 Bong Kpanawa Zota 48 52 100 9.6 6.6 3 2000 3.39 NO YES 16 Bong Kpeyea-ta Jorquelleh 106 114 220 9 5 4 4400 4.52 NO YES 17 Bong Manlonkai-ta Salala 86 94 180 9 6 3 3600 3.39 NO NO 18 Bong Martin-ta Jorquelleh 58 62 120 10.62 6.62 4 2400 4.52 NO YES 19 Bong Moloquelleh #2 Jorquelleh 52 62 114 10.6 7.6 3 2280 3.39 NO YES 20 Bong Palatanda Zota 336 364 700 12.7 8.7 4 14000 4.52 NO NO 21 Bong Sakpala Zota 250 309 559 11 7 4 11180 4.52 NO NO 22 Bong Sengbeh-ta Jorquelleh 67 83 150 7.8 4.9 2.9 3000 3.28 NO YES 23 Bong Sianneh-ta Zota 76 82 158 9.6 6.6 3.6 3160 4.07 NO YES 24 Bong Smith-ta/ Zota 150 180 330 15 11 4 6600 4.52 NO NO 77 # County Community District Population near well Well depth (m) Static water level (m) Water column (m) Req’d water (ltr/day) Calc’d water vol. (m3) Meets standards Male Female Total GOL USAID 25 Bong Suah Gbarngah-ta Kpanta paai 105 230 335 10 6 4 6700 4.52 NO NO 26 Bong Tornor-ta Jorquelleh 45 39 84 12 8.52 3.6 1680 4.07 NO YES 27 Bong Wornukai Village Jorquelleh 60 65 125 10.6 6.6 4 2500 4.52 NO YES 28 Bong Yelepolu-Wannie-ta Salala 50 60 110 11.8 8.9 2.9 2200 3.28 NO YES 29 Lofa Barfelleh Foya 57 63 120 11.7 9.3 2.4 2400 2.71 NO YES 30 Lofa Bolobengu Kolahun 192 208 400 9.6 7.2 2.4 8000 2.71 NO NO 31 Lofa Borley Village Voinjama 62 68 130 11.4 8.4 3 2600 3.39 NO YES 32 Lofa Dedehsu Voinjama 77 83 160 11 5.45 5.55 3200 6.27 YES YES 33 Lofa Fassamah Voinjama 276 299 575 12.2 9.4 2.8 11500 3.17 NO NO 34 Lofa Gondama Vahun 45 68 113 12.85 9 3.85 2260 4.35 NO YES 35 Lofa John Wolobah Zorzor 72 93 165 8.4 4.75 3.65 3300 4.13 NO YES 36 Lofa Karvedu Voinjama 79 85 164 11.6 8.5 3.05 3280 3.45 NO YES 37 Lofa Kebeh Village Zorzor 50 90 140 8 4 4 2800 4.52 NO YES 38 Lofa Koibartomai Voinjama 125 135 260 11.7 8.7 3 5200 3.39 NO NO 39 Lofa Koisua Foya 132 180 312 12.3 9.2 3.1 6240 3.50 NO NO 40 Lofa Kolubah-Gilleh-ta Voinjama 50 55 105 10 5.16 4.84 2100 5.47 YES YES 41 Lofa Korlehawai/ Helbalahun Kolahun 120 130 250 11.5 8.4 3 5000 3.39 NO NO 42 Lofa Kortuhun Kolahun 98 79 177 11.4 7.2 4.2 3540 4.75 NO YES 43 Lofa Kpassay Foya 120 130 250 10.2 7.2 3 5000 3.39 NO NO 44 Lofa Kpelloe Foya 110 170 280 12.2 9.2 3 5600 3.39 NO NO 45 Lofa Mamah Foya 61 68 129 7.8 4.8 3 2580 3.39 NO YES 46 Lofa Mar-Custom Foya 192 208 400 7.2 4.8 2.4 8000 2.71 NO NO 47 Lofa Peter town Voinjama 41 44 85 14 7.7 6.3 1700 7.12 YES YES 48 Lofa Sayanin Foya 40 71 111 12 9 3 2220 3.39 NO YES 49 Lofa Tawalahun Kolahun 181 197 378 11.4 8.4 3 7560 3.39 NO NO 50 Lofa Vonema Voinjama 132 144 276 14 10.5 3.3 5520 3.73 NO NO 51 Lofa Whykenedu Voinjama 233 629 862 8.78 4.78 4 17240 4.52 NO NO 78 # County Community District Population near well Well depth (m) Static water level (m) Water column (m) Req’d water (ltr/day) Calc’d water vol. (m3) Meets standards Male Female Total GOL USAID 52 Lofa Yormbu Vahun 50 65 115 7.8 2.4 5.4 2300 6.10 YES YES 53 Lofa Ziamai Village Voinjama 65 70 135 10.8 7.8 3 2700 3.39 NO YES 54 Nimba Siaplay School Zoe-Geh 73 84 157 12 9 3 3140 3.39 NO YES 55 Nimba Beindin Saclepea-mah 400 655 1055 12.6 9.6 3 21100 3.39 NO NO 56 Nimba Blean Play Zoe-Geh 144 156 300 8.4 5.2 3.2 6000 3.62 NO NO 57 Nimba Borpea # 1 Gbehlay-Geh 65 85 150 11.4 8.1 3.3 3000 3.73 NO YES 58 Nimba Deepeh Village Saclepea-mah 64 86 150 9.4 5.2 4.2 3000 4.75 NO YES 59 Nimba Dolopa Village Sanniquellie￾Mah 63 67 130 11.6 6.1 5.5 2600 6.22 YES YES 60 Nimba Fehnplay Zoe-Geh 440 800 1240 11.4 8.1 3.3 24800 3.73 NO NO 61 Nimba Flumpa Saclepea-mah 1413 2099 3512 11.4 8.1 3.3 70240 3.73 NO NO 62 Nimba Gaussi Village Sanniquellie￾Mah 120 130 250 10.1 6.3 3.8 5000 4.30 NO NO 63 Nimba Gbanzeryeepea Gbehlay-Geh 175 189 364 10.9 4.8 6 7280 6.78 YES NO 64 Nimba Geelar Gbehlay-Geh 168 182 350 10.2 7.2 3 7000 3.39 NO NO 65 Nimba Guantar-old town Gbehlay-Geh 151 158 309 6.7 3.2 3.5 6180 3.96 NO NO 66 Nimba Johnny Blohn Village Saclepea-mah 100 150 250 8.6 5.5 3.1 5000 3.50 NO NO 67 Nimba Karpoplay Zoe-Geh 134 146 280 14.1 11 3 5600 3.39 NO NO 68 Nimba Kpeingatuo Zoe-Geh 99 125 224 10.5 7 3.5 4480 3.96 NO NO 69 Nimba Larkeh Comm. Saclepea-mah 75 82 157 8.6 5.4 3.2 3140 3.62 NO YES 70 Nimba Mahn-play Zoe-Geh 371 413 784 8.2 6.4 1.8 15680 2.03 NO NO 71 Nimba Mankeryor/Dolopei Sanniquellie￾Mah 98 107 205 10.3 5.7 4.6 4100 5.20 YES YES 72 Nimba Nalah Zoe-Geh 91 99 190 7.4 3.4 4 3800 4.52 NO NO 73 Nimba New Blowee Saclepea-mah 165 179 344 8.5 4.7 3.8 6880 4.30 NO NO 74 Nimba Nuontar Gbehlay-Geh 108 130 238 6.6 3.2 3.4 4760 3.84 NO NO 75 Nimba Nyateh Village Saclepea-mah 49 60 109 7.8 4.8 3 2180 3.39 NO YES 76 Nimba Siaplay New Town Zoe-Geh 500 750 1250 10.8 7.8 3 25000 3.39 NO NO 79 # County Community District Population near well Well depth (m) Static water level (m) Water column (m) Req’d water (ltr/day) Calc’d water vol. (m3) Meets standards Male Female Total GOL USAID 77 Nimba Tekeiplay Gbehlay-Geh 11.4 5.7 5.7 0 6.44 YES YES 78 Nimba Tomah Village Zoe-Geh 67 73 140 12 9 3 2800 3.39 NO YES 79 Nimba Vanyenpa Sanniquellie￾Mah 126 137 263 13.9 8.4 5.5 5260 6.22 YES NO 80 Nimba Willie Gio Gbehlay-Geh 33 34 67 7.2 6.9 3.3 1340 3.73 NO YES 81 Nimba Zain/Wimpea Village Gbehlay-Geh 100 110 210 9.9 7 2.9 4200 3.28 NO NO 82 Nimba Zekeh Village Saclepea-mah 90 120 210 8.7 4.9 3.8 4200 4.30 NO YES 80 ANNEX XV: LIST OF FIELD SITES VISITED FOR DATA COLLECTION No: County District Communities Date of Visit Nimba a. Sanquellie-Mahn Saniquellie Town June 30th –July 3rd, 2017 2. Nimba Sanquellie-Mahn Duo-Tiayee June 30, 2017 3. Nimba Sanquellie-Mahn Ganta City July 4, 2017 4. Nimba Gbellaygeh Karnplay Town June 30th & July 1st , 017 5. Nimba Gbellaygie Keplay July 1, 2017 6. Nimba Gbellaygie Peela July 1, 2017 7. Nimba Gbellaygie Kaiplay July 1, 2017 8. Nimba Saclepea-Mahn Flumpa July 3, 2017 9. Nimba Saclepea-Mahn Gbahn July 3, 2017 10. Nimba Zoe-geh Siaplay July 1, 2017 11. Nimba Zoe-geh Beleglay July 1, 2017 12. Nimba Zoe-geh Weplay July 1, 2017 13. Bong Suakoko Pbebe Hospital July 17, 2017 14. Bong Suakoko Blonmue July 18, 2017 15. Bong Suakoko Bassa Village July 18, 2017 16. Bong Suakoko Gbartala Clinic July 21, 2017 17. Bong Jorquelleh Gbarnga City July 17th -21st, 2017 18. Bong Jorquelleh Buckita July 21, 2017 19. Bong Jorquelleh Tonorta July 21, 2017 20. Bong Sanoyea Gbonota July 18, 2017 21. Bong Sanoyea Sanoyea Town July 18, 2017 22. Bong Sanoyea Konee July 18, 2017 23. Grand Gedeh Konobo Turglor July 6, 2017 24. Grand Gedeh Cavalla Gboleken July 6, 2017 25. Grand Gedeh Tchien Zwedru City July 5th -7 th, 2017 26. Grand Cape Mount Garwulu Sinje July 12th -14th, 2017 27. Grand Cape Mount Gola kenneh Mbaloma July 13, 2017 28. Grand Cape Mount Tewor Manboe July 13, 2017 29. Lofa Salayea Lomata July 18, 2017 30. Lofa Salayea Salayea Town July 18, 2017 31. Lofa Salayea Memeta July 18, 2017 32. Monserrado Monrovia Ministry of Health multiple 33. Monserrado Monrovia Ministry of Public Works multiple 34. Monserrado Monrovia PACS offices multiple 35. Monserrado Monrovia USAID multiple 36. Monserrado Monrovia MOGCSP multiple 81 ANNEX XVI: DISCLOSURE OF CONFLICTS OF INTEREST 82 83 84 85 86 U.S. Agency for International Development - Liberia 502 Benson Street Monrovia, Liberia