EVALUATION USAID/LIBERIA PARTNERSHIP FOR ADVANCING COMMUNITY-BASED SERVICES FINAL EVALUATION This publication was prepared independently by Social Impact, Inc. at the request of the United States Agency for International Development under the Liberia Strategic Analysis Activity. RAVI M RAM FOR SOCIAL IMPACT ABSTRACT The Partnership for Advancing Community-based Services (PACS) Activity implemented an integrated community health, water, sanitation, hygiene, and behavior change model in Bong, Lofa, and Nimba counties of Liberia. The final performance evaluation focused on PACS’ performance, primarily fidelity of implementation and outcomes. The evaluation used a range of methods, including primary qualitative data (interviews and group discussions), as well as secondary quantitative and geospatial data analysis, and employed a gender lens in its approach. Key findings are that PACS was generally effective, with some gaps in both implementation fidelity and outcomes related to behavior change and sustainability of community sanitation. USAID/LIBERIA PARTNERSHIP FOR ADVANCING COMMUNITY-BASED SERVICES FINAL EVALUATION USAID/Liberia Liberia Strategic Analysis Contract No: AID-669-C-16-00002 September 17, 2019 Contact: Liberia Strategic Analysis Social Impact, Inc. UN Drive Adjacent Gate 3 US Embassy Monrovia, Liberia POC: Carla Trippe, ctrippe@socialimpact.com DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. TABLE OF CONTENTS I. EXECUTIVE SUMMARY....................................................................................................... i II. INTRODUCTION................................................................................................................. 1 ACTIVITY BACKGROUND...............................................................................................................................................1 DEVELOPMENT HYPOTHESIS.........................................................................................................................................2 EVALUATION PURPOSE AND AUDIENCE...............................................................................................................3 EVALUATION QUESTIONS..............................................................................................................................................3 III. METHODOLOGY................................................................................................................. 4 EVALUATION DESIGN.......................................................................................................................................................4 LIMITATIONS..........................................................................................................................................................................6 IV. FINDINGS.............................................................................................................................. 7 EQ 1. PERFORMANCE.........................................................................................................................................................7 V. CONCLUSIONS................................................................................................................. 28 EQ 1. PERFORMANCE......................................................................................................................................................28 VI. RECOMMENDATIONS...................................................................................................... 31 RECOMMENDATIONS FOR MOH (CENTRAL AND CHTS).........................................................................31 RECOMMENDATIONS FOR MOGCSP.....................................................................................................................31 RECOMMENDATIONS FOR DEOH (IN MOH) AND MPW...........................................................................31 RECOMMENDATIONS FOR PACS AND PACS-SUPPORTED CSOS..........................................................32 VII. UTILIZATION PLAN......................................................................................................... 32 VIII.ANNEXES............................................................................................................................ 33 Annex A: Full Listing of References and Report Utilized.......................................................................................34 Annex B: Full Listing of Persons Interviewed.............................................................................................................35 Annex C: Geospatial equity analysis of PACS’ CLTS sites vs other (non-CLTS) communities...............36 Annex D: Data Collection Instruments.......................................................................................................................38 Annex E: Evaluation Statement of Work.....................................................................................................................93 Annex F: Disclosure of Any Conflicts of Interest..................................................................................................102 TABLES Table 1: Evaluation Questions for PACS Final Evaluation..............................................................................................3 Table 2: Summary of Qualitative Respondents by Qualitative Method and Gender...........................................5 Table 3: Training Outcomes and Achievements for Community Health Workers Training.........................11 Table 4: Outcomes and Achievements for Community Health Service Delivery..............................................16 Table 5: Achievements in Community Sanitation and Hygiene Behavior ............................................................21 Table 6: Outcomes and Achievements for WASH Services and Practices..........................................................23 Table 7: Comparison of CLTS and Hygiene Promotion Costs (USD) for PACS and International Peer Comparators Implementing WASH in Liberia................................................................................................................25 Table 8: Comparison of CLTS and Hygiene Promotion Costs (USD) for PACS-Supported CSOs............26 Table 9: Geospatial equity analysis of PACS’ CLTS sites vs other (non-CLTS) communities (Bong, Lofa and Nimba counties, 2019)....................................................................................................................................................36 FIGURES Figure 1: CSO OCA Scoring Averages (1-4 scale, n=5).................................................................................................9 Figure 2: PACS Areas of Intervention and the Integrated Approach.....................................................................27 Figure 3: PACS CLTS Sites in Relation to Major Roads, Bong, Lofa and Nimba counties (2019)................37 ACRONYMS ALMEP Activity-Level Monitoring & Evaluation Plan BCC Behavior Change Communication CBIS Community-based Information System CDCS Country Development Cooperation Strategy CEBS Community Event-Based Surveillance CHA Community Health Assistant CHC Community Health Committee CHDD Community Health Department Director (County-level) CHFP Community Health Focal Person CHO County Health Officer CHSD Community Health Services Division (MOH) CHSPSP Community Health Services Policy and Strategic Plan CHSS Community Health Services Supervisor CHT County Health Team CHV Community Health Volunteer CHW Community Health Worker CLTS Community-Led Total Sanitation CSO Civil Society Organization CWC Community WASH Committee CWSC Community Water & Sanitation Committee DEOH Division of Environmental and Occupational Health (NPHIL) DHIS District Health Information Systems DHT District Health Team DSA Daily Subsistence Allowance EEF Ebola Emergency Fund EHT Environmental Health Technicians EOHD Environmental and Occupational Health Division EPHS Essential Public Health Services EPSS Essential Public Social Services EQ Evaluation Question ETL Education Through Listening EVD Ebola Viral Disease FGD Focus Group Discussion GC Global Communities GHSA Global Health Security Agenda GOL Government of Liberia HFDC Health Facility Development Committee HMERD HMIS, M&E and Research Division (MOH) HMIS Health Management Information System HPD Health Promotion Division iCCM Integrated Community Case Management IRC International Rescue Committee KII Key Informant Interview LOE Level of Effort LSA Liberia Strategic Analysis M&E Monitoring & Evaluation MFDP Ministry of Finance and Development Planning MIA Ministry of Internal Affairs MOGCSP Ministry of Gender, Child, and Social Protection MOH Ministry of Health MPW Ministry of Public Works NCHAP National Community Health Assistant Program NGO Non-Governmental Organization NHPD National Health Promotion Division (MOH) NPHIL National Public Health Institute – Liberia NTD Neglected Tropical Disease NWSHPS National Water, Sanitation & Hygiene Promotion Secretariat OD/F Open-Defecation/Open-Defecation Free OIC Officer In-Charge PACS Partnership for Advancing Community-based Services PD Program Description PIP Performance Improvement Plan PSI Population Services International – Liberia RDF Revolving Drug Fund SBCC Social and Behavior Change Communication SI Social Impact, Inc. SOW Statement of Work TTM Trained Traditional Midwife USAID United States Agency for International Development WASH Water, Sanitation and Hygiene YMCA Young Men’s Christian Association (Liberia) i | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV I. EXECUTIVE SUMMARY EVALUATION PURPOSE AND EVALUATION QUESTIONS This final evaluation of the Partnership for Advancing Community-Based Services (PACS) provides an independent, in-depth examination of the achievements and lessons learned from the Activity. PACS was a community health, water, sanitation and hygiene (WASH), and behavior change Activity in Bong, Lofa, and Nimba counties of Liberia. The evaluation examined the fidelity of implementation and outcomes, specifically on effectiveness and potential for sustainability, as well as lessons for future community health and WASH activities. The key intended users are USAID and Government of Liberia (GOL) partners including the Ministry of Health (MOH), Ministry of Public Works (MPW) and National WASH Commission of Liberia, Ministry of Gender, Children and Social Protection (MOGCSP), and County Health Teams (CHTs). USAID will use the evaluation to inform the design of future community health and WASH activities in Liberia. Secondary users include donors, GOL local authorities, non-governmental organizations (NGOs), and other Civil Society Organizations (CSOs) working on community health and/or WASH in similar contexts. One overarching evaluation question (EQ) framed the final evaluation around PACS’ performance, with six specific questions as listed below. EQ 1. PERFORMANCE: For key objective areas listed below, to what extent did PACS implement with fidelity (i.e., did they do what they said they were going to do) and to what extent do the available data suggest improved outcomes as a result of implementation in Bong, Lofa and Nimba counties, with results disaggregated by gender? EQ 1.1a. Build capacity of GOL and communities for the Community Health Assistant (CHA) program? EQ 1.1b. Build capacity of the MOH and MPW to sustain the CHA and WASH program? EQ 1.2. Implement the CHA program? EQ 1.3. Increase community demand for healthy behaviors and health services? EQ 1.4a. Improved WASH at the community level? EQ 1.4b. What is the cost of implementing CLTS compared to other international NGOs? The full phrasing of each evaluation question is provided in the main report. ACTIVITY BACKGROUND The purpose of PACS was to advance USAID Liberia’s Country Development and Cooperation Strategy Development Objective 3, “Improved Health Status of Liberians." PACS had four intermediate results(IR), areas, which formed the basis for EQs 1.1, 1.2, 1.3 and 1.4, respectively: ⋅ IR 1. Broadened capacity of central MOH, CHT, local CSOs and NGOs to implement and manage community services; ⋅ IR 2. Increased quality and availability of community-based health and social welfare services; ⋅ IR 3. Increased demand for health services and improved health-seeking behavior and practices; ⋅ IR 4. Improved access to safe WASH services at the community level. EVALUATION DESIGN, METHODS, AND LIMITATIONS The evaluation team (ET) collected qualitative data from a sample of key stakeholders at the central, county, and district level, as well as from community and traditional leaders. The ET conducted Key Informant Interviews (KIIs) and Focus Group Discussions (FGDs) with 251 total respondents, of whom 155 (62 percent) were male and 96 (38 percent) female. The ET captured and USAID.GOV USAID/LIBERIA PACS FINAL EVALUATION REPORT ii systematically processed KII and FGD data using a coding system for each topic area. The ET complemented the primary qualitative data with geospatial analysis of PACS intervention sites in relation to main roads and health facilities, as a measure of equity in coverage, as well as a survey of capacity development among participating CSOs. This process informed development of key findings, conclusions and recommendations that comprise the bulk of this report. Limitations in this evaluation included a strong male bias among interview and discussion respondents, reflecting the structural gender bias in health workers at all levels of the health system. In response, the ET made special efforts to obtain perspectives and information from female leaders, health workers and community members. Additionally, recall and response bias among respondents may have over-reported positive or negative findings based on their individual experience and memory. The evaluation design did not include community-based quantitative data collection, and existing quantitative data in PACS and GOL databases did not address key outcomes. Accordingly, conclusions are based on primary qualitative data collected in the evaluation and PACS’ monitoring data. Wherever possible, the ET triangulated qualitative findings with observations and secondary data to provide a balanced response to the evaluation questions. FINDINGS AND CONCLUSIONS EQ 1: PERFORMANCE Areas of PACS success included the following: ⋅ EQ 1.1a and 1.1b (IR 1): Capacity building for six CSOs (inclusive of one consortium partner) and three CHTs to continue implementation of the CHA program after PACS’ closure and provision of material support to improve internal resources and aid activity management; ⋅ EQ 1.2 (IR 2): Rollout and implementation of the national CHA program in the three counties, aligned with national policy and in coordination with the CHT and other county stakeholders; ⋅ EQ 1.3 (IR 3 and 4): Community empowerment for local engagement in health and WASH, through the reactivation of CHCs and CLTS; ⋅ EQ 1.4a (IR 4): Construction or rehabilitation of wells and implementation of Community-led Total Sanitation (CLTS) in communities in the three counties. Gaps and challenges in PACS’ implementation revolve around the following areas: ⋅ IR 1: Unsustainable capacity building efforts with MOH, MPW and MOGCSP, largely due to GOL disinterest, competing capacity building from other Implementing Partners (IPs) and donor dependency; capacity improvements were not measured; ⋅ IR 2: Insufficient involvement of GOL and CHT in the rollout and implementation of the CHA program, with PACS undertaking a larger role in order to meet its targets, and CHT counterparts less engaged in part due to PACS’ use of lower daily subsistence allowance rates; ⋅ IR 3: Behavior change messaging that was not backed by structural change such as an enabling environment to access necessary commodities – which did not precipitate targeted behavior change or resulted in short-term, unsustainable outcomes only; ⋅ IR 3: Considerable underperformance in promoting home-based water treatment through Waterguard; ⋅ IR 4: Non-response to evidence of community reversion from open defecation free status back to open defecation; ⋅ IR 4: Late and inadequate employment of CSOs supported by PACS for implementing CLTS and conducting community monitoring; ⋅ Overall: Lack of a truly integrated community health and WASH implementation model; PACS operated more as a parallel set of actors rather than a unified force, thereby limiting the potential for synergy of intentional reinforcement of community health and WASH. iii | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV RECOMMENDATIONS ⋅ In the planned revision of the NCHAP strategy, MOH should introduce a focus on gender in order to mitigate the growing gender bias in CHA and CHSS staffing and other aspects of the program, through close involvement of the MOGCSP and an external gender technical expert. ⋅ To address ongoing needs for continuous monitoring and training support, the MOH should prioritize budget line items for CHSS incentivization and in-service trainings for CHAs, as well as continued quarterly review meetings at both district and county level. ⋅ MOH should ensure regular drugs and commodities to CHAs through a dedicated, prioritized stream within the supply chain that focuses on drugs and commodities for children under five years and pregnant women, as well as family planning commodities, keeping with the equity provisions of the proposed drug-revolving fund. ⋅ MOH can engage with social anthropologists to ensure a relevant and sustainable approach to behavioral change programming to address socio-cultural factors around open defecation reversion and community motivation driven by NGOs. ⋅ PACS-supported CSOs should collaborate closely with CHTs and other county GOL stakeholders (County WASH Officers, ICWSC), with monitoring of ODF communities as an early agenda item. ⋅ MOH should enact a mandatory order for points of contact from MPW to attend MOH meetings (and vice versa) to identify and coordinate activities around WASH infrastructure site selection and CHA collaboration and enable joint goal setting and accountability. 1 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV II. INTRODUCTION ACTIVITY BACKGROUND In February 2015, USAID contracted the consortium of International Rescue Committee (IRC), Global Communities (GC), Population Services International (PSI), Young Men’s Christian Association (YMCA), and the Planned Parenthood Association of Liberia (PPAL) 1 to implement a$24.8 USD million2 community health, water, sanitation and hygiene (WASH) Activity called the Partnership for Advancing Community￾based Services (PACS). The overall purpose of PACS was to advance USAID/Liberia’s Country Development Cooperation Strategy (CDCS), Development Objective (DO) 3, “Improved Health Status of Liberians." USAID specifically designed PACS to “support sustainable country ownership for community-based health, social welfare, and Water, Sanitation and Hygiene (WASH) services [by] ensuring [Government of Liberia] leadership on policy and planning down to service delivery.” Key in-country partners for PACS included the Ministry of Health (MOH) and its Divisions on County Health Services (CHSDs) and National Health Promotion (NHPD), Ministry of Public Works (MPW), the National Water, Sanitation and Hygiene Promotion (NWSHP) Secretariat, and the Ministry of Gender, Children and Social Protection (MOGCSP). In addition, the Division of Environmental and Occupational Health (DEOH) in the MOH also served as a key stakeholder for PACS’ work in all three counties. USAID designed PACS to contribute to the CDCS through the following results: ⋅ IR 1: Broadened capacity of central MOH, County Health Teams (CHTs), local CSOs, and NGOs to implement and manage community services through stakeholder agreement at national and county levels; partnership agreements with MOH units and CHTs; and performance improvement for MOH systems and skills to manage and monitor community health services. ⋅ IR 2: Increased service quality and availability of community-based health and social welfare services. This result supported the MOH to review, develop, and roll out an updated, standardized, and integrated package of community health and social welfare services to be delivered by community health cadres. PACS also engaged CSOs to strengthen linkages between communities and health facilities and improve accountability within the health system. ⋅ IR 3: Improved health-seeking behavior and practices. Activities generated information and behavior change strategies at the community level to inform communities about services and ways to adopt more positive health behaviors in order to increase demand for quality health services. ⋅ IR 4: Improved access to safe WASH services by building capacity of water and sanitation personnel to effectively manage WASH infrastructure; supporting the Government of Liberia (GOL) in deploying trained and equipped water and sanitation staff, and establishing a pump fund to prepare MPW for future management of WASH infrastructure improvement programs. PACS’ expected results aligned with those of the National Health Policy and Plan (2011- 2021), the Investment Plan to Building a Resilient Health System (2016-2021), and the Revised Community Health Services Policy and Strategic Plan (CHSPSP) (2016-2021). In addition to alignment with specific policies and strategies of the MOH, MOGCSP, and MPW, PACS also sought to address the GOL’s Poverty Reduction Strategy Paper Pillar 3 and post Ebola objective: “To rehabilitate infrastructure and rebuild systems to deliver basic service in order to create the conditions and linkages needed to achieve broad￾based growth and poverty reduction.” 1 Planned Parenthood Association of Liberia exited the PACS consortium in Year 3 of implementation (August 2017), following USG’s implementation of the Mexico City policy earlier that year, and is not included in this final evaluation. 2 Later expanded to $36.8 million to include a time-bound intervention in three additional counties affected by the Ebola epidemic up to December 2017 and to transition all stakeholders to a sustainable surveillance system through Community Event-Based Surveillance. USAID.GOV USAID/LIBERIA PACS FINAL EVALUATION REPORT 2 During the second year of implementation, the USAID Global Health Bureau under the Ebola Emergency Fund (EEF) funded PACS with an additional 7 million USD to expand the Activity to three additional counties affected by the Ebola Virus Disease outbreak (Grand Bassa, Margibi and Rural Montserrado). With this modification, PACS quadrupled its performance targets under the original Activity-Level Monitoring and Evaluation Plan (ALMEP) in terms of the number of people PACS trained, equipped, and supervised. After the Ebola epidemic of 2014-2015, the MOH’s rapid restoration of community-based health services and the remuneration of community health workers through a more robust community health worker (CHW) program (including Community Health Assistants (CHAs) and Community Health Services Supervisors (CHSSs)) led to the modification of the PACS award. This shift in the GOL’s priorities for community health service implementation altered the scope of programming for PACS, as USAID had not included many of these service requirements in the original design of the award. After the mid-term evaluation in 2017, PACS revised its Program Description (PD) to better reflect the country’s conditions and priorities, including new national policies and health strategies such as the Revised CHSPSP, MOH’s Investment Plan for Building a Resilient Health System, and the National Community Health Assistant Program (NCHAP). The revised PD (October 2017- February 2020) also incorporated mid-term evaluation recommendations and realigned its budget to reflect changes to the scope and associated interventions. PACS implemented interventionsin collaboration with the MOH and CHTs in the USAID priority counties of Bong, Lofa, and Nimba, as well as the three EEF counties of Margibi, Montserrado, and Grand Bassa. DEVELOPMENT HYPOTHESIS PACS made critical assumptions and developed several hypotheses about the conditions that should be changed in the Liberian health system in order for USAID to realize DO3 in Liberia. Those assumptions and hypotheses form the basis for the PACS Theory of Change: ⋅ If PACS provides the MOH and local CSOs well-targeted, high-quality, and scalable technical and financial assistance, and if the Liberian health system has the absorptive capacity, then MOH and local CSOs will possess broadened capacity to implement and manage community-based health services in remote communities that are more than five kilometers from health facilities. ⋅ If the MOH and CSOs have broadened capacity to conduct health promotion and behavior change communication activities, and if caregivers consistently practice healthy behaviors – such as ensuring their children sleep under insecticide-treated nets, washing hands before preparing food, eating and after toileting, adhering to schedules to fully immunize children, following recommendations for exclusive breastfeeding and appropriate complementary feeding, using effective methods of family planning, and seeking care for antenatal and supervised delivery at health facilities – then incidence of Acute Respiratory Infections (ARI), malaria, diarrhea and malnutrition will reduce and pregnancy outcomes will improve. ⋅ If MOH and CSOs have broadened capacity to implement and manage health services at the community level, and if there are sufficient numbers and spread of trained community health workers (CHAs and Community Health Volunteers [CHVs]) in communities located five kilometers or farther away from health facilities, and if treatment services for sick children are available and accessible in these communities through trained CHAs on the one hand, and caregivers seek prompt and effective treatment for communicable diseases for their children on the other hand, then prevalence of ARI, malaria, and diarrhea among children under-five will decline. ⋅ If households have access to improved sanitation services, and household water treatment options are available at an affordable cost, and households have access to safe drinking water sources, then prevalence of diarrhea and other waterborne diseases among children will further reduce. 3 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV ⋅ If morbidity rates (ARIs, malaria, and diarrhea) among children under five are reduced, then child mortality rate will decline. EVALUATION PURPOSE AND AUDIENCE The purpose of this evaluation was to provide an independent and in-depth examination of the overall progress of community health services, WASH services, and health-seeking behaviors supported by the PACS Activity in Liberia. The evaluation identified achievements, performance issues, and constraints related to Activity implementation and effectiveness. The evaluation team will share and discuss findings and recommendations with USAID/Liberia, Implementing Partners, and relevant GOL units. The key intended users of the evaluation are USAID and GOL partners, primarily MOH, the County Health Teams (CHTs), Ministry of Public Works (MPW), and the Inter-County Water & Sanitation Commission (ICWSC). Secondary users include other donors and development partners, GOL health facilities and local authorities, NGOs, and other CSOs working in similar settings. The intended uses of the evaluation are: (1) to account for the PACS investment in integrated community health and WASH interventions in the three counties; and (2) to document lessons from the PACS Activity. EVALUATION QUESTIONS The evaluation questions (EQs) presented below encompass PACS’ four result areas. The evaluation team (ET) evaluated the four PACS result areas against their contribution to the overall performance objective. TABLE 1: EVALUATION QUESTIONS FOR PACS FINAL EVALUATION EVALUATION QUESTIONS AND SUB-QUESTIONS EQ 1. PERFORMANCE: For key objective areas listed below, to what extent did PACS implement with fidelity (i.e., did they do what they said they were going to do) and to what extent do the available data suggest improved outcomes as a result of implementation in Bong, Lofa and Nimba counties, with results disaggregated by gender? EQ 1.1a. Build capacity (management systems, knowledge transfer, awareness of and attention to gender and equity factors, and political will) of MOH, County Health Teams, CSOs and communities for the Community Health Assistant program? EQ 1.1b. Build capacity of the MOH and MPW to sustain implementation of the CHA and WASH program and demand creation activities? EQ 1.2. Increase the availability of sustainable and quality community-based health services (for child health, family planning, maternal health, nutrition and WASH) including 2-way referrals with health facilities? EQ 1.3. Increase community demand for healthy behaviors and health services (child health, family planning, maternal health, nutrition and WASH)? EQ 1.4a. Improve access to and management of safe water supply and sanitation, with attention to gender roles, at the community level? EQ 1.4b. What is the cost of the access – as defined by the administration cost of PACS providing access to CLTS water and sanitation services per beneficiary – as compared to the administration cost under an international NGO as was previously used? USAID.GOV USAID/LIBERIA PACS FINAL EVALUATION REPORT 4 III. METHODOLOGY EVALUATION DESIGN The evaluation used a mixed-methods design to conduct data collection in the focal counties of Bong, Lofa, and Nimba between May 22 and June 11, 2019. To address EQ 1 and subcomponent questions 1.1a through 1.4b, the ET analyzed primary qualitative data and quantitative data from a CSO survey and available secondary quantitative data from PACS consortium partners and the GOL’s District Health Information Systems (DHIS)-2 database. The evaluation used information from the desk review and semi-structured qualitative data collection instruments (see Annex D) for KIIs and FGDs the ET designed to solicit in-depth knowledge from key stakeholders on implementation and outcomes of PACS interventions in the three priority counties. The ET tailored these tools for each respondent type (e.g., USAID staff, GOL, community members, county, and local health staff), and organized them by evaluation question. QUALITATIVE DATA Desk Review The ET reviewed available program documents and relevant literature to understand PACS’situational and programmatic context. The ET assessed material by document type (e.g., annual progress reports, KAP assessment, other special studies) and document source (e.g., PACS, GOL, other NGOs), and mapped them against the EQs. This framework allowed the evaluation to highlight gaps in the literature for probing during data collection. Desk review findings served as a source of triangulation for primary qualitative data collected during fieldwork. Annex A contains the full list of documents the ET reviewed. Key Informant Interviews The ET conducted 52 Key Informant Interviews with stakeholders and beneficiaries at the central, county, health facility, and community levels to provide context for the overall evaluation and gain a deeper understanding of PACS implementation. The KIIs explored Activity engagement and gauged the development of institutional capacity, along with effectiveness and scale-up of approaches, successes, challenges, and lessons learned. The ET purposively sampled respondents based on their roles and specialized knowledge, primarily regarding PACS, community health, WASH, and gender. Using an initial key informant contact sheet that implementing partners provided the ET, the evaluation employed a snowballing approach to identify additional, relevant key informants as fieldwork progressed. Where possible, the ET considered gender representation in its selection of KII respondentsto ensure it did not omit women’s perspectives, particularly in the context of a male-dominated culture and health workforce. Focus Group Discussions The ET conducted 18 FGDs, randomly selecting community-level participants within the sampled districts based on listings of CHAs, Community Health Services Supervisor (CHSS) and facility staff, as well as Community Health Committee (CHC) members in the villages the ET visited. The evaluation engaged Monrovia-based GOL, USAID Health Office personnel and PACS staff, and captured the perspectives of community members and leaders, health facility staff, senior-level GOL representatives, and PACS field staff. Table 2 presents an overview of the planned and actual qualitative data. A total of 251 respondents participated in the qualitative data collection exercise, of whom 38 percent were female. 5 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV TABLE 2: SUMMARY OF QUALITATIVE RESPONDENTS BY QUALITATIVE METHOD AND GENDER RESPONDENT GROUP KII FGD* TOTAL CONTACTS PLANNED ACTUAL PLANNED ACTUAL PLANNED ACTUAL MALE FEMALE TOTAL COMMUNITY 4 14 12 15 16 29 122 76 198 COUNTY 11 12 3 3 14 15 17 7 24 CENTRAL 14 25 1 1 15 26 16 13 29 TOTAL 29 51 16 18 45 69 155 96 251 Community-level respondents included lay community members, 14 CHC members, 36 CHAs, six CHSSs, trained traditional midwives (TTMs), four community WASH personnel (natural leaders PACS identified to lead CLTS in their communities; and WASH entrepreneurs PACS trained to maintain and repair hand pumps and latrines), town and paramount chiefs and other local leaders. Respondents from the county level included two County Health Officers, three health facility officers-in-charge (OICs), 11 staff from civil society organizations (CSOs) supported by PACS, three county officers for WASH and social protection, 12 CHT members and six PACS field staff. Central-level respondents included five senior staff from MOH, MPW and MOGCSP; 10 PACS senior management; and five USAID health office personnel. The evaluation exceeded the planned number of respondents in each group due to the “snowball” methodology the ET used to recruit additional respondents.The ET could not conduct the FGD it planned with the CHT in Bong County due to nation-wide political protests on that day (June 7, 2019); however, members of CHTs in all three counties were included in the qualitative data collection, including the County Health Officers. The ET employed the U.S. National Institutes of Health’s procedures for confidentiality, informed consent and ethical treatment of respondents among all respondents in KIIs and FGDs. Direct Observation To validate the information from qualitative respondents, particularly regarding drug stock-outs in health facilities, functionality of hand pumps and hand-dug wells, use of improved sanitation and hygiene practices in communities, the ET conducted direct observation in communities and health facilities visited in Bong, Lofa and Nimba. QUANTITATIVE DATA The ET worked with PACS consortium partners who provided up-to-date monitoring data on programmatic activity outputs and outcomes aligned with activity indicators and timelines.The ET analyzed this data to generate findings around the activity’s progress in meeting targets over the course of implementation and identify any strengths and weaknesses in monitoring and measurement. The ET also gained access to the GOL DHIS-2 monitoring database to identify overarching trends in community health indicators in the districts and counties where PACS implemented. Geospatial Analysis As a means of evaluating PACS’ coverage across the three counties, the ET conducted geospatial analysis using ArcGIS software. The analysis matched community settlement data to PACS intervention areas, including sites with iCCM and WASH (wells or CLTS) interventions, and included additional layers for roads and health facilities. The analysis focused on measuring the equity in site selection for communities with or without PACS-targeted interventions. *This number represents focus group discussions held, not number of participants. FGDs included 2-24 participants in total. The column for total represents number of activities (interviews or FGDs) undertaken, not number of persons. USAID.GOV USAID/LIBERIA PACS FINAL EVALUATION REPORT 6 Financial Costing Analysis In order to assess PACS’ costs of CLTS implementation in relation to other implementing partners (IPs), the evaluation gathered cost data from PACS, two international peer IPs in Liberia (UNICEF and Living Waters International), and the five local CSOs supported by PACS. A costing template was provided with guidance to all respondent organizations. Cost data was annualized to facilitate comparisons. Because organizations allocated personnel and some other costs across CLTS and community hygiene activities, further analysis was conducted by combining those two components. Other potential discrepancies were considered to reduce the observed differences, and the analysis as presented can be seen as conservative. Quantitative Mini-Surveys To supplement qualitative data collection, the evaluation employed two mini-surveys: (1) a comprehensive assessment of CSO capabilities; and (2) a prioritization exercise administered to GOL and civil society stakeholders. Five PACS-supported CSOs completed the CSO survey, as well as the YMCA, a PACS consortium member that is also a national CSO, and the ET analyzed the results of this mini survey to gauge the changes in capacity noted by CSO partners during their involvement with PACS. The ET designed the prioritization survey to develop participatory rankings on respondents’ priorities for health service delivery improvement over the next five years. KII respondents noted key areas that the ET transformed into a series of both closed and open-ended questions. Respondents included participants from the primary qualitative data collection list, as well as individuals identified through the snowballing methodology. However, due to the low response rate among all respondent types, the data from the prioritization mini-survey were insufficient for the ET to consider valid for use in the analysis. LIMITATIONS The evaluation relied extensively on qualitative data, to which respondents may have introduced recall and/or response biases. The evaluation mitigated these risks by triangulating data sources wherever possible. The evaluation was able to include a larger number of respondents in the KIIs and FGDs than had been anticipated. However, the overall qualitative sample is still small compared to the PACS’ intended reach, and cannot be considered statistically representative. As such, the evaluation does not attempt to assign attribution at the population or outcome level to the PACS Activity. The ET made efforts to obtain a balance in views among female and male respondents, but due to the gender imbalance in the staffing of GOL positions – both in Monrovia and in the counties – a gender bias appears in the overall qualitative and prioritization data. The ET mitigated this bias by deliberately recruiting female respondents and by analyzing responses from female respondentsseparately from males, when possible and appropriate. Financial data for CLTS and WASH overall reported by PACS, other IPs and CSOs were subject to differences in each organization’s programmatic approach and costing model. The evaluation adjusted for these differences by combining costs for CLTS and community hygiene, the two components where appreciable differences were found. The final performance evaluation design did not include a community-based survey to assess quantitative outcomes or impact from PACS’ implementation. Consequently, the findings provide qualitative information related to sustainability of community health and WASH interventions but cannot describe quantitative data for key outcomes. Such data were also unavailable in PACS and GOL databases and therefore cannot be answered in this evaluation. 7 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV IV. FINDINGS EQ 1. PERFORMANCE For key objective areas listed below, to what extent did PACS implement with fidelity (i.e., did they do what they said they were going to do) and to what extent do the available data suggest improved outcomes as a result of implementation in Bong, Lofa and Nimba counties, with results disaggregated by gender? The ET addressed the performance questions under EQ 1 by examining implementation with fidelity, then presenting available outcome data. Fidelity refers to effectiveness (delivering results according to the Activity design and approved workplans) with additional attention to quality and sustainability. This evaluation defines outcomes as changes that arose as a result of PACS’ outputs and through the work of GOL, CSOs, and communities. EQ 1.1A. BUILD CAPACITY (MANAGEMENT SYSTEMS, KNOWLEDGE TRANSFER, AWARENESS OF AND ATTENTION TO GENDER AND EQUITY FACTORS, AND POLITICAL WILL) OF MOH, COUNTY HEALTH TEAMS, CSOS AND COMMUNITIES FOR THE COMMUNITY HEALTH ASSISTANT PROGRAM PACS’ capacity development and strengthening interventions covered a range of areas including: logistics, materials, technical and managerial skill development, and system improvements for the Central Ministry (MOH), County Health Teams (CHTs), Civil Society Organizations (CSOs), and communities (CHCs, CHSS, CHAs, CHVs). MOH: PACS provided support to the MOH to assess internal management strengths and weaknesses, resulting in the development of performance improvement plans (PIPs) for the Community Health Services Division (CHSD),the National Health Promotion Division (NHPD) and DEOH. The plans encouraged units to create clear job descriptions and defined roles and responsibilities to improve communication bottlenecks. PACS supported the MOH to develop agendas and take minutes at department meetings which helped enforce organized internal management. PACS also procured vehicles for central level coordination and provided financial support to secure internet services. PACS also supported CHSDand NHPDstaff to attend additional trainingon leadership and management, both within and outside of Liberia. However,there was some GOL staff reluctance to attend trainings or collaborative capacity development activities when no monetary daily subsistence allowance (DSA) was provided, or DSA was less than government staff expected. PACS stated in their implementation plan that they would not increase DSA allotments at the risk of impacting activities only because the government did, and has not done so. Training on project management and the assignment of short-term technical assistants to MOH technical working groups ended when PACS determined that the MOH was becoming overly reliant on technical assistance, were not fully engaged or committed to implementing PIPs, and because the MOH was receiving similar support from another IP. OUTCOMES FOR MOH: Despite the support provided by PACS and the opportunities to engage in several capacity development activities to improve a variety of skills, MOH staff and other stakeholders identified a number of gaps where continued investments were needed, namely in project management, monitoring and evaluation, and strategic planning. CHTs: PACS provided quarterly financial support for material goods and logistical support, as well as skills training and management strengthening for Bong, Lofa and Nimba CHTs. This included office stationery, computers, and printers, along with fuel and motorbikes to transport drugs to facilities, WASH reagents to county WASH officers, and supervision of DHTs and facilities. Skill building and process improvements focused on staff management capacity development. In addition, 15 CHT staff including Community Health Focal Persons and Health Promotion Focal Persons received Education Through Listening (ETL) training from PACS (focused on encouraging community self-awareness, USAID.GOV USAID/LIBERIA PACS FINAL EVALUATION REPORT 8 proactiveness and self-diagnoses), becoming Master Trainers for delivering ETL training to CHVs, CSOs, CHSS, DHTs and OICs. CHT officers anecdotally noted that ETL helped them with community engagement. One CHT member stated, "We have specific supervision skills now and know how better to stimulate and ask the right questions when monitoring." For monitoring and evaluation (M&E), PACS provided training on the Community-based Information System (CBIS) on reporting responsibilities and timelines and standard operating procedures for 149 MOH staff focusing on data clerks and M&E managers, as well as DHOs, CHFPs, OICs and CHSS. This training also sought to improve understanding of revised iCCM indicators and data entry protocols. Additionally, PACS provided supportto M&E by assisting staff with the monthly (district), quarterly (county) and semi￾annual (central MOH) supervision follow-ups in order to detect gaps in data and reporting. Every quarter, PACS conducted a Data Quality Assessment (DQA) with a M&E officer from the CHT, utilizing CHA registers and monthly reports and CHSS reports, complemented by the CBIS data. OUTCOMES FOR CHTs: CHT staff respondents believe the financial and logistic support to be the most notable contribution of PACS, and came to rely on these material supplies and support the most. CHT staff in Lofa and Nimba noted that the M&E support from PACS helped them to better understand indicators, and introduced a newly-occurring quarterly data review. However, to date, the CHTs in Bong, Lofa or Nimba counties have not conducted any DQAs independent of PACS.Indeed, respondents noted that M&E is a key area where continued capacity development is needed, namely in data quality and reporting, for the central level, CHT and DHT. PACS did not establish targets for capacity building activities for MOH and CHT and is not collecting outcome monitoring data or measurements of capacity improvements for CHT members to measure actual skill development or improved knowledge and practices after trainings. CHT respondents reported that PACS’ internal decision-making took time. CHTs were also critical of long delays in receiving equipment and consumables,such as motorbikes. However, PACS’ principal focus was not on procurement, but on capacity building to strengthen managerial and technical skills for supportive supervision, along with providingfuel for existing motorbikes. Procurement of new motorbikes was notincluded in PACS implementation plan or operating budget until year 5 and was only made possible through additional funds from the Global Health Security Agenda (GHSA) for community surveillance activities in early 2018. Material items, such as motorbikes,required special, additional approvals, and the CHTs’ expectations were such that motorbikes were to be delivered with other support materials; in some cases, CHTs unfavorably compared PACS to other IPs that delivered immediate material goods rather than long-term capacity development. There is evidence of lack of coordination between PACS and CHTs. PACS withheld payments of the quarterly financial support ($2,500) to the CHTs because of a report of “double dipping” of incentive payments for five CHSSs. While all respondents confirmed that PACS informed them of the suspension of financial support, the suspension and the CHTs’ delay in providing requested information regarding CHSS payroll to the MOHfurther affected the counties’ anticipated activities and receipt of supplies.Many CHSSs and CHAs also noted that PACS was often late in making its monthly incentive payments, and asserted that they had not received payment in several months (discussed further under EQ 1.2.). CSOs: PACS initially supported 25 CSOs, including consortium partners YMCA and PPAL, to conduct capacity self-assessments to assess their strengths and weaknesses in Governance, Administration, Human Resources (HR), Project Performance, Technical Capacity and Financial, Organizational and Project Management, guided by a modified version of USAID’s Organizational Capacity Assessment tool, and worked with them to develop PIPs customized to each CSO’s existing level of capacity. Each PIP for the 25 CSOs included a framework identifying a specific timeline for improvement and the specific resources required. In 2016, PACS worked with the MOH to define the future role of CSOs in community health promotion, namely for the CHSPSP. In late 2017, PACS aligned its CSO engagement approach with the Plan. PACS limited support only to those five CSOs 9 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV receiving sub-grants for community health work in which PACS could provide continuous oversight to achieve their PIP goals, and who could directly support PACS’ community health promotion activities (SEARCH, DEN-L, RUCEP, CSI and EQUIP), meeting its target to support a total of six CSOs including YMCA. PACS capacity building for CSOs involved strategic planning and policy development (including aligning their community engagement and MOH coordination strategies to those in the CHSP), institutional vision and mission, report writing, financial management, internal protocols and standard operation procedures for guidance and accountability, identifying critical operational gaps in M&E and organizational development, and technical capacity development for community health engagement. PACS also provided CSOs training to work closely with the CHCs and CHAs to support their community engagement on Behavior Change Communication (BCC) messaging, CLTS, and health promotion activities. PACS’ capacity building support to CSOs was consistent, and all CSOs noted that their MOUs with PACS remained unchanged. In some cases, CSOs received increased funding from PACS to allow for further capacity building activities and for procurement of material supplies for operations. PACS staff worked with CSOs on a monthly mentorship basis to check on PIP progress. However, several CSOs noted that the mentorship support from PACS was not readily available and that “they were supposed to help you get from point A to point B, but they weren’t always there.” OUTCOMES FOR CSOs: CSOs reported that the key areas and most useful types of capacity building support included both institutional and technical capacities. CSOs conducted organizational self￾assessments using OCAs in 2016 (conducted by PACS), and again in 2019 (conducted internally), to rank themselves and gauge their progress. Across the five CSOs receiving sub-grants, improvements in institutional capacity were on average highest in organization management (57.9 percent increase), followed by financial management (45.6 percent increase), and project performance (38.1 percent increase); technical capacity skills decreased for two CSO, and two CSOs did not assess themselves in this category. Skill development was appreciated by all CSOs. One CSO noted, “PACS helped us realize that we needed to budget for full time M&E staff to make sure we were on track, and we never thought about that before.” CSOs’ capacity development also focused on institutional strengthening to better position them to receive funding and improve their systems to be more attractive to new donors, and CSOs have since been successful in acquiring new funding beyond PACS. Additionally, CSOs improved their accuracy and timeliness of reporting, and several CSOs who noted gender considerations as an area of weakness, developed gender policies as part of their PIPs. There were several areas where CSOs noted remaining gaps. These included institutional strengthening, especially for policy design and roll-out, and in meeting international standards for accounting (e.g., accessing and utilizing accounting software licenses). In addition, many CSOs lack computers and sufficient typing skills, and CSOs expressed the need for further capacity strengthening in data analysis and reporting, as well as improved skills in marketing, website management and social media. Further, all CSOs agreed that they are dependent on additional 0 1 2 3 4 Governance Admin HR Finan. Mgmt Org Mgmt Proj. Mgmt Proj. Perform Tech Capacity 2016 average 2019 average FIGURE 1: CSO OCA SCORING AVERAGES (1-4 SCALE, N=5) USAID.GOV USAID/LIBERIA PACS FINAL EVALUATION REPORT 10 and continuing financial support from grants to uphold operations and activities after their PACS sub-grants conclude. One senior CSO staff noted that challenges will likely arise in the future, stating, “You need motorcycles, fuel, and maintenance for movement. There are expectations for monitoring and supervision, but you can’t do the work even when you have love for the work.” As an apparent outcome of PACS capacity strengthening for community engagement with CSOs, the five CSOs developed Community Risk Reduction Plans (CHRRP) with communities and CHCs to assist them with identifying areas of strength and weakness in their community’s health practices and the types of preventive and health improvement measures they could take on independently. Several CSO representatives and PACS staff believed this to be a valuable activity for them to engage with communities to improve capacity at the village level, but noted that it felt more like a ‘pilot’ intervention and lacked the time to complete adequate community follow-up. PACS did not select the CSOs it engaged with;rather they were identified and selected by the CHT based on their existing presence and work in each county. Monthly coordination meetings were held between CHT, DHT, and CSOs where each could share updates on activities and priorities. Despite this, CHT members in each county voiced dissatisfaction in the collaboration between PACS, CSOs, and the CHT in that they did not feel included in CSOs’ work once the project was underway and questioned the extent of the work the CSOs engaged in. CHT respondents felt that CSO allegiance lay only with the implementing partner, and that once they were working with PACS, the CSOs ceased regular communication or meaningful collaboration with the CHT. Other CHT members stated that CSOs did not widely share their reports with the CHT, that they were unaware of the CSOs activities within their districts, and that CSOs do not see the county as partners to whom they are accountable for their day-to-day work. This was despite the fact that monthly meetings were well attended and a stark contradiction to reports from PACS and CSOs who attended meetings and felt CHT engagement was timely and successful. COMMUNITY HEALTH EDUCATORS AND SERVICE PROVIDERS: PACS assisted CHTs with the roll out of the NCHAP and CHSP validation exercises in each county. The CHSP included the NCHAP framework, and introduced three new categories of community health workers: CHSSs, CHAs and CHVs. CHSSs serve as trained medical staff to provide oversight to the CHAs, collect reports, and serve as a liaison between the health facility and the community, including supplying the CHAs with necessary drugs. CHAs are identified by communities to serve in a leadership capacity, by providing health education and medical treatment to community members for easily treatable concerns like diarrhea, pain management and fever, particularly for children under five years of age, and providing facility referrals for more complicated cases. CHVs’primary role is to provide community health education. The new policy required incentives for CHAs (based on monthly performance and reporting), and provided guidelines for CHSS supervision. The CHPSP was initiated in 2016, and PACS adjusted its program to accommodate and respond to the policy by omitting prior activities and re-directing funds to the NCHAP in line with the specifications around incentivization and supervision in Bong, Lofa and Nimba counties. With MOH facilitation and PACS support, 1,237 gCHVs and 99 CHSS in six counties were trained on basic primary care, and then 720 CHAs and CHSSwere targeted to receive a training package under PACS in three counties. This package was standardized by the GOL and covered four modules: (1) community engagement; (2) Reproductive, Maternal, Neonatal and Child Health; (3) Integrated Community Case Management (iCCM) or treatment of children under five for illness; and, (4) Special programs (Mental Health, HIV, TB, NTDs). Some CHAs starting after January 2019 have not yet received training in all four modules. CHA respondents in all three counties believed that their strongest capacity is in Module 3 and delivering treatment for malaria and diarrhea to children under five years of age, but felt they have not had enough time to absorb or utilize their training on neglected diseases and community based surveillance (CEBS) and feel ill-equipped in their knowledge 11 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV of WASH, a critical area that training does not cover in detail. PACS also provided CHAs with supplies and materials following training, including stationery, waterproof binders, raincoats, boots, backpacks, and flashlights, though this package of supplies was not uniformly distributed and received across counties, namely due to attrition and turnover of CHAs. PACS encouraged community engagement through the establishment of Community Health Committees (CHCs), community structures comprised of community members to encourage health promotion activities and accountability. Health Facility Development Committees (HFDCs) monitor internal health facility operations and patient care, work with town chiefs and CHCs on outreach and health communications, and compile monthly reports for the OIC in the facilities. PACS worked with CSOs to establish 1,817 CHCs and 158 HFDCs in communities that did not have one, and to reactivate these structures in communities that had a CHC or HFDC present, but had ceased to function or were not functioning effectively. The CHC and HFDC promote ownership, identify community roles and responsibilities around health promotion, and foster relationships with the clinics to have community representation at facility meetings. The CHC complements the CHA work in identifying community health concerns and providing education. However, there is a lack of clarity between the HFDC and CHC members and OICs on their roles. OUTCOMES OF CAPACITY BUILDING FOR PROVIDERS OF COMMUNITY HEALTH SERVICES: Support for the CHA program was the most notable form of support PACS provided.PACS trained the CHAs, equipped them, and initiated the monitoring and supervision of CHSS for work in the community, effectively rolling out the community health service delivery structure as NCHAP designed. PACS support for the NCHAP reinforced the importance of meaningful engagement with the community to address health concerns. PACS exceeded its targets for training CHVs, CHAs and CHSSs (see Table 3). The project’s initial reach across six counties trained 1,237 gCHVs and 99 CHSSs in primary health care and 531 CHVs in BCC through ETL. After the introduction of NCHAP in 2017, only CHAs and CHVs categories of community health cadres existed, and once the EEF funds expired in December 2017, PACS scaled back to working in three counties only. In Bong, Lofa and Nimba 664 new CHAs (83 percent male, 17 percent female) were trained, and PACS retained 56 of the trained CHSSs (62 percent male, 38 percent female) and 371 of the trained CHVs. TABLE 3: TRAINING OUTCOMES AND ACHIEVEMENTS FOR COMMUNITY HEALTH WORKERS TRAINING INDICATOR ACTUAL TARGET EOP % ACHIEVED Number of CHVs* trained under communication strategy who are providing health communication messages 531 450 118 percent Number of people trained in child health and nutrition through USG-supported program) (F)* 2,000 1,498 133 percent Number of USG-assisted CHWs (CHAs/CHVs) providing family planning (FP) information, referrals, and/or services during the year (F)* 640 617 103 percent Number of CHVs* trained under communication strategy who are providing health communication messages 531 450 118 percent * Note: Figures received as of July 1, 2019. CHV is an umbrella term that includes TTMs and natural leaders. (F) denotes those standard foreign assistance indicators required by USAID. PACS involvement in training included assisting MOH with curriculum/module development and validation, the printing and dissemination of materials, and conducting a training of trainers for MOH staff. The CHA USAID.GOV USAID/LIBERIA PACS FINAL EVALUATION REPORT 12 and CHSS training itself was facilitated by MOH trainers in line with MOH standards and followed the finalized curriculum approved by the MOH. While training was generally well received, CHAs noted training areas that could be enhanced. CHAs in each county expressed interest in learning more clinical skills in first aid and disease diagnosis to complement their work in iCCM and treatment of children under five, which is further discussed under EQ 1.2. Several CHT respondents agreed that the CHAs could benefit from more advanced training in order to both treat community members and comprehensively collect data and report on their work, but also cited limitations with the selection criteria around minimal levels of literacy. Some felt the NCHAP should standardize selection to community members with high school diplomas instead of just literacy competencies, as well as those with more specialized medical knowledge. However, given the amount of pay CHAs receive and existing responsibilities, it would not be an attractive job option for higher-skilled community members, and adding more responsibilities may not necessarily improve their ability to deliver services. It is widely noted that “CHAs are overloaded…their TOR is bigger than them and they can’t even do what’s assigned,” and that resource and logistical constraints (discussed further under EQ 1.2) impede the CHAs ability to consistently offer quality and comprehensive services. PACS did not capture any outcome data relating to improved knowledge and practicesfor CHAsfollowing training and during the duration of their work. Per the implementation plan (original and revised), PACS measurable outputs are related to CHA training completion and report submission frequency only; PACS did not measure actual increases in CHA capacity in terms of technical health and health service knowledge over time. EQ 1.1B. BUILD CAPACITY OF THE MOH AND MPW TO SUSTAIN IMPLEMENTATION OF THE CHA AND WASH PROGRAM AND DEMAND CREATION ACTIVITIES MOH: PACS provided capacity development in internal management and policy development, areas in which long-term community health programming effects (as stipulated under the Essential Package of Health Services, EPHS) could be more readily felt by improving systems. PACS targeted three units at the MOH (Community Health Services Division [CHSD], Health Promotion Division [HPD] and Environmental and Occupational Health Division [EOHD]) in response to the capacity assessment results that highlighted internal system bottlenecks and challenges to communication that impacted health services. PACS intended for its implementation of management system checks, as well as support to revise guidelines and disseminate strategy materials, to address the bottlenecks and influence long-term operational improvements that would continue to support the CHA program and promote more systematic future community health programming. While there was anecdotal evidence from MOH and CHT of skills sustained as a result of PACS assistance, there was no evidence of ownership over the revised systems and processes. Moreover, the consensus from MOH was that the “support for human resources” PACS provided by directly funding the CHAs and CHSSs themselves was the most valued form of support provided by the Activity over the course of implementation, not the capacity development support to uphold the CHA program on their own. PACS also provided policy revision and dissemination support, working with the Ministry of Gender, Children and Social Protection (MOGSCP) to support the development of vulnerable children guidelines, and with CHTs in each of the three focus counties on social and behavioral change communication (SBCC) policy development. PACS intended for this support to be a sustainable means to guide system improvements and activities after interventions concluded. MOGCSP provided support to policy development as part of support for the Essential Package of Social Services (EPSS), where PACS worked with MOGCSP to revise orphanage accreditation and foster care guidelines. PACS ended its support to the MOGCSP at the end of its third year after conducting a workshop to revise and validate the guidelines, noting that additional support for printing would be provided once 13 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV MOGSCP had finalized the guidelines. While PACS support was to provide revisions and printing only, the guidelines have remained in a near-final draft and have not been finalized by MOGCSP. PACS also provided key support to draft the SBCC Strategic Plan and Communication Policy in 2015, later aligning SBCC messaging with the MOH’s CHSPSP in 2016. Overall, CHTs were satisfied with the support provided by PACS in streamlining BCC messaging and operationalizing the Policy and Strategic Plan with their county focal persons, and improving CHT’s ability to supervise community BCC activities in line with the policy in the future. The three CHTs PACS supported have implemented health promotion plans that reflect the national Strategic Plan and health promotion policies that meets the end-of-project target set by PACS. In preparation for the end of the PACS Activity and to transfer activities to the GOL, PACS engaged in a number ofmeetingsfor government stakeholdersto discuss what the exit of financial support would mean for the MOH and how best to transition key responsibilities. MPW: PACS provided logistical and technical assistance to the MPW for WASH, which consisted of training on hand pump and latrine installation and maintenance, and the provision of vehicles to assist with the monitoring of Community-Led Total Sanitation (CLTS),a community-driven WASH initiative.Under PACS’ implementation plan forYear 2, PACS established a $200,000 pump fund to assist MPW with identifying communities in Bong, Lofa and Nimba without water points and to design and manage water point construction. PACS provided salaries for two WASH County Coordinators and CLTS focal persons but did not intend to provide organizational capacity development support to the MPW as it did with the MOH. MPW respondents felt that the largest gaps in capacity were not related to internal management training or soft skill development (as was provided to MOH and CHTs), but appreciated the material and logistic support PACS provided. However, unlike the MOH who stressed the need for further skill development, MPW respondents highlighted outstanding needs in human resource capacity alone. As one MPW representative stated, the lack of continued attention to WASH at the central level and availability of needed WASH engineers throughout each county was the most significant concern. He noted, “we don’t need additional skills, what we’ll need is additional support. We have limited staff in the counties, and we can’t be everywhere at the same time.” To address the identified shortage in WASH personnel, PACS had intended for trained Wash Entrepreneurs to fill gaps within district and communities, but as noted under EQ 1.4, the Entrepreneurs face their own limitations with funding for repair parts and travel, making them a less reliable source of support. MPW staff were aware that their technical and financial support would end following the end of PACS, and they were unsure of the amount of logistic support they would receive moving forward; this would likely affect the necessary ongoing monitoring of open defecation-free (ODF) status and WASH infrastructure (like latrines and hand pumps) in each county. OUTCOMES OF CHT AND WASH CAPACITY DEVELOPMENT: Genuine capacity building and ownership for MOH at the central and county level, CSOs, and CHAs is subjective, and each actor expressed uncertainty as to how improvements to capacity are effectively measured and the level of responsibility each group has for ensuring continued investment in community health under the CHAP. CHT respondents expressed strong doubts about the capacity and commitment of CSOs to work collaboratively with GOL in advancing the CHA program due to their lack of consistent financing and limited geographic scope. On the other hand, some respondents lauded the CSOs as being the key actors for sustainable community health activities, since they speak local languages, are from the community, and have the community trust. There was a strong consensus among PACS and CSOs that CSOs did not receive deserved recognition from the MOH. CSOs also did not feel confident that the CHTs have the capacity to continue the CHA program after PACS. CSOs and some CHT members cited the inability of CHTs to fully address gaps in capacity due to frequent government staff turnover. Staff undergoestraining in a number of areas and then leaves their government position (or move to another position in another division), resulting in a loss of institutional knowledge USAID.GOV USAID/LIBERIA PACS FINAL EVALUATION REPORT 14 (especially around systems, standards and policy) and requiring new staff to be trained, which does not occur systematically. The CHSPSP called for government to assume full operations of the NCHAP from the implementing partners by 2021. All interviewed stakeholders were doubtful that the MOH will have the capacity at the central or county level to continue to provide support to the CHAsor CHSSs after PACS has concluded. Challenges to continuity include poor supervisory skills, data management, financial management, project management, comprehensive reporting, M&E and knowledge management as hindrances to institutional development at the county and central level. Many MOH and CHT respondents defined “sustainability” as continued USAID (or other donor) support to implementing partners and GOL. One MOH staff stated, “We will continue to do our best to continue the use of the skills after PACS but will still need additional support.” Respondents repeatedly raised GOL’s financial constraints, as well as a relentless dependency on external financing support due to GOL’s insufficient fund allocation, not only for the continuation of the NCHAP, but for upholding the implementation and monitoring of WASH activities. As one MPW representative stated, “We are greatly dependent on UNICEF and USAID to be able to drive us through. Now the government has not passed any budget for WASH. We can’t say ‘we have prepared this plan; we have prepared this program’ and they will fund it. We have not reached that level yet.” Central level and county ministry staff, along with CSOs, agreed that the continuous need for CHSSs supervision and CHAs supplies and incentivization on a monthly basis is potentially the largest challenge for the MOH to address after PACS has closed. PACS’ initial attempts at handing over key functions of the CHAP to GOL were unsuccessful despite the numerous exit meetings and discussions of timelines and responsibilities. One aspect of the MOH transition strategy was to bring CHSSs onto the GOL payroll, as noted in an agreement between PACS and MOH from 2016-17. To date, only five of 56 CHSSs have been placed on government payroll. PACS staff noted that “we haven’t seen evidence that they are taking on a role. By year five we would expect something like 80 percent of CHSSs would be under GOL payroll.” The likelihood that CHA incentives and CHA material supplies will be supplied by the government is poor. While training and knowledge building can easily be replicated at a lower cost by MOH staff themselves, government funding for materials, should those materials continue to be deemed necessary and required by the MOH, is not guaranteed. EQ 1.2. INCREASED THE AVAILABILITY OF SUSTAINABLE AND QUALITY COMMUNITY-BASED HEALTH SERVICES (FOR CHILD HEALTH, FAMILY PLANNING, MATERNAL HEALTH, NUTRITION AND WASH) INCLUDING 2-WAY REFERRALS WITH HEALTH FACILITIES PACS responded to the needs of the CHSP by training and supplying CHAs in nine priority districts in the three focus counties, and ensuring their training curricula and work aligned with the policy. The Revised National Community Health Services Policy states, “Beyond five kilometers from health facilities, CHAs shall provide an integrated package of preventive, promotive, and curative services.” CHAs carry the responsibility for community-based service provision, and are instrumental in carrying out preventative community health education and providing services through clinic referrals and provision of essential medical treatment and drugs to children under five,3 with monitoring and oversight from CHSSs. PACS supported the MOH to train CHAs in the basic expectations of service delivery, including drug provision at the community level. CHAs can carry essential drugs like 3 The national iCCM guidelines, as highlighted in the CHSPSP, restrict community case management to prevention and treatment of diarrhea, respiratory infections and malaria through administration of ORS and conducting RDT tests for children under five years only. 15 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV paracetamol for all community members, and are mandated to only treat children between the ages of two months and five years under the national integrated community case management (iCCM) strategy under which they were trained. The CHA curriculum developed by PACS and the MOH follows the NCHAP and its associated national and global standards for treatment of children under five. CHA practices are limited to be in line with these standards. PACS does not select CHAs or have responsibility over the selection process; however, PACS field staff have encouraged community leaders to consider and select women as CHA candidates. Under the NCHAP, CHAs are selected by community members and required to reside in the community, speak the community language, and possess high school level literacy (deemed necessary for reporting). CHAs are selected by the community – many were previously gCHVs or natural leaders in the community – but none have a formal health background. Female CHAs are generally underrepresented, and less women apply due to community and cultural norms that dissuade women’s participation, including fears about safety. While the NCHAP states a preference for female candidates and encourages women to apply, CHAs are male dominated largely due to the literacy requirement and abovementioned challenges. CHA supervision includes monthly visitation by PACS and up to two monthly visits by the CHSS to review reports, exchange stock of medicine from the facility, complete drug consumption tracking forms, and discuss health facility referrals made. CHAs shared that supervision should be more frequent and consistent, and sometimes the CHSS have limited time and cannot cover all the topics for supervision. OUTCOME OF PACS SUPPORT FOR CHA SERVICE DELIVERY: CHT and PACS respondents reported that PACS support for CHAs reduced the need for facility visits and services by providing health education, prevention, and first-line treatment to communities. Community members stated that they seek out the CHA for treatment of minor ailments before their symptoms become more severe, warranting a visit to the clinic or hospital. Community members, CHAs, and OICs noted that facilities are less crowded because more people are treated in the community. Respondents believed that children contract diarrhea and malaria less frequently, and if they do contract it, are treated quickly before their condition worsens. Community members, TTMs, CHCs, and HFDCs stated that pregnant women seek out and visit facilities more frequently than before, rather than staying in the community. However, many communities impose a fine on women if they do not deliver in a facility, with the hope that this will deter them from a potentially risky home birth that can often necessitate emergency intervention. Community members did not cite challenges with speaking to male or female CHAs, and claimed they are comfortable sharing health concerns with a man or woman just as they would be if they visited a clinic. However, other stakeholders including CHAs, PACS and CHT members noted that there are unique sensitivities that must be considered in addressing male and female health needs and providing health information, especially for adolescents. KII respondents agreed that greater gender balance in CHA representation would enhance community engagement and trust. Nationwide data from DHIS-2 confirms improvements in key indicators for child and maternal health in Bong, Lofa and Nimba counties over the course of PACS, including a substantial reduction in the average number of malaria deaths of children under five from 24.2 percent in 2015 to 4.5 percent in 2018, and an almost 30 percent increase in facility deliveries during the same time. It is important to note that it is not possible to attribute these improvements to PACS, as district specific data is not available, but it is important that there are notable improvements within counties. CHT and PACS staff in Lofa and Bong anecdotally claimed a noticeable difference in community health outcomes between districts with PACS-supported CHAs and districts without, including more cases of child diarrhea and pneumonia at facilities outside PACS USAID.GOV USAID/LIBERIA PACS FINAL EVALUATION REPORT 16 areas, and fewer deliveries in facilities. This level of assessment was not included in PACS ALMEP or implementation plans, and PACS did not track or measure this level of specific outcome data at facilities to ascertain attribution to the Activity, so progress towards improving health outcomes cannot be accurately measured quantitatively. The data that PACS tracked to assess actual progress was provided by CHAs’self-reported activities (noted below in Table 4) alone. This data show that the CHAs exceeded the target for malaria treatment, but were below target for treating pneumonia, and well under target for diarrhea treatment. TABLE 4: OUTCOMES AND ACHIEVEMENTS FOR COMMUNITY HEALTH SERVICE DELIVERY INDICATOR ACTUAL TARGET EOP % ACHIEVED Proportion of children under age five treated for malaria within 24 hours of onset of fever 73 percent 60 percent 113 percent Number of children under age five with suspected pneumonia receiving antibiotics by trained facility or community health workers in USG-assisted program 21,075 36,306 58 percent Number of cases of diarrhea treated in USG assisted programs (F)* 7,166 50,716 14 percent * Note: Figures received as of July 1, 2019. (F) denotes those standard foreign assistance indicators required by USAID. CHAs, CHCs, and community members noted that numerous resource and logistical constraints impeded CHAs’ ability to consistently offer quality and comprehensive services. CHAs were trained to share health messages encouraging receiving treatment for pneumonia, diarrhea and malaria, but CHAs unanimously noted that clinics were consistently out of stock of the drugs needed to treat these illnesses, such as ORS, amoxicillin and zinc, which the evaluation also observed in clinics. Some clinics reported stock-outs of six months or more, which significantly affect the achievement of PACS targets highlighted above (see section on ‘Availability of Drugs and Commodities’ for further information). CHAs also cited inconsistencies in receipt of materials such as thermometers, personal protective gear like face masks and sterile gloves, and supplies like rain jackets and boots, which limited their ability to perform to the standards of their training. PACS mandate was to provide rain boots and jackets, not to procure or distribute medical commodities (noted under EQ1.1), but the lack of other materials due to stock-outs at clinics affects the overall efficacy of the CHAs and the targets PACS has set for treatment. One CHA in Bong county noted frustrations with conducting RDT tests on children with malaria without sterile gloves: “I can’t be treating people with my bare hands. We don’t have a lot of the materials that we are trained to use, and this greatly affects the work.” CHAs are also trained on family planning methods and counseling and can provide information about methods, but they can only carry pills and are unable to provide injections which are often in higher demand. This prompts CHAs to refer patients to the nearest health facility. Facilities can provide injections, but often at a cost, and male condoms are carried by CHAs and may be stocked out at health facilities as frequently as other drugs. The MOH is currently piloting the community-based distribution of Sayana Press, a contraceptive injection that could be administered by CHAs, in counties outside of PACS areas. Both CHAs and CHSSs reported significant payment delays that demotivated them, constrained their ability to take care of themselves and their families, and was a significant contributor to attrition. PACS provided CHAs mobile phones as part of their training package to ease the efficiency of incentive payments using Mobile Money transfers, but there were challenges with payment timeliness. 17 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV There are a range of reasons for the delays in payments, most of which were outside of PACS direct control, such as recipients providing incorrect transaction or account information for their Mobile Money accounts. In other instances, at the request of the CHO, PACS made payments only after the CHO or other CHT member signed off on the reports, or followed a performance-based payment model to be review by the CHT, creating delays when the CHO or CHT members were out of office. Errors in CHA reporting are frequent, and data collection and report completion require continued monitoring. During quarterly DQAs, PACS found similar patterns of data discrepancies that linked back to CHAs’ human error and differing levels of understanding of the indicators and worked with CHTs on identifying these errors through in-depth data reviews. CHT members noted that the CHA training “needs to be reinforced through targeted mentoring” because it causes “serious problems with their recordkeeping.” There is also a concern about the level of detail included in reports, for example around gender disaggregated data. In addition, CHAs cited confidentiality concerns with reporting on community members with HIV and TB. It is worth noting that while the NCHAP specifies a 10:1 ratio of CHA to CHSS, it was not possible for PACS to address the intervention specifications exactly due to the financial constraints of the Activity. As a result, there are some CHSS who supervise more than 20 CHAs. This created a larger challenge for monitoring and supervision, as well as the ability to perform quality data checks moving forward Limitations: Availability of Drugs and Commodities It was not within the mandate or scope of PACS to provide essential drugs or medicines or include supply chain management as part of its programming, as this was a direct responsibility of the Supply Chain Management Unit of the MOH. However, drug non-availability hindered the ability of CHAs to provide community-based services and affects PACS treatment indicators as noted in Table 4. CHAs should have access to select drugs in providing direct treatment in the community (e.g., paracetamol, malaria, ORS), but this supply is not guaranteed, and the CHAs – and facilities – often did not have drugs available for the population they serve, a finding that the PACS mid-term evaluation also presented. In the revised program description, the renewal of PACS commitment to paying CHA and CHSS incentives was contingent on “the availability of funds and sufficient progress being made on drug supply by the MOH," but PACS continued to make incentive payments to prevent interruption of services. In addition, several respondents added that the public drug supply chain also served private pharmacies and clinics, which typically did not experience stock-outs while GOL health facilities frequently had low or no stock, or only drugs close to their expiry. The CHSS carried drugs directly from the facility to the CHAs during monthly supervisory meetings. CHAs keep a “drug box” with them to carry essential medicines, but aside from this, they lack a temperature controlled, safe storage area for the drugs. Each clinic associated with PACS communities should have a drug supply both for the clinic (80 percent of all medicines per NCHAP policy) and for the community (20 percent of all medicines per NCHAP policy), but often when the facility supply is low, facilities take drugs from the CHA supply, leaving the CHAs with no drugs. OICs, CHSSs, and CHAs noted that when there are no drugs available in the community, the patient volume at facilities rises due to simple, yet untreated ailments at the community level that become severe enough to force community members to travel to the clinic for treatment. Facilities’ staff became accustomed to referring patients to private clinics or pharmacies for medicine due to regular stock-outs of often six months or more. The CBIS (which includes data from CHA reports) does not currently include consumption data, which can identify what is being used, what is replaced, and specific trends in demand and stock-outs, also informing county and national level forecasting for drug supply needs. Constraints due to nation and county-wide drug shortages threaten the efficacy of the CHA intervention. The lack of essential drugs and commodities directly impacts the ability of CHAs and the facilities they advocate for to effectively deliver services, and severely impacts health-seeking behaviors. USAID.GOV USAID/LIBERIA PACS FINAL EVALUATION REPORT 18 Patients are less willing to seek out services to which they are referred because facilities are continually out of drugs, and transport to the clinic is costly and time-consuming, especially when treatment is unavailable. The opportunity cost is a gamble for community members who may return home empty￾handed. In addition to drugs, facilities often ask or expect community members to pay for materials like sterile gloves, bags for medicine, candles, matches, and stationery. Participants in all community FGDs stated that the government clinics are “just like private” now because “you have to pay for everything.” These limitations substantially affect service demand and health-seeking behaviors at the community level. The Referral System and Service Delivery The referral process is part of PACS’mandate in overseeing an effective community health service delivery structure, included in the CHA training and reinforced by CHSSs.Referral of patients from the community to health facility is also a sub-outcome in the theory of change and an output indicator tracked in PACS’ ALMEP. PACS technical field officers are responsible for tracking data gathered on the referral system and identifying any challenges with the process during monitoring visits. For both children and adults, CHAs cannot diagnose or treat persistent, severe,or complicated symptoms because they are not trained on that level of clinical care and do not have the equipment necessary to run diagnostic tests. In these instances, CHAs are trained to refer patients to the nearest facility based on location/distance of clinic, clinic reputation, and patient preference. The counter-referral from the facility informs the CHA of the diagnosis, treatment and the follow-up care, and any follow-up needed by the CHSS. IMPLEMENTATION OF THE REFERRAL SYSTEM FOR SERVICE DELIVERY: PACS exceeded its target (55,200) for referrals by 40 percent, with CHAs making over 77,384 referrals at the end of Year 4. Referrals are an indication that CHAs are assessing community health concerns but also reflect the high number of patients who cannot be treated in the community due to lack of drugs or severity of symptoms, so more referrals are not necessarily a positive outcome. The data was not available to ascertain the root cause behind the referral output numbers. A complete referral system includes not only referrals from community to facility, but also counter-referral from the facility back to the community, with instruction on continuation of care for provision of home support. While PACS tracked the number of referrals to the facility, it did not track counter-referrals received, and this was not a specified priority in the ALMEP. It was reported that instances where patients forgot to bring the counter-referral form home, or the facility staff did not complete it, happened often. Lack of counter-referral created challenges for the CHA who could not track the information provided to the patient from the facility. This also meant that the CHA (or even the patient) might not know the accurate diagnosis to assist with administration of medication or to protect themselves in case of a contagious illness. In instances where a facility provided no information, the CHSS attempted to follow up with the facility on expected patient care. EQ 1.3. INCREASED COMMUNITY DEMAND FOR HEALTHY BEHAVIORS AND HEALTH SERVICES (CHILD HEALTH, FAMILY PLANNING, MATERNAL HEALTH, NUTRITION AND WASH) Establishment of Community Structures to Support Healthy Behaviors PACS worked with communities that did not have a CHC or HFDC to establish one, and to reactivate these structures for communities where the CHC or HFDC had ceased to function, creating 1,975 structures. CHCs and HFDCs are supported by chiefs and other community leaders, and these structures help create a foundation to enforce community level behavior change and increase the demand for services. The CHCs promote ownership and identify community roles and responsibilities around health promotion, and HFDCs foster relationships with the clinics to have community representation at facility meetings. CHCs and HFDCs provide key support for sustaining behavior changes in the communities, and work with CHAs and CHSSs, town chiefs, and other respected community leaders. 19 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV In some communities, CHCs assumed the role of Community WASH Committees (CWCs) or Community Water and Sanitation Committees (CWSCs) after PACS reduced its involvement when they were declared ODF; CHCs play an integral role in monitoring the ODF communities to ensure they do not regress back to open defecation. Where CWCs/CWSCs still exist, communities do not perceive them as formal committees with a clear role, but instead as an occasional ad hoc group that tends to community water/sanitation needs,such as identifying repair needs for handpumps. CHCs and HFDCs work without compensation, and PACS and GOL provide them ongoing support in the form of monitoring,supervision and meals during meetings.Committee members reported frustration with the long distance and transport costs they incurred to attend meetings without reimbursement, while others stated that increased community expectations warranted their receipt of incentives similar to CHAs. Nevertheless, CHC and HFDC members claimed to be committed to the work regardless of pay. CHTs and CSOs believe that the lack of incentives poses a potential threat to the CHCs continued involvement in promoting healthy behaviors. Community Outreach Activities to Promote Healthy Behaviors and Services PACS worked with the MOH to draft the SBCC Strategic Plan and Communication Policy. PACS and the MOH streamlined thepolicy guidelines into the BCC component of its CHA training, which covered a range of topic areas including health-seeking behavior (go to clinic when sick, follow the immunization schedule for children, seek nutritious foods for a well-balanced diet), sanitation and hygiene (wash hands, use the latrine), and household healthy practices (sleep under a mosquito net, keep the environment clean). There are specific BCC messages targeting pregnant women and their caregivers, including the importance of regular check-ups, delivering safely at a health facility, and breastfeeding. There are also general messages targeting caregivers of children that reinforce timely immunizations, hand washing, nutrition, and disease prevention. There were no BCC messages targeted to specific populations like men, adolescents, or vulnerable populations (e.g., persons with disabilities). PACS provided specific health promotion messages through radio shows, community health talks within the community and at markets and advertising. The messages were on printed, colorful pictures that CHAs carried with them on household visits and shared widely at market outreach events. Some CHAs had videos of the messages, which they played to community members on their mobile phones. PACS conducted two Knowledge, Attitude and Practice (KAP) assessments (one questionnaire provided at baseline and one at midline) to capture information about community health behaviors. One flaw in the design of the KAP was that it included populations from both PACS and non-PACS communities across each county, without a means of differentiating PACS communities from others so PACS was unable to attribute broader health improvements at midline to its own interventions. For the final assessment (expected in June 2019), PACS aims to collect data from both PACS and non-PACS sites of implementation, but plans to disaggregate between the two to assess PACS key indicator performance, rather than community performance overall. OUTCOMES OF COMMUNITY OUTREACH ACTIVITIES AND STRUCTURES: While PACS staff believe radio advertising to be more effectivebecause messaging can adapt to different dialects, staff emphasized the need to employ a customized approach for each county’s specific demographics and needs. For example,several CHT members in each of the three counties noted that the vernacular and dialect used in the radio programming and in the market outreach was not appropriate for their constituency, causing the community to take messages less seriously, while other CHT members felt excluded from the decision-making around when and where community SBCC activities occurred. In Lofa, many CHT members felt that their improved capacity in ETL for BCC was one of their greatest achievements and felt confident to continue BCC activities on their own. USAID.GOV USAID/LIBERIA PACS FINAL EVALUATION REPORT 20 There is general agreement that BCC messaging promoted under PACS has increased positive behavioral changes in communities, noting most widely that community members have improved in seeking treatment for both adults and children, keeping the community environment clean, sleeping under mosquito nets and ensuring pregnant women have a safe facility delivery. Anecdotal evidence from communities and CHAs also shows that community members have increased awareness of how community cleanliness prevents malaria and waterborne diseases through initiatives like grass cutting, emptying standing water containers to prevent mosquito breeding, and promoting CLTS. Community members, MOH, and CHAs believe that there is a reduction of cases that need to go to the clinic because of the uptake in healthy home behaviors and the success of seeking treatment for minor ailments from the CHA in the community. Respondents also noted a number of specific behaviors that are more difficult to consistently uphold due to a lack of resources or motivation. Behavior is often driven by social expectations and controls, and not necessarily individual motivation. Many communities expect families to pay a fine if pregnant women do not deliver in a facility. Local authorities also impose fines to enforce environmental and community cleanliness behaviors, such as drying clothes or dishes on the ground instead of on a clothesline or drying rack. Some community members noted that the threat of fines was “just for show,” and that no one was really expected to pay the full amount or expect to face substantial repercussions as a result of poor behavior. Paying the fine shows respect to the town chief. Legal or financial pressure can enforce changes in behavior, and as representatives from the MPW stated, “there must be a legal backing…citizen laws must be upheld to back up behavior change.” For example, the Public Health Law passed in 1957 stipulates creating community structures for cleanliness and proper guidance for latrine construction, but MPW representatives noted that this law is not often known or enforced at the community level. Behavioral messages themselves are not always applicable for the community. For example, messages focused on using latrines and drinking clean water from the well is not possible in communities where these WASH structures are in disrepair. Some community members simply “prefer” open defecation because they are accustomed to it, and several communities visited were completely unfamiliar with the concept of ODF and term when asked. Additionally, the unavailability of essential drugs significantly affects the health-seeking behavior of community members. There are a number of health messages targeted to prevention or treatment of illness that are much more difficult for community members to be motivated by or follow if they know drugs are unavailable. As noted under EQ 1.2, the widespread drug shortage undermines behavior change efforts to seek care. Both community members and facility staff reported that because of the health promotion work of the CHAs under PACS, clinics are less crowded, and they are receiving fewer emergency cases. However, there are no feedback mechanisms at the facility level to capture the number of referred cases from the CHAs in the community, or to determine attribution of a reduced facility patient load to the work of the CHAs or to broader improvements in health behaviors. Some CHT and CSO members felt clinics were less crowded solely due to iCCM, and that while areas like BCC and WASH were integral to community level service delivery, CHAs placed more importance on iCCM over BCC. As one CHT member in Lofa noted, “When the drugs don’t come, they sit down. They have BCC work to do, but they let it lapse.” The ALMEP tracks two indicators with corresponding targets solely for community sanitation and hygiene improvements stemming from BCC activities, tracked from CHA self-reported data (see Table 5 below). PACS achieved just over half of its target reach for household level water treatment and handwashing but did not track or measure other community health behaviors. BCC outreach activitiessuch asmarket days and radio programming also did not produce any documentation of improved knowledge or behaviors directly resulting from these events. 21 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV TABLE 5: ACHIEVEMENTS IN COMMUNITY SANITATION AND HYGIENE BEHAVIOR INDICATOR ACTUAL TARGET EOP % ACHIEVED Percentage of households in target areas practicing correct use of recommended household water treatment technologies (F) 13 percent 25 percent 52 percent Percentage of households with soap and water at a hand￾washing station commonly used by family members 24 percent 45 percent 53 percent * Note: Figures received as of July 1, 2019. (F) denotes those standard foreign assistance indicators required by USAID. There were no real or perceived challenges with accessing soap for hand washing. Community members were confident in their knowledge of disease prevention by handwashing;however,the ETdid not observe fully equipped handwashing stations in any community or health facility visited. In most locations, there was no soap or sanitizer in proximity to the latrine. Some households kept the soap inside the home. Healthy Behaviors and Services: Promotion of Waterguard to Prevent Waterborne Disease One of the key tenets of the PACS BCC approach was in the sale and marketing of Waterguard, a household water treatment agent sold by various retailers and CHAs, who also directly distributed to community members to prevent waterborne and diarrheal disease and encourage good water treatment practices. There are currently two wholesalers in each district where PACS implemented. PACS distributed Waterguard to CHAs at cost to sell the product directly to community members and relieve them from having to buy it retail. However,some CHAs said it was unethical to be selling products to the community when their purpose was to provide free drugs and services. Some community members would not purchase Waterguard from them, and it damaged the trust between some communities and CHA and left the CHAs with a product for which they had already paid. Further, CHAs reported that PACS had often simultaneously marketed Waterguard in their communities, using “buy one, get one free” offers and/or distributing free merchandise, such as t-shirts and bags. CHAs believed those marketing promotions undermined their sales. As seen in Table 5, PACS was just over halfway towards meeting its target for households practicing water treatment, with 13 percent of PACS communities reporting Waterguard usage. In the sample of communities visited, no respondents reported regularly using Waterguard, even those with limited or no access to clean water and using untreated creeks as their main source of water. In several FGDs, participants wore Waterguard bracelets or t-shirts – relics of strong marketing campaigns – but when asked about the product, they denied currently usingit. CHAs and community members noted deterrents in the use of Waterguard:respondents found the scent of the product off-putting, and if used in incorrect doses, it affected the flavor of the water or led to reports of community illness, leading CHAs and PACS to conduct information sessions to re-inform community members about correct usage. Community members knew where to purchase Waterguard and how to use it, but the main deterrent to its use was the steadily increasing price. PACS exercised no control over the cost imposed by retailers, or the fluctuating value of the Liberian dollar, and some community members stated that the price had nearly doubled since the previous year. PACS did not maintain any consistent tracking of Waterguard sales outside of number of cases sold to retail vendors and location, nor has PACS worked with communities to modify its social marketing to respond to low acceptance of the product outside of providing additional information to communities on correct usage. The KAP midline report noted, “Despite high levels of knowledge linking untreated drinking water to illness, adequate water treatment practices remain low,” and this finding was substantiated during the evaluation. Despite the ALMEP tracking numbers of households reporting usage of a water treatment USAID.GOV USAID/LIBERIA PACS FINAL EVALUATION REPORT 22 technology, neither PACS, MOH, nor the CHTs assessed any differences in specific behaviors among community members based on use of the product, or to measure if the demand and use of Waterguard in communities with non-functional water infrastructure had changed over time. EQ 1.4A. IMPROVED ACCESS TO AND MANAGEMENT OF SAFE WATER SUPPLY AND SANITATION, WITH ATTENTION TO GENDER ROLES, AT THE COMMUNITY LEVEL? WASH Activities and Structures in PACS-supported Districts In the three counties, PACS constructed 90 and rehabilitated 66 hand-dug wells with hand pumps (with 45 additional wells under construction at the time of the evaluation), and assisted 1,490 communities to begin to trigger community-led total sanitation (CLTS), including support for community-driven latrine construction with a goal of becoming ODF, in addition to initiatives on community hygiene, such as grass clearing, clotheslines and dish￾drying racks. In those communities, PACS trained 2,507 natural leaders (individuals residing in communities who advocated an end to open defecation and led CLTS in their communities), as well as 280 WASH Entrepreneurs on handpump, well, and latrine maintenance and repair and 493 CWSCs/CWCs on community WASH and CLTS.PACS also engaged Wash Entrepreneurs in soap making and commode construction for sale to community members and product marketing activities to raise awareness on the services and products available. PACS engaged with county WASH officers and Environmental Health Technicians (EHTs) for joint site visits, but several respondents from PACS and the county WASH teams noted these visits, including site selection for wells, latrines and CLTS, took place after activities had already started; several WASH officers within counties noted that PACS often engaged them only at the beginning of an activity, or informed them of its work only after it had already been initiated or completed. In turn, MPW did not actively engage with CHAs or grasp the role of CHAs in supporting community WASH. At the central MPW level, respondents were aware of CLTS through Global Communities’ work nationally, but were unaware of any linkage with community health programming in the three counties or elsewhere and were unaware that Global Communities was within the PACS consortium. As noted in EQ 1.1, PACS provided salary for two MPW WASH Coordinators for almost two years in addition to providing a vehicle and fuel for monitoring and assisting with coordination meetings among key county-level WASH stakeholders. MPW field technicians oversaw final drilling activities to ensure that PACS followed both GOL and international standards for WASH infrastructure. Additionally, there were often delays in site selection because of inaccuracies in government figures for population, topography, water tables and location. Multiple respondents from PACS, MOH, and the CHTs commented that PACS “prioritized women and children” in WASH interventions. Indeed, PACS program description states that “females and children are the center piece of all WASH project designs.” However, none of the respondents could elaborate on what priority meant in practical terms of PACS’ WASH programming, and none of the stakeholders could describe how WASH was affected or influenced by gender dynamics, including changes in workload, health and safety for community women. Out of those PACS trained, 35 percent of natural leaders and WASH Entrepreneurs were women, and 68 percent of CWC members were women, a significantly more balanced proportion than with CHAs. One representative from the MPW noted that there were relative successes in recruiting women as WASH Entrepreneurs (even with initial pushback from male spouses),since women view the activity as increased potential for income generation, and also that women were consulted, albeit inconsistently, to recommend access to water sources since they were primary users. Universally, respondents agreed that the more favorable gender balance among WASH Entrepreneurs compared to that among CHAs could be attributed to the absence of a literacy requirement for Entrepreneurs. 23 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV OUTCOMES FOR TARGETED WASH ACTIVITIES AND STRUCTURES IN PACS￾SUPPORTED DISTRICTS: PACS nearly achieved its targets for access to safe water and sanitation in the numbers of hand-dug wells and community-built latrines, and exceeded its targets for constructing and rehabilitating water points. These considerable successes with WASH activities were echoed among all stakeholder groups. However, it was noted that these outputs are at risk for poor sustainability due to a number of factors related to a lack of continued CHT engagement and oversight by CSOs with a limited geographic reach. The disappearance of active CWSCs in many communities underlies this risk. Without an active cash box system, communities do not have funds for maintenance and repair of the hand pumps or the communal latrines. Unfortunately, financial issues prohibited many entities from proactively addressing community WASH concerns, despite the training and monitoring support received from PACS. WASH Entrepreneurs reported low demand for their services and attributed it to cost associated with repairs, lack of community organization around repairs and community members not knowing that Entrepreneurs could provide a resource. Only one of the community FGDs noted familiarity with Wash Entrepreneurs and knew how to contact them for services. Further, the absence of a functioning CWSC was thought to contribute to relapse to open defecation and the rapid decay of locally constructed latrines, hand-washing stations, and community hygiene infrastructure.CWSCs had previously managed a cash box system that collected and saved community funds for WASH servicing and repairs but were largely inactive at the time of the evaluation, namely if there was a repair needed that community resources could not fund. Community respondents said that after they achieved key milestones – either a functioning well or ODF certification – PACS no longer engaged them, and they did not receive regular follow-up from government entities like the MOH or MPW. At the time of the evaluation, natural leaders were uncertain how long their role as volunteers would continue through the CHT. This disengagement from CLTS communities is further seen in the absence of any post-ODF monitoring data by PACS or the CHTs, preventing insight as to the causes of relapse and sustainability. TABLE 6: OUTCOMES AND ACHIEVEMENTS FOR WASH SERVICES AND PRACTICES INDICATOR ACTUAL TARGET EOP % ACHIEVED Number of communities certified as open defecation free (ODF) as a result of USG assistance (F) 1,061 1,076 99 percent Number of people gaining access to a basic sanitation service as a result of USG assistance (F) 442,609 630,000 70 percent Number of people gaining access to basic drinking water services as a result of USG- assistance (F) 33,069 33,750 98 percent Number of trained WASH entrepreneurs selling WASH products and services in target communities 280 (207m/73f) 280 100 percent # water points constructed or rehabilitated in target communities 156 135 115 percent * Note: Figures received as of July 1, 2019. Targets as included in Year 5 ALMEP. (F) denotes those standard foreign assistance indicators required by USAID. Table 6 shows that PACS has nearly achieved its target for ODF-certified communities, and a number of stakeholders within MPW, CHT, CSOs and PACS agreed that monitoring communities to support ODF practices and prevent relapse should be a key priority in USAID.GOV USAID/LIBERIA PACS FINAL EVALUATION REPORT 24 promoting healthy WASH behaviors moving forward. As both focus groups with CHAs and community members noted, even communities that were certified still admitted to having members who openly defecate. ODF status is directly related to behavioral change, and there are a number of reasons why community members may have relapsed to open defecation. Community members said that visitors or migrants came from neighboring communities that were not ODF; while communities should have signs declaring their town as ODF, these signs were often in disrepair, hard to see, or had not yet arrived, so routine community visitors did not know the community bylaws regarding OD. Others said that ODF, like other behavior change, “is a process” and cannot be sustained after a few months of practice without support. Many people simply preferred open defecation “in the breeze,” while others firmly believed that digging a hole and covering the feces with soil after defecation was sufficient. Many community members, CHAs, and CHT members noted that relapse to open defecation was a byproduct of the lack of functional latrines, or latrines being in constant disrepair without the ability to fund repairs. Community members noted that some people in their community reverted to OD rather than using a latrine that was crumbling or did not meet communities' expectations of health or safety. During planting and harvest seasons, residents spent all or a substantial part of their time away from home on their farms, where it is not considered feasible to construct new latrines so holes were dug for defecation during those periods. A final reason noted by several CHT and CSO respondents was that the lack of community adherence was due to the association of ODF with NGOs such as PACS. As a CHT member stated, “people feel like ODF is a project from an NGO, so it doesn’t trigger community ownership.” Several others agreed that community members often follow NGO ‘directives’ in order to create opportunities to receive media attention or visits from the Mayor and NGOs and that because “even the ODF certified sign is covered in the NGO branding,” their motivation to be ODF is driven by NGO interest, not genuine concern for hygienic behavior. CHAs, CHT representatives and community members note that communities that maintained ODF status typically supported positive behaviors by imposing community fines and public accountability to punish reversion to unhealthy practices like open defecation. Many communities relied on traditional law enforcement by town chiefs and active CHCs that had taken on the work of CWSCs in community monitoring and ensuring ODF status and other community hygiene practices. The participation of the chief in the CHC as a member or ally conferred traditional respect for the chief and the authority to impose fines,reinforcing the WASH behavior change processes. Respondents from CHCs, CHAs, and lay community groups related that the CLTS process contributed substantially to community attention to health. In particular, communities that had maintained ODF status also reported more active CHCs and a wider range of BCC results, including midwives’ supporting facility-based deliveries and mothers’ reporting positive interactions with CHAs regarding treatments for children. Together, those changes in sanitation, hygiene and community health appeared to be linked to active CHCs along with community empowerment from CLTS. EQ 1.4B. WHAT IS THE COST OF THE ACCESS – AS DEFINED BY THE ADMINISTRATION COST OF PACS PROVIDING ACCESS TO CLTS WATER AND SANITATION SERVICES PER BENEFICIARY – AS COMPARED TO THE ADMINISTRATION COST UNDER AN INTERNATIONAL NGO AS WAS PREVIOUSLY USED? The evaluation examined CLTS implementation costs between PACS and two international peer organizations: UNICEF and Living Water International (LWI), an international NGO working on WASH in Liberia. The evaluation further included costing data for the five CSOs supported by PACS for local comparison. The ET considered costs as reported by each organization, as were associated results of implementation (numbers of communities certified as ODF, engaged with hygiene promotion and estimated number of beneficiaries). 25 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV The ET standardized the cost data to facilitate comparisons by annualizing the data. Specifically, the ET compared the single-year cost data that UNICEF, LWI and the CSOs provided to the five-year average of PACS’ cost data. Similarly, the ET compared cost data for CLTS and community hygiene work separately and combined for each organization, primarily because each organization used different methods for allocating personnel and travel costs to those WASH sub-activities and also because programmatically, IPs typically implement sanitation and hygiene interventions together. Some discrepancies remained, as the UNICEF data may omit costs by its sub-partners that execute some of the CLTS community work; and PACS-supported CSOs as a group showed high variability in their costs. Furthermore, UNICEF and LWI reported some level of post-ODF monitoring of communities, which PACS did not. Cost could not be disaggregated from the total; thus, a qualitative difference remains. On balance, the cost comparison is indicative of each IP’s approach. Any further refinement in precision is not considered likely to alter the accuracy or rank order in the comparisons, and instead would likely show larger differences between international IPs and local CSOs. Reported costs for CLTS and community hygiene by PACS and its two international peer comparators are shown in Table 7. The table presents data as the total annualized costs reported by each organization, as well as costs per key result, community or beneficiary, based on data provided by the organizations. The table includes average total and unit costs for the international organizations and for the local CSOs. TABLE 7: COMPARISON OF CLTS AND HYGIENE PROMOTION COSTS (USD) FOR PACS AND INTERNATIONAL PEER COMPARATORS IMPLEMENTING WASH IN LIBERIA PACS* UNICEF LWI AVERAGE Annual cost: CLTS $785,890 $76,620 $98,044 $320,185 Cost per ODF-certified community $3,957 $1,095 $2,391 $2,481 Annual cost: Hygiene promotion $152,729 $9,500 $199,872 $120,700 Cost per community site $513 $50 $2,562 $1,042 Combined cost: CLTS & Hygiene Cost per community site $4,470 $1,145 $4,954 $3,523 Cost per beneficiary (estimated) $6.30 $3.18 $8.96 $6.15 Key results (annualized) # newly ODF-certified communities 199 70 41 103 # communities engaged (hygiene) 298 190 78 189 # beneficiaries (estimated) 149,000 47,500 33,260 76,587 * Notes: Costs and results data for PACS are based on the five-year average for the activity. Table 7 shows that PACS dedicated a larger budget for CLTS and hygiene promotion, as a whole and per ODF-certified community. PACS’ costs for hygiene promotion wererelatively more moderate. Combining the CLTS and hygiene promotion costing data, PACS and LWI reported higher costs per community and per beneficiary than UNICEF, though the UNICEF data may not include some costs from its sub-partners engaged in CLTS and community hygiene; if so, UNICEF’s costs would be higher. Combined CLTS and hygiene implementation costs per beneficiary ranged from $3.18 to $8.96 USD, with PACS in the middle of that range. Notably, PACS also reported a higher scale of implementation, with the numbers of communities and estimated beneficiaries exceeding the combined total of its international comparators. Averaging the international CLTS IPs, the costs per community were US$2,481 for CLTS and US$1,042 for hygiene promotion. Similar CLTS and community hygiene costsfor the PACS-supported CSOs are provided in Table 8, below, along with the average for the five local CSOs. USAID.GOV USAID/LIBERIA PACS FINAL EVALUATION REPORT 26 TABLE 8: COMPARISON OF CLTS AND HYGIENE PROMOTION COSTS (USD) FOR PACS￾SUPPORTED CSOS CSI SEARCH RUCEP DEN-L EQUIP AVERA GE Annual cost: CLTS $8,786 $78,000 $55,000 $132,000 $19,730 $58,703 Cost per ODF-certified community $204 $1,560 $458 $1,760 $1,973 $1,191 Annual cost: Hygiene promotion $5,924 $43,960 $56,100 $25,000 $31,680 $32,533 Cost per community site $61 $586 $442 $167 $3,168 $885 Combined cost: CLTS & Hygiene Cost per community site $265 $2,146 $900 $1,927 $5,141 $2,076 Cost per beneficiary (estimated) $7.58 $3.25 $1.91 $4.19 $4.85 $4.36 Key results (annualized) # newly ODF-certified communities 43 50 120 75 10 59.6 # communities engaged (hygiene) 97 75 127 150 10 91.8 # beneficiaries (estimated) 1,940 37,500 58,071 37,500 10,590 29,120 The CSOs supported by PACS also reported their CLTS and hygiene promotion costs, along with associated results. Both cost and results data were more variable across the CSOs, in some instances raising questions about the quality of data. In particular, the estimated number of beneficiaries are atypically low for two CSOs and high for one CSO, and the costs varied considerably without clear linkage to scale of results. Because the cost allocations between CLTS and hygiene promotion varied appreciably across the CSOs, the average cost data is of greater value to this analysis: US$1,191 per ODF-certified community and US$885 per community for hygiene promotion. A comparison of the average CLTS and hygiene costing data for international organizations and local CSOs provides additional information. PACS-supported CSOs reported lower costs per community for both CLTS (52 percentlower) and hygiene promotion (15 percentlower), against the costs presented by PACS and its international peers. Combining CLTS and hygiene promotion, the average CSO costs were 41 percent lower on a per-community basis and 29 percent lower on a per-beneficiary basis compared to the average of the international comparators. The analysis is likely to have underestimated this difference, given the under-reporting of cost data by UNICEF and the under-reporting of beneficiary data by CSOs – both errors reduce rather than exaggerate the per unit cost differences between international and local organizations working in CLTS and hygiene.On the other hand, the comparison assumes an equal standard of quality of implementation of CLTS, hygiene promotion and durability of infrastructures, which the ET did not assess in this evaluation (but would require MOH, MPW and community input). Additional questions remain regarding the success rate (adherence to ODF practices versus relapse to OD) among communities triggered and guided by the CSOs, two years or more after ODF status (declared or certified) and how that rate compares to rates by international NGOs implementing CLTS. SUMMATIVE FINDINGS: PACS PERFORMANCE USING AN INTEGRATED APPROACH While PACS has largely achieved fidelity of implementation within each of its four result areas, it did so without integrating its components of iCCM (through CHAs) and WASH. The PACS project description notes that the Activity intends to “promote linkages between health and WASH” and the annual implementation plans following the mid-term evaluation findings stated PACS’ intent to better integrate activities across its four result areas and among partners (namely for Year 4 and 5). However, PACS’ presentation as an integrated approach to community health and WASH overstates the reality of a largely parallel, rarely coordinated set of activities, and multiple stakeholders expressed that the community health and WASH activities operated in ‘siloes.’ 27 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV PACS shared both BCC/iCCMand WASH/CLTSmessaging universally across all districts,but the number of communities receiving intentional integrated interventions with both CHAs and infrastructure development was limited. While half of the PACS districts in Bong (20 communities) and Lofa (39 communities), and one-third of the Nimba districts (a total of 26 communities) had communities that received integrated iCCM/WASH interventions, the actual number of communities PACS reached in those districts was proportionally inadequate. PACS’ internal monitoring data shows that out of the 1,961 total communities reached, 75.4 percent of communities received WASH/CLTS interventions, 20.2 percent received iCCM and only 4.3 percent of PACS communities received both iCCM and WASH/CLTS interventions. Further, as seen in Figure 3, 35 percent of the districts reached with WASH only interventions (referred to as CLTS on the map) were in districts where Plan International was providing support to CHAs for iCCM, preventing any integration of activities between the two IPs and preventing PACS from determining attribution of positive health outcomes to their interventions alone. As one PACS respondent noted, the integration model “was not a deliberate design.” For half of its duration, PACS field staff worked from separate partner offices in each county; the PACS agreement with USAID did not stipulate that the consortium was required to work from a common office or with a shared identity, and therefore, they did not do so. Thus, different consortium partners with separate workplans, separate budgets and separate reporting mechanisms implemented PACS interventions. The head office staff in Monrovia never worked from the same office, and PACS staff often identified themselves first through their home organization, and secondly as working for PACS; this finding was confirmed with feedback from some stakeholders who referred to Global Communities or PSI individually and did not recognize these organizations as consortium partners. PACS did prepare an ALMEP and theory of change that encompassed the work of its partners for over three years into the Activity, and PACS staff over time began to share information about their respective work and identify some areas of coordination. OUTCOMES OF THE INTEGRATED APPROACH: All of these factors hindered joint planning and a more intensive means of reinforcing the results under PACS which numerous respondents felt had too wide a breadth and not enough depth. Multiple stakeholders including PACS staff themselves commented that a jointly focused effort to target the full set of interventions (WASH, iCCM and BCC) within all communities in one county and progressively expand outward rather than implement some activities only in some districts would have provided more “quality over quantity.” This more targeted FIGURE 2: PACS AREAS OF INTERVENTION AND THE INTEGRATED APPROACH USAID.GOV USAID/LIBERIA PACS FINAL EVALUATION REPORT 28 and comprehensive approach would have also ensured adequate supervision from the limited PACS field staff, who oversaw large geographic areas. PACS staff had limited opportunities to share skills and develop internal capacities that would complement their work and ensure effective linkages between community health and WASH in all counties. Stakeholders interviewed agreed that an integrated design will have more positive effects than a single￾track intervention, and that WASH, BCC, and iCCM would be most effective when addressed together, especially relating to service delivery and creating demand for infrastructure and good health behaviors. Stakeholders also believed that stronger linking of appropriate BCC and WASH messaging within communities would lead to more sustainable outcomes. However, PACS did not collect any outcome measurements to capture or validate these claims, or any differences in districts receiving integrated services with those who did not. V. CONCLUSIONS EQ 1. PERFORMANCE: FOR KEY OBJECTIVE AREAS LISTED BELOW, TO WHAT EXTENT DID PACS IMPLEMENT WITH FIDELITY (I.E., DID THEY DO WHAT THEY SAID THEY WERE GOING TO DO) AND TO WHAT EXTENT DO THE AVAILABLE DATA SUGGEST IMPROVED OUTCOMES AS A RESULT OF IMPLEMENTATION IN BONG, LOFA AND NIMBA COUNTIES, WITH RESULTS DISAGGREGATED BY GENDER? Conclusions for the Performance EQs consolidate the evidence related to fidelity of implementation and outcomes for each sub-question. EQ 1.1. BUILD CAPACITY (MANAGEMENT SYSTEMS, KNOWLEDGE TRANSFER, AWARENESS OF AND ATTENTION TO GENDER AND EQUITY FACTORS, AND POLITICAL WILL) OF MOH, COUNTY HEALTH TEAMS, CSOS AND COMMUNITIES FOR THE COMMUNITY HEALTH ASSISTANT PROGRAM? The capacity development support PACS provided included training, skill building, process management, material provision, and logistic support. MOH and CHTs placed more emphasis on the continued reliance on supplies and logistical support, and less emphasis on the impact of skill development in areas like M&E and supervision. However, respondents consistently noted these as areas of weakness in which capacity remains underdeveloped. “Capacity” and “support” have become synonymous with material provisions and less about necessary skill development, except in the cases of CSOs, whose skill building helped identify organizational strengths and gaps and created opportunities to better engage with communities. The PACS design was flexible in responding to evolving country priorities and provided substantial support to development of key MOH policies for community health and health communication. However, Activity adaptations did not always include key county-level stakeholders, and collaboration between PACS, CSOs, and counties, was often tenuous. This resulted in some stakeholders feeling ignored or underutilized, rather than stakeholders capitalizing on one another’s new skill sets. CHAs and CHVs played a key role in promoting health messages and education, and PACS equipped CHTs to continue this training. CHAs were trained in the areas under the CHSPSP, but require further, ongoing training and continued incentivization to secure their continued work. There was a missed opportunity for PACS to capture data to quantify and measure capacity development outcomes for MOH, CHTs, CSOs, or CHAs. Because of PACS’ intent and mandate to operationalize the MOH policies, community members, CHAs and MOH respondents seemed to critically misunderstand what activities and directives were PACS’ initiatives and responsibility and what was the responsibility of the GOL, which confused expectations and led to misplaced frustrations. EQ 1.1B. BUILD CAPACITY OF THE MOH AND MPW TO SUSTAIN IMPLEMENTATION OF THE CHA AND WASH PROGRAM AND DEMAND CREATION ACTIVITIES? 29 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV While the capacity development support PACS provided to the MOH and MPW at both central and county levels emphasized internal system management and promoted essential community health program functions, it did not succeed in overcoming a prevailing culture of dependency in GOL that hindered sustainability of CHA and WASH interventions. GOL dependency was reinforced not only by PACS but also by informal competition among USAID and other donors. The CHA program provided a range of critical community-based health services that stakeholders viewed positively, and they saw the supervision of these community-based health assistants as the overarching achievement of the PACS Activity. However, there is no evidence that the CHTs, MOH and other ministries have sufficiently prepared themselves to assume key functions of the CHA intervention, as outlined in the NCHAP or WASH infrastructure monitoring, after the conclusion of PACS. EQ 1.2. INCREASED THE AVAILABILITY OF SUSTAINABLE AND QUALITY COMMUNITY-BASED HEALTH SERVICES (FOR CHILD HEALTH, FAMILY PLANNING, MATERNAL HEALTH, NUTRITION AND WASH) INCLUDING 2-WAY REFERRALS WITH HEALTH FACILITIES? There was qualitative evidence of service delivery improvements as a result of PACS implementation, and support to CHAsincluding reductions in malaria, diarrheal disease, and home birth deliveries. Community members appreciated the door-to-door service CHAs offered, as it reduced their need to visit a clinic. PACS designed the CHAs link to facilities to rebuild community trust in health services available to them; however, chronic drug unavailability has seriously hampered the CHAs’ ability to treat illnesses at the community level. Further, lack of drugs has impeded facility staff ability to treat patients within facilities, necessitating outside referrals for costly drugs, discouraging community health-seeking behaviors and degrading public trust in the health system. Notably, the public drug supply chain kept private pharmacies in stock, but not public facilities. PACS had the responsibility to pay CHA and CHSS salaries alone and were not responsible to procure drugs or commodity packages including personal protective gear or diagnostic equipment to CHAs. However, there were misunderstandings from CHAs and expectations about what PACS was supposed to be providing, compared to what was the responsibility of the MOH. PACS made inconsistent progress toward meeting targets for service delivery. The two-way referral system operated as a one-way referral, where counter-referrals from the health facility were inconsistent and excluded CHAs from receiving information for follow-up care. PACS and CHTs missed this opportunity to systematically track and report on referrals from CHAs within facilities to better identify trends in care-seeking behaviors, services rendered, or recurring health concerns. Payment delays in CHA/CHSS incentives and CHT transport funds affected field monitoring, which threatened the sustainability of the supervision model and de-motivated CHAs and CHSSs who relied on this income, increasing attrition. Instances of “double-dipping” affect the integrity of the CHSS model and undermined the relationship between PACS and the CHT. EQ 1.3. INCREASED COMMUNITY DEMAND FOR HEALTHY BEHAVIORS AND HEALTH SERVICES (CHILD HEALTH, FAMILY PLANNING, MATERNAL HEALTH, NUTRITION AND WASH)? Community members have shown improvements in positive health behaviors within their homes and environments, but behavior change is substantially influenced by resource availability. In the absence of functioning and accessible wells and latrines, communities struggle to uphold good behaviors while attempting to rebuild or repair structures; there are other instances in which behaviors and intentions are good, but the community still relies on external forces to supply it with resources like latrines and mosquito nets. Further, it is difficult to gauge sincere demand creation and long-term behavioral change, as community members’ actions and improved behaviors around environmental cleanliness and facility deliveries are often the result of fines imposed by town leaders. USAID.GOV USAID/LIBERIA PACS FINAL EVALUATION REPORT 30 The key community-level support for healthy behavior change and demand for formal health services came from CHCs supported by PACS, chiefs, and other community leaders. As volunteers, it is uncertain if CHCs will continue to uphold their roles or take on additional responsibilities. Beyond the forthcoming KAP endline assessment, there is no evidence of consistent or ongoing process measurements of behavioral change that would indicate efficacy of PACS BCC interventions outside of anecdotal evidence from community members and CHAs.A lack of solid PACS-collected data to regularly measure outcomes of BCC initiatives, including their effect on knowledge, attitudes and practices,hampers the ability to gauge real change, identify linkages or monitor the influence of PACS on healthy behaviors and service use. PACS achieved only half of its BCC outcome targets. There was also a missed opportunity for PACS to measure demand creation and community behavioral effects for key interventions,such as the marketing and use of Waterguard. PACS included Waterguard as an intervention with promise to treat water and prevent waterborne illness, but miscommunication about dosage, increasing costs and often unethical direct marketing from PACS undermined CHA sales and prevented its consistent use or adoption in communities. EQ 1.4A. IMPROVED ACCESS TO AND MANAGEMENT OF SAFE WATER SUPPLY AND SANITATION, WITH ATTENTION TO GENDER ROLES, AT THE COMMUNITY LEVEL? PACS nearly met or exceeded its targets for improved access to water and sanitation services and infrastructure. PACS intended the CWSC initiative – steeped within the community – to serve as a sustainable means through which to promote community ownership and agency over identifying WASH needs and working collectively to solve WASH infrastructure problems like necessary repairs. However, there is little evidence that these committees were effective structures in PACS’ final year of implementation. Within the communities implementing WASH, there was no evidence of a uniform strategy for community WASH Entrepreneurs and natural leaders, nor any uniform operation of CWSCs. There is a clear need to monitor ODF communities’reversion back to OD; the rejection of community￾built and owned latrines, or their rapid decay, raised fundamental questions about CLTS. However, PACS lacked a systematic process to follow up with these specific communities to review the structural integrity of hand pumps or community-built latrines, the degree of ownership in WASH decision-making, and the socio-cultural challenges to maintaining ODF status. Instead, CHCs, CSOs, MPW, and some PACS staff have conducted monitoring superficially and on an ad-hoc basis without systematic data collection or support to communities that had relapsed. MOH recognized PACS for its integrated approach to addressing both WASH and BCC/iCCM, as it is the only partner with that mandate. However, the PACS approach was more siloed than integrated and missed the opportunity to provide narrower coverage with a more comprehensive package of services. PACS’ experience did reveal that even modest coordination among community health and WASH had some benefits, which focused integration might achieve with greater scale, efficiency, and – particularly for WASH – sustainability. EQ 1.4B. WHAT IS THE COST OF THE ACCESS – AS DEFINED BY THE ADMINISTRATION COST OF PACS PROVIDING ACCESS TO CLTS WATER AND SANITATION SERVICES PER BENEFICIARY – AS COMPARED TO THE ADMINISTRATION COST UNDER AN INTERNATIONAL NGO AS WAS PREVIOUSLY USED? Based on the comparative costings, PACS is in the middle cost range of international partners implementing CLTS and hygiene promotion. The international partners are more expensive than the PACS-supported CSOs, assuming similar levels of experience and quality. PACS did not clearly show economy of scale, despite implementing CLTS and hygiene across a larger number of community sites. 31 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV On average, the PACS-supported CSOs showed a lower implementation cost, and indirect evidence suggests that these CSOs implemented CLTS in communities that are more difficult to reach than those triggered by international NGOs (PACS and its peers). The level of quality that these CSOs provide in CLTS and hygiene promotion is unknown. The variability in the costs and results reported by the PACS￾supported CSOs suggests a need for PACS to review the findings with the CSOs to ensure high quality financial accounting, M&E, and CLTS implementation models. VI. RECOMMENDATIONS The evaluation structures recommendations according to their intended users, as below. RECOMMENDATIONS FOR MOH (CENTRAL AND CHTS) • IN RELATION TO EQ 1.1A: • In the planned revision of the NCHAP strategy, introduce a focus on gender in order to mitigate the growing gender bias in CHA and CHSS staffing and other aspects of the program, through close involvement of the MOGCSP and an external gender technical expert for gender analysis and mainstreaming and to better understand and address gender in WASH programming. • To address ongoing needs for continuous monitoring and training support, prioritize budget line items for CHSS incentivization and in-service trainings for CHAs, as well as continued quarterly review meetings at both district and county level. • Initiate standardized processes for data quality assurance involving CBIS and other data from the NCHAP at the facility, county and national levels, directed by the MOH HMERD. • In relation to EQ 1.2, ensure regular drugs and commodities to CHAs through a dedicated, prioritized stream within the supply chain that focuses on drugs and commodities for children under five years and pregnant women, as well as family planning commodities, keeping with the equity provisions of the proposed drug-revolving fund (see below). Enact a process tracing activity to analyze the numerous factors affecting drug availability and determine responsible actors necessary to ensure a functioning system. • In relation to EQ 1.4b, further engagement with UNICEF, LWI and other CLTS IPs might provide lessons on post-ODF monitoring, an essential element for sustainability that PACS and the involved CSOs did not incorporate. RECOMMENDATIONS FOR MOGCSP • In relation to EQ 1.1a and EQ 1.4a, support the MOH with the gender technical team in examining and finding solutions to the gender bias in the NCHAP and WASH programming. • In relation to EQ 1.1b, finalize the nearly completed foster care guidelines that were developed with PACS up to 2017, and roll out those guidelines at the county and local levels. RECOMMENDATIONS FOR DEOH (IN MOH) AND MPW • IN RELATION TO EQ 1.4A: • DEOH to lead a review of the CLTS approach with all partners engaged in community sanitation, to assess and respond to the rising issues of communities that relapse from ODF to OD, socio-cultural factors that hinder sustained behavior change in use of toilets, and the sustainability of community-built sanitation. Engage with social anthropologists to ensure a culturally relevant and sustainable approach. • MPW to include community health in the ICWSC mandate, as well as a small budgetary allocation, to support monitoring of ODF communities, and share that data with the CHTs and partners. • Enact a mandatory order for points of contact from MPW to attend MOH meetings (and vice versa) to identify and coordinate activities around WASH infrastructure site selection and CHA collaboration and enable joint goal setting and accountability. USAID.GOV USAID/LIBERIA PACS FINAL EVALUATION REPORT 32 RECOMMENDATIONS FOR PACS AND PACS-SUPPORTED CSOS Recommendations for PACS and the CSOs it supported are focused on supplementing and enhancing the transition and handover processes that had already begun at the time of the final evaluation. • Build on discussions with GOL on transition planning, specifically by mentoring CHT members on priority tasks for community health and WASH (e.g., CHSS supervision, CHA reporting and BCC skills refresher, water quality testing, CLTS site selection and post-ODF monitoring, etc.). • In relation to EQ 1.4a, PACS’ WASH specialists and Partnership unit should collaborate on improving sustainability and community maintenance of ODF status, for instance, by exploring monitoring for ODF communities, documenting lessons about socio-cultural factor that lead to OD relapse, developing remedial support for communities that show signs of relapse to OD, and sharing those methods with the PACS-supported CSOs for continued work after PACS has closed. • CSOs should collaborate closely with CHTs and other county GOL stakeholders (County WASH Officers, ICWSC), with monitoring of ODF communities as an early agenda item. • PACS and CSOs should clarify the gender aspects of WASH in their communities and prioritize local women as partners. • In relation to EQ 1.4b, clarify the variability in the costs and results (or targets) that CSOs reported for CLTS and hygiene promotion as well as differences in their approaches to CLTS and hygiene (compared to other IPs), as part of shared learning for the CSOs and facilitating the handover of community hygiene and sanitation to those organizations. VII. UTILIZATION PLAN The evaluation engaged GOL stakeholders from the MOH and MPW, as well as CHT representatives throughout the preparation, field work, and preliminary analysis phases of the evaluation. These meetings included a briefing on the design of the evaluation, a participatory analysis workshop, and a debrief on the initial findings and recommendations. With this relatively intensive engagement, GOL is well-informed of the evaluation and is positioned to engage further. LSA will facilitate a learning event after submission of the final evaluation report with USAID, PACS, GOL, and other stakeholders,such as UNICEF and other NGOs working in community health and WASH. This will provide an opportunity to share findings and lessons learned, as well as to consider use of the evaluation’s recommendations for relevant decision-making, including policy development and larger sectoral assessment needs. LSAwill facilitate the event with the aim of ensuring that relevant stakeholders understand the evaluation findings and conclusions and are motivated to implement evaluation recommendations. 33 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV VIII. ANNEXES USAID.GOV USAID/LIBERIA PACS FINAL EVALUATION REPORT 34 ANNEX A: FULL LISTING OF REFERENCES AND REPORT UTILIZED S/N Desk Review Documents Source 1 USAID/Global Documents: • Journey to Self-Reliance Overview Fact Sheet • USAID Policy Framework Full Report • USAID Policy Framework Summary • USAID FY 2019 Liberia Country Roadmap USAID 2 GOL Documents: • CLTS Guidelines & Other WASH Supporting docs. • Investment-Plan for Building-a-Resilient Health System • National Health Communication Strategy • National Health Policy 2011-2021 • National Nutrition Policy Revised Final 2019-2024 • Revised Community Health Services Policy 2016- 2021 GOL Ministries 3 Liberia Context: • Liberia Census Report • Malaria Indicator Surveys • 2013 Demographic & Health Survey • Health Sector Performance Report Final Nov. 2016 GOL and other public resources 4 PACS Yearly Reports & Project Document/Description • PACS KAP Studies (Baseline & Mid-term) PACS Consortium 5 SI Studies: • CDCS Stakeholder Consultations • CHT Capacity Assessment SOW • PACS DQA Draft Report-11-08-18 • PACS DQA Final Report-31-10-2016 • PACS Mid-term Evaluation 2017 LSA 6 Sites & Stakeholders List/Matrix PACS 35 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV ANNEX B: FULL LISTING OF PERSONS INTERVIEWED The full listing of persons interviewed was submitted separately in line with data de-identification policies. Please contact Carla Trippe, ctrippe@socialimpact.com, to request the data. USAID.GOV USAID/LIBERIA PACS FINAL EVALUATION REPORT 36 ANNEX C: GEOSPATIAL EQUITY ANALYSIS OF PACS’ CLTS SITES VS OTHER (NON-CLTS) COMMUNITIES To examine PACS’ CLTS site selections, distances to the closest road were calculated for all community settlements in the three counties for which geospatial data were available, 1474 communities in total. A comparison was then conducted for PACS’ CLTS communities against other, non-PACS CLTS communities, based on the proportion of PACS communities for each road type and the average distance between each community and its closest road. Two road types (local and service roads) were dropped because the numbers of communities were five each, below the threshold for stable estimates. Table 9 summarizes the results of this analysis, below. TABLE 9: GEOSPATIAL EQUITY ANALYSIS OF PACS’ CLTS SITES VS OTHER (NON-CLTS) COMMUNITIES (BONG, LOFA AND NIMBA COUNTIES, 2019) Closest Road Type No. of communities Proportion of communities with PACS CLTS Average distance (km) from closest road Percentage difference PACS CLTS Other in distance Primary Road 290 41.0% 1.01 1.44 29.8% Feeder Road 768 31.0% 1.30 1.67 22.1% Residential Road 113 31.9% 0.90 1.16 22.8% Secondary Road 293 23.5% 1.74 1.84 5.7% Total 1464 32.0% 1.25 1.62 23.1% Two patterns emerge in this geospatial analysis relating to the equity of PACS’ CLTS site selection. First, PACS relatively oversampled communities closest to primary roads (41 percent) and under-sampled those closest to secondary roads (23.5 percent), compared to the overall average of 32 percent across all communities. Second, the analysis showed that sites selected for PACS’ CLTS interventions were 23 percent closer to any road (average 1.25 km) than communities not selected by PACS for triggering CLTS (average 1.62 km). This second factor was highest for communities closest to a primary road, with a difference of nearly 30 percent in distance between PACS’ CLTS sites and other, non-CLTS communities. These patterns are easily visualized in a map, as shown in Figure 3 below. PACS CLTS sites, plotted in red color, cluster alongside the primary roads in all three counties. Communities without PACS CLTS, plotted in black color, dominate the areas farther off-roads, particularly in Lofa and western districts of Bong. In southern Nimba, close to Tappita City, non-CLTS communities are apparent both along and away from roads, as PACS did not extend its CLTS to that area. 37 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV FIGURE 3: PACS CLTS SITES IN RELATION TO MAJOR ROADS, BONG, LOFA AND NIMBA COUNTIES (2019) Given that communities closest to secondary roads or farther from any road typically lack resources or score higher on poverty measures (socio-economic data for Bong, Lofa and Nimba communities were not available to the evaluation), this analysis implies that PACS-selected CLTS sites were easier to reach and may have been relatively better resourced than the communities that were farther from main roads, raising questions about the effect of equity on PACS’ results. It is important to note that there is no evidence that PACS or its stakeholders deliberately selected CLTS sites based on proximity to primary roads. Nevertheless, CSO respondents called attention to their perception of this bias in CLTS site selection. They asserted that they were well-placed to initiate CLTS in remote communities because they could travel by motorcycle rather than 4x4 vehicle, and not be obligated to return to town before dusk as PACS staff were. USAID.GOV USAID/LIBERIA PACS FINAL EVALUATION REPORT 38 ANNEX D: DATA COLLECTION INSTRUMENTS Form A - KII: MOH Directors About the Tool This tool will capture the experiences of MOH Directors/senior staff to gather a deeper understanding of the PACS design, implementation, results and lessons learned, stakeholders’ engagement, challenges faced in working with PACS, and their views on effectiveness and sustainability. INTERVIEWER NAME__________________________________ DATE (DD/MM/YY)…………………………….…………………… START TIME………………………………………………………………. H M N END TIME………………………………………………….…………… H M N Introduction AT THE START, inform the participant about the purpose of the interview/discussion, and go through the informed consent process. Keep the signed consent form with you. We are interested in learning about your experience with PACS, including your perceptions, opinions and experiences regarding implementation, and your views on its effectiveness and sustainability. Your frank responses and participation will be most helpful to us to understand your views. Questions Respondent Details Information Name of the Organization Organization code (if any) Location Locality (…………………………………..) Sub-Locality (…………………………………..) Respondent(s) Name(s) Position Title Gender Female Male How many years have you been in this position? 39 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV Code Question Probes/prompts and notes A01 E11 E12 Briefly describe your role, and how you have worked with PACS. What support has been provided to you or your division through PACS? Examples: logistics, human resources, physical resources, and programmatic information? A02 E11 E12 What do you consider as your strongest skills and competencies with your involvement with PACS? How are those skills and competencies applied in practice? Eg. QRMs, Coord. meetings, superv. Meetings - Weekly, monthly, yearly How will you use those skills after PACS ends? A03 E11 What areas of PACS capacity building did not work very well? Why did that happen? Encourage a balanced view on capacity activities. A04 E11 E12 How would you describe or identify the main changes in PACS programming around increasing the availability of community-based health care from the inception of the project to its current state? child health, family planning, maternal health, nutrition and WASH M&E – DQAs? use of data for planning, etc. A05 E11 E12 How did PACS help the MOH contextualize/adapt/operationalize new policy (CHSPSP, NCHSP, NCHAP) and best practices? Could you illustrate with examples? To what extent will these policies continue after PACS ends? Policy, strategies, BCC curriculum, etc. Quality services and assurances strategies How sustainable are these efforts? A06 E11 What considerations went into designing a technical assistance strategy for CSOs? How has your office worked with the CSOs supported by PACS? What system put in place to gauge level of skills and knowledge increase? Level of GOL involvement with CSOs. A07 E11 E12 What are the government’s plans for financing for NCHAP (or CLTS, wells, etc.) following the end of PACS? Probe beyond funding, ask about technical areas, engagement by OICs, DHTs, CHTs with CHAs, etc. A08 E11 E13 Can you please share how the NCHAP operational plan guides PACS activities around health promotion? To what extent did PACS create BCC tools to cover select health areas? Have any tools been operationalized by the MOH, CSO partners or others? What is the feedback on their utility? Behavioral change communication Gender as criteria for CHA selection, but mostly male result (80 percent+) A09 E11 E13 What are the key messages from PACS related to healthy behaviors and health education, and how were these messages delivered? Which messages have community members been the most receptive to? Any that have not been as successful? Why do you think this is? radio, posters, mass media Which stakeholders were critical in initiating messaging and how were these stakeholders identified? Were there any differences by gender, in messages or acceptance by community members? A10 E11 E12 How would you prioritize the key areas of capacity that still need to be built or supported in the next 5 years for the MOH, CHT, CSOs, DHTs to best support community health service delivery? Are there any skill areas that have improved, remained stagnant, or declined? USAID.GOV USAID/LIBERIA PACS FINAL EVALUATION REPORT 40 Code Question Probes/prompts and notes A11 E21 Could you identify specific benchmarks that you believe would be necessary for these GOL entities to monitor? How might this internal accountability system work? Benchmarks can be the following: - Financial - Technical - Ownership - others as suggested by respondent A12 E21 To what extent do you believe that your office will take on the role(s) that PACS has done, or where PACS has built capacity? Why do you think so? Kindly describe the indications of where the gap may be, in their taking on that work? What other things need to happen for your office to take on that role, if anything? Step by step, how could your office move ahead in doing the work after PACS has closed? When do you think your office will be ready to take on that role? What signs or evidence show that your office has the capacity for those roles, or not? What about the commitment to take on that role? Besides funding, what gaps still exist that hinder your office from taking on the roles from PACS? Has some change started already? Now, or will it be when PACS ends, or another year? Or, does it depend on something else, such as policy, directive from Monrovia, funding, changes in priorities, etc.? Kindly explain. A13 E22 Looking at PACS integrated community health and water, sanitation and hygiene (WASH) together - how well do you think that integration worked? Do you think that integrating would work for community health and WASH with other services, such as nutrition? Social protection? Any other services or sectors that are ready for integration with community health? Probe for examples and evidence to support the claims. How was gender a factor? Community engagement? A14 E22 In hindsight, what worked and what didn’t, for increasing access and improving delivery of community health and WASH? Looking forward, how do you think access and quality in community health can be improved – anything new that could be tried? How did PACS work with women and men, for instance in the CHA program or drilling wells or sanitation work, and what lessons are there? What about other social factors, such as ethnicity, or poverty, or distance? What could have been done differently or better, to improve the access and quality of community health and WASH? What kinds of innovations are you aware of or can be tried, given the experience with PACS? A15 E24 Likewise, for working with communities, what parts of PACS worked well and what didn’t work as expected? What could have been done differently or better with communities? What about future work, should the Ministry of Internal Affairs or another ministry be involved, specifically to include traditional structures? Tell me more about how PACS worked with women, and what lessons you see? Anything that could have been better or different, to achieve good community health and WASH with women? How did PACS work with traditional groups – and, what worked or could have been done differently/better? 41 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV Form B - KII: MPW Directors About the Tool This tool will capture the experiences of MPW senior staff to gather a deeper understanding of the PACS design, implementation, results, lessons learned, stakeholder engagement, the challenges they have faced in working with PACS, and their views on effectiveness and sustainability. INTERVIEWER NAME__________________________________ DATE (DD/MM/YY)…………………………….………………………… START TIME……………………………………………………………… H M N END TIME………………………………………………….…………… H M N Introduction AT THE START, inform the participant about the purpose of the interview/discussion, and go through the informed consent process. Keep the signed consent form with you. We are interested in learning about your experience with PACS, including your perceptions, opinions and experiences regarding implementation, and your views on its effectiveness and sustainability. Your frank responses and participation will be most helpful to us to understand your views. Questions Respondent Details Information Name of the Organization Organization code (if any) Location Locality (…………………………………..) Sub-Locality (…………………………………..) Respondent(s) Name(s) Position Title Gender Female Male How many years have you been in this position? Code Question Probes/prompts and notes B01 E11 E14 Briefly describe your role, and how you have worked with PACS. What support has been provided to you or your division through PACS? Examples: logistics, human resources, physical resources, and programmatic information? B02 E11 E14 What do you consider as your strongest skills and competencies with your involvement with PACS? How are those skills and competencies applied in practice? Eg. QRMs, Coord. meetings, superv. Meetings (annual, quarterly, monthly, etc.) How will you use those skills after PACS ends? USAID.GOV USAID/LIBERIA PACS FINAL EVALUATION REPORT 42 Code Question Probes/prompts and notes B03 E11 What areas of PACS capacity building did not work very well? Why did that happen? Encourage a balanced view on capacity activities. B04 E11 E14 How would you describe or identify the main changes in PACS programming around increasing the availability of WASH services from PACS start to its current state? Water points/wells, CLTS, hygiene education and equipment use of data for planning, etc. B05 E11 E14 How did PACS help the MPW contextualize or adapt/operationalize new policy and best practices? Could you illustrate with examples? To what extent will these policies continue after PACS ends? Policy, strategies, guidelines, etc. Quality services and assurances strategies How sustainable are these efforts? B06 E11 What considerations went into designing a technical assistance strategy for CSOs? How has your office worked with the CSOs supported by PACS? What system put in place to gauge level of skills and knowledge increase? Level of MPW involvement with CSOs. B07 E11 E14 What are the government’s plans for financing and/or maintaining CLTS, ODF status, wells, hygiene activities following the end of PACS? Probe beyond funding, ask about technical areas, engagement by county WASH, communities, natural leaders, town chiefs B08 E11 E14 Can you please share how the MPW operational plan guides PACS activities around WASH? ODF status certification and maintenance Inspection of wells, quality testing Hygiene? WASH Joint Sector Review B09 E11 E14 What brought about the creation of the Integrated County WASH Team, and how does it work? Who are the key players, and how has PACS involved them? There are a lot of stakeholders involved- how do they work together? B10 E11 E14 How would you prioritize the key areas of capacity that still need to be built or supported in the next 5 years for MPW to best support WASH services? Are there any skill areas that have improved, remained stagnant, or declined? Involvement of women? B11 E13 E14 What types of WASH structures were set up within the community under PACS? How was community need in different locations determined/assessed? What is the community feedback about the utility of these structures? How are men and women involved in these WASH structures? B12 E21 Could you identify specific benchmarks that you believe would be necessary for these GOL entities to monitor? How might this internal accountability system work? Benchmarks can be the following: - Financial - Technical - Ownership - others as suggested by respondent 43 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV Code Question Probes/prompts and notes B13 E21 To what extent do you believe that your office will take on the role(s) that PACS has done, or where PACS has built capacity? Why do you think so? Kindly describe the indications of where the gap may be, in their taking on that work? What other things need to happen for your office to take on that role, if anything? Step by step, how could your office move ahead in doing the work after PACS has closed? When do you think your office will be ready to take on that role? What signs or evidence show that your office has the capacity for those roles, or not? What about the commitment to take on that role? Besides funding, what gaps still exist that hinder your office from taking on the roles from PACS? Has some change started already? Now, or will it be when PACS ends, or another year? Or, does it depend on something else, such as policy, directive from Monrovia, funding, changes in priorities, etc.? Kindly explain. B14 E22 Looking at PACS integrated community health and water, sanitation and hygiene (WASH) together - how well do you think that integration worked? Do you think that integrating would work for community health and WASH with other services, such as nutrition? Social protection? Any other services or sectors that are ready for integration with community health? Probe for examples and evidence to support the claims. How was gender a factor? Community engagement? B15 E22 In hindsight, what worked and what didn’t, for increasing access and improving delivery of community health and WASH? Looking forward, how do you think access and quality in community health can be improved – anything new that could be tried? How did PACS work with women and men, for instance in drilling wells or sanitation work, and what lessons are there? What about other social factors, such as ethnicity, or poverty, or distance? What could have been done differently or better, to improve the access and quality of community health and WASH? What kinds of innovations are you aware of or can be tried, given the experience with PACS? B16 E24 Likewise, for working with communities, what parts of PACS worked well and what didn’t work as expected? What could have been done differently or better with communities? What about future work, should the Ministry of Internal Affairs or another ministry be involved, specifically to include traditional structures? Tell me more about how PACS worked with women, and what lessons you see? Anything that could have been better or different, to achieve good community health and WASH with women? How did PACS work with traditional groups – and, what worked or could have been done differently/better? B17 E14 E23 E24 Which other partners or NGOs are doing similar work in community WASH, and what are their advantages or disadvantages relative to PACS? Find names of local CSOs that do … - CLTS (+) - Water points - Hygiene education and support Ask about comparisons in technical work, engagement with GOL and communities, and cost. USAID.GOV USAID/LIBERIA PACS FINAL EVALUATION REPORT 44 Form C - FGD: MOGCSP About the Tool This tool will capture the perceptions of MOGCSP senior staff regarding the PACS design, implementation, results and lessons learned, how stakeholders have implemented the project, the challenges they have faced in working with PACS, and their views on effectiveness and sustainability. MODERATOR NAME __________________________________ NOTE TAKER NAME ___________________________________ DATE (DD/MM/YY)…………………………….……………………… ORGANIZATION NAME ______________________________ ORGANIZATION CODE ______________________________ LOCALITY ___________________________________ SUB-LOCALITY ___________________________________ FGD CODE …………………………………………….……………… No. PARTICIPANTS AT START OF FGD……………(FEMALE)………….. No. PARTICIPANTS AT START OF FGD……………(MALE)……………….. No. PARTICIPANTS AT THE END OF FGD…………(FEMALE)…..………. No. PARTICIPANTS AT THE END OF FGD…………(MALE)……..………. RECORDER NUMBER………………………………………………… START TIME…………………………………………………………… H M N END TIME………………………………………………….…………… H M N *: Focus-Group Codes: Team Leader to provide FGD codes, if required. Introduction AT THE START, inform the participants about the purpose of the interview/discussion, and go through the informed consent process. Keep the signed consent forms with you. We are interested in learning about your experiences and that of your group regarding PACS, including your engagement, understanding and opinions. Your frank responses and participation will be helpful to us as we try to really understand your views. Your responses will not be considered “right” 45 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV or “wrong” as we are mainly interested in knowing what you think. Everyone’s views are equally important. It’s fine to disagree with other people’s views, but if you do, it’s important to disagree in a respectful and polite manner. It’s important for you to talk in turns, because if you all speak at once, we will not be able to hear you and understand. If you disagree with something anyone says, you can say ‘I disagree’ and then wait for them to finish before you speak. MODERATOR ● EXPLAIN THE ROLE OF NOTE-TAKERS AND AUDIO-RECORDER ● ASK THE PARTICIPANTS TO FILL IN THE SIGN-IN SHEET (ADULT or CHILD TEMPLATE PROVIDED) ● ALLOW RESPONDENTS A FEW MINUTES FOR ANY QUESTIONS REGARDING THE FGD __________________________________________________________________________ Code Question Probes/prompts and notes C01 E21 What does MOGCSP see as good practice in sustainability or transition/phase-out for an activity? How is that practice followed by GOL and your ministry? Discussions/agreement with GOL and IPs? Clarity and frequency of communication? C02 E22 Where does PACS fit in MOGCSP priorities, alongside other health activities and other sectors? And community health? Given that internal positioning, what areas of integration does MOGCSP already expect to explore? Nutrition, food security, agriculture, livelihoods/economic growth Social protection, DRG C03 E21 E23 Describe the quality and frequency of your interactions with GOL partners (Minister, CMO, CHSD Director, others, Min PW, Minister of Internal Affairs, others). Sustainability v pilot/trial C04 E21 Similarly, describe the ministry’s interactions with other DPs: UNICEF, WB, JICA. What coord around sustainability? Harmonization, coordination Competition C05 E21 What do you think is required for GOL and local CSOs to take on the role(s) that PACS has done, or where PACS has built their capacity? Why do you think so? Step by step, how could the MOH move ahead in doing the work after PACS has closed? When do you think the MOH, MPW and/or CSOs will be ready to take on those roles? Has any of this started already, through PACS or other activities? Does that depend on something else, such as policy, directive from Monrovia, funding, changes in priorities, etc.? Kindly explain. Can you point to signs or evidence that they have the capacity to do that role? What signs or evidence do you see of their commitment to take on that role? Do you think that the gaps are in capacity, or commitment, or both, or something else? Kindly describe the indications of where the gap may be, in their taking on that work? C06 E22 Why do you think that integration would work for community health and WASH with other services, such as nutrition? Social protection – which aspects? In what ways is community engagement a factor in integrating those areas? C07 E23 E24 Looking at PACS in hindsight, what worked and what didn’t (capacity building, services or engagement? What could have been done differently or better, to improve the access and quality of gender, children and social protection? Community health and WASH? 47 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV FGD D: USAID About the Tool This tool will capture the perceptions and experiences of USAID staff regarding the PACS design, implementation, results and lessons learned, how the stakeholders have implemented the project, the challenges they have faced in working with PACS, and their views on effectiveness and sustainability. MODERATOR NAME __________________________________ NOTE TAKER NAME ___________________________________ DATE (DD/MM/YY)…………………………….……………………… ORGANIZATION NAME _______________________________ ORGANIZATION CODE ________________________________ LOCALITY ___________________________________ SUB-LOCALITY ___________________________________ FGD CODE …………………………………………….………………… No. PARTICIPANTS AT START OF FGD……………(FEMALE)………….. No. PARTICIPANTS AT START OF FGD……………(MALE)……………….. No. PARTICIPANTS AT THE END OF FGD…………(FEMALE)…..………. No. PARTICIPANTS AT THE END OF FGD…………(MALE)……..………. RECORDER NUMBER………………………………………………… START TIME……………………………………………………………… H M N END TIME………………………………………………….…………… H M N *: Focus-Group Codes: Team Leader to provide FGD codes, if required. Introduction AT THE START, inform the participants about the purpose of the interview/discussion, and go through the informed consent process. Keep the signed consent forms with you. We are interested in learning about your experiences and that of your group regarding PACS, including your engagement, understanding and opinions. Your frank responses and participation will be helpful to us as we try to really understand your views. Your responses will not be considered “right” USAID.GOV USAID/LIBERIA PACS FINAL EVALUATION REPORT 48 or “wrong” as we are mainly interested in knowing what you think. Everyone’s views are equally important. It’s fine to disagree with other people’s views, but if you do, it’s important to disagree in a respectful and polite manner. It’s important for you to talk in turns, because if you all speak at once, we will not be able to hear you and understand. If you disagree with something anyone says, you can say ‘I disagree’ and then wait for them to finish before you speak. MODERATOR ● EXPLAIN THE ROLE OF NOTE-TAKERS AND AUDIO-RECORDER ● ASK THE PARTICIPANTS TO FILL IN THE SIGN-IN SHEET (ADULT or CHILD TEMPLATE PROVIDED) ● ALLOW RESPONDENTS A FEW MINUTES FOR ANY QUESTIONS REGARDING THE FGD ___________________________________________________________________________ Code Question Probes/prompts and notes D01 E10 Describe what you mean by the phrasing “implement with fidelity” in EQ-1 Clarifications D02 E21 E23 As the evaluation team, we want the recommendations to be relevant to USAID’s intended uses. Accordingly, please describe the types of funding mechanisms you envision for a future design of community health activities in Liberia. Specifically, how would USAID want to handle deviations from the Journey to Self-Reliance? Contingent phases, based on IP and GOL completion of milestones. Can USAID reasonably terminate funding, or move to go-slow, with its obligation cycles? D03 E21 What does USAID Liberia see as good practice in sustainability or transition/phase-out for an activity? How is that practice followed by USAID? Discussions/agreement with GOL and IPs? Clarity and frequency of communication? D04 E22 Where does CH fit in USAID Liberia’s priorities, alongside other health activities (FARA, HSS, others) and other sectors? Given that internal positioning, what areas of integration does USAID already expect to explore? Nutrition, food security, agriculture, livelihoods/economic growth Social protection, DRG D05 E21 E23 Describe the quality and frequency of your interactions with GOL partners (Minister, CMO, CHSD Director, others, Min PW, Minister of Internal Affairs, others). How do you handle ‘difficult’ topics, such as funding requests, e.g., payment of CHA/SS incentives, or drugs? Are those discussions around CH typically technical, political, activity￾focused? Sustainability v pilot/trial D06 E21 Similarly, describe the mission’s and health office’s interactions with other DPs: UNICEF, WB, JICA. What coord around sustainability? DSA-type incentives? Harmonization, coordination Competition D07 E21 E24 What about internal USAID discussions, e.g., Washington and other Monrovia activities? e.g., LMH funded from DC, with CHA incentives, but not initially favored by USAID Monrovia, nor in line with J2SR Coherence D08 E21 What do you think is required for GOL and local CSOs to take on the role(s) that PACS has done, or where PACS has built their capacity? Why do you think so? Step by step, how could the MOH move ahead in doing the work after PACS has closed? When do you think the MOH, MPW and/or CSOs will be ready to take on those roles? Can you point to signs or evidence that they have the capacity to do that role? What signs or evidence do you see of their commitment to take on that role? Do you think that the gaps are in capacity, or commitment, or both, or something else? Kindly describe the 49 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV Code Question Probes/prompts and notes Has any of this started already, through PACS or other activities? Does that depend on something else, such as policy, directive from Monrovia, funding, changes in priorities, etc.? Kindly explain. indications of where the gap may be, in their taking on that work? D09 E22 Why do you think that integration would work for community health and WASH with other services, such as nutrition? Social protection – which aspects? Are there other sectors that USAID is considering? In what ways is community engagement a factor in integrating those areas? D10 E23 E24 Looking at PACS in hindsight, what worked and what didn’t (capacity building, services or engagement? What could have been done differently or better, to improve the access and quality of community health and WASH? D11 E14D Regarding the costing for WASH, we have tried to contact UNICEF and have not had a response, regarding the main person to speak with. Can USAID assist? And, for identifying a local WASH IP, would a different organization be preferable to UNICEF? 51 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV Form G - KII: County Health Officer/Department Director (CHO/CHDD) About the Tool This tool will capture the perceptions of senior county staff (CHO/CHSS) to gather a deeper understanding of the PACS design, implementation, results, lessons learned, stakeholder engagement, challenges faced in working with PACS, and their views on effectiveness and sustainability. INTERVIEWER NAME__________________________________ DATE (DD/MM/YY)…………………………….………………………… START TIME………………………………………………………………. H M N END TIME………………………………………………….………… H M N Introduction AT THE START, inform the participant about the purpose of the interview/discussion, and go through the informed consent process. Keep the signed consent form with you. We are interested in learning about your experience with PACS, including your perceptions, opinions and experiences regarding implementation, and your views on its effectiveness and sustainability. Your frank responses and participation will be most helpful to us to understand your views. Questions Respondent Details Information Name of the Organization Organization code (if any) Location Locality (…………………………………..) Sub-Locality (…………………………………..) Respondent(s) Name(s) Position Title Gender Female Male How many years have you been in this position? Code Question Probes/prompts and notes G01 E11a Briefly describe your role, and how you have worked with PACS. What support has been provided to you or your division through PACS? (logistics, human resources, physical resources, and programmatic information?) QRMs, coordination or supervisory meetings, development of plans and guidelines G02 E11a What do you consider as your strongest skills and competencies gained from your involvement with PACS? How are those skills and competencies applied in practice? How will you use those skills after PACS ends? USAID.GOV USAID/LIBERIA PACS FINAL EVALUATION REPORT 52 Code Question Probes/prompts and notes G03 E11a What areas/types of PACS capacity building for CHOs/CHDDs did not work very well? Anything that you did not find to be helpful in your work? Why did that happen? Encourage a balanced view on capacity activities. G04 E11a E13 One of the PACS areas of work was in behavioral change or BCC. What specific areas did PACS provide you support with for BCC in the intervention districts? How effective was this support? BCC tool development CLTS and ODF maintenance/monitoring WaterGuard G05 E13 What are the key messages from PACS related to healthy behaviors and health education, and how were these messages delivered? What key areas of BCC do community members respond to most? What are the behaviors that are the most difficult for them to keep up? Why do you think this is, and what are the challenges moving forward? Which stakeholders were critical in initiating messaging and how were these stakeholders identified? Were there any differences by gender, in messages or acceptance by community members? G06 E11a E14 One of the PACS areas of work was in supporting WASH activities. In what ways did PACS support the CHO/CHDD with WASH in the intervention districts? How effective was this support? What key areas of WASH are the most difficult to uphold in this county and why? What will be the challenges moving forward? WASH Market Facilitation Plan Water quality training Capacity building on WASH G07 E10 E22 PACS had some districts that had integrated WASH support along with BCC and iCCM. Were there were any differences in community health outcomes in districts who had integrated support compared to those communities that only received WASH support? Why or Why not? What things went well with integration, and what things were more difficult to uphold? What resources are necessary for integration? How well did the individual sector officers collaborate? Any areas that you think benefited more than others? G08 E22 Do you think that integrating would work for community health and WASH with other services, such as nutrition? Social protection? Any other services or sectors that are ready for integration with community health? Gender considerations Community engagement G09 E10 How would you describe the main changes in PACS programming around increasing the availability of community-based health care from the beginning of the project to its current state? child health, family planning, maternal health, nutrition and WASH use of data for planning, etc. G10 E11a How has the CHT worked with the CSOs supported by PACS? What worked well with this collaboration, and was anything not so successful? In what areas do you feel that CSOs have the most added value in your county? Are there activities you would like to see them more involved in? Functionality of CHCs, HFDCs & CSOs leaderships Monitoring BCC G11 E12 E23 To what extent did PACS’s interventions with the CHA’s make any improvements in their work with the communities in your county? What else do CHAs and CHSSs need to do their job well? Is there anything that PACS missed in engaging with the CHAs? 53 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV Code Question Probes/prompts and notes G12 E12 What were the outcomes of the CHC mapping exercises? Were there any lessons learned from this exercise? Have there been any discussion about what/how this data can be used in the future? How would you describe the quality of the data collected by CHAs? How is this used for decision making? Has the data quality changed over time? CBIS data collection tools? Other data collected? DQAs? G13 E21 What are the government’s plans for financing for NCHAP (or CLTS, wells, etc.) following the end of PACS? Probe beyond funding, ask about technical areas, engagement by OICs, DHTs, CHTs with CHAs, etc. G14 E11a E21 How would you prioritize the key areas of capacity that still need to be built or supported in the next 5 years for the CHT, DHTs and CSOs, to best support community health service delivery? Are there any skill areas that have improved, remained stagnant, or declined? Are there any specific benchmarks to monitor that will identify progress? (financial, technical, ownership) G15 E21 To what extent do you believe that the CHT will take on the role(s) that PACS has done, or where PACS has built capacity? Why do you think so? Kindly describe the indications of where the gap may be, in their taking on that work? What other things need to happen for the CHT to take on that role, if anything? Step by step, how could your office move ahead in doing the work after PACS has closed? Is there any one specific activity you would be able to take on? When do you think your office will be ready to take on that role? What signs or evidence show that your office has the capacity for those roles, or not? What about the commitment to take on that role? Besides funding, what gaps still exist that hinder your office from taking on the roles from PACS? Has some change started already? Now, or will it be when PACS ends, or another year? Or, does it depend on something else, such as policy, directive from Monrovia, funding, changes in priorities, etc.? Kindly explain. G16 E12 E23 In hindsight, what worked and what didn’t, for increasing access and improving delivery of community health and WASH? Looking forward, how do you think access and quality in community health can be improved – anything new that could be tried? How did PACS work with women and men, and what lessons are there? What about other social factors, such as ethnicity, or poverty, or distance? Any key lessons learned? What could have been done differently or better, to improve the access and quality of community health and WASH? What kinds of innovations are you aware of or can be tried, given the experience with PACS? G17 E13 E24 Likewise, for working with communities, what parts of PACS worked well and what didn’t work as expected? What could have been done differently or better with communities? What about future work, should other ministries or agencies be involved, specifically to include traditional structures? MIA, etc, private sector Anything that could have been better or different, to achieve good community health and WASH with women? How did PACS work with traditional groups – and, what worked or could have been done differently/better? USAID.GOV USAID/LIBERIA PACS FINAL EVALUATION REPORT 54 Form H - FGD: County Health Team About the Tool This tool will capture the perceptions and experiences of CHT staff regarding the PACS design, implementation, results and lessons learned, how the stakeholders have implemented the project, the challenges they have faced in working with PACS, and their views on effectiveness and sustainability. MODERATOR NAME __________________________________ NOTE TAKER NAME ___________________________________ DATE (DD/MM/YY)…………………………….……………………… ORGANIZATION NAME _______________________________ ORGANIZATION CODE ________________________________ LOCALITY ___________________________________ SUB-LOCALITY ___________________________________ FGD CODE …………………………………………….……………… No. PARTICIPANTS AT START OF FGD……………(FEMALE)………….. No. PARTICIPANTS AT START OF FGD……………(MALE)……………….. No. PARTICIPANTS AT THE END OF FGD…………(FEMALE)…..………. No. PARTICIPANTS AT THE END OF FGD…………(MALE)……..………. RECORDER NUMBER…………………………………………………… START TIME……………………………………………………… H M N END TIME………………………………………………….…… H M N *: Focus-Group Codes: Team Leader to provide FGD codes, if required. Introduction AT THE START, inform the participants about the purpose of the interview/discussion, and go through the informed consent process. Keep the signed consent forms with you. We are interested in learning about your experiences and that of your group regarding PACS, including your engagement, understanding and opinions. Your frank responses and participation will be helpful to us as we try to really understand your views. Your responses will not be considered “right” 55 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV or “wrong” as we are mainly interested in knowing what you think. Everyone’s views are equally important. It’s fine to disagree with other people’s views, but if you do, it’s important to disagree in a respectful and polite manner. It’s important for you to talk in turns, because if you all speak at once, we will not be able to hear you and understand. If you disagree with something anyone says, you can say ‘I disagree’ and then wait for them to finish before you speak. MODERATOR ● EXPLAIN THE ROLE OF NOTE-TAKERS AND AUDIO-RECORDER ● ASK THE PARTICIPANTS TO FILL IN THE SIGN-IN SHEET (ADULT or CHILD TEMPLATE PROVIDED) ● ALLOW RESPONDENTS A FEW MINUTES FOR ANY QUESTIONS REGARDING THE FGD ___________________________________________________________________________ Code Question Probes/prompts and notes H01 E11 VERY BRIEFLY – describe your role(s), and how you have worked with PACS. What support has been provided to you or your team through PACS? Examples: capacity building, materials, technical assistance, managerial, operational, etc. H02 E11 What do you consider as your strongest skills and competencies based on your involvement with PACS? How do you apply those skills and competencies in your work, day to day, weekly, monthly? Examples – new skills in management or technical work, community engagement, collection or use of data? How will you use those skills after PACS ends? H03 E11 What areas of PACS capacity building did not work very well? Why did that happen? Encourage a balanced view on capacity activities. H04 E13 How would you describe the current capacity of CHTs to promote health-seeking behaviors and encourage community members to seek out services? What about the capacity of District Health Teams and Health Facility staff? H04 E21 Looking forward, what kind of milestones would show that PACS support to you has led to lasting changes? And milestones for the CHT? DHTs? OIC? Examples – respondent and/or CHT can plan, budget for community health, supervise CHAs, monitor WASH, etc. DHTs and OICs – drugs for CHAs, etc. H05 E21 Step by step, how could your office move ahead in doing the work after PACS has closed? When do you think your office will be ready to take on that role? Besides funding, what gaps still exist that hinder your office from taking on the roles from PACS? Capacity? Commitment? Has some change started already? Now, or when PACS ends, or another year? Or, does it depend on something else, such as policy, directive from Monrovia, funding, etc.? H06 E11 How has your office worked with the CSOs supported by PACS? What level of county involvement with CSOs, how can CSOs continue PACS-type activities after PACS closes? H07 E12 E13 E14 What are the government’s plans for financing and/or maintaining the work started under PACS on its own, after PACS ends: - support to the NCHA program - health promotion - siting, drilling, testing wells - CLTS, ODF status, hygiene activities Probe beyond funding, ask about technical areas, engagement by - CHAs and CHSSs - ICWSC, CHC and community WASH, - natural leaders, WASH Entrepreneurs, - town chiefs, others USAID.GOV USAID/LIBERIA PACS FINAL EVALUATION REPORT 56 Code Question Probes/prompts and notes H08 E11 E14 How would you prioritize the new areas of capacity that still need to be built or supported in the next 5 years for the CHT to best support community health and WASH services? Are there any skill areas that have improved, remained stagnant, or declined? Involvement of women? H09 E11- 4 What would you consider as the top three achievements of PACS related to increasing access and improving community health and/or WASH service delivery, and why? H10 E11 E12 E14 In what ways has gender been part of the CHTs work in community health and WASH? How has PACS contributed to any specific work on gender, or equity, if at all? Prompts: - 80 percent+ of CHAs are men, how can that be balanced? - is there evidence that women are involved in site selection for wells, triggering communities for CLTS? H11 E22 Looking at PACS integrated community health and WASH together - how well do you think that integration worked? Do you think that integrating would work for community health and WASH with other services, such as nutrition? Social protection? Any other services or sectors that are ready for integration with community health? Probe for examples and evidence to support the claims. What specifically has the integration affected positively and negatively? How was gender a factor in integrating health and WASH? Community engagement? H12 E23 In hindsight, what worked and what didn’t, for increasing access and improving delivery of community health and WASH? Looking forward, how do you think access and quality in community health can be improved – anything new that could be tried? What could have been done differently or better, to improve the access and quality of community health and WASH? What kinds of innovations are you aware of or can be tried, given the experience with PACS? H13 E24 Likewise, for working with communities, what parts of PACS worked well and what didn’t work as expected? What could have been done differently or better with communities? What about future work, should any traditional groups or authorities like the Ministry of Internal Affairs or another ministry be involved, to support community health and WASH? Tell me more about how PACS worked with women, and what lessons you see? Anything that could have been better or different, to achieve good community health and WASH with women? How did PACS work with traditional groups – and, what worked or could have been done differently/better? H14 E11 E14 E23 E24 Which other LOCAL CSO partners or NGOs are doing similar work in community health WASH, and what are their advantages or disadvantages relative to PACS? Find names of local CSOs that do … - Support to CHAs and Health Facilities - Health promotion and BCC - CLTS (+) and/or Water points - Hygiene education and support Ask about comparisons in technical work, engagement with CHT, facilities and communities, and costs. H15 Kindly share with us plans or reports related to community health/WASH, such as the county development agenda or annual plans. Examples: - CHT Implementation Plan - Reports for the QRM 57 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV Form J - KII: County WASH Officer About the Tool This tool will capture the experiences of County WASH Officers to gather a deeper understanding of the PACS design, implementation, results, lessons learned, stakeholder engagement, the challenges they have faced in working with PACS, and their views on effectiveness and sustainability. INTERVIEWER NAME__________________________________ DATE (DD/MM/YY)…………………………….……………………… START TIME………………………………………………………………. H M N END TIME………………………………………………….…………… H M N Introduction AT THE START, inform the participant about the purpose of the interview/discussion, and go through the informed consent process. Keep the signed consent form with you. We are interested in learning about your experience with PACS, including your perceptions, opinions and experiences regarding implementation, and your views on its effectiveness and sustainability. Your frank responses and participation will be most helpful to us to understand your views. Questions Respondent Details Information Name of the Organization Organization code (if any) Location Locality (…………………………………..) Sub-Locality (…………………………………..) Respondent(s) Name(s) Position Title Gender Female Male How many years have you been in this position? Code Question Probes/prompts and notes J01 E11 E14 Briefly describe your role, and how you have worked with PACS, which includes Global Communities as well as IRC, PSI/L and YMCA. What support has been provided to you or your division through PACS? Examples: logistics, human resources, physical resources, and programmatic information? J02 E11 E14 What brought about the creation of the Integrated County WASH Team, and how does it work? Who are the key players, and how has PACS involved them? There are a lot of stakeholders involved- how do they work together? Explore possibilities of integrating WASH with other sectors: nutrition, social protection, food security, agriculture, etc. USAID.GOV USAID/LIBERIA PACS FINAL EVALUATION REPORT 58 Code Question Probes/prompts and notes J03 E11 What areas of PACS support to WASH worked well, or did not work very well? Why did that happen? Encourage a balanced view on capacity activities. J04 E14 What have been the specific roles of natural leaders and WASH entrepreneurs and how effective have they been? Has the role of natural leaders changed after their communities become ODF? How do WASH entrepreneurs J05 E11 E14 What are the government’s plans for financing and/or maintaining CLTS, ODF status, wells, hygiene activities following the end of PACS? Probe beyond funding, ask about technical areas, engagement by county WASH, communities, natural leaders, town chiefs J06 E11 E14 How would you prioritize the key areas of capacity that still need to be built or supported in the next 5 years for MPW to best support WASH services? Are there any skill areas that have improved, remained stagnant, or declined? Involvement of women? J07 E21 What can you suggest as future milestones for WASH in the county? Probe for milestones that belong to the partners and to the County WASH/ICWSC J08 E11 E14 E23 What would you consider as the top three achievements of PACS related to increasing access and improvingWASH service delivery, and why? Are there specific activities that you found to be particularly innovative? J09 E21 To what extent do you believe that your office will take on the role(s) that PACS has done, or where PACS has built capacity? Why do you think so? Kindly describe the indications of where the gap may be, in their taking on that work? What other things need to happen for your office to take on that role, if anything? Step by step, how could your office move ahead in doing the work after PACS has closed? When do you think your office will be ready to take on that role? What signs or evidence show that your office has the capacity for those roles, or not? What about the commitment to take on that role? Besides funding, what gaps still exist that hinder your office from taking on the roles from PACS? Has some change started already? Now, or will it be when PACS ends, or another year? Or, does it depend on something else, such as policy, directive from Monrovia, funding, changes in priorities, etc.? Kindly explain. J10 E22 Looking at PACS integrated community health and water, sanitation and hygiene (WASH) together - how well do you think that integration worked? Do you think that integrating would work for community health and WASH with other services, such as nutrition? Social protection? Any other services or sectors that are ready for integration with community health? Probe for examples and evidence to support the claims. How was gender a factor? Community engagement? 59 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV Code Question Probes/prompts and notes J11 E22 In hindsight, what worked and what didn’t, for increasing access and improving delivery of community health and WASH? Looking forward, how do you think access and quality in community health can be improved – anything new that could be tried? How did PACS work with women and men, for instance in drilling wells or sanitation work, and what lessons are there? What about other social factors, such as ethnicity, or poverty, or distance? What could have been done differently or better, to improve the access and quality of community health and WASH? What kinds of innovations are you aware of or can be tried, given the experience with PACS? What approach of the integrated activities would be more effective within the county/district, and why? J12 E24 Likewise, for working with communities, what parts of PACS worked well and what didn’t work as expected? What could have been done differently or better with communities? What about future work, should the Ministry of Internal Affairs or another ministry be involved, specifically to include traditional structures? Tell me more about how PACS worked with women, and what lessons you see? Anything that could have been better or different, to achieve good community health and WASH with women? How did PACS work with traditional groups – and, what worked or could have been done differently/better? J13 E14 E23 E24 Which other partners or NGOs are doing similar work in community WASH, and what are their advantages or disadvantages relative to PACS? Can you recommend any local partners that work well in community WASH and health? Find names of local CSOs that do … - CLTS (+) - Water points - Hygiene education and support Ask about comparisons in technical work, engagement with GOL and communities, and cost. 61 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV Form K - KII: County Social Workers About the Tool This tool will capture the perceptions and experiences of County Social Workers regarding the PACS design, implementation, results, lessons learned, stakeholders’ implementation, the challenges they have faced in working with PACS, and their views on effectiveness and sustainability. Code Question Probes/prompts and notes K01 Tell us about your role, and how the PACS project provided any support Guidelines for foster care. K02 E22 Where does PACS fit in priorities for social work, alongside other health activities and other sectors? Given that internal positioning, what areas of integration does MOGCSP already expect to explore? Nutrition, food security, agriculture, livelihoods/economic growth Social protection, DRG K03 E21 Similarly, kindly describe support you receive from other donors or partners, such as UNICEF, WB, JICA. What coordination is there around sustainability? Harmonization, coordination Competition K04 E21 What does County social office/MOGCSP see as good practice in sustainability or transition/phase-out for an activity? How is that practice followed by GOL and your ministry? Discussions/agreement between GOL and PACS? Clarity and frequency of communication? K05 E21 What do you think is required for GOL and local CSOs to take on the role(s) that PACS has done, or where PACS has built their capacity? Why do you think so? Step by step, how could the government move ahead in doing the work after PACS has closed? Can you point to signs or evidence that they have the capacity to do that role? What signs or evidence do you see of their commitment to take on that role? K06 E22 How might social protection be integrated into community health and WASH? Are there areas already being integrated with social protection? In what ways is community engagement a factor in integrating those areas? K07 E22 How will the new social registry being developed support health or WASH? K08 E23 E24 Looking at PACS in hindsight, what worked and what didn’t (capacity building, services or engagement? What could have been done differently or better, to improve the access and quality of gender, children and social protection? Community health and WASH? 63 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV Form L - KII: PACS field staff at County Level About the Tool This tool will capture the experiences of PACS county coordinator and field staff to gather a deeper understanding of the PACS design, implementation, results and lessons learned, stakeholders’ engagement, challenges faced in working with PACS, and their views on effectiveness and sustainability. INTERVIEWER NAME__________________________________ DATE (DD/MM/YY)…………………………….…………………… START TIME……………………………………………………………… H M N END TIME………………………………………………….………… H M N Introduction AT THE START, inform the participant about the purpose of the interview/discussion, and go through the informed consent process. Keep the signed consent form with you. We are interested in learning about your experience with PACS, including your perceptions, opinions and experiences regarding implementation, and your views on its effectiveness and sustainability. Your frank responses and participation will be most helpful to us to understand your views. Questions Respondent Details Information Name of the Organization Organization code (if any) Location Locality (…………………………………..) Sub-Locality (…………………………………..) Respondent(s) Name(s) Position Title Gender Female Male How many years have you been in this position? Code Question Probes/prompts and notes L01 Briefly describe your role, and how you have worked with PACS. To put the respondent at ease. Listen for points to follow-up. L02 E10 What do you consider the strongest contributions by PACS to community health and WASH, related to your own work or your colleagues? Explain (why/how). How will that work continue after PACS closes? Who are the county and local owners? Is there local capacity to continue? L03 E10 What parts of PACS work did not work as expected? Why is that? Encourage a balanced view on PACS activities. Code Question Probes/prompts and notes L04 E10 In each area of PACS work, what do you consider to be the (a) lasting accomplishment, and (b) less successful? - Capacity building: MOH, other government, CSOs - CHA program and iCCM - Behavior change, Water Guard - CLTS, wells, hygiene After the responses, be sure to cover the two points: - community engagement - capacity of GOL and partners L05 E12 To what extent did PACS’s interventions with the CHA’s make any improvements in their work with the communities in your county? What else do CHAs and CHSSs need to do their job well? L06 E10 There are some examples of work by PACS that were initially successful, but then was not sustainable, e.g., - GOL capacity building (some aspects) - MOH – community engagement with PACS - Communities not adopting new behaviors/WaterGuard - ODF certified communities reverting to open defecation Why do you think that may have happened, and what could the next project do differently to improve these? L07 E20 E23 Looking back at the work of PACS and the situation today, what are the remaining gaps in community health and WASH, that a new project will have to take on? L08 E22 PACS has some districts with integrated WASH support along with BCC and iCCM. Were there were any differences in community health outcomes in districts who had integrated support compared to those communities that only received WASH support? Why or Why not? What things went well with integration of PACS field activities, and what things were more difficult to uphold? What resources are necessary for integration? How well did the individual sector officers collaborate? Any areas that you think benefited more than others? L09 E22 Do you think that integrating would work for community health and WASH with other services, such as nutrition? Social protection? Any other services or sectors that are ready for integration with community health? Gender considerations Community engagement L10 E23 How would you describe the main changes in PACS programming around increasing the availability of community-based health care from the beginning of the project to its current state? child health, family planning, maternal health, nutrition and WASH use of data for planning, etc. L11 E14 E23 With PACS coming to a close, who are the local partners that can take on or continue the work that PACS started? CHT members, other GOL CSOs Functionality of CHCs, HFDCs & CSOs leaderships L12 E21 How would you prioritize the key areas of capacity that still need to be built or supported in the next 5 years for the CHT, DHTs and CSOs, to best support community health and WASH service delivery? Are there any skill areas that have improved, remained stagnant, or declined? Are there any specific benchmarks to monitor that will identify progress? (financial, technical, ownership) 65 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV Code Question Probes/prompts and notes L13 E21 To what extent do you believe that the CHT will take on the role(s) that PACS has done, or where PACS has built capacity? Why do you think so? Kindly describe the indications of where the gap may be, in their taking on that work? What other things need to happen for the CHT to take on that role, if anything? Step by step, how could the CHT move ahead in doing the work after PACS has closed? Is there any one specific activity you would be able to take on? When do you think the CHT will be ready to take on that role? What signs or evidence show that the CHT has the capacity for those roles, or not? What about the CHT’s commitment to take on that role? Besides funding, what gaps still exist that hinder the CHT from taking on the roles from PACS? Has some change started already? Now, or will it be when PACS ends, or another year? Or, does it depend on something else, such as policy, MOH, funding, changes in priorities, etc.? Kindly explain. L14 E24 How did PACS work with women and men, and what lessons are there? What about other social factors, such as ethnicity, or poverty, or distance? Any key lessons learned? What kinds of innovations are you aware of or can be tried, given the experience with PACS? L15 E24 Likewise, for working with communities, what parts of PACS worked well and what didn’t work as expected? What could have been done differently or better with communities? What about future work, should other ministries or agencies be involved, specifically to include traditional structures? Anything that could have been better or different, to achieve good community health and WASH with women? How did PACS work with traditional groups – and, what worked or could have been done differently/better? Ministry of Internal Affairs, etc, private sector 67 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV Form M - FGD: Community Water Supply and Sanitation Committee (CWSC) About the Tool This tool will capture the experiences of the CWSC to gather a deeper understanding of the PACS design, implementation, results and lessons learned, how the stakeholders have implemented the project, the challenges they have faced in working with PACS, and their views on effectiveness and sustainability. INTERVIEWER NAME__________________________________ DATE (DD/MM/YY)…………………………….…………………… START TIME…………………………………………………….…… H M N END TIME………………………………………………….………… H M N Introduction AT THE START, inform the participant about the purpose of the interview/discussion, and go through the informed consent process. Keep the signed consent form with you. We are interested in learning about your experience with PACS, including your perceptions, opinions and experiences regarding implementation, and your views on its effectiveness and sustainability. Your frank responses and participation will be most helpful to us to understand your views. Questions Respondent Details Information Name of the Organization Organization code (if any) Location Locality (…………………………………..) Sub-Locality (…………………………………..) Respondent(s) Name(s) Position Title Gender Female Male How many years have you been in this position? Code Question Probes/prompts and notes M01 E11 What have been the CWSC committee specific roles and responsibilities with the WASH component in PACS? What specific support/capacity have you received under PACS? M02 E11 What have been your top 3 successful contributions/achievements to WASH activities? How has CWSC provided needed support in the community? Code Question Probes/prompts and notes M03 E11 There are several actors involved in CLTS. Which actors within the PACS’ WASH Activity have you worked or collaborated with easily and closely as compared to the others? What makes it easy for such collaboration? (MPW, NTCU, County/District Sanitation Committees, ICWSC, CHTs, CHSS, CHAs, CSOs) M04 E11 What are the feedbacks and requests normally received through the CWSC on WASH areas? And how are they channeled? Please mention few specific requests and how were they handled/addressed? M05 E11 To what extent are there clear linkages between CLTS and community behavioral health messages from PACS? Are there things that could have been done differently? M06 E11 What types of WASH structures were set up within the community under PACS? How would you describe the quality and functions of these structures? (pumps, water point construction) M07 E11 How is gender considered in CWSC committee and activities? M08 E11 What is the CWSC’s role in the water sources productivity and water quality measurement M09 E11 In general, how would you prioritize the key areas of capacity that still need to be built or supported in the next 5 years for the WASH Committees and WASH entrepreneurs to best support community health service delivery? What are the specific skills areas that have improved, remained or dropped? M10 E11 What opportunities available for WASH that are still not fully utilized to improve community health programs/service delivery, and which future initiatives would you make use of? M11 E11 What would you consider as the top three achievements of PACS related to increasing access and improving service delivery under WASH and why? M12 E11 What would you consider as the top three areas where PACS could have done better with WASH? That is; collaboration, communication, etc) Please give reasons. M13 E11 What were the main limitations and challenges faced by the WASH entrepreneurs doing PACS’ implementation? How did were they addressed and to what extent? What went well and what didn’t, and why? 69 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV Form N - KII: WASH Entrepreneur About the Tool This tool will capture the experiences of WASH Entrepreneurs to gather a deeper understanding of the PACS design, implementation, results and lessons learned, how the stakeholders have implemented the project, the challenges they have faced in working with PACS, and their views on effectiveness and sustainability. INTERVIEWER NAME__________________________________ DATE (DD/MM/YY)…………………………….…………………… START TIME…………………………………………………………… H M N END TIME………………………………………………….………… H M N Introduction AT THE START, inform the participant about the purpose of the interview/discussion, and go through the informed consent process. Keep the signed consent form with you. We are interested in learning about your experience with PACS, including your perceptions, opinions and experiences regarding implementation, and your views on its effectiveness and sustainability. Your frank responses and participation will be most helpful to us to understand your views. Questions Respondent Details Information Name of the Organization Organization code (if any) Location Locality (…………………………………..) Sub-Locality (…………………………………..) Respondent(s) Name(s) Position Title Gender Female Male How many years have you been in this position? Code Question Probes/prompts and notes N01 E14a What have been your specific roles as a WASH Entrepreneur? To what extend have you been involved with PACS’ activities, specifically the WASH component? What have been your top 3 successful contributions/achievements to WASH activities? What specific support/capacity have you received under PACS? How involved are you with the WASH water sources productivity and water quality measurement? N02 E14a What are the requests normally received from CLTS￾WASH to you? Please mention few specific requests and how were they handled/addressed? Code Question Probes/prompts and notes N03 E14a How do you work with various water-sanitation groups, such as the community CWSC or the county WASH office? ICWSC? CHT? Women in communities? N04 E11 E22 E14a What systems or methods are setup between you and the community/CHAs/CHSS/CWSC to receive WASH request and feedback? N05 E14a In general, how would you prioritize the key areas of capacity that still need to be built or supported in the next 5 years for the WASH entrepreneurs to best support community WASH and health service delivery? What are the specific skills areas that have improved, remained or dropped? N06 E24 What opportunities available for WASH entrepreneurs that are still not fully utilized to improve community WASH and health programs/service delivery, which future initiatives can make use of? N07 E24 Are there any stakeholders who did not play much visible role in PACS WASH component that you believe should have? And why? N08 E22 What would you consider as the top three achievements of PACS related to increasing access and improving service delivery under WASH and why? N09 E23 What would you consider as the top three areas where PACS could have done better with WASH? Please give reasons. Examples: collaboration, communication, etc. N10 E14a E24 What were the main limitations and challenges faced by the WASH entrepreneurs doing PACS’ implementation? How did were they addressed and to what extent? 71 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV Form P – FGD: CHC and HFDC (including DHO) About the Tool This tool will capture the experiences of CHC/HFDC/DHO to gather a deeper understanding of the PACS design, implementation,results and lessons learned, how the stakeholders have implemented the project, the challenges they have faced in working with PACS, and their views on effectiveness and sustainability. The FGD will be conducted by trained moderators and note-takers from among the enumerators and supervisors MODERATOR NAME __________________________________ NOTE TAKER NAME ___________________________________ DATE (DD/MM/YY)…………………………….…………………………… ORGANIZATION NAME _______________________________ ORGANIZATION CODE ________________________________ LOCALITY ___________________________________ SUB-LOCALITY ___________________________________ FGD CODE …………………………………………….………………… No. PARTICIPANTS AT START OF FGD……………(FEMALE)………….. No. PARTICIPANTS AT START OF FGD……………(MALE)……………….. No. PARTICIPANTS AT THE END OF FGD…………(FEMALE)…..………. No. PARTICIPANTS AT THE END OF FGD…………(MALE)……..………. RECORDER NUMBER………………………………………………… START TIME……………………………………………………………….… H MN END TIME………………………………………………….………………… H MN *: Focus-Group Codes: Team Leader to provide FGD codes, if required. Introduction AT THE START, inform the participants about the purpose of the interview/discussion, and go through the informed consent process. Keep the signed consent forms with you. We are interested in getting information about your experiences, and that of CHC/HFDC/DHO regarding PACS including your engagement, understanding and opinions. Your frank responses and participation will be most helpful to us as we try to really understand your views. Your responses will not be considered “right” or “wrong” as we are mainly interested in knowing what you think. Everyone’s views are equally important. It’s fine to disagree with other people’s views, but if you do, it’s important to disagree in a respectful and polite manner. It’s important for you to talk in turns, because if you all speak at once, we will not be able to hear you and understand. If you disagree with something anyone says, you can say ‘I disagree’ and then wait for them to finish before you speak. MODERATOR • EXPLAIN THE ROLE OF NOTE-TAKERS AND AUDIO-RECORDER • ASK THE PARTICIPANTS TO FILL IN THE SIGN-IN SHEET (ADULT or CHILD TEMPLATE PROVIDED) • ALLOW RESPONDENTS A FEW MINUTES FOR ANSWERING QUESTIONS REGARDING THE FGD ___________________________________________________________________________ Code Question Probes/prompts and notes P01 E12 VERY BRIEFLY – describe your role(s), and how you have worked with PACS. What support has been provided to you or your team through PACS? Examples: capacity building, materials, technical assistance, managerial, operational, etc. P02 E12 What do you consider as your strongest skills and competencies based on your involvement with PACS? How do you apply those skills and competencies in your work, day to day, weekly, monthly? Examples – new skills in management or technical work, community engagement, collection or use of data? How will you use those skills after PACS ends? P03 E12 What areas of PACS capacity building did not work very well? Why did that happen? Encourage a balanced view on capacity activities. P04 E10 E22 Looking at PACS integrated community health and water, sanitation and hygiene (WASH) together - how well do you think that integration worked? Do you think that integrating would work for community health and WASH with other services, such as nutrition? Social protection? Any other services or sectors that are ready for integration with community health? Probe for examples and evidence to support the claims. What specifically has the integration affected positively and negatively? How was gender a factor in integrating health and WASH? Community engagement? P05 E13 Which health promotion and behavior change messages have community members been the most receptive to? Any that have not been as successful? Why do you think this is? How, if at all, have these messages changed from the beginning of the PACS project to now? Radio, posters, mass media Waterguard ODF P06 E11 How have you worked with the CSOs supported by PACS? What worked well with this collaboration, and was anything not so successful? In what areas do you feel that CSOs have the most added value in your district? Are there activities you would like to see them more involved in? Level of county and district involvement with CSO Health promotion & BCC, Monitoring How can CSO’s continue PACS’ work after PACS closes? P07 E12 E23 What were the outcomes of the CHC mapping exercises? Were there any lessons learned from this exercise? Have there been any discussion about what/how this data can be used in the future? How would you describe the quality of the data collected by CHAs? How is this used for decision making? Has the data quality changed over time? CBIS data collection tools? Other data collected? DQAs? 73 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV P08 E11 E12 Are there any key activities or support provided to you that has changed considerably since the beginning of the project to now? Why do you think this happened? P09 E13 Where might the community face challenges in upholding new behaviors over time? And for maintaining ODF status, or going back to OD? What happens when support to the community is stopped? How can CHAs continue the changes? P10 E10 How do you work differently with women and men? What lessons are there? Anything that could have been better or different, to achieve good community health and WASH with women? P11 E12 E23 In hindsight, what worked and what didn’t, for increasing access and improving delivery of community health and WASH? Looking forward, how do you think access and quality in community health can be improved – anything new that could be tried? What could have been done differently or better, to improve the access and quality of community health and WASH? What kinds of innovations are you aware of or can be tried, given the experience with PACS? P12 E13 E24 Likewise, for working with communities, what parts of PACS worked well and what didn’t work as expected? What could have been done differently or better with communities? Tell me more about how PACS worked with women, and what lessons you see? P13 E24 What about future work, should any traditional groups or authorities be involved, to support community health and WASH? Chiefs, women’s groups, clan elders. How did PACS work with traditional groups – and, what worked or could have been done differently/better? 75 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV Form Q - FGD: Community Health Assistants (CHAs) About the Tool: This tool will capture the experiences of CHAs about the PACS design, implementation, results and lessons learned, how the stakeholders have implemented the project, the challenges they have faced in working with PACS, and their views on effectiveness and sustainability. The FGD will be conducted by trained moderators and note-takers. MODERATOR NAME __________________________________ NOTE TAKER NAME ___________________________________ DATE (DD/MM/YY)……………….…………………………………….. ORGANIZATION NAME _______________________________ ORGANIZATION CODE ________________________________ LOCALITY ___________________________________ SUB-LOCALITY ___________________________________ FGD CODE …………………………………………….……… No. PARTICIPANTS AT START OF FGD……………(FEMALE)………….. No. PARTICIPANTS AT START OF FGD……………(MALE)……………….. No. PARTICIPANTS AT THE END OF FGD…………(FEMALE)…..………. No. PARTICIPANTS AT THE END OF FGD…………(MALE)……..………. RECORDER NUMBER…………………………………………………… START TIME……………………………………………………………… H MN END TIME………………………………………………….……………… H MN Introduction AT THE START, inform the participants about the purpose of the interview/discussion, and go through the informed consent process. Keep the signed consent forms with you. We are interested in getting information about your experiences, and that of other CHAs regarding PACS including your engagement, understanding and opinions. Your frank responses and participation will be most helpful to us as we try to really understand your views. Your responses will not be considered “right” or “wrong” as we are mainly interested in knowing what you think. Everyone’s views are equally important. It’s fine to disagree with other people’s views, but if you do, it’s important to disagree in a respectful and polite manner. It’s important for you to talk in turns, because if you all speak at once, we will not be able to hear you and understand. If you disagree with something anyone says, you can say ‘I disagree’ and then wait for them to finish before you speak. MODERATOR • EXPLAIN THE ROLE OF NOTE-TAKERS AND AUDIO-RECORDER • ASK THE PARTICIPANTS TO FILL IN THE SIGN-IN SHEET (ADULT or CHILD TEMPLATE PROVIDED) • ALLOW RESPONDENTS A FEW MINUTES FOR ANSWERING QUESTIONS REGARDING THE FGD Code Question Probes/prompts and notes Q01 E10 VERY BRIEFLY – describe your role(s), and how you have worked with PACS. What support has been provided to you or your team through PACS? Q02 E11 What areas of PACS capacity building did not work very well? Why did that happen? What are the current gaps in capacity? Encourage a balanced view on capacity activities. Q03 E13 Which health promotion and behavior change messages have community members been the most receptive to? Any that have not been as successful? Why do you think so? Child health, Delivery in facilities, Water Guard, ODF, hygiene Q04 E12 In what way do the referrals to health facilities work, and how can those be improved? And referrals from facilities to CHAs? Quality or frequency of communication, feedback on diagnosis and treatment, types of diseases being referred, etc. Q05 E12 E13 What can you tell us about the provision of drugs and other health commodities to CHAs, for use in communities? Regular supply? What happens during stock-outs? Community perceptions? Q06 E12 What kinds of changes have taken place in your communities because of the CHA program? Any changes not related to health? Nutrition, farming, how CHAs incentive payments are used? Q07 E14 E22 How much are CHAs involved in the community WASH activities? Probe for examples and evidence to support the claims. Community engagement? Q08 E22 What other, additional skills do you think would be useful for your work, without adding too much as a burden? Nutrition, gender-based violence, social protection (birth registration, orphaned children, etc.), Q09 E23 In hindsight, what worked and what didn’t, for increasing access and improving delivery of community health and WASH? Looking forward, how do you think access and quality in community health can be improved – anything new that could be tried? What could have been done differently or better, to improve the access and quality of community health and WASH? What kinds of innovations are you aware of or can be tried, given the experience with PACS? Q10 E12 How do you work with women and men differently in your communities? Most CHAs are male – how does that work in the community? Q11 E13 E24 Likewise, for working with communities, what parts of PACS worked well and what didn’t work as expected? What could have been done differently or better with communities? Tell me more about how PACS worked with women, and what lessons you see? Anything that could have been better or different, to achieve good community health and WASH with women? Q12 E24 What about future work, should any traditional groups or authorities l be involved, to support health and WASH? Traditional groups – and, what could have been done differently/better? Q13 E12 E23 What happens to you and your work as a CHA after PACS closes? Continue without incentives, teach others to start as CHA, just stop working? Q13 E10 Finally, tell us what communities value the most from your work as CHAs? Form R – KII: Community Health Services Supervisors (CHSS) and Facility Officers-in-Charge (OICs) About the Tool 77 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV This tool will capture the perceptions of Community Health Services Supervisors (CHSS) and Health Facility Officers-in-Charge (OICs) about the PACS design, implementation, results, lessons, stakeholders’ engagement, challenges faced in working with PACS, and their views on effectiveness and sustainability. INTERVIEWER NAME__________________________________ DATE (DD/MM/YY)…………………………….………… START TIME……………………………………………………………… H M N END TIME………………………………………………….…………… H M N Introduction AT THE START, inform the participant about the purpose of the interview/discussion, and go through the informed consent process. Keep the signed consent form with you. We are interested in learning about your experience with PACS, including your perceptions, opinions and experiences regarding implementation, and your views on its effectiveness and sustainability. Your frank responses and participation will be most helpful to us to understand your views. Questions Respondent Details Information Name of the Organization Organization code (if any) Location Locality (…………………………………..) Sub-Locality (…………………………………..) Respondent(s) Name(s) Position Title Gender Female / Male How many years have you been in this position? Code Question Probes/prompts and notes R01 E10 VERY BRIEFLY – describe your role(s), and how you have worked with PACS. Tell us about the work in supervising CHAs. What support has been provided to you or your team through PACS? R02 E11 What areas of PACS capacity building did not work very well? Why did that happen? What are the current gaps in capacity? Encourage a balanced view on capacity activities. R03 E13 Which health promotion and behavior change messages have community members been the most receptive to? Any that have not been as successful? Why do you think so? Child health, Delivery in facilities, Water Guard, ODF, hygiene R04 E12 In what way do the referrals to health facilities work, and how can those be improved? Quality or frequency of communication, feedback on And referrals from facilities to CHAs? diagnosis and treatment, types of diseases being referred, etc. R05 E12 E13 What can you tell us about the provision of drugs and other health commodities to CHSSs and CHAs, for use in communities? Regular supply? What happens during stock-outs? Community perceptions? R06 E12 What kinds of changes have taken place in your communities because of the National CHA program? Any changes not related to health? Nutrition, farming, how CHSS/CHAs incentive payments are used? R07 E12 How do you work with women and men differently in your communities? Most CHSSs and CHAs are male – how does that work in the community? R08 E22 What other, additional skills do you think would be useful for your work, without adding too much as a burden? Nutrition, gender-based violence, social protection (birth registration, orphaned children, etc.), R09 E14 E22 How much are OICs / CHSSs and CHAs involved in the community WASH activities? Probe for examples and evidence to support the claims. Community engagement? R10 E23 In hindsight, what worked and what didn’t, for increasing access and improving delivery of community health and WASH? Looking forward, how do you think access and quality in community health can be improved – anything new that could be tried? What could have been done differently or better, to improve the access and quality of community health and WASH? What kinds of innovations are you aware of or can be tried, given the experience with PACS? R11 E13 E24 Likewise, for working with communities, what parts of PACS worked well and what didn’t work as expected? What could have been done differently or better with communities? Tell me more about how PACS worked with women, and what lessons you see? Anything that could have been better or different, to achieve good community health and WASH with women? R12 E24 What about future work, should any traditional groups or authorities like other ministry be involved, to support community health and WASH? How did PACS work with traditional groups – and, what worked or could have been done differently/better? R13 E10 Finally, tell us what communities value the most from your work? 79 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV Form T - FGD: Community members (male and female, conducted separately) About the Tool: This tool will capture the experiences of female and male community members (separately) about the PACS design, implementation, results, lessons learned, stakeholders’ engagement, the challenges they have faced in working with PACS, and their views on effectiveness and sustainability. The FGD will be conducted by trained moderators and note-takers. MODERATOR NAME __________________________________ NOTE TAKER NAME ___________________________________ DATE (DD/MM/YY)……………….…………………………………….. ORGANIZATION NAME _______________________________ ORGANIZATION CODE ________________________________ LOCALITY ___________________________________ SUB-LOCALITY ___________________________________ FGD CODE …………………………………………….…………………… No. PARTICIPANTS AT START OF FGD……………(FEMALE)………….. No. PARTICIPANTS AT START OF FGD……………(MALE)……………….. No. PARTICIPANTS AT THE END OF FGD…………(FEMALE)…..………. No. PARTICIPANTS AT THE END OF FGD…………(MALE)……..………. RECORDER NUMBER…………………………………………………… START TIME……………………………………………………………….……. H MN END TIME………………………………………………….……………………………… H MN Introduction AT THE START, inform the participants about the purpose of the interview/discussion, and go through the informed consent process. Keep the signed consent forms with you. We are interested in getting information about your experiences, and that of other CHAs regarding PACS including your engagement, understanding and opinions. Your frank responses and participation will be most helpful to us as we try to really understand your views. Your responses will not be considered “right” or “wrong” as we are mainly interested in knowing what you think. Everyone’s views are equally important. It’s fine to disagree with other people’s views, but if you do, it’s important to disagree in a respectful and polite manner. It’s important for you to talk in turns, because if you all speak at once, we will not be able to hear you and understand. If you disagree with something anyone says, you can say ‘I disagree’ and then wait for them to finish before you speak. MODERATOR 80 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV • EXPLAIN THE ROLE OF NOTE-TAKERS AND AUDIO-RECORDER • ASK THE PARTICIPANTS TO FILL IN THE SIGN-IN SHEET (ADULT or CHILD TEMPLATE PROVIDED) • ALLOW RESPONDENTS A FEW MINUTES FOR ANSWERING ANY QUESTIONS REGARDING THE FGD Code Question Probes/prompts and notes T01 E10 You may remember that starting in 2016-17, CHAs began working in your community. The CHAs also work with health facility staff and with other people on water and sanitation (toilets). We want your open opinions on what is working, what needs improvement and what new things can be added. Think of us as your messengers, to bring your voice and ideas to the officials in government and partners. VERY BRIEFLY – describe changes you have experienced or seen in health facilities and CHA/CHVs. What can be better? What new things should be added? [Set the context] Listen for changes in CHA program: - what types of services CHAs do: ANC, postnatal, family planning, advice on disease prevention, treating malaria, diarrhea, pneumonia - CHAs make referrals and follow-up after you go to the clinic - CHAs have drugs and commodities T02 E13 Which health promotion and behavior change messages have community members been the most receptive to? Any that have not been as successful? Why do you think so? Child health, Delivery in facilities, Water Guard, ODF, hygiene T03 E12 In what way do the CHAs make referrals to health facilities, and how can those be improved? What happens after you come back from the clinic? Communication about treatment Does the clinic give you a paper/referral to the CHA? Does CHA come to see you again? T04 E12 E13 Does the CHA usually have the right medicine for you when you or your child is unwell? Explain…. If not, where do you find the medicines you need? ORS for Diarrhea Family planning methods Malaria medication Antibiotic for fever or cough T05 E12 What kinds of changes have taken place in your communities because of the CHA program? Any changes not related to health? Nutrition, farming, how CHAs incentive payments are used? T06 E12 How to male CHAs work with women who have special female health needs? Do the husbands get involved? Are other women there to assist? T07 E14 E22 How much are CHAs involved in the community WASH activities? Probe for examples and evidence to support the claims. Community engagement? T08 E22 E24 What other thing would you like to see the CHAs do in the community, without adding too much as a burden on them? Nutrition, gender-based violence, social protection (birth registration, orphaned children, etc.), T09 E24 Which other people can help improve health in your community, such as traditional groups or leaders? Chiefs – which types? Other types of community leaders? T10 E10 Finally, tell us what you and your community like the most from the CHAs? 81 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV Form U - KII: Civil Society Organizations (CSO) – county level About the Tool This tool will capture the experiences and perceptions of CSO directors and program managers to gather a deeper understanding of the PACS design, implementation, results and lessons learned, stakeholders’ engagement, challenges faced in working with PACS, and their views on effectiveness and sustainability. INTERVIEWER NAME__________________________________ DATE (DD/MM/YY)…………………………….…………………… START TIME……………………………………………………………….… H M N END TIME………………………………………………….…………………… H M N Introduction AT THE START, inform the participant about the purpose of the interview/discussion, and go through the informed consent process. Keep the signed consent form with you. We are interested in learning about your experience with PACS, including your perceptions, opinions and experiences regarding implementation, and your views on its effectiveness and sustainability. Your frank responses and participation will be most helpful to us to understand your views. Questions Respondent Details Information Name of the Organization Organization code (if any) Location Locality (…………………………………..) Sub-Locality (…………………………………..) Respondent(s) Name(s) Position Title Gender Female Male How many years have you been in this position? Code Question Probes/prompts and notes U01 E11 Briefly describe your role, and how you have worked with PACS. To put the respondent at ease. Listen for points to follow-up. U02 E11 Tell us about the capacity building you received from PACS – which aspects were more useful, and less useful? Did the support you received change at all over your time with PACS? If so, in what ways? - Governance, strategy - Institution and management - Technical areas: health, WASH, … 82 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV Code Question Probes/prompts and notes U03 E10 In each area of PACS work, what do you consider to be the (a) lasting accomplishment, and (b) less successful? - Capacity building: MOH, other government, CSOs - CHA program and iCCM - Behavior change, Water Guard - CLTS, wells, hygiene Encourage a balanced view on PACS activities. After the responses, be sure to cover the two points: - community engagement - capacity of GOL and partners U04 E12 To what extent did PACS’s support to CHAs make any improvements in their work with the communities? What else do CHAs and CHSSs need to do their job well? U05 E20 E23 Looking back at the work of PACS and the situation today, what are the remaining gaps in community health and WASH, that a new project will have to take on? U06 E21 E23 With PACS coming to a close, who are the local partners that can take on or continue the work that PACS started? Other CSOs? CHT, other GOL Functionality of CHCs, HFDCs U07 E21 How would you prioritize the key areas of capacity that still need to be built or supported in the next 5 years for your CSO and other CSOs, to improve community health and WASH service delivery? And for the CHT, DHTs and other government offices? Are there any skill areas that have improved, remained stagnant, or declined? Are there any specific benchmarks to monitor that will identify progress? (financial, technical, ownership) U08 E21 To what extent do you believe that the government – county or national – will take on the role(s) from PACS? Why do you think so? Which other groups might take on some of the roles from PACS, after the project ends? Kindly describe the indications of where the gap may be, in their taking on PACS’ work? What signs or evidence show that the GOL has the capacity for those roles, or not? What about the commitment to take on that role? U09 E11a We have seen your OCA from 2016. Is there any update? What has changed in your CSO’s capacity? How has your CSO used the information from any update? Request a copy of any update or other document that tracks the CSO capacity. U10 E24 From your perspective, what could be done differently or better with communities, to improve health and WASH? What about future work, should other ministries or agencies be involved, specifically traditional structures? Anything that could have been better or different, to achieve good community health and WASH? Traditional groups? Ministry of Internal Affairs, etc, private sector U11 E23 Based on experience with PACS, what suggestions would you make to others about building capacity of CSOs? 83 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV Form V – FGD: Civil Society Organizations (CSOs) representatives About the Tool This tool will capture the perceptions and experiences of CSO representatives regarding the PACS design, implementation, results, lessons learned, how the stakeholders have implemented the project, the challenges they have faced in working with PACS, and their views on effectiveness and sustainability. MODERATOR NAME __________________________________ NOTE TAKER NAME ___________________________________ DATE (DD/MM/YY)…………………………….…………………… ORGANIZATION NAME _______________________________ ORGANIZATION CODE ________________________________ LOCALITY ___________________________________ SUB-LOCALITY ___________________________________ FGD CODE …………………………………………….………………… No. PARTICIPANTS AT START OF FGD……………(FEMALE)………….. No. PARTICIPANTS AT START OF FGD……………(MALE)……………….. No. PARTICIPANTS AT THE END OF FGD…………(FEMALE)…..………. No. PARTICIPANTS AT THE END OF FGD…………(MALE)……..………. RECORDER NUMBER…………………………………………………. START TIME……………………………………………………………….……… H M N END TIME………………………………………………….…………………… H M N *: Focus-Group Codes: Team Leader to provide FGD codes, if required. Introduction AT THE START, inform the participants about the purpose of the interview/discussion, and go through the informed consent process. Keep the signed consent forms with you. We are interested in learning about your experiences and that of your group regarding PACS, including your engagement, understanding and opinions. Your frank responses and participation will be helpful to us as we try to really understand your views. Your responses will not be considered “right” 84 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV or “wrong” as we are mainly interested in knowing what you think. Everyone’s views are equally important. It’s fine to disagree with other people’s views, but if you do, it’s important to disagree in a respectful and polite manner. It’s important for you to talk in turns, because if you all speak at once, we will not be able to hear you and understand. If you disagree with something anyone says, you can say ‘I disagree’ and then wait for them to finish before you speak. MODERATOR ● EXPLAIN THE ROLE OF NOTE-TAKERS AND AUDIO-RECORDER ● ASK THE PARTICIPANTS TO FILL IN THE SIGN-IN SHEET (ADULT or CHILD TEMPLATE PROVIDED) ● ALLOW RESPONDENTS A FEW MINUTES FOR ANY QUESTIONS REGARDING THE FGD ___________________________________________________________________________ Code Question Probes/prompts and notes V01 E11 Briefly describe how you have worked with PACS. To put the group at ease. Listen for points to follow-up. V02 E11 Tell us about the capacity building you received from PACS – which aspects were more useful, and less useful? - Governance, strategy - Institution and management - Technical areas: health, WASH, … V03 E10 In each area of PACS work, what do you consider to be the (a) lasting accomplishment, and (b) less successful? - Capacity building: MOH, other government, CSOs - CHA program and iCCM - Behavior change, Water Guard - CLTS, wells, hygiene Encourage a balanced view on PACS activities. After the responses, be sure to cover the two points: - community engagement - capacity of GOL and partners V04 E12 To what extent did PACS’s support to CHAs make any improvements in their work with the communities? What else do CHAs and CHSSs need to do their job well? V05 E20 E23 Looking back at the work of PACS and the situation today, what are the remaining gaps in community health and WASH, that a new project will have to take on? V06 E21 E23 With PACS coming to a close, who are the local partners that can take on or continue the work that PACS started? Other CSOs – besides respondent? CHT, other GOL Functionality of CHCs, HFDCs V07 E21 How would you prioritize the key areas of capacity that still need to be built or supported in the next 5 years for your CSO and other CSOs, to improve community health and WASH service delivery? And for the CHT, DHTs and other government offices? Are there any skill areas that have improved, remained stagnant, or declined? Are there any specific benchmarks to monitor that will identify progress? (financial, technical, ownership) V08 E21 To what extent do you believe that the government – county or national – will take on the role(s) from PACS? Why do you think so? Which other groups might take on some of the roles from PACS, after the project ends? Kindly describe the indications of where the gap may be, in their taking on PACS’ work? What signs or evidence show that the GOL has the capacity for those roles, or not? What about the commitment to take on that role? 85 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV Code Question Probes/prompts and notes V09 E10 E23 There are several areas of work under PACS, such as – - CHT capacity building - Support to CHA program, iCCM - community engagement - gender mainstreaming - healthy behaviors: bednets, antenatal care, nutrition - safe water/WaterGuard use/drilling wells - CLTS/ODF; community hygiene Who are the other groups that have the capacity to continue this work after PACS? What other support – besides funding – is required to do this work successfully? V10 E21 Step by step, how would CSOs move ahead in doing the work after PACS has closed? What are the specific activities you would be able to take on? V11 E24 From your perspective, what could be done differently or better with communities, to improve health and WASH? What about future work, should other ministries or agencies be involved, specifically traditional structures? Anything that could have been better or different, to achieve good community health and WASH? Traditional groups? Ministry of Internal Affairs, etc, private sector V12 E23 Based on your experience with PACS, what suggestions would you make to others about strengthening the capacity of CSOs? 87 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV Form W: Civil Society Organization (CSO) – project profile About the Tool: This tool will capture basic information about PACS-supported CSOs, in relation to their project portfolios and organizational capacity as of May 2019. Kindly complete the information below for each project (current or recent past) implemented by the CSO. You may enter the requested information in the white unshaded rows. For project documentation, you may list the documents that are available, and send email them to Ben bkitson@socialimpact.com and Ravi rram@socialimpact.com . You may also call us on 0886-945144 or 0777-474507 with any questions. Project 1 Project name Start date End date Total budget US$__._ Donor(s) County & districts of implementation Local and international partners GOL partners - Health - Other Community engagement process Project description and scope of work List of documentation for the project (e.g., funding contract, donor agreement, etc.) (to be provided by email) Project 2 Project name Start date End date Total budget US$__._ Donor(s) County & districts of implementation Local and international partners GOL partners - Health - Other Community engagement process Project description and scope of work List of documentation for the project (e.g., funding contract, donor agreement, etc.) (to be provided by email) 88 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV Project 3 Project name Start date End date Total budget US$__._ Donor(s) County & districts of implementation Local and international partners GOL partners - Health - Other Community engagement process Project description and scope of work List of documentation for the project (e.g., funding contract, donor agreement, etc.) (to be provided by email) Project 4 Project name Start date End date Total budget US$__._ Donor(s) County & districts of implementation Local and international partners GOL partners - Health - Other Community engagement process Project description and scope of work List of documentation for the project (e.g., funding contract, donor agreement, etc.) (to be provided by email) Organizational Capacity Using the table below, kindly fill in your OCA scores from 2016 when you worked with PACS on the Capacity Self-Assessment. In the 2019 column, enter scores for any update on your OCA. You may make additional comments in the space provided. 89 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV Competency area Score Comments 2016 2019 1. Governance 2. Administration 3. Human resources 4. Financial management 5. Organization management 6. Project management 7. Project performance 8. Technical capacities Describe your CSO’s organizational capacity and experience in working with communities. What is your model for community engagement? Include any specific experience to work with CHCs, CWSCs or other community self-help group. There are several areas of community health and WASH work under PACS. Kindly select and rank the top 1, 2 and 3 that you feel are most closely in alignment with your CSO’s mission and vision? PACS work Ranking Comments CHT capacity building Support to CHA program, iCCM Community engagement Gender mainstreaming Behavior change/communication Safe water Drilling wells CLTS/ODF Community hygiene How does your CSO demonstrate or measure its capacity in these areas? (Examples: policy document, specific training, specific resources, etc.) What other support – besides funding – would you need to do this work successfully? Step by step, how would your organization move ahead in doing the work after PACS has closed? What are the specific activities you would be able to take on? 90 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV Form Y – KII: Ministry of Internal Affairs (MIA), Traditional Chiefs About the Tool This tool will capture the perceptions and experiences of MIA and Chiefs about the PACS design, implementation, results, lessons, stakeholders’ engagement, challenges faced in working with PACS, and their views on effectiveness and sustainability. INTERVIEWER NAME__________________________________ DATE (DD/MM/YY)…………………………….……………………… START TIME……………………………………………………………….……. H M N END TIME………………………………………………….…………………… H M N Introduction AT THE START, inform the participant about the purpose of the interview/discussion, and go through the informed consent process. Keep the signed consent form with you. We are interested in learning about your experience with PACS, including your perceptions, opinions and experiences regarding implementation, and your views on its effectiveness and sustainability. Your frank responses and participation will be most helpful to us to understand your views. Questions Respondent Details Information Name of the Organization Organization code (if any) Location Locality (…………………………………..) Sub-Locality (…………………………………..) Respondent(s) Name(s) Position Title Gender Female / Male How many years have you been in this position? Code Question Probes/prompts and notes Y01 E10 VERY BRIEFLY – describe your role, and how you have worked with PACS. How do you work with or oversee CHAs and WASH? When did your office get involved, and why? What is your role in the activities? Y02 E11 What support has been provided to you or your team through PACS? How has that been useful? What are the current gaps in capacity? Encourage a balanced view on capacity activities. Y03 E12 E13 What parts of the program have been most successful? Any that have not been as successful? Why do you think so? Capacity building, CHA program, Child health, Delivery in facilities, Water Guard, ODF, wells, hygiene 91 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV Y04 E12 E23 From your understanding of the PACS community health and WASH project, what can you suggest about how it works, and how it can be improved in the future? Accountability with drugs/supplies CBIS reporting Y05 E12 What kinds of changes have taken place in your communities because of the National CHA program? Any changes not related to health? Nutrition, farming, how CHSS/CHAs incentive payments are used? Y06 E12 How do you work with women and men differently in your communities? Who are the traditional female leaders? Other focus on women? Y07 E22 E23 Tell us about the new Social Registry. What is the purpose? What will it do? How will the social registry support health services? How will it support WASH or other public services? Find timeline for Social Registry. What does it build on? Y08 E22 E23 E24 For a new project, what other, new areas do you think would be useful to include, to support community health and WASH? Some groups say that MIA has a role in community health and WASH. Do you agree? If so, what would that role ? Nutrition, gender-based violence, social protection (birth registration, orphaned children, etc.), Chiefs apply laws and rules on community sanitation, hygiene? Y09 E24 For a new project, how should any traditional groups or authorities be involved, to support community health and WASH? Types of Chiefs, other community leaders, religious leaders, chair of women’s group, youth leaders, etc. Y10 E24 Which other ministries or government offices should also be involved in community health and WASH? What would be their roles? How would those office contribute to the project, or is their involvement for other reasons? Y11 E23 In what ways have you worked with CSOs on health and WASH? What do you think about their work? Relationships between CSOs and GOL Can CSOs sustain PACS activities? 92 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV Form Z – KII: Trained Traditional Midwives (TTM) About the Tool: This tool will capture the experiences of TTMs about the PACS design, implementation, results, lessons learned, stakeholders’ engagement, the challenges they have faced in working with PACS and the National CHA Program, and their views on effectiveness and sustainability. INTERVIEWER NAME__________________________________ DATE (DD/MM/YY)…………………………….……………………… START TIME……………………………………………………………… H M N END TIME………………………………………………….……………… H M N Introduction AT THE START, inform the participant about the purpose of the interview/discussion, and go through the informed consent process. Keep the signed consent form with you. We are interested in learning about your experience with PACS, including your perceptions, opinions and experiences regarding implementation, and your views on its effectiveness and sustainability. Your frank responses and participation will be most helpful to us to understand your views. Respondent Details Information Name of the Organization Organization code (if any) Location Locality (…………………………………..) Sub-Locality (…………………………………..) Respondent(s) Name(s) Position Title Gender Female Male How many years in position? Code Question Probes/prompts and notes Z01 E12 You may remember that starting in 2016-17, CHAs began working in your community. The CHAs also work with health facility staff about pregnancy and baby deliveries. We want your open opinions on what is working, what needs improvement and what new things can be added. [Set the context] How doe TTMs see the CHA program Z02 E12 Compared to 4-5 years ago, how has your work as a TTM changed? Care for pregnant women? Respect and role in the community? Z03 E12 E24 In what way do the CHAs and health clinic staff involve you as a TTM in caring for the baby ma (pregnant/after delivery) How do you think that can be improved, if at all? Communication about care for the baby ma. Respect for the TTM Z04 E24 In what ways to you interact with other TTMs? What kinds of topics do you talk about? Do you know of any group or organization of TTMs, even if it meets irregularly or infrequently? How do TTMs interact or coordinate? How do TTMs train new midwives? Z05 E12 What other traditional leaders or groups do you suggest should be part of health projects, particularly the CHA program? Nutrition, farming, how CHAs incentive payments are used? Z06 E24 Which other people can help improve health in your community, such as traditional groups or leaders? Chiefs – which types? Other types of community leaders? Z07 E10 Finally, tell us what you and your community like the most about your work as a TTM? 93 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV ANNEX E: EVALUATION STATEMENT OF WORK PARTNERSHIP FOR ADVANCING COMMUNITY-BASED SERVICES (PACS) Final Performance Evaluation Statement of Work 1. Background This Statement of Work (SOW) describes the conditions of work and terms of reference for a final performance evaluation of USAID’s Partnership for Advancing Community-based Services (PACS) activity. The objective of PACS is to strengthen the delivery of high quality community health, water, sanitation and hygiene, and social welfare services by providing technical and operational support to the Ministries of Health (MOH), Public Works (MPW), and to civil society organizations (CSO). The evaluation will measure the progress and sustainability of PACS while also providing an evidence base for the new community health activity design. The Mission will use findings from the evaluation to inform more strategic USAID investment in the sector and continue support to the delivery of community health services. Overview of PACS Activity: Partnership for Advancing Community-based Services (PACS) Contract Number: AID-669-A-15-00001 Activity Dates: February 23, 2015 – February 22, 2020 Funding: $36.8 Million (original 24.8 Million + EEF 7 Million + CEBS 5 Million) Implementing Partner: International Rescue Committee Subs: Global Communities, Population Services International, and Young Men Christian Association of Liberia AOR: Jannie M. Horace A/AOR: Wondwossen Teffera Geographical focus: Bong, Lofa, Nimba Since 2008, USAID has supported the rebuilding of Liberia’s health system through its flagship Rebuilding Basic Health Services Activity (RBHS) that provided critical support to the Ministry of Health. To build on the successes of RBHS and to support the MOH in the implementation of the 2011-2021 National Health Policy and Plan, USAID entered into a Fixed Amount Reimbursement Agreement (FARA) with the MOH to build stewardship of basic health service delivery. Part of this holistic approach included investing in community health delivery. USAID signed a Cooperative Agreement with the International Rescue Committee and its partners in February 2015 to implement the Partnership for Advancing Community-based Services (PACS) activity. The overall purpose of PACS is to advance USAID Liberia’s Country Development and Cooperation strategy (CDCS) Development Objective 3 (DO3), of “Improved Health Status of Liberians." The USAID/Liberia Results Framework is provided in Annex I. To link PACS to the overall USAID/Liberia objectives, refer to the PACS Results Framework in Annex II. PACS is designed to contribute to DO3 through the following objectives: • Broadened capacity of MOH, County Health Teams (CHTs), Non-Governmental Organizations and community organizations to implement and manage community services; • Increased the availability of community-based health and social welfare services; • Improved health-seeking behavior and practice; and • Improved access to safe water supply and sanitation at the community level. 94 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV The main priority of PACS is to enhance the capacity of the MOH, MPW, and CHTs to support community-based delivery of health and social welfare services and increase utilization through health communications and support of community structures. The expected results of PACS are closely aligned with those of the National Health Policy and Plan (2011-2021) and the Investment Plan to Building a Resilient Health System (2016-2021), the Revised Community Health Services Policy and Strategic Plan (2016-2020) and implementation has been done in close collaboration with the MOH and the CHTs in the USAID priority counties of Bong, Lofa, and Nimba. PACS areas of implementation are co-located with FARA-managed health facilities, although other donors work in the same counties also providing community health services. PACS activities are focused in the three counties: Bong, Lofa, and Nimba. PACS does not support NCHAP activities in all districts of the three counties. The map in Annex III shows districts where PACS supports activities to scale-up and strengthen the NCHAP (e.g., health service delivery, health promotion and behavior change communication) and expand access to WASH facilities; and districts where only WASH activities are supported by PACS. Results Framework: In addition to indicators to measure progress toward achievement of targets set for the four result areas and the overall development objective, the Activity Level Monitoring and Evaluation Plan (ALMEP) includes results and sub-results for tracking lower-level indicators of progress and outcome indicators of achievements. The PACS Results Framework contributes to the four IRs of USAID’s DO3, “Improved Health Status of Liberians.” The Results Framework and how it aligns with DO3 is included in Annex II. Result 1: Broadened capacity of central MOH, CHTs, local CSOs and NGOs to implement and manage community services. Activities include obtaining and maintaining stakeholder agreement at national and county levels; establishing partnership agreements with MOH units and CHTs; and facilitating self-assessments and performance improvement plans will lead to ensuring that the MOH have the systems and skills necessary to effectively plan manage and monitor community health services. These activities will enable the GOL to successfully increase availability of community-based health and social welfare services. Result 2: Increased service quality and availability of community-based health and social welfare services. Activities include supporting MOH to review, develop, and roll out an updated, standardized, and integrated package of community health and social welfare services to be delivered by community health assistants (CHA) working in areas more than 5 kilometers from a health facility. CHSS are trained and deployed to supervise the CHAs and CHVs thereby improving the provision of health services according to MOH guidelines and protocols, at the community level. PACS is also engaging CSOs to strengthen linkages between communities and health facilities and improve accountability within the health system. Result 3: Improved health-seeking behavior and practices. Ensuring the availability of health services at the community will only be effective if those communities are informed about the services and adopt more positive health behaviors. Result 3 activities will generate information and behavior change strategies at the community level to increase demand for quality health services. Result 4: Improved access to safe WASH services. Activities include capacity building of water and sanitation personnel to effectively manage WASH infrastructure; supporting GOL in deploying trained and equipped water and sanitation staff; supporting 95 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV GOL staffing and logistics gaps to improve WASH services; and strengthening county and district level focal points for WASH infrastructure and maintenance. PACS Theory of Change (TOC): - If the MOH and local CSOs receive technical and financial assistance that are well targeted, of high quality, and to scale, and if the Liberia health system has the absorptive capacity, then they [MOH and local CSOs] will have broadened capacity to implement and manage community-based health services in remote communities that are more than 5 km from health facilities. - If the MOH and CSOs have broadened capacity to conduct health promotion and behavior change communication activities. And if caregivers consistently practice healthy behaviors such as ensuring their children sleep under insecticide-treated nets; practicing hand washing before preparing food, eating and after toileting; adhering to schedules to fully immunize children; following recommendations for exclusive breastfeeding and appropriate complementary feeding; using effective methods of family planning; and seeking care for antenatal and supervised delivery at health facilities. Then incidence of Acute Respiratory Infections (ARI), malaria, diarrhea and malnutrition will be reduced and pregnancy outcomes will be improved. - If MOH and CSOs have broadened capacity to implement and manage health services at the community level, and if there are sufficient numbers and spread of trained community health workers (CHAs and CHVs) in communities located 5 km or farther away from health facilities. And if treatment services for sick children are available and accessible in these communities through trained CHAs on the one hand, and caregivers seek prompt and effective treatment for communicable diseases for their children on the other hand. Then prevalence of Acute Respiratory Infections (ARI), malaria, and diarrhea among children under-five will be reduced. - If households have access to improved sanitation services. And household water treatment options are available at an affordable cost. And households have access to safe drinking water sources. Then prevalence of diarrhea and other waterborne diseases among children will be reduced even further. - If morbidity rates (ARIs, malaria, and diarrhea) in children under five is reduced. Then child mortality rate will be reduced. 2. Purpose of PACS Evaluation The purpose of this evaluation is to provide an independent and in-depth examination of the overall progress of community health services, WASH services, and health-seeking behaviors as supported by the PACS Activity in Liberia. The evaluation will identify achievements, performance issues, and constraints related to activity implementation and effectiveness. Evaluation findings and recommendations will be shared and discussed with USAID/Liberia, Implementing Partners, and relevant GOL units. 3. Evaluation Questions Performance: For key objective areas listed below, did PACS implement with fidelity (i.e., did they do what they said they were going to do) and does the available data suggest significant outcomes as a result of implementation? 1. Build capacity (management systems, knowledge transfer and political will) of MOH, County Health Teams, CSOs and communities for the Community Health Assistant program?4 4 For CSO capacity, this includes: 1. the quality of the work: technical know-how and ability to deliver; and 2) management systems: presence and implementation of strategic plans, human resource and finance functions, and revenue-generation strategies? 96 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV a. What is the cost of the access – as defined by the administration cost of PACS providing access to CLTS water and sanitation services per beneficiary – as compared to the administration cost under an international NGO as was previously used? 2. Build capacity of the MOH and MPW to sustain implementation of the CHA and WASH program and demand creation activities? 3. Increased the availability of sustainable and quality community-based health services (for child health, family planning, maternal health, nutrition and WASH) including 2-way referrals with health facilities? 4. Increased community demand for healthy behaviors and health services (child health, family planning, maternal health, nutrition and WASH)? 5. Improved access to and management of safe water supply and sanitation at the community level? 4. Evaluation Methods The evaluation is expected to apply both quantitative and qualitative methods for data collection and analysis. The evaluation team will conduct a desk review of available literature including activity and project documents. The evaluation team will also look at sectoral assessments that were used to determine and adapt interventions. Visits to the relevant partners and sites will provide primary qualitative data for analysis through methods such as in-depth and key informant interviews, focus groups, and direct observation. The qualitative data collection can be supplemented through quantification of qualitative information, especially through participatory ranking exercises that will provide numeric insight on respondent priorities. It is expected that the evaluation team will use data provided by the implementing partners in regular quarterly and annual reports, performance reporting, and studies for supplemental quantitative data. The team will also draw on GOL datasets such as DHIS2. Primary collection of quantitative data through large-scale structured surveys is beyond the scope of this evaluation. The evaluation should consider Bong, Nimba, and Lofa counties as field visit options. Site lists provided by IPs will be further used to select field visit sites that cover the breadth of interventions being reviewed by the evaluation. The USAID/Liberia Health team will provide additional documents when necessary for the desk review, as well as contact information for prospective interviewees. The evaluation team will be responsible for identifying and reviewing additional materials relevant to the evaluation, as well as additional contacts. Illustrative data sources include: 1. Award and modifications 2. Annual Work Plans 3. Quarterly and Annual reports 4. Activity Monitoring, Evaluation, and Learning Plan 5. Revised PACS Program Description 6. Internal assessment reports (population based and KAP) 7. Performance Indicator Database System data 8. Organizational capacity assessments reports of CSOs 9. PACS Midterm Evaluation 10. Data quality assessment report 11. National Community Health Services Policy and Strategic Plan 12. MOH Evaluation of Community Health Policy 13. Other related national data and reports 14. Social Cash Transfer intervention documentation 97 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV To engage the key evaluation users in analysis of the preliminary findings and help shape the recommendations, the evaluation team will facilitate the debriefs in a more collaborative, half or full-day session. During the session, preliminary findings will be shared, and the evaluation users will add interpretation to the “so what” and “now what” of the evaluation results. This can also help the evaluation team identify areas where more information is needed (e.g., through additional document review or remote interviews). This may take place separately or jointly with USAID, Implementing Partners, and GOL depending on the nature of the evaluation results. 5. Deliverables and Timeline Evaluation deliverables include: a. Evaluation Team Planning Meetings b. Inbrief with USAID/Liberia, Implementing Partners, and GOL c. Inception Report with work plan and data collection instruments d. Participatory analysis session with USAID/Liberia, Implementing Partners, and GOL - Session that allows for stakeholders to engage with the preliminary evaluation findings and help shape the recommendations prior to report drafting. e. Formal Debrief – Presentation to USAID/Liberia prior to team departure. f. Draft Evaluation Report - A draft report should be submitted to USAID/Liberia for review within three weeks after the in-country work is conducted. The written report should clearly describe findings, conclusions, and recommendations and conform to USAID requirements outlined below. USAID will provide comments on the draft report within five working days of submission. Then, LSA will submit the draft report to the Implementing Partners and GOL for another five working days to provide comments and disclose a statement of difference, if applicable. g. Final Report - LSA will submit a final report that incorporates USAID and Implementing Partners’ comments no later than ten days after final, written comments on the team’s draft report have been submitted by all parties. h. Learning Event – LSA will facilitate a learning event one to three months after the report has been finalized to engage USAID/Liberia, Implementing Partners, GOL, and other relevant stakeholders in the utilization of recommendations. This will be combined with an After-Action Review, prescribed as good learning practice per ADS 201, to support USAID in the incorporation of recommendations into work plans, project or activity design, and the next Country Development Coordination Strategy. The draft evaluation report should meet the following criteria: i. The report should be in line with USAID Evaluation Policy (see Appendix I – Criteria to Ensure the Quality of the Evaluation Report) and USAID Secretariat Style guide. ii. The report should be no longer than 20 pages, excluding executive summary, table of contents, and annexes. iii. The report should include a one to two page Executive Summary highlighting findings and recommendations. iv. The report should represent a thoughtful and well-organized effort to objectively respond to the evaluation questions. v. The report shall address all evaluation questions included in the SOW. vi. Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides shall be included in an Annex in the final report. vii. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, etc.). 98 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV viii. Evaluation findings and conclusions should be specific, concise and supported by strong quantitative or qualitative evidence. ix. Recommendations should be action-oriented, practical, specific, and evidence-based. x. The final report should be edited and formatted. xi. LSA must submit the final evaluation to the Development Experience Clearinghouse. The report will be submitted electronically. The final report will be edited/formatted by the contractor and provided to USAID/Liberia five working days after the Mission has reviewed the content and approved the final revised version of the report. The final evaluation report must be 508 compliant and comply with the USAID Evaluation Policy: http://www.usaid.gov/sites/default/files/documents/1868/USAIDEvaluationPolicy.pdf Timeline: The evaluation should follow the timeline for producing deliverables outlined in the Gantt chart below. 99 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV Performance Evaluation Gantt Chart Activity # of days Wee k 1-2 Wee k3-5 Wee k6 Wee k7 Wee k8 Wee k9 Wee k10 Wee k11 Wee k12 Wee k13 Wee k14- 15 Wee k16- 17 USAID shares SOW with LSA 1 LSA publishes call for CVs 14 LSA recruits and onboards team members 21 Desk Review 6 Team travels to Liberia 1 Team planning meeting - internal 1 Submit draft Inception Report to USAID Planning Meetings with USAID and PACS/GOL 1 Inbrief with USAID and PACS/GOL 1 Prepare for field work 2 Submit final Inception Report to USAID Field Work 20 Participatory analysis sessions with USAID, PACS, and GOL 1 Official debrief with USAID 1 Travel from Liberia 1 Analysis and report drafting 13 LSA quality control and revisions 6 LSA submits Draft Report to USAID USAID, PACS, and MOH review the draft report 10 Evaluation team incorporates comments and feedback 5 LSA quality control 5 LSA submits Final Report to USAID LSA facilitates Learning Event with USAID, PACS, GOL, and stakeholders 1 1-2 mont hs after PE Prep work by Team Leader and national consultant 1 100 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV 6. Team Composition The evaluation team will be composed of four individuals: team leader, two technical experts, and an evaluation specialist. The evaluation team will be supported by LSA staff, including a Project Manager based in Monrovia to facilitate client relations and a Project Director based at HQ to enhance quality of deliverables. LSA staff may be used to fill the evaluation team roles if available. 1. Senior Evaluation Specialist/Team Leader: S/he should have at least ten years of experience managing mixed-method evaluations and conducting analysis of health interventions in developing countries. Extensive experience evaluating and implementing community health systems development and/or institution building is required. The candidate must have a background in facilitating and measuring progress of self-assessments, performance improvement plans, and capacity strengthening plans. Experience implementing USAID performance evaluations and linking to CDCS design is preferred. S/he must have a proven track record supervising teams in the field and producing high quality and concise reports, as well as extensive experience working in West Africa and similar settings. Must hold an advanced degree in a discipline related to Public Health, Community Health, International Development, research, or related field. 2. Integrated Health and WASH Analyst: The candidate should have at least eight years of experience designing, implementing, and evaluating community-led WASH activities that reinforce community health outcomes, preferably through USAID. S/he should have a background in measuring capacity building progress of CSO and GOL water and sanitation personnel and implementing infrastructure development and maintenance in close coordination with government institutions. Knowledge of USAID rules, regulations, and procedures in the sector is desirable. The candidate should have previous work experience conducting mixed￾method evaluations. Strong interpersonal skills and American English language speaking and writing skills are also essential. 3. Evaluation Specialist: The candidate should have at least five years of experience conducting mixed-method evaluations in West Africa, with both qualitative and quantitative data collection. The specialist must have experience evaluating community health programs and have demonstrated logistics and planning skills. S/he should have experience analyzing DHIS2 data and broad knowledge of Liberian health issues. The specialist must have strong social and gender analytical skills. Experience managing or implementing research programs is preferable. Must be able to conduct interviews and focus group discussions and analyze the resulting data. 4. Liberian Health Context Specialist: The candidate should have at least five years of experience working on County, District, and community level health system administration in Liberia, specifically regarding community health initiatives. The specialist should have a background in behavior change strategies and working in partnership with CSOs to improve community-based health service delivery. S/he should have experience conducting mixed-method evaluations, with both qualitative and quantitative data collection. Collectively the team members must have experience in conducting both quantitative and qualitative data collection and analysis. Prior to their arrival in Liberia, all team members are required to familiarize themselves with USAID’s Evaluation Policy, with USAID’s publication outlining a good evaluation report, and with USAID’s checklist for assessing an evaluation report. Additionally, all team members should possess a strong familiarity with the political, economic, policy and educational context in Liberia. 7. Logistics and Level of Effort 101 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV LSA is responsible for making meeting and logistical arrangements for the evaluation team including hotel, air travel, and local transportation arrangements. For planning purposes, contractors should be aware of Liberian and US holidays during the evaluation time frame. Evaluation team members are authorized and expected to work a six-day week. Travel over weekends may be necessary. Work should commence as soon as practicable, but no later than May 2019. The evaluation should follow the illustrative level of effort given below. Task/Deliverable Estimated time (Days) Team Leader Int’l Team Member Lib Team Member Review background documents & preparatory work (offshore) 6 6 4 Travel to/from Liberia 2 2 0 Team Planning Meetings in Monrovia with USAID/Liberia and IP 2 Inbrief with USAID/Liberia, IP, and GOL and prepare for field work Submit Inception Report 3 Field work: Data collection and on-going data analysis 20 (including field travel) Participatory analysis sessions with USAID/Liberia, IPs, and GOL 2 Debrief to USAID 1 Analysis and report drafting 13 10 6 LSA performs quality assurance check and edits draft report 5 LSA submits draft report to USAID/Liberia 1 USAID/Liberia and IP provide comments on draft report 10 Team Leader revises draft report to incorporate comments and submit final report 5 2 1 LSA performs quality assurance check and submits final report 5 LSA facilitates learning event 1 0 1 Total time required 76 Working days for consultants 55 48 40 102 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV ANNEX F: DISCLOSURE OF ANY CONFLICTS OF INTEREST 103 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV 104 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV 105 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV 106 | USAID/LIBERIA PACS FINAL EVALUATION REPORT USAID.GOV U.S. Agency for International Development - Liberia 502 Benson Street Monrovia, Liberia