Final Evaluation Report Community Health Worker Support for Advancing Liberia Livelihood (CHWS for ALL) Project December 2018 ________________________________________________________________________________ Copyright © Final End-Line Evaluation - CHWS for ALL Project – 2018 ii FINAL PROJECT END-LINE EVALUATION REPORT Community Health Worker Support for Advancing Liberian Livelihoods (CHWS for ALL) CHWS for ALL is working to strengthen the capacity of the Ministry of Health (MoH) and other community health stakeholders to oversee the scaling up of a high-quality National Community Health Assistant (CHA) Program that is informed by and held accountable to the remote communities it has been designed to serve. Award Number: AID-OAA-A-16-00028 Award Office: USAID Global Health Period of Performance: June 10, 2016 – January 9, 2019 ________________________________________________________________________________ Copyright © Final End-Line Evaluation - CHWS for ALL Project – 2018 iii DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. ACKNOWLEDGMENT The success of this evaluation was a result of the dedicated efforts of staff of Last Mile Health and selected staff of the Ministry of Health (MoH) at the central and county levels. Last Mile Health (LMH) contributed extensively to this project end-line evaluation report through the provision of technical support for the evaluation setup, evaluation publicity and provision of relevant documents of the project for the desk review. The Evaluation Team (ET) wishes to express its gratitude for the tremendous and vital support provided by all study participants including the Community Health Services Division (CHSD), the Community Health Services Supervisors (CHSS), the County Health Team (CHT), and the Health Information System Monitoring, Evaluation and Research (HMER) Department; whose participation made this research possible. The evaluation process was challenging but encouraging as the evaluation team gained additional valuable experiences from implementing the evaluation project. It is our hope that this final report of the evaluation of the Community Health Worker Support for Advancing Liberian Livelihoods (CHWS for ALL), implemented by LMH, will provide useful lessons for the scale-up of the National Community Health Assistance (NCHA) program in the most remote communities in Liberia. Note: The contents of this report are the responsibility of Top Consulting Incorporated (TCi) with appropriate inputs from LMH and do not necessarily reflect the views of USAID or the Government of the United States. Maxwell S. B. Nimely, Team Leader Director of Research and M & E Top Consulting Incorporated (TCi). Wilson O. Idahor, Project Consultant Managing Consultant/CEO Top Consulting Incorporated (TCi) ________________________________________________________________________________ Copyright © Final End-Line Evaluation - CHWS for ALL Project – 2018 iv TABLE OF CONTENTS ACKNOWLEDGMENT ........................................................................................................................ iii TABLE OF CONTENTS........................................................................................................................ iv LIST OF TABLES..................................................................................................................................... v ACRONYMS.......................................................................................................................................... vi 1.0 EXECUTIVE SUMMARY........................................................................................................... vii 2.0 EVALUATION PURPOSE, OBJECTIVES, AND SCOPE..........................................................16 2.1 Purpose ................................................................................................................................. 16 2.2 Core Objectives....................................................................................................................16 2.3 Scope of Work......................................................................................................................16 3.0 EVALUATION APPROACH AND METHODOLOGY...........................................................16 3.1 Data Analysis.........................................................................................................................18 3.2 Compilation of Data Bibliography.........................................................................................18 3.3 Final Evaluation Criteria........................................................................................................18 3.4 Limitation of the End-Line Evaluation Project......................................................................18 3.5 Challenges of the Evaluation................................................................................................. 18 3.6 Organization of the Report ..................................................................................................19 4.0 PROJECT OVERVIEW..............................................................................................................20 4.1 Project Background and Context.........................................................................................20 4.2 Project Objectives.................................................................................................................21 4.3 Results Framework ...............................................................................................................23 4.4 Implementation Approach and Methodology .......................................................................23 4.5 M&E APPROACH.................................................................................................................24 5.0 EVALUATION FINDINGS .......................................................................................................26 5.1 Profile of Respondents..........................................................................................................26 5.2 Evaluation Results .................................................................................................................28 ________________________________________________________________________________ Copyright © Final End-Line Evaluation - CHWS for ALL Project – 2018 v 5.3 Project Achievements...........................................................................................................38 5.4 Sustainability of the Project Achievements...........................................................................38 6.0 IMPLEMENTATION CHALLENGES, LESSONS LEARNED AND RECOMMENDATIONS..41 6.1 Implementation Challenges...................................................................................................41 6.2 Lessons Learned....................................................................................................................41 6.3 Recommendations.................................................................................................................42 7.0 CONCLUSION ........................................................................................................................44 ANNEXES.............................................................................................................................................46  Annex 1: Evaluation Terms of Reference (Attached)...............................................................46  Annex 2: Data Collection Tools (Attached) ............................................................................46  Annex 3: List of the Keys persons Interviewed (Attached).....................................................46  Annex 4: Logical Framework for the Project (Initial Project Log frame)................................. 46 LIST OF TABLES  Table 1 – Population Distribution  Table 2 – Results & Indicators  Table 3 – Profile of Respondents  Table 4 – Project Contributions  Table 5 – Project Logical frame ________________________________________________________________________________ Copyright © Final End-Line Evaluation - CHWS for ALL Project – 2018 vi ACRONYMS ALL Advancing Liberian Livelihoods CBIS Community-Based Information System CBO Community-Based Organization CSO Civil Society Organization CHA Community Health Assistant CHAI Clinton Health Access Initiative CHC Community Health Committees CHSD Community Health Services Division CHSS Community Health Services Supervisor CHT County Health Team CHTWG Community Health Technical Working group CHWS Community Health Worker Support CHWS for ALL Community Health Worker Support for Advancing Liberian Livelihoods DCT Data Collection Tool DHIS2 District Health Information System EVD Ebola Virus Disease ET Evaluation Team GoL Government of Liberia HFDC Health Facility Development Community HMIS Health Management Information System HMER HMIS Monitoring, Evaluation and Research ICH Integrating Community Health IFI Implementation Fidelity Initiative IRC International Rescue Committee KII Key Informant Interview LMH Last Mile Health LMIS Logistics Management Information System M & E Monitoring and Evaluation MoH Ministry of Health NCHA National Community Health Assistant NCHAP National Community Health Assistant Program NGO Non-Governmental Organization PPS Program Perception Study QRM Quarterly Review Meeting SOW Scope of Work SOP Standard Operating Procedure RM & E Research, Monitoring and Evaluation TCi Top Consulting Incorporated TL Team leader TOR Terms of Reference TWG Technical Working Group UNICEF United Nations International Children's Emergency Fund USAID United States Agency for International Development WHO World Health Organization ________________________________________________________________________________ Copyright © Final End-Line Evaluation - CHWS for ALL Project – 2018 vii 1.0 EXECUTIVE SUMMARY Introduction: The Community Health Worker Support for Advancing Liberian Livelihoods (CHWS for ALL) Cooperative Agreement aims to strengthen the capacity of the Liberian Ministry of Health (MoH) and other community stakeholders to oversee the scaling up a high-quality National Community Health Assistant (CHA) Program that is informed by and held accountable to the remote communities it has been designed to serve. To ensure the success and sustainability of the National CHA Program, the LMH through the CHWS for ALL Project collaborated with MoH and other community health stakeholders to: 1. Establish environmental and operational readiness at the central and county levels for the introduction of the National CHA Program 2. Support the MoH to integrate and institutionalize the National CHA Program 3. Ensure continuous learning and quality improvement for National CHA Program implementation and accountability. The CHWS for ALL Project was awarded under the USAID-UNICEF Integrated Community Health Program to Last Mile Health (LMH). LMH was mandated under the contract award to provide technical and systems strengthening support to the central Ministry of Health and three (3) out of Liberia's Fifteen (15) County Health Teams (Rivercess, Grand Gedeh, and Grand Bassa). In addition to the technical assistance provided through the CHWS for ALL Project, Last Mile Health also provided other direct support to the CHT, to support their implementation of the National CHA Program. The LMH contracted the services of Top Consulting Incorporated (TCi) to conduct an End-Line Evaluation of the project’s result and achievements, and to provide informed decisions for future projects. Purpose: The core purpose of the End-line evaluation of the project was to assess how the technical support provided through the CHWS for ALL Project as implemented by LMH was able to build the operational and environmental readiness of the central and county level MoH to implement and scale the National CHA Program. Specifically, the project end-line evaluation sought to answer the following key learning questions, which formed the core objectives of the end line evaluation:  To what extent has the CHWS for ALL Project increased the operational and environmental preparedness of the central MoH and relevant County Health Teams (CHTs) to manage the implementation of the National CHA Program?  To what extent did the technical support provided under the CHWS for ALL Project support the performance of a county level model (focusing on Rivercess)?  To what extent has the CHWS for ALL Project developed systems that allow for the continuous learning and quality improvement of the National CHA Program? Scope: The scope of the End Line Evaluation was limited to:  Examination of the systems strengthening approach CHWS for ALL took at the central and county level and how it supported the quality of implementation of the ________________________________________________________________________________ Copyright © Final End-Line Evaluation - CHWS for ALL Project – 2018 viii NCHA specifically to examining CHWS for ALL technical support related to M&E, health financing, and creating supportive supervision structures and other interventions.  Review of model CHWS for ALL bringing together qualitative and quantitative data to link inputs, outputs, and outcomes for the CHW logic model with focus on its implementation in Rivercess County. Approach and Methodology: In order to conduct a comprehensive end-line evaluation of the CHWS for ALL Project, TCi used a systematic approach and methodology to obtain the most relevant project data from all stakeholders. The evaluation process included desk review, meetings with key project stakeholders, development of the evaluation tools, training of evaluators, administering the tools, analyzing the collected data and writing the final report. Purposive sampling was used to conduct key informant interviews (KIIs) with relevant stakeholders. A total of 48 persons participated in the evaluation, with the overall evaluation sample size consisting of males (65%) and females (35%). Key Findings: Given the challenges of accessing and providing technical support to Rivercess County, LMH was able, to a large extent, build the operational and environmental readiness of the central and county level MoH to implement and scale the National CHA Program. The evaluation findings confirms that while important challenges still exist, desk review analysis and key informant interviews (KII) indicated a general consensus that CHWS for ALL Project has greatly improved the ability of central MoH and the Rivercess County Health Team (CHT) staff, to implement the NCHA program in Rivercess County. The key findings of the project End-Line Evaluation are presented based on the core objectives of the evaluation; the key project achievements and their sustainability; project implement challenges, lessons learnt and recommendations for future project design and implementation. Below are a summary of the key findings of the evaluation:  Environmental and Operational Readiness: The CHWS for ALL project provided support for planning, organizing, and conducting orientation workshops on the NCHA strategic plan and policy in Rivercess County1 . The CHWS for ALL project worked with CHSD to develop advocacy materials and support targeted outreach for National CHA Program with key stakeholders within MoH Leadership2 . These materials explained the operations and benefits of National CHA Program to the target counties and were used to create awareness and promote the NCHA program. 1 5. AID-OAA-A-16-00028_Annual Report 2018_Activity Table 2 ibid ________________________________________________________________________________ Copyright © Final End-Line Evaluation - CHWS for ALL Project – 2018 ix The data collected and analyzed showed that the CHWS for ALL project interventions have strengthened the MoH community health county and national level leadership. Specifically the CHTs and Central MoH staff have improved their leadership skills as they are using supervision tools and processes for standard supervision and monitoring, accessing data from DHIS2 to make decisions, conducting training, coaching and mentoring to improve the capacity of other NCHAP stakeholders (CHSS, CHCs etc), conducting monthly revision of supervision work plans. Moreover, at the central MoH, staffs are using data analysis tools such as Stata to review and analysis data, creating and managing governance structures through the development and use of tools including the development of SOPs, Tools, Processes and development TORs for TWGs. While the project did not involve the deployment of quality assurance officers, other LMH projects directly implementing the NCHA program provided such support, from which the project benefited. Hence, quality assurance procedures were provided at every stage of the implementation. For example, they assigned quality assurance officers to each district to support these values at the county level. The CHWS for ALL Project supported the CHT to strengthen the community health structures (including CHCs, HFDCs) and engage local CSOs/CBOs to improve local leadership and sustainability within Rivercess. The CHT also supported the establishment of new CHCs in communities that did not have CHCs prior to the project interventions. It was said during the evaluation interviews, that the project intervention also supported the coordination, supervision and meetings of the Community Health Technical Working Groups (CHTWGs) and District health teams. The CHWS for ALL Project also supported the delineation of responsibilities and accountability for oversight, coordination, and implementation of the NCHA Program. These structures are essential in the sustainability and ownership of the NCHA program. Community engagement seems to be at its best in the CHA implementation. HFDC meetings are held with Community Health Committee (CHC) participation. There are established CHCs in all communities with a CHA. Mobilizing such CHCs is still a challenge with the project because they are not yet fully responsive to their roles and responsibilities. The rate of CHCs holding monthly meetings has been dropping steadily from 92% in Q4 to 15% in Q7. Although At the time of preparing this evaluation report; it has risen to 75% in Q8 based on based on recent project data received from LMH. Towards long-term sustainability of the program, CHWS for ALL continues to draw upon a comprehensive suite of financial planning and budgeting tools to support the identification and cultivation of new funding mechanisms, while also charting towards increased fiscal commitment by the Government of Liberia. Of the estimated $95 million total cost to reach and maintain full coverage via the National CHA Program between 2016 and 2021, over $32 million has been committed to date with additional ________________________________________________________________________________ Copyright © Final End-Line Evaluation - CHWS for ALL Project – 2018 x coordination underway to support alignment between the Government of Liberia and USAID financing priorities3 . While it is true there is still room for improvement to attain environmental and operational readiness, the synergy of the essential support provided by the CHWS for ALL Project, to a great extent, created some level of readiness and has increased the operational and environmental preparedness of the central MoH and relevant CHTs to manage the implementation of the National CHA Program. Specifically these improvements involve the development of a standard operation procedure (SOP) for orientating new staff on the National CHA Policy and Strategic Plan; consideration of payment mechanism from the perspective of the local communities by examining local financial management schemes such Village Saving Groups (‘Susu clubs etc.) as a payment mechanism and; the need to support MoH to place emphasis on sourcing funding for the NCHA program internally including GoL budget allocation, charging of fees for services, etc. , besides the current case of overly dependence on external donor funding.  Institutional readiness to integrate the NCHA program: The CHWS for ALL project built capacities to ensure that staff have the necessary technical, managerial and leadership knowledge and skill to carry out their responsibilities, and that they know when and how to use these skills best. The CHWS for ALL Project consulted with IRC/PACS, CHAI, UNICEF, and other community health stakeholders to align and provide technical assistance and capacity building for CHSD and relevant MoH divisions/programs. At the central MoH level, the respondent data analysis indicated that 50% of the respondents (5 persons) were part of the planning committees for the development of the NCHA program. The data also showed that 3 of this number are still in their current position while 2 persons have moved on to different positions in the MoH. Based on data collected, 100% of the respondents overwhelmingly acknowledged and demonstrated their role and responsibilities as linked to the objectives of the NCHA program. Furthermore, there is evidence that the coordination mechanisms within the Government have been strengthened, with draft terms of reference (TORs) developed for the Community Health Technical Working Group, the Community Health Steering Committee, and the National CHA Program Inter-Ministerial Coordination Committee. CHWS for ALL also supported representatives of the National CHA Program within forums such as the Health Sector Coordination Committee, chaired by the Minister of Health, and the Human Resources for Health, Supply Chain, Health Information System, and Health Financing Technical Working Groups. 3 1. AID-OAA-A-16-00028_Annual Report 2018_vf.docx ________________________________________________________________________________ Copyright © Final End-Line Evaluation - CHWS for ALL Project – 2018 xi CHWS for ALL supported the communication and information sharing, learning, and advocating for the quality of the NCHA program. Improvements and changes were recorded, using data to demonstrate results and how the results were achieved in QRMs. Data from IFI, PP and CBIS were shared and analyzed quarterly at the national QRMs. The CHTs and MoH staff also has access to the DHIS2 and IFI data on an ongoing basis. The evaluation data showed that an aggregate of 75% of Central and County MoH staff respondent interviewed; expressed strong confidence in their ability to oversee both management and technical aspects of the National CHA program in their roles, based on the support and training acquired through the CHWS for ALL Project. The respondents confirmed that they had acquired skills and competences from the Project including the ability to develop TORs, use of supervision tools and processes for standard supervision and monitoring; analysis and use of data for decision making; provision of training, coaching and mentoring skills, revision of supervision work plans on a monthly basis; use of data analysis tools such as Stata to review and analysis data; creation and management of governance structures and tools including the development of SOPs, Tools etc; enhanced qualitative research skills through the participation in joint research during the Program Perceptions study with LMH.  Continuous learning and Quality improvement for the NCHA program: The Liberia’s Community Based Information System (CBIS) serves as the National CHA program’s primary data collection toolkit, and is fully integrated within DHIS2. With rollout complete across 14 of Liberia’s 15 counties, increased focus is being placed on individualized support to County Health Teams to improve the frequency and quality of reporting. Strengthening of NCHA program to ensure sharing and dissemination of important program information, continuous learning and quality improvement is required. Thus the CHWS for ALL supported platforms to enhance the program activities. According to the respondents, the support given includes participant transport, DSAs/lunch, logistics, etc. in the table below indicates the meetings covered: Although the Evaluation Team did not access documentation such as minutes of the meetings and activities mentioned above, responses from the interviews show that some of these continued learning activities did not occur as planned. Some reasons proffered for these discrepancies include constraints with logistics, unforeseen emergency activities, and competing priorities (i.e. national plans competing with county plans). For example, the quarterly review meetings should be held 4 times a year; however, the ET observed that the last one held was the third QRM for the year which was mainly due to the change in the leadership of the government of Liberia. To support more learning, the CHWS for ALL supported the development and submission of publication in a peer-reviewed academic journal. Document reviewed show that two Research Projects have been approved as part of the MoH Research Agenda relating to NCHA program. During the time of preparation of this evaluation report; 2 publications have been developed and submitted in a peer-reviewed ________________________________________________________________________________ Copyright © Final End-Line Evaluation - CHWS for ALL Project – 2018 xii academic journal while 6 of the research studies has been approved as part of national research agenda.  Key Project Achievements and their Sustainability: The following are key achievement of the project: o The development of key processes that enabled the Ministry of Health to regularly monitor the progress of and anticipate challenges with the National CHA Program. o Well-established processes and data collection and analysis system enable successful advocates for the National CHA Program to be included in the Ministry of Health 100 day priorities during the political transition. o Technical support to develop and maintain health financing tools (costing of the NCHA program, resources mapping etc.) o Roll-out of CBIS in Rivercess and training on accessing and analyzing CBIS data from DHIS2 that ensures program implementers can now clearly review and use data. o Provision of support to CHT and Central MoH in conducting the QRMs established a major platform to review program achievements, identify gaps and necessary actions to mitigate the gaps.  Sustainability of Project Achievements: Overall, the mechanisms to ensure long-term sustainability of the project achievement were developed. 90% of the respondents were confident that the CHWS for ALL project has been of tremendous support to the implementation of the NCHA Program in Rivercess County. While respondents interviewed are hopeful that the CHWS for ALL interventions will be sustained in the long-term.; however, they also noted that the progress achieved by the project can only be sustained if certain processes and activities particularly those that require financial support are adequately funded by MoH. The majority of the respondents are of the view that without the adequate funding support currently being provided by external donors such as LMH for implementing processes and activities such as QRMs, all forms of field supervisions, training, coaching, monitoring and evaluation etc. the degree of sustainability of the project achievements will decline. The following are key Sustainability issues: o Financial Sustainability: As mentioned above, CHWS for ALL project implemented by LMH in Rivercess, provided financial support to enable some key processes and activities in the implementation of the NCHA program. Based on desk review and respondents’ data, there are no definite financial sustainability strategies to fund some of these technical and logistics support provided by project although, consultations are ongoing. The lack of internal government funding for the NCHA Program in general presents a serious risk in the program. Several respondents indicated that they are concerned that the NCHA program will not survive without the continuous support of donors. o Technical Sustainability: The CHWS for All Project supported the development of several tools, data collection, management and analysis systems, ________________________________________________________________________________ Copyright © Final End-Line Evaluation - CHWS for ALL Project – 2018 xiii training standard operating procedure (SOPs), supportive supervision tools, processes for standard supervision, monitoring, the use of data for decision making; the use of Google Suit and other tools that can be used to manage the NCHA program. These processes and activities supported by the CHWS for ALL project ensured that the Project was successful in delivering it core objectives to achieving its overarching goal. Notwithstanding, to ensure technical sustainability of the processes and activities developed and supported by the Project; it is essential that top management of MoH institutes procedures (and/or directives) that mandates all NCHA Program activities including reporting to use the tools and follow the processes developed and supported by the project. o Institutional Sustainability: The project supported the CHTs in the organization and management of Community-based Structures (CHCs & HFDCs) to ensure continuous provision of the NCHA program services in the rural communities. Training support were provided through the CHTs to strengthen the CHCs, HFDCs, CHSS and CHAs structures to ensure post-project institutional sustainability. Furthermore, the CBIS serves as the National CHA Program’s primary data collection toolkit and is fully integrated within DHIS2 and Health information system, further promoting institutional sustainability of the data system. In additional, joint research processes i.e. IFI, PP served as additional sources of monitoring and evaluation that is being built into regular CHSD and HMER work plans. Information sharing of all from sources: IFI, PP and CBIS are done at the national QRMs. The CHTs and MoH staff also have access to the DHIS2 system. However, some stakeholders have challenges with accessing the data due to poor internet service and non-use of their access to the DHIS2. All the data are shared quarterly with CHSD at the county and national QRMs. While the primary responsibility for sustainability depends greatly on the communities themselves, continuous support, monitoring and evaluation will be required from all stakeholders especially the local and national government to ensure sustainability.  Lessons Learnt: The project presented the following important key lessons learnt from the evaluation findings: o Constant engagements improve information-gathering and sharing which enables successful project delivery: o Understudy (joint activity implementation) increases project performance o Pre-implementation Capacity Assessment enables more effective capacity development o Coaching and Mentoring are essential for effective capacity development o Effective Data collection, processing, storage, analysis and access for use, significantly improves project implementation and support advocacy  Recommendations: the following are key recommendations for ensuring continued sustainability of the project achievements: ________________________________________________________________________________ Copyright © Final End-Line Evaluation - CHWS for ALL Project – 2018 xiv o GOL should allocate Specific National Budget for the NCHA Program: Lack of direct and adequate Government funding for the program presents a serious risk to the NCHA program. The project has supported the MoH to develop health financing tools, develop the estimated program budget as well as the development of resource mapping template for fund sourcing for the NCHA program. These tools and corresponding MoH staff efforts have been mainly geared towards solicitation of external funding for the NCHA program. It is recommended that the project support MoH to place considerable more effort and emphasis in using these tools to solicit for revenue from within the country i.e. from the national government and other internal sources instead of the overly dependence on solicitation of funds from external donors. o MoH should demonstrate practical ownership through funding and the mandatory use of the Project developed tools and processes: The lack of direct MoH funding and general civil service bureaucracy can affect sustainability of the progress achieved by the project. MoH should commit to funding to support key program processes and activities such as QRMs, all forms of field supervisions, research studies, data gathering, storage and access including regular training, coaching, monitoring etc. This way, the MoH will demonstrate practical of the program. MoH should also demonstrate ownership by mandating that all NCHAP activities including reporting system be strictly implemented using the tools, conducting the activities and implementing the processes developed by the CHWS for All project for the program daily management and periodic reporting. o The Project implementer should develop and implement a Strategic Exit Plan to support the sustainability of the project achievements: Such project strategic exit plan should incorporate the following actions points: Financial Sustainability, Technical Sustainability, Institutional Sustainability, Continuous Learning, other measures to enhance sustainability. o MoH should mandate the use of developed Project Processes and Activities: The CHWS for ALL Project has successfully contributed to the implementation of NCHA Program in Rivercess County through its technical assistance support. However, since MoH currently lacks adequate funding to internally self-finance the developed project processes and activities, it is critical, that donors such as LMH continue to provide funding required for technical and other support for the NCHA Program.  Other Important Recommendations: - MoH should create employment pathways for CHSS and CHAs: - MoH and partners should increase motivation for CHAs - MoH should improve security of CHSS and CHAs - MoH should develop Strategies to Mitigate Program Staff Attrition - MoH should support and improve the drug supply chain - MoH should improve the capacity of other MoH department particularly at the Central MoH - GOL should provide and/or improve basic infrastructure especially roads ________________________________________________________________________________ Copyright © Final End-Line Evaluation - CHWS for ALL Project – 2018 xv Conclusion: The desk review of project documents and analysis of data collected from key stakeholders during interviews demonstrated a general consensus that CHWS for ALL Project has greatly improved the ability of MoH staff at Central office and particularly at Rivercess County to implement the NCHA program. However, lack of direct MoH funding and general civil service bureaucracy can affect sustainability of the progress achieved by the project. The Project was able to increase the operational and environmental preparedness of the central MoH and relevant CHTs to manage the implementation of the National CHA Program. The MoH still needs to demonstrate practical ownership of the program through increase finance commitment and to the program. As one key respondent aptly said, if external donors do not continue to support this program, we are dead”. The technical support provided under the CHWS for ALL Project increased the performance of a county level model in Rivercess, the progress report of the NCHA program reflects this performance (the scope of this evaluation project does not include the performance NCHA program results). However, in terms of staff capacity development, the CHTs and Central MoH staff have greatly improve their skills and competences which reflects in their use and analysis of CBIS data from DHIS2 for decision making, operations and presentation; their improvement in the organization and management of QRMs; the use of supervision tools and processes for standard supervision and monitoring etc. Notwithstanding the achievements, to ensure high quality service delivery and sustainability, the MoH should mandate all staff involved in the NCHA program to strictly adhere to using the tools, conducting the activities and implementing the processes developed by the CHWS for All project for the daily management and periodic reporting of the program. The CHWS for ALL project developed key processes that have enabled the Ministry of Health to regularly monitor the progress of and anticipate challenges with the National CHA Program. This includes programmatic challenges highlighted by CBIS, IFI and Program Perceptions data and are tracked through the program improvement tracker. This has contributed to continuous learning and quality improvement of the National CHA Program. The Project also supported the development of health financing tools enabling the MoH to develop program budget estimates as well as anticipates financial challenges through resource mapping. MoH should place more emphasis on using these tools to advocate for increased revenue from within the country rather than solely depending on raising funds from external donors. Finally it is important to emphasis that NCHA program structures developed and supported by the CHWS for All project should be owned by the beneficiary communities; the MoH and fully integrated, institutionalized and supported financially by the MoH to ensure overall sustainability of the CHWS for All project achievement and the entire NCHA program. __________________________________________________________________________________ Copyright © CHWS for ALL Project Final Evaluation - 2018 16 2.0 EVALUATION PURPOSE, OBJECTIVES, AND SCOPE 2.1 Purpose The purpose of this evaluation was to assess how the CHWS for ALL Project, as implemented by Last Mile Health, was able to build the operational and environmental readiness of the central and county level MoH to implement and scale the National CHA Program. 2.2 Core Objectives The core objectives of the end-line evaluation were as follows:  To what extent has the CHWS for ALL Project increased the operational and environmental preparedness of the central MoH and relevant CHTs to manage the implementation of the National CHA Program? How might the MoH and CHTs increase their operational readiness to take on ownership of the National CHA Program?  To what extent did the technical support provided under the CHWS for ALL Project support the performance of a county level model (focusing on Rivercess)? How might future interventions and how they are implemented be more effective in building capacity for the central and county MoH to manage implementation of the National CHA Program and ensure high quality service delivery?  To what extent has the CHWS for ALL Project developed systems that allow for the continuous learning and quality improvement of the National CHA Program? How might the MoH better utilize data for decision making to inform and improve the National CHA Program? What additional interventions or capacity is needed? 2.3 Scope of Work  To Examine the systems strengthening approach CHWS for ALL took at the central and county level and how it supported the quality of implementation (this includes but it is not limited to examining CHWS for ALL technical support related to M&E, health financing, and creating supportive supervision structures).  To conduct a deep dive into a full-scale model that brings together qualitative and quantitative data to link inputs, outputs, and outcomes for the CHW logic model (Rivercess). 3.0 EVALUATION APPROACH AND METHODOLOGY In order to conduct a comprehensive end-line evaluation of the CHWS for ALL Project, a systematic methodology was used to obtain data and information about the project from a representative sample of all stakeholders. The following approach and methodology was used to evaluate the CHWS for ALL Project:  Meetings with Key Project Stakeholders The Evaluation Team (ET) held consultations with LMH to clarify project objectives and agreed on timelines and deliverables. The team also held meetings with Central MoH officials and Rivercess CHT to introduce the evaluation team, communicate the purpose and objectives of the evaluation, and clarify the activities that would take place at the county and at the central MoH. __________________________________________________________________________________ Copyright © CHWS for ALL Project Final Evaluation - 2018 17  Desk Review of Documents A Desk Review of relevant project documents and reports was done. This involved the review of project documents, existing data, and other resources related to the National CHA Program. The desk review provided supporting data and information for the end￾line evaluation of the project.  Development of the Evaluation tools In consultation with the LMH, relevant evaluation tools, to capture both quantitative and qualitative data, were developed. The tool was used for key informant interviews (KIIs) which were semi-structured evaluation tools used to capture both quantitative and qualitative data.  Evaluation Team Refresher training An evaluation team of 5 people comprising of 3 principal evaluators and 2 assistant evaluators were involved in the evaluation of the project. The evaluation team participated in a refresher course on research methods, techniques and ethics as well as reviewed the proposed evaluation tools. After inputs from the LMH, the final sets of evaluation tools were updated and approved.  Analysis of the Target population The target population for this evaluation included:  Community Health Services Supervisors (CHSS)  County Health Team (CHTs)  Selected Central MoH Staff The above participants included past and present project staff based on their duration on the project and the position held. Table 1 – Distribution of Respondents Gender CHSS CHT LMH Central MoH Total Target Male 10 14 4 6 31 Female 12 0 4 4 17 Total 22 14 8 10 54  Sampling Methods The sampling method was purposive sampling that targeted key people who had knowledge about the CHWS for ALL project support for the NCHA program.  Data Collection Quantitative and qualitative data were gathered using the following: o Comprehensive Desk Review: The evaluation team review about 25 reports, including annual reports, work plans, monthly reports etc. o Key Informant Interviews (KIIs): The evaluation team conducted KIIs with the different stakeholders. These included CHSS, CHT staff, LHM staff and central MoH staff. __________________________________________________________________________________ Copyright © CHWS for ALL Project Final Evaluation - 2018 18 o Observation of Interventions: Observation of the CHWS for ALL interventions was particularly valuable. The evaluating team visited some of the health facilities and observed the data entry processes and procedures such as the use of DCTs by CHSS. The observations were aimed at identifying the appropriateness of the interventions. 3.1 Data Analysis The evaluation team compiled various primary and secondary data sets. The data were collated and subjected to additional validation and scrutiny. The process of screening the data sets was to ensure the quality in close consultation with the sources. Where further clarification was needed, the evaluators worked directly with the data source to fill gaps in existing data sets to address the missing data. The information gathered through the various evaluation methods, including in-depth interviews with key informants and physical observation, were compared, triangulated and analyzed. Simple tabulations were constructed using custom table mechanism. Microsoft Excel was used to capture fields for each respondent attribute and responses from the completed evaluation questionnaires. The result was used to generate frequency distributions, charts, tables and graphs. 3.2 Compilation of Data Bibliography After the data had been cleaned, a consolidated bibliography of all the existing data sources was compiled based on the scope of the evaluation. This bibliography focused on the entire analysis of the data to meet the objective of CHWS for ALL Evaluation. The sources that were provided by the client and partners when conducting this evaluation were analyzed and referenced in the final evaluation report. 3.3 Final Evaluation Criteria The evaluation team developed a detailed project review framework, relating to the overall objectives of the End line evaluation, with specific focus on the following criteria for the CHWS for ALL Project Final End line Evaluation: ▪ Environmental and operational readiness of the central MoH and relevant CHTs to manage the implementation of the National CHA Program ▪ Technical support provided through the CHWS for ALL Project to support the central MoH and relevant CHTs performance (i.e. manage implementation of the National CHA Program and ensure high quality service delivery) at Rivercess county ▪ Establish mechanism for Continuous learning and quality improvement of the National CHA Program 3.4 Limitation of the End-Line Evaluation Project The evaluation was limited in scope as defined by the CHWS for ALL Project Final Evaluation Consultancy TOR. (See the section: Scope of Work). The interviews did not include the CHAs, CHCs and the HFDCs, as the end line evaluation project was focused on the evaluation of technical support provided LMH to the NCHA program through the CHWS for ALL project; not the evaluation of the NCHA program itself. 3.5 Challenges of the Evaluation In conducting the final evaluation, the evaluation team (ET) noted the following challenges: __________________________________________________________________________________ Copyright © CHWS for ALL Project Final Evaluation - 2018 19 • Limitation in the scope of the evaluation. The CHWS for ALL Project was designed primarily to provide Technical Assistance to CHSD staff and some selected staff of Central MoH with the aim of enabling the successful implementation of NCHA program. Therefore evaluating the progress out the CHWS for ALL Project sometime required evaluating some activities of the NCHA program which was outside the scope of the end line. • Difficulty in mobilizing participants - The mobilization of participants was difficult especially the CHT and central MoH staff. Most MoH staff were absent from the office and/or too busy to avail themselves for the interviews at scheduled times; and where these were not the case, official bureaucracy delayed the process significantly. There was extensive rescheduling of interviews as well as difficulty in reaching participants to schedule interviews. • Limitation of access to information - Some participants were very reluctant to share basic information even after receiving due authorization from relevant authorities.. • Deplorable road condition – impassable roads slowed down the travel time in and out of Rivercess County as well as within the county itself. This delayed the execution of the evaluation activities. • Challenge in conducting evaluation during the Holiday Season – the end line evaluation although schedule to begin at the early part of the quarter of the year, due to delay started towards the end of the year. Most staff ware already in the Christmas mood, therefore getting the attention was challenge. 3.6 Organization of the Report The evaluation report is structured as follows: • Session 1: Provides an Executive Summary of the entire report • Session 2: presents the evaluation purpose, objectives and scope including the key evaluation criteria/questions, • Session 3: give the evaluation approach and methodology including description of the data collection and analysis, compilation of data bibliography, the key evaluation criteria/questions, limitation of the evaluation, the challenges of the evaluation and the organization of the report • Session 4: looks at the Project overview; Project background and description, project objectives, key activities, Results Framework Implementation Approach and Methodology, and Monitoring & Evaluation. • Session 5: describes the evaluation findings, given the Profile of Respondents, the actual Evaluation Results and Sustainability of the Project Achievements • Session 6: itemize the project implementation challenges, lessons learned and proffer relevant recommendations. • Session 7: Captures the conclusion of the report. __________________________________________________________________________________ Copyright © CHWS for ALL Project Final Evaluation - 2018 20 4.0 PROJECT OVERVIEW4 4.1 Project Background and Context In 2015, Last Mile Health (LMH) along with strategic partners like UNICEF, USAID, WHO, and others worked closely with the Ministry of Health (MoH) and others to revise the National Community Health Services Policy and Strategic Plan. Under the pre-existing policy, the community health landscape was highly fragmented, resulting in varied service delivery coverage and quality, as well as weak central MoH governance and oversight. Moving from many poorly aligned and disparate community health volunteer programs towards a single, cohesive and ‘fit for purpose’ professionalized community health workforce, the revised National Community Health Services Policy and Strategic Plan 2016-2021 seeks to address these issues. Incorporating the National CHA Program; the Policy and Strategic Plan positions the MoH to institutionalize a unified approach to providing quality primary health care services to the approximately 1.2 million Liberians living farther than five kilometers from the nearest health facility. Recognizing the challenges posed by fragmentation of community health programming, and conversely the potential of a unified community health program to support a resilient and strengthened health system following Ebola, the MoH embarked on a collaborative effort with partners to develop a plan to rebuild Liberia’s health system. The MoH envisioned a health system that was able to provide more inclusive and equitable health services, readily prepared for the possibility of future outbreaks, and resilient to public health shocks. The MoH articulated this ambition in its Investment Plan for Building a Resilient Health System in Liberia: 2015-2021 (“Investment Plan”) as well as a Health Workforce Program, aimed at revitalizing Liberia’s health workforce following the Ebola outbreak. The Community Health Worker Support for Advancing Liberian Livelihoods (CHWS for ALL) Cooperative Agreement was developed to addressed the above issues and aimed to strengthen the capacity of the Liberian Ministry of Health (MoH) and other community stakeholders to oversee the scaling up a high-quality National Community Health Assistant (CHA) Program that is informed by and held accountable to the remote communities it has been designed to serve. The CHWS for ALL Project was awarded under the USAID-UNICEF Integrating Community Health Program. It is being implemented by Last Mile Health. The core aim of the project is to provide technical and systems strengthening support to the central Ministry of Health and three (3) out of Liberia's Fifteen (15) County Health Teams (CHTs) (Rivercess, Grand Gedeh, and GrandBassa). In addition to the technical assistance provided through the CHWS for ALL Project, Last Mile Health also provides direct support to the three County Health Teams to implement the National CHA Program (Grand Gedeh, Rivercess, Grand Bassa). Today, the National CHA Program is implemented fully in ten counties and partially in an additional four (out of 15 counties), having trained 2,808 CHAs and 357 Community Health Services Supervisors The CHWS for ALL Project provides direct technical assistance to the central MoH to support the scaling of the National CHA Program across the country and 4 19. PROJECT WORK PLAN AND M&E PLAN __________________________________________________________________________________ Copyright © CHWS for ALL Project Final Evaluation - 2018 21 provides embedded technical assistance to the three counties where Last Mile Health serves as the direct implementing partner. Other key implementing and funding partners include: Plan International; PACS/USAID; UNICEF; the World Bank; Global Fund; USAID and various philanthropic funders. Informed by the MoH’s Investment Plan for Building a Resilient Health System in Liberia – 2015 to 2021, CHWS for ALL has aimed to improve the quality and sustainability of the National CHA Program by developing MoH governance and oversight procedures and capacity, accountability and feedback mechanisms, and continuous learning and quality assurance frameworks. Highlighted under Priority Investment Area A: Building a Fit-for￾Purpose Productive and Motivated Health Workforce that Equitably and Optimally Delivers Quality Services’, the National CHA Program in turn should impact health-sector- wide goals to restore Liberia’s gains lost through the EVD crisis, provide health security, and improve progress towards universal health coverage.5 Specifically, to ensure the success and sustainability of the National CHA Program, which forms a key component of this plan, the CHWS for ALL is collaborating with MoH counterparts and other community health stakeholders to: ▪ Establish environmental and operational readiness at the central and county levels Establish environmental and operational readiness at the central and county levels for the introduction of the National CHA Program ▪ Support the MoH to integrate and institutionalize the National CHA Program ▪ Ensure continuous learning and quality improvement for National CHA Program implementation and accountability 4.2 Project Objectives In order to achieve the overarching goal of this project, CHWS for ALL seeks to address the following:  Objective 1: Establish environmental and operational readiness at central and county levels for the introduction of the NCHA Program. CHWS for ALL will create an enabling environment for the successful introduction of the NCHA Program among national, county and community-based stakeholders. This objective seeks to effectively orient critical stakeholders to the recently revised National Community Health Services Policy and Strategic Plan as part of the NCHA Program that is inclusive of community-based stakeholders and community structures, and begins to establish mechanisms for the MoH to remunerate CHAs and plan for long-term financial needs. This objective takes the first step to effectively engaging community stakeholders including traditional and religious leaders, as well as women’s and youth groups to set the foundation where perspectives of these groups are captured, responded to, and where appropriate, reflected in the national plans, protocols and standards that make up the NCHA Program. The focus on inclusiveness and accountability at the introduction of the NCHA Program sets the groundwork for equitable distribution of high-quality 5 Scope of Work – End-Line Evaluation __________________________________________________________________________________ Copyright © CHWS for ALL Project Final Evaluation - 2018 22 primary health care services to remote communities that considers local norms and is responsive to the needs of women, youth and vulnerable populations.  Objective 2: Support MoH to integrate and institutionalize the NCHA Program The second objective will chart a pathway towards enhanced country ownership through the CHSD, CHTs, and other community health stakeholders to establish the NCHA Program as an integrated component of the health system. CHWS for ALL will provide technical assistance to the CHSD to: conduct supervision and oversight of NCHA Program implementation, ensure compliance to national policy and service delivery guidelines, coordinate partner efforts, and document any recommendations for the planned revision of the National community Health Services Policy and Strategic Plan in 2021 and the NCHA package post-2017. These efforts will result in more optimal alignment and coordination of partner resources and ensure that the MoH and partners monitor and ensure compliance with the equitable distribution of primary health care services envisioned as part of the CHS Policy. CHWS for ALL will collaborate with the CHSD and implementing partners to improve community engagement, ensuring that formal mechanisms exist for community-based actors, including women’s groups, youth groups, community leaders, CBOs and community health structures have a stake in the NCHA Program and provide regular feedback at the county and national levels. CHWS for ALL will operate through the CHTWG and Steering Committee to ensure that input and feedback is incorporated into policy, guideline and protocol revisions over the course of the project.  Objective 3: Ensure continuous learning and quality improvement for NCHA Program implementation and accountability The ability of the NCHA Program to increase access to high quality primary health care services and reduce the burden of disease will be anchored in a continuous learning and quality improvement approach. CHWS for ALL will support the MoH to ensure uniformity of core programmatic components across geographies in Liberia while promoting creative solutions to local implementation barriers. CHWS for ALL will support the CHSD to establish and maintain an inclusive, participatory process to review and learn from implementation efforts nation-wide, provide knowledge management capacity, and enable the MoH to capture, retain and share critical lessons and decision points in the scale-up of the NCHA Program. __________________________________________________________________________________ Copyright © CHWS for ALL Project Final Evaluation - 2018 23 4.3 Results Framework The following framework below illustrates how the activities envisioned as part of CHWS FOR ALL will contribute to the immediate and ultimate project aims: 4.4 Implementation Approach and Methodology Last Mile Health will leverage its involvement at every level of the health system to undertake CHWS for ALL using a bottom-up approach with feedback mechanisms at each level. This approach will ensure that CHWS for ALL’s technical assistance and international knowledge sharing contributions are informed by on-the-ground experience and local perspectives. CHWS for ALL’s engagement at every level of the health system, detailed below, will impact how the objectives and activities in this project are implemented. A. Community-level engagement, through CHTs and local NGOs/CBOs, to work with CHAs, community health structures, and community-based actors to identify needs, gaps and areas of ongoing improvement in provision of quality primary care services. Under CHWS for ALL, Last Mile Health will leverage its presence in Grand Gedeh and Rivercess counties implementing the NCHA Program to engage with Community Health Committees (CHCs), community-based organizations, women’s groups, youth groups, and community elders, as well as CHAs and beneficiaries of the NCHA Program. Last Mile Health is pursuing complementary funding that would allow it to leverage USAID ICH to achieve more robust county-level support and sub-county-level support. B. County- and facility-level collaboration and capacity building, prioritizing CHT engagement. Through this support, CHTs will work with facility staff, CHSSs, and implementing partners to identify and address barriers to service linkages while improving inclusion of community-based actors in operational planning and execution of the NCHA Program from the county level. __________________________________________________________________________________ Copyright © CHWS for ALL Project Final Evaluation - 2018 24 C. National-level coordination and technical assistance in engaging partners, aligning resources, and incorporating lessons learned at the community and county-level into national quarterly and annual review meetings that can catalyze revisions to national policies, implementation plans and protocols. D. Global-level contributions to regional and international bodies through research, presentations and case studies. 4.5 M&E APPROACH Last Mile Health is committed to saving lives by addressing the top killers of people who live in the most remote areas of Liberia, and as the MoH scales up a professionalized, incentivized, and clinically supervised cadre of community health assistants (CHAs), and over the next five years, the Liberia MoH will work with assistance from Last Mile Health as a technical partner to train and deploy approximately 4,100 professionalized CHAs and 230 supervisors to provide care for approximately 1.2 million individuals living in remote areas, specifically targeting children under five years and women of reproductive age, who together make up roughly 40% of the population. Last Mile Health’s Research, Monitoring, and Evaluation (RM&E) division played a vital role in the pursuit of this mission. As a health services organization, Last mile Health believes that the use of high quality data will enable better decision-making and improved patient outcomes; data should inform activities at every level. This M&E Plan is designed to describe how Last Mile Health will monitor and evaluate progress against the overall goals and activities related to the USAID ICH award APS-OAA-15-000005. The indicators that will be used to measure progress against grant deliverables, as well as the rationale for the particular selection of the indicators, are described in detail. Additionally, this plan outlines the data collection processes that will be used in order to measure values of the selected indicators.  M&E Strategy and Goals Monitoring, evaluation, and learning for the proposed activities will proceed within a complexity-aware Plan-Do-Study-Act framework to synthesize routinized data from multiple stakeholders into formats that are responsive to the needs of policymakers, program managers, and community members in accordance with best practices.  M&E Objectives There are four overarching objectives of the M&E component of this project: 1. Process monitoring. Continually ensure that the achievement of project deliverables is on track by periodically measuring progress against elected indicators, allowing for transparency among all government and NGO stakeholders across program domains. 2. Research, global learning, and knowledge management. Support a continuous learning and improvement cycle within the MoH through the design and execution of research and global learning activities, and facilitate dissemination and institutional knowledge management __________________________________________________________________________________ Copyright © CHWS for ALL Project Final Evaluation - 2018 25 3. National monitoring system design. Support the MoH to bring together various governmental and non-governmental stakeholders to design, implement, and refine a high-quality system for monitoring the national CHA program. 4. End-line evaluation. Facilitate the completion of an end-line evaluation of the entire project, in order to enhance internal and external accountability by assessing the extent to which grant activities have been implemented as designed, as well as to identifying all major and minor obstacles that prevented the achievement of overarching programmatic goals. __________________________________________________________________________________ Copyright © CHWS for ALL Project Final Evaluation - 2018 26 5.0 EVALUATION FINDINGS 5.1 Profile of Respondents The profile of respondents captured during the assessment included their gender, education, age, and number of years spent in their current positions. The tables and graphs below summarize the profile information gathered from the respondents. Composition of KII Participants 63% 37% Male Fema 10 14 4 6 12 0 4 4 0 2 4 6 8 10 12 14 16 CHSS CHT LMH MOH Male Female 0 2 4 6 8 10 12 14 16 26 - 30 years 31 - 35 years 36 - 40 years 41 - 45 years 46 - 50 years Above 50 yrs Not mention Age range of respondents __________________________________________________________________________________ Copyright © CHWS for ALL Project Final Evaluation - 2018 27 Box 1: Respondent Profile Summary  54 participants • 22 Community Health Services Supervisors (CHSS) • 14 Rivercess County Health Team (CHT) staff • 8 representative of Last Mile Health (LMH) • 10 Central MoH staff, especially the CHSD  63% males participants  37% Females participants  35% obtained bachelor degrees, 30% obtained associate degrees, 17% obtained High school education or in college currently while 18% obtained master degrees  The ages of the participants fell 26 years mid 50 years • 54% participants were 35 years or below • 29% participants were fell between 36 years and 45 years • 17% participants were above 45 years  All the participants interviewed indicated spending at least one month to 8 years in their current positions  19% participants spent less than a year in current positions  50% participants spent one to two years in current positions  20% participants spent three to four years in current positions  11% participants spent more than four years 8 6 15 5 1 10 4 5 0 2 4 6 8 10 12 14 16 Male Female __________________________________________________________________________________ Copyright © CHWS for ALL Project Final Evaluation - 2018 28 5.2 Evaluation Results  Environmental Readiness The CHWS for ALL project provided support for planning, organizing, and conducting orientation workshops on the NCHA strategic plan and policy in Rivercess County6 . Although during the evaluation interviews, a majority of participants could not remember attending the orientation workshops which could be due to general attrition and the movement of MoH staff between departments due to the change in government. Notwithstanding, these respondents stated that they got oriented on the NCHA strategic plan and policy either by reviewing copies of the policy or by acquiring such knowledge of the policy from discussions about them from other workshops and meetings attended. The CHWS for ALL project worked with CHSD to develop advocacy materials and support targeted outreach for National CHA Program with key stakeholders within MoH Leadership7 . These materials explained the operations and benefits of National CHA Program to the target counties and were used to create awareness and promote the NCHA program. The project worked with the county health team and community health structures to improve leadership to promote a learning environment and model the desired quality of the NCHA program. The data collected and analyzed shows that the CHWS for ALL interventions have strengthened the county and national level leadership. Specifically the CHTs and Central MoH staff have improved their leadership skills as they are using supervision tools and processes for standard supervision and monitoring, accessing data from DHIS2 to make decisions, conducting training, coaching and mentoring to improve the capacity of other NCHAP stakeholders (CHSS, CHCs etc), conducting monthly revision of supervision work plans. Moreover, at the central MoH, staffs are using data analysis tools such as Stata to review and analysis data, creating and managing governance structures through the development and use of tools including the development of SOPs, Tools, Processes and development TORs for TWGs. The CHWS for ALL core values emphasized quality and continued improvement. It supported and acculturated these core values throughout the implementation process to ensure that all stakeholders see their contributions to the quality of the NCHA program. While the project did not involve the deployment of quality assurance officers, other LMH projects directly implementing the NCHA program provided such support, from which the project benefited. Hence, quality assurance procedures were 6 5. AID-OAA-A-16-00028_Annual Report 2018_Activity Table 7 ibid __________________________________________________________________________________ Copyright © CHWS for ALL Project Final Evaluation - 2018 29 provided at every stage of the implementation. For example, they assigned quality assurance officers to each district to support these values at the county level.  Operational Readiness The CHWS for ALL Project supported the CHT to strengthen the community health structures (including CHCs, HFDCs) and engage local CSOs/CBOs to improve local leadership and sustainability within Rivercess8 . The CHT also supported the establishment of new CHCs in communities that did not have CHCs prior to the project interventions. It was said during the evaluation interviews, that the project intervention also supported the coordination, supervision and meetings of the Community Health Technical Working Groups (CHTWGs) and District health teams. The CHWS for ALL Project also supported the delineation of responsibilities and accountability for oversight, coordination, and implementation of NCHA Program. These structures are essential in the sustainability and ownership of the NCHA program. Community engagement seems to be at its best in the CHA implementation. HFDC meetings are held with Community Health Committee (CHC) participation. There are established CHCs in all communities with CHAs,9 however, some of the respondents mentioned that some of these CHCs are not very functional at the moment as they do not hold regular meetings. Mobilizing such CHCs is still a challenge with the project because they are not yet fully responsive to their roles and responsibilities. The rate of CHCs holding monthly meetings has been dropping steadily from 92% in Q4 to 15% in Q7. Although At the time of preparing this evaluation report; it has risen to 75% in Q8 based on based on recent project data received from LMH.10 Towards long-term sustainability of the program, CHWS for ALL continues to draw upon a comprehensive suite of financial planning and budgeting tools to support the identification and cultivation of new funding mechanisms, while also charting towards increased fiscal commitment by the Government of Liberia. Of the estimated $95 million total cost to reach and maintain full coverage via the National CHA Program between 2016 and 2021, over $32 million has been committed to date with additional coordination underway to support alignment between the Government of Liberia and USAID financing priorities11. While it is true there is still room for improvement to attain environmental and operational readiness, the synergy of the essential support provided by the CHWS for ALL Project, to a great extent, created some level of readiness and has increased 8 5. AID-OAA-A-16-00028_Annual Report 2018_Activity Table 9 CHSD QRM_August_9_V1 presentation 10 LMH Indicator data Table for End-Line Evaluation Project 11 1. AID-OAA-A-16-00028_Annual Report 2018_vf.docx __________________________________________________________________________________ Copyright © CHWS for ALL Project Final Evaluation - 2018 30 the operational and environmental preparedness of the central MoH and relevant CHTs to manage the implementation of the National CHA Program. Currently new staffs are oriented on the policy through on-going on the job coaching conducted with CHT to ensure they remained familiar with the documents and to orient new staff members. Due to the attrition of personnel at the MoH, it is recommended that future project should include a standard operation procedure (SOP) for orientating new staff on the National CHA Policy and Strategic Plan. Cash payment mechanism is still used for payment of CHAs in Rivercess. Mobile money is not yet fully feasible as a payment mechanism due to poor internet (GSM network coverage). While the project has made significant effort in identifying various payment mechanisms, no alternative payment mechanism besides direct cash payment is been used to pay CHAs. It is recommended that future support project examines other payment mechanism from the perspective of the local communities such using local Village Saving Groups (‘Susu clubs etc.) as a payment mechanism. The project provided technical support to MoH to complete proposals for various funding sources, to cost the National CHA Program in order to anticipate and mobilize funds to sustain and expand the program as well as to conduct resource mapping on a regular basis to identify current needs and plans for funding gaps12. Based on the desk review of the project document13, the amount of funding from GOL and donors earmarked for CHA Program beyond current levels is US$38,810,000. However, there are no available data (based on the scope and timing of this End-Line evaluation) to indicate the portion of the above amount that MoH has been able to solicit directly from internally funding i.e. (from the GOL). It is noted that GoL institutions are providing only marginal funding for NCHAP implementation, potentially jeopardizing the program financial sustainability14. This conclusion is also supported by the result of central MoH respondents’ data analysis as most respondents indicated that the GOL is not financially supporting the program. It is recommended that future support project should also place considerable emphasis on supporting the MoH to advocate; lobby and engage GOL to source and/or solicit internally generated revenue from the national budget to support funding for the NCHA program besides the overly dependence on solicitation of funding from international donors. This is particularly important to improve the MoH environmental and operational readiness to owe the NCHA program. For example, support MoH capacity to advocate for a percentage of all concession tax revenues receipt by the GOL in a particular county to be directly allocated to support the funding of NCHA program in that county amongst other innovative fund generation mechanism. 12 National Community Health Assistant Program: Financing Advocacy Brief 13 2. AID-OAA-A-16-00028_Annual Report 2018_MEL Table 14 Resource Mapping Update __________________________________________________________________________________ Copyright © CHWS for ALL Project Final Evaluation - 2018 31  Support to MoH to integrate and institutionalize the NCHA Program The CHWS for ALL project built capacities to ensure that staff have the necessary technical, managerial and leadership knowledge and skill to carry out their responsibilities, and that they know when and how to use these skill best. The CHWS for ALL consulted with IRC/PACS, CHAI, UNICEF, and other community health stakeholders to align and provide technical assistance and capacity building for CHSD and relevant MoH divisions/programs. It assisted CHSD to revise TORs for the Health Services Coordination Committee15. At the central MoH level, the respondent data analysis indicated that 50% of the respondents (5 persons) were part of the planning committees for the development of the NCHA program. The data also showed that 3 of this number are still in their current position while 2 persons have moved on to different positions in the MoH. Based on data collected, 100% of the respondents overwhelmingly acknowledged and demonstrated their role and responsibilities as linked to the objectives of the NCHA program. This indicates a high level of integration and institutionalization of the NCHA Program. In addition, the CHWS for ALL Project’s interventions have reinforced the importance of capacity building and the need for a core of local quality assurance coaches and trainers to keep up with the demand and expansion of the NCHA program. Furthermore, there is evidence that the coordination mechanisms within the Government have been strengthened, with draft terms of reference (TORs) developed for the Community Health Technical Working Group, the Community Health Steering Committee, and the National CHA Program Inter-Ministerial Coordination Committee. CHWS for ALL also supported representatives of the National CHA Program within forums such as the Health Sector Coordination Committee, chaired by the Minister of Health, and the Human Resources for Health, Supply Chain, Health Information System, and Health Financing Technical Working Groups16 CHWS for ALL supported the communication and information sharing, learning, and advocating for the quality of the NCHA program. Improvements and changes were recorded, using data to demonstrate result and how the results were achieved in QRMs. Information sharing of all from sources: IFI, PP and CBIS is done at the national QRMs. The CHTs and MoH staff also has access the DHIS2. However, the evaluation team observed that some stakeholders such as the District Health Officers (DHOs) and some CHTs were having challenges with providing data due to poor internet service and non-use of their access to the DHIS2. All the data are shared quarterly with 15 5. AID-OAA-A-16-00028_Annual Report 2018_Activity Table 16 1. AID-OAA-A-16-00028_Annual Report 2018_vf.docx __________________________________________________________________________________ Copyright © CHWS for ALL Project Final Evaluation - 2018 32 CHSD at the county and national QRMS. Documentation review shown that there is 100% reporting rate for supervision for stakeholders implementing the National CHA program17. The achievements of the CHWS for ALL are gaining strong recognition from key stakeholders. The technical support provided under the CHWS for ALL Project support has increase the performance of a county level model in Rivercess. The evaluation data showed that an aggregate of 75% of Central and County MoH staff respondent interviewed; expressed strong confidence in their ability to oversee both management and technical aspects of the National CHA program in their roles, based on the support and training acquired through the CHWS for ALL Project. The respondents confirmed that they had acquired the following skills and competences from the Project: • Development of TOR for monthly TWGs, • Use of supervision tools and processes for standard supervision and monitoring • Use of data use for decision making • Training, coaching and mentoring skills • Revision of supervision work plans on a monthly basis • Use of data analysis tools such as Stata to review and analysis data • Creation and management of governance structures and tools including the development of SOPs, Tools, Processes and TORs • Joint research for Program Perceptions study with LMH that enhanced their qualitative research skills. • Continuous learning and quality improvement for the NCHA Program implementation and accountability The Liberia’s Community Based Information System (CBIS) serves as the National CHA program’s primary data collection toolkit, and is fully integrated within DHIS2. With rollout complete across 14 of Liberia’s 15 counties, increased focus is being placed on individualized support to County Health Teams to improve the frequency and quality of reporting18. Strengthening of NCHA program to ensure sharing and dissemination of important program information, continuous learning and quality improvement is required. Thus the CHWS for ALL supported platforms to enhance the program activities. According to the respondents, the support given include participant transport, DSAs/lunch, logistics, etc. in the table below indicates the meetings covered: Activity/meeting19 Frequency 17 National Community Health Assistant Program Improvement Tracker 18 1. AID-OAA-A-16-00028_Annual Report 2018_vf.docx 19 Activities and their frequencies were indicated during key informant interviews __________________________________________________________________________________ Copyright © CHWS for ALL Project Final Evaluation - 2018 33 Coordination Meetings Monthly Technical Working group Monthly CBIS sub-working group Monthly Integrated Supportive supervision and monitoring Monthly HMER Technical Working group Monthly HMER Coordination Committees Monthly Performance Review meetings Monthly Program Review Meetings Quarterly Quarterly review Meetings Quarterly Community Health Services Supervisor Retreat Quarterly National Community Health Services Retreat Quarterly Quality Assurance Officers Retreat Quarterly District Health Officers Retreat Quarterly Quality improvement Training Yearly HMIS Refresher Training Yearly Logistics Management Information System (LMIS) Yearly Personal Appraisal Management (PAM) Yearly Community Even Based Surveillance Yearly Community based Information System forms Review Yearly Integrated MCH Meetings Regular* Staff Mentoring and Coaching Regular* Trainings and workshops Regular* *There are some of these kinds of workshops, meetings and trainings that are not held regularly as highlighted by the respondents. Although the Evaluation Team did not access documentation such as minutes of the meetings and activities mentioned above, responses from the interviews show that some of these continued learning activities did not occur as planned. Some reasons proffered for these discrepancies include constraints with logistics, unforeseen emergency activities, and competing priorities (i.e. national plans competing with county plans). For example, the quarterly review meetings should be held 4 times a year; however, the ET observed that the last one held was the third QRM for the year which was mainly due to the change in the leadership of the government of Liberia. To support more learning, the CHWS for ALL supported the development and submission of publication in a peer-reviewed academic journal. Document reviewed show that two Research Projects have been approved as part of the MoH Research Agenda relating to NCHAP20. During the time of preparation of this evaluation report; reporting, 2 publications have been developed and submitted in a peer-reviewed academic journal while 6 of the research studies has been approved as part of national research agenda. 20 CBIS Update to CHSD QRM_August_9_V1(1) __________________________________________________________________________________ Copyright © CHWS for ALL Project Final Evaluation - 2018 34 The table below summarizes the contributions made by the project at the county and central Ministry level Table 2: Project Contributions (Outputs)21 Goal Objectives Expected Results Contributions (support) Goal: Develop the capacity for the MoH and community-based stakeholders to oversee the scaling up of a high-quality NCHA Program that is informed by and accountable to the remote communities it has been designed to serve. Objective 1: Establish environmental and operational readiness at the central and county levels for the introduction of the NCHA Program Result 1: Stakeholders well￾oriented on NCHA Policy and Strategic Plan ● Large scale orientation of the community health policy and strategic plan was conducted at the launch of the program ● 3 orientation Workshops (Q2, Q4 & Q5) conducted with 49 government bodies, partner or CBO organizations represented ● Supported CHSD to conduct follow-up activities to address questions on Policy, SOPs, and other National CHA Program Materials and Data Collection Tools ● Worked with CHSD to develop advocacy materials & support targeted outreach for National CHA Program with key stakeholders within MoH Leadership ● Ongoing on the job coaching conducted with Rivercess CHT to ensure they remained familiar with the documents and to orient new staff members. ● Overview of the NCHA program integrated into various presentations given to major stakeholders (Annual Health Conference 2017 & 2018, GFF meeting, etc.) Result 2: NCHA Program implementers empowered to more effectively and efficiently pay CHAs ● Mobile Money and Cash payments were identified as the two main payment mechanisms ● Cash payments are used in Rivercess due to poor GSM network coverage and lack of mobile money vendors. Efforts are ongoing to explore transitioning to Mobile Money. This includes speaking with Mobile Money 21 Annual Report 2018_MEL Table, CHWS for ALL Activity Table and based on analysis of interview results __________________________________________________________________________________ Copyright © CHWS for ALL Project Final Evaluation - 2018 35 providers to ensure that cash required to be paid to CHAs is available each month ● Report on payment mechanisms will be completed and circulated in Q1 FY19 to relevant stakeholders Result 3: MoH empowered to better support and sustain the NCHA program through Innovative financing mechanisms ● Technical support provided to the MoH to complete proposals for various funding sources, including Global Fund and the World Bank. ● Budget forecasting tools and accompanying documents developed and submitted to GFF ● 6 of additional sources of finance earmarked for NCHA program beyond the sources available today ● Costing of the National CHA Program was completed allowing the MoH to anticipate and mobilize funds to sustain and expand the program. ● Resource mapping with the MoH is done on a regular basis to identify current needs and plans for funding gaps to ensure there is no disturbance in the implementation of the program Objective 2: Support MoH to integrate and institutionalize the NCHA Program Result 1: CHSD capacitated to oversee and manage NCHA Program ● Supportive supervision tools and processes that allows for standard supervision, monitoring and data use for decision making. By ensuring that that CHSD is involved in reviewing supervision work plans on a monthly basis, they are better prepared to manage the process on a regular basis. ● CHSD staff had accompanied LMH when conducting research for Program Perceptions study in order to build their qualitative research skills. ● HMER staff coached on the use of Stata in order to be able to review and analysis. ● Coaching on Google Suit and other tools that can be used to manage the NCHA program. ● Technical support to develop and maintain health financing tools (costing of the NCHA program, resources mapping ect.) ● Roll-out CBIS in Rivercess and training on accessing and analyzing CBIS data from DHIS2 to ensure program implementers can clearly review and use data. Result 2: Rivercess CHT capacitated to ● Supported orientation on the community health supply chain SOP __________________________________________________________________________________ Copyright © CHWS for ALL Project Final Evaluation - 2018 36 integrate and implement NCHA Program ● Coaching on LMIS and how to request sufficient commodities for the NCHAP ● Supported orientation on CBIS and how to analysis data for decision making ● Supported orientation on MoH human resources handbook and coaching on how to address motivation and other staffing concerns related to CHSSs and CHAs ● Supported planning and implementing QRMs at the county level to review data and address key challenges identified. ● Supported the re-establishment of County Health Boards to ensure proper governance structures ● Coaching on conducting monthly supervision to ensure that CHT staff are correctly filling out the supervision tools and asking probing questions Result 3: Community health structures strengthened and local CSOs/CBOs engaged to improve local leadership and sustainability within Rivercess ● General mapping of CBOs/CSOs were done to ensure that Rivercess CHT was familiar with organizations present ● Orientation on the community health policy and strategy was conducted with CBOs/CSOs ● Rejuvenated the County Health Board to ensured that there is a sustainable platform for a CSO to continue to collaborate with the CHT (as a CSO is required to sit on the board) ● Supported the strengthening of community health structures (e.g. CHCs, HFDCs) to improve local leadership and sustainability within Rivercess. Objective 3: Ensure continuous learning and quality improvement for the NCHA Program implementation and accountability Result 1: NCHA Program stakeholders have improved program quality and accountability based on feedback from regular, participatory NCHA Program Performance review ● 2 QRMs (Q6 & Q7) for which dashboard for National CHA Program M&E framework and implementation fidelity metrics was generated (for cross￾county/partner comparison of performance and consequent corrective action planning to improve quality and accountability) ● Reviewed and analysis of IFI and Program Perceptions data and development of presentations in partnership with MoH. ● Coaching on Stata for HMER staff to ensure that they can analysis and share data collected through the IFI study. ● Maintained of the National CHA Program Improvement Tracker that is used to track action points identified at the QRMs. __________________________________________________________________________________ Copyright © CHWS for ALL Project Final Evaluation - 2018 37 Result 2: NCHA Program stakeholders have Improved understanding of perceptions of NCHA Program ● 2 publications developed and submitted in a peer-reviewed academic journal ● 6 of research studies approved as part of national research agenda ● Joint Integrated Supportive Supervision tool and IFI could be integrated to allow for one supervision structure at the MoH ensuring that it is more sustainable. One QRM that brings together multiple divisions would also ensure that all partners and divisions are aligned and not duplicating efforts or resources. ● To ensure that the National CHA Program are integrated into the larger health system, it requires collaboration with all divisions (HRH, Nutrition, Family Health Division, EPI, NTDs etc.). __________________________________________________________________________________ Copyright © CHWS for ALL Project Final Evaluation - 2018 38 5.3 Project Achievements The following are key achievement of the project: • The development of key processes that enabled the Ministry of Health to regularly monitor the progress of and anticipate challenges with the National CHA Program. This includes programmatic challenges that were highlighted by CBIS, IFI and Program Perceptions data and were tracked through the program improvement tracker as well as financial challenges that were anticipated through resource mapping. As a result of these well-established processes and data collected, the CHWS for ALL project was able to advocate for the National CHA Program to be included in the Ministry of Health 100 day priorities during the political transition. • Technical support to develop and maintain health financing tools (costing of the NCHA program, resources mapping etc.) • The Roll-out of CBIS in Rivercess and training on accessing and analyzing CBIS data from DHIS2 that ensures program implementers can clearly review and use data. • Provision of support to CHT and Central MoH in conducting the QRMs established a major platform to review program achievements, identify gaps and necessary actions to mitigate the gaps. 5.4 Sustainability of the Project Achievements The Project Sustainability presents the extent to which the CHWS for ALL project activities, results and effects are expected to continue after the project intervention has ended. The ET measured the level at which MoH and direct beneficiaries are able to autonomously manage the NCHA program with or without future donor support. The desk review of various documents and interviews conducted showed that the CHWS for ALL project placed strong efforts in the establishment of technical and institutional sustainability, but lacks strong mechanisms to ensure financial sustainability. Overall, the mechanisms to ensure long-term sustainability of the project achievement were developed. The CHSS, CHT, Central MoH and partners interviewed are hopeful that the CHWS for ALL interventions will be sustained in the long-term. 90% of the respondents were confident that the CHWS for ALL project has been of tremendous support to the implementation of the NCHA Program in Rivercess County. However, it is also noted that the progress achieved by the project can only be sustained if certain processes and activities that require financial support are adequately funded by MoH. Majority of the respondents are of the view that without the adequate funding support currently being provided by external donors such as LMH for implementing processes and activities such as QRMs, all forms of field supervisions, training, coaching, monitoring and evaluation etc. the degree of sustainability of the project achievements will decline. __________________________________________________________________________________ Copyright © CHWS for ALL Project Final Evaluation - 2018 39  Financial Sustainability All the respondents mentioned that NCHA program and CHWS project have no assured or clear financial sustainability strategies but rather consultations are ongoing. For example, many of the respondents mentioned that relevant authority such as the National legislature is being engaged to consider budgetary allocation for the NCHA program, while a 5-year budget forecasting tool and accompanied documents were developed and submitted to the GFF and awaiting response and action. These current conditions of the program led to some of the participants lamenting; as one said “There is no sustainability strategy at the moment except that we are pushing that GOL must consider budget allocation for the NCHA program”. Another mentioned “as Government, we do not have money to support the program and the program will continue if donors give money”. Several respondents indicated that they are concerned that the NCHA program will not survive without the continuous support of donors. One concluded that “if donors do not provide money, the program will die”. One of the respondents had a recommendation, stating “Since there is no plan to sustain the program, GOL should stop free services and attach affordable fees to the services provided.” Clearly there is a concern by most respondents with regards to the lack of any available GoL strategies to effectively fund the NCHA program. During the preparation of this report, the Evaluation Team did not access any existing documentation of MoH strategies __________________________________________________________________________________ Copyright © CHWS for ALL Project Final Evaluation - 2018 40 to financially sustain the NCHA program when the CHWs for ALL project comes to an end. While sustainability strategies may have been developed by top MoH/GoL management, the lack of access to such strategies by most respondents especially at the county level, presented an evaluation challenge for the ET as one cannot expect the respondents to know about high level strategic discussions on NCHAP sustainability which have not been communicated across board. The scope limitation of the Evaluation assignment including the non-inclusion of other donors and financiers of the NCHA Program as part of the target group for interviews was also a challenge.  Technical Sustainability The CHWS for All Project supported the development of several tools, data collection, management and analysis systems, training standard operating procedure (SOPs), supportive supervision tools, processes for standard supervision, monitoring, the use of data for decision making; the use of Google Suit and other tools that can be used to manage the NCHA program. These processes and activities supported by the CHWS for ALL project ensured that the Project was successful in delivering it core objectives to achieving its overarching goal. Notwithstanding, to ensure technical sustainability of the processes and activities developed and supported by the Project; it is essential that top management of MoH institutes procedures (and/or directives) that mandates all NCHAP activities including reporting to use the tools and follow the processes developed and supported by the project.  Institutional Sustainability The project supported the CHTs in the organization and management of Community￾based Structures (CHCs & HFDCs) to ensure continuous provision of the NCHA program services in the rural communities. Training support were provided through the CHTs to strengthen the CHCs, HFDCs, CHSS and CHAs structures to ensure post￾project institutional sustainability. Furthermore, the CBIS serves as the National CHA Program’s primary data collection toolkit and is fully integrated within DHIS2 and Health information system, further promoting institutional sustainability of the data system. In additional, joint research processes i.e. IFI, PP served as additional sources of monitoring and evaluation that is being built into regular CHSD and HMER work plans. Information sharing of all from sources: IFI, PP and CBIS are done at the national QRMs. The CHTs and MoH staff also have access the DHIS2 system. However, some stakeholders have challenges with accessing the data due to poor internet service and non-use of their access to the DHIS2. All the data are shared quarterly with CHSD at the county and national QRMs. While the primary responsibility for sustainability depends greatly on the communities themselves, continuous support, monitoring and evaluation will be required from all stakeholders especially the local and national government to ensure sustainability. __________________________________________________________________________________ Copyright © CHWS for ALL Project Final Evaluation - 2018 41 6.0 IMPLEMENTATION CHALLENGES, LESSONS LEARNED AND RECOMMENDATIONS 6.1 Implementation Challenges From the project perspective, the main implementation challenge identified during the evaluation was the Change in Government due to presidential election of 2017: The political transition led to the reorientation of MOH staff members not only on the CHWS for ALL project but also on the NCHA Program as a whole. There was also a slow reappointment of staff which led to further delays. As a result, improvement or changes to the NCHA Program was paused until the MOH was fully orientated to the NCHA Program. 6.2 Lessons Learned The project presented important key lessons learnt from the evaluation findings such as:  Constant engagements improve information-gathering and sharing which enables successful project delivery: Constant engagement of targeted communities by the CHTs improved cooperation among stakeholders and the overall results of the project. This enagagment built trust among stakeholders that enable easy gathering and sharing of information that was essential to the success of the project.  Understudy increases performance – the process of CHSD staff accompanying LMH when conducting research for Program Perceptions study contributed to them improving their own qualitative research skills.  Pre-implementation Capacity Assessment enables more effective capacity development – the Project conducted comprehensive capacity assessments to examine the environmental and operational readiness of the CHTs to implement the National CHA Program which lead to the design of the capacity development plans tailored to improve the specific needs of each CHT instead of conducting general capacity development that would not have achieved optimal result.  Coaching and Mentoring are essential for effective capacity development – the project incorporated the provision of hands-on on the job coaching and mentoring support to CHTs which improve the ability of CHTs to transfer skills and knowledge to other stakeholders in the NCHA Program.  Effective Data collection, processing, storage, analysis and access for use, significantly improves project implementation and support advocacy - The development of key processes enabled the Ministry of Health to regularly monitor the progress of and anticipate challenges with the National CHA Program. Also data collection and __________________________________________________________________________________ Copyright © CHWS for ALL Project Final Evaluation - 2018 42 analysis systems enable successful advocate for the National CHA Program to be included in the Ministry of Health 100 day priorities during the political transition. 6.3 Recommendations Based on the evaluation findings, the following are key recommendations for ensuring continued sustainability of the project achievements and for consideration in the design and/or implementation of future technical interventions: o GOL should allocate Specific National Budget for the NCHA Program: Lack of direct and adequate Government funding for the program presents a serious risk in the NCHA program. The project has supported the MoH to develop health financing tool, develop the estimated program budget estimates as well as the development of resource mapping template for fund sourcing for the NCHA program. These tools and corresponding efforts have been mainly geared towards solicitation of external funding for the NCHA program. It is recommended that the project support MoH to place considerable more effort and emphasis in using these tools to solicit for revenue from within the country i.e. from the national government and other internal sources instead of the overly dependence on solicitation of funds from external donors. This is particularly important to improve the MoH environmental and operational readiness to owe the NCHA program. For example, the MoH can advocate for a percentage of all concession tax revenues receipt by the GOL in a particular county to be directed to funding of NCHA program in that county amongst other innovative fund generation mechanism. o MoH should demonstrate practical ownership through funding and the mandatory use of the Project developed tools and processes: Overall, the mechanisms to ensure long-term sustainability of the project achievement were developed and the CHWS for ALL project contributed tremendously to the implementation of the NCHA Program in Rivercess County. However, lack of direct MoH funding and general civil service bureaucracy can affect sustainability of the progress achieved by the project. Lack of adequate MoH/GoL funding support key program processes and activities such as QRMs, all forms of field supervisions, research studies, data gathering, storage and access including regular training, coaching, monitoring etc. will effect sustainability. By committing its own funds to the program, the MoH will demonstrate ownership that will also improve the commitment of those engaged in the program. MoH should also demonstrate ownership by mandating all NCHAP activities including reporting system to strictly adhere to using the tools, conducting the activities and implementing the processes developed by the CHWS for All project for the program daily management and periodic reporting. __________________________________________________________________________________ Copyright © CHWS for ALL Project Final Evaluation - 2018 43  The Project implementer should develop and implement a Strategic Exit Plan to support the sustainability of the project achievements: Such project strategic exit plan should include the incorporate the following actions points: o Financial Sustainability: a phased mechanism for generating and allocating funds for the NCHA program from different sources including MoH/GOL budget, Service charge, external donors and other sources. o Technical Sustainability: top management of MoH should institutes procedures and directives that mandate all CHSD personnel to strictly adhere to using the tools, conducting the activities and implementing the processes supported by the project for the daily and periodic management of the NCHA program. o Institutional Sustainability: technical support for the CHTs to enable them to provide continuous training, coaching, supervision and monitoring to ensure community￾based Structures (CHCs & HFDCs) are functional and improving through the use of collection, analysis and use of appropriate data. o Continuous Learning: Conduct of periodic research and survey to evaluate the functionality and impact of the program. o Other measures to enhance sustainability.  MoH Prioritize Support for the developed Project Processes and Activities: The CHWS for ALL Project has successfully contributed to the implementation of NCHA Program in Rivercess County through its technical assistant. However, since MoH currently lacks adequate funding to internally self-finance the project processes and activities, it is critical, that donors such as LMH continue to provide funding required for technical and other support for the NCHA Program.  Other Important Recommendations: The following important cross cutting recommendations. Although, outside the scope of this evaluation, these issues effect the central MoH and CHTs staff (which the CHWS for All Project) ability to manage and oversee the NCHA program: o MoH should create employment pathways for CHSS and CHAs: the lack of official employment pathways for CHSS and CHAs into the MoH official civil service payroll is an issue. They expressed that they have no sense of job security, which is affecting their commitment to the program implementation. o MoH and partners should increase motivation for CHAs: Technically, CHAs are community dwellers who in addition to providing local medical services are still farmers. As the main focal point for basic medical services at the community level, there is need for increase motivation for them to ensure they are providing effective services to their communities as well as providing the correct and complete data for the effective implementation of NCHAs Program. . It is necessary to motivate them to focus more on their CHAs activities. __________________________________________________________________________________ Copyright © CHWS for ALL Project Final Evaluation - 2018 44 o MoH should improve security of CHSS and CHAs: There is a need to develop strategies to improve security for CHSS & CHAs, especially the females; who travel long distances (sometime in thick forests and isolated areas) to access hard to reach rural communities. These strategies may include pairing them with male counterparts, providing security and safety training (e.g. self-defense training), providing security equipment, such as torch lights, pepper spray, etc. There is also a need to develop a security protocol, with hotlines and chain of communication in cases of security issues. o MoH should develop Strategies to Mitigate Program Staff Attrition: The Project should develop strategies to reduce attrition among CHSS and CHAs, while providing a structured training and development process that quickly brings new CHSS and CHAs into the program. Currently, respondents indicated that CHSS & CHAs are not replaced in line with the standard national protocol. Such Strategies may include incentive packages for hard-to-reach areas, performance incentives such as end-of￾year bonuses, etc. o MoH should support and improve the drug supply chain: One of these critical support areas (although outside the scope of this Project) is the need for improvement of the national drug supply chain mechanism to ensure availability of essential drugs in the communities. Improvement in the supply chain of drugs and other logistics which are critical to achievement and sustainability of the NCHA program. o MoH should improve the capacity of other MoH department particularly at the Central level: The evaluation findings reveal that while the Project has greatly improve the capacity of the CHTs and staff of the CHSD and finance department at the Central MoH, the lack of adequate capacity of some staff of other departments whose functions directly impact that of the CHSD, inadvertently effects the overall performance of the CHTs and the general CHSD staff. MoH should consider implementing a capacity development project with based on approach and methodology applied in the CHWS for All project. o GOL should provide and/or improve basic infrastructure especially roads: Poor basic infrastructure especially access roads are implement elect in the success of the NCHA program,. MoH should engage GoL and other partners to build or rehabilitate roads and bridges access to rural health facilities. 7.0 CONCLUSION Given the challenges of accessing, providing and supporting technical support to Rivercess County, LMH was able to a large extent build operational and environmental readiness of the central and county level MoH to implement and scale the National CHA Program. The desk review of project documents and analysis of data collected from key stakeholders during interviews demonstrated a general consensus that CHWS for ALL Project has greatly __________________________________________________________________________________ Copyright © CHWS for ALL Project Final Evaluation - 2018 45 improved the ability of MoH staff at Central office and particularly at Rivercess County to implement the NCHA program. The CHWS for ALL project contributed tremendously to the implementation of the NCHA Program in Rivercess County. However, lack of direct MoH funding and general civil service bureaucracy can affect sustainability of the progress achieved by the project. The Project was able to increase the operational and environmental preparedness of the central MoH and relevant CHTs to manage the implementation of the National CHA Program. The MoH still needs to demonstrate practical ownership of the program through increase finance commitment and to the program. As one key respondent aptly said’ if external donors do not continue to support this program’ we are dead”. The technical support provided under the CHWS for ALL Project increased the performance of a county level model in Rivercess, the progress report of the NCHA program reflects this performance (the scope of this evaluation project does not include the performance NCHA program results). However, in terms of staff capacity development, the CHTs and Central MoH staff have greatly improve their skills and competences which reflects in their use and analysis of CBIS data from DHIS2 for decision making, operations and presentation; their improvement in the organization and management of QRMs; the use of supervision tools and processes for standard supervision and monitoring etc. Notwithstanding, to ensure high quality service delivery, MoH should mandate all NCHAP activities including reporting system to strictly adhere to using the tools, conducting the activities and implementing the processes developed by the CHWS for All project for the program daily management and periodic reporting. The Project developed key processes that have enabled the Ministry of Health to regularly monitor the progress of and anticipate challenges with the National CHA Program. This includes programmatic challenges that were highlighted by CBIS, IFI and Program Perceptions data and were tracked through the program improvement tracker. This has contributed to continuous learning and quality improvement of the National CHA Program. The Project also provided technical support which developed health financing tools that enabled the MoH develop the program budget estimates as well as to anticipate financial challenges through resource mapping. Going forward, the MoH has to place additional emphasis on using these tools to advocate for increased revenue from within the country rather than solely depending on raising funds from external donors. Finally it is important to emphasis that NCHA program structures developed and supported by the CHWS for All Project should be owned by the beneficiary communities, the MoH and; should be fully supported financially by the MoH in order to ensure overall sustainability of the CHWS for All project achievements and that of the entire NCHA program. __________________________________________________________________________________ Copyright © CHWS for ALL Project Final Evaluation - 2018 46 ANNEXES  Annex 1: Evaluation Terms of Reference (Attached)  Annex 2: Data Collection Tools (Attached)  Annex 3: List of the Keys persons Interviewed (Attached)  Annex 4: Logical Framework for the Project (Initial Project Log frame)