W F P C ag A Final Endline ev WASH pre FHE Addis Prepared Contact Pe germekuriaw Addis Ababa, l Report valuation o paredness Be Ababa, Et d by MA erson: Ag w1@gmail.c , Ethiopia of a six mo project at nishangul thiopia College ger Mekur om for onth emerg health care Gumuz an Sub e riaw, Telep Feb 1 gency respo e facilities nd Oromia bmitted to: ephone: M ruary 12, 20 A onse “Imp (HCFs)” i Regions in Mobile: +2 021 Addis Ababa proving CO in seven w n Ethiopia 251/ 0911 a, Ethiopia OVID-19 IP oredas in A 1460767 E PC and Amhara, Email: 2 Table of Contents Acronyms List of Tables and Figures 1. Introduction or background describing the project 8 2. Description of the evaluation area 9 3. Objective and scope of the evaluation 9 3.1. Scope of the Evaluation 10 4. Methodology and Approach 10 4.1. Evaluation Design and Criteria 10 4.2. Evaluation Sample Woredas and Number of Respondents 11 4.3. Data Collection Methods and Tools 11 4.4. Data Limitations 12 5. Evaluation Findings 12 5.1 Project relevance and appropriateness 12 5.2 Project efficiency 14 5.3. Project Effectiveness 15 5.4. Changes and impact of the project 25 5.5. Project Partnership, Coordination and Coherence 29 5.6. Project sustainability 30 6. Challenges of the Project 31 7. Strength and weakness of the project 33 8. Lessons Learned/Good Practices 33 9. Conclusion and Recommendation 34 3 Acronyms/Abbreviations FHE Food for the Hungry Ethiopia HCF Health Care Facilities HCW Health Care Workers HEW Health Extension Workers IPC Infection Prevention and Control ODK Open Data Kit PPE Personal Protection Equipment SPSS Statistical Packages for Social Science SWOT Strengths, Weakness, Opportunities and Threats UNDP United Nations Development Program WASH Water, Sanitation, and Hygiene 4 List of Tables Table 1: Sample Woredas and Number of Respondents 11 Table 2: Budget Utilization as of December 2020 15 Table 3: Key Indicators Achievement-Amhara, Oromia and Benishangul G. Regions 17 Table 4: Mention the Effective Hand Washing Procedures to Prevent COVID-19 Transmission Pathways 21 List of Figures Figure 1: What to do to prevent coronavirus? 20 Figure 2: How is COVID-19 Spread? 21 Figure 3: Three of the Most Important Times When Someone Should Wash Their Hands? 22 5 Executive Summary FH Ethiopia (FHE) implemented an emergency project to improve preparedness and prevention of COVID-19 in Amhara, Benishangul Gumuz, and Oromia regions in seven selected woredas. The objective of the project was to reduce mortality and morbidity from COVID-19 at health care facilities (HCFs) and provide appropriate capacity strengthening to frontline health care worker (HCWs)on controlling sporadic cases and clusters; suppressing transmission in HCFs and communities through context-appropriate infection, prevention and control (IPC) measures; and providing appropriate care, including water, sanitation, and hygiene (WASH) services. The program ran from July 20, 2020 to January 19, 2021. In December 2020, MA College was contracted by FHE to undertake the endline evaluation of the project. The purpose of the evaluation was to assess the coherence (relevance), efficiency, effectiveness, impact and sustainability of COVID-19 IPC and WASH preparedness and response at HCFs interventions and the progress made towards achieving the planned objectives in Sasiga, Diga, Bambasi and Assosa Woredas of Oromia and Benishangul Gumuz Regions; and Nefas Mewcha, Arbe Gebeya and Wogeda town administrations in Amhara region. The data collection was carried out from December 23, 2020 to January 6, 2021. Both quantitative and qualitative data collection methods (key informant interviews, one-to-one interviews, case stories, observations) were used to collect the data from the beneficiaries. Android mobile device based platform ODK (kobol tool box) was used for gathering data of the beneficiary survey, WASH and HCFs assessment. Highly qualified data collectors and supervisors were deployed for the data collection under the guidance of the evaluation team. The quantitative and qualitative data were analyzed, triangulated and supplemented to seek the credible evaluation result. Statistical analysis using Statistical Packages for Social Science (SPSS) and advanced excel used for quantitative data analysis while content based/narration techniques used for the qualitative data. The evaluation results showed that the project achievement is outstandingly high considering the emergency circumstances and the short duration (i.e. six months) and area coverage of the project. It has covered 134 HCFs, 7 woredas in three regions (Oromia, Amhara and Benishangul-Gumuz). The project was important, relevant, appropriate and valuable in supporting HCF needs in preventing COVID-19 pandemic- the global health crisis, in line with the government priorities and in supporting the Ethiopian Government efforts to reduce the impact of COVID-19 at country level. The project intervention and implementation was timely and systematic to address a large section of the population. The project has followed value for money principals. The materials purchased and the services rendered were in the least cost combination in which the project cost was spent mainly on durable items which have a long-term use value. The project has achieved its key outcomes. Most of the project outcomes have recorded overachievement. For instance, healths related and WASH outcomes achieved 140.1% and 1,828% respectively. The project implementation was mainly on time, very innovative and swift. The project focused on institutional capacity building as well as the provision of required materials for COVID-19 prevention and control. Communication materials and dissemination strategy was effective and innovative andconsidered the realities on the ground. The project was also effective in the capacity building of HCFs, in raising the awareness level of the community members in COVID-19 prevention messages and practices. As evidence, 98.2% of the beneficiaries knew the COVID-19 prevention measures, 95.1% of the beneficiary respondents had knowledge of COVID-19 transmission vectors, and 98.3% were aware of the effective and hand washing procedures and 97% mentioned the most important times to wash hands. 6 The impact of emergency interventions can be seen in terms of short term benefits such as saving life or reduction in morbidityamong the targeted communities. The project has successfully achieved the following intended outcome level results: ● Prevented frontline health workers from being exposed to infection ● Reduced morbidity and mortality ● Increased face mask utilization among project beneficiaries ● Increased the habits of hand washing practices among target communities ● Increased health workers knowledge on management of risk mitigation ● Motivated health workers to work on management of risk mitigation ● Enabled the target beneficiaries as well as the community members to have access to the health facilities, and as result,help gainthe necessary knowledge and skill sets related to COVID-19 prevention mechanisms such as hand washing practices, social distancing, and wearing face masks. The evaluation team realized that participation in project design, collaboration/partnership with stakeholders, inclusiveness, flexibility in project management and value for money were the major reasons for the successful achievement of FHE. Though the project was successful, the overall sustainability of the project outcomesremains questionable in the project woredas due to limited resources and budget constraints at the government level. Based on the evaluation results, the following recommendations are suggested for future activities or similar projects ● One aspect of sustainability that requires close attention in the future is the supply chain, which is relatively weak at the HCF level. Although the project provided essential IPC items and PPEs during the implementation period, due to financial constraints the Woreda Health Offices or HCFs could not make such supplies readily available in a sustainable manner. This could be a major factor in limiting the sustainability of the project in the future unless a more sustainable solution is found through collaboration efforts among relevant stakeholders/donors. In addition, the project has completed necessary basic capacity works in order to help the target beneficiaries handle increased outputs. However, to make the changes sustainable, it will be important for the projectto follow-up with beneficiariesfor some time.More importantly, the project must coach the concerned HFs their regular service delivery programs.Beneficiarieswill be more likely to maintain sustainable change if adequate supervision and monitoring systems are put in place. ● Additional efforts are required to work with the partners to strengthen the sustainability of the results and the positive outcomes. There is also a need to extend the program in order to continually support the HCFs until they are able access adequate supply of various core IPC supplies as well as medical and other pharmaceutical commodities.In addition, refresher training on COVID-19 prevention should be provided in an organized manner both in a classroom and on￾site. This is especially necessary given the global emergence of new variants of COVID-19. ● Even though this project has achieved its intended objectives, because of the insufficient budget its coverage was relatively limited: the other non- intervention Kebeles or HCFs are still in need of assistance. Therefore, attempts should be made by the FHE to obtain funding from donors that will include those HCFs found in the target Woredas in future interventions. 7 ● There is also a need to design an innovative system and find ways to engage youth and women in manufacturing qualitylocal products related to WASH services in a cost-efficient way. This will help meet the local demand and ensureaccess to affordable supplies in nearby areas. This also creates a viable employment opportunity for the unemployed youth population. Any such project would require a serious business feasibility study. 8 1. Introduction or background describing the project FHE began operations in Ethiopia in 1985, and has been implementing multi-sectoral programs across the country. Currently, FHE manages food security, education, water and sanitation, health and nutrition, OVC projects, child focused community transformation and emergency relief and rehabilitation programs, among others. It operates in six regions: Amhara, SNNPR, Sidama, Oromia, Benishangul Gumuz and Tigray with over 1,000 staff. FHE strives to work with communities to improve their living conditions through sustainable and participatory development programs. In response to the COVID-19 pandemic, FHE implemented a project to increase preparedness and prevention of COVID-19 in Amhara, Benishangul Gumuz, and Oromia regions in seven selected woredas. The program ran from July 20, 2020 to January 19, 2021. The program’s focus was to reduce mortality and morbidity from COVID-19 at HCFs and provide appropriate capacity strengthening to frontline HCWs on controlling sporadic cases and clusters; suppressing transmission in HCFs and communities through context-appropriate infection, prevention and control (IPC) measures; and providing appropriate care, including water, sanitation, and hygiene (WASH) services. These interventions contribute toward a sustained continuity of essential health and social services and protect HCWs and vulnerable populations. The major expected outputs of the project were capacity enhancement, communication/messaging, infrastructure maintenance, and provision of material and infrastructure facilities. The details of the outputs by aforementioned category are below: a. Capacity Building ● Community level including government staff trained on hand washing adapted to COVID-19 prevention ● Health Extension Workers (HEWs) and HCWs trained on hand washing adapted to COVID-19 prevention ● HCWs trained on gender issues, inequality and implication on COVID-19 pandemic prevention and control b. Communication/Messaging ● Development, printing and dissemination of posters on COVID-19 IPC ● Development, recording and dissemination of COVID-19 messages via mini-media including hand washing sensitization campaign ● Dissemination of key messages on COVID-19 prevention and control using digital media in HCFs reaching about 11,187 patients and caregivers each month ● Regular risk communication and hygiene promotion sessions conducted for patients and caregivers c. Provision of Material and Infrastructure Facilities and Maintenance of Infrastructure ● Setting up hand washing stations at critical locations through the provision of detergents, liquid soap, and tissue supplies for hand washing stations within HCFs for effective hand washing ● Provision of waste disposal baskets at hand washing stations and at other critical locations within 134 HCFs (three hospitals, 21 health centers and 110 health posts) ● Provision of sanitizers/hand rubs for HCWs, inpatient, outpatients, caregivers and CHWs ● Provision of Personal Protection Equipment (PPE) to HCWs ● Installation of 10 water storage tanks to maintain adequate water supply at vulnerable HCFs ● Water supply maintenance at 24 HCFs, including repair of any broken taps, sinks or pipes, and pipe extension works to existing and new hand washing stations within the HCFs 9 ● Maintenance of existing water supply sources at HCFs Currently, the project is in the last phase; therefore, evaluation is needed to measure to what extent the project has achieved its stated objectives, identify the observed changes (planned, unintended), perform SWOT analysis and draw reliable and valid recommendations for future programming. 2. Description of the evaluation area The evaluation areas consist ofworedas targeted from three regional government states which include Amhara, Oromia and Benishangul-Gumuz regional states. Abrief description of assessment areas by regions are provided as follows: (1) Amhara Region Three woredas towns of Amhara Region (Nefas Mewcha, Arbe Gebeya and Wogeda town administrations)are included in this final evaluation. Wogeda Woreda is one of the 11 Woredas capital in South Gondar Administrative Zone of the Amhara National Regional State. It is located about 205 kilometres southeast of the regional capital city, Bahir Dar. Nefas Mewucha is the capital of Lay Gayint district. Lay Gayint is found in South Gondar administrative zone 210 km from North East of Bahir Dar. Lay Gaint has 31 kebeles. Areb Gebeya is the woreda capital of Tach Gayint woreda. Tach Gayint woreda is located in South Gondar Zone, Amhara Regional State Ethiopia. It is about 770 kms northeast of Addis Ababa and about 200kms from Bahir Dar; the regional capital. Tach Gayint has 18 kebeles. (2) Oromia Region Diga Woreda is one of the 262 districts in the Oromia regional state, located in the southwest of the Abay basin approximately 343 kms west of Addis Ababa. The Woreda is bordered in the northeast by Guto Gida district, in the west by the Dhidhessa River, in the north by Sasiga district and in the south and southeast by Jimma Arjo and Leka-Dulecha districts. Sasiga is also one of the woredas in the Oromia region of Ethiopia in East Wollega Zone. Sasiga is bordered on the south by Diga, on the West by the Benishangul-Gumuz region, on the North West by Limmu (3) Benishangul-Gumuz Region Assosa and Bambasi woredas in Benishangul-Gumuz region are covered in the final evaluation. Assosa is one of the 20 woredas in the Benishangul-Gumuz Region of Ethiopia. Part of the Asosa Zone, it is bordered by Kurmuk and Komesha in the north, by Menge in the northeast, by Oda Buldigilu in the east, by Bambasi in the southeast, by Mao-Komo special woreda in the south and by Sudan in the west. Bambasi Woreda is one of the twenty Woredas of the Benishangul-Gumuz Regional state. It is found 45 km from Asossa town which is the capital city of the region. The woreda is bordered by Oromia regional state and Maokomo special Woreda in the south and south west and, Asossa Woreda in the west and Oda Buldegelu Woreda in the north east. Administratively, the Woreda is divided into 38 kebeles. The people found in this Woreda are composed of a variety of ethnic groups. 3. Objective and scope of the evaluation The purpose of this evaluation is to assess the coherence (relevance), efficiency, effectiveness, impact and sustainability of COVID-19 IPC and WASH Preparedness and Response at HCFs interventions and the progress made towards achieving the planned objectives in Sasiga, Diga, Bambasi and Assosa Woredas of Oromia and Benishangul Gumuz Regions; and Nefas Mewcha, Arbe Gebeya and Wogeda town administrations of Amhara region. The results of this evaluation will contribute to a culture of learning and evidence-based decision-making, and will improve the quality of future programs in FHE. 10 Therefore, the specific objective of this assessment is to generate a credible, evidence based, actionable analytical report with insight on the following issues: ● Provide an independent assessment of progress achieved in the COVID-19 response project and identify key factors that negatively or positively affected the project performance; ● Draw lessons learned and best practices and identify challenges encountered during implementation of the program ● Provide insights on how outcomes were achieved with COVID-19 response project ● Provide evidence on synergy in partnership, sustainability and cross cutting issues including impacts achieved. 3.1. Scope of the Evaluation The final evaluation will cover the achievement of the project in 134 target HCFs found in seven woredas of the three operational regions (Amhara, Benishangul Gumuz, and Oromia). The woredas to be included in the evaluation are as follows, by region: (i) Amhara Region: Nifas Mewecha, Arab Gebeya and Wogeda (ii) Benishangul Gumuz Region: Assosa and Bambasi (iii) Oromia Region: Diga and Sasiga The evaluation unit was HCFs, HCWs, inpatients, outpatients and communities. The evaluation focused primarily on measuring the impacts of the project on the target population. The evaluation will also look into cross-cutting issues such as gender and inclusion of marginalized groups (e.x. disabled people, the poor and other disadvantaged populations) and the impact of COVID-19 on women and children. 4. Methodology and Approach This evaluation is a cross-sectional survey design and assesses the performance, results and impacts of the project interventions. Mixed methods of data collection were deployed in order to gain a more complete understanding of the issue, more confidence in the findings and to maintain data quality. Using both quantitative and qualitative data collection methods provided the consulting team the advantage of triangulating data and findings and ensured that non-quantitative issues, which would ordinarily be treated as outliers, were not excluded. Quantitative data relied mainly on the use of beneficiary-based questionnaires and project progress reports. Qualitative data relied on the use of in-depth key informant interviews and case studies that enabled the evaluation team to get answers on questions related to the attitudes and opinions of communities, beneficiaries and stakeholders. It also enabled the evaluation team to draw lessons that will be used to develop actionable recommendations for future program implementation. 4.1. Evaluation Design and Criteria The project is an emergency response designed for a six-month period,not as an experimental design. Thus, this evaluation employed the pre-experimental evaluation design. The pre-experimental design involves pre- and post-intervention focused on collecting information primarily from the project beneficiaries and stakeholders. The evaluation analysis was undertaken with the common project evaluation criteria, which includerelevance, appropriateness, efficiency, effectiveness/impact, coherence and coordination, sustainability, coverage and connectedness. 11 4.2. Evaluation Sample Woredas and Number of Respondents The evaluation was undertaken in three regions (Amhara, Benishangul Gumuz and Oromia) in the selected woredas. The woredas included in the evaluation were Sasiga and Diga of Oromia; Bambasi and Assosa Woredas from Benishangul Gumuz Regions; and Nefas Mewcha, Arbe Gebeya and Wogeda town administrations from Amhara region. The following table depicts the number of respondents by region and target areas Table 1: Sample Woredas and Number of Respondents Region Woreda HCF Assessment WASH Assessment Beneficiary Survey KII Observation Case stories Validation workshop Oromia Diga 3 28 109 7 6 2 1 Sasiga 3 38 124 6 3 1 Sub-Total 6 66 233 13 9 3 1 Amhara Nefas Mwecha 2 6 99 4 2 2 1 Arbe Gebya 2 4 99 4 2 1 Wogeda 2 4 99 4 2 1 Sub-Total 6 14 297 12 6 4 Beni￾Shangul Gumuz Assosa 2 31 96 3 3 1 1 Bambasi 2 23 115 3 3 1 Sub-Total 4 54 211 6 6 2 Grand Total 16 134 741 31 21 9 3 4.3. Data Collection Methods1 and Tools Primary and secondary data sources were used for gathering relevant information. Document review, in￾depth key informant interviews, case studies and observations were used as data collection methods. ODK Platforms mobile data collection (Kobotoolbox) was used for gathering data using the structured questionnaire-based surveys. The WASH Assessment Tool developed from the WHO/UNICEF “Water and Sanitation for Health Facility Improvement Tool (WASH FIT)” (2018) and modified for COVID-19 response based upon WHO/UNICEF “Water, Sanitation, Hygiene and Waste Management for the COVID-19 Virus, Technical Brief “(23 April 2020) was used. Discrete observation on targeted HCFs was undertaken to gain insight about the project output/ outcome/results and relevant pictureswere taken to supplement the evaluation report. An observation checklist was used for conducting on-site observations. Case stories were collected to show a more vivid picture of the project outputs/outcomes from each of the HCFs, and the cases were supported with pictures of the HCFs. The validation workshop was also used, not only for validating preliminary evaluation findings, but also to help the evaluation team to further explorethe thematic issues of the evaluation and to clarify information gathered.                                                             1 The data collection was conducted with COVID-19 safety measures (such as mask wearing, social distancing, the use of hand sanitizers, regular hand washing etc.).While conducting the data collections, the evaluation team members should use the necessary personal protective equipment (PPE) including facemasks and hand sanitizers. The group gatherings and interviews will be conducted through appropriate social distancing adapted for COVID-19 pandemic spread prevention protocols. 12 4.4. Data Limitations Data collection was successful, with a response rate of 100%. However, to meet the proposed sample size, it was necessary to visit more HCFs as we could not get secure interviews with the expected number of beneficiaries in some locationsduring the data collection period. We acknowledgethe project staff (both the head office and field office) and extend them our sincere appreciation for their quick and wonderful facilitation of the data collection process, which enabled the evaluation team to finish the field work on time. 5. Evaluation Findings The report is based on field data (both qualitative and quantitative) collected from different sources triangulated for the purpose of ensuring data validity. The following describes the major findings of the evaluation 5.1 Project relevance and appropriateness Relevance implies the extent to which the objectives of a project intervention are consistent with beneficiaries’ requirements, country needs, institutional priorities and partner and donor policies. It also entails an assessment of project coherence in achieving its objectives. In assessing the relevance of the project, this evaluation determines whether or not the project was timely and adequately prepared to meet the needs of increasing numbers of beneficiaries and the level of access required by those beneficiaries. The evaluation assesses how well the implementation of the project was coordinated with local structures and community stakeholders and the level of government involvement, as well as an analysis of opportunities and challenges. This evaluation also assesses whether the project included adequate strategies, approaches, institutional arrangements and financing to achieve the objectives. Overall, the evaluation team found that the project was important, relevant, appropriate and valuable in supporting HCFs as they worked to preventthe spread of the COVID-19 pandemic. In response to this pandemic, the Ethiopian Government declared a state of emergency on April 8, 2020. Many stakeholders from the government, non-government and public society in Ethiopia participate in the response against the COVID-19 Pandemic. FHE committed to its role through designing the “Improving COVID-19 IPC and WASH Preparedness Project at HCFs” project which extends from July 20, 2020-January 19, 2021. The evaluation team observed that the design process of the project was participatory and followed a problem-solving approach-seeking a best fit solution. As assessed from the documents as well as the information obtained from direct interview of key informants, the project planned different interventions based on the gap assessment findings. In this regard, FHE conducted a Rapid Facility-Gap Assessment at 134 HCFs to inform the design of this program. Consequently, the following gaps were identified: ● Inadequate knowledge and skills-gap among HCWs and HEWs in preparedness and response to COVID-19 ● Inadequate access to quality WASH services within the HCFs in the targeted areas of Benishangul Gumuz, Oromia and Amhara Regions of Ethiopia. The assessment confirmed that, out of 134 HCFs targeted, 55% of them have inadequate water supplies, 39% are without sanitation facilities, and 55% do not have any hand hygiene materials. ● The baseline assessment found that WASH and broader related health initiatives, including quality care, IPC, COVID-19 preparedness and response levels and emergency preparedness were found to be inadequate in some HCFs and completely lacking in other HCFs. 13 ● Out of 134 HCFs, 42 HCFs were found to have existing hand-washing facilities which were damaged and out of use while the rest (92 HCFs) did not have these facilities at all. Consequently, 137 handwashing facilities were identified, requiring a total of 6 hand washing facilities in 3 hospitals and 131 handwashing facilities in 21 health centers and 110 vulnerable health posts. ● 35 HCFs require maintenance or repair of water sources and infrastructure. Out of those 35 HCFs, 16 HCFs (1 hospital and 15 health centers) have malfunctioning water sources which are inadequate to effectively respond to the COVID-19 pandemic. ● 80% of the HCFs have no solid and hazardous waste temporary disposal baskets. Any solid waste that comes in touch with COVID-19 patients or caregivers should be safely disposed of as part of infection prevention and control. At the moment 243 solid waste disposal baskets needs have been identified for 134 HCFs in 7 woredas. ● Shortage of medical supplies and other pharmaceutical commodities for IPC Preparedness and response to COVID-19. ● Availability of equipment and crucial supplies such as medical oxygen, ventilators and PPE and other basic supplies (for example, drugs and detergents) have all been identified as critical gaps during the HCFs Gap-Assessment. Identifying gaps enhances the relevance, appropriateness and feasibility of the project intervention. Key informants interviewed in all project areas have appreciated the project and many consider it “live saving”. For instance, a key informant recalled the situation during the case of COVID 19 identified in Ethiopia: “The health care staff members were very reluctant to provide treatment services for patients for fear of being infected with COVID-19 and the patients in turn also claim the health care professionals are carriers and transmitters of COVID-19. Consequently, the health services provision system was disrupted. This is also fuelled up by lack of budget for purchase of medical supplies as well as the shortage of water in the hospital. With this desperate situation, the FHE project has contributed a lot to reinstate the facilities in normal operation-(KII, Nefas Mwecha).” The evaluation team found that the project intervention and implementation was timely and systematic to address a large section of the population. It systematically integrates and works vertically and horizontally to address the felt needs around COVID-19. Making the HCWs the primary target for the project was a good approach as it made it easier to communicate and disseminate the necessary information to the public at large. Moreover, the FHE’s response was aligned with government priorities, seen inthe Ministry of Health’s call for collaborative action between private sector, government and non-governmental organizations. The respondents from all project areas felt that FHE was at the forefront in preparedness for and prevention of COVID-19. At the validation workshop in East Wollega zone the participants appreciated FHE’s timely involvement and persistent support to the area:“FHE has supported the zone health office and health institutions with supplies for the prevention of COVID-19 beyond its target areas. It has rendered support to us in its capacity without reservation and I want to admire FHE’s implementation in East Wollega zone where many NGOs retreat or are not willing to come to our zone without reason. Therefore, I can say FHE is our principal development supporter (Workshop Participant, East Wollega Zone).” In addition, the respondents from all project areas affirmed that the project is in conformity with different health policies such as sanitation, community health transformation and, principally,the prevention strategy of the Ethiopian government. Some even asserted that the project has contributed significantly in 14 cases where the healthcare system struggled to continue its regular operations. For instance, a key informant from Nefas Mwecha mentioned that “the woreda health office had a serious budget deficit infacilitating the expected health care provision in all the health facilities. It was such a big problem even to pay salary for the HCWs. Because of that the HCFs were expected to collect revenue from different sources such as health insurance, which was not effective. So, it surely reached usat an appropriate time and covered the required costs when we were in need of searching for the way out (KII, Nefas Mwecha).” 5.2 Project efficiency Efficiency measures the outputs – qualitative and quantitative – achieved as a result of inputs. This generally requires comparing alternative approaches to achieving an output to see whether the most efficient approach has been used. Efficiency measures how economically inputs (usually financial, human, technical and material resources) were converted to outputs. The evaluation team has considered the following major project efficiency measurement criteria to measure the project’s efficiency: (a) Value for Money The evaluation team has observed that the project has followed the value for money principal. The materials purchased and the services rendered were the most cost-effective available, and the budget was spent mainly on durable items which have a long-term use value. Key informants from the Oromia and Amhara project target areas have said that the project brought a tremendous change in the HCFs. similarly, the results of all key informants from other project areas showed that the service provisions by HCFs have greatly improved. For instance, a key informant from Nefas Mwecha said “ourhospital has never had the standard type of bin since its establishment in 2015. But, currently thanks to FHE project, we have got the access to it.” (b) Output quality Procured items, service provisions and project supplies maintained quality standards and the project outputs met their expected targets. Most of the respondents replied that the project output was of a high quality that can serve as a role model for many other organizations (both government and non￾government) which sometimes are unable to provide quality outputs. According to the evaluation team’s observation, respondents from the visited HCFs were mostly satisfied with the quality and quantity of materials and supplies. In some areas, however, the respondents wished for additional surgical gloves. (c) Timeliness The respondents from all project target areas stated that the project implementation was mainly on time, innovative and swift. From document review and interviews, the evaluation team confirmed that all the project activities were accomplished on time, though there were some challenges. The challenges can be divided between pre-implementation and implementation.During the project pre-implementation phase, there were problems in gettingqualified suppliers of PPE and hygiene supplies in some project locations. Consequently, it was decided to procure PPEs and hygiene supplies from Addis Ababa. This procurement incurred additional costs, as supplies had to be transported from Addis Ababa to each project location. Because of this, the project was slightly delayed. In the implementation phase, security was a challenge in the Oromia region during the second week of August 2020. During this time, roads were closed and travel restricted, so trainers were unable to drive to the project site to provide training to the HCWs and HEWs. In addition, the restriction set by the government on gatherings/meetings due to COVID-19 prevention measures delayed activities slightly. 15 (d) Robust Investment and Innovativeness The evaluation team understood, from document review of project performance reports, on-site observations and key informant interviews, that most of the project budget was spent on durable items, fixed goods and items which have a long-term effect on the health care system. The project focused on institutional capacity building as well as the provision of required materials for COVID-19 prevention and control. The dissemination of communication materials was also effective and innovative. (e) Budget Utilization As indicated in the table 2, the budget utilization of the project during this evaluation period was 99.1%. The project woredas (Diga,Wogeda and Tach Gayint woredas) have used all of the allocated budget. The target woredas in Benishangul Gumuz region seem to have the lowest rate of budget utilization. Table 2: Budget Utilization as of December 20202 Region Planned (USD) Utilized (USD) Oromiya Diga 67,472.42 68,279.37 Sasiga 48177.64 53,969.58 Amhara Nefas Mewecha 78,360.26 90,651.41 Wogeda 50,714.89 43,157.50 Tach Gayint 50,691.13 51,730.58 Benishangul Gumz Assosa and Bambasi 121,042.10 104,917.18 Total 416,458.44 412,705.62 NICRA (Negotiated Indirect Cost Rate Agreement) (20.06%) 83,541.56 82,788.75 Grand Total 500,000.00 495,494.37 5.3. Project Effectiveness This section of the evaluation report describes the effectiveness of the project: i.e., the extent to which the planned results specified in the project’s log-frame are achieved. The evaluation team’s observation as well as the information gathered by the team confirmed that, overall; the project met its goals of improving the HCFs’ services and enhancing health providers’ responsiveness to and communication with patients in the project woredas. The planned activities of the project were effectively implemented. The following section further describes the extent to which each project core was effective. Project activities can be summed up in the following three categories: capacity building, messaging on COVID￾19 preparedness and prevention and provision of materials/supplies and maintenance of infrastructure. A project will succeed if its impetus comes from the real needs of the target beneficiaries. The project conducted a baseline assessment and a stakeholder workshop to discuss priority issues and shared                                                             2 This budget does not incorporate the January 2021 expenses. 16 responsibilities and roles with the relevant stakeholders. The project has given space for participation of stakeholders from design to implementation, which has positively impacted its effectiveness. a. Effectiveness of Capacity Building The project planned capacity building activities which were intended to enhance the sustainability of project results. These activities include:providingIPC training for HCWs at primary HCFs in accordance with WHO protocols and guidelines; ensuring that all HCWs adhere to IPC protocols, laboratory specimens in target HCFs are handled properly, and safe waste disposal is in place; regular monitoring to make sure IPC is maintained; and organizing training for all staff on WASH, including a specific training for cleaners based on the protocols.As shown in the following table, the results of these activities were, in most cases, overachieved. Key informant interviews and analysis of secondary data confirmed that all planned activities were delivered in all project woredas and the trainings were relevant. Key informants testified to the importance of the interventionsto the target beneficiaries as well as their strong collaboration with the HCFs, woredas and regional strategies and priorities. The project involved HCFs as well as the Woreda and Zonal stakeholders in reviewing,sensitizing, and planning in the course of the first phase. One of the most important supporting factors to project effectiveness was the engagement and contribution of stakeholders as well as the dedication and commitment of the FHE project staff. 17 The overall achievement of project outcomes is high/over-achieved.The following table depict the key indicators of achievement disaggregated by project regions: Table 3.0 Key Indicators Achievement-Amhara, Oromia and Benishangul G. Regions Indicators Target Achievement by Region Achievement Amhara Oromia Benishangul G Total Achievement % Sub￾sector Public Health Emergencies of International Concern and Pandemics Ind 1 Number of health care staff trained 63 314 371 224 909 1443% Ind 2 Percentage of target population who can recall two or more protective Measures 80% 78% 73% 68% 73% 91% Ind 2 Number of outpatient health facilities supported 3 6 8 10 24 800% Ind 3 Number of inpatient health facilities supported 3 3 0 0 3 100% Ind 4 Number of individuals screened or triaged for COVID-19 at supported health facilities 2,431.00 12,966.00 12,737.00 1,386.00 27,089 1114% Ind 5 Number of people reached through risk communication activities by channel 19029 12169 12708 10920 35,797 188% Hygiene Promotion Ind 1 Number of people receiving direct hygiene promotion (excluding mass media campaigns and without double￾counting) 16214 17582 14,737.00 2,803.00 35122 217% Ind 2 Percent of people targeted by the hygiene promotion program who know effective handwashing procedures to prevent COVID-19 transmission pathway (Custom) 85% 75% 71% 78% 75.1 88.35% Ind 3 Percent of households targeted by the hygiene promotion program with soap and water at a designated handwashing location (HCF) 100% 79% 81% 77% 79.5 79.50% Ind 4 Percent of people targeted by the hygiene promotion program who know at least three (3) of the five (5) critical times to wash hands 85% 75% 71% 78% 63.6 74.82% Sanitation 18 Ind 1 Number of people directly utilizing improved sanitation services provided with USAID/OFDA funding 19029 12169 12708 10920 35,797 188% Ind 2 Percent of hand washing stations built or rehabilitated in health facilities that are functional 100% 100% 100% 100% 100 100% Ind 3 Percent of latrines/defecation sites in the target population with handwashing facilities that are functional and in use 85% 100% 100% 100% 85 100% Water Supply Ind 1 Number of people directly utilizing improved water services provided with OFDA funding 363 1088 0 812 1,900 523% Ind 2 Percent of water points developed, repaired, or rehabilitated that are clean and protected from contamination 100% 100% 100% 100% 100 100% Ind 3 Estimated safe water supplied per beneficiary in liters/person/day 7.5 17.7 0 23.2 20.4 272% 19 The key informant interviews asserted that trainings provided by the project helped participants avoid confusion and myths about Covid-19 spread by unreliable sources. They helped the HCWs to have an accurate understanding about COVID-19. A key informant explained that before the training provided by FHE, almost all HCWs had a different level of knowledge about and varying attitudes towards COVID￾19. Some health workers adhered to information obtained only from browsing websites, and believed that any information related to COVID-19 posted on a website is accurate. Some health workers even doubted the COVID-19 related information released from WHO and the Ministry of Health of Ethiopia. After FHE’s training these inaccuracies were corrected. The training modalities as well as the skill of the trainers and facilitators have enabled the health workers to increase their knowledge of COVID-19 within a short period of time. This has really helped us to run the normal operation of the health services system (Key Informant Interview from Diga woreda, Oromia region). Similarly, a key informant in the Amhara region project target Wogeda woreda stated that the HCFs staff as well as the community members were of the belief that COVID-19 could not appear in Ethiopia without any sound justification. Therefore, I give utmost credit for FHE’s training related to COVID-19, which gave a clear understanding among the HCWsas well as the community members. Therefore, the training wasimportant to change the attitudes and practices of the health workers. Before the training, the health workers who wore face masks faced unnecessary sarcasm from their colleagues who did not practice wearing masks. Consequently, some had abandoned wearing masks. The project also contributed to improving the attitude and practices of using sanitation and hygiene to prevent COVID 19(Key Informant Interview, Wogeda town, Simada Woreda, Amhara region. Related to this, the respondents recognized the following preventive measures for COVID-19: washing hands with water and soap (93.5%), maintaining social distance (95.7%), not touching the eyes or nose with hands (77.1%), using a mask (98.2%), avoiding contact with infected people (80.6%), taking all family members into home quarantine (26.0%), and maintaining self-quarantine (42.9%) to prevent COVID-19 (26.7%). F F be (e igure 1: Wh or the mode e during cou e.g., meat, m at to do to p e of coronav ughing, touc milk) (66.2%) revent coron virus transmi ching contam ), and only 0 navirus? (N= ission, 95.1% minated sur 0.3% did not 20 =707) % of the res rfaces (87.6% t know abou spondents an %), through ut the mode o nswered dire eating infe of transmissi ect transmis ected animal ion. sions will products F A C (9 pa fi T (N A W R R igure 2: How As indicated COVID-19 tr 98.3%), wet alm over the ingers interla Table 4: Men N=707) Apply enough Wet hands wi Rubbing the h Rubbing the R w is COVID in the tabl ransmission t hands with e left dorsum aced (81.9% ntion the Eff h soap to cove ith water hands palm w Right palm ov D-19 Spread? e 6, the res pathways, s h water (98% m with inter %). fective Hand er all hand su with the other ver left dorsum ? (N=707) spondents id specifically t %), rub the rlaced finger d Washing P urfaces palm (rotary m with interla 21 dentified eff the need to a palmstogeth rs and vice Procedures to movement) aced fingers a fective hand apply enoug her (rotary m versa (86.7) o Prevent CO and vice d washing p gh soap to co movement) ( ) and palm t OVID-19 Tra procedures to over all hand (94.1%) rub to palm rub ansmission P 9 9 o prevent d surfaces b the right bing with Pathways 98.3 98.02 94.06 86.7 v P R R h R W D U U A ti pr F b T in an ha ju “b sk co versa Palm to palm Rotational rub Rotational rub hand in left pa Rinsing the ha Wash hands w Drying the ha Use alcohol-b Use clean item As indicated i mes that som reparation (7 igure 3: Thr . Inclusiven The project h nterviews, th ntiseptic solu ad made by ust ordered th barakina”. H kilful on mix overed both rubbing with bbing of left t bbing, backw alm and vice ands with wa with water and ands with clea based hand ru m to turn of th in the figure meone shoul 77.5%). ree of the Mo ness of the P has tried to he cleaners/j ution (prepa guess or esti he cleaners t However, the xing the righ h urban and h fingers inter thumb clasped ward and forwa versa ater d soap for at an item/towel ub/hand sanitiz he faucet or o e, most of the ld wash their ost Importan Project o include all janitors rece aring a soluti imate. A key to disinfect t ey didn’t hav ht type of th rural areas rlaced d in the right ard with clasp least 20 secon zer outlet e respondent r hands follo nt Times Wh l HCWs in eived releva ion mixture y informant the rooms an ve adequate he mixture to in 7 wored 22 palm and vic ped fingers of nds ts (96.8%) m owed by afte hen Someone capacity bu ant knowled of water and from Nefas nd some of t knowledge o bring accu das/township ce versa f the right mentioned be r defecation e Should Wa uilding effor dge and lear d “Barakina” Mwecha aff the utensils l on preparin urate preven ps and all t efore eating (79.55) and ash Their Ha rts. Accordi rned how to ”), which, in firmed that “ like bed pan ng the mixtu tion of infec types of 134 7 6 6 4 4 4 as the most i d before cook ands? (N=70 ing to key o properly p n previous ti “in previous ns with deter re. Now the ction. The pr 4 HCFs (HC 81.9 71.15 65.77 62.38 49.93 47.38 43.14 39.1 important king/meal 07) informant prepare an imes, they times, we rgents like ey became roject has Cs, HP).” 23 C. Effectiveness of COVID-19 Related Preparedness and Prevention Communication and Dissemination A proper communication strategy is an important element in changing the attitudes and behaviours of direct and indirect beneficiaries. In this respect, the evaluation team observed that communications were appropriately tailored to target areas’ language, culture, and understanding, using properchannels, language, clarity of messaging, place and time. For instance, in Oromia project areas, communications use Afan Oromo to raise awareness on COVID-19 prevention methods and facts. Likewise, the Amharic language is used in project areas in the Amhara region. In addition, the sign boards/placards displaying basic information about the prevention of COVID-19 are placed in visible and accessible places. All materials use easy-to-understand language. These communication tools have contributed to the rising awareness among community members on COVID-19 prevention methods. The following pictures describe the reality on the ground. As shown in the pictures below the communication materials can accommodate equally the literate and the illiterate. Those who cannot read can use the visual elements of these materials to learn about COVID-19 prevention. The beneficiaries appreciated the visibility, clarity, inclusiveness, structure, flow and content of FHE’s COVID-19 messaging. Communication materials d. Provision of Material and Infrastructure Facilities and Maintenance ofInfrastructure During the COVID-19 pandemic, many of the HCFs in Ethiopia have experienced supply chain shortages. This has created more strain on COVID-19 prevention measures particularly in resource poor health facilities. The FHE assessment also revealed that there were stock-outs and an inadequate supply of various core IPC supplies as well as medical and other pharmaceutical commodities. As the outbreak evolved, HCFs across the targeted three regions have experienced further supply chain disruptions in medical supplies and medications. FHE operates in extremely resource-constrained woredas without access to private HCFs to reinforce HCW capacity and strengthen IPC to prevent the spread of COVID-19 and protect HCWs, patients, and communities in the targeted woredas. According to key informant in in T in ha A co ce re fa N w hy de pr bi re pr pr L nterviews, FH n normal tim The evaluatio nfrastructure ardware and Additional c ommunities. enter activit educing the acilities visit Non-governm were serious ygiene mate epartment (i rovide servic ins with the elation tohe rotection ma revention of Loud speaker HE has play mes. on team und e maintenanc d infrastructu apacity has . For instanc ties. Likewi transmission ted for the mental organ ly compelle erials, shorta i.e. the deliv ces. There w e expected s ealth care aterials like f COVID 19 r for Awarene yed a great ro derstood that ce provided ure is becom s been crea ce, reservoir ise, the was n not only o evaluation h nizations. A d to stop th age of water very service) was a poor h tandard to p provision m face masks addressed th ess creation ole in reinst t the HCFs d by the FH ming functio ated in the rs like the R ste bins pro of COVID 19 have given key informa he provision r supply. Co ) and labora ospital wast prevent spre materials li s and gloves he real need 24 ating the he improved th HE project. onal to contr woreda he Rotos were u ovided to lo 9 but also o higher value ant from the n of health onsequently, atory service te disposal sy ead of infec ike antisept s, and liquid ds of the hosp Tem W alth services heir services According ribute to the ealth institu used to store ocal health of other infec e for FHE’s e Amhara re services du , due to the es were the m ystem becau ction. The ho tic solutions d detergents. pital (KII fro mperature mea ater Storage s provision s s due to the to the key prevention utions to im e the water facilities co ctious diseas s Project inv egion reporte ue to shorta shortage of most affecte use of lack of ospital had s (Alcohol, . So, the FH om Nefas Me asurement Tankers system as it material su informant i of infectiou mprove care needed for ontributed g ses. Almost volvement t ed “The hos age of sanita f water the d; almost st f appropriat many defic sanitizer), H Ethiopia p ewecha).” happened upport and interview, us disease. e to their all health greatly to all health than other pital staff ation and maternity topping to te hospital iencies in personal project on 25 A key informant from Benishangul-Gumuz region also highly appreciated the efforts and commitments of FHE’s project compared to those of other similar development organizations. He said “There are many international development organizations working with us in Assosa and Bambabsi woredas in COVID-19 prevention and control, but I have never seen any organization compete with FHE's project outcomes and impacts. It excels surprisingly. This project is really one of the most successful projects that I have been involved with (KII, Assosa).” 5.4. Changes and impact of the project It is known that the impact of emergency interventions can be seen in terms of short term benefits such as saving life, reduction in morbidity among the targeted communities. The long term benefits can be seen in terms of potential improvement in health care service provider’s knowledge and skills and as indicated in the results of the project activity in terms of outcome and final impact. What can be determined at this stage is that the project has successfully achieved the intended outcome level result. According to the beneficiaries and key informant’s including representatives of woreda government sector offices, the impact/benefits of the FHE ‘s emergency responseinterventions include: ● Prevented frontline health workers from being exposed from infection ● Reduced morbidity and mortality ● Increased face mask utilization among project beneficiaries ● Increased the habits of hand washing practices among target communities ● Increased health workers knowledge on management of risk mitigation ● Motivated health workers to work on management of risk mitigation ● Enabled the target beneficiaries as well as the community members to have access to the health facilities, and as result, help gain the necessary knowledge and skill sets related to COVID-19 prevention COVID-19 prevention mechanisms such as hand washing practices, social distancing, and wearing face masks The following excerpts from interview notes, observation and structured questionnaires reflect to what extent the project has brought impacted changes: a. Hand Washing Practices One of the key positive long-term outcomes of this project was the practice of proper hand washing in response to COVID-19 prevention. According to the key informant interview in almost all visited HCFs, the community members were not following the proper hand washing practices even though the HEWs had communicated its importance amongst the community. Currently, as a result of the effective awareness and health education messaging, the community members followed the hand washing procedures recommended for COVID-19 prevention. A key informant interviewee from Oromia region reported that let alone the community members, the HCWs did not follow even the proper hand washing procedures at critical times than the hand washing procedures for COVID-19 prevention. It is only after the project intervention of FHE, that proper hand washing is practiced among the HCWs. The patients coming to HCFs also observed the proper hand washing practices. Some community members were also observed exercising this practice in their homes. 26 b. Health Care Staff Attitudes Changed According to the key informant interview, in many HCFs visited for the evaluation, during the first period of COVID-19 pandemic, many HCWs were unable to continue their services in HCFs out of fear of getting infected by COVID-19. Similarly, patients were hesitant to come to HCFs to get the needed health services thinking that they were easily vulnerable to the infection of COVID-19. Before FHE intervention, these attitudes continued for some time because there weren't necessary supplies/materials for the prevention of COVID-19 such as masks, hand sanitizers, gloves, soaps and detergents. This situation has negatively impacted the maternal and child health care services. This service was staggering. According to the key informants it was after the intervention of the FHE’s project that the normal health system provision services continued due to the behavioural change created among HCWs by the project. The following case supports this statement: “Health Care Provisions System backs to Normal” Didessa health post found in Diga Woreda.The health post provides mainly services on maternal and child care, ANC/PNC services, immunization and treatment services. In the area, malaria, Diarrhea, Typhoid and other communicable diseases are the most common diseases. AMREF, FHE and Government work in the area on health related issues. During the first COVID-19 cases diagnosed in Ethiopia, HCWs in Didessa health post were frustrated and unable to continue the normal services due to the fear of COVID-19 infection. They only provided care for 5 to 15 patients per day; this was 80% lower than from normal times. The service provision was also not satisfactory. The immunization services, ANC/PNC, family planning, vitamin supplements, and overall maternal and child health care services were seriously affected due to the fear of infection of COVID-19. Almost 80% of the clients had discontinued coming to health post. However, after the intervention of FHE’s project, all types of HCFs services continued due to the project capacity building, necessary material support, and on-site monitoring and follow-up. The training provided by the FHE’s project COVID-19 was very articulate, professional, and understandable which led to HCWs having a common understanding and avoiding unfounded myths on COVID-19. This has boosted HCWs psychology and knowledge. Almost all HCWs 27 stopped their fears and came to understand it is possible to control the infection of COVID-19 if prevention measures are followed properly. In addition, the project has provided the necessary materials and supplies (i.e. face mask, gloves, hand sanitizers, alcohol etc) for prevention of COVID-19. The qualities of the PPEs provided to HCWs are very good. After this the HCWs gained courage not only in providing services tothe health post, we also tried to reach the community members at home and taught them about COVID-19 prevention measures. Awareness raising messages were put on the banners, leaflets, and posts with local language including pictorial presentation to sensitize the community members. As a result, we witnessed the community members follow and strictly respect the COVID-19 prevention measures like wearing masks/handkerchiefs/cloth and keeping social distance in kebele meetings. In addition due to the proper hand washing practices at the community level, the diarrhoeal cases reduced by about 30% (rough estimation). These are all positive impacts are results of the collaboration efforts of FHE project intervention and the woreda health office and other woreda offices included in the woreda COVID-19 task force. And the informant wishes this project continues for the next five years for the sustainability of the project outcome considering the unfinished COVID-19 health pandemic. It is also important to extend the project beneficiaries both at institutional and community level at a wider scale than the current area coverage of the project. The lessons/good practices of the project will have a great impact if replicated to other health issues like maternal care. The last thing the informant suggested was that the project should consider delivering long-lasting particularly in the lowland areas. “Never Dreamt of Access to Pipe Water in Short Period of Time “ Anbessa Maseria health post is located in Simada woreda. It provides services for 7,245 people (i.e. 1,684 households, 981 children under 5 years of age, 1,780 women at the reproductive age groups). 2 HEWs are assigned to the health post. Before the project intervention, the health post had no water supply; therefore, it was very difficult to provide quality services, to keep cleanliness of the health post and personal hygiene of the staff. We used to buy water of 30 liters (i.e. one Jerry Can/container and even had to travel a long distance to purchase water. Besides, due to the shortage of water we were unable to practice good hand washing practices. We have never thought the problem of water would be solved in a short period of time. However, we now have an installed water pipe. This was realized with the support of the relevant sector offices of the government as well as FHE. However, FHE’s project support was unreserved and deserves the highest recognition for the success of health post’s access to water. We are now able to teach our clients about proper hand washing both theoretically and practically. Every client that comes to visit our health post must wash before getting into the room. In the future, we plan to cultivate the land in the compound with different vegetables and fruits with our water. By doing so, we will be able to teach the community about nutrition and healthy diets. c. Health and Economic Related Unintended Impacts The distribution of water storage/containers by the FHE’s project has brought significant changes on the provision of health services in HCFs as well as reduced the medical costs/expenses of the clients. According to a key informant interview, supply of water in the hospital is adequate and persistent following FHE’s interventions. The containers /water storage tanks received from the project have enabled the HCFs to store water in sufficient quantity for the services they provided. For instance, “the laboratory department provides laboratory services efficiently as this is because the daily performance of the body organs function testing machine works effectively. This situation has indirectly reduced costs of the client since the patients are not exposed for additional expenses including travel and stay out of home. T In d A in th pr th in di be du fo N Th U E se w at fo go w 19 ap is H po This has an nterview, Nef . Disease Bu According to n the respect he HEWs in ractices. On he health pa ntervention t iseases in th eneficiaries ue to the s ollowing cas Nifas Mewcha There are 103 University tak Environmental ervices were wastes, includi t the sides of or the commu overnment off was no budget 9, FHE plan ppropriate w ssue of the san Health Office, oor waste disp economic i efas Mwecha urden Reduc the key info tive project w n the project e HEW note ackages; bu that dramatic he areas ha have shown upport of F se witnesses primary hosp HCWs and 7 ke clinical t l Officer and the most vuln ing dangerou f the corridor unity that ca ffice to purcha t for such item nned and des ay to solve th nitation prob the Woreda posal was ide impact on p a).” ed formant inter woredas redu gave FHE c ed that “we p ut we have c changes we s decreased n an improve FHE’s proje this outcom pital was esta 72 supporting training (app other staff m nerable to C us ones, were walls. This w me to get se ase dust bins f ms (there was signed its pr he hospital’s lem with the WASH Offic entified and d They ha wastes. W material one for wheels contamin hospital The hosp this prob Compare that save patients bec rview of the uced signific credit for th provide awa never seen ere observed . The health ement. More ect avoided me: ablished in 2 staff. The hos prenticeship) members witn OVID-19 infe collected by was dangerou ervice. In pre for the hospi s a serious de eparedness a major sanita FHE staff an ce, the munici dust bins that ve black, red We use the bl ls like expired blood-stained and cleaner nation. By d staff commun pital staff is v blem was ma ed with other es the lives of 28 cause they s visited HCF cantly due to he behaviour areness to th n behavioura d.” Consequ h situations over, the ava risk of infe 007 EC. Hav spital is also based on nessed, the sta fection. There using open b us and a poss evious times ital because th eficit of budg and response ation problem nd started to w ipality and o have 3 differe d and yellow lack one for l d medicines a d and wound rs are well doing so, the nity from infe very grateful andatory and r NGOs or G f many. save their t Fs, the preva o the proper r change of he communit al changes. ently, the rep of the com ailability of ection transm ving 28 beds, the place whe their yearly aff and the p e was no prop buckets that s sible source o it was hard hey were not get in the wor e of COVID￾m. Accordingl work with diff thers. After t ent colors we colors that less risky was nd chemical c d care wastes protected s e FHE projec ection of COV l for FHE staf d needed spec O projects, F time and mo alence of co hand washin the commun ty on hand w It is only ported numb mmunity mem waste dispo mission am it serves mor ere medical s y schedule. patients that c per waste dis stood in front of infection fo for the Hosp available in reda). After th 19 project a ly, we as stak fferent sector thorough disc ere provided t help us to co stes, the red contaminated s. Besides, th o as to no ct prevented VID-19 and ot aff and the in cial attention FHE performe oney (Key I ommunicable ng practices nity on hand washing as it after FHE’ ber of cases mbers and t sal materials mong the cli re than 235,0 students of De This hospita came to seek sposal system of each class or the hospita pital and for the local mar he epidemic o and implemen keholders dis offices like th cussion, the p to us. ollect differen one for very d wastes, and e bins are m t be expose many peopl ther infectiou nstitution itself . They helped ed a very nob Informant e diseases . Many of d washing t is one of ’s project of the top the direct s in HCFs ents. The 000 people. ebre Tabor al, as the the health m. Hospital sroom and al staff and r the local rket. There of COVID￾nted in an scussed the he Woreda problem of nt hospital dangerous the yellow mobile with ed to any le and the s diseases. lf, because d us a lot. ble activity D be T w pr to de O th A ta ac ho W su m ef 5. In th T in re pr re ar ev ke A pa ac w Due to the su e saved. The “No more lo The water sup was so seriou roject helped o collect be etergents for One hospital he organ test After this FH anker that c ctivities of ospital for p We, as the ho upport. We machine is fir ffective proj .5. Project P n any develo he project su This included ntervention w elevant offic roject/progra elevant gove rrangement valuation tea eeping coher According to articipated i ctivities. Th woreda WAS upport of FH e following e oss will happ pply for Nef us in the hosp d us apply p etter water ( r proper han department ting equipme HE project im ollects 10,0 the departm roper hand w ospital staff consider tha rmly depend ects like FH Partnership opment proje uccessful. Th d an assessm with the exis ces and oth ams outcom ernment stak (role and r am has fully rence of the key informa in the need he project w SH sector on HE’s project, excerpt conf pen due to w fas Mewchia pital that wit proper hand (in addition nd washing a experiencing ent can funct mplementati 00 liters. Th ment. This a washing to p community, at FHE bou dent on the a H Ethiopia. p, Coordinat ect partnersh he project w ment of target sting progra her relevant mes and impa keholders w responsibilit y appreciated project with ant interview ds assessme woredas healt n the impro the confiden firms this situ water shortag a woreda is l thout the act washing and to mainten and improvin g difficulties tion properly on, the hosp his is suffic also improve prevent COV want to ack ught us the o availability o tion and Co hip, coordin as participat ting, rationa am, in particu stakeholder acts. Besides was assessed ties, workflo d the project h similar dev ws and docum nt, project th offices w ovement of 29 nce develop uation: ge” limited to on tions of FHE d keep sanit nance work ng awarenes s due to the s y. pital got add cient for the ed the amou VID 19 and s knowledge a organ functi of water. So oherence nation and co tory starting ale/justificati ular the exte rs. This und s, the extent d in terms o ow, etc) an t’s coordina velopment in ment review design and were involve the HCFs w ed among th nce every 2 o E, there woul ation and hy of taps and ss on the tech shortage of w ded value be e distiller pa unt of wate similar infect and thank FH ion testing m o, we sugges oherence we from the pr on for the re ent to which doubtedly co of project’s f task netw nd partnersh ation, team w nitiatives and w, different re during the d in the ove water and sa he hospital s or 3 weeks. T ld have been ygiene by im d pipe lines hniques of p water was th ecause the la art of the m er collected tious disease HE for its he machine, be st that other ere an impor rogram desig esponse, com h the program ontributed t coordinatio orking, influ hip. In this working, par d interventio elevant gove e implement erall project anitation ser taff many pe The shortage n severe dam mproving our s, accessing proper hand he laboratory aboratory go machine and and remain es. eartiest and w ecause the li NGOs must rtant elemen gn to implem mplementarit ms were link to meet the on and coher uence of ins respect, th rtnership bui ons. ernment sect tation of th t implement rvices and t eople will e of water mage. This r capacity different washing). y, but now ot its own for other ned in the wonderful ife of this t work on nt to bring mentation. ties of the ked to the intended rence with stitutional he project ilding and tor offices he project tation, the the health 30 development agents and the HEWs on teaching the community and identifying the suspected cases from the community. In some areas, particularly in Amhara region project woredas, influential people like religious leaders were invited to teach the community members about the prevention of COVID-19. Therefore, the project implementation was participatory, well-coordinated, inclusive and innovative. The FHE’s project participatory and collaborative approaches have received merits from different project beneficiaries and stakeholders. Some reported as follows: “We work with FHE project in good collaboration. We usually consider FHE project employees as our staff members. They are open and friendly in supporting our health transformation efforts. The COVID￾19 project has also strengthened our partnership. We understood from FHE’s action about the value of NGOs in averting any emergencieslike COVID-19 that needs prompt support (KII, Nefas Mwecha).” Similarly, one key informant from the Oromia region said “The FHEs staffs works for us more than expected; therefore, he preferred to consider them as staff member.” He further stated that about 98% of our tasks related to the COVID-prevention were undertaken with the support of FHE’s project. Therefore, he reported that most of the community members inhabited in surrounding HCs awareness on COVID-19 prevention messages have increased significantly. Moreover, the key informant from Sasiga Woreda, East Wollega zone, Oromia region appreciates the timeliness of the project accomplishment. He affirmed “I can witness FHE’s project is keeping its own words. All are accomplished as planned in a timely manner.” Moreover, the establishment of the COVID-19 taskforce for prevention of the pandemic has contributed to the project implementation with good progress. For instance, according to the key informant of Diga woreda, the woreda task force consists of stakeholders (FHE, Save the Children, AMREF and other woreda sector offices). This helps to identify gaps, resource mobilization and start implementation on time. Though different stakeholders were involved in the woreda task force for COVID-19 prevention, the role and contribution of FHE’s project was rated impressively. The key informant estimated that 97% of the COVID-19 prevention support was covered by the FHE’s project. The project is very in alignment with other activities conducted by the woreda government offices such as WASH, Nutrition and disease prevention. In this regard, the evaluation team considers the FHE’s project has great complementarities to other development projects. 5.6. Project sustainability In this respect the evaluation team tried to assess whether the positive effects will continue after external support has been withdrawn. This will include, but is not limited to, analysis of the exit strategy and the extent to which these appear to be sustainable. The evaluation team realized that the Overall sustainability of the project remains questionable in the project woredas due to limited resources and budget constraints at the government level. However, the following outcomes will be sustainable: a. Knowledge Sustainability Due to the capacity building training provided by the project, the beneficiaries have acquired relevant knowledge and skill with regard to the prevention and control mechanisms of COVID-19. These will help the staff to continue to prevent infections like COVID -19. The awareness of the staff is also reflected in the community members and people are aware of the infection prevention especially frequent hand washing though most of the community members do not seem to use masks out of the health facilities. The evaluation team observed that the clients coming to health facilities were given masks at the gate of 31 health facilities. In some woredas, for instance, in Sasiga Woreda,everyone who enters into the health facility was measured his/her temperature. Therefore, it is an indication that the knowledge/skill acquired related to COVID-19 might be sustainable. b. Material Sustainability It is well known that FHE’s project delivered materials; supplies and infrastructure are relevant to the WASH and COVID-19 prevention measures. These include hand washing facilities, water storage tanks, water supply and other consumable items such as detergents, hand sanitizer, soaps and etc. However the nature of these items differs particularly on their durability. Some items can be used for 2-3 years, while some will have a short duration of use. This implies there is a need to allocate budget regularly to replenish when they are worn out or depreciated. However, there may not be a means to access the sanitizer, alcohol and liquid soap in a sustainable way because of the shortage of the government budget for such supplies. Therefore, some key informants from the woreda sector offices wished for the project to continue for at least for five years. The key informant from Diga woreda revealed “It is known that the government has limited and stretched resources or budget to accommodate the need arises particularly at community level with regard to the COVID- 19 prevention and control. Therefore, we need FHE’s support for the minimum of five years considering the issue of COVID-19 is not yet resolved at global level.” c. Sense of Ownership The beneficiaries and the community are aware and took ownership on COVID-19 prevention. The health staff are interested and feel a sense of ownership and care for safety. There is also a committee/task force formed from different government sectors, and other partner organizations/stakeholders. This task force/ committee is responsible to follow the COVID 19 prevention activities, identify gaps and mobilize resources for the prevention and control of COVID-19. 6. Challenges of the Project Although the project was successful, it was challenged during implementation and pre-implementation of the project with mainly external factors. The most critical challenges during the implementation period particularly in the lowland project areas in Beni-shangul Gumuz and Oromia regions were the shortage of water both in quantity and quality in the HCFs. In fact, the evaluation team understood that the FHE’s project was not responsible and beyond the scope of the project to supply water particularly in the Oromia region. According to the key informant interview, budget shortage for WASH activity was mentioned in Amhara and Benishangul-Gumuz regions. The short project period/project duration also found as a main challenge to address such a global issue. The evaluation team also believed that though this is an emergency response project, the six months project period is very short. Moreover, due to the nature of the emergency project, actual budgeting for each planned activity was somewhat difficult for some HCFs, and that made large amounts of money for those that may not need such an amount, and hence the activities were done by shifting budget. The expense for some of the activities was also different due to inflation. There was also difficulty in putting proper specification of items to be procured for COVID-19 prevention messages. The following table gives a summary of SWOT in project implementation. 32 SWOT- Project Implementation Performance Analysis Strengths Weaknesses Opportunities Threats Implementing the project on emergency response to improving COVID-19 IPC and WASH Preparedness and response at HCFs Short duration of the project Build on good image and reputation of FHE - Services have been made available timely by the project Regional Health Bureau and Woreda Health Offices’ commitment willingness to support Effective service delivery Strong support of woreda government/ sector offices, dedication and commitment of FHE project staff HCWs are motivated Strong NGOs support such as FHE Increased knowledge of health workers on management of risk mitigation Strong support of BHA Increased community knowledge about signs and symptoms of COVID-19 Joint regular monitoring with government offices . Dedication and commitment of the FHE project staff Availability and utilization of line health facilities Good coordination and implementation capacity Securing operational Funding Community participation Community support and participation Accountability and Transparency Bringing awareness on importance of management of risk mitigation Positive FHE image among project beneficiaries Good relations with Ethiopian government Good BHA support 33 7. Strength and weakness of the project The evaluation team highly appreciates the project’s approach, selection of target project areas, the project key activities, inclusiveness of the project and its implementation modality. The selected target areas are in high need of support. Besides, FHE’s responsiveness to the call for the government to tackle the COVID-19 pandemic was immediate and in an organized manner. However, the project period was short to ensure the project sustainability. The coverage of the project was very limited though there is high demand from the beneficiaries. The community members in the project target woredas were considered as beneficiaries. 8. Lessons Learned/Good Practices In this project there are important lessons/good practices identified that can be useful for scale up as well as for starting similar projects. The major lessons are described as follows: (a) Value of collaboration, participation and sense of ownership Almost all key informant interviews asserted that if commitment, collaboration, participation and sense of ownership among project partners/stakeholders and beneficiaries developed then the project achievement was successful. They believe that this type of innovative approach makes the FHE’s project to bring astounding changes within a short period of time. One key informant noted that “utilizing materials and time efficiently bring relevant change for the community problems. The FHE staff were committed on this project and the partners too were committed to achieve good result, which taught us how harmony brings change” (b) Efficient utilization of budget In this project, as observed and reviewed from the document, most of the project budget was spent on the purchase of relevant materials and supplies on the COVID-19 infection, prevention and control measures. These procured materials and supplies were utilized efficiently for the intended purpose after serious awareness campaigns to beneficiaries on how to use and the usefulness of it on the prevention of COVID￾19. Therefore, this brought significant impact on behavioural and attitudinal changes of the beneficiaries and the community members. Therefore, in the case of COVID-19 focusing on capital investment on WASH related materials are mandatory to see changes among the society. A key informant added “The NGOs which spent high cost on workshop and visibility on public medias should learn from FHE’s project to bring tangible changes to the community at large.” The evaluation team also appreciates the budget allocation and utilization as it is done in a sensible and change-oriented manner. (c) Parallel Actions for tangible changes In most of the project activities, implementing complementary actions simultaneously has contributed to bringing robust changes and meeting the objectives of the project. The gap between training or capacity building and its real action usually delays the project outcomes and impacts. However, in this project, the capacity building and operations were almost conducted in parallel like on-site or on-job training. This means that every trainee could get an opportunity to exercise the acquired skill/knowledge instantly. This means it enhances the transformation of acquiring skill or awareness to knowledge. (d) Messages/communication modality In communication and messaging, selection of communication strategy, language and beneficiaries profile are an important element to transmit properly the required information to the target beneficiaries. With 34 respect to this, FHE’s communication modality, visibility of the communication instruments and timing of information transformation/messaging was excellent. Both the literate, illiterate and disabled people, with the exception of double disabled (deaf and blind), could access the information relevant to COVID-19 as the messages were transferred visually, audio and written forms. 9. Conclusion and Recommendation a. Conclusion The results of the evaluation confirmed that the project was found to be relevant, appropriate and aligned with the health sector development plan and priorities. The FHE project was also cost efficient in economically converting resources to intended results. Overall, the FHE improving COVID-19 IPC and WASH Preparedness and Response in HCFs project in target woredas can be regarded as a positive intervention response to an emergency situation in which frontline health workers in these areas were at risk of being exposed and vulnerable to infection as a result of IPC items and PPE shortages, due to resource constraints. After the project intervention, HCWs have been prevented from falling into coronavirus. In broad terms, the interventions were carried out successfully, meeting the main objectives as outlined in the project plan. The achievements of the project are also observed in the increased HCW’s’ knowledge and skills on COVID-19 and on the management of risk mitigation; increased the awareness and behavioral change of the beneficiaries in the COVID-19 prevention and in the creation of awareness on health and hygiene in communities. Additional achievements were also recorded by the project such as provision of IPC items and PPEs which have enabled the trained HCWs to directly apply the acquired skill into practice. It was also learned from the key informant interviews from HCFs and as well as from the observation, after the end/exit of the project, the logistic supply will be insufficient and erratic in the project woreda as the result of this, the problem of the supply chain persists; and the HCFs are still in need of assistance. Hence, the finding of the evaluation clearly indicated that extension of the project is needed in the study woredas. b. Recommendations Based on the evaluation results, the evaluation team observation, stakeholders’ workshop participants and implementers’opinion and the increasing of COVID-19 reported cases in Ethiopia, the following recommendations suggested for future activities or the similar projects ● One aspect of sustainability that requires close attention in the future is the supply chain, which is relatively weak at the HCF level. Although the project provided essential IPC items and PPEs during the implementation period, due to financial constraints the Woreda Health Offices or HCFscould not make such supplies readily available in a sustainable manner. This could be a major factor in limiting the sustainability of the project in the future unless a more sustainable solution is found through the collaboration efforts among relevant stakeholders/donors. In addition, the project has done necessary basic capacity works in order to help the target beneficiaries to be able to handle the output that it had availed them. Although that is the case, in order to further underpin the required sustainability, it will be important for the project to follow￾up with the beneficiaries, and more importantly, encourage the concerned HCFs to coach and assist though their regular service delivery program, the beneficiaries inareas thatsustainability will be ensured morelikely if adequate supervision and monitoring system put in place. 35 ● Additional efforts are required to work with the partners to strengthen the sustainability of the results and the positive outcomes. There is also a need to extend the program in order to continually support the HCFs until they become capable enough to have access to adequate supply of various core IPC supplies as well as medical and other pharmaceutical commodities. In addition, refresher training on COVID-19 prevention should be provided in an organized manner both at class room and on-site basis. Since the new COVID-variants come to appear in this time at global level ● Even though this project has successfully achieved its intended objectives in the emergency context and from pertaining to its scope perspective, because of the insufficient availability of the budget its coverage in terms of areas was relatively limited and other HCFs are in need of assistance. Therefore, attempts should be made by the FHE to obtain funding from donors so as to include these other HCFs. ● There is also a need to design an innovative system and find ways to engage youth and women in manufacturing of local products related to WASH services with quality and in a cost-efficient way. This will have a benefit to meet the local demand in affordable costs and access to supplies in nearby areas. This also creates a viable employment opportunity for the unemployed youth population and access to materials with least cost. However, this needs a serious business feasibility study to know how it is feasible in terms of access to raw materials, skill, working capital/finance, working premises and etc.