USAID/GLOBAL HEALTH EVALUATION AND LEARNING SUPPORT ACTIVITY (GH EvaLS) Evaluation Report 013 PERFORMANCE EVALUATION OF USAID/PMI GUINEA STOPPALU+ December 16, 2022 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by ME&A, Inc.; its subcontractor Dexis Consulting Group; and the evaluation team, composed of Joseph Inungu, Alpha Mahmoud Barry, Sidibe Sidikiba, and Roger Emmanuel Millimono. USAID Contract No. GS-10F-154BA/7200AA20M00003 GH EvaLS Assignment No. 013 Period of Performance: October 2021 through June 2022 Submitted by: Andrea Camoens, Chief of Party GH EvaLS 4350 East-West Highway, Suite 210 Bethesda, MD 20814-4410 Tel.: (301) 652-4334 Email: acamoens@engl.com DISCLAIMER The views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. This document is available online. Online documents can be located on the GH EvaLS website at https://ghevals.meandahq.com. Documents are also made available through the Development Experience Clearinghouse (http://dec.usaid.gov). Additional information can be obtained from: Global Health EvaLS 4350 East-West Highway, Suite 210 Bethesda, MD 20814-4410 Tel.: (301) 652-4334 Website: https://ghevals.meandahq.com GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 1 ABSTRACT Program Purpose/Description: In December 2017, USAID/PMI Guinea launched the five-year StopPalu+ activity to assist the Government of Guinea in achieving the collective goal of reducing malaria-related morbidity and mortality by 75 percent compared to 2016. RTI International implemented StopPalu+ in partnership with 10 international and local organizations and works with the National Malaria Control Program (NMCP) at the national level in all five communes of Conakry and 13 prefectures in the western and northern parts of Guinea. GH EvaLS conducted this evaluation from December 2017 to October 2022 to identify the approaches and activity components that were most effective in achieving the U.S. President’s Malaria Initiative (PMI) goals. Key Questions: The evaluation questions include: To what extent has StopPalu+ contributed to improvements in malaria care services across the different levels of the health system in Guinea; what are the sustainability factors that are evident; to what extent has the capacity of the NMCP to lead, coordinate, and manage malaria prevention and control activities been strengthened as a result of StopPalu+ efforts; what kind (if any) of positive and negative knock-on effects did the StopPalu+ supported interventions have on the wider health systems in Guinea; and to what extent did the StopPalu+ activity address gender issues in access and utilization of malaria services. Methodology: Using a mixed-method design, an evaluation team consisting of one international and two Guinean (malaria and monitoring, evaluation, and learning) experts conducted 64 key informant interviews and 14 focus group discussions with selected communities. For analysis and triangulation, document reviews and DHIS2 produced relevant national data. Challenges and Mitigation Strategies: Secondary data may not always contain sufficient elements and results to answer evaluation questions. Alternative data sources, when available, are useful to mitigate these effects. Also, gender data requires a more systematic approach to evaluation data collection, especially where disaggregation is needed. Consistency in methods and elements (e.g., male, female) of collection posed challenges to conclusions. Key Findings: StopPalu+ contributed to improvements in the availability, quality, and use of malaria care services at the community, facility, and district levels while strengthening the NMCP at the center. StopPalu+ objectives aligned with national malaria strategies and activities were implemented as planned, timely, and with quality. StopPalu+ provided technical support to the NMCP in leading, coordinating, and managing malaria prevention and treatment activities through capacity building and health system strengthening. Various communication channels helped raise awareness of the target populations about malaria control strategies, increasing community trust/respect and health services uptake. Factors such as COVID-19 and the seeming inability of the NMCP leadership to effectively coordinate and harmonize activities between PMI and the Global Fund to maximize resource allocation sometimes affected the implementation of some activities. Key Recommendations: The sustainability of StopPalu+-supported activities requires national leadership, community ownership, public-private partnerships, and continued institutional capacity building. GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 2 ACKNOWLEDGMENTS The performance evaluation team of the USAID/PMI-funded RTI/StopPalu+ activity in Guinea wishes to acknowledge the relentless support received for this assignment from the following: ● The Ministry of Health National Malaria Control Program representatives who showed support for the StopPalu+ activity and provided very useful insights from the government’s perspective ● The United States Agency for International Development/PMI Guinea team members who provided insight and guidance from the donor’s perspective ● The StopPalu+ consortium in Guinea and the team in the field who gave valuable time to answer questions and provided support through program documents The team wishes to thank all the staff members of local health facilities that were visited and local resource persons and groups we contacted. They graciously gave time to respond to interview questions, share experiences and concerns, and make suggestions for future improvements. The team is also grateful to the GH EvaLS team for their assistance and direct support during the preparation and field implementation of this evaluation and its final editing. Finally, the team would like to express its deepest appreciation to all those who generously reviewed this report's findings, conclusions, and recommendations and provided feedback. GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 3 TABLE OF CONTENTS 1. EVALUATION PURPOSE AND EVALUATION QUESTIONS..................................... 14 1.1. Assignment Purpose ......................................................................................................................................14 1.2. Assignment Questions...................................................................................................................................14 1.3. Assignment Audience ....................................................................................................................................15 2. BACKGROUND................................................................................................................... 16 2.1. Project Overview ...........................................................................................................................................16 3. ASSIGNMENT METHODS AND LIMITATIONS............................................................ 19 3.1. Assignment Methodology .............................................................................................................................19 3.2. Data Sources....................................................................................................................................................19 3.3. Sample Selection .............................................................................................................................................21 3.4. Data Management and Analysis...................................................................................................................21 3.5. Ethical Considerations...................................................................................................................................22 3.6. Limitations........................................................................................................................................................23 4. FINDINGS ............................................................................................................................ 25 4.1. To what extent has StopPalu+ contributed to improvements in malaria care services across the different levels of the health system in Guinea?..............................................................................................25 4.2. What are the sustainability factors that are evident? ............................................................................35 4.3. To what extent has the capacity of the NMCP to lead, coordinate, and manage malaria prevention and control activities been strengthened as a result of StopPalu+ efforts?........................41 4.4. What kind (if any) of positive and negative knock-on effects have the StopPalu+-supported interventions had on the wider health systems in Guinea?..........................................................................47 4.5. To what extent has the StopPalu+ project addressed gender issues in access and utilization of malaria services? .....................................................................................................................................................53 5. CONCLUSIONS .................................................................................................................. 60 6. RECOMMENDATIONS ...................................................................................................... 62 Consent Statement ..............................................................................................................................................106 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 4 FIGURES Figure 1: Map of Guinea Showing StopPalu+ Activity Districts ............................................. 17 Figure 2: Availability of LLINs.................................................................................................... 28 Figure 3: Number of Routinely Distributed LLINs from 2018 to 2021 (ANC, Extended Programme on Immunization [EPI])........................................................................................ 28 Figure 4: Number of Persons Trained in LLIN Distribution and Promotion........................ 29 Figure 5: Proportion of CHWs Capable of Using RDT at the Household Level................... 30 Figure 6: Proportion of Children Under Age 5 Who Slept Under LLIN the Night Before . 30 Figure 7: Number of Children Under Age 5 Who Tested Positive for Malaria and Receive Treatment From a CHW........................................................................................................... 31 Figure 8: Number of Children Under Age 5 Who Tested Positive for Malaria and Were Referred by a CHW for Facility Treatment ............................................................................. 32 Figure 9: Number of Quarterly Coordination Meetings Held Under NMCP’s Leadership . 38 Figure 10: Change in Parasite Detection Among Lab Technicians in Conakry and Kindia . 39 Figure 11: Number of Community Dialogs Organized by CAGs ........................................... 40 Figure 12: Number of Community Dialogs Organized by CAGs by Prefecture................... 40 Figure 13: Percentage of Households That Know the RECO and Percentage of Households that Received a RECO Visit During the Previous Month (FY 2021) ...................................... 42 Figure 14: Number of Mosquito Generations by Year ............................................................ 44 Figure 15: Total Number of Health Workers and Laboratory Technicians Trained in Malaria Laboratory Diagnostics................................................................................................. 45 Figure 16: Improvement in Parasite Identification Among Lab Technicians in Conakry and Kindia............................................................................................................................................ 46 Figure 17: Proportion of Diagnostic Tests Interpreted Correctly (Positive-Negative) ....... 46 Figure 18: Number of People Reached by NGOs/Community-Based Organizations/CHWs with Malaria Prevention and Control SBCC by Sex................................................................ 48 Figure 19: Number of Community Dialogs Organized by the Community About Malaria Control ......................................................................................................................................... 48 Figure 20: Proportion of Malaria Channels Reported by Household Members in Selected Regions ......................................................................................................................................... 50 Figure 21: Number of CHWs Trained in Malaria Case Management.................................... 51 Figure 22: Malaria Case Management Performance in Hospitals/CMCs in PMI Regions in Q2 FY2022 ................................................................................................................................... 52 Figure 23: Proportion of Pregnant Women Who Received at Least Three Doses of SP During Their Last Pregnancy (Routine).................................................................................... 55 Figure 24: Number of Health Workers Trained in IPTp from 2018 to 2021 ........................ 55 Figure 25: Number of SMC Doses (SP+AQ) Administered to Eligible Children.................. 57 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 5 Figure 26: Number of Trained SMC Distributing Agents and Social Mobilizers .................. 58 Figure 27: Percentage of Children Under Age 5 Who Received SMC During the Different Cycles ........................................................................................................................................... 59 TABLES Table 1: Evaluation Questions and Sub-questions ................................................................... 14 Table 2: StopPalu+ Activity Locations, Populations, and Health Facilities........................... 17 Table 3: Number of KIIs and Affiliations Within StopPalu+ Target Districts....................... 20 Table 4: Number of FGDs in StopPalu+ Target Districts ....................................................... 21 Table 5: FGD Health Districts in StopPalu+ Activity Locations............................................. 21 Table 6: Alignment of the Goal and Objectives of the National Strategic Plan and StopPalu+..................................................................................................................................... 26 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 6 ACRONYMS AND ABBREVIATIONS Acronym Definition AMELP Activity Monitoring, Evaluation, and Learning Plan ANC Antenatal Care APIC Association pour la Promotion des Initiatives Communautaires AQ Amodiaquine BCC Behavior Change Communication CAFD Centre Africain de Formation pour le Développement CAG Community Action Group CAM Club des Amis du Monde CHW Community Health Worker CJMAD Comité des Jeunes Mon Avenir D’abord CMC Central Medical Store COVID-19 Coronavirus Disease 2019 DCS Direction Communale de la Santé (Communal Health Directorate) DHIS2 District Health Information System 2 DHS Demographic and Health Surveys DPS Direction Préfectorale de la Santé (Prefectural Health Directorate) ECAMM External Competency Assessment for Malaria Microscopists EPI Expanded Programme on Immunization FGD Focus Group Discussion GFATM Global Fund to Fight AIDS, Tuberculosis, and Malaria GH EvaLS Global Health Evaluation and Learning Support Activity HRH2030 Human Resources for Health 2030 IEC Information Education Communication INAASPO Initiatives et Actions pour l’Amélioration de la Santé des Populations IP Implementing Partner IPC Infection, Prevention, and Control IPTp Intermittent Preventive Treatment of Malaria in Pregnancy KI Key Informant KII Key Informant Interview LLIN Long-Lasting Insecticidal Net M&E Monitoring and Evaluation MCDI Medical Care Development International ME&A Formerly Mendez England & Associates MEL Monitoring, Evaluation, and Learning MICS Multi-Cluster Indicator Survey MIS Malaria Indicator Survey MOH Ministry of Health NCC National Coordination Committee NGO Non-governmental Organization NMCP National Malaria Control Program NMS National Malaria Strategy PMI U.S. President’s Malaria Initiative RDT Rapid Diagnostic Testing RECO Relais Communautaires (Community Health Worker) RTI Research Triangle Institute International SBCC Social Behavior Change Communication SGPD Solidarité Guinéenne pour le Développement GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 7 Acronym Definition SMC Seasonal Malaria Chemoprevention SNIS National Health Information System SOW Scope of Work SP Sulfadoxine/Pyrimethamine TWG Technical Working Group UGANC University of Gamal Abdel Nasser, Conakry U.S. United States USAID United States Agency for International Development GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 8 EXECUTIVE SUMMARY Introduction The Global Health Evaluation and Learning Support (GH EvaLS) activity conducted a performance evaluation of the USAID/U.S. President’s Malaria Initiative (PMI)-funded StopPalu+ activity in Guinea, which was implemented from December 2017 to October 2022. This evaluation focused primarily on identifying the approaches and components of the StopPalu+ activity that have been most effective in achieving the goals of the PMI. In addition, this evaluation will help identify promising practices and lessons learned, along with weaknesses and gaps that hinder effective malaria service delivery. It will inform the Government of Guinea’s National Malaria Control Program (NMCP) and future program design and implementation for malaria control in the country. The target audiences for this StopPalu+ activity performance evaluation include: (1) the USAID Guinea Mission/PMI Team; (2) StopPalu+ implementing partners (IPs): RTI International (prime) and sub-primes; (3) NMCP and the Ministry of Health (MOH) units involved in malaria care services at all levels; (4) community members and representatives, and community health workers (CHWs, also referred to as RECOs); and (5) other in-country malaria stakeholders. Activity Background Malaria is endemic throughout Guinea. The country has made important progress in malaria control and prevention, substantially reducing malaria prevalence in children under age 5, annual malaria incidence, and in-patient deaths. These gains were driven by the rapid scale-up of malaria prevention and control interventions, led by the country’s NMCP and supported separately by PMI and the Global Fund to Fight AIDS, Tuberculosis, and Malaria (GFATM). The Government of Guinea is currently implementing a five-year National Malaria Strategy (NMS) with the goal of bringing the country toward pre-elimination by the end of 2022. This will be achieved by reducing malaria morbidity and mortality by 75 percent compared to 2016 Guinea Multi-Cluster Indicator Survey (MICS) levels. In December 2017, USAID Guinea, with funding from PMI, launched the five-year StopPalu+ activity, effectively a continuation of the prior StopPalu activity also implemented by RTI International from 2013–2017. StopPalu+ was meant to assist the Government of Guinea in achieving the collective goal of reducing malaria-related morbidity and mortality by 75 percent compared to 2016 levels through the following objectives: 1. Increased use of long-lasting insecticidal nets (LLINs) by the population; 2. Increased use of intermittent preventive treatment of malaria in pregnancy (IPTp) during antenatal care (ANC) visits; 3. Increased prompt care seeking and treatment; 4. Increased numbers of children receiving the full treatment doses of seasonal malaria chemoprevention (SMC), delivered on time; 5. Increased community involvement in and support for malaria prevention and care; and 6. Improved capacity of NMCP at the central, regional, and district levels to manage, implement, and monitor prevention, care, and treatment activities. StopPalu+ is implemented by RTI International in partnership with Jhpiego, Medical Care Development International (MCDI), Centre Africain de Formation pour le Développement (CAFD), and other local GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 9 partners including Association pour la Promotion des Initiatives Communautaires (APIC), Club des Amis du Monde (CAM), Comité des Jeunes Mon Avenir D’abord (CJMAD), Initiatives et Actions pour l’Amélioration de la Santé des Populations (INAASPO), Solidarité Guinéenne pour le Développement (SGPD), Maférinya Rural Research Center, and University Gamal Abdel Nasser Conakry (UGANC). StopPalu+ works with the NMCP at the national level, in all five communes of Conakry, and in 13 districts in the western and northern parts of Guinea. Assignment Purpose and Key Questions The evaluation assessed the extent to which StopPalu+ interventions have contributed to improvements in the availability, quality, and use of malaria care services1 at the community, facility, and district levels, identifying both enabling and hindering factors and approaches. USAID posed five questions that guided this evaluation’s inquiry and that are presented below: 1.1. To what extent has StopPalu+ contributed to improvements in malaria care services across the different levels of the health system in Guinea? 1.2. What are the sustainability factors that are evident? 2. To what extent has the capacity of the NMCP to lead, coordinate, and manage malaria prevention and control activities been strengthened as a result of StopPalu+ efforts? 3. What kind (if any) of positive and negative knock-on effects did the interventions that StopPalu+ supported have on the wider health systems in Guinea? 4. To what extent did the StopPalu+ activity address gender issues in access and utilization of malaria services? Assignment Methods and Limitations This performance evaluation used a cross-sectional, non-experimental mixed-method design (qualitative and quantitative) to produce the evidence and insights to answer the evaluation questions. Qualitative data comprised most of the analysis, drawing upon reviews of documents (including activity reports), planning meetings; key informant interviews (KIIs) and focus group discussions (FGDs); visits to health facilities, and Mission debrief meetings. Quantitative data was secondary and comprised routine monitoring data mined from StopPalu+ results reports (quarterly and annual progress reports) to determine successful activity components over time, data from the Guinea Demographic and Health Surveys (DHS) and Guinea Malaria Indicator Surveys (MIS) to show national level trends in malaria indicators. The composition of data varied depending on the level of the relevant analysis—national, regional, district, or facility. At the national level, the evaluation team collected data virtually and in person from the representatives of the following groups or individuals: USAID/PMI Guinea staff; Program Office/M&E Specialist; Government of Guinea MOH NMCP; StopPalu+ IPs, including RTI International (prime), Jhpiego, MCDI, and CAFD; and other local partners, including APIC, CAM, CJMAD, INAASPO, and SGPD (sub￾recipients). The team conducted 64 key KIIs representing major institutions (public and private) and community and other stakeholders involved in the StopPalu+ activity (more information can be found in Table 3: Number of KIIs and Affiliations) Within StopPalu+ Target Districts. The team also organized 14 FGDs 1 Malaria care services include prevention, diagnostics, treatment, and positive anti-malaria behaviors and practices as adopted by the people of Guinea. Availability means the extent to which the scale and scope of malaria services changed over the period of the activity. Quality means the extent to which these services meet national and international standards. GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 10 within selected communities: seven each in urban and rural districts covered by the activity. Respondents were pregnant women and mothers of children under the age of 5 years. Qualitative data, collected primarily from the interviews and group discussions, were transcribed in Microsoft Word and then the transcripts were analyzed using QDAMiner Lite grounded coding techniques. The results were then triangulated across stakeholder groups and with secondary data sources such as mined data from progress reports and DHS data, according to each evaluation question and sub-question and then used to populate a findings, conclusions, and recommendations matrix. The team paid particular attention to ethical considerations to ensure that informants and participants were protected against risks during data collection. Transcripts are securely stored in GH EvaLS data vaults, no identifiers are included in reports, and information is aggregated and presented by topic. The performance evaluation utilized an informed consent form (see Annex 3) before any interview or FGD to ensure that consent was obtained before data collection. Findings, Conclusions, and Recommendations Key Findings Evaluation Question 1a. The StopPalu+ activity contributed to improvements in the availability, quality, and use of malaria care services at different levels of the health system in Guinea to the satisfaction of activity stakeholders and beneficiaries. The specific objectives of the activity to reduce malaria morbidity and mortality were relevant and consistent with the national malaria strategies. Although activities and interventions were implemented as planned, according to schedule and quality standards, factors such as the COVID-19 pandemic and NMCP leadership’s challenge in harmonizing activities and strategies between the GFATM and PMI affected implementation. Evaluation Question 1b. The government’s leadership, community ownership, public-private partnerships, continuous capacity building, and health system strengthening are critical factors to ensure the sustainability of the activity. Evaluation Question 2. The StopPalu+ activity provided much-needed technical support to the NMCP in achieving its objectives of leading, coordinating, and managing malaria prevention and treatment activities through capacity building and health system strengthening. Evaluation Question 3. The StopPalu+ activity supported interventions that used various communication channels to raise the awareness of the target populations about malaria control strategies, and consequently increased community trust, respect, and uptake of health care services across the levels of the health system in Guinea (such as increased IPTp and prenatal care uptake). Evaluation Question 4. StopPalu+ promotes gender equality and inclusiveness at all levels where the activity intervenes, by ensuring that women are given priority whenever possible during staff recruitment with stakeholders. Conclusions The performance evaluation shows that StopPalu+’s objectives of fighting malaria, the major cause of morbidity and mortality in Guinea, were relevant. The activities were planned and implemented according to the guiding principles of the NMS and the National Health Development Plan. Health care providers and CHWs were trained to carry out MOH directives according to schedule and quality GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 11 standards. Monthly district data quality review meetings were held to evaluate program indicators and make corrections for any incoherence found. Overall, StopPalu+ contributed to improvements in the availability, quality, and use of malaria care services at the different levels of the health system in Guinea to the satisfaction of activity stakeholders and beneficiaries. The rate of LLIN use was described to have increased significantly among the population by a Government of Guinea stakeholder during an interview, following routine distribution and the distribution campaign supported by StopPalu+ in 2019. The positive outcomes of the activity, such as increased uptake of LLINs, contributed to increased satisfaction among most stakeholders, which in turn can contribute to decrease in malaria morbidity and mortality. Beneficiaries’ trust and respect for the health care system were boosted, and more pregnant women sought ANC, as well as immunization services for their children. Because of the activity’s success, most beneficiaries are advocating for changes in policies for children at age 6 years and older to qualify for SMC, for example. The advent of the COVID-19 pandemic and inadequate communication between IPs and between the GFATM and PMI affected the effective coordination of malaria control activities in Guinea. While StopPalu+ followed the national payment scale to remunerate CHWs, other agencies and programs paid more. Therefore, the low remuneration of CHWs residing in StopPalu+ activity zones compared to those in the rest of the country affected their motivation to carry out their daily tasks. While building the capacity of health care providers improved the quality of services, engaging the private sector could contribute to expanding service coverage. StopPalu+ activity beneficiaries at the different levels of the health system have taken ownership of the activity approach. Core team members at all levels are involved in planning, mobilizing, and implementing activities. For example: ● At the national level, support to the NMCP, such as an embedded advisor, was a strategy geared towards improving the NMCP’s capacity to lead, manage, and coordinate malaria prevention and treatment activities in the country, and this was successful to some extent. Capacity building and health systems strengthening has made the NMCP more dynamic. ● At the facility level, the capacity building of health care providers, improvement of the NMCP monitoring and evaluation (M&E) system, and diagnostics strengthened Guinea’s health care system. ● At the community level, the creation of community action groups (CAGs), which enable community members to organize community dialogues, also enabled them to identify problems and develop solutions. Trained CHWs conducted malaria prevention activities and case management. The StopPalu+ activity used a mix of communication channels to raise awareness about malaria control strategies and to reach the target populations. These channels included mass media (TV and radio), interpersonal communication facilitated by CHWs and trained religious leaders, and home visits. Flyers posted at health facilities were also used as means of communication. Community dialogue, an innovative approach to promoting community participation and ownership, placed the community at the forefront of all malaria control interventions. Consequently, the activity increased community trust, respect, and uptake of health care services across the levels of the health system in Guinea (such as increased IPTp and prenatal care uptake in Guinea). As one of the activity priorities, gender consideration was at the center of activities. During recruitment of personnel, priority was given to women whenever possible. StopPalu+ promoted equity and inclusiveness at all levels of the activity. GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 12 StopPalu+ activity was mostly successful. It increased the use of malaria control activities, including uptake of LLINs in the community, IPTp during ANC visits, and SMC among children ages 3 to 59 months. StopPalu+ activities significantly reduced malaria cases and mortality among pregnant women and children under age 5. This success was achieved through capacity building, including operations of the insectary and laboratory for entomology, social mobilization, use of mobile clinics and red booklets, a steady supply of commodities, and strategic behavior change communication. Recommendations Based on the performance evaluation findings and conclusions, the evaluation team makes the following recommendations: Recommendations for StopPalu+ 1. Increase collaboration with the MOH, UGANC, the National Health Information System (SNIS), and NMCP to design relevant interventions that are consistent with country needs and NMCP strategies/objectives, to make the most impact on the health of the population. 2. Collaborate with relevant programs and agencies to ensure that malaria-related commodities (IPTp, SMC, etc.) are always in stock. This collaboration will include capacity development in the areas of supply management, commodity ordering, and distribution. 3. CHWs (RECOs) have played a significant role in the success of StopPalu+ at community and facility levels. The activity can sustain gains by continuously training existing and new community￾based health workers. 4. As a best practice, StopPalu+ should advocate the institutionalization of community dialogs to promote community engagement in malaria control activities. 5. StopPalu+ and similar USAID activities should ensure that all sections of activity monitoring, evaluation, and learning plans (AMELPs), such as performance indicator reference sheets and indicator performance tracking tables, are regularly updated with consistent and complete progress data to appropriately reflect activity progress. Recommendations for USAID 1. Make future funding contingent upon submission of an action plan describing how the government will take over the activities implemented by the activity. 2. Prepare the ground to ensure a gradual takeover of all the activities currently supported through the activity. 3. Organize training on leadership and management to strengthen NMCP leadership’s managerial skills to improve coordination of activities among IPs and manage stocks. 4. Train the NMCP in grant writing to prepare them to seek new funding streams. 5. Support the NMCP to implement a compensation policy that all Government of Guinea and development agencies can adhere to as they implement development activities. 6. Monitor progress being made to promote gender equity across all program and community levels. 7. Set aside a special funding mechanism to support the insectarium after the end of the activity. Considering the high cost of supplies to run genetic and other advanced tests, external financial support is needed in the future to keep the insectarium running. 8. Assume a proactive monitoring role to ensure that the annual performance reports include consistent data for all performance indicators. GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 13 Recommendations for the MOH/NMCP 1. Fulfill its funding obligations to health sector programs. The Government of Guinea must fulfill its responsibility. The government investment in health care has a strong multiplier effect throughout the economy in addition to enhancing the population’s wellbeing. 2. Sustain the current gain achieved by the activity. Because StopPalu+ will come to an end, the Government of Guinea must take full responsibility to sustain what the activity has accomplished so far. The government should identify other revenue streams to maintain the current level of activities. 3. Standardize effective interventions such as community dialogue by CAGs across the country. Considering the limited resources, the Government of Guinea must invest in these resources only in effective interventions expected to yield a high return on investment. 4. Take immediate corrective action to address the recurrent stockouts of commodities at health facilities. The NMCP should organize additional training in leadership and management to improve staff skills in coordinating activities among IPs, as well as stock management. 5. Take immediate corrective action to address staff’s lack of motivation and turnover. The quality of services is associated in part with the staff competencies and level of satisfaction. The NMCP should analyze the root causes of the lack of motivation and address it expeditiously to maintain the gains of the activity. 6. Develop more public-private partnerships to improve access to and coverage for health care services. Considering that many people seek medical attention from private providers (pharmacists or health centers) or resort to self-medication, the private sector can play a significant role in the fight against malaria. Because public-private partnerships leverage the strengths of the public and private sectors simultaneously, they could create economies of scale and expand access to care for people at high risk and/or underserved populations. 7. Implement a gender equity policy across all programs and community levels. Gender equity is a powerful social determinant for health. Malaria in pregnancy contributes to significant perinatal morbidity, causing higher rates of miscarriage, intrauterine demise, premature delivery, low birth-weight neonates, and neonatal death. Considering these factors, the evaluation team recommends that the NMCP actively pursues gender equity and inclusiveness across all program levels. 8. Standardize monthly report review meetings to improve the quality of data collected and support good decision-making and quality of care. GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 14 1. EVALUATION PURPOSE AND EVALUATION QUESTIONS 1.1. ASSIGNMENT PURPOSE The purpose of the performance evaluation of the StopPalu+ was to establish which approaches and components of the activity have been most effective in achieving the U.S. President’s Malaria Initiative (PMI) goals. The evaluation was also intended to help identify promising practices and lessons learned, along with weaknesses and gaps that hinder effective malaria service delivery. In addition, this evaluation will inform the Government of Guinea’s National Malaria Control Program (NMCP) and future program design and implementation for malaria control in the country. 1.2. ASSIGNMENT QUESTIONS2 This evaluation followed a retrospective and descriptive design, using methods that align with the objectives of the evaluation which include: (1) they are aligned with the purpose of the evaluation and the intended use of the results, (2) they are defined to produce the necessary evidence and results, and (3) they are tailored to the time and budget parameters of the assignment. The evaluation addressed the four evaluation questions presented in Table 1: Evaluation Questions and Sub-Questions. The evaluation questions are divided into sub-questions that can help answer the main questions. Table 1: Evaluation Questions and Sub-questions Evaluation Question Sub-question 1.1. To what extent has StopPalu+ contributed to improvements in malaria care services across the different levels of the health system in Guinea? 1.1.a. To what extent have the project activities/interventions been implemented as planned? 1.1.b. Were the project activities/interventions implemented according to schedule and quality standards? 1.1.c. Are the specific objectives of the project relevant and consistent with national malaria orientations, objectives and strategies? 1.1.d. To what extent have the project activities/interventions contributed to improvements in the availability, quality, and use of malaria care services in the community, facility, and district levels? 1.1.e. How have various factors (socio-political, cultural, etc.) hampered and/or facilitated the implementation of the project? 1.1.f. What is the level of satisfaction of the stakeholders and beneficiaries of the project? 1.2. What are the sustainability factors that are evident? 1.2.a. What are the main factors that could influence the sustainability of the malaria program in Guinea that have facilitated or not the implementation of the project? 1.2.b. To what extent beneficiaries at the community, facility, and community levels have taken ownership of the project approach? 1.2.c. Who are the main factors that could play a role and be involved to ensure sustainability of the project? 2 The evaluation team modified the evaluation questions included in the original scope of work (SOW) by expanding them based on consultations with USAID. The amended SOW is presented in Annex 1: Scope of Work. GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 15 Evaluation Question Sub-question 2. To what extent has the capacity of the NMCP to lead, coordinate, and manage malaria prevention and control activities been strengthened as a result of StopPalu+ efforts? 2.a. What support has the project provided to the NMCP in achieving its objectives of leading, coordinating, and managing malaria prevention and treatment activities? 2.b. To what extent have the capacity building activities of the project improved the overall management of the NMCP? 3. What kind (if any) of positive and negative knock￾on effects has the StopPalu+ supported interventions had on the wider health systems in Guinea? 3.a. What are the communication channels used to reach the project’s target populations (how will you rank those channels from your preference point of view)? 3.b. What are the effects derived from project activities that have enabled the strengthening or not of the health systems in Guinea? 4. To what extent has the StopPalu+ project addressed gender issues in access and utilization of malaria services? 4.a. To what extent has the promotion of gender equality been integrated into the project activities? 4.b. To what extent has the project improved the factors associated with intermittent treatment in pregnant women? 4.c. To what extent has the project improved the factors associated with intermittent treatment in children under five? How are trends for girls compared with boys? 1.3. ASSIGNMENT AUDIENCE The target audiences for the StopPalu+ activity performance evaluation include: ● The USAID Guinea Mission/PMI team ● StopPalu+ IPs: RTI (prime) and sub-recipients ● USAID/PMI HQ Backstops (Washington DC/Atlanta) ● NMCP and the MOH units involved in malaria prevention and care services at all levels ● Community members and representatives, as well as CHWs ● Other in-country malaria stakeholders GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 16 2. BACKGROUND Malaria is endemic throughout Guinea. The country has made important progress in malaria control and prevention, substantially reducing malaria prevalence in children under age 5, annual malaria incidence, and patient deaths. These gains were driven by the rapid scale-up of malaria prevention and control interventions, led by the country’s NMCP and supported by the PMI and GFATM. Comparison of the 2012 DHS with the 2016 MICS shows a substantial decrease in the prevalence of malaria parasitemia in children aged 6–59 months in Guinea. However, there is substantial regional variation in malaria parasitemia prevalence. With high prevalence in some regions, the entire population of 12.1 million remains at risk; approximately 1.7 million cases were reported in 2019, accounting for 37 percent of outpatient visits.3 The 2016 MICS found that malaria prevalence in Guinea is heterogeneous, with prevalence ranging from 1.9 percent in Conakry to 30.2 percent in N’Zérékoré. The Government of Guinea is currently implementing a five-year NMS with the goal of bringing the country toward pre-elimination by the end of 2022. This will be done by reducing malaria morbidity and mortality by 75 percent from 2016 Guinea MICS levels. In addition to funding from PMI, implementation of the NMS has benefited from grants from GFATM, including the third round of the New Funding Model (NFM3, 2021–2023). 2.1. PROJECT OVERVIEW Area Info Project/Activity Title: StopPalu+ Award/Contract Number: 72067518CA00001 Award/Contract Dates: December 2017–October 2022 Project/Activity Funding: $28 million (currently $44 million) Implementing Partner(s): RTI International (Prime) Project/Activity AOR/COR: Lamine Bangoura In December 2017, USAID Guinea, with funding from PMI, launched a second serial contract for RTI/StopPalu+ to assist the Government of Guinea to achieve the collective goal of reducing malaria￾related morbidity and mortality by 75 percent from 2016 levels through the following objectives: ● Increased use of LLINs by the population ● Increased use of IPTp during ANC visits ● Increased prompt care seeking and treatment ● Increased numbers of children receiving full treatment doses of SMC, delivered in a timely manner ● Increased community involvement in and support for malaria prevention and care ● Improved capacity of NMCP at the central, regional, and district levels to manage, implement, and monitor prevention, care, and treatment activities 3 Source: Assignment SOW (Annex 1: Scope of Work). GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 17 StopPalu+ is implemented by RTI International in partnership with Jhpiego, MCDI, CAFD, and other local partners (APIC, CAM, CJMAD, INAASPO, Maférinya Rural Research Center, University of Gamal Abdel Nasser Conakry, and SGPD). StopPalu+ works with the NMCP at the national level, in all five communes of Conakry and in 13 districts in the western and northern parts of Guinea. See Figure 1: Map of Guinea Showing Activity Districts and Table 2: StopPalu+ Activity Location, Population, and Health Facilities. Figure 1: Map of Guinea Showing StopPalu+ Activity Districts Table 2: StopPalu+ Activity Locations, Populations, and Health Facilities Region No. Health District Covered Population of Women of Reproductive Age Number of Existing Health Centers Conakry 1 Kaloum 66,797 7 2 Dixinn 145,193 5 3 Matam 153,210 3 4 Ratoma 697,744 13 5 Matoto 712,185 7 Total 1,775,129 35 Boké 6 Boké 48,101 15 7 Boffa 22,722 8 8 Fria 103,385 6 9 Gaoual 207,033 8 10 Koundara 13,890 7 Total 395,131 44 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 18 Region No. Health District Covered Population of Women of Reproductive Age Number of Existing Health Centers Kindia 11 Forécariah 259573 12 12 Coyah 281757 10 13 Dubréka 353859 8 Total 895,189 30 Labé 14 Labé 33,984 18 15 Mali 30,755 13 16 Lelouma 17,420 11 17 Tougue 13,292 10 18 Koubia 10,670 6 Total 106,121 58 Grand Total 18 3,171,570 167 Source: USAID/GH EvaLS Assignment 013 SOW The StopPalu+ technical approach draws on recent national and subnational survey data; best practices developed and implemented under the previous StopPalu activity; continued close collaboration with the MOH/NMCP, USAID/PMI, and other stakeholders; and the imperative to work with providers, civil society organizations, community-based organizations, and communities to identify and address specific needs in the demand for and coordination and provision of malaria services. In 2020, in addition to ongoing polio, measles, and yellow fever epidemics, Guinea experienced the onset of the devastating COVID-19 pandemic, resulting in the death of at least 87 Guineans by February 25, 2021. To minimize the negative effects of increased demands on health care provision, the MOH, through the NMCP and with the support of partners including GFATM and its IPs, increased efforts to continue to implement interventions to combat malaria while incorporating COVID-19 infection, prevention, and control (IPC) accommodations for field staff and their target audiences. These included prevention (routine distribution of LLINs, IPTp, SMC, household visits, entomology surveillance, and other means), case management (diagnosis and treatment) in health facilities and at the community level, and interpersonal communication and mass media to support the uptake and maintenance of the aforementioned interventions. Guinea still faces the unique challenge of trying to revive its health system and malaria activities in a country affected by the underlying threat of multiple viral hemorrhagic fevers, vaccine-preventable disease resurgence, and COVID-19. This includes the need to constantly adjust the implementation of malaria case management guidelines in the context of COVID-19, along with the ongoing need to promote infection prevention and control in health facilities by providing personal protective equipment to health workers in health facilities and communities (e.g., CHWs) to examine patients, including conducting rapid diagnostic testing (RDT) for suspected malaria cases. GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 19 3. ASSIGNMENT METHODS AND LIMITATIONS 3.1. ASSIGNMENT METHODOLOGY The performance evaluation of StopPalu+ used a cross-sectional, non-experimental mixed-method design (desk review, qualitative, and quantitative) to produce the evidence and insights to answer the evaluation questions. A team of three evaluators conducted the evaluation, supported by a logistics specialist. The team will share this report with USAID and StopPalu+ activity IPs to inform further program implementation. Qualitative data comprised most of the analysis, drawing on a review of documents (including activity reports), planning meetings, KIIs, FGDs, health facility visits, and mission debrief meetings). Quantitative data consisted of extracts from activity quarterly and annual reports, annual reports of entomological surveillance activities, and secondary sources such as the Guinea DHS and Guinea MIS, from which malaria indicators data were extracted. These data were analyzed for trends over the life of award. The evaluation went beyond listing StopPalu+ achievements that were previously recorded under the routine performance monitoring system to assess whether those contributions have been of the required quality, timely, and relevant to result in improvements in services at different levels. Process The evaluation was divided into three major phases: preparation, field work, and reporting. Preparation phase. The evaluation team reviewed activity-related documentation provided by USAID/PMI Guinea and by StopPalu+ (see Annex 2, Sources of Information). The aim was to gain insights into the activity’s design and implementation based on its goals, objectives, strategies, and approaches. The team developed an inception report that contained the evaluation methods and tools, such as the interview guides (see Annex 3, Data Collection Tools). Field work phase. The second phase focused on field work in Guinea: KIIs (see Annex 3), and field visits at selected StopPalu+-supported sites, and FGDs with pregnant women and mothers of children under age 5. The team selected sites to visit based on the activity’s geographical, epidemiological, and technical coverage. Reporting phase. During this phase, the evaluation team triangulated information from desk reviews, the qualitative data collected, and secondary quantitative data (DHS, MIS) to populate a findings, conclusions, and recommendations (FCR) matrix. This was followed by a topline presentation of findings to the USAID/Guinea team. Feedback from the presentation was incorporated into a draft evaluation report. 3.2. DATA SOURCES This evaluation considered primary and secondary data sources that originated largely from qualitative research including KIIs, FGDs, and a desk review. Quantitative sources included secondary data from activity progress reports, the DHS (2018/2021), and the MIS (2018/2021). The evaluation methods and data sources depended on the evaluation questions and whether a participant was at the national, regional, district, or facility level. Data Management and Data Analysis are discussed in detail below. GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 20 3.2.1. Desk Review The evaluation team conducted a desk review through a systematic analysis of documents made available by USAID/PMI Guinea and StopPalu+. The desk review provided information regarding StopPalu+ progress toward meeting its objectives over time, as well as achievements by strategic objective and target. The evaluation team extracted quantitative data and historical information related to the evaluation questions from reports to complement data from other sources. (see Annex 2, Sources of Information). 3.2.2. Key Informant and Group Interviews A purposive selection of key informants (KIs), stratified by stakeholder group and geography (central, regional, and district) was selected for in-depth and group interviews. The evaluation team identified informants through the desk review and recommendations from USAID and IPs. Table 3 presents the number of KIIs based on informants’ affiliations and StopPalu+ target districts. Table 3: Number of KIIs and Affiliations Within StopPalu+ Target Districts Stakeholder Central Forécariah Dubréka Boké Fria Labé Lelouma Koubia Total USAID /PMI 3 3 MOH4 3 1 1 1 1 1 1 1 10 IPs 13 13 Partner institutions 3 3 StopPalu+ field agent 1 1 1 1 1 1 1 7 Health management committee 1 1 1 1 1 1 1 7 Facility supervisor 1 1 1 1 1 1 1 7 Community agent 2 2 2 2 2 2 2 14 Total 6 6 6 6 6 6 6 64 The evaluation team conducted interviews largely in French. All interviews followed interview guides, with topics tailored to the stakeholder group (see Annex 3, Data Collection Tools) responding to the evaluation questions. All interview sessions were recorded with the consent of respondents. Interview notes and recordings were kept in a GH EvaLS locked electronic folder, accessible only to the evaluation team; these will be deleted once the report is completed. Information from each interview was recorded using a data extraction log for interviews (see Annex 3, Data Collection Tools) to improve systematic analysis. 3.2.3. Focus Group Discussions The evaluation team conducted FGDs in the language of the environment in selected communities to obtain beneficiaries’ perspectives of StopPalu+ activity interventions. FGDs targeted pregnant women 4 Including the Direction des Grandes Endémies, Programme National de Lutte contre le Paludisme and Direction Préfectorale de la Santé. GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 21 and mothers of children under age 5 regarding malaria care services. The evaluation team visited health facilities in the communities selected for FGDs and observed facilities’ provision of malaria care services. The team developed and used a FGD guide to ensure topics of discussion in each selected community focused on responding to the evaluation questions. The team conducted 14 FGDs—seven each in rural and urban areas, each consisting of about six to eight participants. (See Table 4, Number of FGDs in StopPalu+ Target Districts). The FGDs and KIIs were conducted during January and February 2022. Table 4: Number of FGDs in StopPalu+ Target Districts Stakeholder Capital Forécariah Dubréka Boké Fria Labe Lelouma Koubia Total Pregnant women and mothers of children under age 5 in urban areas - 1 1 1 1 1 1 1 7 Pregnant women and mothers of children under age 5 in rural areas - 1 1 1 1 1 1 1 7 Total 2 2 2 2 2 2 2 14 3.3. SAMPLE SELECTION The team interviewed KIs including representatives of public and private institutions involved in the StopPalu+ activity, such as USAID Program Office and PMI staff, activity partners, relevant stakeholders, and host government counterparts across the levels of Guinea’s health system. USAID/PMI selected KII respondents, including stakeholders at the global, headquarters, and country/field levels. FGDs with pregnant women and mothers of children under age 5 were organized within selected urban and rural communities. A total of 14 FGDs were conducted across seven health districts—seven each in urban and rural areas (see Table 5, FGD Health Districts in StopPalu+ Activity Locations). Table 5: FGD Health Districts in StopPalu+ Activity Locations Health Districts of Project Area For écariah Coyah Dubr éka Boffa Boké Fria Gaoual Koundara Labé Mali Lelouma Tougue Koubia Total Selected districts (urban) 1 0 1 0 1 1 0 0 1 0 1 0 1 7 Selected districts (rural) 1 0 1 0 1 1 0 0 1 0 1 0 1 7 Total selected districts (communities) 2 0 2 0 2 2 0 0 2 0 2 0 2 14 3.4. DATA MANAGEMENT AND ANALYSIS Each data collection method was implemented and analyzed separately. The evaluation team extracted data and historical information from the desk review of activity documents and secondary datasets (DHS, MIS) and conducted KIIs and FGDs that were transcribed using Microsoft Word, and analyzed using grounded coding techniques in QDAMiner Lite qualitative data analysis software. GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 22 The evaluation team triangulated (compared, contrasted, and validated) findings from the different sources of data and information (data from interviews, group discussions, and secondary data from the DHS and MIS) according to the evaluation questions and sub-questions, and organized analysis results in a FCR matrix. 3.4.1. Document Review The assessment team performed targeted reviews of activity reports and other relevant documents provided by USAID/PMI Guinea. These included the StopPalu+ Project M&E Plan, annual and quarterly progress reports, Human Resources for Health 2030 (HRH2030) reports, Health Systems Assessment, USAID Health Systems Strengthening Framework, Assessment of long-term technical assistance advisors to NMCP (scope of work [SOW] and reports), National Malaria Strategy, National Community Health Workers Strategy, Service Continuity Plan for COVID-19, and DHS 2018 report, among others. The team reviewed these documents and available secondary data to understand the activity context. 3.4.2. Qualitative Data Analysis The transcription of KIIs and FGDs began immediately after field interviews were completed. The evaluation team analyzed the KII and FGD transcripts and notes using content and thematic analysis techniques in which text was coded according to key themes of interest that aligned with the evaluation questions and sub-questions, using the QDAMiner Lite qualitative data analysis software. The team analyzed transcripts relying on the content analysis method, which required: ● Using verbatim remarks and having sentences as the unit of analysis ● Grouping results by key themes of interest ● Identifying different sub-themes positions in relation to each major theme ● Summarizing each theme and assessing its strength or degree of importance The team structured the findings in a descriptive narrative based on the evaluation questions 3.4.3. Quantitative Data Analysis The team collated quantitative data from secondary sources including activity progress and results reports (quarterly and annual), DHS Guinea, and MIS Guinea. This data was used to help understand the changes that might have resulted from StopPalu+ interventions. The data were analyzed to show progress in national- and activity-level malaria and related indicators, and to respond to the assignment evaluation questions. These secondary sources were triangulated with findings from document reviews and from qualitative KIIs and FGDs to respond to the evaluation questions and to reach conclusions. Analysis of quantitative data was descriptive in nature, with activity indicators progress and results compared over the life of the activity where possible. 3.5. ETHICAL CONSIDERATIONS The evaluation methodology, data collection, analysis, and dissemination follow the GH EvaLS Data Management Plan. This plan outlines data handling from pre-data collection to reporting and dissemination, and it provides guidance regarding data quality, flow, storage, and privacy throughout the evaluation process. The Data Management Plan provides guidelines to ensure privacy and confidentiality in all data collection. GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 23 The evaluation will rely on the complementary approaches stated above (document review, KIIs, group interviews, observations, and data triangulation. All data collection during the evaluation will begin with an informed consent process (Annex 5) and written documentation that includes strategies for the protection of respondents, such as de-identification of data collected and the right to non-participation without consequence. Information will be collected or recorded only with respondents’ consent. Information obtained from interviews and discussions will be secured in a password-protected electronic folder that is accessible only to the evaluation team (during the evaluation period) and GH EvaLS. Interview notes, extraction forms, recordings, and all data collected belong to GH EvaLS; all consultants must deposit these materials in the secure folder and shall retain no data. In any reports or publications, the identity of the respondents will not be associated with the information they provided during data collection. Evaluation team members did not identify any conflicts of interest that could influence or limit the scope and rigor of the evaluation or limit the disclosure of or weaken or bias the findings. The evaluation team does not have professional, personal, or financial relationships with the Missions to be approached or with involved implementing organizations. No member of the evaluation team was involved in designing, planning, or implementing the activities under evaluation or has had any kind of contractual obligation to them. The evaluation team can provide independent, objective, and unbiased evaluation due to: ● Organizational independence ● The absence of any kind of conflict of interest in the case of all international and local experts involved in the evaluation ● The professionalism of team members, as demonstrated in previous assignments 3.6. LIMITATIONS Due to the descriptive nature of this evaluation and the focus on learning from document reviews and stakeholders’ experiences with the activity, the evaluation team anticipated possible limitations resulting from the availability and collection of data at hand. These limitations that needed to be mitigated through methodological and analytical means, included, among others: ● Qualitative approach. The primary approach for this evaluation is qualitative data collection via KIIs and FGDs and, to a lesser degree, a review and analysis of secondary qualitative data. The opinions of stakeholders are, by their nature, subjective, and may also reflect certain vested interests. This is the reason the evaluation identified a wide array of stakeholders to minimize subjectivity. ● Selection and representation biases. The evaluation team identified a large and diverse group of stakeholders at the central and peripheral levels. It is possible that some key stakeholders were inadvertently excluded, and that some participants in a KII or group interview might have introduced self-selection bias (either beneficial or detrimental) into the results. To mitigate this concern, the team remained flexible to the extent possible to accommodate as many KIs and FGD participants as they could. ● Response bias. Response bias is a common problem for assessments, evaluations, and reviews and takes several forms. For instance, respondents may give the interviewers positive statements about the activity because they have a vested interest in seeing the activity succeed and continue. Respondents may be inclined to shape their responses according to gender or GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 24 other social norms or to provide responses that they think the interviewer wants to hear. Expanding and aggregating responses on specific topics mitigates this issue. ● Secondary quantitative data sufficiency and quality. Use of secondary quantitative data assumes that these data are valid and reliable. However, this may also mean that, if there were biases in the existing data, they will be introduced in the current use. In addition, secondary data may not always contain sufficient elements and results to answer current evaluation questions. To mitigate these effects, expanding into alternative sources of data, when available, is useful. ● Gender data. Gender data that enables conclusions require more systematic approaches to data collection within evaluations, especially where disaggregation is required. Consistency in methods and elements (especially male, female) of collection posed challenges to conclusions. GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 25 4. FINDINGS This section presents the key findings of the performance evaluation based on the four evaluation questions. 4.1. TO WHAT EXTENT HAS STOPPALU+ CONTRIBUTED TO IMPROVEMENTS IN MALARIA CARE SERVICES ACROSS THE DIFFERENT LEVELS OF THE HEALTH SYSTEM IN GUINEA? Key Finding 1.1. Overall, StopPalu+ implemented a wide variety of interventions at the central, regional, and district levels that contributed, to a large extent, to improvements in the availability, quality, and use of malaria care services to the satisfaction of both the stakeholders and beneficiaries of the activity. The following national indicators that StopPalu+ may have contributed to showed increases between 2018 and 2021— households with at least one insecticide-treated mosquito net (from 44 percent to 63 percent); children under age 5 who slept under an insecticide-treated net (from 27 percent to 38 percent), and pregnant women who slept under an insecticide-treated net (from 28 percent to 39 percent). 4.1a. To what extent have the project activities/interventions been implemented as planned? Although the overall perception among the respondents is that the activity was implemented as planned, the delay in obtaining funds in 2018 due to the U.S. government shutdown and the advent of COVID-19 prevented or delayed the implementation of some major activities. Most respondents indicated that the “activity evolved according to plans and schedule” and that it placed strong emphasis on planning. A KI from Forécariah confirmed this general perception: Absolutely, based on the project work plan, you know that every year the project developed a plan of action at the national level. Every region followed suit and developed its action plan also. So, everything planned was also realized in the field. (KII, Forécariah) 4.1b. Were the project activities/interventions implemented according to schedule and quality standards? Notwithstanding the delay in obtaining funds in 2018 and the advent of COVID19, activity reports across the life of award indicate that interventions were implemented as planned and according to schedule. This is corroborated in the following statements: As for the level of achievement of the specific objectives of the project, if we take the broad lines of the project, we can say that it is following its normal course. It is up to date in terms of the implementation of its activities. The project is evolving according to what is planned and on schedule. (KII, NGO) In general, there was close coordination between the project and the structures involved in the implementation of interventions through the [Direction Préfectorale de la Santé] DPS/Health District Authorities. We had monthly meetings with the actors involved who brought back data to assess the progress of all the interventions underway and to schedule those that are planned. (KII, Labé) GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 26 With regard to standards, the activity “carried out the Ministry of Health’s directives.” The activity conducted several supervision visits to ensure that they were implemented according to the defined quality standards. Corrective measures were taken immediately to address any discrepancy. For example, where the evaluation team noted that ANC staff completed the ANC forms during consultations but did not update the registers, the observers encouraged ANC agents to update the registers daily and recommended collecting the monthly IPTp data through the ANC forms. This project brought a monthly meeting; we did not know that. Of course, we produce reports on the other projects; but the fact of organizing meetings is a very good thing because it allows us to correct the reports and give feedback. (KII, Physician) 4.1c. Are the specific objectives of the project relevant and consistent with national malaria orientations, objectives, and strategies? In its efforts to assist the Government of Guinea in the fight against malaria, the leading cause of morbidity and mortality in the country, StopPalu+ is addressing the country’s most important national health issue. StopPalu+’s objectives are relevant to and consistent with national malaria objectives and strategies. The activity’s objectives are drawn from the Guinea National Strategic Plan’s vision, objectives, and strategies as shown in Table 6: Alignment of Goals and Objectives of the National Strategic Plan and StopPalu+. While the National Strategic Plan called for effective preventive measures against malaria, StopPalu+ spelled out these effective preventive measures and how the activity would intervene. Table 6: Alignment of the Goal and Objectives of the National Strategic Plan and StopPalu+ Goal Objectives National Strategic Plan By the end of 2022, bring the country towards pre￾elimination by reducing morbidity and malaria-related mortality by 75% compared to 2016 1. Ensure the protection of at least 90% of the population with effective preventive measures against malaria 2. Ensure correct and early management of at least 90% of malaria cases 3. Strengthen the management, partnership, coordination, communication, and ES capacities of the malaria program at all levels StopPalu+ Assist the Government of Guinea in its efforts to reduce malaria-related morbidity and mortality by 75% compared with 2016 levels 1. Increased use of LLINs by the population 2. Increased use of (IPTp) of malaria in pregnancy during ANC visits 3. Increased prompt care seeking and treatment 4. Increased numbers of children receiving the full dose of SMC delivered in a timely manner 5. Increased community involvement in and support for malaria prevention and care 6. Improved capacity of the NMCP (central, regional, and district levels) to manage, implement, and monitor prevention, care, and treatment activities The general perception among stakeholders interviewed is that “the project perfectly aligns with the guidelines of the [Sustainable Development Program] and NMCP.” Furthermore, “the project annual work plans were developed under the leadership of the NMCP and broken down by strategies and operational levels.” The following narratives emphasized the relevance and alignment of StopPalu+: GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 27 This project is very relevant. Before the project came along, not a minute went by in the rainy season without me hearing a woman crying about the loss of her child. Now I assure you that this has changed drastically. A week or more goes by without losing a child. This is really a change. (KII, Labé) Obviously, malaria was the leading cause of morbidity and mortality in our community. The project is in line with the objectives of the national malaria control program. It respects all the directives issued by the Ministry of Health, which means that the project is perfectly in line with the [National Summary Data Page. (KII, Labé) I believe that all we (project) do is support the Ministry of Health. The project supports the overall development plan and is under the umbrella of the Ministry of Health and works in close collaboration with the National Malaria Control Program. For example, if the project wants to develop its annual plan, consultations are organized with the NMCP for the development of the work plan, and all technical staff are invited. (KII, USAID) 4.1.d. To what extent have the project activities/interventions contributed to improvements in the availability, quality, and use of malaria care services in the community, facility, and district levels? The activities/interventions contributed significantly to improvements in the availability, quality, and use of malaria care services in the district, community, and facility, levels. To make long-lasting insecticide-treated nets available in the activity-supported areas, StopPalu+ conducted LLIN mass campaign distribution and routine distribution at ANC clinics and health facilities. According to the Guinea DHS, there was an increase at national level in the proportion of households with at least one LLIN from 31 percent in 2018 to 42 percent in 2021 and in the proportion of households with access to an insecticide-treated net within their household from 44 percent to 63 percent in 2018 and 2021, respectively. At the StopPalu+ intervention level, increasing numbers of LLINs distributed throughout the activity areas of operation through routine distribution for instance, (Figure 2: Availability of LLINs and Figure 3: Number of Routinely Distributed LLINs from 2018 to 2021) increased population access. These routine and mass campaign distributions can significantly contribute to availability at national level. GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 28 Figure 2: Availability of LLINs 31% 42% 44% 63% 0% 20% 40% 60% 80% 100% 2018 2021 Access to LLINs Proportion of Population with Access to an ITN within their Household Households with at least one insecticide-treated mosquito net (ITN) Source: Guinea DHS Reports 2018, 2021 Figure 3: Number of Routinely Distributed LLINs from 2018 to 2021 (ANC, Extended Programme on Immunization [EPI]) 33517 55795 49432 87526 31471 60147 54970 77558 27987 54912 55394 171971 36758 60989 84470 100512 0 20000 40000 60000 80000 100000 120000 140000 160000 180000 200000 2018 2019 2020 2021 Source: StopPalu+ Annual reports Boke Kindia Labe Conakry Key informants and group discussants were unanimous about the contribution of the activity to improving availability of and access to malaria services in the community. For example, … the population is very happy with the interventions being carried out in their locality. All the households own the LLINs, and 80 percent use them. Following the distribution conducted in 2019, GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 29 children and people are not getting sick. Even our statistics before and after are showing a significant decline in the prevalence of malaria. Patients who come to the center are those suffering from acute respiratory infection, not those with malaria. Malaria is rare during this season. (Centro Sur, Dalein) You can see where we were in terms of the use of mosquito nets before 2012. In the StopPalu+ zone, the rate of net use was estimated at less than 10 percent before their intervention, but after the three distribution campaigns with StopPalu+ intervention, we noticed that the rate of use had risen to 80 percent. To maintain this coverage, they do routine distribution of LLINs. (KII, NMCP) The first improvement that can be noted is the decrease in malaria, there is the attendance of pregnant women in health facilities; there is social mobilization, and the involvement of all layers of society; these are aspects that should not be overlooked nowadays. (KII, NGO) To assure the quality of malaria care services in the community, facility, and district levels, StopPalu+ conducted regular micro-planning meetings to ensure a systematic and organized process for LLIN distribution. Distribution personnel were trained, for instance, on mobilization and how to assist households to hang up their LLINs. Figure 4: Number of Persons Trained in LLIN Distribution and Promotion shows that a total of 18,446 people were trained to carry out the LLIN mass distribution by 2019 and subsequent years. Training included areas such as micro-planning, enumeration, and LLIN hang-up. Figure 4: Number of Persons Trained in LLIN Distribution and Promotion Furthermore, the activity trained community health workers on the correct way to administer RDT in the community to ensure that only people who have malaria receive the appropriate treatment. Figure 5: Proportion of CHWs Capable of Using RDT at the Household Level shows that at least 77 percent of CHWs were consistently capable of using RDT at the household level. GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 30 Figure 5: Proportion of CHWs Capable of Using RDT at the Household Level 62.41% 77.39% 77.39% 77.39% 0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00% 70.00% 80.00% 90.00% 2018 2019 2020 2021 Source: StopPalu+ Annual Reports Regarding use of malaria care services in the community, facility, and district levels, national data from the Guinea DHS (Figure 6: Proportion of Children Under Age 5 Who Slept Under LLIN the Night Before) shows that the proportion of children under age 5 who slept under LLIN increased from 28 percent in 2018 to 39 percent in 2021, and the proportion of pregnant women who slept under LLIN increased from 27 percent in 2018 to 38 percent in 2021. Figure 6: Proportion of Children Under Age 5 Who Slept Under LLIN the Night Before 28% 39% 27% 38% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 2018 2021 Source: DHS Data % of pregnant women who slept under an LLIN the previous night % of children U5 who slept under a long-lasting insecticide-treated net (LLIN) the previous night GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 31 Specific malaria services such as treatment and referrals are highlighted by results reports from StopPalu+ (Figures 7 and 8). Children under age 5 who are tested and report positive for malaria received treatment from CHWs at household level, and CHWs also referred cases to higher-level facilities in the health system for further treatment. Male and female children under age 5 received the same attention to treatment at the community and facility levels over the three years for which data are available. In addition, the data presented show that the number of male and female children testing positive for malaria decreased over the three year period between 2018 and 2021. Figure 7: Number of Children Under Age 5 Who Tested Positive for Malaria and Receive Treatment From a CHW 32836 17681 23553 32852 17628 24039 0 10000 20000 30000 40000 50000 60000 70000 2019 2020 2021 Source: StopPalu+ Annual Reports Male Female GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 32 Figure 8: Number of Children Under Age 5 Who Tested Positive for Malaria and Were Referred by a CHW for Facility Treatment 442 141 90 470 124 113 0 100 200 300 400 500 600 700 800 900 1000 2019 2020 2021 Source: StapPalu+ Annual Reports Male Female The perceived effectiveness of StopPalu+’s interventions in reducing malaria morbidity in children under age 5 and pregnant women motivated some respondents to advocate for changes in policy regarding the administration of SMC and a reduction in the interval between two consecutive LLIN distribution campaigns. They advocated that the activity review the requirements for the SMC to include children age 6 and older and to reduce the interval between two consecutive LLIN distribution campaigns from three years to two. The following narratives explain the rationale for this demand: Considering the seasonal malaria chemoprevention (SMC) medication, we only give them from 3 months to 5 years old. Now kids 6 years and up need them too. So, I believe the project should think of them too because they are now the ones who are getting sick. (FGD, Dalein) If you only have one bed, you will only have one net, now when the delivery time is too long, the nets we have can get damaged. (FGD, Koubia) For one KI from Koubia, StopPalu+ interventions can have a multiplier effect because “the use of SMC will boost many other interventions such as the vaccination program and prenatal care visits.” 4.1.e. How have various factors (socio-political, cultural, etc.) hampered and/or facilitated the implementation of the project? Overall, although the activity’s capacity building interventions and the health authorities’ ownership of these interventions facilitated implementation, the advent of COVID-19, the delay in funding, and the inability of CHWs to operate without direct assistance from StopPalu+ slightly hampered the implementation of the activity. The evaluation identified socio-political and cultural factors that facilitated or hampered activity implementation. For instance, “the involvement of … health authorities who work in the communities, the GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 33 mayor, the sub-prefect, and the presidents of the districts at all levels facilitated program implementation.” Also, CHWs were essential to the success of the activity as they were involved in implementing, monitoring, and evaluating malaria control activities in the community. “Their effectiveness in administering SMC enhances the confidence of the community in the program.” Furthermore, commodities such as LLINs could not be distributed without CHWs’ involvement. One KI from the NMCP described how StopPalu+ developed a new culture in the organization that facilitated the implementation of the activity: From an organizational point of view, StopPalu+ has put a structural mechanism that allows us to capitalize on what we have learned with ease in implementing activities. For example, when you look at the CTRs (regional technical advisors), a team around them coordinates the activities to be carried out in the field at the regional level. Beyond that, in each prefecture, there is also a team. In their way of doing things, we see a strengthening of capacity and, even better, a transfer of skills, especially by working with local NGOs, which helps to sustain the achievements. (KII, NMCP) Conversely, the COVID-19 pandemic slowed down activities. In some areas, community members were hostile to CHWs because they likened COVID-19 response to the Ebola virus disease response, which they perceived negatively. They were uncooperative with CHWs who were focused on malaria prevention and treatment. Other external factors that hampered program implementation included COVID-19, the lack of medical supplies in the community, CHWs’ lack of initiative, and the seeming lack of coordination among implementing agencies. The following narratives refer to some of these factors: The context of COVID-19 has also slowed down activities a little, but we have adapted. (KII, NGO) There were socio-political factors that hindered the interventions. In this region, for example, the year 2020 was marked by social unrest; even if you wanted to go and supervise, you couldn’t. All the roads were barricaded, and when people don’t move, you can’t work. That’s what made us more tired. Fortunately, it didn’t last; it just took a few months, and activities resumed as normal. (KII, Labé) The factors that can block these activities are socio-political, and I would like to emphasize this a lot, such as election periods, which have a negative influence on these activities. (KII, LeySere) Considering the discourse regarding the shape of LLINs and the motivation of CHWs on the activity implementation, the evaluation team analyzed each influencing factor individually in the following paragraphs. 1. Although Koenker and Youkich (2017) reported that net preference does not influence use,5 the shapes of respondents’ beds were a determinant in the preference and use of nets. Most respondents, based on KIIs and FGDs, indicated that they preferred either conical or rectangular nets because they match the shapes of their beds. FGD participants in Wonkifon and Doumbouya districts expressed the following opinions. In Wonkifon, “all women prefer the round LLINs and the 5 Hannah Koenker and Joshua O. Yukich. Effect of user preferences on ITN [insecticide-treated net] use: a review of literature and data. Malar J. 2017; 16: 233. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5455118/ GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 34 white ones,” whereas in Doumbouya “many women prefer rectangular or square LLINs.” This constitutes a challenge for the program, as some FGD participants noted: The net I received is very small for my bed; I have a 4-seater bed, so the net is small. The conical LLINs are bigger and easier to use; while the rectangular ones are difficult to hang up. (FGD, Kakossa) The rectangular nets are not big enough, sometimes the beds are bigger than the net. When this happens, I take the rest of the old net to the tailors to get what I need. (FGD, Koubia) 2. Trained RECOs’ lack of motivation due to insufficient compensation could threaten the implementation of several activities in the community. RECOs are selected by the community to carry out malaria prevention activities and case management. But their remuneration is said not to cover the cost of basic needs such as transportation or uniforms. Several respondents expressed the RECOs’ concerns regarding the use of a different and higher pay scale for RECOs operating in the GFATM activities area compared to those of StopPalu+: I know that “relais communautaires” (RECOs) work in their community, but there is some concern about their remuneration. I think that the Government should help to address some difficulties. This is a challenge. (KII, USAID) As community workers, what we are given for transport is insufficient. We do not have gloves, we do not have uniforms for work, sometimes we lack the supplies. The number of CHWs is not sufficient to cover all the area. Maintaining motivation is the main difficulty. There is no compassion on the part of the project managers. When we fall sick, it is on us. Although we are chosen by the community, they must be interested in our problems. I had an accident here and no one came to greet me. That’s not encouraging. (KII, Ramatara) But we RECOs have difficulties. We have no means of transportation; you can walk many kilometers between two villages. The project must assist the RECOs, because we live far away (25km to 30km) to get to the center here. Moving the motorbikes costs us 50,000 GNF and the remuneration is only 80,000 GNF. The villages are far from each other, so we ask the project to find us motorbikes, or increase the motivation from 400,000 GNF to 500,000 GNF. (KII, Koubia) 4.1.f. What is the level of satisfaction of the stakeholders and beneficiaries of the project? Stakeholders and beneficiaries are very satisfied with the activity. KIs at the central level reminded the evaluation team that “StopPalu+ came to Guinea when the country had limited resources to run malaria prevention activities. Funds from the GFATM were suspended. The medical commodities that StopPalu+ brought saved countless lives. In addition to the financial support, the project helped the NMCP develop normative documents that laid down clear strategies to implement malaria prevention activities in the country.” They built the capacity of the NMCP at all levels. So, in terms of satisfaction, on a scale of one to 10, their level of satisfaction was between eight and nine. The only area of concern for them is the fact that the country did not reach universal coverage with LLIN. But this was not the sole responsibility of the activity. At the district and community levels, the beneficiaries consistently expressed satisfaction with StopPalu+ interventions. They were “highly appreciative of all the services provided and hoped that the project would last GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 35 until malaria is completely eradicated in Guinea.” The fact that StopPalu+ made malaria commodities (LLIN, Sulfadoxine/Pyrimethamine [SP], and Combinaisons thérapeutiques à base d'artemisinine) available free of charge in the community and health facilities, following Guinea national guidelines, is greatly appreciated. The following narratives illustrate the high level of satisfaction and appreciation perceived from most stakeholders and beneficiaries at different levels: StopPalu+ brought medical commodities that saved lives in Guinea. They helped the NMCP develop normative documents that became the basis for planning malaria programs. I am very satisfied.” (KII, NMCP) The population is very happy with the interventions being carried out in their locality. All the households own the LLINs and most use them. Following the distribution conducted in 2019, children and people are not getting sick. Even our statistics before and after are showing a significant decline in the prevalence of malaria. Patients who come to the center are those suffering from acute respiratory infection, not those with malaria. Malaria is rare during this season. (KII, Dalein) We are very happy about the activities of the project … the magnitude of malaria has really diminished, now we no longer suffer as before. The children are healthy, and we have time to work to feed our families. I am also happy; children born now start running around from the age of 3. This makes us very happy. We thank God and we thank StopPalu+ too. (FGD, Pillimini) There is a significant demand for LLINs and SPC in the communities. This means that people have understood the importance of the activities implemented within the population as part of malaria prevention. To my understanding, the communities are very satisfied, and they adhere to the activities that are carried out in the field. (KII, NGO) 4.2. WHAT ARE THE SUSTAINABILITY FACTORS THAT ARE EVIDENT? Key Finding 1.2. Government ownership, community engagement, public-private partnerships, behavioral change communication (BCC), the establishment of an entomology system (insectarium and laboratory in collaboration with UGANC), and a stable, backed-up health management information system are critical factors to ensure the sustainability of the activity in Guinea. 4.2.a. What are the main factors and who are the main actors that could influence the sustainability of the malaria program in Guinea that have facilitated or not the implementation of the project? USAID defines sustainability as “the capacity to maintain program services at a level that will provide ongoing prevention and treatment for a health problem after termination of major financial, managerial and technical assistance from an external donor.”6 Respondents identified several factors and/or actors that facilitated (or did not facilitate) implementation of the activity in Guinea. The most reported factors were: 6 LaPelle NR, Zapka J, Ockene JK. Sustainability of public health programs: the example of tobacco treatment services in Massachusetts. Am J Public Health. 2006;96(8):1363–9. GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 36 1. The government’s ownership. Participants are concerned that the lack of ownership from the Government of Guinea (Minister of Health) would jeopardize the gains of the StopPalu+ activity. One key informant stated: We all know that a project has a beginning and an end, so to capitalize on the achievements, the government must move through the Ministry of Health to take its responsibilities and become more involved in the sustainability of this project. Based on my experiences, I can make recommendations for sustainability. For this, the government must assume more responsibility. Especially for the SMC campaign, get involved in purchasing commodities to have more sovereignty. (KII, USAID) The low contribution of the Minister of Health in the fight against malaria is visible ... if you take the interventions carried out in the field, the partners take care of everything. Therefore, this makes it difficult to sustain because the day the partner decides to leave, all your achievements will be reduced to nothing. (KII, IP) 2. Community involvement. There is no sustainability without community engagement. The local community engagement through RECO training and the implantation of CAGs is critical to ensuring sustainability. Through training and supervision, CHWs have gained skills in malaria control activities that enable them to successfully carry out the activities in the community even after the life of the activity. The ability of CAGs to organize community dialogues and find solutions to community priority issues will survive the StopPalu+ activity. People no longer travel to treat simple malaria from one village to another or from one village to the health post; these are villages that are very far from each other, but today the RECOs are there to facilitate this; to manage simple malaria in the households even during home visits. And they are really raising awareness that people understand today. (KII, NGO) If you see today StopPalu+, it has built the capacity of all the members of the NGO, the staff, the community workers that we supervise. You will notice that there are no doctors among us, especially at the coordination level, but the knowledge we have in managing malaria cases; it is a long-term experience. (KII, NGO) The RECOs mobilize us in our communities and sensitize us on how to behave when we get pregnant, they explain the whole procedure to avoid premature deliveries, childbirth, and low birth-weight children. As far as children are concerned, we have what is called [Artemisinin-based combination therapies], since I started going to the health center to get these products, I am really satisfied because my children no longer fall ill. (FGD, CSO) 3. The behavioral change in communication strategies. StopPalu+ helped develop several communication strategies that have increased people’s knowledge of and behavior about malaria. Almost anywhere in StopPalu+-supported areas, people know how to protect themselves against malaria. One KI was amazed to see the impact of these strategies on people. “It is amazing to hear the speeches of the former prefect of Labé. He knows everything about malaria.” This knowledge will survive the StopPalu+ activity. The project has done an excellent job in this regard. Before in our communities, people thought that malaria is a disease of the devil or of birds; or of witch doctors who create conflicts between neighbors. GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 37 Now with the awareness, people have understood that malaria is caused by a mosquito and they have understood the barriers, the prevention measures. Now they are starting to use them effectively in the households and the incidence of malaria is decreasing in our communities. The campaign you see is very much in demand by the community, especially when the rainy season comes. It is very much appreciated because it has reduced the infant mortality rate in the households. (KII, NGO) 4. The implantation of an insectarium and laboratory is an important factor for sustainability in Guinea. This activity is a big catalyst. Many people are being trained in Benin, even in the U.S. to staff these insectary and laboratory. Upon return home, these experts will be able to conduct genetic analysis at the center. The training of entomologists at the peripheral level in the country will address the dependency on the three currently at the central level. 5. Private sector engagement. In many developing countries, more than half of first encounters with the health system by people with a new illness are through the private sector. The private sector should be engaged to improve the quality of care for malaria. In addition, mobilization of the private sector in a public-private partnership could generate more funds for the malaria control program. One KI expressed this idea as follows: One major challenge for the project is the mobilization of financial resources to purchase commodities (antimalarial drugs, RDTs, LLINs) especially as we know that malaria care services are offered for free. The government must mobilize resources at all levels, including the private sector and NGOs, to ensure the availability of those commodities because the project will be of no use without these commodities. (KII, MOH) 6. A strong health management information system and monitoring and evaluation system. A strong health management information system is essential to ensure sustainability. Access to the right data and information will enable the program managers to make the right decisions in managing programs. During FY 2018, the activity monitoring, evaluation, and learning (MEL) team focused on improving activity data management during the first year of the award. The team improved activity data management by establishing an internal instance of the District Health Information System 2 (DHIS2) as the activity database. Using the internal DHIS2 gave the activity a central storage location for all routine activity data related to malaria, enabled interactive data visualization to be used during learning activities, and links the activity directly to the National Health Management Information System to ensure that the activity and the NMCP are using the same data for reporting and are working together to improve national malaria data completeness and quality. StopPalu+ participated in quarterly results-based management meetings at the regional and national levels, providing financial and logistics assistance to support the NMCP and the DPS in analyzing malaria and commodities data, identifying gaps and challenges such as low internal supervision and stockouts, and ensuring that district representatives determined corrective actions to address gaps and challenges. GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 38 4.2.b. To what extent beneficiaries at the central, facility, and community levels have taken ownership of the project approach? Health and administrative authorities (MOH/NMCP) took ownership of the activity approach to a large extent. Among the 64 individual interviews conducted, 24 were with administrative and/or health authorities. During the interviews, they described in detail their involvement in the conception and implementation of the activity. The NMCP was deeply involved in planning the different activities. For example, to ensure a smooth mass distribution of LLIN, the MOH/NMCP established National Coordination Committee (NCC), regional committees as well as prefectural committees in collaboration with StopPalu+. The NCC assisted with the development of training materials, micro-plans, and data collection tools, as described below. The training on micro-planning reinforced participants’ capacities to plan a successful campaign. They learned household calculation methods based on the size of the population as well as the number of LLINs needed, the amount of storage space required, the transport of LLINs, the number of distribution sites based on the number of households per site, and the criteria for identifying a distribution site. Figure 9: Number of Quarterly Coordination Meetings summarizes the number of coordination meetings held under the NMCP’s leadership by fiscal year. The NMCP held at least three coordination meetings per year. Figure 9: Number of Quarterly Coordination Meetings Held Under NMCP’s Leadership 2 3 4 4 0 1 2 3 4 FY 18 FY19 FY 20 FY21 Source: StopPalu+ Annual Reports The ownership of the StopPalu+ approach at the health facility level is appreciable. Among the 64 individual interviews conducted, 14 were with heads of health facilities. They unanimously (14/14) agreed that the activity provided them with remarkable support. Through training received, they are appropriating the components of the activity. To improve the quality of malaria diagnosis in health facilities, the activity conducted a training-of￾trainers for 12 malaria diagnosis trainers. Among those, four experts were identified who also helped GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 39 train 74 laboratory technicians on malaria diagnosis and use of rapid diagnostic testing. The activity conducted post-training supervision in the laboratory that benefited from these trainings in Kindia and Conakry. Figure 10: Change in Parasite Detection Among Lab Technicians shows a significant increase in their performance between 2018 and 2019. It is important to highlight the External Competency Assessment for Malaria Microscopists (ECAMM), a WHO-accredited training and test-based certification organized by NMCP in collaboration with StopPalu+ and the WHO, toward certification of microscopists in Guinea. Based on WHO standards, by 2021,12 training participants qualified to participate in the ECAMM, (previously implemented only in Kenya and Senegal) in accredited reference laboratories supervised by the WHO. Of the 12 participants in the Guinea microscopist training, 10 achieved Level 1 and 2 certifications (Level 1 being higher) based on test scores). Figure 10: Change in Parasite Detection Among Lab Technicians in Conakry and Kindia Source: StopPalu+ Annual Report FY 2019 At the community level, 12 of the 14 health center managers who were interviewed, as well as community members who participated in the 14 FGDs, indicated that they have taken ownership of the activity approach to a large extent. To increase community involvement and support for malaria prevention and care, StopPalu+, with the support of the national social behavior change communication (SBCC) consultant, implemented the national SBCC and Capacity Strengthening Plans, including community dialogs carried out by CAGs, training of school and religious leaders, and implementation of the School and Religious Leaders against Malaria. Figure 11: Number of Community Dialogs Organized by CAGs shows that the number of community dialogs increased significantly from 81 in 2018 to 567 in 2021. GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 40 Figure 11: Number of Community Dialogs Organized by CAGs 81 263 488 567 0 258 478 603 0 500 1000 FY 18 FY19 FY 20 FY21 Community Ownership Source: StopPalu+ Annual Reports # of community dialogs organized by the community about malaria control The number of dialogs organized in communities by the CAGS in different StopPalu+ areas of intervention are presented by prefecture (Figure 12). Throughout StopPalu+ implementation, ownership through the CAGs at the community level is an area of continuous improvement, as seen in the increased community engagement through dialogs. Figure 12: Number of Community Dialogs Organized by CAGs by Prefecture 0 11 7 47 21 54 95 128 29 102 177 197 31 93 209 194 0 50 100 150 200 250 2018 2019 2020 2021 Source: StopPalu+ Annual Reports Conakry Kindia Boke Labe GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 41 4.3. TO WHAT EXTENT HAS THE CAPACITY OF THE NMCP TO LEAD, COORDINATE, AND MANAGE MALARIA PREVENTION AND CONTROL ACTIVITIES BEEN STRENGTHENED AS A RESULT OF STOPPALU+ EFFORTS? Key Finding 2. The capacity of the NMCP to lead, coordinate, and manage malaria prevention and control activities has been strengthened to a large extent as a result of StopPalu+ efforts. 4.3.a. What support has the project provided to the NMCP in achieving its objectives of leading, coordinating and managing malaria prevention and treatment activities? The activity provided a range of support to the NMCP in achieving its objectives of leading, coordinating, and managing malaria prevention and treatment activities. StopPalu+ supported the NMCP coordination capacity; strengthened M&E surveillance and research capacity; and strengthened the NMCP’s vector control capacity in obtaining information on species, composition, abundance, physiological status, and vector-biting behavior. The section below describes these activities. 1. StopPalu+ supported the NMCP coordination capacity by offering the following support: 1.1. Facilitating technical working groups’ (TWGs’) monthly meetings. The activity team facilitated the NMCP Behavior BCC, Vector Control, and Diagnosis and Case Management TWGs’ monthly meetings. The activity team also participated in the Supply Chain Management TWG. Several points were discussed during the TWG monthly meetings throughout the life of the activity, such as the orientation of health providers on the new malaria prevention and treatment protocols in the COVID-19 context, the production and distribution of the new protocols, and the SBCC materials developed for the COVID-19 context. 2.1. Conducting malariology course. To strengthen the skills of the districts and the activity team regarding implementation of malaria control activities, StopPalu+ collaborated with the NMCP and WHO to organize a malariology course for the 19 malaria focal points from PMI-supported prefectures, the four activity regional coordinators, the activity medical advisor in Kindia, and the doctor in charge of training in the prefecture of Dinguiraye. 3.1. Supporting the participation of NMCP/MOH officials in international meetings. StopPalu+ supported the participation of the National Director of Disease Control to attend the annual American Society of Tropical Medicine and Hygiene conference in Seattle in 2022. 4.1. Implementing the community champion approach. 5.1. Reinforcing the integration of private health facilities. 2. StopPalu+ supported M&E surveillance and research by: 1.1. Improving M&E data quality and use at the national level and among statistical analysts at the regional and district levels. To ensure the quality of the data transmitted by the health facilities, the activity regional team and the malaria focal points conducted data quality reviews in different health facilities. The results of these reviews in each district were GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 42 presented during the monthly monitoring meetings. Data quality assessments were conducted and health facilities with discrepancies reports and primary tools were asked to explain the discrepancies and commit to avoiding them in future months. These monthly data review meetings have improved the quality of data reported by the health facilities and entered into the national DHIS2. 2.1. Conducting periodic health facility and community supervision visits in areas with high incidences of malaria. During FY 2020 and FY 2021 (for which data are available), the activity conducted community visits in health facilities with high malaria incidences in the four regions supported by the activity. A total of 18,592 and 32,912 visits, respectively, were conducted in the two years. The post-visit surveys for both years indicate that, over 70 percent of the time, RECOs are well-known by households in the communities they serve, and these households benefited from RECOs’ services in the months preceding the survey (Figure 13). Results of these supervision visits, including the performance of health facilities at different levels and RECOs in communities, were presented at monthly meetings where NMCP, DPS, and Direction Communale de la Santé (DCS) representatives were present. Figure 13: Percentage of Households That Know the RECO and Percentage of Households that Received a RECO Visit During the Previous Month (FY 2021) Source: StopPalu+ Annual Reports 3. StopPalu+ strengthened the NMCP’s vector control capacity in obtaining information on species, composition, abundance, physiological status, and vector-biting behavior. StopPalu+ supported NMCP to improve entomology surveillance capabilities and capacities by establishing an insectarium (breeding and holding cages) and associated laboratory (entomological surveillance) at UGANC; building a rabbit enclosure to support mosquito production as needed; breeding Kisumu strain mosquitoes for multiple generations; identifying priority, secondary, and seasonal sentinel sites (based on malaria prevalence) in Labé, Kankan, Kissidougou, Siguiri, Lola, Maférinya, Boké, and Faranah, where mosquito collection was performed to facilitate national entomological surveillance; GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 43 and developing a National Vector Control Strategy (2018–2022), including an insecticide management plan to address the spread of insecticide-resistant vectors. An insectarium has been set up at the University of Conakry, and everyone knows it; people are recruited and work there. These entomologists are deployed at the level of the sentinel sites in the country's countryside. They collect and bring back entomology information. … The installation of focal points everywhere in their zones, the execution or implementation of activities with NGOs by transferring skills is critical. So, when they are not there, we just have to activate this system by putting the means at their disposal. (KII, NMCP) About capacity building, for us at NMCP, StopPalu+ strengthened our capacities outside the country. Example: for prevention activities (vector control or entomology), every year entomologists are trained in Benin under StopPalu+ funding, other partners also intervene but StopPalu+ does more. (KII, NMCP) Since the establishment of the entomology laboratory and insectary at UGANC, StopPalu+ has continued supporting the operation of the laboratory and insectary, providing needed equipment, supporting efforts to maintain a susceptible Anopheles gambiae s.s. population (Kisumu strain), holding meetings organized by the staff of the NMCP Vector Control Unit’s Vector Control TWG, and training NMCP staff and student interns in all mosquito breeding techniques and insectary maintenance. StopPalu+ supported the NMCP in developing and implementing the National Insecticide Resistance Management Plan to propose vector control strategies to limit the expansion of vector insecticide resistance. The number of mosquitoes generated by the insectary between 2017 and 2020 is presented in Figure 14: Number of Mosquito Generations by Year. In addition to the routine phenotypic and genotypic resistance tests on mosquito strains, in 2018 StopPalu+ evaluated the resistance status of malaria vectors in six sites (Boké, Labé, Maférinya, Faranah, Kankan, and Kissidougou). An. gambiae s.l. was resistant to DDT, permethrin, alpha-cypermethrin, and lambda-cyhalothrin, and the Kdr (knockdown resistance) West mutation was widespread. This enabled the introduction of dual active ingredient LLINs (chlorfenapyr-alpha-cypermethrin and deltamethrin-pyriproxyfen) in selected sub￾prefectures to mitigate the threat of widespread pyrethroid resistance and sustain malaria reduction gains. GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 44 Figure 14: Number of Mosquito Generations by Year 14 13 7 17 0 2 4 6 8 10 12 14 16 18 2017 2018 2019 2020 Source: Annual report of Entomological Surveillance Activities StopPalu+ routinely conducted entomological surveillance for seasonality monitoring at the sentinel sites till date, except during the COVID-19 pandemic. Different species of mosquitoes were frequently collected indoors and outdoors from the sentinel sites for both insectary and laboratory using human landing collection, Centers for Disease Control light traps, and pyrethrum spray collection. 4.3.b. To what extent have the capacity building activities of the Activity improved the overall management of the NMCP? StopPalu+’s capacity building activities contributed significantly to the improvement of the overall management of the NMCP. Although the list of activities is not exhaustive, the following are some of the activities that the NMCP is implementing with the activity’s support: ● Coordinating meetings with TWGs to discuss various topics including LLIN mass campaign in the COVID-19 context and the digitalization of the enumeration and distribution phases of the campaign. ● Improving M&E data quality review and use at the national level and among statistical analysts at the regional and district levels. ● Organizing the national supervision system to supervise malaria activities at both the health facility and community levels. ● Conducting periodic health facility and community visits in areas with high incidences of malaria. ● Strengthening the NMCP’s vector control capacity. ● In the context of strengthening malaria control, training entomologists and lab technicians. Figure 16: Improvement in Parasite Identification Among Lab Technicians shows the improvement of diagnostic skills for reading slides for parasite identification among lab technicians. From FY 2018 through FY 2021, NMCP was supported by StopPalu+ to conduct malaria diagnosis supervision. A case in point is the 2020 supervision exercise conducted in certain facilities in Conakry GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 45 and Kindia regions to assess the level of implementation of recommendations from previous supervision visits. Activities during these supervision visits included testing technicians’ diagnostic skills for reading slides from both facilities (internal quality control) and WHO-validated slides (external quality control). For internal quality control, the supervision team checks slides read by the technicians and rates the agreement (concordance) between the two readings. For external quality control, technicians read slides provided by WHO, and their responses are checked against the WHO-validated results. This is in addition to the routine training of health workers, including laboratory technicians, each year (Figure 15) by the activity to support NMCP. Different cadres of health workers were trained in malaria diagnostics, including laboratory technicians, especially those that eventually staffed the insectarium and mosquito breeding laboratories. Figure 15: Total Number of Health Workers and Laboratory Technicians Trained in Malaria Laboratory Diagnostics 1171 1345 1578 1674 86 72 58 0 0 200 400 600 800 1000 1200 1400 1600 1800 2018 2019 2020 2021 Source: StopPalu+ Annual Reports Health workers Lab Technicians Supervision activities usually involve hundreds of staff, mostly female, and facilities at different levels of the health system— from university hospitals to communal medical centers and health centers. Feedback from the activity provides information regarding the status of knowledge and skills from training. For instance, in the Conakry and Kindia locations noted above for supervision visits, feedback indicates that participants improved their performance in species identification in both regions (Figure 15), with an average performance of 31.25 percent in Quarter 4 of FY 2018 increasing to 57 percent in Quarter 1 of FY 2019. GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 46 Figure 16: Improvement in Parasite Identification Among Lab Technicians in Conakry and Kindia Source: StopPalu+—FY 2019 Annual Report The improvement in laboratory technicians’ diagnostic skills assists health care providers in making correct therapeutic decisions and therefore improves the quality of care at health facilities. The proportion of malaria cases treated has increased from 94 percent in 2018 to 96 percent in 2021. There have also been advances in detection, identification, and classification of plasmodial species for better diagnosis. These are just some of the results of our efforts in the field. (KII, Conakry) Almost all patients with suspected malaria received a diagnostic test either by microscopy or RDT (99 percent, 99 percent, 98 percent, and 98 percent in 2018, 2019, 2020, and 2021, respectively), according to StopPalu+ health facility records. Relatedly, accurate interpretation of tests (positive or negative) also increased (Figure 17), which may mean that training was achieving its outcomes and improving the expertise of health workers. Figure 17: Proportion of Diagnostic Tests Interpreted Correctly (Positive￾Negative) 65% 87.85% 90.52% 95.38% 0% 20% 40% 60% 80% 100% 120% 2018 2019 2020 2021 Source: StopPalu+ Annual Reports GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 47 Although the activity’s capacity building activities, such as training, development of strategy documents, and joint supervision visits, contributed significantly to the improvement of the overall management of the NMCP, there are a few areas that require attention. The recurrent loss/replacement of trained staff at different levels in the MOH/NMCP because of redeployment, regime change, or low remuneration present potential challenges. When there is redeployment or a regime change, for instance, new government appointees may not always share the organization’s vision or priority, let alone understand the policy in place. At the district and community levels, many people are leaving their posts for reasons such as those stated above. Additional training will be required to bring the replacements up to speed. However, it is not realistic to think that funds will always be available to organize new trainings to accommodate newcomers. These facts could explain, in part, the inability of the NMCP leadership to effectively coordinate activities between PMI and GFATM leading supposedly to recurrent stockouts of commodities at health facilities. A KI acknowledged the coordination challenges and discussed its potential impact on the program: I think there are challenges. Those that are internal to the project, we don’t know them. However, challenges that affect the collaboration and the implementation can be detected. The first challenge is the lack of communication that we have. This gap led to poor performance in the year 2020 for StopPalu+ where they had no commodities for the 4th round of the [Chimioprophylaxie seasonale— SMC]. The communication was not well handled and yet there were commodities in the Global Fund area. Because we are using a common basket, the information must circulate to compensate for the lack of commodity. So, a large population was not served, namely children from 3 to 59 months. This affected the performance of the indicators. (KII, NGO) This specific challenge identified by the KI above is incongruent with StopPalu+ and PMI records, which indicate that the SMC campaigns were implemented as planned and with adequate doses of SMC available. A possible gap in communication regarding collaboration between partners may require further study. An NGO partner cannot be reporting challenges, including lack of commodities, while the activity is reporting no challenges and adequate commodities. 4.4. WHAT KIND (IF ANY) OF POSITIVE AND NEGATIVE KNOCK-ON EFFECTS HAVE THE STOPPALU+-SUPPORTED INTERVENTIONS HAD ON THE WIDER HEALTH SYSTEMS IN GUINEA? The complex nature of the StopPalu+ activity meant that there could have been various challenges with implementation and adaptation. However, this does not appear to be the case, as StopPalu+ interventions are widespread, accepted, and owned in activity target regions and communities. Key Finding 3. The activity implemented several BCC strategies to help people in Guinea adopt healthy behaviors about malaria prevention, including uptake of vector-control measures, use of preventive treatments, and care-seeking behavior. These are positive effects of the activity. StopPalu+ reached very many and a steadily increasing number of community members between 2018 and 2021 with SBCC (Figure 18), especially women. Communication and collaboration at the community level enhances engagement and transparency, and strengthens ownership—which are positive knock-on effects of StopPalu+. GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 48 Figure 18: Number of People Reached by NGOs/Community-Based Organizations/CHWs with Malaria Prevention and Control SBCC by Sex 609662 1090178 1301823 1409520 816443 1331658 1769850 1787212 0 500000 1000000 1500000 2000000 2500000 3000000 3500000 2018 2019 2020 2021 Source: StopPalu+ Annual Reports Male Female In the same vein, every year since 2018, the community dialogs initiated by StopPalu+ and in collaboration with CAGs to bring community members together to identify and solve their peculiar problems were conducted with increased frequency (Figure 19) in the four focus regions (except Conakry, where there was a dip in 2020, maybe as a result of the COVID-19 pandemic). Nonetheless, the community dialogs have been positive knock-on effects of the StopPalu+ interventions. Figure 19: Number of Community Dialogs Organized by the Community About Malaria Control 0 11 7 47 21 54 95 128 29 102 177 197 31 93 209 194 0 50 100 150 200 250 2018 2019 2020 2021 Source: StopPalu+ Annual Reports Conakry Kindia Boke Labe GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 49 4.4.a. What are the communication channels used to reach the project’s target populations (how will you rank those channels from your preference point of view)? StopPalu+ used a variety of communication channels to reach the activity’s target population. These communication channels include: Radio and television. The activity used private radio stations as well as community or rural radio stations and television to communicate health education messages. CHWs visited homes to educate children and their parents. The IPC component was powerful in changing people’s behavior. Community action groups organized meetings and community dialogs to determine and address priority community issues. Mobile clinics (supplies, health workers, managers, etc.) were dispatched to areas where peak malaria cases were reported to treat and distribute LLINs at no charge to those communities. Mobile clinics were very successful in the community. Trained school children and religious leaders against malaria. This strategy consists of pairing malaria awareness with education in schools. If all social groups are included, students’ dissemination of malaria prevention, messages would be beneficial for everyone. Once a group of students is trained, they would raise awareness among their peers in schools who would in turn disseminate information to their families and neighbors. Imams (religious leaders) trained in the PMI-supported zones passed malaria prevention messages during the religious worships. Supervision visits were opportunities to convey activity messages. Flyers were hung on walls at health facilities to educate patients. One key informant explained the activity’s reliance on BCC in these terms: In terms of communication, for example, I would say that StopPalu+ is the only project for the moment to have a clear strategy adapted to the field. These are strategies by and for the community. I’ll just take the example of community dialogue, it’s only in the StopPalu+ zone where community dialogues are held (where a problem is identified by the community itself in the fight against malaria, this problem is identified by this same community and resolved by itself). KII, NMCP) As noted earlier, between FY 2020 and FY 2021, when data were available, the activity conducted a total of 51,504 community visits to health facilities and surrounding locations with the highest malaria incidences in the four regions supported by the activity. During these visits, rapid surveys were conducted to assess households’ knowledge of malaria, the number of households visited by a CHW, and the channels of communication preferred by respondents to receive information on malaria. The highest ranking channel of communication on malaria among community members is the CHWs (in this case, RECOs), followed by radio, and then NGO facilitators (Figure 20). Religious leaders and CAGs have also played important roles in raising awareness in communities. GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 50 Figure 20: Proportion of Malaria Channels Reported by Household Members in Selected Regions 49% 26% 10% 48% 23% 30% 7% 23% 6% 46% 23% 11% 58% 16% 25% 5% 22% 4% 47% 28% 18% 48% 12% 22% 6% 18% 5% 45% 22% 14% 53% 11% 21% 4% 18% 4% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Radio TV Religious RECO MOSO/COSAH Animators GAC Health care providers Others Oct-Dec 20 Jan-Mar 21 Apr-Jun 21 Jul-Sept 21 Source: StopPalu+ Annual Report 4.4.b. What are the effects derived from project activities that have enabled the strengthening or not of the health systems in Guinea? USAID defines health systems strengthening as strategies, responses, and activities designed to sustainably improve country health system performance. The activity was instrumental in building the capacity of many health care providers, laboratory technicians, and CHWs to improve the quality of services through training and refresher trainings, on-the-job capacity support, supervision visits, revision of training manuals and protocols, collaboration to develop national strategy documents, and SBCC implementation. Figure 21 is a compilation of some training targets, types, and numbers over the four fiscal years of the StopPalu+ activity. GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 51 Figure 21: Number of CHWs Trained in Malaria Case Management 1432 1171 1085 879 795 1,345 1273 1884 1,546 1,578 1,320 1456 1,883 1620 1903 1683 0 500 1000 1500 2000 # of health workers trained in IPTp with USG funds* # of health workers trained in malaria laboratory diagnostics with USG funds* # of health workers trained in case management with ACT with USG funds* # of members of local NGOs, CBOs who have received training in malaria control activities, monitoring the use of public services in malaria-related services, and advocacy with project support in the last reporting period Capacity Strengthening Source: StopPalu+ Annual Report FY21 FY 20 FY19 FY 18 To address the evaluation question regarding the effects derived from activities that strengthened Guinea health systems, the evaluation team relied on an article by Hatt et al., (2015).7 The authors described the following effects of health system strengthening: Improved service provision quality, increased financial protection, increased service utilization, and uptake of healthy behaviors. This section focuses on one of these four effects: Improved service provision quality. Hospitals, Central Medical Stores (CMCs), and advanced health centers are referral facilities with the technical capacity to ensure the correct management of all forms of malaria. StopPalu+ organized joint supervision visits focused on the diagnosis and case management to assess the quality of the services of these facilities and strengthen the quality of interventions for malaria diagnosis and treatment. Figure 22 shows malaria case management performance in hospitals/CMCs in PMI regions (Boké, Kindia, Labé, and Conakry) in as reported by StopPalu+ for FY 2022. The results show that all the regions exceeded the 85 percent overall performance required during the evaluation, with overall performance of 92 percent in the PMI zones. Provider-patient interaction and simple malaria case management were highest at 93 percent and 97 percent, respectively. The Labé region had the best overall performance at 100 percent, followed by Conakry (91 percent) and then Kindia (89 percent). During supervision visits, 7 Hatt, Laurel, Ben Johns, Catherine Connor, Megan Meline, Matt Kukla, and Kaelan Moat. June 2015. Impact of Health Systems Strengthening on Health. Bethesda, MD: Health Finance and Governance Project, Abt Associates. GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 52 the weaknesses observed were shared with the teams and measures were taken to improve facilities with low performance. Figure 22: Malaria Case Management Performance in Hospitals/CMCs in PMI Regions in Q2 FY2022 Source: StopPalu+—FY 2022 Annual Report The following testimonies from respondents support StopPalu+ capacity strengthening interventions: The project’s capacity-building activities helped NMCP above and beyond expectations. (KII, NMCP) The project has helped NMCP to review the malaria national policies, update the training documents, the normative documents of the NMCP and if there is something that also shows the project’s support, it is the thematic group, which is one of the few groups that work with the NMCP. (KII, NGO) The health district is the implementation point of the government's health policy. For this to happen, the actors, the district management team must understand the directives very well and the fact that the capacity of members of the district management team have been reinforced on the new management directives allows for the correct implementation of activities in the field. The core team members are involved in the planning, mobilization, and implementation of activities. The holding of quarterly reviews with all core team members to share data related to trends in performance indicators; I think this is an important factor that needs to be highlighted. (KII, Koubia) The project’s collaboration with the NMCP team contributed to strengthening the experience between the partners at the managerial level. (KII, NGO) Capacity building in diagnosis and management: with the deployment of young doctors and nurses after Ebola, the project carried out training and supervision activities for these personnel. One of the successes is especially around diagnosis. In our zone in 2018, there was no level 1 expert in diagnosis but to date we have 3 ISO certified experts [World Health Organization]. (KII, Boké) GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 53 4.5. TO WHAT EXTENT HAS THE STOPPALU+ PROJECT ADDRESSED GENDER ISSUES IN ACCESS AND UTILIZATION OF MALARIA SERVICES? Key Finding 4. Promotion of gender is one of StopPalu+’s priorities. The activity promoted equity and inclusiveness at all levels of the activity. Gender issues are given priority in all StopPalu+ activities. 4.5.a. To what extent has the promotion of gender equality been integrated into the project activities? The activity addressed gender issues in access to and utilization of malaria services to a large extent through varied activities that the activity undertook to achieve this objective: StopPalu+ deliberately included training in staff orientation about how to integrate gender into all project activities to emphasize the importance of the issue and allow activity staff to acquire the necessary skills to promote access to and utilization of malaria services among women. StopPalu+ implemented varied BCC strategies to promote the adoption of healthy behaviors and create demand for malaria services among men and women. Health providers and CHWs (male and female) were trained in IPC and quality care delivery with emphasis on gender. In line with the national policy, the activity provided malaria prevention services free of charge, removing the cost barrier to access to care for people of low socio-economic status, including women. During the recruitment of personnel, priority was given to women when possible. As a result, women are well represented within StopPalu+ activity and partners’ staff. Women were not only the recipients of services (LLIN and IPTp), but they were also providers of these services. “Seeing women on motorcycles delivering services sent a strong message to the community,” said one KI. “They were very effective.” Although there are inadequate quantitative data to support the gender reports of StopPalu+ activities, KIs and focus group discussants discussed results from the activity’s gender interventions. Some corroborative statements include the following: Yes, I think that in the whole process there is the involvement of women. If we take the staff at the national level; there are many women who occupy high positions. In the implementation, the RECOs of the female gender are very involved. We also have women’s associations that are involved. I believe that all components are involved. (KII, USAID) We have integrated a training module on gender/equity in our training tools and it has been validated by the Ministry. During each training module, we remind people of the inequities of the most vulnerable (disabled, visually impaired, pregnant women and children under one year old...) and we often organize mobile clinics on the islands and on the mainland to reach the needy who cannot take care of themselves. (KII, Conakry) The project involves women, for example in advocacy we know that women are one of the most vulnerable groups. And to this effect, women are really involved in mobilizing and sensitizing their peers to promote the project's activities. (KII, NMCP) GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 54 Yes, first of all, at the level of gender, the project takes a lot of account of gender because even when you notice during our community activities, when you see the number of community relays that are used by the center, there are either more women than men or somewhere you will see an equal number of shares. (KII, Labé) Another important finding is that the StopPalu+ activity promotes equity and inclusiveness at all levels of the activity. At Boké, StopPalu+ carried out a special activity to increase the uptake of LLIN among fishermen. The project takes into account the poor and people who have no means. (FGD, Koubia) Officially, prenatal care and delivery are free of charge, including malaria treatment for pregnant women and children without distinction. As far as the implementation of the project is concerned, we work a lot with women (midwives and [technical health agents ATS] in ANC and delivery activities. But men are also present in the laboratory team, nurses ... so the proportion is qualitative and not quantitative in the implementation of project activities. (KII, NGO) 4.5.b. To what extent has the project improved the factors associated with intermittent treatment in pregnant women? The activity significantly improved the factors associated with intermittent treatment of malaria in pregnant women. StopPalu+ increased women and family members’ knowledge on IPTp increased availability of SP and ancillary supplies and improved CHWs’ and health facility providers’ technical and interpersonal skills in effective administration of IPTp. The section below describes some of these activities. StopPalu+ increased awareness of women and family members of the benefits of ANC visits, including IPTp through SBCC activities to promote ANC and IPTp. The activity team increased women’s and men’s awareness primarily through the community dialogs conducted regularly (see Figure 19). Facilitators educated participants on the importance of IPTp for preventing malaria in both mother and baby using clear and strong messages about the “no charge” aspect of antimalarial commodities and services for both health providers and the population. Providing malaria services free of charge will encourage people to consult a health center as soon as they are sick. The activity increased the availability of SP and ancillary supplies. The activity team collaborated with the Procurement and Supply Management activity on macro-planning and stock management. StopPalu+ continued to work closely with the Procurement and Supply Management activity, district malaria focal points, and the NMCP team to monitor the supply of commodities at health facilities. This increased availability may have enabled pregnant women to complete the recommended dosage of at least three SP doses during the life of any pregnancy (see Figure 23). Although there was a dip in 2019, more pregnant women are completing three doses of SP during their pregnancies from year to year. Figure 23: Proportion of Pregnant Women Who Received at Least Three Doses of SP During Their Last Pregnancy (Routine) Source: StopPalu+ Annual Reports 100% 90% 67.02% 68% 79% 60% 70% 80% 50% 57% 40% 30% 20% 10% 0% FY 18 FY19 FY 20 FY21 The activity trained health workers on IPTp. Figure 24 shows an increase in the number of health workers trained in IPTp from 2018 to 2021. The health workers training module (1) explained the importance of IPTp in preventing malaria for pregnant women and their babies, (2) explained IPTp administration protocols (during which stage of pregnancy to begin, the number of tablets per dose, where pregnant women can receive IPTp, etc.), (3) shared key message on IPTp promotion, and (4) explained the roles of health workers in improving IPTp uptake (promotion during home visits, referral). Figure 24: Number of Health Workers Trained in IPTp from 2018 to 2021 1432 795 1386 1883 0 500 1000 1500 2000 2018 2019 2020 2021 Source: StopPalu+ Annual Reports The activity improved providers’ supervision and tracking of pregnant women. StopPalu+ routinely conducted supervision activities throughout activity implementation and supported the DPS and DCS to conduct their bimonthly supervision visits. These supervision visits integrated all malaria control activities, including IPTp. The activity supported provider outreach services to improve access to IPTp services rolled out. StopPalu+ supported the health centers to conduct monthly outreach services to provide health GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 55 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 56 services such as IPTp and EPI to eligible patients, test all cases of fever, treat positive simple malaria cases, and refer negative or severe cases to the necessary higher health facility. This support is provided during training and during supervision visits, and the data collected during these activities are recorded in the health facilities’ data records. Several KIs confirmed that the activity made significant improvement in the uptake of IPTp among pregnant women. Prevention: starting with the use of LLINs, the desired coverage was 85 percent among pregnant women, but we were below it at 60 percent before our implementation. So, our activities led to the following results: Mass distribution campaign every 3 years and in 2019, also routinely. Since the first campaign and in routine we have a clear improvement from year to year until 2021 of availability and distribution of LLINs especially among pregnant women and children (under 5 years). Very clear improvement of IPTp in our region from a desired coverage of 60 percent. We have gone beyond this coverage through an innovation called MELA (Monitoring Evaluation Learning in Adapting). (KII, Conakry) We have specific activities that are oriented towards women. When we talk about routine LLINs, it's for pregnant women, IPTp are also activities aimed at women. In this area we have made a lot of effort because even to improve attendance at health centers for women, we have used several innovative approaches: namely SMS reminders and tracking women in the registers. During the monthly meetings, we try to identify women who have not come for their consultation, then we call them on the phone and ask the RÉCO to go and get them. These two approaches have allowed us to improve women's use of ANC services. (KII, RTI) IPTp for pregnant women is an important element in the fight against malaria. This was made possible thanks to the activities of the social mobilizers through household medical visits and supervision…These medical visits made it possible to raise all the health indicators: whether ANC1, ANC2, or ANC3; but also, the use of malaria prevention in pregnant women as well as in children. …. These medical visits allowed the project to achieve good indicators for ANC and IPTp among women. (KII, Koubia) 4.5.c. To what extent has the project improved the factors associated with intermittent treatment in children under five? How are trends for girls compared with boys? StopPalu+ significantly increased caregivers’ knowledge about SMC, increased availability of SMC (SP+AQ), and improved CHWs’ and health facility providers’ technical and interpersonal skills in effective administration of SMC. StopPalu+ increased awareness of the community and family members about the need for infants and children under age 5 to be protected from malaria by: • Developing and implementing SBCC and outreach to increase community and family members’ awareness and knowledge of the importance of SMC to protect children under age 5 from malaria. • Strengthening community mobilization through advocacy meetings: StopPalu+ organized advocacy meetings in the districts that implemented the SMC campaign to inform community leaders and administrative and health authorities of the SMC campaign strategy and to obtain these leaders’ adherence to and appropriation of the strategy. GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 57 • Using mobile clinics to circulate in the neighborhoods and sectors of communities to inform and educate them on the importance of keeping children ages 3–59 months at home so they would benefit from the first dose of drugs. • Using village vigils. StopPalu+ regional team conducted village vigils in the rural communes of Kamaby (Gaoual district) and Sinthiourou Kebou (Koundara district) to encourage mothers and caregivers to have their children take SMC medicines due to the low coverage of children who received SP-AQ doses during the first two rounds of the 2021 SMC campaign. The activity increased availability of SMC (SP+AQ) and commodities for continuous and mass distribution during the high-transmission season in targeted districts by: • Organizing micro-planning workshops with the participation of stakeholders at the district, community, and facility levels to quantify the resources needed for a successful SMC campaign in each district. • Coordinating procurement and distribution of SMC commodities through participation in the various meetings of the NMCP supply chain TWG to discuss malaria commodities procurement, distribution, and use at all levels. • Digitalizing the SMC campaign to significantly reduce reliance on paper forms, ensure the collection of quality data, measure the impact of distribution in the community, and make decisions in real time. Available data indicate large numbers of SMC doses administered to eligible children underage 5 years in StopPalu+ activity locations (Figure 25). Increased availability of commodities leads to higher uptake. Figure 25: Number of SMC Doses (SP+AQ) Administered to Eligible Children 1,054,598 2,467,454 0 1,277,214 0 500,000 1,000,000 1,500,000 2,000,000 2,500,000 3,000,000 FY 18 FY19 FY 20 FY21 Source: StopPalu+ Annual Reports Data unavailable for FY 2020 The activity improved providers’ (facility- and community-based) technical and interpersonal skills in effective administration of SMC by: GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 58 • Organizing training-of-trainers, supervisors, mobilizers, facilitators, and distributors. StopPalu+ supported the NMCP to conduct cascade training-of-trainers at the regional, prefectural, and health center levels. StopPalu+ supported training of drug distributors on SMC strategy, eligibility criteria, key messaging, SP+AQ administration, pharmacovigilance, and program monitoring. • Supporting the multi-layer supervision of SMC campaigns, which consisted of different levels of supervisors: national, regional, prefectural, and health center supervision, and improved the quality of the administration of SMC medicines and compliance with guidelines related to the control of COVID-19 prevention. To improve the availability of SMC, the activity trained similar numbers of SMC distributing agents and social mobilizers before organizing the four rounds of SMC campaigns each year. For example, data available at the time of the evaluation indicate that 2,404 distributing agents and 486 social mobilizers previously selected by the community and trained, began distributing SMC drugs to children in the target age group. At the end of the third quarter of FY 2021, 2,202 distributing agents and 438 social mobilizers had been trained and distributed SMC drugs to children in the target age group (Figure 26). Figure 26: Number of Trained SMC Distributing Agents and Social Mobilizers 2404 2202 486 438 0 500 1000 1500 2000 2500 3000 2018 2021 Source: StopPalu+ Annual Reports Distribution agent Social mobilizer During StopPalu+, almost all eligible children received SMC, from 95 percent upwards (Figure 27). StopPalu+ contributions can be traced to the training of providers, distribution agents, and social mobilizers, who jointly improved the system of reaching children under age 5 with SMC. The only gap is the unavailability of data to assess SMC trends by sex (boys and girls). GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 59 Figure 27: Percentage of Children Under Age 5 Who Received SMC During the Different Cycles 95.5 98 99 0 96 99 99 99 95 99 99 0 98 100 101 99 0 20 40 60 80 100 120 % of children U5 who received SMC during the first cycle of the transmission season % of children U5 who received SMC during the second cycle of the transmission season % of children U5 who received SMC during the third cycle of the transmission season % of children U5 who received SMC during the fourth cycle of the transmission season Source: StopPalu+ Annual Reports 2018 2019 2020 2021 Data unavailable for 2018 and 2020 for fourth cycle SMC at the time of reporting. A key informant summed up the activity’s success in these words: There is a 100 percent demand for LLINs and SMC in the communities. This means that people have understood the importance of the activities implemented within the population as part of malaria prevention. To my understanding, the communities are very satisfied, and they adhere to the activities that are carried out in the field. (KII, NGO) GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 60 5. CONCLUSIONS The performance evaluation found that the StopPalu+ activity’s objectives of fighting malaria, the major cause of morbidity and mortality in Guinea, were relevant and that the activities were planned and implemented according to the guiding principles of the NMS and the National Health Development Plan. Health care providers and CHWs were trained to carry out MOH directives according to schedule and quality standards. Monthly meetings were held to evaluate the program indicators and make corrections for any gaps identified. Overall, the StopPalu+ project contributed to improvements in the availability, quality, and use of malaria care services at the community, facility, and district levels in Guinea to the satisfaction of activity stakeholders and beneficiaries. Use of LLINs increased, with more being distributed each year of the activity. The positive outcomes of the activity (such as high percentages of children under age 5 who received SMC) contributed to increased satisfaction among most stakeholders. Beneficiaries’ trust in and respect for the health care system were boosted, and more pregnant women were mobilized and informed by StopPalu+ to receive IPTp up to three times during their pregnancy. This, by multiplier, means that more women are visiting ANC clinics to get the SP doses. The advanced strategy mobilization, distribution, and community following implemented by StopPalu+ has benefits. The advent of COVID-19 and the inability of the NMCP leadership to effectively coordinate activities between the GFATM and PMI affected the implementation of programs. Furthermore, the low remuneration of CHWs in the project-supported areas compared to those in the GFATM areas affected their motivation to perform their daily tasks. To ensure the sustainability of the StopPalu+ activity, the Government of Guinea should proactively identify other funding streams to keep the activity running and possibly standardize and scale up the national community health strategy based on specific successful StopPalu+ interventions such as community dialogs facilitated by CAGs, mobile clinics, etc. While building the capacity of health care providers improved the quality of services, engaging the private sector could contribute to expanding service coverage. Beneficiaries of the StopPalu+ activity at the central, facility, and community levels have taken ownership of the activity approach through CAGs. Core team members at all levels are involved in planning, mobilizing, and implementing activities. ● At the central level, support to the NMCP improved its capacity to lead, manage, and coordinate malaria prevention and treatment activities in Guinea. Capacity building and health system strengthening transformed the NMCP and improved its functions. ● At the facility level, the capacity building of health care providers and improvement of the M&E system and diagnostics strengthened the country’s health care system. ● At the community level, the creation of CAGs enabled community members to organize community dialogs, and also allowed them to identify problems and develop solutions. Trained CHW conduct malaria prevention activities and case management. The activity used a mix of communication channels to reach different target populations. These included mass media (TV and radio), interpersonal communication with CHWs and trained religious leaders, supportive supervision, and home visits. Flyers hung at health facilities were also a means of communication. Community dialogue, facilitated by CAGs is an innovative approach to promoting GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 61 community engagement and ownership that places the community at the forefront of all malaria control interventions. Strengthening malaria diagnostics, treatment, and prevention services helped strengthen the overall health care system in Guinea by contributing to increased community engagement and trust in and respect for the activity, leading to increased utilization of health care services (e.g., IPTp during ANC and SMC for children under age 5). StopPalu+ implemented varied SBCC strategies to promote the adoption of healthy behaviors and create demand for malaria prevention and treatment services among men and women alike. The activity trained health providers and male and female CHWs in IPC and quality care delivery. The delivery of malaria control services free of charge removed barriers for people of low socio-economic status, including women, to access and utilize malaria control services. Quality data is available on the StopPalu+ activity for the most part, showing adequate tracking and reporting on indicators. However, some indicators lack consistent data and disaggregation. While some data could not be easily located annually, many others where disaggregation could have been useful and standard, e.g., data by region, and data by gender, did not have either data for all required periods or disaggregation. The StopPalu+ AMELP contained all components, but the performance indicator reference sheets were not complete in important sections such as “data quality issues” and “performance indicator values” for each year of the activity. Additionally, the Indicator Performance Tracking Tables (IPTT) contained only targets for each year of the activity, without any actual results. Overall, StopPalu+ successfully contributed to the increased use of malaria control activities, including uptake of LLINs in the community, IPTp during ANC visits as well as SMC among children ages 3–59 months. This success was achieved through capacity building, social mobilization, use of mobile clinics and red booklets, a steady supply of commodities, and SBCC. GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 62 6. RECOMMENDATIONS Based on the performance evaluation findings and conclusions, the evaluation team makes the following recommendations: RECOMMENDATIONS FOR STOPPALU+ 1. Increase collaboration with the MOH, UGANC, SNIS, and NMCP to design relevant interventions that are consistent with country needs and NMCP strategies/objectives, to make the most impact on the health of the population. 2. Collaborate with relevant programs and agencies to ensure that malaria-related commodities (IPTp, SMC, etc.) are always in stock. This collaboration will include capacity development in the areas of supply management, commodity ordering, and distribution. 3. CHWs (RECOs) have played a significant role in the success of StopPalu+ at community and facility levels. The activity can sustain gains by continuously training current and new community￾based health workers. 4. As a best practice, StopPalu+ should advocate the institutionalization of community dialogs to promote community engagement in malaria control activities. 5. StopPalu+ and similar USAID activities should ensure that all sections of AMELPs, such as performance indicator reference sheets and indicator performance tracking tables, are regularly updated with consistent and complete progress data to appropriately reflect activity progress. RECOMMENDATIONS FOR USAID 1. Make future funding contingent upon the submission of an action plan describing how the government will take over the activities implemented by StopPalu+. 2. Prepare the ground to ensure a gradual takeover of all the activities currently supported through StopPalu+. 3. Organize training on leadership and management to strengthen the NMCP leadership’s managerial skills to improve coordination of activities among IPs and manage stocks. 4. Train the NMCP in grant writing to prepare them to seek new funding streams. 5. Support the NMCP to implement a compensation policy that all Government of Guinea and development agencies can adhere to as they implement development activities. 6. Monitor progress being made promoting gender equity across all program and community levels. 7. Set aside a special funding mechanism to support the insectarium after the end of the activity. Considering the high cost of supplies to run genetic and other advanced tests, external financial support is needed in the future to keep the insectarium running. 8. Assume a proactive monitoring role to ensure that the annual performance reports include consistent data for all performance indicators. GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 63 RECOMMENDATIONS FOR THE MOH/NMCP 1. Fulfill its funding obligations to health sector programs. The Government of Guinea must fulfill its responsibility. The government investment in health care has a strong multiplier effect throughout the economy in addition to enhancing the population’s wellbeing. 2. Sustain the current gain achieved by the activity. Because StopPalu+ will come to an end, the Government of Guinea must take full responsibility to sustain what the activity has accomplished so far. The government should identify other revenue streams to maintain the current level of activities. 3. Standardize effective interventions such as community dialogue by CAGs across the country. Considering the limited resources, the Government of Guinea must invest these resources only in effective interventions expected to yield a high return on investment. 4. Take immediate corrective action to address the recurrent stockouts of commodities at health facilities. The NMCP should organize additional training in leadership and management to improve staff skills in coordinating activities among IPs and stock management. 5. Take immediate corrective action to address staff’s lack of motivation and turnover. The quality of services is associated, in part, with staff competencies and level of satisfaction. The NMCP should analyze the root causes of the lack of motivation and address it expeditiously to maintain the gains of the activity. 6. Develop more public-private partnerships to improve access to and coverage for health care services. Considering that many people seek medical attention from private providers (pharmacists or health centers) or resort to self-medication, the private sector can play a significant role in the fight against malaria. Because public-private partnerships leverage the strengths of the public and private sectors simultaneously, they could create economies of scale and expand access to care for people at high risk and/or underserved populations. 7. Implement gender equity policy across all programs and community levels. Gender equity is a powerful social determinant for health. Malaria in pregnancy contributes to significant perinatal morbidity, causing higher rates of miscarriage, intrauterine demise, premature delivery, low birth-weight neonates, and neonatal death. Considering these factors, the evaluation team recommends that the NMCP actively pursues gender equity and inclusiveness across all program levels. 8. Standardize monthly report review meetings to improve the quality of data collected and support good decision-making and quality of care. GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 64 ANNEX 1. SCOPE OF WORK GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 65 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 66 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 67 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 68 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 69 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 70 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 71 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 72 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 73 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 74 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 75 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 76 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 77 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 78 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 79 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 80 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 81 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 82 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 83 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 84 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 85 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 86 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 87 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 88 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 89 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 90 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 91 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 92 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 93 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 94 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 95 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 96 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 97 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 98 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 99 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 100 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 101 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 102 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 103 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 104 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 105 ANNEX 2. SOURCES OF INFORMATION Activity Documents: StopPalu+ ● Request for Application/solicitation document ● Activity Monitoring and Evaluation Plan ● Annual work plans ● Annual and quarterly progress reports o StopPalu+ PMI Program Component Annual Report, (FY 2020). October 1– September 30, 2020 o StopPalu+ PMI Program Component Quarterly Report, April 1–June 30, 2021 o StopPalu+ PMI Program Component Quarterly Report, January 1–March 31, 2019 ● HRH2030 reports ● StopPalu+’s Knowledge, Attitude, and Practice Survey reports ● Global Health Supply Chain Program Procurement and Supply Management reports USAID ● Health Systems Assessment ● Gender Assessment 2020 ● Local Systems Assessment Framework ● USAID Health Systems Strengthening Framework ● Malaria Operational Plans and New Funding Model of Global Funds ● Assessment of long-term technical assistance advisors to NMCP (SOW and reports) ● PMI Annual Report of Entomological Surveillance Activities 2018–2021 Guinea Documents ● National Malaria Strategy, protocols, and standard operating procedures ● National Community Health Workers Strategy ● National Health Development Plan 2015–2024 ● National Ebola Recovery Strategy 2015–2017 ● Service Continuity Plan for COVID-19 ● Monthly malaria bulletins ● 2018 Demographic and Health Survey report ● 2017 Service Availability and Readiness Assessment report ● 2016 MICS report ● Global Health Supply Chain Program Procurement and Supply Management reports GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 106 ANNEX 3. DATA COLLECTION TOOLS 3A. Informed Consent Statement StopPalu+ Performance Evaluation CONSENT STATEMENT The Assignment Team, prior to any data collection, must develop protocols and create an informed consent process in collaboration with GH EvaLS to ensure that there are adequate provisions to protect the privacy of subjects and to maintain the confidentiality of the data. Abstracting Data from Existing Reports and Materials: • During the process of this assignment, if data are abstracted from existing documents that include unique identifiers, data can only be abstracted without this identifying information, unless the identifiers are referencing a public figure. Collecting Data from Participants in KIIs and Group Interviews • An informed consent process is required to ensure that participants have the information they need to determine whether or not they wish to participate. • Informed consent may be obtained in writing or verbally, depending upon the nature of the information being collected, but consent must be documented. • If using a written informed consent (preferred), the following elements are to be included: • Introduction of facilitator/note-taker • A statement that explains the purpose of the Assignment including: • The expected duration of the subject’s participation • A description of the procedures to be followed (if applicable), and • Identification of any procedures that are experimental (if applicable) • A description of any reasonably foreseeable risks or discomforts to the subject • A description of any benefits to the subject • A statement that participation is voluntary, refusal to participate will involve no penalty or loss of benefits to which the subject is otherwise entitled, and the subject may discontinue participation at any time • A statement describing the extent to which confidentiality of records identifying the subject will be maintained • A statement that all information provided is confidential and information provided will not be connected to the individual in published reports, unless the respondent is designated as a key informant and provides permission to be named in published reports • A statement that identifiers might be removed from the identifiable private information and that, after such removal, the information may be used for future research studies or distributed to another investigator for future research studies without additional informed consent • An explanation of whom to contact for answers to questions about the assignment Note: Only adults can consent as part of this assignment. Minors cannot be respondents to any interview or survey and cannot participate in a focus group discussion without going through an Institutional Review Board. The only time minors can be observed as part of this assignment is as GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 107 part of a large community-wide public event, when they are part of a family and community in the public setting. INFORMED CONSENT Introduction Thank you for the opportunity to speak with you. My name is _________________, and I represent the USAID-supported Global Health Evaluation and Learning Support (GH EvaLS) Activity as a consultant. USAID has contracted GH EvaLS to conduct a performance evaluation of the StopPalu+ Activity implemented by RTI International as prime. This evaluation aims to help identify promising practices and lessons learned, along with weaknesses and gaps that hinder effective malaria service delivery. It will inform the Government of Guinea’s National Malaria Control Program (NMCP) and future program design and implementation for malaria control in the country. Purpose of the KII/FGD/survey You were suggested as a key individual to inform this evaluation. We greatly appreciate your perspectives, experiences, and expertise on the effectiveness and outcomes of the StopPalu+ in reducing the burden of malaria in Guinea by working with the National Malaria Control Program and other agencies. We seek your expertise and experience to provide your perspectives on the effectiveness and outcomes of the activity, “SSBH, or System Strengthening for Better Health.” We want you to share your thoughts, based on your experience and your involvement with the project. Your participation is entirely voluntary; you can withdraw at any time during the process. The interview is voluntary; you can withdraw at any time during the process. The interview may take 60 minutes. Please know that anything you say during the interview will be kept confidential and that in our report we will not be attributing specific comments to any specific individual. In order to ensure accuracy, we would like to record the interview so we may refer back to it. After entering the information into a database, we will erase the recording, and remove all information such as your name, and comments that could link any responses to you. You have the right to not participate in this interview/discussion/survey and you may stop participation at any time. The information that you provide will be kept confidential. Any information that you provide that will contribute to any reports or publications will not be associated with your name, unless you are considered to be a public figure and tell us that you would be willing to have your responses to be either quoted in the report, or otherwise attributed to you. In the future, your de-identified information may be used by other researchers for other studies. In this case, we will not be seeking additional consent from you. Before we begin, do you have any questions about this interview? _________ [Interviewer/Recorder initials] GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 108 Request for consent prior to initiating data collection Our interview will take about one hour. Do I have your permission to begin? Yes _____ (Proceed with interview) No _____ (End the interview) A separate consent for recording the interview/discussion (as necessary) Optional: I would like to record this interview to ensure that I do not miss any important points. Do I have your permission? Yes, record ______ No, don’t record ______ Note: Only adults can consent as part of this assignment. Minors cannot be respondents to any interview or survey and cannot participate in a focus group discussion without going through an Institutional Review Board. The only time minors can be observed as part of this assignment is as part of a large community￾wide public event when they are part of a family and community in a public setting. GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 109 3B. Key Informant Interview Guides STOPPALU+ PERFORMANCE EVALUATION KEY INFORMANT INTERVIEW GUIDE 3.B.1 USAID Guinea Mission Questions Responses Interviewer : Interview Date : ______ /______/2021 Interview time : Start Time : Interviewee’s Name : Last Name : First Name : Current Position : Contact e-mail : Contact Phone : Has the KI affirmed Informed Consent? (Interviewer’s initials) Y____ N____ Has the KI consented to recording the interview? Y____ N____ Respondent’s Unique ID ID: ______________________ N° Questions/Topics Responses I. Background information 1 What activities are you responsible for in the Mission? 2 How long have you worked at the mission? 3 What have been your or the mission’s roles or involvement in the design and implementation of StopPalu+ in Guinea? GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 110 4 Would you say you are more involved with the management aspects or the technical/policy/strategy aspects of the StopPalu+ process? Probe: Why? Why not? (This is a screening question) II. Core Questions 5 How do you think StopPalu+ objectives align with those of the PNDS? 6 How do you think StopPalu+ objectives align with those of the Ministry of Health malaria program? 7 What were the mission’s expectations of the StopPalu+ process and the resulting activities? As implemented, is the project meeting those expectations? Why? Why not? 8 To your opinion, how well StopPalu+ is strengthening the skills of partner structures including those of the NMCP, and government and NGO health structures? 9 What are some of the improvements in the Guinean overall health system (health policy, capacity for health leadership and inter-sectoral collaboration) you have noted that you could attribute to StopPalu+? 10 In your opinion, what are some distinguishing aspects or features of the StopPalu+ projects, both positive and negative? 11 Do you see those characteristics as linked to the StopPalu+ processes through which they were created? Or to other things? (Probe for specifics) GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 111 III. Questions tailored for different stakeholder groups 12 Was the StopPalu+ process sufficiently understood by all stakeholders at its initiation? If not, what were the biggest knowledge gaps? 13 Do you believe that expectations (management and technical) were clearly understood by all stakeholders (MOH, IPs and communities) prior to award signing? If not, what were some of the gaps? IV. Challenges 14 What have been the primary successes and challenges of the StopPalu+ project implementation? How have M&E and project oversight worked? Are you satisfied with the information you have? Can you make informed decisions? How attributable are those successes and challenges to StopPalu+ process? What else worked well and what didn’t? What makes you say so? V. Gender inequalities 15 What efforts have you noted in this project to increase the involvement of women and girls in leading and benefiting from malaria care services in the country due to the project? 16 What progress is being made, or is needed, to address the gender inequalities that influence the nation’s health? GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 112 VI. Sustainability 17 What do you think of the sustainability of project activities? Probe: For example, what steps do you think were taken for supported activities to continue after the project’s end? VII. Recommendations 18 What are your recommendations to correct weaknesses of the project and to ensure sustainability? (list the 5 major ones) Probe: For example, do you think of other activities that the project should add? (If yes, list them) Other comments: Time at End of Interview: Interviewer’s observations or interpretations: GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 113 StopPalu+ Performance Evaluation Key Informant Interview Guide 3.B.2. MOH (Central and Periphery) and Other Institutions Involved Questions Responses Interviewer: Interview Date: ______ /______/2021 Interview time: Start Time: Interviewee’s Name: Last Name: First Name: Current Position: Contact e-mail: Contact Phone: Has the KI affirmed Informed Consent? (Interviewer’s initials) Y____ N____ Has the KI consented to recording the interview? Y____ N____ Respondent’s Unique ID ID: ______________________ N° Questions / Topics Responses I. Background information 1 What activities are you responsible for in your structure? 2 How long have you worked in this structure? 3 What have been your or your structure’s roles or involvement in the design and implementation of StopPalu+ in Guinea? (Examples: social mobilization, supervision, capacity building, etc.) GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 114 4 Would you say you are more involved with the management aspects or the technical/policy/strategy aspects of the StopPalu+ process? Probe: Why? Or Why not? (This is a screening question) II. Core Questions 5 What do you think of the relevance of the StopPalu+ project (objectives, services offered, adequacy of the services offered to the needs of the populations, etc.)? 6 How do you think StopPalu+ objectives align with those of the PNDS? 7 How do you think StopPalu+ objectives align with those of the Ministry of Health malaria program? 8 What were your expectations of the StopPalu+ process and the resulting activities? As implemented, is the project meeting your expectations? Why? Why not? 9 In your opinion, how well is StopPalu+ strengthening the skills of partner structures, including yours, those of the NMCP, and government and NGO health structures? 10 What are some of the improvements in the Guinean overall health system (health policy, capacity for health leadership and inter-sectoral collaboration) you have noted that you could attribute to StopPalu+? 11 In your opinion, what are some distinguishing aspects or features of the StopPalu+ projects, both positive and negative? GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 115 12 Do you see those characteristics as linked to the StopPalu+ processes through which they were created? Or to other things? (Probe for specifics) III. Questions tailored for different stakeholder groups 13 How do you assess the effectiveness of StopPalu+ communication strategies in the fight against malaria? (List 3 determining success factors) 14 Tell us about the effectiveness of the StopPalu+ participatory management strategy and how it has facilitated implementing partners’ adherence to the technical standard of excellence guidelines. Probe: What are the technical standards that IPs followed? Was there any effect on the pace and quality of the intervention? 15 What do you think of t the StopPalu+ coordination with your organization and others? Probe: What were the communication mechanisms between the organizations? Provide examples of coordinated activities implemented. What worked well and what was challenging? 16 What do you think of the use of LLIN distributed by StopPalu+ in Guinea? 17 What is most satisfying in your work with StopPalu+? GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 116 IV. Challenges 18 What have been the main challenges you have faced with the StopPalu+ process or activities (of the two)? What is working well and what needs improvements? V. Gender inequalities 19 What efforts have you noted in this project to increase the involvement of women and girls in leading and benefiting from malaria care services in the country due to the project? 20 What progress is being made, or is needed, to address the gender inequalities that influence the nation’s health? VI. Sustainability 21 What do you think of the sustainability of project activities? Probe: For example, what steps were taken for supported activities to continue after the project’s end? VII. Recommendations 22 What are your recommendations to correct weaknesses of the project and to ensure sustainability? (list the 5 major ones) Probe: For example, can you think of other activities that the project should add? (If yes, list them) Other comments: GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 117 Time at End of Interview: Interviewer’s observations or interpretations: GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 118 StopPalu+ Performance Evaluation Key Informant Interview Guide 3.B.3. Implementing Partners (RTI and Others) Questions Responses Interviewer : Interview Date : ______ /______/2021 Interview time : Start Time : Interviewee’s Name : Last Name : First Name : Current Position : Contact e-mail : Contact Phone : Has the KI affirmed Informed Consent? (Interviewer’s initials) Y____ N____ Has the KI consented to recording the interview? Y____ N____ Respondent’s Unique ID ID: ______________________ N° Questions/Topics Responses I. Background information 1 What activities are you responsible for in your organization? 2 How long have you worked in this organization? 3 What have been your or your organization’s roles or involvement in the design and implementation of StopPalu+ in Guinea? GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 119 4 Would you say you are more involved with the management aspects or the technical/policy/strategy aspects of the StopPalu+ process? Probe: Why? Why not? (This is a screening question) II. Core Questions 5 How do you think StopPalu+ objectives align with those of the PNDS? 6 How do you think StopPalu+ objectives align with those of the Ministry of Health malaria program? 7 What were your expectations of the StopPalu+ process and the resulting activities? As implemented, is the project meeting your expectations? Why? Why not? 8 In your opinion, how well is StopPalu+ strengthening the skills of partner structures, including those of the NMCP, and government and NGO health structures? 9 What are some of the improvements in the overall Guinean health system (health policy, capacity for health leadership and inter-sectoral collaboration) you have noted that you could attribute to StopPalu+? 10 In your opinion, what are some distinguishing aspects or features of the StopPalu+ projects, both positive and negative? 11 Do you see those characteristics as linked to the StopPalu+ processes through which they were created? Or to other things? (Probe for specifics) GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 120 III. Questions tailored for different stakeholder groups 12 How do you assess the effectiveness StopPalu+ communication strategies in the fight against malaria in Guinea? (list 3 determining success factors) 13 Tell us about the effectiveness of the StopPalu+ participatory management strategy and how it has facilitated implementing partners’ adherence to the technical standard of excellence guidelines. Probe: what are the technical standards that IPs followed? Was there any effect on the pace and quality of the intervention? 14 What do you think of the StopPalu+ coordination with your organization and others? Probe: What were the communication mechanisms between the organizations? Provide examples of coordinated activities implemented. What worked well and what was challenging? 15 What is the level of completeness of the periodic activity reports? 16 What is the level of achievement of specific project objectives (related/planned activities) 17 What do you think of the use of LLIN distributed by StopPalu+ in Guinea? 18 What is most satisfying in your work with StopPalu+? GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 121 IV. Challenges 19 What have been the main challenges you have faced with the StopPalu+ process or activities (of the two)? Probe: What is working well and what needs improvements? V. Gender inequalities 20 What efforts have you noted in this project to increase the involvement of women and girls in leading and benefiting from malaria care services in the country due to the project? 21 What progress is being made, or is needed, to address the gender inequalities that influence the nation’s health? VI. Sustainability 22 What do you think of the sustainability of project activities? Probe: For example, what steps were taken for supported activities to continue after the project’s end? VII. Recommendations 23 What are your recommendations to correct weaknesses of the project and to ensure sustainability? (list the 5 major ones) Probe: For example, do you think of other activities that the project should add? (If yes, list them) Other comments: GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 122 Time at End of Interview: Interviewer’s observations or interpretations: GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 123 StopPalu+ Performance Evaluation Key Informant Interview Guide 3.B.4. Community Leaders/Health Workers Questions Responses Interviewer : Interview Date : ______ /______/2021 Interview time : Start Time : Interviewee’s Name : Last Name : First Name : Current Position : Contact e-mail : Contact Phone : Has the KI affirmed Informed Consent? (Interviewer’s initials) Y____ N____ Has the KI consented to recording the interview? Y____ N____ Respondent’s Unique ID ID: ______________________ N° Questions/Topics Responses I. Background information 1 What activities are you responsible for in the community? 2 How long have you worked in this capacity? 3 What are the main health problems encountered in your community? 4 Do you know the StopPalu+ project? What specifically do you know? GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 124 5 What roles do you (or your institution) play in the implementation of the StopPalu+ project? (Examples: social mobilization, supervision, capacity building, etc.) II. Core Questions 6 What significant changes (positive or negative) regarding good practices in the fight against malaria have occurred in your community as a result of the StopPalu+ activities? 7 What were your expectations of the StopPalu+ process and the resulting activities? As implemented, is the project meeting your expectations? Why? Why not? 8 What is your level of satisfaction with StopPalu+ activities regarding the fight against malaria in your communities? III. Questions tailored for different stakeholder groups 3.1. Information, Education, and Communication (IEC)/Behavior Change Communication (BCC) 9 What are your sources of health information (list at least 3)? 10 What did you take away from the content of messages delivered through malaria awareness campaigns conducted by the project? 11 How have these messages had an impact on attitudes of people in your community towards malaria? GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 125 12 Is there anything in these messages that bothered you with respect to socio￾cultural norms? • The form and nature of the messages and • The behavior of educators Why? 13 What other message do you think should be added to improve your understanding of malaria in your communities (list a maximum of five)? 3.2. LLINs 14 Which organizations gave distributed LLINs in your community? (list 3 organizations in order) 15 In your opinion, what are the reasons that would prevent the use of LLINs against malaria? (List 5 organizations in order) 16 List 5 ways to fight malaria that you recommend in your communities. 17 In your opinion, how could the project better support your community in the fight against malaria, especially for children under 5? IV. Gender inequalities 18 What efforts have you noted in this project to increase the involvement of women and girls in leading and benefiting from malaria care services in the country due to the project? 19 What progress is being made, or is needed, to address the gender inequalities that influence the nation’s health? GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 126 V. Challenges/Recommendations 20 In your opinion: • What worked most in the implementation of the project (list 4)? • What should we improve or change (list 4)? • What is needed for the sustainability of the achievements of the project (list 4)? Other comments: Time at End of Interview: Interviewer’s observations or interpretations: GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 127 StopPalu+ Performance Evaluation Mothers of Children Under the Age of 5 Years 3.B.5. Focus Group Discussion Guide Questions Responses Region: Health District Sub-Division (Sous-Prefecture): Interviewer: Discussion Date: ______ /______/2021 Discussion time: Start Time : Has the Group affirmed Informed Consent? (Interviewer’s initials) Y____ N____ Has the Group consented to recording the interview? Y____ N____ Group’s Unique ID ID: ______________________ N° Questions/Topics Responses I. Background information 3 What are the main health problems encountered in your community? 4 Do you know the StopPalu+ project? What specifically do you know? II. Core Questions 5 What significant changes (positive or negative) regarding good practices in the fight against malaria have occurred in your community as a result of StopPalu+ activities? GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 128 6 What were your expectations of the StopPalu+ process and the resulting activities? As implemented, is the project meeting your expectations? Why? Why not? 7 What is your level of satisfaction with StopPalu+ activities regarding the fight against malaria in your communities? III. IEC/BCC 8 What are your sources of health information (list at least 3)? 9 What did you take away from the content of messages delivered through malaria awareness campaigns conducted by the project? 10 How have these messages had an impact on your attitudes towards malaria? 11 Is there anything in these messages that bothered you with respect to socio￾cultural norms? • The form and nature of the messages and • The behavior of educators Why? 12 What other messages do you think should be added to improve your understanding of malaria in your communities (list a maximum of five)? IV. LLINs 13 Which organizations gave you LLINs? (list 3 organizations in order) 14 In your opinion, what are the reasons that would prevent the use of LLINs against malaria? (list 5 organizations in order) GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 129 15 List 5 ways to fight malaria that you recommend in your communities. 16 In your opinion, how could the project better support your community in the fight against malaria, especially for children under age 5? V. Gender inequalities 17 What efforts have you noted in this project to increase the involvement of women and girls in leading and benefiting from malaria care services in the country due to the project? 18 What progress is being made, or is needed, to address the gender inequalities that influence the nation’s health? VII. Challenges and Recommendations 19 In your opinion: • What worked most in the implementation of the project (list 4)? • What should we improve or change (list 4)? • What is needed for the sustainability of the achievements of the project (list 4)? 20 In your opinion, how could the project better support your community in the fight against malaria, especially children under age 5? Other comments: Time at End of the Discussion: GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 130 Interviewer’s observations or interpretations: GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 131 StopPalu+ Performance Evaluation 3.B.6. Review of the IEC/BCC Strategy and Materials N° Questions Summary responses 1 Could you show us your communication plan (strategy) in use for the implementation of the StopPalu+ project, please? (RTI and NGO partners) 2 What are the IEC/BCC tools used by the project (disaggregated by type/site)? 3 Have the IEC/BCC tools used in the project been pre￾tested (see report)? 4 What are the contents of the messages conveyed to the target population (appreciate clarity, objectivity and comprehension)? 5 What are the communication channels used in the project for each target (disaggregated by type/site)? 6 What are the strengths and weaknesses of the IEC/BCC tools used in the life of the Project? Summary and comments of quality of the IEC/BCC approaches used by the project GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 132 StopPalu+ Performance Evaluation 3.B.7. Health Facility Observation Checklist Questions Responses Region: Health District Sub-Division (Sous-Prefecture): Interviewer: Interview Date: ______ /______/2021 Interview time: Start Time : Facility Name: Type _____________/ Code: ____________/ Ask a health worker to show you around the facility. Look and ask to complete the following questions. N Questions Responses 1 Characteristics of health workers with case management responsibilities for children Qualification Number Physician______________ Nurse ________________ Midwife_______________ Health Assistant________ Others________________ Total_________________ _______________ _______________ _______________ _______________ _______________ _______________ 2 Does the facility have the following equipment and materials? a. Accessible and working adult scale? (1) Yes (2) No b. Accessible and working baby scale? (1) Yes (2) No c. Working watch/timing device available to every health worker managing children? (1) Yes (2) No d. Supplies to mix ORS, cups and spoons (1) Yes (2) No e. Source of clean water (1) Yes (2) No f. Stock cards/drug logbook (1) Yes (2) No g. Child vaccination cards (1) Yes (2) No h. A functioning fridge? i. Mothers’ counseling cards? (1) Yes (2) No j. Waste disposal equipment? (1) Yes (2) No k. Other ………………………………… (1) Yes (2) No GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 133 3 Does the facility have the following drugs available the day of visit? (Check the drug stocks. Answer the following questions based on what you see). a. Recommended antimalarial: (1) Yes (2) No c. Vitamin A (1) Yes (2) No d. Iron (1) Yes (2) No e. Paracetamol/aspirin (1) Yes (2) No f. Quinine IM (1) Yes (2) No g. Sterile water for injection (1) Yes (2) No h. Needles and syringes for injectable drugs (1) Yes (2) No f. Recommended IV fluid for severe dehydration (1) Yes (2) No g. Other…………………………………… (1) Yes (2) No Other comments: Interviewer’s observations or interpretations: GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 134 StopPalu+ Performance Evaluation 3.B.8. Exit interview Mothers of Children Age 2 Months–5 Years Who Received Malaria Care Services at the Facility Questions Responses Region: Health District Sub-Division (Sous-Prefecture): Interviewer: Interview Date: ______ /______/2021 Interview time: Start Time : Has the Group affirmed Informed Consent? (Interviewer’s initials) Y____ N____ Has the Group consented to recording the interview? Y____ N____ Facility Name: Type _____________/ Code: ____________/ Child Name: ID: ___________/ Age: ___________/ N Questions Responses 1 What brought you to the health facility today? 2 Take a look at the health card and note: ● The diagnosis ● Was temperature taken? ● Was the weight recorded? 3 What do you think of services for your sick child at this facility? Read all options to the mother. (1) Good as they are (2) Should improve What should improve: 4 Did the health worker give you or prescribe any oral medicines for at the health facility today? 1) Yes ________ (2) No______ If No, Skip to # 8 5 Does the oral treatment given or prescribed by the health worker include an antimalarial? 1) Yes ________ (2) No______ If No, Skip to # 8 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 135 6 If the oral treatment includes an antimalarial, record what health worker says: a. First antimalarial name: ……………………………… b. Formulation: ………………………………………………. c. Amount each time: …………………………………….. d. number of times per day: …………………………… e. total days: ……………………………………………… f. Second antimalarial name: …………………………… g. Formulation: ………………………………………………. h. Amount each time: ……………………………………. i. # Times per day: ………………………………………….. j. Total days: ……………………………………………… 7 Did the health worker explain how to administer the oral treatment for malaria prescribed? (1) Yes______ (2) No ______ NA______ 9 Did the health worker give or ask the mother to give the first dose of the oral drug at the facility? (1) Yes______ (2) No ______ NA______ 9 Sometimes children’s condition may worsen and they should be taken immediately to a health facility. What types of symptoms would cause you to take your child to a health facility right away? Do not prompt—keep asking for more signs/symptoms until the caretaker cannot recall any additional ones. a. Child not able to drink or breastfeed (1) Mentioned (2) Not mentioned b. Child becomes sicker (1) Mentioned (2) Not mentioned c. Child develops a fever (1) Mentioned (2) Not mentioned d. Child has fast breathing (1) Mentioned (2) Not mentioned e. Child has difficulty breathing/pneumonia (1) Mentioned (2) Not mentioned f. Child has blood in the stools (1) Mentioned (2) Not mentioned g. Child is drinking poorly (1) Mentioned (2) Not mentioned h. Other, specify ………………………………………… i. Other, specify …………………………………………… GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 136 10 Do you and your child sleep under a bed net at home? If yes, was it received from the project? (1) Yes______ (2) No ______ If Yes: Source: ____________________ 11 Did you receive medicine to prevent malaria during your pregnancy? (1) Yes______ (2) No ______ NA______ 12 Did the health worker ask at least one question about the mother’s health (ask about her own health, access to malaria prevention services, family planning or vaccination status)? (1) Yes______ (2) No ______ NA______ Other comments: Time at End of the Discussion: Interviewer’s observations or interpretations: End of Exit interview: Thank the mother for answering your questions and ask if she has any questions. Be sure that she knows how to give the prescribed medications, and when to return if the child becomes worse at home. GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 137 StopPalu+ Performance Evaluation 3.B.9. Document Data Extraction Sheet DATE REVIEWED: REVIEWER: TITLE: AUTHORS (APA FORMAT): CITATION (APA FORMAT): QUOTES FOR THE REPORT WITH PAGE LOCATION FOR CITING This document contains data supporting or not supporting the following Evaluation Questions Pages with info (Supporting) Pages with info (Not Supporting) 1.1.a To what extent have the program activities/interventions been implemented as planned? 1.1.b Were the programs implemented according to schedule and quality standards? 1.1.c Are the specific objectives of the project relevant and consistent with national malaria orientations, objectives, and strategies? 1.1.d. How have socio-political and cultural factors hampered and/or facilitated the implementation of the program? 1..2.a What are the main factors that can influence the sustainability of the malaria program in Guinea that have facilitated or not the implementation of this project? 1.2.b To what extent did beneficiaries take ownership of the project approach? 1.2.c Who are the main actors to be more involved for the sustainability of the project? 2.a To what extent have the capacity building activities of this project improved the financial management, organizational planning and procurement capacity of the NMCP? 2.b What support has the project provided to the NMCP in achieving its objectives of coordinating malaria prevention and treatment activities? 3.a To what extent did the resources allocated to the project allow the implementation of activities? 3.b What are the communication channels used to reach the target populations (Preference)? 3.c What is the level of satisfaction of the stakeholders and beneficiaries of the project? 3.d What are the effects derived from project activities that have enabled the strengthening or not of the health system in Guinea? 4.a To what extent has the promotion of gender equality been integrated into the activities carried out? 4.b To what extent has the project improved the factors associated with intermittent treatment in pregnant women? 4.c To what extent has the project improved the factors associated with intermittent treatment in children under age 5? GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 138 ANNEX 4. DISCLOSURE OF CONFLICTS OF INTEREST GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 139 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 140 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 141 GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 142 ANNEX 5. EVALUATION TEAM MEMBERS Joseph Inungu—Team Lead Dr. Joseph Inungu is a professor of global health in the School of Health Sciences at Central Michigan University and the founding Director of the Master of Public Health Program at the university. Previously he served as the director of socio-behavioral research for the International AIDS Vaccine Initiative in Southern Africa, a New York-based NGO that specializes in HIV vaccine research. Prior to joining the initiative, he was the Regional Researcher for Population Services International in West and Central Africa. In this role, he provided technical assistance and oversaw the design and implementation of social marketing research in Benin, Burkina Faso, the Democratic Republic of Congo, Liberia, Nigeria, and Senegal, dealing with the prevention of HIV/AIDS and malaria, reproductive health, water, and sanitation and hygiene. Dr. Inungu conducted program evaluations and built capacity to conduct research in participating countries. He has authored more than 40 articles in peer-reviewed journals and presented several papers at national and international conferences. He co￾authored the textbook Foundation of Rural Public Health in America, published by Jones & Barletts Learning in 2022. He holds master’s and doctoral degrees in public health from Tulane University School of Public Health and an MD degree from the University of Kinshasa in the Democratic Republic of Congo. Sidikiba Sidibé—Local Malaria Specialist Dr. Sidikiba Sidibé is a multilingual public health professional with 21 years of national, regional, and global experience (including 13 years with Africare). He has a track record of success as an independent international and national consultant; country director/chief of party, project coordinator, study coordinator and M&E director for the malaria program at RTI; research and M&E director for HC3; M&E advisor for JSI/APC/USAID; and senior program manager for IMC Guinea. Dr. Sidibé has excellent partnership building skills at the national and international levels, and extensive experience in Guinea, Burkina Faso, Burundi, Chad, Mali, and Rwanda. He earned his MD degree in 1995 and has since earned diplomas in research methods and epidemiology, an MPH, and a PhD in public health epidemiology. For this assignment, Dr. Sidibé provided expert knowledge of the local malaria environment in Guinea. Alpha Barry—Evaluation Specialist Alpha Barry has more than 20 years of experience in public health performance program management and community health interventions mainly with USAID health projects. He has strong expertise in public health project monitoring and evaluation, performance indicators, development, quantitative and qualitative research methods, and data analysis. He also has relevant expertise in the use of mobile android technology to improve data collection and GH EvaLS Draft Evaluation Report: USAID Guinea StopPalu+ / 143 epidemic surveillance, as well as skills in behavior change communication to improve awareness and practices. He currently is a research methodology professor. On this assignment, Dr. Barry was the evaluation specialist, leading research methodology and data collection and analysis. Roger Millimono—Logistics Coordinator Roger Millimono is a monitoring and evaluation research coordinator with more than eight years of experience in management, M&E, translation, advocacy, community mobilization/awareness, and entrepreneurship. His international experience spans Africa, Europe, and the United States, including Guinea, Sierra Leone, and Spain. He has worked with USAID, UNICEF, Action contre la Faim, the U.S. Office of Foreign Disaster Assistance, American Friends of Guinea, and other donors and organizations. As the logistics coordinator for this assignment, Mr. Millimono arranged meetings with key informants, made travel and transportation arrangements, assisted with the final data analysis, and performed other tasks.