USAID/GLOBAL HEALTH EVALUATION AND LEARNING SUPPORT ACTIVITY (GH EvaLS) USAID/PMI IMPACT MALARIA MID-TERM PERFORMANCE EVALUATION September 2021 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by ME&A, Inc., its subcontractor Dexis Consulting Group, and the evaluation team comprised of Deborah McSmith, Peter Bloland, and Marcelo Castrillo. 2 USAID/PMI IMPACT MALARIA MID-TERM PERFORMANCE EVALUATION September 2021 USAID Contract No. GS-10F-154BA/7200AA20M00003, GH EvaLS Assignment Number: 008 Period of Performance: February-July, 2021 Contracting Officer Representative (COR): Virginia Lamprecht Submitted by: Andrea Camoens, Chief of Party GH EvaLS 4350 East-West Highway, Suite 210 Bethesda, MD 20814-4410 Tel: (301) 652-4334 Email: acamoens@engl.com Cover Photo: Regina Kluise, District Deputy Director of Nursing Services, trains nurses on malaria rapid diagnostic tests at the Tetteh Quarshie Memorial Hospital in Ghana. March 23, 2021. Photo Credit: Emmanuel Attramah, PMI Impact Malaria DISCLAIMER The views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. 3 This document is available online. Online documents can be located in the GH EvaLS website at https://ghevals.meandahq.com. Documents are also made available through the Development Experience Clearinghouse (http://dec.usaid.gov). Additional information can be obtained from: Global Health EvaLS 4350 East-West Highway, Suite 210 Bethesda, MD 20814-4410 Tel: (301) 652-4334 Website: https://ghevals.meandahq.com ABSTRACT The mid-term evaluation (MTE) of the five-year (2018-2023) President’s Malaria Initiative (PMI)/Impact Malaria (IM) Contract under United States Agency for International Development (USAID)/Bureau of Global Health (GH)/Office of Infectious Disease (ID) was conducted to inform the structure and content of current and future USAID/PMI investments in malaria case management (CM), prevention of malaria in pregnancy (MIP), and other malaria drug-based interventions. The IM MTE has the following objectives: 1. Assess and document progress toward achieving project objectives and whether desired results have occurred; 2. Determine the effectiveness and efficiency of project operations and management; 3. Capture lessons learned and identify key bottlenecks and gaps that can inform future PMI activities in CM, in the context of the PMI strategy. The three main evaluation questions (EQs) were: EQ1. COUNTRY-LEVEL PERFORMANCE: To what extent has PMI IM achieved its country-level objectives? EQ2. MANAGEMENT: To what extent has PMI IM met the management requirements and functions outlined in the agreement, including planning, allocation of funds, coordination among the IM partners (Population Services International [PSI], Medical Care Development International [MCDI], University of California San Francisco [UCSF], Jhpiego), staffing requirements, and in￾country support? EQ 3. GLOBAL RESULTS: What results have been realized at the global level? Evidence indicates that PMI IM is performing very well across 18 buy-in countries and two USAID Regional Bureaus in terms of activities and outputs, is trusted among consortium partners, and enjoys strong credibility at both country and global levels. The project’s accomplishments for Seasonal Malaria Chemoprevention (SMC) and responses to Coronavirus Disease 2019 (COVID￾19) are particularly impressive. Project outcomes show improvements, although they are highly variable across programmatic areas, countries, sub-country project areas, and time. In terms of project impact, it would be unreasonable to expect major changes in morbidity and mortality (even if not an explicit expectation) after 2.5 years of project implementation. Additionally, available data sources for analysis of impact are not within the project’s control, are often not standard across countries and are often plagued by poor quality and consistency. Methodologic issues (absence of control areas or well-designed independent periodic surveys) also contribute to an inability to demonstrate more effect at this point in time. 5 ACKNOWLEDGEMENTS The evaluation team acknowledges the strong support from the IM Headquarters team hosted by PSI and its partners in sharing the IM project data and documents as requested. We also thank the GH EvaLS and USAID PMI colleagues who provided oversight and support for this mid-term evaluation. Finally, we express our gratitude to all in-depth interview and survey respondents who added rich detail to the evaluation findings. 6 TABLE OF CONTENTS Abstract 4 Acknowledgements 5 Table of Contents 6 Acronyms 8 Executive Summary 11 Introduction 11 PMI IM Project Background 11 PMI IM Mid-Term Evaluation Purpose and Evaluation Questions 12 Methods and Limitations 12 Findings, Conclusions, and Recommendations 14 1. Introduction 25 1.1 Evaluation Purpose 25 1.2 Evaluation Questions 25 1.3 Evaluation Audiences 26 2. Background 27 PMI IM Project Background 27 3. Evaluation Methods and Limitations 30 3.1 Evaluation Methodology 30 3.2 Data Sources 30 3.2.1 Desk Review 30 3.2.2 Key Informant Interviews 30 3.2.3 Online Survey 30 3.2.4 Five In-Depth Country Reviews 31 3.3 Sample Selection 31 3.4 Data Management and Analysis 31 3.5 Ethical Considerations 32 3.6 Limitations 32 4. Findings 39 4.1 EQ1. Country-Level Performance: To What Extent has PMI IM achieved the Country￾Level Objectives? 39 4.2 EQ2. Management: To What Extent has IM Met the Management Requirements and Functions Outlined in the Agreement, Including Planning, Allocation of Funds, Coordination Among the IM Partnership (PSI,MCDI, UCSF, JHPIEGO), Staffing Requirements, and In￾Country Support? 52 4.3 EQ3. Global Results: What Results Have Been Realized at the Global Level? 61 7 5. Conclusions and Recommendations 63 Annexes 75 Annex 1 Statement of Work 76 Annex 2: Data Collection Instruments 125 Annex 3: Documents Reviewed, Key Informants, and Survey Respondents 154 Annex 4: Evaluation Team Members 161 Annex 5: PMI Impact Malaria’s Response to COVID-19 163 8 ACRONYMS Acronym Description ACT Artemisinin-Based Combination Therapy ASTMH American Society for Tropical Medicine and Health ANC Antenatal Care BA Breakthrough ACTION bMDRT basic Malaria Diagnostic Refresher Training CDC Center for Disease Control and Prevention CHO Clinical Health Officer CHTF Child Health Task Force CHW Community Health Worker CM Case Management COP Chief of Party COR Contracting Officer Representative COVID-19 Coronavirus Disease 2019 CRSPC Country/Regional Support Partner Committee DHIS2 District Health Information System 2 DHMT District Health Management Team DHS Demographic Health Survey DOT Directly Observed Therapy ECAMM External Competency Assessment for Malaria Microscopists EQ Evaluation Question FDA US Food and Drug Administration FY Fiscal Year G-ANC Group Antenatal Care GF Global Fund GH Global Health GH EvaLS Global Health Evaluation and Learning Support Activity GHSC-PSM Global Health Supply Chain Program – Procurement and Supply Management GTS Global Technical Strategy HF Health Facility HMIS Health Management Information System HNQIS Health Network Quality Improvement System HQ Headquarters HW Health Worker iCCM Integrated Community Case Management ID Infectious Disease IM Impact Malaria IP Implementing Partner IMaD Improving Malaria Care and Diagnostics INGO International Non-Governmental Organization 9 Acronym Description IPTp Intermittent Preventive Treatment in Pregnancy IRB Institutional Review Board ITN Insecticide Treated Mosquito Net KI Key Informant KII Key Informant Interview LAC Latin America and the Caribbean LOE Level of Effort LQAS Lot Quality Assurance Sampling MAL Malaria Division MCDI Medical Care Development International MCH Maternal and Child Health MDA Mass Drug Administration MDRT Malaria Diagnostic Refresher Training mRDT Malaria Rapid Diagnosis Test ME&A ME&A, Inc. M&E Monitoring and Evaluation MERG Monitoring and Evaluation Reference Group MIP Malaria in Pregnancy MIS Malaria Indicator Survey MM Malaria Microscopy MOP Malaria Operational Plan MOH Ministry of Health MTE Mid-Term Evaluation NMCP National Malaria Control Program OB/GYN Obstetrician-Gynecologist OR Operational Research OTSS Outreach Training and Supportive Supervision OTSS+ Outreach Training and Supportive Supervision plus PMI U.S. President’s Malaria Initiative PMP Performance Monitoring Plan PSI Population Services International PY Project Year Q Quarter QA Quality Assurance RA Resident Advisor RBM Roll Back Malaria RDT Rapid Diagnostic Test RFP Request for Proposal RH Reproductive Health RHMT Regional Health Management Team SBCC Social and Behavior Change Communication SMC Seasonal Malaria Chemoprevention SOP Standard Operating Procedure 10 Acronym Description SOW Scope of Work SP Sulfadoxine-Pyrimethamine SS Supportive Supervision TA Technical Assistance TES Therapeutic Efficacy Studies ToC Theory of Change TOT Training of Trainers TWG Technical Working Group UCSF University of California San Francisco US United States UT University of Thies USAID United States Agency for International Development USG United States Government WASH Water, Sanitation, and Hygiene WHO World Health Organization 11 EXECUTIVE SUMMARY INTRODUCTION The U.S. President’s Malaria Initiative (PMI) Impact Malaria (IM) cost plus fixed fee award, Contract 7200AA18C00014, in the amount of $163,393,540.00, is managed by Population Services International (PSI) as the primary implementing partner (IP) and Jhpiego, Medical Care Development International (MCDI) and University of California San Francisco (UCSF) as sub partners. The five-year award period is 2/13/2018 – 2/12/2023. The three key IM objectives are: Objective 1: Improve quality of and access to malaria case management (CM) and malaria in pregnancy (MIP) interventions. Objective 2: Improve quality of and access to other malaria drug-based approaches and provide support to pilot/scale-up newer malaria drug-based approaches. Objective 3: In support of Objectives 1 and 2, provide global technical leadership, support operational research, and advance program learning. The project’s expected outcomes are: ● A median of at least 80 percent of patients with suspected malaria receiving a diagnostic test; ● An average of 80 percent of confirmed malaria cases receiving effective malaria treatment according to standard national protocols; ● A 15 percent median increase in the percentage of pregnant women receiving two or more doses of intermittent preventive treatment (IPTp) for malaria during their last pregnancy; ● For each round of Seasonal Malaria Chemoprevention (SMC), 80 percent of targeted children receive a dose of SMC. PMI IM PROJECT BACKGROUND PMI, led by USAID and implemented together with the U.S. Centers for Disease Control and Prevention (CDC), is the U.S. Government’s primary vehicle for assisting malaria affected countries to scale-up proven malaria control and elimination interventions. Its original goal in 2005 was to reduce malaria-related mortality by 50 percent across 15 high-burden countries in sub￾Saharan Africa through a rapid scale-up of four proven and highly effective malaria prevention and treatment measures: insecticide-treated mosquito nets; indoor residual spraying; accurate diagnosis and prompt treatment with artemisinin-based combination therapy (ACTs); and IPTp. PMI supports IM, a flagship project aiming to improve access to and quality of CM, prevent MIP, and improve access to new drug-based approaches to malaria prevention. By the end of project year (PY) 3, IM was operating in 18 countries across Africa and Asia and supporting USAID’s Bureaus for Latin America and the Caribbean (LAC) and Africa1. 1 IM is currently operating in Benin, Burkina Faso, Cambodia, Cameroon, Côte d’Ivoire, Democratic Republic of the Congo, Ghana, Kenya, Lao PDR, Madagascar, Malawi, Mali, Niger, Rwanda, Senegal, Sierra 12 PMI IM MID-TERM EVALUATION PURPOSE AND EVALUATION QUESTIONS The mid-term evaluation (MTE) of PMI IM was conducted to inform the structure and content of current and future USAID/PMI investments in malaria CM, prevention of MIP, and other malaria drug-based interventions. It has the following objectives: 1. Assess and document progress toward achieving project objectives and whether desired results have occurred; 2. Determine the effectiveness and efficiency of project operations and management; 3. Capture lessons learned and identify key bottlenecks and gaps that can inform future PMI activities in CM, in the context of the PMI strategy. The results of the evaluation will be used by the USAID Bureau for Global Health (out of which the project is managed), USAID/PMI Headquarters (HQ) and mission staff, as well as by PMI IM project staff. Evaluation Questions: EQ1: COUNTRY-LEVEL PERFORMANCE: To what extent has PMI IM achieved the country-level objectives? EQ2: MANAGEMENT: To what extent has IM met the management requirements and functions outlined in the agreement, including planning, allocation of funds, coordination among the IM partnership (PSI, MCDI, UCSF, Jhpiego), staffing requirements, and in￾country support? EQ3: GLOBAL RESULTS: What results have been realized at the global level? METHODS AND LIMITATIONS The evaluation used a mixed methods methodology that included: a desk review of documents provided by PSI and USAID/PMI; 81 semi-structured key informant interviews (KIIs) with USAID/PMI management staff, global malaria experts, IM HQ management and technical staff and IPs, host country government stakeholders; a bilingual (English/French) survey sent electronically to 308 country-level staff, government officials, other key stakeholders, and PMI HQ backstops across all 18 IM participating countries; and in-depth reviews of four IM buy-in countries (Cameroon, Ghana, Kenya and Niger) with a fifth review of Senegal’s operational research (OR) experience. Unless otherwise specifically noted, analyses of non-IM data were restricted to IM project areas. The project is designed such that country USAID missions can buy-in to or opt out of the range of activities IM offers, leading to a large degree of variation in what IM was responsible for from country to country. While this design offers flexibility and permits better tailoring to country needs, it presents a challenge for standardized evaluation of the program across countries. Additionally, country performance differed as a function of the amount of support the country had received prior to the IM project, with some countries having had much more work and experience with the fundamental interventions than others. As a result, variable targets, performance indicators, and inconsistent numerators and denominators across country data made Leone, Tanzania, Zambia. USAID’s Bureau for Africa has funded IM’s iCCM activities and related Learning Agenda activities. In the LAC region, IM works with the Pan-American Health Organization (PAHO) to provide support to selected countries to determine key policy changes for the treatment of P. vivax through two operational research studies. 13 it impossible to carry out meaningful standardized analysis. Project targets are not representative of what is happening in a country overall; therefore, it is impossible to describe with accuracy the degree to which project activities have influenced national aims and objectives. Evaluation findings are representative of project accomplishments across PY3 but do not comprise an exhaustive list. The evaluation was necessarily conducted virtually, due to COVID-19 restrictions on movement. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS At the end of each recommendation, the evaluation team has included the stakeholder(s) who should consider implementing it. EQ FINDINGS CONCLUSIONS RECOMMENDATIONS EQ1. COUNTRY-LEVEL PERFORMANCE EQ1a. To what extent has PMI IM achieved the technical and programmatic objectives described in annual country and core workplans and IM Performance Monitoring Plan (PMP)? ● IM has been very productive across multiple technical components: Outreach Training and Supportive Supervision plus (OTSS+), case management including for severe malaria, prevention of MIP, malaria diagnosis and iCCM, and in the development of a complex project Data Hub. ● Project outcomes have been highly variable across programmatic areas, countries, and sub-country project areas, and over time. Where progress toward technical objectives has been made, it is mostly modest. Data quality and variability issues that are out of IM’s control continue to be a challenge for project reporting. ● Because most of the project's data (Data Hub/Health Management Information System (HMIS)) are based on data generated at health facilities (HFs), access to malaria services for people who do not reach the facility cannot be examined. IM data, therefore, are not providing enough information to develop strategies to improve access for hard-to-reach groups. The only information available on a population basis (and therefore providing information about those not accessing services from HFs) is through Demographic Health Survey (DHS) and Malaria Indicator Survey (MIS). Because these surveys are typically only conducted every three to five years, they tend not to provide information that is timely enough to inform operational decisions. ● The project’s performance is outstanding in terms of carrying out multiple activities across a rapidly growing number of countries in the face of multiple start up challenges and the arrival of COVID-19. ● Quality of routinely collected malaria data from government systems and variations across countries make it challenging for the project to report improved outcomes. More specifically, insufficient time, underlying poor government data quality and different types of data across countries (e.g., Malaria Indicator Survey/MIS, other national household surveys) that are out of IM’s control and present a wide variation in quality, and methodologic issues (absence of control areas or well￾designed independent periodic surveys) are likely contributors to an inability to demonstrate more effect at this point in time. However, project data does demonstrate high performance in meeting training, supportive supervision, and SMC distribution targets. ● Consider investing in alternative methods to periodically assess progress independent of national HMIS systems or IM’s own OTSS+ data systems, e.g., population-based surveys of project areas, lot quality assurance sampling (LQAS), quasi￾experimental design, or other comparison methods (cohort study, control group). Consider independent quarterly surveys in randomly selected HFs in one country in PY5. (USAID/PMI and IM) EQ1b. Is there evidence of in-country capacity improvements in malaria diagnosis and CM and prevention of MIP at various levels of the health system (national, regional, district, community), taking into account guidelines, training, supervision checklists? 15 EQ FINDINGS CONCLUSIONS RECOMMENDATIONS ●Project outcomes related to capacity improvements show a high degree of variability across countries and sub-country project areas, across indicators, and over time. ● Quality and completeness issues with routinely collected government health data, which are out of IM’s control and present a wide variation in quality, make it difficult to form a complete picture of the levels of capacity and proficiency that the project has helped to build. ● There is wide variation across countries in the proficiency scores related to laboratory malaria diagnostics. ● Project data show evidence of improvements in quality service delivery in a number of IM-focus countries (e.g., Cameroon and Niger). ● While many Health Workers (HW) have been trained, the prioritization of low￾performing facilities for OTSS+, and high HW turnover in targeted HFs, make it difficult for IM to demonstrate with certainty sustained levels of improved capacity and competence, and will ‘raise the bar’ slowly. ● IM is tracking both knowledge and competency data. The more meaningful indicators related to sustained capacity are competency scores that determine how well new training-related knowledge or skills have ‘held’. ● For OTSS+, rely on competency scores more than pre- and post￾test training scores as evidence of improved capacity. (USAID/PMI and IM) ● Consider following a cohort of microscopists to see that their skills improve and remain high. (IM) ● See the EQ1a. recommendation for independent surveys. (USAID/PMI and IM) EQs1c. and 1d. Do checklists and other tools capture useful data on the status and quality of CM? Are they appropriate and informative? Is implementation of OTSS+ disruptive to provision of services (does it take too much time)? Are results from checklists/other tools used by IM to make adjustments to training and supervision to improve quality? 16 EQ FINDINGS CONCLUSIONS RECOMMENDATIONS ● IM has developed, validated, and implemented seven OTSS+ checklists. ● Eleven IM-supported countries have aligned their supervision approach with the OTSS+ package of checklists by end of PY3. ● KIs from all stakeholder groups identified digitalization of the checklists as a major step forward. ● OTSS+ is a time, resource, and effort intensive activity, and a number of KIs raised concerns about sustainability of the intervention. ● Countries using the OTSS+ model find it very helpful to identify and address performance issues in real time but have concerns about sustainability. ● OTSS+ data generally show that performance was relatively high in some countries reviewed (Kenya and Ghana) and there were modest improvements for some indicators. However, more data on the effect of the OTSS+ approach over a longer period of time are needed. The planned independent OTSS+ evaluation should yield useful additional findings. ● There is no standard operating procedure (SOP) included in the OTSS+ checklist package to guide countries in implementing OTSS+ checklists effectively. ● Collect more evidence regarding durability of change (where there are improvements with IM’s well￾funded, intensive approach, would they be maintained or improved upon as MOH takes more independent responsibility for activities?) (IM) ● Add a SOP to the checklist package that guides District Health Management Teams (DHMTs) and Regional Health Management Teams (RHMTs) to effectively conduct OTSS+, and guides supervisors in skillful use of the checklists. (IM) EQ1e. Has the development of the IM Data Hub and the associated efforts to access national HMIS data for PMP reporting resulted in tangible improvements to data use? ● Setting up the Data Hub has been resource-intensive. ● As of February 2021, eight countries were submitting data to the IM Data Hub (Cameroon, DRC, Ghana, Kenya, Madagascar, Mali, Rwanda, and Sierra Leone); one country (Niger) is estimated to be fully operational in 2021; two countries (Malawi, Zambia) are scheduled to start in 2021; one country (Tanzania) is in start-up phase, and one country (Côte d’Ivoire) is closing operations in late 2021. ● There does not appear to be a well-articulated plan for the future of the Data Hub itself and the wealth of data it has collected. ● IM has encouraged data use at all levels and this remains a stated priority for the remainder of the project. ● Key Informants (KIs) raised concerns about the future of IM’s data systems. ● Expectations differ as to the Data Hub’s continuation after IM ends. It would be inefficient to start over under the next malaria project with a new data system. ● Include expectation that Data Hub data will be transitioned in some form in the Request for Proposal (RFP) for the next five-year malaria project and include this in transition and close out planning. (USAID/PMI) 17 EQ FINDINGS CONCLUSIONS RECOMMENDATIONS EQ1f. Have SMC coverage and adherence objectives been met in areas where PMI IM has been supporting SMC implementation? ● IM’s accomplishments include a demonstrated ability to rapidly implement SMC campaigns at scale, achieving high coverage rates in very difficult settings, and piloting and proving utility of approaches to pay large numbers of workers quickly and efficiently using “mobile money” systems. ● IM supported 28 SMC cycles in Cameroon (8), Niger (12), and Mali (8), reaching about 31 million children under five years in age over three years. Coverage was reported to be between 94 percent and 104 percent for individual cycles, with an overall average of 99 percent of targeted children being reached. The project reports that 34,000 campaign field staff were trained and supported for these SMC cycles. ● In PY3, IM supported all three SMC implementation countries to develop and implement rigorous, low-cost, rapid monitoring surveys in order to confirm measurement of coverage and adherence to all three doses. ● Each country’s approach to SMC campaigns had different strengths: Mali had an SMC payment strategy deemed to be a best practice; Cameroon utilized household enumeration before each cycle; and Ghana implemented real-time tracking of every targeted child. ● SMC campaigns have achieved impressively high coverage and adherence targets in the face of many obstacles and security concerns and exceeded expected outcomes. ● The rapid monitoring surveys have helped to confirm high target achievements and seem like a best practice to continue. ● For SMC, the suggestion to conduct a household enumeration in Niger and Mali to achieve a better denominator for targets seems sound. ● Replicate the rapid monitoring surveys and household enumeration where possible for future SMC campaigns. (IM) ● Document this experience with more learning briefs so that the SMC successes and lessons can be fully shared by end of project. (IM) EQs1g. and EQs1h. Is there evidence of integrated management of key project interventions (CM and MIP) and has this integration strengthened their approach and delivery? Has collaboration with maternal health and antenatal care (ANC) services been strengthened in areas where IM is supporting MIP implementation and service delivery? 18 EQ FINDINGS CONCLUSIONS RECOMMENDATIONS ● Areas where KIs reported that Maternal Health and NMCP collaborated well were within MIP Technical Working Groups (TWGs) and committees, in development of training materials and curricula that covered both malaria CM and MIP components, and within the Policy, Planning, Monitoring and Evaluation (M&E) directorate for issues related to data. ● OTSS+ visits “integrate” supervision of and mentorship for both CM and MIP. ● It is less clear how integration works at service delivery level. Efforts are not yet being made to strengthen linkages between HF and Community Health Workers (CHWs) who are tasked with malaria diagnosis, treatment and referrals to HFs and with MIP education and referrals. Ghana seems to have implemented a successful model of this linkage. ● There is a substantial drop-out rate for the IPTp intervention in all countries. ● Coordination of CM and MIP activities occurs mainly at the central level. At regional or district level, there is no evidence of integration apart from OTSS+ related activities. ● Define indicators for integrated CM and MIP so that the project’s success can be better determined. (USAID/PMI and IM) ● If the MIP (and iCCM) components are further developed for this or future PMI projects, investigate whether phone-based electronic data entry by CHWs can be introduced as part of technical support for training and supervision of CHWs, to help document integration of CM and MIP at community level. This could better link CHWs into the country’s health service delivery and malaria data systems and help collect data on populations not accessing services at HFs. (USAID/PMI and IM) EQ2. MANAGEMENT EQ2a. Has PMI IM HQ and PMI Contracting Officer Representative (COR) team oversight and management aided or hindered IM in accomplishing workplan objectives, both at central and country-level? 19 EQ FINDINGS CONCLUSIONS RECOMMENDATIONS ● KIs repeatedly described mutual appreciation between USAID COR team and PSI’s project management. Communication has been regular and transparent. ● Consortium partners acknowledged the IM Project Director as having created trust among the partners. ● IM managers expressed deep appreciation for the entire project team’s dedication to the work. ● Overall, the IM consortium has benefited from the “one team” approach and partnership principles created by PSI to maintain a healthy dynamic. ● PSI’s one partner principle is well appreciated overall; however, there are some instances where this principle is not fully working. ● There is strong evidence that the USAID/PMI COR team and IM leadership have worked well together over PY1-3. The USAID/PMI COR team and IM’s HQ management team have mutually contributed to the positive working relationship. ● The IM partnership principles have served the project well overall, however instances where they are apparently not working are not always addressed. ● Look for opportunities to integrate the “one partner” good practices into future project management expectations. (USAID/PMI and PSI) ● Take steps to ensure that problems with partner roles at country-level are reported and addressed. This may in some cases mean actively stepping in to mediate. Joint meetings inclusive of all in-country partners and IM leadership at HQ level could help to mitigate partners’ struggles in country when they arise. ● Consider adding flexibility to the partnership principles that allows for changes in the lead partner role if necessary. (PSI) EQ2b. Has coordination between IM and partners in country (PMI Resident Advisors [RAs], National Malaria Control Programs [NMCPs], other IPs) aided or hindered IM in accomplishing country workplan objectives? ● There is nearly unanimous appreciation by country-level KIs and survey respondents of the collaboration and coordination among IM consortium partners, and between IM and Ministry of Health (MOH) and other malaria partners. ● Country-level stakeholders who are not IM project staff praised the project as having strong credibility as a malaria partner. ● IM enjoys an excellent reputation among in-country partners as a collaborative and cooperative malaria partner. ● No recommendations - the project is performing with excellence. EQ2c. Is in-country presence of IM staff sufficient and appropriate? ● There is nearly unanimous consensus that IM staff are well suited for their roles. ● By the end of PY3, country staff were viewed as sufficient. ● Most interview and survey respondents consider IM’s in-country staff sufficient and appropriate. ● No recommendations. IM has performed very well. EQ2d. Has IM been adept at adjusting to the rapid growth of country buy-in, from the original nine countries in fiscal year (FY) 2017 to 18 countries and two regional buy-ins in FY 2019? 20 EQ FINDINGS CONCLUSIONS RECOMMENDATIONS ● IM has done an outstanding job of managing the project’s rapid growth from nine countries in PY1 to 18 countries and two regional buy-ins in PY3. ● Kenya and Ghana report expansion of geographic responsibilities without commensurate expansion of funding. ● Start-up has been most successful where at least one consortium partner already had a registered presence in-country; this has been the case in nearly every participating country. ● IM HQ leadership reported not having received clear planning guidance from USAID for country buy-ins. ● Activities in the PMI core work plan were viewed by IM as ambitious and not prioritized. ● Where target areas or activities have expanded, funding envelopes need to be reviewed. ● Simplifying the country buy-in planning process would save time and money. ● Introduce a systematic review of funding envelopes whenever new responsibilities or areas are added to an IM country activity portfolio. (IM) ● Consider abbreviating the country buy-in planning process to the development of a high-level activity plan and budget. (USAID/PMI and IM) ● Prioritize activities in future workplans; allow sufficient time for a startup/set up period and invite feedback from project partners on realistic time frames. (USAID/PMI with PSI) EQ2e. Has IM been able to hire staff, set up offices, launch activities, and continue activities on the agreed upon timelines? 21 EQ FINDINGS CONCLUSIONS RECOMMENDATIONS ● IM has continued to increase staffing across PY1-3 in response to the project’s growth and in line with the approved core workplan’s staffing plan. ● The IM project faced several major start-up challenges, including an award protest that resulted in a stop-work order which interrupted crucial staff hiring, a temporary USG closure, a disrupted partnership with WHO mandated by USG, and the Covid-19 pandemic. ● Despite serious obstacles, the project has been timely overall with submission of core and country workplans, financial and annual reports. ● Many country and HQ level IM project staff described struggles with uncertainties about when pre-approvals are needed under the IM contract that are still not entirely resolved. ● In-country respondents highlighted funding delays as the key challenge for timely completion of activities. ● Overall, IM is performing well in meeting country staffing needs and launching activities in a timely way. ● Several delays in PY1-2 were outside the project’s control. Delays from PSI in PY1 related to contract requirements were seemingly resolved with a staffing change. ● COVID-19 had profound impacts on the project’s ability to carry out activities as planned and in a timely way. IM found many innovative solutions, detailed in Annex 5. ● The contract mechanism presented a learning curve for both USAID/PMI and IM and created delays where pre￾approvals have been needed. It is not clear that it is the optimal mechanism for a service delivery project with many variables. ● It is not clear which funding delays are outside of USAID/PMI’s control and where improvements can realistically be made. ● Consider publishing the project’s COVID- 19 adaptations as useful lessons for future pandemics. (USAID/PMI and IM) ● For future contract agreements, provide guidance on where pre￾approvals are needed and the anticipated timing for completion of the pre-approval process, to help partners better anticipate and plan. (USAID/PMI and PSI) ● All parties proactively communicate where anticipated funding delays are concerned. Consider how to remove pressure to ‘front’ expenses from the prime partner wherever possible. (USAID/PMI, PSI, and in-country partners) EQ2f. Has IM been adept at tackling the logistics of staffing, coordinating and managing logistics for SMC campaigns? Are the campaign activities in conflict with maintaining routine support for CM and MIP activities? ● SMC campaigns are widely considered one of the project’s biggest successes. Refer to findings under 1f. above. ● As with the TESs (see EQ2g. below), funding envelopes are reportedly sometimes too small to complete the campaigns as planned. ● Campaigns are described by IM leadership as logistically heavy lifts; each country has faced unique and severe challenges. ● IM has done an outstanding job of resolving challenges and security risks and finding solutions to complete SMC campaigns. ● IM is handicapped whenever funding is delayed or funding obligations are not sufficient to cover all campaign costs. ● Ensure that lines of communication from country to IM HQ level are effective where reporting of problems is concerned. (PSI) ● Refer to recommendation under 2e. above related to funding delays. EQ2g. Has IM been adept at tackling the logistics of staffing, coordinating and managing logistics for Therapeutic Efficacy Studies (TES) activities? 22 EQ FINDINGS CONCLUSIONS RECOMMENDATIONS ●IM is planning or implementing nine TESs across eight countries with local sub-contractors: Burkina Faso, Cameroon, Côte d'Ivoire, DRC, Kenya, Mali, Niger, and Rwanda. ● PMI HQ is pleased to have IM’s technical support and involvement in these countries, and the competence of IM’s TES staff at both HQ and country-level was praised. ● TESs have faced challenges in Cameroon, Mali and Kenya. ● Multiple IM HQ informants emphasized that the TES budget envelopes ($75k per study arm) are not always sufficient, and the PMI policy needs review. ● The investment in TES is not in all cases adequately funded to yield meaningful results. ● Review and update TES funding ceilings and recognize country by country cost variations. (USAID/PMI) EQ2h. Has IM been adept at tackling the logistics of staffing, coordinating and managing logistics for OR activities? ● Five OR studies are underway in Benin, Mali, Cambodia, Senegal, and LAC. ● There appears to be some role confusion within the IM consortium as to who is in charge of tracking the Senegal study’s timeline and budget, which was developed in country and passed over to IM. ● KIs concurred that the Senegal study would benefit from having an IM project manager in country. ● Delays in obtaining USAID approval for laptops, tablets, and field and laboratory supplies have affected the Senegal Mass Drug Administration (MDA) study’s implementation schedule. ● University of Thies (UT) has been an outstanding research partner in Senegal. ● USAID expressed some disappointment with PSI’s not having stepped in when a stronger hand was needed for OR challenges. ● The partner model incorporates sufficient flexibility in the existing partner principles. It is often other issues such as the available budget, skillset or national registration which would prevent making a change, should one be needed. ● The Senegal MDA study would benefit from having an IM project manager present in country to help troubleshoot. ● IM’s OR experiences and challenges can provide useful learning to PMI for prevention of similar problems in future projects. ● Introduce some flexibility or exceptions into the partner principles to enable change of lead partner if needed. (PSI) ● Explore the possibility of placing a research manager in country until the Senegal study is completed. (IM) ● Refer to recommendation under 2e. above related to funding delays. ● Recommendations from the Senegal study include closer coordination between research partners (UCSF and UT) to monitor the budget and ensure that teams in the field have all the resources they need; continuation of weekly phone calls between UCSF and UT; and development of a plan for how various partners intend to use and publish study data. ● Frame the projects OR challenges and solutions as a useful ‘lessons learned’ close out deliverable. (USAID/PMI) 23 EQ FINDINGS CONCLUSIONS RECOMMENDATIONS EQ2i. Are PMP indicators agreed upon at the HQ and/or the country-level practical from a reporting perspective and are they useful from a programmatic perspective? ● Nearly all interview and survey respondents were positive about IM’s indicators. ● Some respondents felt that there were too many indicators or that some indicators were difficult to measure, even if the indicators themselves were relevant. ● There is consensus that some PMP indicators need clearer definitions or revisions. ● The indicators are clear and aligned with global standards for each technical component. However, they don’t reflect an overall holistic framework for health systems strengthening. ● There is general agreement that the PMP indicators are relevant, even though some interview and survey respondents feel that there are too many. ● There is not yet an indicator related to the integration of gender into the project. ● The project does not have indicators for internal operations, although this appears to be one of its outstanding achievements. ● IM would benefit from a conceptual framework that clearly identifies linkages and complementarities between all technical components and indicators for each. Such a framework would likely help to identify opportunities for an integrated approach across technical interventions and health system levels, including community-level. ● Consider ways to simplify and shorten indicators and emphasize outcomes and impacts over activities and outputs. Provide more guidance on how to use them. (IM) ● Use learning from IM’s gender focused activities and collaborations to develop suggested gender indicators for future projects. (USAID/PMI) ● Consider internal performance indicators for future projects. (USAID/PMI) ● Consider the development of an actual Theory of Change (ToC) in the next annual planning meeting that maps out assumptions that underlie the activities under each technical component and shows how they link with and complement one another. A ToC can also show where the project can best link with other PMI flagship projects. (USAID/PMI) EQ 3. GLOBAL RESULTS EQ3a. To what extent has IM achieved global level results laid out under each objective in the detailed program description of the award, including plans for and progress towards publications, documentation, and dissemination of best practices/lessons learned? What has IM developed in their Learning Agenda and other job aides, such as guidance on implementing SMC campaigns in the context of COVID-19? 24 EQ FINDINGS CONCLUSIONS RECOMMENDATIONS ● IM has been an active and credible participant, and sometimes a leader, in global level malaria meetings and conversations, at conferences, and on social media platforms in PY1-3. ● IM has engaged with Roll Back Malaria through various working groups, World Health Organization (WHO) and the SMC Alliance, which is linked with the Roll Back Malaria (RBM) Country/Regional Support Partner Committee (CRSPC)2 . ● IM partnered with Breakthrough ACTION to produce A Blueprint for Applying Behavioral Insights for Malaria Service Delivery: Methods and Frameworks for Improving Provider Behavior that provides steps for using insights into health provider behavior to improve the quality of malaria service delivery, and co-hosted a webinar on the tool with Breakthrough Action (BA). ● IM launched multiple social media platforms, including a project website, blog posts, a Twitter account, a presence on LinkedIn, and a Flickr photo library with over 500 photos. ● IM presented in English and French on COVID-19 adaptations to Integrated Community Case Management (iCCM) and SMC programming as part of a Child Health Task Force (CHTF) webinar series and produced learning briefs from SMC campaigns. ● IM partner Jhpiego plans to publish the Cameroon and Kenya gender analyses that were completed in PY4, and has submitted a gender-focused abstract to the ASTMH 2021 conference. ● Prime partner PSI has recently contracted with UCSF to provide technical support to countries for abstracts they wish to submit to ASTMH or for manuscripts submitted to peer reviewed journals. ● IM has produced a Learning Agenda through consultation with USAID PMI, to contribute to the body of knowledge on the most effective and efficient ways to deliver malaria services in four project intervention areas: Quality Assurance (QA), iCCM, MIP, and SMC. ● IM has been involved since PY1 in multiple, well-respected contributions to global malaria conversations and has plans in place to publish findings from project activities in both published journals and through ‘softer’ communication channels, including its website. ● IM has defined outputs for a project￾wide Learning Agenda. ● Strengthen opportunities to build the capacity of national colleagues in writing and publishing findings from IM project activities, using both global and national communication. (IM) ● Disseminate Learning Agenda findings and outputs as widely as possible at end of project. (IM) 2 The purpose of the Country/Regional Support Partner Committee (CRSPC) is to provide a platform to engage the RBM Partnership community in coordinating support to countries and regions as they execute their malaria control and elimination implementation programs. 1. INTRODUCTION 1.1 EVALUATION PURPOSE The mid-term evaluation (MTE) of the five-year (2018-2023) PMI IM Contract under the United States Agency for International Development (USAID) Bureau of Global Health (GH)/Office of Infectious Disease (ID)/Malaria Division (MAL) was conducted to inform the structure and content of current and future USAID/Presidents Malaria Initiative (PMI) investments in malaria Case Management (CM), prevention of Malaria in Pregnancy (MIP), and other malaria drug-based interventions. The IM MTE has the following objectives: 1. Assess and document progress toward achieving project objectives and whether desired results have occurred; 2. Determine the effectiveness and efficiency of project operations and management; 3. Capture lessons learned and identify key bottlenecks/gaps that can inform future PMI activities in CM, in the context of the PMI strategy. 1.2 EVALUATION QUESTIONS The evaluation was guided by three evaluation questions (EQs) and 18 sub-EQs, as shown below. EQ1. COUNTRY-LEVEL PERFORMANCE: To what extent has PMI Impact Malaria (IM) achieved the country-level objectives? EQ1a. To what extent has PMI IM achieved the technical and programmatic objectives described in annual country and core workplans and IM Performance Management Plan (PMP)? EQ1b. Is there evidence of in-country capacity improvements in malaria diagnosis and CM and prevention of MIP at various levels of the health system (national, regional, district, community), taking into account guidelines, training, supervision checklists? EQ1c. Do checklists and other tools capture useful data on the status and quality of CM? Are they appropriate and informative? Is implementation of Outreach Training and Supportive Supervision plus (OTSS+) disruptive to provision of services (does it take too much time)? EQ1d. Are results from checklists/other tools used by IM to make adjustments to training and supervision to improve quality? EQ1e. Has the development of the IM Data Hub and the associated efforts to access national HMIS data for PMP reporting resulted in tangible improvements to data use? EQ1f. Have Seasonal Malaria Chemoprevention (SMC) coverage and adherence objectives been met in areas where PMI IM has been supporting SMC implementation? EQ1g. Is there evidence of integrated management of key project interventions (CM and MIP) and has this integration strengthened their approach and delivery? EQ1h. Has collaboration with maternal health and Antenatal Care (ANC) services been strengthened in areas where IM is supporting MIP implementation and service delivery? 26 EQ2. MANAGEMENT: To what extent has PMI IM met the management requirements and functions outlined in the agreement, including planning, allocation of funds, coordination among the IM partners (Population Services International (PSI), Medical Care Development International (MCDI), University of California San Francisco (UCSF), and Jhpiego), staffing requirements, and in-country support? EQ2a. Has PMI IM Head Quarters (HQ) and PMI Contracting Officer Representative (COR) team oversight and management aided or hindered IM in accomplishing workplan objectives, both at central and country-level? EQ2b. Has coordination between IM and partners in country (PMI Resident Advisors (RAs), National Malaria Control Programs (NMCPs), other Implementing Partners (IPs)) aided or hindered IM in accomplishing country workplan objectives? EQ2c. Is in-country presence of IM staff sufficient and appropriate? EQ2d. Has IM been adept at adjusting to the rapid growth of country buy-in, from the original nine countries in fiscal year (FY) 2017 to 18 countries and two regional buy-ins in FY 2019? EQ2e. Has IM been able to hire staff, set up offices, launch activities, and continue activities on the agreed upon timelines? EQ2f. Has IM been adept at tackling the logistics of staffing, coordinating and managing logistics for SMC campaigns? Are the campaign activities in conflict with maintaining routine support for CM and MIP activities? EQ2g. Has IM been adept at tackling the logistics of staffing, coordinating and managing logistics for Therapeutic Efficacy Studies (TES) activities? EQ2h. Has IM been adept at tackling the logistics of staffing, coordinating and managing logistics for Operational Research (OR) activities? EQ2i. Are PMP indicators agreed upon at the HQ and/or the country-level practical from a reporting perspective and are they useful from a programmatic perspective? EQ 3. GLOBAL RESULTS: What results have been realized at the global level? EQ3a. To what extent has IM achieved global level results laid out under each objective in the detailed program description of the award, including plans for and progress towards publications, documentation, and dissemination of best practices/lessons learned? What has IM developed in their Learning Agenda and other job aides, such as guidance on implementing SMC campaigns in the context of COVID-19? 1.3 EVALUATION AUDIENCES The results of the IM MTE will be used by USAID, PMI HQ and mission staff as well as by IM project staff. 27 2. BACKGROUND The World Malaria Report 20203 presents a mixed picture of the status of malaria. Globally, there were an estimated 229 million malaria cases in 2019 in 87 malaria endemic countries, declining from 238 million in 2000. Between 2000 and 2015, global malaria case incidence declined by 27 percent, and between 2015 and 2019 it declined by less than 2 percent, indicating a slowing rate of decline since 2015. The World Health Organization (WHO) African Region, with an estimated 215 million cases in 2019, accounts for about 94 percent of the cases. Globally, an estimated 1.5 billion malaria cases and 7.6 million malaria deaths have been averted between 2000 and 2019. Most of the averted cases (82 percent) and deaths (94 percent) were in the WHO African Region. In 2019, in 33 moderate-to-high transmission countries in the WHO African Region, there were an estimated 33 million pregnancies, of which 12 million (35 percent) were exposed to malaria infection during pregnancy. The Global Technical Strategy (GTS) for Malaria 2016-2030 aims for a reduction in malaria case incidence and mortality rate of at least 40 percent by 2020, 75 percent by 2025, and 90 percent by 2030, from a 2015 baseline. However, the 2020 malaria case incidence of 56 cases per 1,000 population at risk, instead of the GTS target of 35 cases per 1,000 people at risk, indicates that globally the current trajectory is off track by 37 percent. PMI IM PROJECT BACKGROUND PMI, led by USAID and implemented together with the United States (U.S.) Center for Disease Control and Prevention (CDC), is the US Government’s (USG’s) primary vehicle for assisting malaria affected countries to scale-up proven malaria control and elimination interventions. Its original goal in 2005 was to reduce malaria-related mortality by 50 percent across 15 high-burden countries in sub-Saharan Africa. This called for a rapid scale-up of four proven and highly effective malaria prevention and treatment measures: insecticide-treated mosquito nets; indoor residual spraying; accurate diagnosis and prompt treatment with Artemisinin-Based Combination Therapy (ACT); and Intermittent Preventive Treatment in Pregnancy (IPTp). PMI partners with host countries and works with host countries in advancing malaria control and elimination efforts in accordance with their national strategic plans for malaria. The ultimate goal of worldwide malaria eradication by 2040-2050 aligns with the USG vision of ending preventable child and maternal deaths and protecting communities from infectious diseases. PMI’s first five-year project (2007-2011), Improving Malaria Diagnostics (IMaD), focused on building capacity for laboratory-based malaria diagnosis. The second five-year project (2012-2017), MalariaCare, focused on increased capacity for malaria CM - testing, diagnosis and appropriate treatment. PMI is currently working in 24 countries in sub-Saharan Africa as three programs in the Greater Mekong Subregion of Southeast Asia. PMI supports five flagships projects that address: 1. Improved access to and quality of CM, prevention of MIP, and improved access to new drug based approaches to malaria prevention (PMI IM project); 3 World malaria report 2020: 20 years of global progress and challenges. WHO 2020. ISBN 978-92-4- 001580-7. World Malaria Report 2020 (who.int) 28 2. Supply chain strengthening for malaria testing and treatment (USAID Global Health Supply Chain Program-Procurement and Supply Management (GHSC-PSM))4; 3. Malaria prevention through vector control (PMI Vector Link project); and 4. Strengthened country-level capacity to collect, analyze and use routine malaria health data and improved country-level ability to manage health information systems to serve malaria needs (PMI Measure Malaria); 5. Breakthrough ACTION and RESEARCH, USAID’s flagship programs for social and behavioral change. The purpose of the IM contract award is to provide implementation support services and technical assistance to countries to accelerate progress in comprehensive malaria facility and community service delivery including malaria CM, prevention of MIP, and other malaria drug-based interventions. IM has three key objectives: Objective 1: Improve quality of and access to malaria CM and MIP interventions. Objective 2: Improve quality of and access to other malaria drug-based approaches and provide support to pilot/scale-up newer malaria drug-based approaches. Objective 3: In support of Objectives 1 and 2, provide global technical leadership, support operational research, and advance program learning. The IM project builds on prior USAID investments to strengthen malaria diagnosis and CM, prevention and treatment of MIP, and related malaria service delivery efforts. By the end of Project Year (PY) 3 (2020), IM was operating in 18 countries across Africa and Asia and supporting USAID’s Bureaus for Latin America and the Caribbean (LAC) and Africa5 (see Figure 1 in the next page). IM country leadership is distributed across the four consortium partners as follows: PSI: Bureau for Africa funding, Burkina Faso (TES), Cambodia, Cameroon, Côte d’Ivoire, Lao People’s Democratic Republic (PDR), Madagascar, Malawi, Mali, Niger, Sierra Leone, and Tanzania Jhpiego: Burkina Faso (SMC), Ghana, Kenya, and Rwanda MCDI: Benin, Democratic Republic of the Congo (DRC), and Zambia UCSF: LAC, Senegal IM project’s expected outcomes are: ● A median of at least 80 percent of patients with suspected malaria receiving a diagnostic test; ● An average of 80 percent of confirmed malaria cases receiving effective malaria treatment according to standard national protocols; ● A 15 percent median increase in the percentage of pregnant women receiving two or more doses of IPTp for malaria during their last pregnancy; ● For each round of SMC, 80 percent of targeted children under age five receive a dose of SMC. 4 This is an integrated mechanism with a mandate that is broader than malaria supply management. 5 Benin, Burkina Faso, Cambodia, Cameroon, Cote d’Ivoire, Democratic Republic of the Congo, Ghana, Kenya, Lao PDR, Madagascar, Malawi, Mali, Niger, Rwanda, Senegal, Sierra Leone, Tanzania, Zambia 29 Figure 1. IM Country Buy-in Map 30 3. EVALUATION METHODS AND LIMITATIONS 3.1 EVALUATION METHODOLOGY An evaluation team composed of three persons, a Team Lead, a Malaria Specialist, and an Analyst, conducted the IM MTE between February and July 2021. The evaluation team conducted the entire evaluation virtually due to the COVID-19 pandemic restrictions on movement and gathering, including numerous virtual team meetings and data collection. The evaluation team used a mixed-method approach, as described in the Scope of Work (SOW); see Annex 1. 3.2 DATA SOURCES 3.2.1 Desk Review The evaluation team reviewed key project documents provided by PSI and USAID/PMI, including the IM project’s SOW, contract, core and country annual workplans, PMPs, annual reports, and other project performance data. Publications on the project’s website were also reviewed, as well as global malaria documents. The information obtained from the desk review helped to answer EQs. 1 and 3. 3.2.2 Key Informant Interviews The evaluation team conducted in-depth key informant interviews (KIIs) with 81 key informants (KIs) from various stakeholder groups, including: ● USAID/PMI management staff; ● Global malaria experts; ● IM project IPs; ● IM project HQ management and technical staff; ● Host country government stakeholders; and ● Country-level staff and IM project partners in Cameroon, Ghana, Kenya, Niger, and Senegal. For a full list of the KIs, see Annex 3. KIIs provided relevant information and data for all three EQs, except for Senegal, for which they provided information specific to the ongoing Mass Drug Administration (MDA) OR. The evaluation team developed separate KII guides for global/HQ-level interviews versus country-level interviews (see Annex 2). French language interpreters were used when needed with French speaking participants. All interviews were conducted virtually via Zoom calls. 3.2.3 Online Survey A bilingual (English and French) 15-minute survey was sent electronically to 308 country-level project staff, government officials, other key stakeholders, and PMI HQ backstops across all 18 IM countries. The survey questionnaire focused on EQs.1 and 2. Not all survey questions were relevant to all 31 respondents, and the survey questionnaire was designed accordingly, with “Not Applicable” as a possible response for those cases. Weekly and daily reminders, as necessary, were sent to potential survey respondents by USAID/PMI, the GH EvaLS team, and IM staff. The survey deadline was extended by an additional week to allow for a higher response rate. In the end, 156 responses (112 English/44 French) were received out of 308, a 51 percent response rate (see Annex 3 for a list of survey respondents). 3.2.4 Five In-Depth Country Reviews USAID/PMI suggested four IM project countries (Cameroon, Ghana, Kenya, and Niger) for an in￾depth review for the MTE. The in-depth country reviews were conducted to determine: • Whether IM project results were achieved according to country workplans; • Successes that should be replicated/continued and major contributors to these successes; • Major challenges or barriers to IM project implementation/scale-up of malaria CM and prevention of MIP; • Strengths and weaknesses of IM project management; • Level and types of capacity built in malaria diagnosis and CM, and prevention of MIP at the regional, district, and HF levels (evidence of increased knowledge, practice, and skill levels); • Level of capacity built in management of malaria at the community-level, where applicable; • Successes/weaknesses in coordination, planning and implementation of SMC campaigns, where applicable; and • Recommended areas of IM project focus in the future. USAID/PMI selected a fifth country, Senegal, for in-depth country review because the ongoing MDA OR in Senegal offers possible lessons for other IM countries that may conduct similar future research. 3.3 SAMPLE SELECTION The evaluation team in close collaboration with USAID/PMI and PSI, IM’s prime IP, did a purposive sampling of Key Informants (KIs) and survey respondents to ensure that the sample included KIs and respondents from all stakeholder groups. The in-depth review countries were selected by USAID/PMI to intentionally include two countries that have been supported across more than one malaria project (Ghana and Kenya) and two newer buy-in countries (Cameroon and Niger). As mentioned above, a fifth country (Senegal) was added because of the MDA OR currently being implemented in that country. 3.4 DATA MANAGEMENT AND ANALYSIS The evaluation employed a qualitative methodology through KIIs, supplemented by quantitative online survey data and IM project data, where available. 32 During qualitative data analysis, the evaluation team conducted a qualitative content and thematic analysis of the Key Informant Interview (KII) data. Specifically, the evaluation team developed a running list of emerging themes, organized by EQs. These themes provided the basis for thematic content analysis of the reviewed documents and KIIs, allowing the team to formulate the evaluation findings through an iterative process. Evaluation team members first compiled key findings and conclusions individually, and then compared, contrasted, discussed, and validated them against the findings of the rest of the team, to arrive at a consolidated agreed-upon set of findings. For the quantitative online survey data analysis, the survey data was translated into visual graphics using the survey platform’s internal capability. Data were also exported into Excel and simple descriptive analysis was performed. For the in-depth country reviews, data were obtained from: annual, quarterly, and other relevant reports and presentations provided by IM; the IM Data Hub; KIIs; Demographic and Health Survey (DHS)/Malaria Indicator Survey (MIS) data in the public domain; and Health Management Information System (HMIS) data provided by the four focus countries. The Senegal review did not require examination of HMIS data. The evaluation team then collated information across the primary and secondary data sources (PMP indicators contained in project workplans, annual reports, etc.), triangulated the results, and analyzed findings by EQ. 3.5 ETHICAL CONSIDERATIONS The evaluation team did not interview any project consumers. Further, no one under the age of 18 years was interviewed or participated in the MTE survey. All KIIs conducted during the evaluation began with an informed consent process and written documentation that contained:6 ● Introduction of facilitator/note-taker; ● Purpose of the evaluation/assessment; ● Purpose of interview/discussion; ● Statement that all information provided is confidential and information provided will not be connected to the individual; ● Right to refuse to answer questions or participate in interview/discussion and right to stop interview at any time; and ● Request for consent prior to initiating data collection (i.e., interview)/discussion. The survey questionnaire also included an informed consent statement as part of the introduction (see Annex 2). 3.6 LIMITATIONS The project is designed such that country USAID missions can buy-in to or opt out of the range of activities IM offers, leading to a large degree of variation in what IM was responsible for from country to country. While this design offers flexibility and permits better tailoring to country 6 KIIs were developed in alignment with the Common Federal Policy for Protection of Human Subjects in Research (the Common Rule) adopted by USAID. 33 needs, it presents a challenge for standardized evaluation of the program across countries. Additionally, country performance differed as a function of the amount of support the country had received prior to the IM project, with some countries having had much more work and experience with the fundamental interventions than others. As a result, variable targets, performance indicators, and inconsistent numerators and denominators across country data made it impossible to carry out any meaningful standardized analysis. The population-level malaria indicator data used by the evaluation team were from DHS and MIS that were a few years old and did not coincide with the timing of IM PY1-3. Also, the population and demographic data that were reviewed were projections from censuses that were, in some cases, conducted a decade ago. For the in-depth country reviews, the evaluation team utilized routinely collected data from each county’s HMIS, supplied by IM, to look for evidence of improvement in select IM project indicators as well as for evidence of improved public health outcomes and impact. In all cases, substantive data quality and completeness issues were readily apparent. The evaluation team stresses that improving the quality and completeness of data from these national systems is beyond IM’s mandate and IM has had very limited ability to improve those data. It should also be noted that, nonetheless, IM relies on these data to track its own performance indicators. As a result, the underlying quality issues of HMIS data not only limited the evaluation team’s ability to assess IM’s progress, but also limit IM’s ongoing ability to accurately track its own progress. Project targets are not representative of what is happening in a country overall, therefore it is impossible to describe with any accuracy the degree to which project activities have influenced national aims and objectives. Project data primarily describes how well targets were reached for key activities. The OTSS+ data also describes improved performances for a discrete number of trained HWs. However, contributions to the broader national effort are difficult to measure. The evaluation team had limited time to digest and analyze a large amount of project information. Wherever possible, evaluators have attempted to independently verify achievements rather than rely on IM reports alone. Activities highlighted in this report are intended to be representative or illustrative of those that IM has conducted, facilitated, or provided technical support for; the activities that are noted do not comprise a complete or exhaustive list of what the project has accomplished across PY1-3. The evaluation was necessarily conducted virtually, due to COVID-19 restrictions on movement and gathering. Relevant project-wide and country-specific data was nonetheless made available, and the evaluation team believes the MTE findings to be as accurate as possible. However, data quality and completeness issues were obvious in the HMIS-derived data, which could affect the accuracy of some findings. 34 4. FINDINGS This section presents the findings of the PMI IM MTE for each EQ and sub-EQ. 4.1 EQ1. COUNTRY-LEVEL PERFORMANCE: TO WHAT EXTENT HAS PMI IM ACHIEVED THE COUNTRY-LEVEL OBJECTIVES? IM’s first two objectives, shown below, are relevant specifically to the country-level. IM’s third objective (Provide global technical leadership, support operational research, and advance program learning) will be discussed later under EQ3. Objective 1: Improve quality of and access to malaria CM and MIP interventions. Objective 1 has the following core expected outcomes: ● A median of at least 80 percent of patients with suspected malaria receiving a diagnostic test; ● An average of 80 percent of confirmed malaria cases receiving effective malaria treatment according to standard national protocols; and ● A 15 percent median increase in the percentage of pregnant women receiving two or more doses of IPTp for malaria during their last pregnancy. Objective 2: Improve quality of and access to other malaria drug-based approaches and provide support to pilot/scale-up newer malaria drug-based approaches. Objective 2 has the following core expected outcome: ● For each round of SMC, 80 percent of targeted children up to five years in age receive a dose of SMC. For the purposes of this evaluation, evidence for country-level progress comes primarily from four of the four focus country in-depth reviews (Kenya, Ghana, Niger, and Cameroon). Additional information is available for the seven countries7 submitting data to the IM Data Hub by the end of PY3. Where possible, information from other sources have been used (e.g., HMIS, DHS/MIS). 7 DRC, Ghana, Kenya, Mali, Cameroon, Sierra Leone, and Rwanda. 35 Evaluation of IM indicators generally follows the standard approach to programmatic evaluation (https://www.cdc.gov/eval/approach/index.htm): Inputs – Indicators that measure the basics required to enable the program to function: funding, staffing, partnerships, etc. Process – Indicators that address whether the program was implemented as planned: Activities – updating guidelines, policies, Standard Operating Procedures (SOPs), training material, conducting training, OTSS+ visits Outputs – # people trained, # of Health Facilities (HFs)/people visited via OTSS+, number of people reached via SMC campaigns Outcomes – Indicators that measure whether program is achieving its stated short- or medium-term changes: changes in Health Worker (HW)/laboratorian performance indicators; changes in treatment cascade or IPTp coverage; coverage and adherence in SMC campaigns. Impact – Indicators that measure longer term changes: changes in malaria incidence, malaria cases in pregnancy, etc. NOTE: There was no explicit expectation from PMI that IM would achieve improvements in malaria morbidity or mortality. EVALUATION OF IM INDICATORS EQ1a. To what extent has PMI IM achieved the technical and programmatic objectives described in annual country and core workplans and IM PMP? INPUTS: Discussions regarding project inputs can be found in EQ2. PROCESS INDICATORS: Process indicators (activities and outputs) clearly show that IM has been very productive. ACTIVITIES: In terms of project activities, cumulative data from 11 IM-supported countries8 show that IM facilitated 25 rounds of clinically oriented OTSS+ visits and 27 rounds of lab-oriented OTSS+ visits. Checklists associated with the OTSS+ intervention have been developed or revised and translated into French as needed (see EQ1c and EQ1d for more information on checklists). IM has developed and, in some countries, implemented a digital system, the Health Network Quality Improvement System (HNQIS), that allows OTSS+ checklist data to be easily and rapidly entered, analyzed, and reported. 8 Cameroon, Côte d’Ivoire, DRC, Ghana, Kenya, Madagascar, Mali, Niger, Rwanda, Sierra Leone, and Zambia 36 IM has facilitated or contributed to the review, revision, and dissemination of relevant country policies, guidelines, training materials, job aids, SOPs, annual Malaria Operational Plans (MOPs), and other documents; participated in country-level technical working groups (TWGs); and updated existing country checklists. IM has also collaborated with District Health Management Team (DHMTs) and Regional Health Management Team (RHMTs) to schedule OTSS+ rounds, for which they also provided transport. IM has conducted a series of training workshops for health and laboratory personnel. In addition, during OTSS+ visits, additional in-service and follow-up training is conducted for a variety of topics in addition to the topics covered by the workshops. Annex 7 describes how the project rapidly and skillfully adjusted its training strategies and other key activities during the COVID-19 pandemic. OUTPUTS: Project outputs have also been substantial. Across the 11 IM-supported countries noted above, the 25 clinical OTSS+ rounds reflect 4,470 individual OTSS+ facility visits. The 27 lab-oriented OTSS+ rounds reflect 1,351 individual facility visits. In PY3, four countries that had been implementing earlier versions of OTSS for several years, conducted two or more rounds of clinical supervision (Ghana, Kenya, Mali, and Zambia), while five countries that adopted the updated IM OTSS+ approach (Cameroon, Côte d’Ivoire, Madagascar (for clinical OTSS+), Niger, and Sierra Leone) conducted one round and are launching second rounds in the first quarter of PY4. Through these OTSS+ visits, IM trained, mentored, or coached nearly 25,000 HCWs and supervisors (including 12,262 clinical staff; 3,426 laboratory staff; 7,376 additional clinical and laboratory staff in Zambia; 1,503 clinical supervisors; and 331 laboratory supervisors). IM also trained 5,010 Community Health Workers (CHWs). Lastly, IM trained 4,518 facility-based HWs specifically in MIP. IM also supported more traditional, classroom-style training, especially for malaria microscopy (MM). IM supported 1,185 laboratorians to attend basic Malaria Diagnostic Refresher Training (bMDRT) and trained 149 laboratorians to become trainers themselves. Forty-two laboratorians were supported through a national External Competency Assessment for Malaria Microscopists (nECAMM) and ten more went through WHO’s ECAMM program. IM supported 28 cycles of SMC over three years that reportedly reached 30,957,214 children under five years of age in three countries (Cameroon, Mali, and Niger)9. Additionally, technical assistance was provided to Ghana’s SMC efforts early on, but this was later dropped from the IM workplan because the project did not see a need for additional assistance beyond what the government and other partners were able to do. More discussion of SMC can be found in EQ1f below. OUTCOMES: Project outcomes have been highly variable across programmatic areas, countries, sub-country project areas, and time. Where progress has been made, it has mostly been modest. Outcome measures are addressed in more detail under EQ1b below. IMPACT: There was no specific expectation from PMI that IM would demonstrate impact on morbidity and mortality. Regardless, in terms of impact, it would have been unreasonable to expect major changes in morbidity and mortality after only 2.5 years or less of program implementation. In Kenya, numbers of confirmed malaria cases had been declining steadily for the last four years prior to the implementation of IM (between 2015 and Quarter (Q)1 2019). Since the start of IM implementation, that declining trend has continued. In Ghana, the opposite was seen – cases of confirmed malaria have been increasing since 2015 (or 9 Source: IM Data Hub provided to the evaluation team on May 21, 2021. 37 before). A data anomaly in 2019 makes it difficult to assess whether or not that trend continued since IM began, but it is possible that there has been a slight decrease. In Cameroon, although it appears as though cases are increasing since IM began, this is likely due to improved reporting due to IM or others' activities. In Niger, there was no discernable change from 2017 onwards. The number of confirmed malaria cases among pregnant women have generally followed the same pattern and trends seen for the overall number of confirmed cases of malaria. EQ1b. Is there evidence of in-country capacity improvements in malaria diagnosis and CM and prevention of MIP at various levels of the health system (national, regional, district, CHWs)? As mentioned above, project Outcomes related to capacity improvements show a high degree of variability across countries and sub-country project areas, across indicators, and over time. It is important to note that issues with data quality were apparent in the HMIS data of the four focus countries. Review of HMIS data in all countries showed instances of nonsensical data, presumably due to underlying quality problems. Additionally, KIs raised concerns over data quality. In Ghana, for example, one KI said that concerns over the quality of routinely collected data led the country to rely more heavily on population-based surveys (such as DHS and MIS). IM and other PMI country staff KIs in Niger reported that the information system needs to improve data quality at all levels. HMIS data completeness was also an apparent problem. For example, in Kenya, data on the number of women receiving three or more doses of IPTp were not collected at all until Kenya instituted the practice in 2020. In Niger, the proportion of suspected malaria cases receiving a diagnostic test was reported to be over 3,500 percent in one year, probably indicating substantial underreporting of suspected cases. DHS and MIS data are useful alternatives to cross-check HMIS data while also providing important additional data points not available within HMIS (such as ANC coverage). Unfortunately, these population-based surveys are normally conducted every five years or so. Ghana had recently conducted an MIS in 2019, which provided a reasonably recent data source. In Kenya, a DHS and MIS are in progress, but analyzable data sets are not yet available. The most recent population￾based surveys in Kenya with relevant data were conducted in 2014 (DHS) and 2015 (MIS). Cameroon conducted a DHS in 2018 and an MIS is scheduled for late 2021. In Niger, the last DHS conducted in 2017 was voided due to severe flaws in the quality of the data collected, leaving the 2012 DHS as the only available population-based data source. MALARIA IN PREGNANCY IM’s strategies for improving MIP service delivery include support for policy change and implementation, increased ANC attendance, improved quality of care for MIP prevention and treatment, supply chain coordination with GHSC-PSM and other partners, and strengthened routine monitoring and evaluation (M&E), including use of data for decision making. IM uses these strategies to increase the percentage of pregnant women receiving an insecticide-treated mosquito net (ITN) and three or more doses of IPTp through the ANC platform, as well as confirmation of infection among pregnant women with suspected malaria followed by appropriate treatment or referral of those who test positive. MIP activities are led by IM sub-partner Jhpiego. 38 ACTIVITIES AND OUTPUTS: IM has provided technical assistance related to MIP in eight countries.10 Examples of IM activities related to MIP during PY3 include: ● Training and OTSS+, focusing on the WHO-recommended three-pronged approach to MIP (ITNs through ANC, at least three doses of IPTp, and CM of MIP); ● Implementing an MIP module within the OTSS+ checklist (Cameroon, Côte d’Ivoire, Mali, Niger, Rwanda, Sierra Leone, and Zambia); ● Adding a coaching and mentorship component to OTSS+ (Cameroon, Côte d’Ivoire, Ghana, Kenya, and Sierra Leone); ● Training OB/GYNs on the national directives and guidelines (Niger); ● Supporting the NMCP and the Directorate of Reproductive and Child Health to reinvigorate the MIP TWG (Sierra Leone) and helping to launch a new MIP TWG (Niger); ● Supported the NMCP, through the MIP TWG, to conduct a rapid assessment in 28 Western Area HFs, 13 of which were in the private sector, to identify barriers to MIP access (Sierra Leone); ● Facilitated development of a study of Group ANC (G-ANC) to assess its impact on IPTp uptake and investigate the feasibility and acceptability of using pregnant women as a sentinel surveillance population (Benin); ● Published a methods and frameworks guidance tool in partnership with the USAID-funded Breakthrough ACTION (BA) project. The tool provides steps for using insights into health provider behavior to improve the quality of malaria service delivery. IM co-hosted a PMI￾moderated webinar to help people understand the tool and encourage its uptake. Supported planning for a study of group ANC in Benin11 and ANC surveillance in Mali, both scheduled to launch PY4 Q1. OUTCOMES: In Ghana, the proportion of women attending ANC who received an ITN at ANC ranged from 88.2 percent in Q2-4 2019 to 92 percent in 2020. In spite of relatively high overall ANC coverage rates and the high percentage of ANC attendees receiving ITNs, receipt of IPTp112 has remained relatively stable over the years reviewed (2015 to 2020), varying little between 65 percent and 75 percent. The rate decreased from 73 percent in Q2-4 2019 to 66 percent in 2020. The proportion of women “left out” of the IPTp intervention (i.e., those not attending ANC at all plus those attending ANC but not receiving IPTp1) was estimated using ANC coverage rates derived from the 2029 MIS. This proportion has ranged from 26.9 percent in Q1 2019 to 35 percent in both 2016 and 2020. In Kenya, after a period of increasing IPTp1 coverage rates from 2015 (52.5 percent) to 2017 (84.7 percent), the proportion of women receiving IPTp1 plateaued or declined from 2018 to 2020 (ranging from a high of 79.7 percent in Q2-4 2019 to a low of 69.6 percent in 2020) (Kenyan HMIS). In spite of relatively high ANC coverage,13 “Left Out” rates have been increasing from 2017 (17.9 percent) to 32.5 percent (2020). It should be noted, however, that this estimate is based on ANC coverage rates derived from the 2014 DHS and the 2015 MIS surveys. 10 Cameroon, Côte d’Ivoire, Ghana, Mali, Niger, Rwanda, Sierra Leone, and Zambia 11 “Assessing the Impact of G-ANC on IPTp uptake in Benin and the Feasibility and Acceptability of Pregnant Women as a Sentinel Surveillance Population.” 12 One dose of IPTp. 13 The 2014 DHS and 2015 MIS estimated the average ANC coverage rates within the IM project areas to be 96.9 percent, with an average of 2.1 percent of women who had been pregnant within the last 3 years reporting no ANC visits and 57.9 percent reporting having had 4 or more ANC visits. 39 Prior to IM starting in Cameroon, the IPTp1 coverage rate for the IM project areas was about 82 percent. After reaching slightly over 92 percent in 2019, that rate declined to 65.3 percent in 2020 and 52.6 percent in Q1 2021. In Niger, project data show evidence of improvements in quality service delivery, especially for pregnant women. IPTp1 coverage rose from a low of 76.7 percent in 2017, prior to the IM project, to 84.5 percent in 2020. One KI in Niger reported an observed increase when comparing with the earlier participation in ANC services in the IM target areas; it is unclear whether data confirm this. IPTp dropout rates remain high in all four countries, especially in Kenya where the proportion of women receiving three or more doses of IPTp (IPTp3+) in 2020 was 19.3 percent. Ghana and Niger performed better, with 49.4 percent and 59 percent of women receiving 3+ doses of IPTp, respectively. In Cameroon, while the proportion of IPTp3+ was reported to be 69.8 percent, a rate that was higher than the reported for IPTp1 and IPTp2 rates, indicating either an artifact due to better reporting, or data quality issues. Implementation of MIP has met challenges within IM. However, they are mostly challenges that would exist whether MIP was embedded within a malaria service delivery project or not. Many factors, such as cultural and gender norms, weather, transportation and distance, security – many beyond IM’s control – influence MIP outcomes, and it is virtually impossible to separate out the various influencing factors. Figure 1. below shows how survey respondents rated IM’s success in achieving the project’s Objective 1: improving the quality of and access to malaria CM and MIP interventions. Figure 1: Survey Respondents’ Rating of Success in Achieving Objective 1 in PY1-3 MALARIA DIAGNOSTICS IM’s objective has been to (1) improve access to quality malaria diagnostic services, and to increase the proportion of people with suspected malaria who are tested before receiving treatment; (2) improve the competency of HWs to perform diagnostic tests (Rapid Diagnostic tests (RDTs) or microscopy); and (3) increase the proportion of febrile cases that are classified correctly as having malaria or not. The project has given attention to improving microscopists’ ability to correctly identify malaria parasite species and to accurately estimate parasite counts. 40 MCDI, the leading partner for laboratory diagnostics, is active in 14 of the 19 IM countries and leads the overall IM project in three countries (DRC, Benin, and Zambia). ACTIVITIES AND OUTPUTS: As described in EQ1a, IM supported 27 lab-oriented OTSS+ rounds in 11 IM-supported countries, reflecting 1,351 individual facility visits. Through those lab￾oriented OTSS+ visits, 3,426 laboratory staff (and some unspecified portion of 7,376 clinical and laboratory staff in Zambia) and 331 laboratory supervisors received training. Additionally, IM supported 1,185 laboratorians to attend bMDRT and 149 were trained to become trainers themselves. Forty-two laboratorians were supported through a nECAMM and ten more went through WHO’s ECAMM program. OUTCOMES: The evaluation team looked at the proportion of suspected malaria cases receiving a diagnostic test as a primary outcome indicator for IM’s diagnostic work. In Ghana, the proportion of suspected malaria cases getting a parasitologic test was increasing in the years prior to IM (from 70 percent in 2015 to 91 percent in 2018). Since IM started, that trend has continued, increasing from 92.6 percent in 2019 to 93.5 percent in 2020.14 In Kenya, data quality issues pose a problem for assessing changes in proportion of suspected malaria cases that receive a parasitologic test. Specifically, the reported number of suspected malaria cases that received a parasitologic test far exceeded the reported number of reported suspected malaria cases (in 2015, by more than 600 percent on average across PMI IM project counties). Since IM began, the reported proportion of suspected malaria cases receiving a parasitologic test was 76.6 percent in Q2-4 2019 and 74.2 percent in 2020). In Cameroon, the proportion of suspected cases receiving a diagnostic test was reported to be 100 percent in 2017 and 2018 (prior to the start of IM). Since 2019, that proportion has remained quite high (between 98 percent in 2019 to 97.1 percent in 2020). In Niger, data quality and completeness issues prevent a reliable sense of the situation prior to the start of IM. As mentioned previously, the proportion of suspected cases receiving a diagnostic test was reported to be over 3,500 percent, likely indicating substantial underreporting of suspected cases. That proportion has remained above 100 percent until 2020, after which it was reported to be a more believable 95.1 percent. An additional three countries reported some malaria diagnosis data into the IM Data Hub (Mali, Rwanda, and Sierra Leone). The proportion of suspected cases receiving a diagnostic test ranged from a low of 81.9 percent (Mali, 2020) to a high 100 percent or more (Rwanda, 2020; Sierra Leone, 2019). Overall, evaluation of the quality of MM suggests ongoing challenges. In Ghana, for example, a national ECAMM was conducted for six newly created regions in 2021. Of 29 laboratorians assessed, 20 received the lowest rating, and five received the next lowest rating; the remaining four were determined to have adequate skills to be used as trainers themselves. Evaluation of competence in diagnostics was conducted via OTSS visits to facilities over the course of five OTSS rounds, during which 2,953 staff received on-site training. Observation scores for laboratorians use of RDTs were relatively high, ranging from 85.9 percent to a low of 56.9 percent. Among 98 HFs assessed in 2020, all regions had an average competency in malaria rapid diagnostic tests (mRDT) use scores above 90 percent.15 At the HF level, the percentage that scored 90 percent or higher on assessments of RDT use rose from 59 percent to 75 percent. Conversely, the 14 IM Data Hub and HMIS data. 15 Ghana Health Service Institutional Care Division - Report of Laboratory Outreach Training & Supportive Supervision Round 17 and Proficiency Testing Scheme Round 5. 41 percent of facilities that scored 90 percent or higher on assessments of slide staining and reading were reportedly quite low, ranging from 22 to 39 percent. MALARIA CASE MANAGEMENT IM priorities for malaria CM include revamping national guidelines to align with global standards, strengthening capacity of the ministries of health (MOHs) to monitor and improve the quality of facility-level care through training and OTSS+, building capacity of CHWs to expand access to quality community-based services through Integrated Community Case Management (iCCM), launching initiatives to reinforce quality improvement of malaria CM delivered through the private sector, and strengthening systems for improvement of malaria CM. ACTIVITIES & OUTPUTS: IM supported the strengthening of clinical CM in ten countries.16 In PY1, IM worked with NMCPs to assess whether national policies and norms were aligned with global and PMI guidance, and assisted in revising them as needed. In PY2, IM supported the development of training curricula and job aids tailored to country priorities and used tools to develop a cadre of HWs trained in diagnosis and CM, including severe malaria, and the prevention and treatment of MIP. In PY3, eight countries finalized updated guidelines and training curricula that were aligned with global policies and best practices. For iCCM, IM reinforced recruitment, training, and supervision of CHWs by district and HF facility staff, supported regular check-ins visits of CHWs at their supervising facility, and supported community mobilization activities to build community support for the role of CHWs and bolster use of iCCM services. IM updated policies and training for iCCM in five countries (Cameroon, Côte d’Ivoire, Mali, Niger, and Rwanda) and then trained 2,697 CHW using the updated iCCM curriculum. IM focused on providing logistical and technical support for trainings, along with support for supervisory visits of CHWs and regular CHW check-ins visits to their catchment health facility. (PMI funds are excluded by Congressional mandate from being used for activities that don't benefit malaria – e.g., Water, Sanitation, and Hygiene (WASH), FP, immunization - with the exception of integrated activities – e.g., training that includes training in malaria. Even in those cases, Technical Assistance (TA) support would only be for the iCCM component of that training.) IM’s approach has been to bring stakeholders together on cascade of care, better integrate iCCM programs into the national primary health care system and help to establish sustainable structures that the government can continue to support in the future. A pilot activity in Kenya focuses on deploying CHWs already engaged in iCCM to promote attendance at ANC using standard messaging and home follow up. Reportedly deliverables have not yet been defined for the iCCM activities. Evaluators are not clear what IM’s or PMI’s ultimate goals are for iCCM, and it is unclear whether this approach aligns to the OTSS+ Supportive Supervision (SS) model in how it supports SS for CHWs working in remote areas outside of HF catchment areas. USAID/PMI expressed the hope that the creation of an iCCM toolkit (see EQ3) will identify other capacity strengthening/systems strengthening opportunities. A PMI KI expressed interest in exploring models that pay CHW to find active cases that would have otherwise gone untreated. IM also supported NMCPs to improve quality of severe malaria CM in eight countries. (Cameroon, Côte d’Ivoire, Ghana, Kenya, Mali, Niger, Sierra Leone, and Zambia) through improvements in community-level pre-referral care, referral practices, and facility-level care. KIs in Niger report that IM has been instrumental in building capacity of laboratory technicians for parasitological testing that matches WHO norms, and a quality assurance (QA) guide was developed. 16 Cameroon, Côte d’Ivoire, DRC, Ghana, Kenya, Madagascar, Mali, Niger, Sierra Leone, and Zambia. 42 In Ghana, IM supported the implementation of an NMCP Clinical Health Officer (CHO) internship program, designed to improve CHO provider skills, particularly in management and referral of severe febrile illness. Sixty CHOs participated in a two-week program that involved working with a mentor in a referral facility who observed and coached them during patient management. Pre￾and post-internship assessments demonstrated substantial improvement of CHO’s malaria knowledge, including severe malaria pre-referral and referral knowledge (increasing from 37 percent to 70 percent) and an increase in patient assessment skills (from 25 percent to 74 percent). Reportedly, this program will be a model from which IM will develop a standardized curriculum for similar CHW internships for use within other PMI countries. IM Sierra Leone partnered with the NMCP to roll out the use of artesunate rectal capsules across five districts and implemented national and district level training of trainers (TOT) in pre-referral interventions, effective referrals, severe malaria CM, data entry, and best practices for filling out referral forms for 116 government personnel. In PY4, IM will cascade this training to 1,265 HF providers. Of note, in Cameroon one KI reported that there were financial incentives for HFs to over￾diagnose severe malaria; patients with severe malaria must pay for treatment, which generates income for HFs that they wouldn’t receive if treating for uncomplicated malaria. Also, a government informant in Cameroon stated that patients don’t always trust test results and ask for treatment anyway. A KI in Cameroon reported that HWs also do not always trust negative test results. OUTCOMES: As with other indicators, there was a large degree of variation in indicators of competency in malaria CM across countries, sub-country project areas, and over time. A key indicator assessed in the focus country in-depth reviews included proportion of malaria cases receiving appropriate treatment, derived from routinely collected HMIS data. In Ghana, the proportion of uncomplicated malaria cases that received the appropriate first-line antimalarial treatment was mostly over 100 percent prior to IM (albeit declining somewhat from 147 percent in 2015 to 92 percent in 2018). Since IM, that proportion dropped to only 78 percent in 2019 but rose again to 93.5 percent in 2020).17 In Kenya, prior to IM, the number of “uncomplicated malaria cases” that received appropriate malaria treatment was consistently greater than the number of confirmed malaria cases. That trend has continued since IM was implemented, with proportions over 200 percent. Given the recognized and ongoing issues with data quality reported by KIs, this is unlikely to be primarily an issue of overuse of antimalarial treatment (although that may well be occurring). Regardless, at this point, this appears to be an unreliable indicator of IM progress. IM assessed six additional indicators of HW performance, obtained from OTSS data. The indicators include: (1) competence in classifying cases; (2) management of uncomplicated malaria; (3) adherence to treatment guidelines for positive test results; (4) adherence to treatment guidelines for negative test results; (5) treatment of MIP; and (6) prevention of MIP (Figure 2 below). Definitions for these indicators, from the FY2020 PMP, are as follows: “Classifying” = “Percentage of observed health workers demonstrating competence in correctly classifying cases as not malaria, uncomplicated malaria, and severe malaria” defined as “Number of 17 IM Data Hub and HMIS data. 43 observed health workers who score 90% or greater on supervisory or quality improvement checklists measuring competency in malaria classification /Total number of health workers who received supervision for clinical observation” “Mgmt uncomp” = “Percentage of observed health workers demonstrating competence in management of uncomplicated malaria” defined as “Number of observed health workers who score 90% or greater on the clinical observation checklist/Total number of health workers who received supervision for clinical malaria case management” “Compliance pos” = “Percentage of observed health workers demonstrating compliance to treatment according to WHO guidelines for cases with positive malaria test results” defined as “Number of observed health workers who provided the recommended treatment to patients with positive malaria test results during clinical assessment visits measured through direct observation during supervision visits/Total number of health workers who were assessed for compliance to positive malaria test results during supervision visits” “Compliance neg” = “Percentage of observed health workers demonstrating adherence to negative test results according to global standards” defined as “Number of observed health workers who do not provide malaria treatment to patients with a negative test result/Total number of health workers who were assessed for adherence to negative malaria test results during supervision visits” “Comp Tx MIP” = “Percentage of observed health workers demonstrating competence in treatment of MiP” defined as “Number of observed health workers who score d 90% or greater on the case management section of the MIP at ANC checklist/Total number of health workers who were observed for MIP case management” ”Comp Pv MIP” = “Percentage of observed health workers demonstrating competence in prevention of MiP” defined as “Number of observed health workers who scored 90% or greater on the prevention section of the MIP at ANC checklist/Total number of health workers who were observed for MIP prevention” Figure 2: Trends in HW Performance Over Time in Four PMI Focus Countries 44 While performance was relatively high in Ghana and Kenya across all indicators (ranging from mid-70 percent to high 90 percent for both countries), performance in Cameroon and Niger was quite low. In Niger, indicators ranged from a low of 25 percent (competency in prevention of MIP) to a high of 68 percent (adherence to guidelines for a negative test). The data from Cameroon showed quite high values for one indicator (adherence to guidelines for a negative test at 87 percent and 94 percent for the two quarters with available data), but surprisingly low values for all other indicators (most below 10 percent and one observation of 13 percent). The exceedingly large discrepancy between the highest and lowest value raises questions about the reliability of the data overall. In all countries, more data points would be required to smooth out quarter-to-quarter variation and get a good sense of an underlying trend. Ghana had data over the longest period of time (five quarterly reports). Kenya had three quarters’ worth of data, while Cameroon had two and Niger had only one. Indicators generally showed a high degree of volatility over time and between indicators. Trends in the Ghana data showed improvements over time for most indicators, however, data over a longer period of time would be needed to better ensure the upward trends were sustained. As mentioned above, in Kenya, Cameroon, and Niger, there were not enough data to discern any trends over time. IM also supported NMCPs in selected countries to conduct quality assurance activities in targeted private sector outlets. In Côte d’Ivoire, IM trained 83 health care providers from the private non￾profit sector, including 27 physicians, 35 nurses, and 21 midwives, alongside their public sector counterparts. These providers received training on the latest national guidelines for malaria prevention and control. IM Côte d’Ivoire will conduct OTSS+ visits at private facilities in the coming PY. EQ1c. Do checklists and other tools capture useful data on the status and quality of CM and MIP? Are they appropriate and informative? Is implementation of OTSS+ disruptive to provision of services (does it take too much time)? EQ1d. Are results from checklists/other tools used by IM to make adjustments to training and supervision to improve quality? The OTSS+ model comprises a digitized package of OTSS+ checklists that has enabled project buy-in countries to conduct facility-level supportive supervision and on the job training and mentoring of HW competencies related to malaria testing, treatment, CM, and prevention of MIP. The checklist package includes modules on managing both uncomplicated and severe malaria, preventing and treating MIP, and performing mRDTs and MM. Checklists are tailored to needs and data management protocols of specific countries and have been translated into both English and French.18 Two important aspects of the OTSS+ platform are that (a) the system tracks performance at the HF level, not of individual HWs; and (b) the system is designed not only for in-service and mentoring, but also includes modules that examine other aspects of the HF, such as equipment (e.g., whether microscopes are available and what their condition is), availability of consumables (medicines, log books, job aids), and information systems. As mentioned under EQ1a, a digital data entry platform, HNQIS, has been designed to enable real-time feedback for rapid development of quality improvement action plans and allows for 18 IM FY20 Annual Report 45 easier tracking of progress over repeat visits. The OTSS+ data allows IM and countries to identify both strengths and gaps in HW performance and HF malaria service delivery. ACTIVITIES & OUTPUTS: IM has developed, validated, and implemented seven OTSS+ checklists, including: 1. Assessing MIP Interventions at ANC; 2. Assessing RDT use; 3. Assessing MM skills; 4. Assessing clinical management of patients suspected of having malaria; 5. Assessing management of severe malaria; 6. Evaluation form for laboratory supervisors; and 7. Assessing HF readiness. Altogether eleven IM-supported countries aligned their supervision approach with the OTSS+ package of checklists by end of PY3. Some countries have had much more experience using the OTSS model than others; while a few have been using OTSS for many years, others have only just begun using the approach, which was reflected in differences in their overall performance and amount of OTSS data available at the time of the evaluation. As previously mentioned, IM has conducted 4,470 clinical OTSS visits and 1,351 lab-oriented OTSS visits, reaching in total nearly 25,000 HWs to date. IM has also begun to apply the OTSS+ intervention to private sector facilities in Côte d’Ivoire. OUTCOMES: KIs who expressed an opinion were fairly unanimous in their praise of the OTSS+ system as a whole, and its use of checklists. KIs felt that data from OTSS+ were very helpful in allowing for providing rapid feedback to HWs and HFs, assisting with developing action plans, and serving as basis for follow-up to ensure progress being made on issues that had been identified. In both Kenya and Ghana, KIs reported that OTSS+ data are very useful in identifying HFs in need of extra support. One informant suggested that when they identify a specific poor performance problem in the HMIS data, they then use OTSS+ information to identify the HFs that appear to be the source of the majority of that problem, and then focus their efforts on those HFs. KIs from all stakeholder groups also identified digitalization of the checklists as a major step forward. A number of countries used paper-based systems previously and noted that data were very slow to be analyzed and therefore infrequently used. Through the HNQIS system, supervisors can upload and analyze data quickly, allowing them to make decisions about problem areas while still on-site. One HQ/Global KI noted that previous projects used data mainly for monitoring activities and not for actual decision-making. IM has been advocating for and supporting greater data use at all levels and is especially interested in improving the capacity of NMCP personnel to use data to understand their own program effectiveness and to make adjustments accordingly. One KI in Kenya said “…previously, staff looked at CM indicators as independent and unlinked – now they see them as a cascade with one impacting the next…”. Another KI in Kenya reported “…previously, HWs looked at data as a ‘box to tick’ but now they are beginning to see the value of the data they collect....” KIs from Ghana and Kenya suggested that high staff turnover among HW has been a challenge, requiring continuous repetition of orientations, training, planning. One KI from Kenya said “…by the time you do a training assessment, identify gaps and put together a plan, staff have turned over and info no longer current....” Concerns were raised about the OTSS+ model, alongside wide appreciation for the approach: 46 OTSS+ is a time, resource, and effort intensive activity. In Ghana, OTSS+ visits are reported to take approximately five hours for hospitals and polyclinics and three hours for health centers, clinics and community health posts (CHPS compounds). Accordingly, OTSS+ teams can visit one large facility or two smaller facilities in a day (although travel distances between facilities can change the number visited per day). In Ghana, data collection and mentorship, supervision, coaching, and training occur simultaneously so it was not possible to estimate the proportion of time spent on data collection versus supportive supervision. In Kenya, OTSS+ visits are reported to take approximately one day for large HF/hospitals and half a day for dispensaries and health centers. It has been estimated that perhaps a third of the time spent during an OTSS+ visit relates to data collection while two thirds is devoted to targeted training and mentorship. A number of KIs raised concerns about sustainability of the intervention. One HQ/Global KI expressed skepticism about OTSS+, was unclear about what it was truly accomplishing and whether any lasting change would be achieved. A few country-level KIs echoed those concerns. Two KIs in Kenya felt that the approach wasn’t sustainable, especially as financial responsibility transitioned to the country. In Kenya, given its highly decentralized system, funding of OTSS+ visits would, in theory, be left to the counties. While IM is operating in eight counties, Kenya has a total of 47 counties, prompting one KI to wonder how OTSS+ services would be paid for should it be expanded to all counties in need. The KIs also reported that there is no standardized procedure to guide when and how the OTSS+ visits are conducted. Scheduling is left to the district/county or regional teams to determine these procedures, which are probably within their regular supervision plans. A simple SOP could help to institutionalize and maximize effectiveness of the OTSS+ intervention. Furthermore, a good SOP could help define the level of effort (LOE) and time needed for collecting data, mentoring and tutoring, and assessing the presence and conditions of consumables, equipment, and support systems. Experience to date suggests that Kenyan counties have been reluctant to build support into their budgets. One KI in Kenya said that while counties have developed line items within their budgets for HIV/AIDS, malaria is typically lumped with other illnesses, so there are no dedicated funds to support malaria-focused supportive supervision. Further, the KI reported that there were efforts in both lake and coastal malaria endemic counties to prioritize malaria within their budgets, but progress was slow. Notably, Ghana has reportedly been using OTSS+ (or something similar) for ten or more years. As a result, concerns over sustainability were not raised by KIs. IMPACT: OTSS+ data generally show that performance was relatively high in some countries reviewed (Kenya and Ghana) and there were modest improvements for some indicators. However, more data on the effect of the OTSS+ approach over a longer period of time are needed, especially to demonstrate that such an improvement in HW performance translate into meaningful public health impact. Even then, attribution of any public health impact directly to IM activities would be difficult given the current project design. That said, IM is partnering with WHO to develop a handbook to support development of a roadmap for malaria CM in the private sector and an operational manual for supportive supervision based on lessons learned from OTSS+. IM is also planning for an external, independent evaluation of the OTSS+ model that may lend further support for its use in a variety of other countries. 47 EQ1e. Has the development of the IM Data Hub and the associated efforts to access national HMIS data for PMP reporting resulted in tangible improvements to data use? The Data Hub is IM’s project monitoring system. It serves multiple purposes: (1) consolidating results for IM PMP indicators across countries into a central data warehouse; (2) enabling project staff (at country and global levels), NMCPs, and PMI staff to access and use these data to make evidence-based decisions to improve malaria service delivery; and (3) strengthening the global evidence base for decision-making for IM and PMI. ACTIVITIES & OUTPUTS: The process of setting up countries in Data Hub required substantial effort. The IM project team developing the system goes through several iterative steps and phases, including: ● Framework setting: This phase included collating national reporting documents, finalizing country PMP indicators, and receiving the necessary data and metadata from government counterparts. ● Metadata mapping: This phase included setting up health system administrative levels and facilities using data provided from government counterparts, triangulation of subnational and facility names and information between source documents, identification of metrics and disaggregation levels within reporting forms, mapping reporting forms to government HMIS metrics, reviewing IM country-specific PMP indicators, and mapping metrics and disaggregation levels between government source data and IM destination data. ● Setting up data entry forms: Data entry forms were then set up to allow IM country teams to report on global- and country-specific PMP data, including at least one form to capture HMIS data (and its disaggregation, including differentiating between facility- and community-based data, where applicable). The forms capture all thematic areas (diagnosis, treatment, and MIP), with individual forms to collate training data, to collect OTSS+ data, to record for SMC results (where applicable), and to capture technical leadership across administrative levels. ● Import script: The final phase included creating and testing the import script, specifically for HMIS data. This enables the automatic processing and entering of HMIS data that is required for PMP indicators without the need to manually enter per facility. In order to report on the 44 global IM indicators and 13 country-specific PMP indicators, as well as capture OTSS+ data, as of the end of PY3 the Data Hub included the following: 1. A total of 50 data entry forms; 2. Over 19,000 data elements (e.g., numerators, denominators, checklist questions, scores); 3. Almost 300 unique disaggregation combinations; and 4. Almost 20,000 unique administrative units (including subnational levels, such as regions, districts, and facilities). By the end of PY3, seven IM-supported countries were reporting their data into the Data Hub (DRC, Ghana, Kenya, Mali, Cameroon, Sierra Leone, and Rwanda). In Madagascar and Niger, transition of data into the Data Hub was initiated in PY3 but has not yet been completed. The system is aligned with country-level PMPs and captures HMIS and SMC data for IM-supported areas, as well as data on IM-supported training, contributions in technical leadership, and OTSS+ rounds. 48 IM created a series of dashboards that offer the user a number of options to view and analyze data. The OTSS+ dashboards provide details about results against the supportive supervision indicators, details about facility-level performance, results for each component of the checklist, and outlines the action plan jointly developed by the supervisor and HF team. This level of detail supports the NMCPs and IM teams to identify which facilities are performing well and those that need additional support, allowing country teams to prioritize facilities most in need for support for upcoming OTSS+ rounds and enabling a more accurate picture of competency and quality of care performance in IM-supported areas. IM also developed global thematic area dashboards to provide project-level summaries of PMP indicator data aligned with specific thematic areas, based on the key questions of interest, and presented in a way to address these questions and monitor changes. To enable monitoring of trends in OTSS+ data from the beginning of the project, IM has digitized the OTSS+ checklists using other tools and apps, such as HNQIS and KoboCollect. The seven OTSS checklists described in EQ1c & EQ1d have been digitized and directly feed into HNQIS and then into the Data Hub. As of February 2021, eight countries were submitting data to the IM Data Hub (Cameroon, DRC, Ghana, Kenya, Madagascar, Mali, Rwanda, and Sierra Leone); one country (Niger) is estimated to be fully operational in 2021; two countries (Malawi, Zambia) are scheduled to start in 2021; one country (Tanzania) is in start-up phase, and one country (Côte d’Ivoire) is closing operations in late 2021. IM is engaging in discussions with four countries about integrating its information systems into the national systems (Ghana, Kenya, Madagascar, and Zambia). Concerns have been raised about whether the LOE invested in digitalization of the OTSS+ checklists is justifiable. One HQ/Global KI reported that the level of effort needed to digitalize the checklists was, in fact, minimal (a few weeks). Roll-out, however, was complicated and time￾consuming. Implementation of the checklists at the country-level requires a process of consultations with relevant stakeholders, gaining consensus on and finalizing the content, validating the final checklist in-country, piloting the checklists and making revisions as needed, and obtaining final government approval to move forward. In some countries, this process took a year. OUTCOMES: A detailed discussion of the perceived value of OTSS+ and its associated checklists (especially as pertains to the fully digitalized checklists and HNQIS) can be found in EQ1c and EQ1d. Briefly, KIs were nearly unanimous in their enthusiastic support of IM’s efforts to digitalize OTSS+ checklists and of the benefits that having real time data to inform supportive decision￾making brought to the project. Many KIs from the focus in-depth review countries (especially Kenya and Ghana) reported that data coming from the OTSS+ system have encouraged greater data use at all levels. KIs reported that data are routinely used in data review meetings, and to inform planning at sub-national, national, and global levels. KIs also noted that IM has assisted improved data use at peripheral HFs that are not able to access digital systems due to lack of connectivity. For this, IM is supporting the implementation of wall charts that allow local HWs to track, visualize, and use their own data. IM has encouraged data use at all levels and this remains a stated priority for the remainder of the project. Few KIs had criticisms of IM’s information systems. Two KIs in Kenya raised concerns over data fragmentation within the country, citing frustrations when trying to pull necessary information from many different data systems. One KI noted that the Kenyan government was quite sensitive about such fragmentation, wanting partners to help build and improve the country’s system rather than developing parallel systems. A number of KIs, mostly from the host country governments, were either unaware of IM’s Data Hub, or complained about not having access to IM’s data 49 systems. It is unclear whether this is true or if they were just not well informed about how to access them. KIs have raised concerns about the future of IM’s data systems. In general, the malaria indicators component of the Data Hub was created primarily for internal project use, as previously the data were handled with Excel spreadsheets in a time and effort intensive process. With the Data Hub, this process has been streamlined, especially for general progress reporting. However, because it is primarily an IM monitoring tool, it is unlikely that a national program would adopt the Data Hub, per se. Additionally, the Data Hub does not give the results of malaria indicators disaggregated by sex, age, or other potentially useful characteristics which might be helpful to assess access and quality of care. There have been efforts to integrate some components of the IM system into local systems or transition them to MOH control. IM is reportedly working with the University of Oslo to develop a HNQIS module within the District Health Information System 2 (DHIS2) platform. Additionally, the Data Hub and ancillary systems utilize dashboards for easy and rapid data visualization, which countries have shown interest in adopting. There does not appear, however, to be a well-articulated plan for the future of the Data Hub itself and the wealth of data it has collected. One HQ KI described the challenges that IM faced early on in the project, reporting that the data and data systems developed by the preceding project were largely lost when that project ended. PMI and IM had to scramble to salvage what they could before it was lost completely. In spite of this, another HQ KI indicated that the system would likely be retired at the end of the project. IMPACT: As with other technical areas, the IM project has had insufficient time to fully demonstrate the full extent to which improved data use has been institutionalized at country-level, much less been able to demonstrate the any increased data use has led to improved public health outcomes. EQ1f. Have SMC coverage and adherence objectives been met in areas where IM has been supporting SMC implementation? Many KIs, both at country-level and HQ/Global level, have indicated that IM’s work on SMC campaigns represents a major achievement for which they are very proud. IM’s accomplishments include a demonstrated ability to rapidly implement SMC campaigns at scale, achieving high coverage rates in very difficult settings (especially given the ongoing substantial security concerns in the area as well as implications of large scale community based activity during COVID-19), and piloting and proving utility of approaches to pay large numbers of workers quickly and efficiently using “mobile money” systems. ACTIVITIES & OUTPUTS: As highlighted under EQ1a, IM supported 28 SMC cycles in Cameroon (8), Niger (12), and Mali (8), reaching about 31 million children under five years in age19. Coverage was reported to be between 94 percent and 104 percent for individual cycles, with an overall average of 99 percent of targeted children being reached. The project reports that 34,000 campaign field staff were trained and supported for these SMC cycles. Each country’s approach to SMC campaigns had different strengths: Mali had an SMC payment strategy deemed to be a best practice; Cameroon utilized household enumeration before each cycle; and Ghana implemented real-time tracking of every targeted child.20 19 Source: IM Data Hub data provided to the evaluation team on May 21, 2021. 20 IM Annual Reports. 50 In PY3, IM supported all three SMC implementation countries to develop and implement rigorous, low-cost, rapid monitoring surveys in order to confirm measurement of coverage and adherence to all three doses. IM supported Mali to conduct rapid coverage and adherence surveys in IM-supported areas using the same methodology as in previous PYs. Niger replicated Mali’s rapid survey methodology in IM SMC districts. This was the first time that Niger had done a rapid independent survey of their SMC campaign, so IM piloted it in their districts and shared the results with NMCP and the other partners to encourage them to use the same methodology next year. NMCP Niger has endorsed and added the methodology to the strategic plan 2017-2021 (extended to 2023), so the other IPs will likely use it next year. The findings of the rapid independent monitoring surveys after the first cycle in 2020 were as follows: ● In Cameroon, coverage was 96 percent based on caretaker declarations and 88 percent for those providing an SMC card and/or a used blister pack. An estimated 94 percent of children received all three doses according to caretaker declarations. ● In Mali, coverage by declaration was 98 percent and with proof provided, coverage was 84 percent. 98 percent of caregivers declaring that they had administered the second and third doses. ● In Niger, coverage in the two IM regions was 99 percent and 100 percent, respectively, based on caregiver declarations. According to caregiver declaration, adherence to the second dose was 86 percent and 87 percent in the two IM regions and 63 percent and 72 percent for the third dose. Figure 3. below illustrates survey respondents’ perceptions of IM’s success in achieving Objective 2 in PY1-3. Figure 3: IM Success for Objective 2: Improve Quality of and Access to Other Malaria Drug-Based Approaches and Provide Support to Pilot/Scale-up SMC 51 IMPACT: The country-level public health impact of SMC remains an open question as few countries have undertaken rigorous impact analyses. One informant said that, early on, different countries or groups conducting and evaluating SMC used different metrics for measuring “success”. A lack of consistency in approach made it hard to demonstrate impact. IM prioritized bringing partners together to create a more consistent approach across countries for M&E of SMC. IM is also funding a sub-task force to focus on improving administrative data collection within SMC campaigns and adding independent assessments of coverage and adherence. EQ1g. Is there evidence of integrated management of key project interventions (CM and MIP) and has this integration strengthened their approach and delivery? EQ1h. Has collaboration with maternal health and ANC services been strengthened in areas where IM is supporting MIP implementation and service delivery, at both HF and community-levels? Taken together, EQ1g and EQ1h reflect an interest in ascertaining if MIP is best placed within malaria control programs or within maternal and child health (MCH) programs: does the MIP intervention benefit from closer alignment with other malaria control activities, especially CM, or is it more successful when implemented through MCH programs? For the purposes of this evaluation, however, lack of clarity around these questions made evaluation challenging. For example, the evaluation team was unclear what successful integration would look like in practical or operational terms, and what would be meaningful indicators of integration. KIs highlighted areas where they felt Maternal or Reproductive Health (RH) and NMCP coordinated and collaborated well, such as within MIP TWGs and in development of training materials and curricula that covered both malaria CM and MIP components (e.g., in Niger). In Kenya, the Ministry of Health (MOH) RH Division chairs the MIP Committee of Experts while the NMCP acts as secretariat. In Ghana, IM has ensured that all primary partners are represented on a revitalized MIP TWG, including RH Division, NMCP, and the Policy, Planning, M&E directorate (for issues related to data). KIs from both Kenya and Ghana reported that coordination at the higher levels is generally good and productive. IM Niger reported working closely with MOH’s Mother and Child Care Unit and NMCP for “ANC package inclusion” and training midwives on MIP. In these countries IM contributed to strengthening integration of CM and MIP activities. Within the OTSS+ component of IM, supervisory checklists include, among others, modules for both CM and MIP. OTSS+ visits, therefore, would “integrate” supervision of and mentorship for both CM and MIP. “Thanks to OTSS in field, we realized that the weak link in all of this was midwife communication to pregnant woman due to how very busy midwives are. Little by little we’ve been improving that area” (KI, Niger). Niger integrates midwives and laboratory technicians into the supervision teams. It is less clear how integration works at service delivery level. For HFs staffed by only one or two HWs responsible for all aspects of service delivery offered, “integration” occurs by necessity. In larger HFs, CM and MIP are likely to be done in different parts of the facility (ANC versus Outpatient Department) and by different staff. KIs emphasized that having information about MIP is not enough for women to seek out services; there are barriers at play, some of which are gender-related. One KI made reference to USAID projects that have funded couples communication interventions and found them to be strongly 52 linked to uptake of services, “PMI should be thinking how within the service delivery context, and with a community facing program like BA, we can go beyond awareness raising.” (USAID/PMI KI). As previously noted, implementation of MIP has met challenges within IM but mostly challenges that would exist whether MIP was embedded within a malaria service delivery project or not. Furthermore, as also noted previously, IPTp1 coverage remains well below the ANC coverage or even the coverage of ITNs delivered via ANC in all focus countries. In Ghana, two KIs noted that, while initial gains were observed, progress has stagnated and seemed unresponsive to the country’s efforts. There is a substantial drop-out rate for the IPTp intervention in all countries. KIs described a variety of issues that potentially affect the success of MIP interventions, including stockouts, distance to HFs, perceived poor treatment of patients by HWs, and cultural constraints. Within the local Kenyan cultures, for example, pregnant women frequently do not admit to being pregnant until it is unavoidable, which leads to late first ANC visits and not enough time to receive all recommended doses of IPTp. In Ghana, the reverse reportedly happens: women come very early, too early for their first dose of IPTp, causing a disconnect between ANC registration and delivery of IPTp1. In Cameroon, male heads of households refuse to allow their adult female family members to be attended to by male HWs. This is of particular concern since there is a dearth of qualified female HWs in Cameroon. Although not within IM’s mandate per se, some KIs at both the HQ/Global level (one KI) and in Ghana (two KIs) suggested taking a community-based approach to improving MIP, either through increased efforts to encourage timely and continued ANC visits or through community-delivered IPTp, if and when this is approved by WHO. There is a perception among some PMI HQ and sub partner country-level respondents that MIP is not receiving adequate attention. One country KI said, “What we’re doing now for MIP is a drop in the bucket.”. Refer to EQ2a, p. 41, for examples of MIP interventions that were suggested by the MIP technical partner but rejected by lead country partners. Summary comment for EQ1 The evaluation SOW does not define ‘effectiveness’ for the purposes of this evaluation. The evaluators note that a common definition of effectiveness is whether an activity, strategy, or intervention is achieving its intended results (outcomes, impact). Within this definition, there are limits to what can be said about the degree to which Impact Malaria has been effective in PY1-3. However, if the definition for effectiveness is whether a project has established excellent relationships and completed country-specific contract deliverables, we can say without reservation that IM has been an effective project. We note that these dissimilar definitions have different implications for future planning. 4.2 EQ2. MANAGEMENT: TO WHAT EXTENT HAS IM MET THE MANAGEMENT REQUIREMENTS AND FUNCTIONS OUTLINED IN THE AGREEMENT, INCLUDING PLANNING, ALLOCATION OF FUNDS, COORDINATION AMONG THE IM PARTNERSHIP (PSI,MCDI, UCSF, JHPIEGO), STAFFING REQUIREMENTS, AND IN￾COUNTRY SUPPORT? EQ2a. Has IM HQ oversight and management aided or hindered IM in accomplishing workplan objectives, both at central and country-level? 53 “Clarity, honesty, transparency, frequent and open two-way communication” are words used by KIs to describe both PSI’s relationships with sub partners and the interactions between PSI and the USAID PMI COR team. KIs repeatedly described mutual appreciation between USAID COR team and PSI’s project management. Weekly and biweekly meetings have kept everyone up to date on project activities. KIs considered it helpful that USAID COR team members also backstop buy-in countries and can help to troubleshoot country-level challenges. “Impressed with COR team… clarity regarding expectations, and honesty when they were waiting for information or decisions.” (KI from HQ partner) IM organizes internal “Wednesday Webinars” that are open to all IM staff and made available in English and French. Webinars feature presentations and discussions led by both IM HQ team members and IM in-country field staff. The project also holds weekly internal technical, operational, and financial and compliance staff meetings, and the IM Country Operations Director and Project Director hold one-to-one calls with IM Chiefs of Party (COPs) as needed. Respondents described communication as taking place regularly between IM HQ technical support staff and country field teams, who confirmed close ties with a responsive technical advisory team. “The whole project ethos is radical transparency.” (KI from IM HQ partner) Consortium partners acknowledged the IM Project Director as having created trust among the partners. IM managers expressed deep appreciation for the entire project team’s dedication to the work. “We are having this conversation because so many people have worked beyond the call of duty. Strong team spirit, dedication.” (KI from IM HQ). More than one KI from IM HQ partners indicated that this is the best partnership they have been a part of. Partners acknowledged that getting to this point has taken some work and that the partnership has improved over time. Overall, the IM consortium has benefited from the “one team” approach and partnership principles created by PSI to maintain a healthy dynamic. The approach creates opportunities for sub partners to participate in calls with the USAID COR team and keeps communication lines open with the consortium. However, respondents reported instances where the model doesn’t appear to be working as intended. One principle of the “one team” approach is that a partner with an established presence and recognition in a buy-in country, and with expertise in one of the buy-in technical areas, will be the lead partner there. In Benin this has reportedly been problematic. Two IM partners have a presence in the country; the partner with the larger presence, and per policy the lead partner, manages an ANC study21 that the other partner seems technically better positioned to manage. High frustration was shared with the evaluation team about multiple problems and delays the study has encountered. Respondents questioned whether the IM leadership is fully aware of the extent of problems and wondered if perhaps there may be a disconnect due to a two-pronged reporting structure whereby lead partners communicate IM HQ through a management channel and technical partners communicate through a different technical channel. In this instance there appear to be issues with communication between the two in country partners and a less than optimal alignment of partner technical expertise with the research activities. In Cameroon and Côte d’Ivoire, lead partners have reportedly discounted the technical partner’s inputs. For example, a request for ten mentors in Côte d’Ivoire to supplement OTSS+ rounds was reduced to four without explanation, even though there were excess funds in the pipeline to cover mentor costs. A proposal to implement malaria death audits in Cameroon was reportedly 21 “Assessing the Impact of G-ANC on IPTp uptake in Benin and the Feasibility and Acceptability of Pregnant Women as a Sentinel Surveillance Population.” 54 removed from the workplan because it exceeded IM’s scope. These rejections of technical partner suggestions represent missed opportunities. It is not clear whether all country-level managers fully understand the role of technical partners, nor is it clear whether IM HQ leadership is fully aware of and appropriately intervening in situations where the “one team” principle is not being followed or doesn’t meet the project’s technical needs. It may be that joint meetings inclusive of all in-country partners and IM leadership at HQ level would help to mitigate these situations when they arise. Respondents from two sub partners expressed some disappointment about the limits of their portfolios. A respondent from one sub partner expressed the view that the money is ‘ring fenced’ and any work proposed above a specific budget amount would be denied. Another had expected to play a more significant role in CM. Respondents from two partners expressed regret that the informal think tank mechanism, with space for innovative thinking across technical areas discussed during proposal development, did not materialize. All sub partners believed they had more to contribute than what they have been able to contribute based on funding ceilings outside their control. Partners expressed the hope that their contributions are being fully reported to and recognized by USAID. EQ2b. Has coordination between IM and partners in country (PMI RAs, NMCPs, other IPs) aided or hindered IM in accomplishing country workplan objectives? Nearly all country-level respondents who expressed an opinion held very favorable views of the collaboration and coordination among IM consortium partners and between IM and MOH and other PMI-funded partners. The majority of IM HQ respondents also described strong coordination and collaboration between IM and other malaria partners at country-level. One KI from Kenya saw room for improvement in how IM shares information with the government. In-country respondents reported that country-level coordination meetings were facilitated by USAID/PMI and NMCPs. Some countries have created PMI partner coordination groups to discuss shared deliverables or input into deliverables. Communication takes place regularly between IM country offices and USAID Mission PMI staff; in some countries USAID hosts monthly or quarterly meetings with malaria partners. Various partners sit in the same government meetings or on the same TWGs. KIs from IM Niger described good collaboration with the Global Fund and reported that PMI promotes this collaboration. KIs from IM Madagascar reported bringing malaria partners together and knowing who needs to be in the room for each discussion. One country survey respondent praised MCDI for providing “in￾country continuity and credibility going back a decade.” A survey respondent in Ghana appreciated that the “involvement of M&E staff in all CM, MIP and diagnostic activities” helps to resolve implementation issues for these activities. Country-level stakeholders who are not IM project staff praised the project as having strong credibility as a malaria partner. One survey respondent, however, emphasized that “Impact Malaria country and their supervisors (HQ, PMI country office) must treat the country malaria control and [host country] staff with respect and appreciation by engaging them properly and adequately before changes are made and effected especially during (in the middle of implementation of) planned activities.” Another complained that “Poor coordination of activities at sub-national (level) has resulted in some activities not 55 being implemented.” Without any accompanying details, it is impossible to fully understand or address these comments. EQ2c. Is in-country presence of IM staff sufficient and appropriate? There is nearly unanimous consensus that IM staff are well suited for their roles. Although it took some time to fully staff numerous country offices, by the end of PY3 country staff were viewed as sufficient and competent. Considering the project’s scope, this has been accomplished in a timely way. “Impact Malaria has enough staff and very skilled staff in support to the NMCP.” “The current team is adequate to successfully conduct all workplan tasks.” “PMI Impact project is more than adequately staffed for the level of activities being implemented.” “The staff are competent and efficient.” “They execute their work professionally.” “Staff are highly trained and have senior experience in health programs management.” “At most times, the staff are able to meet the demands from the project. There are times when large activities are being implemented when the staff have been stretched.” (Survey respondents) Dissenting survey comments (with no explanations given) were: “The lack of the constant presence of the M&E officer;” “The project missed staff with critical skills in malaria diagnosis and capacity building;” The staffing up has been a little delayed, which caused problems for some IM work in country.” (Survey respondents). Jhpiego was described as having high performing staff already available or easily recruited in Ghana, Rwanda, and Kenya where it had a strong country presence at project’s start. PSI has reportedly supported partners in taking the time to recruit at appropriate skill levels. For buy-ins with Malawi and Tanzania, MCDI had to renew its registration in those countries. PSI helped to identify and hire staff, then transferred staff to MCDI. There have been recruitment challenges in Cameroon’s remote far north. EQ2d. Has IM been adept at adjusting to the rapid growth of country buy-in, from the original nine countries in FY 2017 to 18 countries and two regional buy-ins in FY 2019? The evidence indicates that IM has done an outstanding job of managing the project’s rapid growth from nine countries in PY1 to 18 countries and two regional buy-ins in PY3. Its success is all the more impressive given the multiple start-up delays in PY1 and the arrival of COVID-19 pandemic in PY3. In PY1, nine countries joined IM: Cameroon, Côte D’Ivoire, DRC, Ghana, Kenya, Mali, Niger, Sierra Leone, and Zambia. Four more countries joined in PY2: Benin, Madagascar, Rwanda, and Senegal. In PY3, five more countries - Burkina Faso, Cambodia, Malawi, Tanzania and Lao PDR - also joined, which brought the total to 18 countries by end of PY3. “The speed for buy-ins exceeded expectation and it didn't stop the team's dedication to ensure quality implementation in each country.” (IM HQ partner) IM leadership reports that start-up has been most successful where at least one consortium partner already had a registered presence in-country; this has been the case in nearly every participating country. There were challenges in Sierra Leone, where none of the consortium partners was registered and the registration process was slow. Start-up was also slow, for a variety of reasons, in Senegal for the MDA study managed by UCSF. Although many of those reasons were not under IM’s control, UCSF doesn’t routinely have an in-country presence in the way that 56 International Non-Governmental Organizations (INGOs) like Jhpiego and PSI normally do, leading to some additional delay. IM Kenya has experienced progressive scale up within eight counties, as new sub-counties have been added. Kenya KIs described additional responsibilities without additional accompanying resources, making it impossible for the project to sustain the same level of intensity over time. KIs report that activities must be “diluted” as a result. No further details were provided. In Ghana, IM initially covered five out of ten regions. This has expanded; IM Ghana is now responsible for activities across the whole country, including newly created regions. However, the funding envelope has reportedly not increased. This challenge has been dealt with mostly through the High Burden/High Impact approach, i.e., using data to focus resources where they are most needed. A Kenya KI noted that funding is sufficient for only one day per county of OTSS+ rounds, not enough to do “proper OTSS”. IM HQ leadership reported not having received clear planning guidance from USAID for country buy-ins. Buy-in planning with a country has taken up to several months even though activities frequently have already been negotiated with USAID country Missions. For each country, the project has developed a lengthy planning document that must also be translated into French for Francophone countries. IM managers suggested that a brief high-level summary of agreed on activities accompanied by an activity matrix, budget and budget narrative would suffice and be much more time and resource efficient. The PMI core workplan was described by IM leadership as ambitious at project’s start. Activities weren’t prioritized and expectations that project could attend to multiple buy-ins while starting up was viewed by project managers as unrealistic. EQ2e. Has IM been able to hire staff, set up offices, launch activities, and continue activities on the agreed upon timelines? IM has continued to increase staffing across the PY1-3 in response with the project’s growth and in line with the approved core workplan’s staffing plan. New positions added during PY3 included three technical advisors, Communications Officer, Contract Compliance Coordinator, Contract Compliance Officer, Finance Officer, Data Analyst, Knowledge Management Officer, two M&E Officers, Operational Research Program Manager, Program Manager and two Kenya-based Systems Specialists. The IM project faced several major start-up challenges. These included an award protest that resulted in a stop-work order which interrupted crucial staff hiring, a temporary USG shutdown, and a disrupted partnership with WHO mandated by the US government. IM leadership had to navigate these daunting challenges while negotiating workplans with multiple countries. One IM partner felt that the need to move quickly despite obstacles led to the sacrifice of innovative planning. “At the beginning, project operations moved quickly to meet COR deadlines at the expense of technical strategy.” The project then faced challenges from the COVID-19 pandemic. See Annex 7 for a detailed description of the project’s laudable responses and solutions to COVID-19. Despite serious obstacles, the project has been timely overall with submission of core and country workplans, financial and annual reports. Travel based activities that were disrupted due to COVID￾19 included scoping and technical support country visits, the project’s third annual global work planning meeting, and participation in global meetings. Many KIs at both HQ and country-levels described challenges related to the project’s structure as a contract rather than a Cooperative Agreement. USAID respondents described initial delays 57 that appeared to be related to individual IM staff inexperience with USAID contract mechanisms, resulting in an “excess of caution” that slowed activities unnecessarily. Many country and HQ level IM project staff described struggles with uncertainties about when pre-approvals are needed under the IM contract that are still not entirely resolved. In particular, the contract was not clear about the need for pre-approval of parastatal waivers (most sub￾contractors are parastatals) and for IT equipment. In Senegal, delays in approval of the parastatal waiver for the University of Thiès led to delays in the operational study’s implementation timeline and planned activities to strengthen data collection practices in advance of baseline data collection. Two HQ management staff described slow feedback from USAID on submitted reports and workplans. “Workplans are the engine, not the end goal.” (HQ level partner) In-country respondents highlighted funding delays as the key challenge for timely completion of activities. In Ghana and Kenya, KIs reported a number of situations where financial management restrictions imposed on IM have hindered country-level activities, caused delays, or put substantial pressure on non-IM staff. “There are differences between what PMI thinks things should cost and what they actually cost in reality.” (IM HQ/Global). Delays with obligations were described as putting undue pressure on the project to begin work without funding in place. PSI described challenges in being asked to move money between countries as a short-term solution, which can muddy the audit trail. EQ2f. Has IM been adept at tackling the logistics of staffing, coordinating and managing logistics for SMC campaigns? Are the campaign activities in conflict with maintaining routine support for CM and MIP activities? SMC campaigns are widely considered one of the project’s biggest successes. KII and survey findings varied in the degree to which routine activities were interrupted by SMC, ranging across responses from ‘no interruptions’ to ‘everything else shut down’. There is consensus across the three countries (Cameroon, Mali, and Niger) where IM helped to implement SMC campaigns that collaborations for SMC have been smooth and that involved partners have coordinated activities well for the benefit of countries. In Niger, a country where there was only a PMI CDC RA in place to facilitate buy-in or campaign planning, IM was given just a few weeks’ notice to take on a SMC campaign dropped by Global Fund (GF) and did it well by all accounts. IM found a way to work around differences in the GF’s and USG’s per diem policies to ensure that the campaign could go forward. As with the TES (see EQ2g. below), funding envelopes are reportedly sometimes too small to complete the campaigns as planned. This is seemingly the case in Burkina Faso. The project looks for solutions that will enable a timebound campaign to go forward. These funding issues create undue challenges for a project already carrying a heavy load in terms of rapid scale up and a wide range of technical activities within several countries. Campaigns are described by IM leadership as logistically heavy lifts; each country has faced unique challenges. In Mali, SMC actors worked around a coup. Because Niger was experiencing attacks on NGOs, the campaign required the use of unmarked cars. Cameroon was experiencing severe problems with Boko Haram (“entire villages disappeared”), and it was reportedly necessary to do verifications of households four times before every cycle. Committed teams were described as working in highly insecure areas and in the rainy season with muddy roads. Managing the huge numbers of workers required to carry out a campaign was a huge task. In all three countries, the SMC campaigns yielded very impressive results for targets and adherence. 58 IM reports that although there is no evidence that parents are not giving the medications correctly during SMC campaigns, some countries nonetheless want to introduce Directly Observed Therapy (DOT); this is to be piloted in Burkina Faso. IM is working to strengthen monitoring surveys to reassure countries that there are no adherence problems in the campaigns. EQ2g. Has IM been adept at tackling the logistics of staffing, coordinating and managing logistics for TES activities? IM is planning or implementing nine TESs across eight countries with local sub-contractors: Burkina Faso, Cameroon, Côte d'Ivoire, DRC, Kenya, Mali, Niger, and Rwanda. PMI HQ is pleased to have IM’s technical support and involvement in these countries and appreciates the biweekly study update calls and monthly meetings with the IM HQ TES focal person and USAID IM COR team. The project is described as “helping to keep a finger on the pulse’ of studies that may change daily." TESs have faced challenges. Cameroon had problems getting Institutional Review Board (IRB) or Federal Drug Administration (FDA) approval. IM reportedly thought it needed its own IRB protocol for Cameroon, even though study was already approved in country and by CDC; eventually the project found a more expedient solution. Mali had trouble recruiting enough patients at a certain site. A major obstacle highlighted for the TES in Kenya was the poor quality of malaria diagnostic capacity. This study needed to “import” microscopists from outside the study facilities to ensure quality of slide reading. When COVID-19 delayed progress on the Zambia operational research project, ProAct, leading to a realization that the study would not be completed before the end of the IM project. Therefore, the study was handed over to a bilateral implementing partner. More than one IM HQ management staff member emphasized that the TES budget envelopes ($75k per study arm) are not always sufficient, and the PMI policy needs review. The Zambia ProACT study22 was described as having a bare bones budget. Additionally, TES budgets may need to be reconsidered to accommodate different types of partners. Central partners are always more expensive. The competence of IM’s TES staff at both HQ and country-level was praised. IM hosted a TES webinar last spring that was considered technically strong. To emphasize the value of these studies, in Rwanda, some concerning resistance mutations were found during a prior TES in 2018, and the current study will look for those mutations again. Findings in Rwanda have also been found in Tanzania. EQ2h. Has IM been adept at tackling the logistics of staffing, coordinating and managing logistics for OR activities? Five OR studies are underway in Benin, Mali, Cambodia, Senegal, and LAC: ● Benin (Does group ANC improve uptake of IPTp?); ● Mali (Does a package of Quality Improvement and Behavior Change interventions improve HW skills in managing MIP and increasing ANC and IPTp uptake?); 22 Whether implementing ProACT on a weekly basis may be effective at more rapidly identifying and treating malaria cases and potentially reducing malaria morbidity and transmission. 59 ● Cambodia (How accurate and reliable is a new point of care test for G6PD23 deficiency?); ● Senegal (How does MDA compare with SMC on transmission reduction?); and ● Latin America (What is the relative efficacy and tolerability of different radical cure regimens for Plasmodium vivax?). COVID-19 has created challenges for the progression of OR as well as other activities (see Annex 7 for how IM has responded to challenges introduced by COVID-19). USAID informants as well as some country respondents perceive ongoing challenges with OR project management. Country respondents in Benin and Senegal especially concurred. Multiple respondents described malaria OR as complicated in the best of circumstances. In Senegal, where re-approval waivers have created funding delays, this is especially challenging in the context of malaria’s seasonality. “With OR where the timing is so critical, before transmission season, there are aspects such as calling and vetting local IP that take a long time, you have to anticipate the study two years before you’re going to implement it…have to plan this out carefully and with a realistic time frame. Everyone should understand from the start whether the timing is feasible.” (HQ partner) There appears to be some role confusion within the IM consortium as to who is in charge of tracking the Senegal study’s timeline and budget, which was developed in country and passed over to IM. There were reported discrepancies between the macro level budget developed by the Senegal Mission and later budget development by IM, resulting in resource gaps. Also, the contract between IM and UT was described as inflexible. Perhaps IM could have better prepared its local partner for the realities of time gaps between requesting and receiving funds. UT was often praised as having been responsible for the study’s ability to progress, and as having approached problem solving with determination, innovation and commitment. KIs concurred that the Senegal study would benefit from having an IM project manager in country to help UT troubleshoot and keep all stakeholders updated on progress and challenges as they arise. Evaluators don’t know if the country workplan and budget would support this addition; if so, it is offered as a recommendation. Senegal MDA study’s implementation schedule faced several delays, many of which were beyond the project’s control (i.e., waiting on PMI to make decisions about the protocol, delays with approval of the subcontract at USAID). Furthermore, delays in obtaining USAID approval for laptops, tablets, and field and laboratory supplies have affected the Senegal’s study’s implementation schedule. Tablets for supervisors weren’t budgeted for and implementation of the MDA intervention was also not budgeted for. PSI and UCSF pulled from other financial sources to find solutions. USAID expressed some dissatisfaction with PSI’s not having stepped in when a stronger hand was needed, referring to Benin and previously Zambia, when a sub partner presented higher than expected budgets even though PMI CDC teams had already worked on protocols and budgets. Reportedly essential components like training for field teams were left out; in these instances, PSI is perceived as having allowed too much leeway. One USAID informant perceived the larger issue as a lack of clear communication about what is feasible within given budgets and a need for timely updates when the project realizes funding isn't sufficient. 23 Glucose-6-phosphate dehydrogenase deficiency, a carbohydrate metabolic disorder that is characterized by abnormally low levels of glucose-6-phosphate dehydrogenase (abbreviated G6PD). 60 Project management also has experienced frustrations. IM Benin is reportedly having trouble fitting the OR into its funding envelope. “LOE required for study protocol preparation and IRB submission greatly exceeded our workplan and resulted in 6-9 months of delays even without taking COVID-19 into account.” (Benin survey respondent). Despite challenges, there is appreciation of IM’s OR efforts. “I really appreciate the great efforts of IM HQ and country team who are working so hard to keep OR activities implemented and continuing under COVID-19 situations. Staff are qualified and friendly. IM staff are proactive to set up bi-weekly meetings to update us on the progress of activities and challenges to address.” (Senegal survey respondent). Recommendations from KIs for the Senegal study include closer coordination between the two research partners, UCSF and University of Thies (UT), to monitor the budget and ensure that teams in the field have all the resources they need to complete the work; continuation of weekly phone calls between UCSF and UT; and development of a plan for how various partners intend to use and publish study data, to ensure that all parties are in agreement. EQ2i. Are PMP indicators agreed upon at the HQ and/or the country-level practical from a reporting perspective and are they useful from a programmatic perspective? Nearly all interview and survey respondents were positive toward IM’s indicators. However, some respondents, especially in Kenya, felt that there were too many indicators or that some indicators were difficult to measure, even if the indicators themselves were relevant. One Kenya respondent recommended that indicators be refined to focus on collection of those that are most important and useful to the project. Another suggested that IM should better reconcile their indicators to be more consistent with those Kenya requires and, where there are differences, IM should do a better job of justifying the added indicators. Another thought there should be better guidance on how to use the indicators. There is consensus that some PMP indicators need clearer definitions or revisions. A revised PMP indicator list was reportedly submitted to PMI and approved after joint discussion. The indicators are clear and aligned with global standards for each technical component. However, they don’t reflect an overall holistic framework for health systems strengthening. The project would benefit from an overall framework that shows the linkages between all technical components within IM. It would be useful to distinguish between QA indicators, such as aligning country policies and guidelines with global standards, and quality improvement indicators, such as improved HF and HW performance through the OTSS+ model. The evaluation team saw no indicators relating to how the project will apply a gender lens to the project. Evaluators heard that Jhpiego has presented technical deep dives on the importance of gender in efforts to prevent, test, treat, and eventually eliminate and eradicate malaria, and that interest from USAID and partners has grown over PY1-3. The two gender analyses conducted by IM that were delayed and are being published in PY4 may help to shape the gender focus of the next five-year project. The PMP indicators do not contain any that relate to IM’s internal operations: communication structures, financial operations, etc. Global institutions such as the Global Fund and Gavi have introduced internal performance measures for each Secretariat department and have found them to be a useful incentive for teams and a way to highlight outstanding performance. PMI may find it useful to adopt this practice. 61 4.3 EQ3. GLOBAL RESULTS: WHAT RESULTS HAVE BEEN REALIZED AT THE GLOBAL LEVEL? EQ3a. To what extent has IM achieved global level results laid out under each objective in the detailed program description of the award, including plans for and progress towards publications, documentation, and dissemination of best practices/lessons learned? What has IM developed in their Learning Agenda and other job aides, such as guidance on implementing SMC campaigns in the context of COVID-19? IM has been an active and credible participant, and sometimes a leader, in global level malaria meetings and conversations, at conferences, and on social media platforms. Since PY1, IM has been actively engaged in multiple global malaria working groups and conversations. IM engaged in several ways with the Roll Back Malaria Partnership (RBM), including through the MIP Working Group Secretariat, as Co-Chair for CM Working Group, and through participation in the RBM Monitoring and Evaluation Reference Group (MERG), the Social and Behavior Change Working Group, and the SMC Alliance. “The project has had broad spectrum engagement in relevant work streams, whether leading them like the CM WG or contributing to them like SMC or MIP.” (HQ IM). In PY3, IM produced a Learning Brief in English and French on SMC lessons learned from the 2019 campaigns, including payment strategies, rapid monitoring survey methodologies, and lessons learned from integration with other disease areas (particularly malnutrition screening), which was disseminated to the SMC Alliance and other global SMC stakeholders after a webinar that was jointly presented by PSI’s Malaria Department and the SMC Alliance in September 2020. IM has convened and serves as informal Secretariat for an M&E Task force affiliated with the SMC Alliance, which is linked with the RBM Country/Regional Support Partner Committee (CRSPC)24. This task force focuses on improving administrative data collection within SMC campaigns, adding independent assessments of coverage and compliance, and has drafted a set of indicators as part of a toolkit that is intended to help standardize metrics for SMC campaigns. In PY2, IM launched multiple social media platforms, including a project website, blog posts, a Twitter account, a presence on LinkedIn, and a Flickr photo library with over 500 photos. In PY3, IM contributed 15 country pages to its website and also packaged this content as fifteen country briefs, with French versions for francophone countries. Furthermore, in PY3, the project created multimedia products for World Malaria Day 2020 that included a short video featuring Cote d’Ivoire’s HF based work on MIP and a feature article on USAID’s medium channel that six global health organizations cross-posted to their websites In PY3, IM worked with PMI to catalogue a series of tools and best practice documents on malaria CM for the CMWG website. “IM emphasizes sharing of best practices and looking for where it can add value.” (HQ IM). In PY3, IM partnered with Breakthrough ACTION to produce a tool entitled A Blueprint for Applying Behavioral Insights for Malaria Service Delivery: Methods and Frameworks for Improving Provider 24 The purpose of the Country/Regional Support Partner Committee (CRSPC) is to provide a platform to engage the RBM Partnership community in coordinating support to countries and regions as they execute their malaria control and elimination implementation programs. 62 Behavior that provides steps for using insights into health provider behavior to improve the quality of malaria service delivery. IM and BA co-hosted a webinar about the Blueprint, moderated by PMI. In PY3, IM presented in English and French on COVID-19 adaptations to iCCM and SMC programming as part of a Child Health Task Force (CHTF) webinar series. IM describes support from PMI for more work with Global Child Health Task Force on increasing the profile of iCCM, updating toolkits and documenting best practices, scaling up of iCCM in three countries, and testing payment approaches. IM is partnering with WHO to develop a handbook for malaria CM in the private sector and an operational manual for supportive supervision based on learnings from OTSS+. IM plans to share findings from an external, independent evaluation of its OTSS+ model. IM’s sub partner Jhpiego has published an external facing brief with general learnings about gender and malaria. Across PY1-3, IM reports having contributed nine posters, three oral presentations, a joint symposium on the topics of IPTp, MM, malaria CM guidelines, SMC, prioritization of facilities, severe malaria from DRC, Ghana, Kenya, Mali, and Sierra Leone for the American Society of Tropical Medicine and Hygiene (ASTMH). Reportedly, for ASTMH 2021 three symposium proposals and various country abstracts are under development. Jhpiego plans to publish the Cameroon and Kenya gender analyses that were completed in PY4, and has submitted a gender-focused abstract to the ASTMH 2021 conference. PSI has recently contracted with UCSF to provide technical support to enable all IM buy-in countries to produce at least one manuscript related to their participation and accomplishments by the end of the project. IM has also produced a Learning Agenda through consultation with USAID PMI, to contribute to the body of knowledge on the most effective and efficient ways to deliver malaria services in four project intervention areas: QA, iCCM, MIP, and SMC. Learning Agenda outputs are expected to include an OTSS+ independent evaluation, a CHW internship curriculum, an iCCM toolkit, an assessment of the effectiveness of the OTSS+ model on MIP service delivery, and more consistent and accurate methods of measuring coverage and adherence to SMC. Two KIs expressed the view that academic journals are not being optimally used to share project findings, especially for OR and TESs. One informant suggested that the best strategy is to “get a publication on the record, amplifying the local teams and NMCP voices….and partner that with a one￾page summary brief for policy makers and decision makers.” Another suggested that greater attention should be paid to the dissemination of information via webinars, technical briefs, and annual meetings, as well as published papers. (IM HQ and country-level). 5. CONCLUSIONS AND RECOMMENDATIONS This section summarizes the findings, conclusions and recommendations of the IM MTE by EQ and sub-EQ. At the end of each recommendation, the evaluation team has included the stakeholder(s) who should consider implementing it. EQ FINDINGS CONCLUSIONS RECOMMENDATIONS EQ1. COUNTRY-LEVEL PERFORMANCE EQ1a. To what extent has PMI IM achieved the technical and programmatic objectives described in annual country and core workplans and IM PMP? ● IM has been very productive across multiple technical components: OTSS+, case management including for severe malaria, prevention of MIP, malaria diagnosis and iCCM, and in the development of a complex project Data Hub. ● Project outcomes have been highly variable across programmatic areas, countries, and sub-country project areas, and over time. Where progress toward technical objectives has been made, it is mostly modest. Data quality and variability issues that are out of IM’s control continue to be a challenge for project reporting. ● Because most of the project's data (Data Hub/HMIS) are based on data generated at health facilities (HFs), access to malaria services for people who do not reach the facility cannot be examined. IM data, therefore, are not providing enough information to develop strategies to improve access for hard-to-reach groups. The only information available on a population basis (and therefore providing information about those not accessing services from HFs) is through DHS and MIS. Because these surveys are typically only conducted every three to five years , they tend not to provide information that is timely enough to inform operational decisions. ● The project’s performance is outstanding in terms of carrying out multiple activities across a rapidly growing number of countries in the face of multiple start up challenges and the arrival of COVID-19. ● Quality of routinely collected malaria data from government systems and variations across countries make it challenging for the project to report improved outcomes. More specifically, insufficient time, underlying poor government data quality and different types of data across countries (e.g., Malaria Indicator Survey/MIS, other national household surveys) that are out of IM’s control and present a wide variation in quality, and methodologic issues (absence of control areas or well-designed independent periodic surveys) are likely contributors to an inability to demonstrate more effect at this point in time. ● Consider investing in alternative methods to periodically assess progress independent of national HMIS systems or IM’s own OTSS+ data systems, e.g., population￾based surveys of project areas, lot quality assurance sampling (LQAS), quasi-experimental design, or other comparison methods (cohort study, control group). Consider independent quarterly surveys in randomly selected HFs in one country in PY5. (USAID/PMI and IM) 64 EQ FINDINGS CONCLUSIONS RECOMMENDATIONS However, project data does demonstrate high performance in meeting training, supportive supervision, and SMC distribution targets. EQ1b. Is there evidence of in-country capacity improvements in malaria diagnosis and CM and prevention of MIP at various levels of the health system (national, regional, district, community), taking into account guidelines, training, supervision checklists? ●Project Outcomes related to capacity improvements show a high degree of variability across countries and sub-country project areas, across indicators, and over time. ● Quality and completeness issues with routinely collected government health data, which are out of IM’s control and present a wide variation in quality, make it difficult to form a complete picture of the levels of capacity and proficiency that the project has helped to build. ● There is wide variation across countries in the proficiency scores related to laboratory malaria diagnostics. ● Project data show evidence of improvements in quality service delivery in a number of IM-focus countries (e.g., Cameroon and Niger). ● While many HW have been trained, the prioritization of low-performing facilities for OTSS+, and high HW turnover in targeted HFs, make it difficult for IM to demonstrate with certainty sustained levels of improved capacity and competence, and will ‘raise the bar’ slowly. ● IM is tracking both knowledge and competency data. The more meaningful indicators related to sustained capacity are competency scores that determine how well new training-related knowledge or skills have ‘held’. ● For OTSS+, rely on competency scores more than pre- and post￾test training scores as evidence of improved capacity. (USAID/PMI and IM) ● Consider following a cohort of microscopists to see that their skills improve and remain high. (IM) ● See the EQ1a. recommendation for independent surveys. (USAID/PMI and IM) EQs1c. and 1d. Do checklists and other tools capture useful data on the status and quality of CM? Are they appropriate and informative? Is implementation of OTSS+ disruptive to provision of services (does it take too much time)? Are results from checklists/other tools used by IM to make adjustments to training and supervision to improve quality? 65 EQ FINDINGS CONCLUSIONS RECOMMENDATIONS ● IM has developed, validated, and implemented seven OTSS+ checklists. ● Eleven IM-supported countries have aligned their supervision approach with the OTSS+ package of checklists by end of PY3. ● KIs from all stakeholder groups identified digitalization of the checklists as a major step forward. ● OTSS+ is a time, resource, and effort intensive activity, and a number of KIs raised concerns about sustainability of the intervention. ● Countries using the OTSS+ model find it very helpful to identify and address performance issues in real time but have concerns about sustainability. ● OTSS+ data generally show that performance was relatively high in some countries reviewed (Kenya and Ghana) and there were modest improvements for some indicators. However, more data on the effect of the OTSS+ approach over a longer period of time are needed. The planned independent OTSS+ evaluation should yield useful additional findings. ● There is no standard operating procedure (SOP) included in the OTSS+ checklist package to guide countries in implementing OTSS+ checklists effectively. ● Collect more evidence regarding durability of change (where there are improvements with IM’s well￾funded, intensive approach, would they be maintained or improved upon as MOH takes more independent responsibility for activities?) (IM) ● Add a SOP to the checklist package that guides DHMTs and RHMTs to effectively conduct OTSS+, and guides supervisors in skillful use of the checklists. (IM) EQ1e. Has the development of the IM Data Hub and the associated efforts to access national HMIS data for PMP reporting resulted in tangible improvements to data use? ● Setting up the Data Hub has been resource-intensive. ● As of February 2021, eight countries were submitting data to the IM Data Hub (Cameroon, DRC, Ghana, Kenya, Madagascar, Mali, Rwanda, and Sierra Leone); one country (Niger) is estimated to be fully operational in 2021; two countries (Malawi, Zambia) are scheduled to start in 2021; one country (Tanzania) is in start-up phase, and one country (Côte d’Ivoire) is closing operations in late 2021. ● There does not appear to be a well-articulated plan for the future of the Data Hub itself and the wealth of data it has collected. ● IM has encouraged data use at all levels and this remains a stated priority for the remainder of the project. ● KIs raised concerns about the future of IM’s data systems. ● Expectations differ as to the Data Hub’s continuation after IM ends. It would be inefficient to start over under the next malaria project with a new data system. ● Include expectation that Data Hub data will be transitioned in some form in the Request for Proposal (RFP) for the next five￾year malaria project and include this in transition and close out planning. (USAID/PMI) EQ1f. Have SMC coverage and adherence objectives been met in areas where PMI IM has been supporting SMC implementation? 66 EQ FINDINGS CONCLUSIONS RECOMMENDATIONS ● IM’s accomplishments include a demonstrated ability to rapidly implement SMC campaigns at scale, achieving high coverage rates in very difficult settings, and piloting and proving utility of approaches to pay large numbers of workers quickly and efficiently using “mobile money” systems. ● IM supported 28 SMC cycles in Cameroon (8), Niger (12), and Mali (8), reaching about 31 million children under five years in age over three years. Coverage was reported to be between 94 percent and 104 percent for individual cycles, with an overall average of 99 percent of targeted children being reached. The project reports that 34,000 campaign field staff were trained and supported for these SMC cycles. ● In PY3, IM supported all three SMC implementation countries to develop and implement rigorous, low-cost, rapid monitoring surveys in order to confirm measurement of coverage and adherence to all three doses. ● Each country’s approach to SMC campaigns had different strengths: Mali had an SMC payment strategy deemed to be a best practice; Cameroon utilized household enumeration before each cycle; and Ghana implemented real-time tracking of every targeted child. ● SMC campaigns have achieved impressively high coverage and adherence targets in the face of many obstacles and security concerns, and exceeded expected outcomes. ● The rapid monitoring surveys have helped to confirm high target achievements and seem like a best practice to continue. ● For SMC, the suggestion to conduct a household enumeration in Niger and Mali to achieve a better denominator for targets seems sound. ● Replicate the rapid monitoring surveys and household enumeration where possible for future SMC campaigns. (IM) ● Document this experience with more learning briefs so that the SMC successes and lessons can be fully shared by end of project. (IM) EQs1g. and EQs1h. Is there evidence of integrated management of key project interventions (CM and MIP) and has this integration strengthened their approach and delivery? Has collaboration with maternal health and antenatal care (ANC) services been strengthened in areas where IM is supporting MIP implementation and service delivery? 67 EQ FINDINGS CONCLUSIONS RECOMMENDATIONS ● Areas where KIs reported that Maternal Health and NMCP collaborated well were within MIP TWGs and committees, in development of training materials and curricula that covered both malaria CM and MIP components, and within the Policy, Planning, M&E directorate for issues related to data. ● OTSS+ visits “integrate” supervision of and mentorship for both CM and MIP. ● It is less clear how integration works at service delivery level. Efforts are not yet being made to strengthen linkages between HF and CHWs who are tasked with malaria diagnosis, treatment and referrals to HFs and with MIP education and referrals. Ghana seems to have implemented a successful model of this linkage. ● There is a substantial drop-out rate for the IPTp intervention in all countries. ● Coordination of CM and MIP activities occurs mainly at the central level. At regional or district level, there is no evidence of integration apart from OTSS+ related activities. ● Define indicators for integrated CM and MIP so that the project’s success can be better determined. (USAID/PMI and IM) ● If the MIP (and iCCM) components are further developed for this or future PMI projects, investigate whether phone-based electronic data entry by CHWs can be introduced as part of technical support for training and supervision of CHWs, to help document integration of CM and MIP at community level. This could better link CHWs into the country’s health service delivery and malaria data systems and help collect data on populations not accessing services at HFs. (USAID/PMI and IM) EQ2. MANAGEMENT EQ2a. Has PMI IM HQ and PMI COR team oversight and management aided or hindered IM in accomplishing workplan objectives, both at central and country-level? 68 EQ FINDINGS CONCLUSIONS RECOMMENDATIONS ● KIs repeatedly described mutual appreciation between USAID COR team and PSI’s project management. Communication has been regular and transparent. ● Consortium partners acknowledged the IM Project Director as having created trust among the partners. ● IM managers expressed deep appreciation for the entire project team’s dedication to the work. ● Overall, the IM consortium has benefited from the “one team” approach and partnership principles created by PSI to maintain a healthy dynamic. ● PSI’s one partner principle is well appreciated overall; however there are some instances where this principle is not fully working. ● There is strong evidence that the USAID/PMI COR team and IM leadership have worked well together over PY1-3. The USAID/PMI COR team and IM’s HQ management team have mutually contributed to the positive working relationship. ● The IM partnership principles have served the project well overall, however instances where they are apparently not working are not always addressed. ● Look for opportunities to integrate the “one partner” good practices into future project management expectations. (USAID/PMI and PSI) ● Take steps to ensure that problems with partner roles at country-level are reported and addressed. This may in some cases mean actively stepping in to mediate. Joint meetings inclusive of all in-country partners and IM leadership at HQ level could help to mitigate partners’ struggles in country when they arise. ● Consider adding flexibility to the partnership principles that allows for changes in the lead partner role if necessary. (PSI) EQ2b. Has coordination between IM and partners in country (PMI Resident Advisors [RAs], National Malaria Control Programs [NMCPs], other IPs) aided or hindered IM in accomplishing country workplan objectives? ● There is nearly unanimous appreciation by country-level KIs and survey respondents of the collaboration and coordination among IM consortium partners, and between IM and MOH and other malaria partners. ● Country-level stakeholders who are not IM project staff praised the project as having strong credibility as a malaria partner. ● IM enjoys an excellent reputation among in-country partners as a collaborative and cooperative malaria partner. ● No recommendations - the project is performing with excellence. EQ2c. Is in-country presence of IM staff sufficient and appropriate? ● There is nearly unanimous consensus that IM staff are well suited for their roles. ● By the end of PY3, country staff were viewed as sufficient. ● Most interview and survey respondents consider IM’s in-country staff sufficient and appropriate. ● No recommendations. IM has performed very well. EQ2d. Has IM been adept at adjusting to the rapid growth of country buy-in, from the original nine countries in fiscal year (FY) 2017 to 18 countries and two regional buy-ins in FY 2019? 69 EQ FINDINGS CONCLUSIONS RECOMMENDATIONS ● IM has done an outstanding job of managing the project’s rapid growth from nine countries in PY1 to 18 countries and two regional buy-ins in PY3. ● Kenya and Ghana report expansion of geographic responsibilities without commensurate expansion of funding. ● Start-up has been most successful where at least one consortium partner already had a registered presence in-country; this has been the case in nearly every participating country. ● IM HQ leadership reported not having received clear planning guidance from USAID for country buy-ins. ● Activities in the PMI core work plan were viewed by IM as ambitious and not prioritized. ● Where target areas or activities have expanded, funding envelopes need to be reviewed. ● Simplifying the country buy-in planning process would save time and money. ● Introduce a systematic review of funding envelopes whenever new responsibilities or areas are added to an IM country activity portfolio. (IM) ● Consider abbreviating the country buy-in planning process to the development of a high-level activity plan and budget. (USAID/PMI and IM) ● Prioritize activities in future workplans; allow sufficient time for a startup/set up period and invite feedback from project partners on realistic time frames. (USAID/PMI with PSI) EQ2e. Has IM been able to hire staff, set up offices, launch activities, and continue activities on the agreed upon timelines? 70 EQ FINDINGS CONCLUSIONS RECOMMENDATIONS ● IM has continued to increase staffing across PY1-3 in response to the project’s growth and in line with the approved core workplan’s staffing plan. ● The IM project faced several major start-up challenges, including an award protest that resulted in a stop-work order which interrupted crucial staff hiring, a temporary USG closure, a disrupted partnership with WHO mandated by USG, and the Covid-19 pandemic. ● Despite serious obstacles, the project has been timely overall with submission of core and country workplans, financial and annual reports. ● Many country and HQ level IM project staff described struggles with uncertainties about when pre-approvals are needed under the IM contract that are still not entirely resolved. ● In-country respondents highlighted funding delays as the key challenge for timely completion of activities. ● Overall, IM is performing well in meeting country staffing needs and launching activities in a timely way. ● Several delays in PY1-2 were outside the project’s control. Delays from PSI in PY1 related to contract requirements were seemingly resolved with a staffing change. ● COVID-19 had profound impacts on the project’s ability to carry out activities as planned and in a timely way. IM found many innovative solutions, detailed in Annex 5. ● The contract mechanism presented a learning curve for both USAID/PMI and IM and created delays where pre￾approvals have been needed. It is not clear that it is the optimal mechanism for a service delivery project with many variables. ● It is not clear which funding delays are outside of USAID/PMI’s control and where improvements can realistically be made. ● Consider publishing the project’s COVID- 19 adaptations as useful lessons for future pandemics. (USAID/PMI and IM) ● For future contract agreements, provide guidance on where pre￾approvals are needed and the anticipated timing for completion of the pre-approval process, to help partners better anticipate and plan. (USAID/PMI and PSI) ● All parties proactively communicate where anticipated funding delays are concerned. Consider how to remove pressure to ‘front’ expenses from the prime partner wherever possible. (USAID/PMI, PSI, and in￾country partners) EQ2f. Has IM been adept at tackling the logistics of staffing, coordinating and managing logistics for SMC campaigns? Are the campaign activities in conflict with maintaining routine support for CM and MIP activities? 71 EQ FINDINGS CONCLUSIONS RECOMMENDATIONS ● SMC campaigns are widely considered one of the project’s biggest successes. Refer to findings under 1f. above. ● As with the TESs (see EQ2g. below), funding envelopes are reportedly sometimes too small to complete the campaigns as planned. ● Campaigns are described by IM leadership as logistically heavy lifts; each country has faced unique and severe challenges. ● IM has done an outstanding job of resolving challenges and security risks and finding solutions to complete SMC campaigns. ● IM is handicapped whenever funding is delayed or funding obligations are not sufficient to cover all campaign costs. ● Ensure that lines of communication from country to IM HQ level are effective where reporting of problems is concerned. (PSI) ● Refer to recommendation under 2e. above related to funding delays. EQ2g. Has IM been adept at tackling the logistics of staffing, coordinating and managing logistics for TES activities? ●IM is planning or implementing nine TESs across eight countries with local sub-contractors: Burkina Faso, Cameroon, Côte d'Ivoire, DRC, Kenya, Mali, Niger, and Rwanda. ● PMI HQ is pleased to have IM’s technical support and involvement in these countries, and the competence of IM’s TES staff at both HQ and country-level was praised. ● TESs have faced challenges in Cameroon, Mali and Kenya. ● Multiple IM HQ informants emphasized that the TES budget envelopes ($75k per study arm) are not always sufficient, and the PMI policy needs review. ● The investment in therapeutic efficacy studies (TES) is not in all cases adequately funded to yield meaningful results. ● Review and update TES funding ceilings and recognize country by country cost variations. (USAID/PMI) EQ2h. Has IM been adept at tackling the logistics of staffing, coordinating and managing logistics for OR activities? 72 EQ FINDINGS CONCLUSIONS RECOMMENDATIONS ● Five OR studies are underway in Benin, Mali, Cambodia, Senegal, and LAC. ● There appears to be some role confusion within the IM consortium as to who is in charge of tracking the Senegal study’s timeline and budget, which was developed in country and passed over to IM. ● KIs concurred that the Senegal study would benefit from having an IM project manager in country. ● Delays in obtaining USAID approval for laptops, tablets, and field and laboratory supplies have affected the Senegal MDA study’s implementation schedule. ● University of Thies has been an outstanding research partner in Senegal. ● USAID expressed some disappointment with PSI’s not having stepped in when a stronger hand was needed for OR challenges. ● The partner model incorporates sufficient flexibility in the existing partner principles. It is often other issues such as the available budget, skillset or national registration which would prevent making a change, should one be needed. ● The Senegal MDA study would benefit from having an IM project manager present in country to help troubleshoot. ● IM’s OR experiences and challenges can provide useful learning to PMI for prevention of similar problems in future projects. ● Introduce some flexibility or exceptions into the partner principles to enable change of lead partner if needed. (PSI) ● Explore the possibility of placing a research manager in country until the Senegal study is completed. (IM) ● Refer to recommendation under 2e. above related to funding delays. ● Recommendations from the Senegal study include closer coordination between research partners (UCSF and UT) to monitor the budget and ensure that teams in the field have all the resources they need; continuation of weekly phone calls between UCSF and UT; and development of a plan for how various partners intend to use and publish study data. ● Frame the project’s OR challenges and solutions as a useful ‘lessons learned’ close out deliverable. (USAID/PMI) EQ2i. Are PMP indicators agreed upon at the HQ and/or the country-level practical from a reporting perspective and are they useful from a programmatic perspective? 73 EQ FINDINGS CONCLUSIONS RECOMMENDATIONS ● Nearly all interview and survey respondents were positive about IM’s indicators. ● Some respondents felt that there were too many indicators or that some indicators were difficult to measure, even if the indicators themselves were relevant. ● There is consensus that some PMP indicators need clearer definitions or revisions. ● The indicators are clear and aligned with global standards for each technical component. However, they don’t reflect an overall holistic framework for health systems strengthening. ● There is general agreement that the PMP indicators are relevant, even though some interview and survey respondents feel that there are too many. ● There is not yet an indicator related to the integration of gender into the project. ● The project does not have indicators for internal operations, although this appears to be one of its outstanding achievements. ● IM would benefit from a conceptual framework that clearly identifies linkages and complementarities between all technical components and indicators for each. Such a framework would likely help to identify opportunities for an integrated approach across technical interventions and health system levels, including community-level. ● Consider ways to simplify and shorten indicators and emphasize outcomes and impacts over activities and outputs. Provide more guidance on how to use them. (IM) ● Use learning from IM’s gender focused activities and collaborations to develop suggested gender indicators for future projects. (USAID/PMI) ● Consider internal performance indicators for future projects. (USAID/PMI) ● Consider the development of an actual Theory of Change (ToC) in the next annual planning meeting that maps out assumptions that underlie the activities under each technical component and shows how they link with and complement one another. A ToC can also show where the project can best link with other PMI flagship projects. (USAID/PMI) EQ 3. GLOBAL RESULTS EQ3a. To what extent has IM achieved global level results laid out under each objective in the detailed program description of the award, including plans for and progress towards publications, documentation, and dissemination of best practices/lessons learned? What has IM developed in their Learning Agenda and other job aides, such as guidance on implementing SMC campaigns in the context of COVID-19? 74 EQ FINDINGS CONCLUSIONS RECOMMENDATIONS ● IM has been an active and credible participant, and sometimes a leader, in global level malaria meetings and conversations, at conferences, and on social media platforms in PY1-3. ● IM has engaged with Roll Back Malaria through various working groups, WHO and the SMC Alliance, which is linked with the RBM Country/Regional Support Partner Committee (CRSPC)25. ● IM partnered with Breakthrough ACTION to produce A Blueprint for Applying Behavioral Insights for Malaria Service Delivery: Methods and Frameworks for Improving Provider Behavior that provides steps for using insights into health provider behavior to improve the quality of malaria service delivery, and co-hosted a webinar on the tool with BA. ● IM launched multiple social media platforms, including a project website, blog posts, a Twitter account, a presence on LinkedIn, and a Flickr photo library with over 500 photos. ● IM presented in English and French on COVID-19 adaptations to iCCM and SMC programming as part of a Child Health Task Force (CHTF) webinar series and produced learning briefs from SMC campaigns. ● IM partner Jhpiego plans to publish the Cameroon and Kenya gender analyses that were completed in PY4, and has submitted a gender￾focused abstract to the ASTMH 2021 conference. ● Prime partner PSI has recently contracted with UCSF to provide technical support to countries for abstracts they wish to submit to ASTMH or for manuscripts submitted to peer reviewed journals. ● IM has produced a Learning Agenda through consultation with USAID PMI, to contribute to the body of knowledge on the most effective and efficient ways to deliver malaria services in four project intervention areas: QA, iCCM, MIP, and SMC. ● IM has been involved since PY1 in multiple, well-respected contributions to global malaria conversations and has plans in place to publish findings from project activities in both published journals and through ‘softer’ communication channels, including its website. ● IM has defined outputs for a project￾wide Learning Agenda. ● Strengthen opportunities to build the capacity of national colleagues in writing and publishing findings from IM project activities, using both global and national communication. (IM) ● Disseminate Learning Agenda findings and outputs as widely as possible at end of project. (IM) 25 The purpose of the Country/Regional Support Partner Committee (CRSPC) is to provide a platform to engage the RBM Partnership community in coordinating support to countries and regions as they execute their malaria control and elimination implementation programs. ANNEXES 76 ANNEX 1: STATEMENT OF WORK Assignment #: 008 [assigned by GH EvaLS] Global Health Evaluation and Learning Support Activity (GH EvaLS) Contract No. GS-10F-154BA STATEMENT OF WORK (SOW) Date of Submission: 10/29/2020 Last update: 2/08/2021 INSTRUCTIONS: Complete this template in MS Word to develop a SOW for your assignment, that may be an evaluation, a DQA, an assessment, or other analytic activity to be passed to the GH EvaLS team. From this point on, this assignment will be referred to as Assignment 008. Please be as thorough as possible in completing this SOW. The GH EvaLS team will assist you in refining your SOW which will be finalized when the Assignment 008 team is in place. Some of the sections below have been pre-populated with information that is common to most evaluation/analytic activities. Please review these details and edit as needed to fit the needs of your specific analytic assignment. Note: Refer to the USAID How-To Note: Evaluation SOW and the Evaluation SOW: Good Practice Examples when developing your SOW. I. SOW SPECIFIC INFORMATION A. TITLE: MID-TERM PROGRAM EVALUATION OF PMI IMPACT MALARIA B. FUNDER/REQUESTER/CLIENT USAID/Washington Office/Division: Office of Infectious Diseases/Malaria Division C. FUNDING ACCOUNT SOURCE(S): (Click on box(es) to indicate source of payment for this assignment) HIV TB X Malaria PIOET Other public health threats MCH FP/RH WSSH Nutrition Other (specify): 77 D. BUDGET CEILING: $(omitted) (Note: GH EvaLS will provide a cost estimate based on this SOW) E. PERFORMANCE PERIOD Expected start/end date (on or about): Mid-January – end of June 2021 F. LOCATION(S) OF ASSIGNMENT (Indicate where work will be performed): This will be a remote evaluation. II. TYPE OF ASSIGNMENT (Check the box to indicate the type of assignment) A. EVALUATION: 1. Performance Evaluation (Check timing of data collection)  Mid-term Endline Other (specify): Performance evaluations encompass a broad range of evaluation methods. They often incorporate before–after comparisons but generally lack a rigorously defined counterfactual. Performance evaluations may address descriptive, normative, and/or cause-and-effect questions. They may focus on what a particular project or program has achieved (at any point during or after implementation); how it was implemented; how it was perceived and valued; and other questions that are pertinent to design, management, and operational decision making 2. Impact Evaluation (Check timing(s) of data collection) Baseline Mid-term Endline Other (specify): ________ Impact evaluations measure the change in a development outcome that is attributable to a defined intervention. They are based on models of cause and effect and require a credible and rigorously defined counterfactual to control for factors other than the intervention that might account for the observed change. Impact evaluations in which comparisons are made between beneficiaries that are randomly assigned to either a treatment or a control group provide the strongest evidence of a relationship between the intervention under study and the outcome measured. B. ANALYTIC ASSIGNMENT: Assessment Assessments are designed to examine country and/or sector context to inform project design, or as an informal review of projects. Costing and/or Economic Analysis Costing and Economic Analysis can identify, measure, value and cost an intervention or program. It can be an assessment or evaluation, with or without a comparative intervention/program. Other Analytic Activity (Specify) 78 ________________________________________________________________ ________________________________________________________________ ________________________________________________________________ C. PEPFAR EVALUATION (see PEPFAR Evaluation Standards of Practice v3.1_October 2019): Note: If this is a PEPFAR-funded, check the box for the type of evaluation: 1. Process Evaluation (Check timing of data collection) c Mid-term Endline Other (specify): Process Evaluations focus on program or intervention implementation, including, but not limited to access to services, whether services reach the intended population, how services are delivered, client satisfaction and perceptions about needs and services, management practices. In addition, a process evaluation might provide an understanding of cultural, socio-political, legal, and economic context that affect implementation of the program or intervention. Example evaluation question: Are activities delivered as intended, and are the right participants being reached? 2. Outcome Evaluation  Outcome Evaluations determine if and by how much, intervention activities or services achieved their intended outcomes. They focus on outputs and outcomes (including unintended effects) to judge program effectiveness but may also assess program process to understand how outcomes are produced. It is possible to use statistical techniques in some instances when control or comparison groups are not available (e.g., for the evaluation of a national program). Example evaluation question: To what extent are desired changes occurring due to the program, and who is benefiting? 3. Impact Evaluation (Check timing(s) of data collection) c Baseline Mid-term Endline Other (specify): Impact evaluations (IEs) measure the change in an outcome that is attributable to a defined intervention by comparing actual impact to what would have happened in the absence of the intervention (the counterfactual scenario). IEs are based on models of cause and effect and require a rigorously defined counterfactual to control for factors other than the intervention that might account for the observed change. There are a range of accepted approaches to applying a counterfactual analysis, though IEs in which comparisons are made between beneficiaries that are randomly assigned to either an intervention or a control group provide the 79 strongest evidence of a relationship between the intervention under study and the outcome measured to demonstrate impact. Example evaluation question: What are the net effects of the program in achieving long term outcomes (e.g., changes in prevalence, incidence, mortality, sustainability)? 4. Economic Evaluation  Economic Evaluations identify, measure, value and compare the costs and outcomes of alternative interventions. Economic evaluations are a systematic and transparent framework for assessing efficiency focusing on the economic costs and outcomes of alternative programs or interventions. This framework is based on a comparative analysis of both the costs (resources consumed) and outcomes (health, clinical, economic) of programs or interventions. Main types of economic evaluation are cost-minimization analysis (CMA), cost-effectiveness analysis (CEA), cost-benefit analysis (CBA) and cost-utility analysis (CUA). Example evaluation question: What is the cost-effectiveness of this intervention in improving patient outcomes as compared to other treatment models. III. ASSIGNMENT BACKGROUND A. PROJECT/PROGRAM BEING EVALUATED/ANALYZED Project/Activity Title: PMI Impact Malaria Award/Contract Number: CA# 7200AAI8C00014 Award/Contract Dates: 2/13/2018 – 2/12/2023 Project/Activity Funding: USD $163,393,540.00 Implementing Partner(s): Population Services International (PSI); Medical Care Development International (MCDI); Jhpiego; University of California at San Francisco (UCSF) Project/Activity AOR/COR: Kimberly Connolly B. BACKGROUND OF PROJECT/PROGRAM/INTERVENTION Provide a brief background on the country and/or sector context; specific problem or opportunity the intervention addresses; and the development hypothesis. The mid-term evaluation of the five-year USAID/ID/PMI project (2018-2023) PMI Impact Malaria is being conducted to inform the structure and content of current and future USAID/PMI investments in malaria CM, prevention of MIP and other malaria drug￾based interventions. The evaluation is expected to accomplish the following objectives: 1. Assess and document progress toward achieving project objectives and whether desired results have occurred; 80 2. Determine the effectiveness and efficiency of project operations and management; 3. Capture lessons learned and identify key bottlenecks/gaps that can inform future PMI activities in CM, in the context of the PMI strategy. The results of the evaluation will be used by USAID Global Health Bureau/ID/PMI headquarters and mission staff as well as by PMI Impact Malaria project staff. The purpose of the PMI/Impact Malaria award is to provide implementation support services and technical assistance to countries to accelerate progress in comprehensive malaria facility and community service delivery including malaria CM, prevention of MIP, and other malaria drug-based interventions. The past decade has seen extraordinary progress in malaria control efforts, especially in the sub-Saharan Africa region. The scale-up of proven, cost-effective malaria interventions – such as effective CM and prevention of MIP – has contributed to substantial progress in malaria control and prevention. The risk of malaria is declining as a result of cumulative efforts and funding by the United States Government (USG) through the President’s Malaria Initiative (PMI), the USG and other governments through the Global Fund to Fight AIDS, Tuberculosis and Malaria (the Global Fund), national governments, the World Bank, the Bill and Melinda Gates Foundation, and many other donors and foundations. The World Health Organization’s (WHO) 2015 World Malaria Report26 estimates that more than 6.2 million malaria deaths were averted worldwide between 2000 and 2015. During that same time period, new malaria cases fell by 37%, and malaria mortality declined by an estimated 48% worldwide. Even greater reductions in malaria mortality were recorded in sub-Saharan Africa, where deaths among children under the age of five years declined by 71%. Based on these results, WHO and UNICEF reported that the Millennium Development Goal for malaria (halting and reversing malaria incidence by 2015) was achieved. However, despite historic gains, WHO reported that there were still an estimated 228 million new cases of malaria and approximately 405,000 malaria-attributed deaths worldwide in 2018 alone.1 The overwhelming majority of these cases and deaths occurred among young children in sub-Saharan Africa.1 Malaria infection during pregnancy contributes to maternal and newborn morbidity and mortality (with an estimated 10,000 maternal deaths and up to 200,000 newborn deaths each year globally). Gaps in intervention quality and coverage contribute to continued malaria related mortality and morbidity. Millions of people still do not receive the malaria prevention and treatment services that they need. For example, approximately 18% of women living in malaria-endemic settings who access antenatal services do not receive a dose of sulfadoxine pyrimethamine (SP) to prevent malaria during pregnancy (intermittent preventive treatment in pregnancy - IPTp). 26 World malaria report 2019 (mmv.org) 81 One area of great success in malaria prevention is the scale-up of seasonal malaria chemoprevention, an intervention that provides treatment doses of antimalarial medication to young children in areas of highly seasonal malaria transmission in the Sahel subregion. In the 12 countries that implement SMC in this region, 62% of eligible children (19 million of 31 million eligible) were treated in 2018.27 President’s Malaria Initiative The U.S. President’s Malaria Initiative, led by USAID and implemented together with the U.S. Centers for Disease Control and Prevention (CDC), is the U.S. Government’s primary vehicle for assisting malaria affected countries to scale-up proven malaria control and elimination interventions. When it was launched in 2005, the goal of PMI was to reduce malaria-related mortality by 50% across 15 high-burden countries in sub￾Saharan Africa through a rapid scale-up of four proven and highly effective malaria prevention and treatment measures: insecticide-treated mosquito nets; indoor residual spraying; accurate diagnosis and prompt treatment with ACTs; and intermittent preventive treatment of pregnant women (IPTp). PMI developed a U.S. Government Malaria Strategy for 2009–2014, which articulated a long-term vision for malaria control in which sustained high coverage with malaria prevention and treatment interventions would progressively lead to malaria-free zones in Africa, with the ultimate goal of worldwide malaria eradication by 2040-2050. The contributions of PMI, together with those of other partners, led to dramatic improvements in the coverage of malaria control interventions in PMI-supported countries, and all 15 original countries have documented substantial declines in all-cause mortality rates among children less than five years of age. The PMI Strategy for 2015-202028 accounts for the progress over the past decade and new and emerging challenges. Malaria prevention and control remains a major U.S. foreign assistance objective and PMI’s strategy fully aligns with the U.S. Government’s vision of ending preventable child and maternal deaths and protecting communities from infectious diseases. Under the PMI Strategy 2015-2020, the USG’s goal is to work with PMI-supported countries and partners to further reduce malaria deaths and substantially decrease malaria morbidity, towards the long-term goal of elimination. To do so, PMI works with National Malaria Control Programs (NMCPs) and partners to accomplish the following objectives by 2020: 1. Reduce malaria mortality by one-third from 2015 levels in PMI-supported countries, achieving a greater than 80% reduction from PMI’s original 2000 baseline levels. 2. Reduce malaria morbidity in PMI-supported countries by 40% from 2015 levels. 27 https://www.pmi.gov/docs/default-source/default-document-library/pmi￾reports/pmi_strategy_2015-2020.pdf 28 https://www.pmi.gov/docs/default-source/default-document-library/pmi￾reports/pmi_strategy_2015-2020.pdf 82 3. Assist at least five PMI-supported countries to meet the WHO criteria for national or subnational pre-elimination.29 These objectives are being accomplished by emphasizing five core areas of strategic focus: 1. Achieving and sustaining scale of proven interventions 2. Adapting to changing epidemiology and incorporating new tools 3. Improving countries’ capacity to collect and use information 4. Mitigating risk against the current malaria control gains 5. Building capacity and health systems towards full country ownership PMI’s approach to programming resources and defining support at the country level is to work within the framework of the National Malaria Control Program’s existing strategic plan. PMI prioritizes investments to support high priority malaria program needs and gaps, taking into account investments by partner countries and other donors and partners active in malaria control. Each year, PMI undertakes a planning process, working closely with partner country counterparts and key stakeholders, to develop a country Malaria Operational Plan (MOP). These plans summarize the current policies and status of malaria prevention and control intervention scale-up; describe the surveillance, monitoring, and evaluation strategies to be supported; identify challenges and programmatic and resource gaps; and provide a description of planned USG investments by activity, budget amount, and implementing partner. Annual MOPs thus serve as the guide for implementing partners on PMI’s expectations across technical areas and for each specific activity, informing the basis for partner annual work plan development. PMI MOPs undergo rigorous interagency technical review and are ultimately approved by the U.S. Global Malaria Coordinator and made publicly available at www.pmi.gov in advance of the availability of fiscal year new funding. The work that is carried out under this contract builds on more than a decade of PMI and USAID investments at the country level in strengthening malaria diagnosis and CM, prevention and treatment of MIP, and related malaria service delivery efforts, and takes into account learning across countries and the current evidence base for these interventions and approaches. During the first five years of the initiative, PMI invested in the Improving Malaria Diagnostics project, which was designed to expand, improve, and build capacity for laboratory-based diagnosis of malaria in PMI partner countries. This project was followed by the MalariaCare project, which was designed with an expanded scope to support the scale-up of and build country capacity for malaria CM, which included strengthening of prompt malaria diagnosis and appropriate treatment. The overall purpose of this contract is to strengthen malaria services delivered in health facilities and community settings. Specifically, the scope includes technical assistance and 29 PMI has not yet decided on the five countries Contract No: 7200AA18C00014 3 83 implementation support for malaria CM (the primary focus), MIP services (i.e., IPTp and CM of pregnant women), and other malaria drug-based interventions appropriate for the given epidemiological setting and as agreed to with partner countries and PMI. To complement these activities, the scope also includes technical and implementation support for strengthening facility- and community-level case reporting, addressing provider behaviors, conducting operational research, and provision of technical leadership at the global level. For the purposes of this contract, malaria drug-based interventions encompass WHO recommended and/or proven prevention and treatment approaches using antimalarials, such as seasonal malaria chemoprevention (SMC) and mass drug administration (MDA). However, newer drug-based approaches, which may be endorsed by WHO during the timeframe of the contract, may also be included for operational research, pilots, and/or scale-up, as directed by PMI. Malaria Case Management Prompt, effective CM is a crucial component of reducing malaria morbidity and mortality and a cornerstone of malaria control. In 2010, WHO revised its treatment guidelines, calling for all patients with suspected malaria to undergo quality-assured diagnostic testing with either microscopy or rapid diagnostic tests (RDTs), and for treatment decisions to be based on test results. Consistent with these guidelines, PMI supports partner countries to implement universal diagnostic testing, and when a test is positive, provide immediate treatment with appropriate antimalarial drugs at public health facility and community-levels, and in the private sector when/if relevant. The goal is for all patients with suspected malaria to seek care, all patients with suspected malaria to be quickly identified and tested, and all patients with confirmed malaria to receive effective treatment without delay. Over the last decade, PMI has invested significant resources to support the scale-up of malaria CM at health facility and community-levels. PMI has worked closely with ministries of health to invest in efforts aimed at building capacity for effective implementation of malaria CM, providing support for all elements of a comprehensive quality program to diagnose and treat patients appropriately. PMI has worked alongside partner countries to promote and expand quality assurance of CM by providing support for strengthening expert diagnostic capacity at reference laboratories and through support for training and supervision activities at health facilities, including on-site training, mentoring, and troubleshooting with routine supervision that assesses and documents health worker performance through direct observation, facility and record review, and re-checking of blood slides. In collaboration with partner countries, PMI has also supported the implementation of integrated community CM (iCCM) for malaria, pneumonia, and diarrhea. To date, all 19 PMI focus countries in sub-Saharan Africa have received PMI financial and technical support for iCCM efforts. Because of these efforts, RDTs and ACTs are now widely available and thousands of facility and community health workers have been trained in their use. The proportion of suspected malaria cases confirmed with a diagnostic test and treated with a recommended antimalarial has increased from baseline measures in nearly every PMI focus country, and most countries are scaling-up quality assurance systems for CM. However, despite these advances, progress in CM (which varies significantly across PMI focus countries) 84 continues to lag behind expectations for several reasons. Country-level barriers include poor provider practices (e.g., continued presumptive treatment or treatment despite a negative diagnostic test) and insufficient ability of individuals (including caregivers) to recognize symptoms of malaria and seek prompt diagnosis and appropriate care. Broader service delivery challenges include stock outs of essential drugs and commodities (including laboratory supplies, ACTs, RDTs, and SP); lack of access to public sector facilities and to laboratory services; poor health provider knowledge and practices in diagnosing and/or treating patients; and lack of or poor-quality supervision. There also continue to be critical policy barriers to effective malaria CM such as outdated country-level policies and guidelines related to treatment of severe malaria (including treatment of severe malaria for pregnant women) and regulations preventing private sector or community health workers from performing diagnostic tests. MIP Services Malaria infection during pregnancy contributes to maternal and newborn morbidity and mortality. Approximately 125 million women living in malaria-endemic countries throughout the world become pregnant every year, more than 30 million of whom live in tropical areas of Africa where there is intense transmission of Plasmodium falciparum.30 In these areas, malaria infection directly contributes to adverse outcomes in maternal and newborn health. Pregnant women, particularly those in their first or second pregnancies, are particularly vulnerable to malaria as pregnancy reduces a woman’s immunity to malaria, making her more susceptible to malaria infection and increasing the risk of illness, severe anemia, and death. For the unborn child, maternal malaria increases the risk of miscarriage, stillbirth, premature delivery, and low birth weight - a leading cause of child mortality. Preventive interventions, including IPTp, promotion of long-lasting insecticide treated net usage, and promoting prompt health care seeking for fever during pregnancy have been shown to significantly reduce the risk of maternal anemia, low birth weight, and perinatal deaths. In line with WHO guidelines, PMI works with partner countries to support a three-pronged approach to reducing MIP in areas with moderate to high levels of malaria transmission: (1) provision and promotion of insecticide-treated mosquito nets to pregnant women; (2) administration of IPTp with SP, where indicated; and (3) prompt diagnosis and effective treatment of malaria and anemia among pregnant women.31 PMI may work with more than one PMI implementing partner at country level to support this three-pronged approach. PMI supports the delivery of these activities through the antenatal care (ANC) platforms in partner countries and promotes collaboration between NMCPs and reproductive, maternal, newborn and child health (RMNCH) programs at the country level. Coordination and planning between NMCP and RMNCH units are essential to increasing coverage of MIP interventions including successful uptake of IPTp delivered through ANC services. Coordination with other infectious 30 Dellicour S, Tatem AJ, Guerra CA, Snow RW, ter Kuile FO. Quantifying the Number of Pregnancies at Risk of Malaria in 2007: a Demographic Study. PLOS Med. 2010;7:1, e1000221 31 http://www.who.int/malaria/areas/high_risk_groups/pregnancy/en/ 85 disease programs (including HIV) is also important for the successful delivery of MIP (MIP) services provided to pregnant women. With support from PMI, coverage of pregnant women with at least two doses of IPTp in PMI focus countries has increased from a baseline of 14 to 38%.32 While progress on IPTp uptake has been slower than expected, PMI recognizes the need for continued efforts and creative approaches to scaling-up IPTp through addressing remaining barriers hindering uptake. In addition to the service delivery challenges listed above, some of the key challenges and barriers to progress in uptake of MIP interventions at the country level include inadequate provider adherence to national MIP IPTp and diagnosis and treatment guidelines; slow country roll-out and implementation of the updated 2012 WHO IPTp policy recommendations; lack of effective communication between and inconsistency across NMCP and RMNCH program policies and guidelines; delays in pregnant women initiating their first ANC visit and failure to return regularly; and consumer perceptions of ANC services. Global partners, including PMI, are working with countries to develop and evaluate new strategies and approaches for prevention and control of MIP in service delivery settings – including the implementation of intermittent screening and treatment in pregnancy (ISTp) and single screening and treatment in pregnancy (SSTp), with potential applicability for low transmission and pre-elimination areas, alternative drugs for IPTp, and possible implementation of community delivery of IPTp to improve uptake. PMI may choose to support countries in the evaluation of new approaches in the context of operational research but does not support new approaches in a routine matter prior to WHO policy recommendations. Furthermore, the new 2016 WHO ANC Guidelines which call for a minimum of 8 contacts with a health provider have had implications for MIP programming.33 Other Drug-Based Service Delivery Interventions/New Tools Over the past several years, WHO has recommended new drug-based interventions for prevention and reduction of malaria that are targeted at areas of high transmission for specific high risk groups (e.g., Intermittent Preventive Treatment of infants (IPTi) and SMC). Although IPTi has not been taken up by most countries, SMC - the administration of a curative dose of antimalarial drugs in areas of highly seasonal transmission at monthly intervals to children aged 3–59 months without malaria symptoms – is being rolled out in countries that meet the WHO criteria for SMC implementation.34 PMI currently supports Mali and Senegal’s NMCPs to implement SMC, and plans are underway to support SMC implementation in additional PMI partner countries where the intervention is recommended, and where PMI, together with the NMCP, has prioritized this support. However, reaching sustained high uptake of SMC in targeted 32 PMI Tenth Annual Report to Congress, April 2016: https://www.pmi.gov/about/pmi-annual￾report-2016 33 http://apps.who.int/iris/bitstream/10665/250800/1/WHO-RHR16.12-eng.pdf?ua=1 34 http://www.who.int/malaria/areas/preventive_therapies/children/en/ 86 geographic areas will be challenging for two main reasons. First, implementation often requires building on an existing community-based platform that is designed for routine public health service delivery, rather than mass, campaign-like services. Second, high coverage and adherence to the full treatment course at each round of delivery requires strong community sensitization and buy-in. In areas approaching interruption of P. falciparum transmission, there has been a renewed interest in the use of MDA as a tool to reduce the parasite reservoir and malaria transmission from a population. Although PMI has not yet invested in MDA implementation, there are several pilot and research activities testing various MDA approaches with support from a variety of research and development partners, including PMI, in different transmission settings across Africa and Asia. In November 2015, WHO reviewed the available evidence and recommended that MDA could be considered in certain situations, including geographic areas approaching elimination.35 However, questions remain about its effectiveness and feasibility in specific country contexts. As further evidence is developed on the appropriate indications for MDA, PMI may support additional operational research or piloting of MDA to assess feasibility and effectiveness of this strategy. Decisions regarding when and where MDA would be deployed will be made by PMI country teams, in consultation with NMCP and PMI technical leadership. Finally, as countries move toward pre￾elimination at national or sub-national levels, identifying and treating all malaria patients and individuals carrying parasites is critical. As in any other setting, in the pre￾elimination setting, all malaria cases should be confirmed with a diagnostic test and treated with effective antimalarials. Community health workers often become the foundation for malaria CM as the need for rapid diagnosis, treatment, and response necessitates quick and easy access to care. In addition, single, low-dose primaquine for Plasmodium falciparum could be considered. Surveillance in the pre-elimination setting must be reinforced. Timely, complete, and accurate reporting of passively detected and confirmed malaria cases diagnosed in both the public and private sectors is required. Surveillance system components may require active/reactive case detection (RCD), treatment, follow up, and reporting. PMI is currently working with NMCPs to support implementation and/or evaluation of RCD activities in areas of Cambodia, Ethiopia, Madagascar, Rwanda, Senegal, and Zanzibar that are targeted for elimination, and may consider support for scaling- up these efforts based on the results and implementing and/or evaluating RCD in additional countries, as appropriate. C. THEORY OF CHANGE (TOC) OF PMI IMPACT MALARIA The objectives below are listed in ascending order in terms of both anticipated funding levels as well as in total level of effort that will be requested to achieve each objective (e.g., work requested under Objective 1 will require more time and resources to achieve than Objective 2 requires). While implementing requested services and technical support, the contractor will build capacity and skills of ministry of health and other in- 35 http://www.who.int/malaria/publications/atoz/role-of-mda-for-malaria.pdf?ua=1 87 country staff at all levels of the health system (i.e., provincial, district, health facility, community level and private sector where appropriate) without creating parallel structures. Technical and material assistance is to be delivered in support of the national malaria control and related health policies and strategies. The intent is to build the capacity of the government and other key stakeholders to lead planning and implementation of all malaria control service delivery strengthening program activities. Objective 1: Improve quality of and access to malaria CM and MIP interventions. The contractor will provide technical assistance and/or implementation support for malaria CM and MIP interventions at public health facilities, community level through iCCM or other community-based platforms, and the private sector, as appropriate. Activities must be informed by routine data, supervision data, and other information collected through monitoring and evaluation efforts to strategically identify needs and target interventions/activities. In addition, activities must be carried out in coordination with PMI and USAID central and bilateral partners working in the areas of supply chain, social and behavior change communication (SBCC), and surveillance monitoring and evaluation (SM&E), and other partners (as appropriate) working in RMNCH, HIV/AIDS, and infectious diseases. Prioritization of country-level activities will be guided by the approved country malaria operational plan. Illustrative tasks include, but are not limited to: • Provide technical assistance to ministries of health in assessing, developing, and updating country diagnostic quality assurance frameworks, and operational support to disseminate these materials to all levels of the health care system. • Participate in and/or provide material assistance to functional in-country technical working groups and stakeholder workshops, or advocate for the formation or revitalization of such groups. • Provide technical and material assistance for training and supervision of health care providers, at all levels, in the diagnosis and treatment of uncomplicated malaria, treatment and/or referral of severe malaria, and MIP. • Provide technical assistance to support implementation of quality assurance/quality control systems on diagnostic testing for malaria. • Provide technical and implementation support for strengthening facility and community level case reporting. • Provide technical assistance to support the introduction and/or expansion of diagnostic and clinical malaria services at the community level and in the private sector, where appropriate. • Identify and prioritize key technical and programmatic bottlenecks at the country level that require an evidence‐based response for malaria CM and MIP interventions. • Provide technical assistance to support provider/health facility linkages to the supply chain management systems and collaborate with ministry of health supply chain and pharmacy management departments and relevant implementing partners. 88 • Provide technical and material assistance to support the development and implementation of SBCC to increase provider adherence to diagnostic testing, appropriate malaria treatment, and uptake of MIP interventions, and to support provider counseling and promotion of malaria service delivery interventions for clients. • Provide technical and material assistance for training and supervision of health providers and data managers in the accurate recording and reporting of CM and MIP data through the routine health information system, and the analysis and use of data by supervisors and program managers. • Facilitate greater linkages in malaria service delivery programming between NMCPs, RMNCH, HIV/infectious disease units, and other relevant ministry of health entities. Objective 2: Improve quality of and access to other malaria drug-based approaches and provide support to pilot/scale-up newer malaria drug-based approaches. The contractor will provide technical assistance and/or implementation support for WHO-recommended and/or proven drug-based approaches using antimalarials. The contractor will also provide technical assistance and/or implementation support for piloting and/or scaling-up of newer drug-based approaches. Illustrative tasks include, but are not limited to: • Provide comprehensive technical assistance and/or implementation support for SMC in malaria policies and guidelines, training and supervision materials, and countries that meet the criteria for its implementation.36 • Provide a wide range of assistance to pilot and/or scale-up MDA, additional CM and surveillance approaches (e.g., primaquine and reactive case detection) in pre-elimination/elimination areas)37, and other drug-based approaches as appropriate given the malaria transmission setting. • Conduct situation analyses on drug-based interventions to obtain information on where they should be implemented, existing activities, availability of human resources, procurement and supply chain management in the areas targeted for the interventions, functionality of the existing pharmacovigilance system, and SBCC partner activities. • Work with NMCPs and PMI to provide evidence for identifying the most effective drug-based approaches for areas with changing malaria transmission risk, including pre-elimination and elimination areas. • Facilitate coordination between ministries of health and implementing partners working in SM&E, supply chain, and SBCC (e.g., facilitate collaboration with supply chain partners to ensure adequate forecasting of SMC-related drugs, etc.). • At national and district levels, assist with implementation plans and budgeting, organize stakeholders’ meetings, and facilitate health care worker training. 36 http://www.who.int/malaria/areas/preventive_therapies/children/en/ 37 http://www.who.int/malaria/publications/atoz/role-of-mda-for-malaria.pdf?ua=1 89 • Identify approaches to improve coverage, quality, and efficiency of interventions by addressing country-specific bottlenecks. Objective 3: In support of Objectives 1 and 2, provide global technical leadership, support operational research, and advance program learning. The Contractor will provide global technical leadership and contribute to the development of global policies on standards in malaria CM, MIP interventions, SMC, and pre-elimination/elimination approaches, including participation in relevant WHO/Roll Back Malaria (RBM) committees and technical workshops/conferences. The contractor will also design, conduct, evaluate, and/or use results from operational research and/or monitoring activities to advance program learning (both globally and in the field) in support of Objectives 1 and 2. Illustrative tasks include, but are not limited to: • Provide leadership and guidance to the global malaria service delivery community, particularly through RBM’s relevant technical working groups, to identify priority activities or actions to accelerate progress in malaria service delivery. • Facilitate collaboration and participation of research, program, and technical partners to work together on improving global and country level efforts and aide in the prioritization of activities. • Drawing on experiences across multiple countries, identify common gaps in technical areas and/or policies that require a global and/or regional response. • Support the establishment or updating of evidence‐based international guidelines, norms, and standards in collaboration with WHO and other international partners. • Inform policy development at the global level by ensuring that data and best practices gathered through operational research studies and implementation efforts, under Objectives 1 and 2, are disseminated across global channels. • Participate in relevant, high-impact conferences and meetings that cover malaria service delivery. • Support operational research studies to improve the delivery of existing interventions; inform where and how to implement newer pre￾elimination/elimination interventions based on transmission level and local setting; and test the feasibility, acceptability, and cost effectiveness of novel drug-based approaches or diagnostic tests. • Support monitoring the efficacy of ACTs and SP. • Coordinate with in-country research partners to design site-appropriate projects and access key information. • Collect, analyze, and disseminate data that will inform global guidance and policy development, and program adoption and implementation. • Draft and disseminate peer-reviewed publications through an array of channels. Expected Outcomes: The contractor, in collaboration with National Malaria Control Programs, is expected to achieve the following outcomes. There might be unforeseen circumstances outside of 90 the management interest of the contractor that could prevent or delay completion of the expected outcomes identified below. USAID will take this into consideration when reviewing progress toward completion of the expected outcomes. By 2023 (the end of the project), the partner will be expected to achieve the following outcomes where programmatically applicable*: • A median of at least 80% of patients with suspected malaria receiving a diagnostic test • An average of 80% of confirmed malaria cases receiving effective malaria treatment according to standard national protocols • A 15% median increase in the percentage of pregnant women receiving two or more doses of IPTp for malaria during their last pregnancy • For each round of Seasonal Malaria Chemoprophylaxis, 80% of targeted children receive a dose of SMC** * Programmatically applicable means the countries who ask the partner to provide services related to that indicator will be part of the calculation. ** Countries will define SMC targets in their annual workplans. D. STRATEGIC OR RESULTS FRAMEWORK FOR THE PMI IMPACT MALARIA One cohesive technical vision for Impact Malaria (IM) is set by PMI in its 2015-2020 strategy. • IM’s contributions are clearly defined as supporting NMCPs in their work to: • Get the right diagnostics and treatment to more patients with suspected fever and confirmed malaria cases; • Increase the provision of intermittent preventive treatment (IPTp) for pregnant women; • Deploy innovative approaches including season malaria chemoprevention (SMC), or other drug-based approaches, as appropriate; • Work at subnational, national, and global levels to bolster the linkage of country systems with global policies and dialogue; and • Strengthen malaria health systems and the rigorous use of data for decision making. IM aims to achieve these by applying its “Impact Model” (below) to each of these areas of work. Figure 1. IM’s malaria service delivery “Impact Model” 91 E. GEOGRAPHIC COVERAGE What is the geographic coverage and/or the target groups for the project or program that is the subject of analysis? All 20 countries/regions currently buying in for PMI Impact Malaria support. Countries/focus regions include: Africa Bureau, Benin, Burkina Faso, Cambodia, Cameroon, Côte d’Ivoire, Democratic Republic of Congo, Ghana, Kenya, LAC Bureau, Madagascar, Malawi, Mali, Niger, RDMA - Laos, Rwanda, Senegal, Sierra Leone, Tanzania, and Zambia. IV. PURPOSE, AUDIENCE & APPLICATION A. PURPOSE Why is this assignment being conducted (purpose of assignment)? Provide the specific reason for this assignment linking it to future decisions to be made by USAID leadership, partner governments, and/or other key stakeholders. The mid-term evaluation of the five-year USAID/ID/PMI project (2018-2023) PMI Impact Malaria is being conducted to inform the structure and content of current and future USAID/PMI investments in malaria CM, prevention of MIP and other malaria drug-based interventions. B. AUDIENCE Who is the intended audience for this analysis? Who will use the results? If listing multiple audiences, indicate which are most important. 92 The results of the evaluation will be used by USAID Global Health Bureau/ID/PMI headquarters and mission staff as well as by PMI Impact Malaria project staff. C. APPLICATIONS AND USE How will the findings be used? What future decisions will be made based on these findings? PMI Impact Malaria will use the results to adjust their current activities and management activities as reasonable within the remaining period of the contract. USAID Global Health Bureau/ID/PMI will use the findings to help shape potential future investments in malaria service delivery projects. V. EVALUATION/ANALYTIC QUESTIONS & MATRIX: Instructions: Questions should be: a) aligned with the assignment purpose and the expected use of findings; b) clearly defined to produce needed evidence and results; and c) answerable given the time and budget constraints. Include any disaggregation (e.g., sex, geographic locale, age, etc.), they must be incorporated into the assignment questions. USAID Evaluation Policy recommends 1 to 5 evaluation questions. State the method and/or data source and describe the data elements needed to answer the questions. Evaluation Questions Data Source(s) Sampling Selection Criteria Data Analysis Method 1 COUNTRY-LEVEL PERFORMANCE: To what extent has PMI Impact Malaria achieved the country-level objectives? 1. a To what extent has PMI Impact Malaria achieved the technical and programmatic objectives described in annual country and core work plans and PMI Impact Malaria performance monitoring plan (PMP)? Desk/docume nt review Descriptive analysis by Objective in Project Program description 93 1. b Is there evidence of in-country capacity improvements in malaria diagnosis and CM and prevention of MIP at various levels of the health system (national, regional, district, CHW)? [Taking into account guidelines, training, supervision checklists.] a. Consider analysis of routine data on malaria testing, treatment (including presumptive treatment), IPTp, etc. b. Consider a focus on health provider behaviors (quality of care) utilizing supervision and training data (pre￾and post-test scores from trainings, OTSS+ results from the PMI Impact Malaria Data Hub, etc.) c. Consider evidence or perceptions of overall improvements at the health facilities targeted by PMI Impact Malaria beyond individual Online survey Key informant interviews (KIIs) Project/countr y data (from the data hub) and reports Online survey – all 20 PMI Impact Malaria countries/regio ns Purposive sampling of key informants (KIs) Selected project/country data (from the data hub) and reports Analysis of online survey data Qualitative analysis of KII transcripts Data abstraction (from the data hub), desk review of project/country documents 94 Evaluation Questions Data Source(s) Sampling Selection Criteria Data Analysis Method improvements in HW behaviors and malaria services. 1. c Do checklists and other tools capture useful data on the status and quality of CM? Are they appropriate and informative? Is implementation of OTSS+ disruptive to provision of services (does it take too much time)? Project/countr y data (from the data hub) and reports Project/country data (from the data hub) and reports for four countries: Ghana, Kenya, Cameroon, Niger Data abstraction (from the data hub), desk review of project/country documents 1. d Are results from checklists/other tools used by PMI Impact Malaria to make adjustments to training and supervision to improve quality? Online survey KIIs Online survey – all 20 PMI Impact Malaria countries/regio ns Purposive sampling of KIs Analysis of online survey data Qualitative analysis of KII transcripts 1. e Has the development of the PMI Impact Malaria Data Hub and the associated efforts to access national HMIS data for PMP reporting resulted in tangible improvements to data use? Consider the LOE required for the digitization of OTSS+ checklists. Online survey KIIs Online survey – all 20 PMI Impact Malaria countries/regio ns Purposive sampling of KIs Analysis of online survey data Qualitative analysis of KII transcripts 95 Evaluation Questions Data Source(s) Sampling Selection Criteria Data Analysis Method 1.f Have SMC coverage and adherence objectives been met in areas where PMI Impact Malaria has been supporting SMC implementation? Online survey KIIs Project/countr y data (from the data hub) and reports Online survey Purposive sampling of KIs Project/country data (from the data hub) and reports Focus: 3 PMI Impact Malaria countries with SMC implementatio n support: Cameroon, Mali, Niger Analysis of online survey data Qualitative analysis of KII transcripts Data abstraction (from the data hub), desk review of project/country documents 2 MANAGEMENT: To what extent has PMI Impact Malaria met the management requirements and functions outlined in the agreement, including planning, allocation of funds, coordination among the PMI Impact Malaria partnership (PSI, MCDI, UCSF, Jhpiego), staffing requirements, and in-country support? 2. a Has PMI Impact Malaria headquarters and PMI COR team oversight and management aided or hindered PMI Impact Malaria in accomplishing work plan objectives, both at central and country level? Online survey KIIs Online survey – HQ and all 20 PMI Impact Malaria countries/regio ns Purposive sampling of KIs Analysis of online survey data Qualitative analysis of KII transcripts 96 Evaluation Questions Data Source(s) Sampling Selection Criteria Data Analysis Method 2. b Has coordination between PMI Impact Malaria and partners in country (PMI RAs, NMCPs, other implementing partners) aided or hindered PMI Impact Malaria in accomplishing country work plan objectives? Online survey KIIs Online survey – HQ and all 20 PMI Impact Malaria countries/regio ns Purposive sampling of KIs Analysis of online survey data Qualitative analysis of KII transcripts 2. c Is in-country presence of PMI Impact Malaria staff sufficient and appropriate? Online survey KIIs Online survey – HQ and all 20 PMI Impact Malaria countries/regio ns Purposive sampling of KIs Analysis of online survey data Qualitative analysis of KII transcripts 2. d Has PMI Impact Malaria been adept at adjusting to the rapid growth of country buy-in, from the original 10 countries in FY 2017 to 17 countries and 2 Regional buy-ins in FY 2019? Online survey KIIs Online survey – HQ and all 20 PMI Impact Malaria countries/regio ns Purposive sampling of KIs Analysis of online survey data Qualitative analysis of KII transcripts 97 Evaluation Questions Data Source(s) Sampling Selection Criteria Data Analysis Method 2. e Has PMI Impact Malaria been able to hire staff, set up offices, launch activities, and continue activities on the agreed upon timelines? • For example, have the trainings planned for malaria diagnosis, CM, management of severe malaria, prevention of MIP, etc. been implemented as planned (understanding COVID-19 likely disrupted more recent activities)? Online survey KIIs Online survey – HQ and all 20 PMI Impact Malaria countries/regio ns Purposive sampling of KIs Analysis of online survey data Qualitative analysis of KII transcripts 2.f Has PMI Impact Malaria been adept at tackling the logistics of staffing, coordinating and managing logistics for seasonal malaria chemoprevention (SMC) campaigns? Are the campaign activities in conflict with maintaining routine support for CM and MIP (MIP) activities? Online survey KIIs Online survey Selected KIs Focus: 3 PMI Impact Malaria countries with SMC implementatio n support: Cameroon, Mali, Niger Analysis of online survey data Qualitative analysis of KII transcripts 98 Evaluation Questions Data Source(s) Sampling Selection Criteria Data Analysis Method 2.g Has PMI Impact Malaria been adept at tackling the logistics of staffing, coordinating and managing logistics for Therapeutic Efficacy Study (TES) activities? Online survey KIIs Online survey Selected KIs Focus: 6 PMI Impact Malaria countries with TES activities: Cote d’Ivoire, DRC, Cameroon, Mali, Rwanda, Burkina Faso Analysis of online survey data Qualitative analysis of KII transcripts 2.i Has PMI Impact Malaria been adept at tackling the logistics of staffing, coordinating and managing logistics for Operational Research (OR) activities? Online survey KIIs Online survey Selected KIs Focus: 4 PMI Impact Malaria countries with OR activities: Mali, Benin, Cambodia, and Senegal Analysis of online survey data Qualitative analysis of KII transcripts 2.j Are PMP indicators agreed upon at the HQ and/or the country level practical from a reporting perspective and are they useful from a programmatic perspective? Online survey KIIs Online survey – HQ and all 20 PMI Impact Malaria countries/regio ns Purposive sampling of KIs Analysis of online survey data Qualitative analysis of KII transcripts 3 GLOBAL RESULTS: What results have been realized at the global level? 99 Evaluation Questions Data Source(s) Sampling Selection Criteria Data Analysis Method 3. a To answer this question, consider the extent to which PMI Impact Malaria has achieved global level results laid out under each objective in the detailed program description of the award, including plans for and progress towards publications, documentation, and dissemination of best practices/lessons learned. In addition, consider consulting PMI Impact Malaria’s Learning Agenda and other job aides that have been developed such as guidance on implementing SMC campaigns in the context of COVID￾19. Document/des k review KIIs Review of PMI Impact Malaria publications and products KIIs with global stakeholders Desk review of documents Qualitative analysis of KII transcripts VI. DATA COLLECTION METHODOLOGY Instructions: Describe the recommended methods for this assignment. Selected methods should be aligned with the assignment questions and fit within the time and resources allotted for the assignment. Also, include the sample or sampling frame in the description of each method selected. 100 Note Related to Methods A critical part of the methodology will be to assess the situation during the COVID-19 pandemic. It is anticipated that for the assignment, especially where borders are closed and access is restricted, highly qualified national/regional evaluators and experts will be contracted. Additionally, virtual approaches to data collection will be used, including virtual stakeholder meetings, key informant interviews, and focus groups, where possible. See also USAID Guide to Remote Monitoring in COVID-19. The Evaluation Team (ET), in collaboration with USAID, will finalize the evaluation methods before fieldwork begins. USAID/PMI expects that, at a minimum, the ET will: • Upon award, familiarize themselves with documentation about the project and USAID/PMI’s current assistance in the health area (specifically malaria) in the region. USAID/PMI will ensure that this documentation is available to the Team prior to their start of work; • Review and assess the existing performance and effectiveness information or data; • Virtually meet and interview USAID/PMI project beneficiaries, partners, and host government counterparts at appropriate levels; • Interview USAID/PMI staff and a representative number of experts working in the sector; • Submit the evaluation workplan (including the design, methodology, and data collection tools) to USAID to be approved prior to the start of data collection. PMI’s vision for the structure of the evaluation will include four components. • Review of key project documents outlined below to understand project goals and assess progress in achieving major milestones – will inform Evaluation Questions 1 and 3. • Survey across all 19 PMI Impact Malaria countries/regions aimed at all Mission and PMI Headquarters staff – will inform EQ 2. • Key informant interviews (KIIs) with PMI Impact Malaria and PMI staff about the management and working relationship with PMI Impact Malaria in 4 countries with investments in service delivery activities (Ghana, Kenya, Cameroon, and Niger], and 1 country with a limited buy-in for operational research activities (selected among Benin or Senegal) – will inform EQs 1 and 2; Interviews with global stakeholders about contributions and successes at global level – will inform EQ 3. • Analysis of supervision and monitoring data including checklist tools in 4 countries (Ghana, Kenya, Cameroon, and Niger) to assess improvements in CM of malaria and provision of MIP services as well as quality of SMC intervention where relevant – will inform EQ 1. 101 Document and Data Review Please list of documents and data recommended for review. The desk review includes at a minimum: • PMI Impact Malaria SOW; • PMI Impact Malaria materials: Annual and Quarterly Reports, Annual Work Plans, MEL/PMP Plans, sector assessments, trip reports, performance reports, gender analyses, relevant sections of the Project Appraisal Document, and miscellaneous thematic reports from other sources. This desk review will be used to provide background information on PMI Impact Malaria Project and will also provide data for analysis for this evaluation. The Evaluation Team (ET) will compare PMI Impact Malaria’s achievements and targets reached to project goals and milestones using the following documents: • Final contract (with goals/objectives and technical content) • Project and Performance monitoring plans/PMP • Annual work plans • Annual and semiannual project reports • PMI Impact Malaria publications and any other written products/documents/technical reports • Any other relevant project documents • PMI Strategy 2015-2020 (or updated strategy if it is available) Secondary analysis of existing data This is a re-analysis of existing data, beyond a review of data reports. Please list the data source and recommended analyses. Data Source (existing dataset) Description of data Recommended analysis PMI Impact Malaria’s Data Hub including PMP data Data extraction Training data Data extraction Outreach training and supportive supervision (OTSS) reports Desk review OTSS checklists Desk review/data extraction Data collected from other monitoring tools Data extraction 102 Key Informant Interviews Please list categories of key informants, and purpose of inquiry. Interviews with stakeholders and partners of PMI Impact Malaria, both at country and global level. The Evaluation Team (ET) will develop a semi-structured interview guide that will be used to conduct the key informant interviews (KIIs). KIIs will be conducted through face-to-face contact or by telephone as necessary. Respondents will be identified by PMI and PMI Impact Malaria. A list of potential respondents will be developed prior to the start of the evaluation process. ⮚ Key informants for 4 countries (Ghana, Kenya, Cameroon, and Niger): • PMI Impact Malaria staff at headquarters and in country (PSI, MCDI, Jhpiego, UCSF) • PMI staff at headquarters and in country; including COR team and country backstops • USAID Health Office leadership and other mission health team staff as appropriate • NMCP staff at headquarters and regional/district level • Other PMI implementing partners or other key malaria stakeholders in country, as appropriate Purpose of inquiry for 4 countries: • Were results achieved according to country workplan? • Successes of program that should be replicated/continued; major contributors to these successes • Major challenges or barriers to project implementation/scale-up of malaria CM • Strengths and weaknesses of management of project • Capacity built in malaria diagnosis and CM and prevention of MIP at the regional, district, and health-facility levels • Capacity built in management of malaria at the community level (iCCM) where applicable • Successes/weaknesses in coordination, planning and implementation of SMC campaigns where applicable • Areas of focus in the future ⮚ In addition to the above, the evaluation will also key informants from a 5th country, Benin or Senegal, as one of the countries with a limited buy-in for operational research (OR) activities. ⮚ Key informants for global level: • Stakeholders at WHO, RBM, and other international organizations or partnerships 103 Purpose of inquiry for global level: • What are PMI Impact Malaria’s contributions to advocacy and technical advancement at the global level? How effective have they been? • Successes at global level that should be replicated/continued • Suggested areas of focus in the future Focus Group Discussions Please list categories of groups, and purpose of inquiry. Group Interviews Please list categories of groups, and purpose of inquiry. Some of the key informant interviews can be clustered, as long as there are no power differentials, and all respondents feel comfortable in voicing their opinions within the group. (See list and description above under KII.) Client/Participant Satisfaction or Exit Interviews Please list who is to be interviewed, and purpose of inquiry. Survey Please describe content of the survey and target responders, and purpose of inquiry. A brief structured survey that will take approximately 15 minutes to complete, using Survey Monkey, will be sent to all PMI Impact Malaria countries’ key informants inquiring about PMI Impact Malaria implementation, management, results, strengths, and shortcomings. Stakeholders from all countries engaged with PMI Impact Malaria will be invited to participate. The Evaluation Team (ET), supported by USAID, will attempt to reach a survey response rate of 80% or more. ET will develop a survey to gauge stakeholders view of the project including: ● If results were achieved according to country workplan ● Successes of program that should be replicated/continued; major contributors to these successes ● Major challenges or barriers to project implementation 104 ● Proposed future areas of focus ● Strengths and weaknesses of management of project ● Capacity built in country ● How well staffing and programming were tailored to meet country needs Please note that not all questions will be relevant for all countries and the survey will be tailored accordingly. Facility or Service Assessment/Survey Please list type of facility or service of interest, and purpose of inquiry. Observations Please list types of sites or activities to be observed, and purpose of inquiry. Cost Analysis Please list costing factors of interest, and type of costing assessment, if known. Data Abstraction Please list and describe files or documents that contain information of interest, and purpose of inquiry. Data abstraction will be conducted for four countries to analyze changes in knowledge, practice, and skills of health workers participating in PMI Impact Malaria training and supervision interventions. These include proficiency in diagnostic testing and adherence to test results, and provision of MIP services. Successes in achieving SMC coverage and adherence goals should also be assessed where relevant. Documents of interest: ● Country work plans and annual and semiannual (quarterly) reports ● PMI Impact Malaria documents capturing program activities: supervisor documents, outreach training and supportive supervision (OTSS) reports, OTSS checklists, data collected from other monitoring tools ● Training data ● PMP data (available on the IM Data Hub) Case Study 105 Please describe the case, and issue of interest to be explored. Verbal Autopsy Please list the type of mortality being investigated (i.e., maternal deaths), any cause of death and the target population. Rapid Appraisal Methods Please (ethnographic/participatory) list and describe methods, target participants, and purpose of inquiry. Other Please list and describe other methods recommended for this assignment and purpose of inquiry. If this is an Impact Evaluation, then: Is technical assistance needed to develop full protocol and/or IRB submission? c Yes g No List or describe “case” and “counterfactual”. Case Counterfactual Note on Human Subject Protection The Assignment Team must develop protocols to insure privacy and confidentiality prior to any data collection. Primary data collection must include a consent process that contains the purpose of the assignment, the risk and benefits to the respondents and community, the right to refuse to answer any question, and the right to refuse participation in the assignment at any time without consequences. Only adults can consent as part of this assignment. Minors cannot be respondents to any interview or survey and cannot participate in a focus group discussion without going through an IRB. The only time minors can be observed as part of this 106 assignment is as part of a large community-wide public event, when they are part of family and community in the public setting. During the process of this assignment, if data are abstracted from existing documents that include unique identifiers, data can only be abstracted without this identifying information. An Informed Consent statement included in all data collection interactions must contain: ● Introduction of facilitator/note-taker ● Purpose of the assignment ● Purpose of interview/discussion/survey ● Statement that all information provided is confidential and information provided will not be connected to the individual ● Right to refuse to answer questions or participate in interview/discussion/survey ● Request consent prior to initiating data collection (i.e., interview/discussion/survey) VII. ASSIGNMENT ANALYSIS PLAN Instructions: Describe how the quantitative and qualitative data will be analyzed. Include method or type of analyses, statistical tests, and what data it to be triangulated (if appropriate). For example, a thematic analysis of qualitative interview data, or a descriptive analysis of quantitative survey data. The box below has been filled out to provide you with an example that you should edit, as necessary. All analyses will be geared to answer the assignment questions. Additionally, the assignment will review both qualitative and quantitative data related to the project/program’s achievements against its objectives and/or targets. Quantitative data will be analyzed primarily using descriptive statistics. Data will be stratified by demographic characteristics, such as sex, age, and location, whenever feasible. Other statistical test of association (i.e., odds ratio) and correlations will be run as appropriate. Thematic review of qualitative data will be performed, connecting the data to the assignment questions, seeking relationships, context, interpretation, nuances and homogeneity and outliers to better explain what is happening and the perception of those involved. Qualitative data will be used to substantiate quantitative findings, provide more insights than quantitative data can provide, and answer questions where other data do not exist. Use of multiple methods that are quantitative and qualitative, as well as existing data (e.g., project/program performance indicator data, DHS, MICS, HMIS data, etc.) will allow the Assignment Team to triangulate findings to produce more robust results. 107 The assignment will describe analytic methods and statistical tests employed. 108 VIII. ACTIVITIES Instructions: List the expected key activities, such as Team Planning Meeting (TPM), briefings, verification workshop with IPs and stakeholders, etc. Activities and deliverables may overlap. Please give as much detail as possible. 1. Desk Review – Several documents are available for review for this assignment. These include [name of project] proposal, annual work plans, M&E plans, quarterly progress reports, and routine reports of project performance indicator data, as well as survey data reports (i.e., DHS and MICS). This desk review will provide background information for the Assignment Team and will also be used as data input and evidence for the assignment. 2. Assignment Launch/In-brief with USAID – A call/meeting among the USAID, GH EvaLS project staff and the Evaluation Team to initiate the assignment and review expectations. USAID will review the purpose, expectations, and agenda of the assignment. GH EvaLS will introduce the Team and review the initial schedule and other management issues. 3. Team Planning Meeting – A three to four-day team planning meeting (TPM) will be held at the initiation of the assignment and before the data collection begins. During the TPM, the Team will: ● Review and clarify any questions on the assignment SOW ● Clarify team composition from EvaLS and USAID, and members’ roles and responsibilities ● Establish a team atmosphere, share individual working styles, and agree on procedures for resolving differences of opinion ● Review and finalize the assignment questions ● Review and finalize the assignment timeline ● Develop a draft of the data collection methods, instruments, and guidelines ● Review and clarify any logistical and administrative procedures for the assignment ● Develop a preliminary data collection plan ● Draft the assignment workplan ● Develop a preliminary draft outline of the team’s report ● Assign drafting/writing responsibilities for the final report or final presentation. 4. Workplan and Methodology submitted to USAID and followed by a review meeting. Workplan will include: ● Assignment timeline ● Assignment questions ● Proposed methodology ● Data collection strategy, sampling frame and selection criteria ● Data analysis plan describing procedures that will be used to analyze 109 qualitative and quantitative data ● Data and resource requirements ● Data collection instruments 5. In-brief with the target Project/Program to review the assignment plans and timeline, and for the project to give an overview of the project to the Assignment Team. 6. USAID and Stakeholder Briefings – The Team Lead will brief the USAID POC weekly to discuss progress. As preliminary findings arise, the TL will share these during the routine briefing, and in an email. A final debrief between the Assignment Team and USAID will be held at the end of the assignment and before the preparation of the final report, to present preliminary findings to USAID. During this meeting a summary of the data will be presented, along with high level findings and draft recommendations. For the debrief, the Team will prepare a PowerPoint Presentation of the key findings, issues, and recommendations. The Team will incorporate comments received from USAID during the debrief in the assignment report. (Note: preliminary findings are not final and as more data sources are developed and analyzed these finding may change.) 7. IP and Stakeholders’ debrief/workshop will be held with the project staff and other stakeholders identified by USAID. This will occur following the final debrief with the Mission and will not include any information that may be procurement deemed sensitive or not suitable by USAID. [OPTIONAL] 8. Fieldwork: Site Visits and Data Collection – The Assignment Team will conduct site visits to for data collection. Selection of sites to be visited will be finalized during TPM in consultation with USAID. The Team will outline and schedule key meetings and site visits prior to departing to the field. During the time of COVID, especially in countries where borders are closed and access is restricted, highly qualified national/regional evaluators and experts will be contracted. In addition, alternative means of data collection will be used, such as virtual stakeholder meetings, key informant interviews, and focus groups, where possible. See also USAID Guide to Remote Monitoring in COVID-19. 9. Assignment Report – The Assignment Team under the leadership of the Team Lead will develop a report with findings and recommendations. Report writing and submission will include the following steps: ● Team Lead will submit draft final report to GH EvaLS for review and formatting ● GH EvaLS will submit the draft report to USAID ● USAID will review the draft report in a timely manner, and send their comments and edits back to GH EvaLS 110 ● USAID will manage implementing partner(s)’s (IP) review of the report and compile and send their comments and edits to GH EvaLS. (Note: USAID will decide what draft they want the IP to review.) ● GH EvaLS will share USAID’s comments and edits with the Team Lead, who will then do final edits, as needed, and resubmit to GH EvaLS ● GH EvaLS will review and reformat the final report, as needed, and resubmit to USAID for approval. ● Once the final report is approved, GH EvaLS will re-format it for 508 compliance and post it to the DEC. The evaluation/analytic report excludes any procurement-sensitive and other sensitive but unclassified (SBU) information. This information will be submitted in a memo to USAID separate from the report. 10.Submission of Datasets to the Development Data Library – Per USAID’s Open Data policy (ADS 579, USAID Development Data), GH EvaLS will submit all quantitative data to USAID and the Development Data Library (DDL), at www.usaid.gov/data, in a machine-readable format (CSV or XML). The datasets created as part of this evaluation/analytic will be accompanied by a data dictionary that includes a codebook and any other information needed for others to use these data. It is essential that the datasets are stripped of all identifying information, as the data will be public once posted on USAID’s DDL. Where feasible, qualitative data that do not contain identifying information should also be submitted to GH EvaLS. 11.Submission of Final Evaluation Report to the Development Experience Clearinghouse – Per USAID policy (ADS 201.3.5.18), GH EvaLS will submit the final evaluation/analytic report to the Development Experience Clearinghouse (DEC) within three months of final approval by USAID. IX. TASKS, DELIVERABLES AND TIMELINES Instructions: Select all deliverables and products required on this analytic assignment. For those not listed, add rows as needed or enter them under “Other” in the table below. Provide timelines and deliverable deadlines for each. Tasks/Deliverables Timelines & Deadlines (estimated) Assignment Launch/In-brief with USAID Week 1 of assignment launch Desk Review January 2021 Team Planning Meeting/In-depth discussion with USAID on workplan and methodology January/February 2021 Workplan and methodology review briefing January/February 2021 111 Tasks/Deliverables Timelines & Deadlines (estimated) Workplan submission (includes assignment questions, methods, timeline, data analysis plan, and data collection instruments) January/February 2021 In-brief with target PMI Impact Malaria February 2021 Fieldwork: site visits and data collection February/March 2021 Routine USAID briefings Weekly Debrief with USAID with PowerPoint presentation on progress of the assignment and preliminary findings April 2021 IP & stakeholders findings review workshop with PowerPoint presentation April 2021 Draft report May 2021 Final report Mid-June 2021 Raw data (cleaned datasets in CSV or XML with code sheet) submitted End of June 2021 Report posted to the DEC Requires USAID approval. Potentially end of June 2021 for posting Estimated USAID review time Average number of business days USAID will need to review the Report? business days X. TEAM COMPOSITION, SKILLS, LEVEL OF EFFORT (LOE) AND LOGISTICAL NEEDS A. TEAM COMPOSITION AND SKILLS: Instructions: Please list technical areas of expertise required for this assignment: ● List desired qualifications for the team as a whole ● List the key staff needed for this analytic assignment and their roles. ● Sample position descriptions are posted on USAID/GH EvaLS webpage ● Edit as needed GH EvaLS provided position descriptions Please also consider: ● Key staff should have methodological and/or technical expertise, regional or country experience, language skills, team lead experience and management skills, etc. ● Team leads for evaluations must be an external expert with appropriate skills and experience. ● Additional team members can include research assistants, enumerators, translators, logisticians, etc. 112 ● Teams should include a collective mix of appropriate methodological and subject matter expertise. ● Evaluations require an Evaluation/Analytics Specialist, who should have evaluation/analytic methodological expertise needed for this assignment. Similarly, other analytic activities should have a specialist with methodological expertise. ● Note that all team members will be required to provide a signed Non￾Disclosure and Conflict of Interest statements attesting that they will keep all information confidential and have no conflict of interest (COI) or describing the conflict of interest if applicable for further consideration. Team Lead (Key Staff 1) Roles & Responsibilities: The Team Lead (TL) will be responsible for: • Providing team leadership • Managing the team’s activities • Ensuring that all deliverables are met in a timely manner • Serving as a liaison between the USAID and the Evaluation Team (ET), and • Leading briefings and presentations Qualifications: • Minimum of 10 years of experience in public health, which included experience in implementation of health activities in developing countries • Experience in evaluation design, methods, management, and implementation • Public health expertise in child health, malaria, and delivery of health-facility based care in Africa, especially in sub-Saharan Africa • Understanding and knowledge of USAID/GH/HIDN and USAID regional missions and programs • Demonstrated experience leading health sector project/program activities, utilizing both quantitative and qualitative methods • Excellent skills in planning, facilitation, and consensus building • Excellent interpersonal skills, including experience successfully interacting with host government officials, civil society partners, and other stakeholders • Excellent organizational skills and ability to keep to a timeline • Good writing skills, with extensive report writing experience • French language skills preferred Senior Malaria Specialist/Subject Matter Expert (Key Staff 2): The Senior Malaria Specialist will provide expertise on malaria program design, implementation, and evaluation. 113 Qualifications: • Extensive experience (at least 10 years) in malaria program design, implementation and evaluations/assessments • Knowledge and experience in the design and implementation of malaria evaluations/assessments, including use of quantitative and qualitative methods • At least 5 years managing Monitoring & Evaluations • Experience implementing key informant interviews, focus groups, observations and other evaluation/assessment methods that assure reliability and validity of the data • Experience in data management and able to analyze quantitative and qualitative data • Experience using analytic software • Demonstrated experience using qualitative evaluation methodologies • Excellent skills in planning, facilitation, and consensus building • Excellent interpersonal skills, including experience successfully interacting with host government officials and other stakeholders • Excellent organizational skills and ability to keep to a timeline • Good writing skills, with extensive report writing experience • Familiarity with USAID policies and practices • French language skills preferred Senior Analyst (Key Staff 3): The Analyst will support the Evaluation Team in all the data analysis aspects of the evaluation. Roles & Responsibilities: The Analyst will be responsible for: • Performing data analysis • Assuring that all quantitative and qualitative data analyses are done to meet the needs for this evaluation • Providing quality assurance on analytic issues, including methods, development of data collection instruments, protocols for data collection, data management and data analysis Qualifications: • Experience in the implementation of project/program evaluations, particularly assessing M&E systems and data quality • Experience in data management • Strong knowledge, skills, and experience in qualitative data analysis • Experience conducting secondary analysis of existing quantitative datasets • Strong data interpretation and presentation skills 114 • Good writing skills, including experience writing evaluation reports • Able to analyze quantitative and qualitative data • Experience using analytic software • Demonstrated experience using qualitative evaluation methodologies, and triangulating with quantitative data • Experience conducting secondary analysis of existing quantitative datasets 1. USAID Participation Will USAID participate as an active team member or designate other key stakeholders to as an active team member? This will require full time commitment during the evaluation assignment. c Full member of the Team (including planning, data collection, analysis and report development) – If yes, specify who: c Some Involvement anticipated – If yes, specify who: ___________________ g No B. STAFFING LEVEL OF EFFORT (LOE) MATRIX AND ANTICIPATED TRAVEL 1. LOE Chart Instructions: The LOE Matrix below will help you estimate the LOE needed to implement this assignment. If you are unsure, GH EvaLS can assist you to complete this table. Please note: a) For each column, replace the label "Position Title" with the actual position title of staff needed for this assignment. b) Immediately below each staff title enter the anticipated number of people for each titled position. c) Enter row labels for each assignment, task and deliverable needed to implement this assignment. d) Then enter the LOE (estimated number of days) for each activity/task/deliverable corresponding to each titled position. e) At the bottom of the table total the LOE days for each consultant title in the ‘Sub-Total’ cell, then multiply the subtotals in each column by the number of individuals that will hold this title. Staffing Level of Effort (LOE) Matrix The Level of Effort (in days) for each Evaluation Team member is shown below: 115 Activity/Deliverable Evaluation/Analytic Team Team Lead (Key Staff 1) Senior Malaria Specialist (Key Staff 2) Senior Analyst (Key Staff 3) Evaluation Logistics Coordinator/ Program Assistant Number of persons 1 1 1 1 1 Assignment Launch/In-brief with USAID 1 1 1 0 2 Desk review 7 5 4.5 0 3 Team Planning Meeting (TPM) 4 4 4 1 4 Workplan and methodology briefing with USAID 0.5 0.5 0.5 0 5 Workplan submission (includes assignment questions, methods, timeline, data analysis plan, and data collection instruments) 2 2 0 1 6 Data collection 20 17 10 9 7 Data analysis 5 5 15 8 Debrief with USAID/Preliminary findings (PPT slides) 1 1 1 1 9 Draft report 10 6 6 1 10 Submission of draft Evaluation Report to USAID 0 0 0 0 11 Revision of Evaluation Report per USAID comments 4.5 3.5 3 0 12 Finalization and submission of final Evaluation Report to USAID 0 0 0 0 13 Approval of the final Evaluation Report by USAID 0 0 0 0 14 Uploading of the final Evaluation Report to the DEC and submission of clean datasets 0 0 0 0 Total LOE per person 55 45 45 12 Total LOE 55 45 45 12 A 6-day workweek permitted c Yes g No 116 6-day workweek approved for travel to/from work locations c Yes g No 2. Anticipated Travel Please list international and local travel anticipated by what team members. N/A C. LOGISTICS 1. Work week Billing up to seven (7) days in any consecutive seven (7)-day period is approved when traveling to or from the consultant’s home of record  Yes  No 117 2. Visa Requirements List any specific Visa requirements or considerations for entry to countries that will be visited by consultant(s): List recommended/required type of visa for entry into counties where consultant(s) will work: Name of Country Type of Visa  Tourist  Business  No preference  Tourist  Business  No preference  Tourist  Business  No preference  Tourist  Business  No preference 3. Clearances & Other Requirements Check all that the consultant will need to perform this assignment, including USAID Facility Access, GH EvaLS workspace and travel (other than to and from post). c USAID Facility Access (FA) Specify who will require Facility Access: Electronic County Clearance (ECC) (International travelers only) High Threat Security Overseas Seminar (HTSOS) (required in most countries with ECC) Foreign Affairs Counter Threat (FACT) (for consultants working on country more than 45 consecutive days) GH EvaLS workspace Specify who will require workspace at GH EvaLS: Travel, other than posting (specify): Other (specify): Specify any country-specific security concerns and/or requirements: 118 Note on Workspace and Clearances Most Teams arrange their own workspace, often in conference rooms at their hotels. However, if a security clearance or facility access is preferred, GH EvaLS can submit an application for it on the consultant’s behalf. GH EvaLS can obtain Facility Access (FA) and transfer existing Secret Security Clearance for our consultants, but please note these requests, processed through AMS at USAID/GH (Washington, DC), can take 4-6 months to be granted. If you are in a Mission and the RSO is able to grant a temporary FA locally, this can expedite the process. FAs for non-US citizens or Green Card holders must be obtained through the RSO. If FA or Security Clearance is granted through Washington, DC, the consultant must pick up his/her badge in person at the Office of Security in Washington, DC, regardless of where the consultant resides or will work. If Electronic Country Clearance (eCC) is required prior to the consultant’s travel, the consultant is also required to complete the High Threat Security Overseas Seminar (HTSOS). HTSOS is an interactive e-Learning (online) course designed to provide participants with threat and situational awareness training against criminal and terrorist attacks while working in high threat regions. There is a small fee required to register for this course. [Note: The course is not required for employees who have taken FACT training within the past five years or have taken HTSOS within the same calendar year.] If eCC is required, and the consultant is expected to work in country more than 45 consecutive days, the consultant may be required complete the one-week Foreign Affairs Counter Threat (FACT) course offered by FSI in West Virginia. This course provides participants with the knowledge and skills to better prepare themselves for living and working in critical and high threat overseas environments. Registration for this course is complicated by high demand (consultants must register approximately 3-4 months in advance). Additionally, there will be the cost for additional lodging and M&IE to take this course. 119 X. GH EvaLS ROLES AND RESPONSIBILITIES GH EvaLS will coordinate and manage the team and provide quality assurance oversight, including: ● Review SOW and recommend revisions as needed ● Provide technical assistance on methodology, as needed ● Develop budget for assignment ● Recruit and hire the team, with USAID POC approval ● Arrange international travel and lodging for international consultants ● Request for country clearance and/or facility access (if needed) ● Review and assist with development of methods, workplan, evaluation/analytical instruments, reports, and other deliverables as part of the quality assurance oversight, as appropriate ● Report production - If the report is public, then coordination of draft and finalization steps, editing/formatting, 508ing required in addition to and submission to the DEC and posting on GH EvaLS website. If the report is internal, then copy editing/formatting for internal distribution. XI. USAID ROLES AND RESPONSIBILITIES Below is the standard list of USAID’s roles and responsibilities. Add other roles and responsibilities as appropriate. USAID will provide overall technical leadership and direction for the analytic team throughout the assignment and will provide assistance with the following tasks: Before Field Work ● SOW. o Develop SOW o Peer Review SOW o Respond to queries about the SOW and/or the assignment at large. ● Consultant Conflict of Interest (COI). To avoid conflicts of interest or the appearance of a COI, review previous employers listed on the CV’s for proposed consultants and provide additional information regarding potential COI with the project contractors evaluated/assessed and information regarding their affiliates. ● Documents. Identify and prioritize background materials for the consultants and provide them to GH EvaLS, preferably in electronic form, at least one week prior to the inception of the assignment. ● Local Consultants. Assist with identification of potential local consultants, including contact information. ● Site Visit Preparations. Provide a list of site visit locations, key contacts, and suggested length of visit for use in planning in-country travel and accurate estimation of country travel line items costs. 120 ● Lodgings and Travel. Provide guidance on recommended secure hotels and methods of in-country travel (i.e., car rental companies and other means of transportation). During Field Work ● Mission Point of Contact. Throughout the in-country work, ensure constant availability of the Point of Contact person and provide technical leadership and direction for the team’s work. ● Meeting Space. Provide guidance on the team’s selection of a meeting space for interviews and/or focus group discussions (i.e., USAID space if available, or other known office/hotel meeting space). ● Meeting Arrangements. Assist the team in arranging and coordinating meetings with stakeholders. ● Facilitate Contact with Implementing Partners. Introduce the analytic team to implementing partners and other stakeholders, and where applicable and appropriate prepare and send out an introduction letter for team’s arrival and/or anticipated meetings. After Field Work ● Timely Reviews. Provide timely review of draft/final reports and approval of deliverables. XII. FINAL REPORT Provide any desired guidance or specifications for Final Report. (See How-To Note: Preparing Evaluation Reports) FINAL REPORT FORMAT 1. Executive Summary 2. Evaluation Purpose 3. Background on the Context and the Strategies/Projects/Activities being Evaluated 4. Evaluation Questions 5. Methodology 6. Limitations to the Evaluation 7. Findings, Conclusions, and Recommendations 8. Annexes See the Evaluation Toolkit for the How-To Note on Preparing Evaluation Reports and ADS 201mah, USAID Evaluation Report Requirements. An optional Evaluation Report Template is also available in the Evaluation Toolkit. The evaluation abstract of no more than 250 words should describe what was 121 evaluated, evaluation questions, methods, and key findings or conclusions. The executive summary should be 2–5 pages and summarize the purpose, background of the project being evaluated, main evaluation questions, methods, findings, and conclusions (plus recommendations and lessons learned, if applicable). The evaluation methodology shall be explained in the report in detail. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methods (e.g., in sampling; data availability; measurement; analysis; any potential bias such as sampling/selection, measurement, interviewer, response, etc.) and their implications for conclusions drawn from the evaluation findings. The evaluation should include a summary of lessons learned from PMI Impact Malaria’s activities at all levels that could inform future programming in malaria CM, prevention of MIP or implementation of seasonal malaria chemoprevention. A focus on key bottlenecks or gaps identified that should be addressed in future activities is requested. Annexes to the report must include: ● Evaluation SOW (updated, not the original, if there were any modifications); ● Evaluation methods; ● All data collection and analysis tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides; ● Tables of survey results ● All sources of information or data, identified and listed; ● Statements of difference regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team, if applicable; ● Signed disclosure of conflict of interest forms for all evaluation team members, either attesting to a lack of or describing existing conflicts of interest; and ● Summary information about evaluation team members, including qualifications, experience, and role on the team. CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT Per ADS 201maa, Criteria to Ensure the Quality of the Evaluation Report, draft and final evaluation reports will be evaluated against the following criteria to ensure quality. ● Evaluation reports should represent a thoughtful, well-researched, and well￾organized effort to objectively evaluate the strategy, project, or activity; ● Evaluation reports should be readily understood and should identify key points clearly, distinctly, and succinctly; ● The Executive Summary should present a concise and accurate statement of the most critical elements of the report; 122 ● Evaluation reports should adequately address all evaluation questions included in the SOW, or the evaluation questions subsequently revised and documented in consultation and agreement with USAID; ● Evaluation methodology should be explained in detail and sources of information or data properly identified; ● Limitations to the evaluation should be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.); ● Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or simply the compilation of people’s opinions; ● Conclusions should be specific, concise, and include an assessment of quality and strength of evidence to support them supported by strong quantitative and/or qualitative evidence; ● If evaluation findings assess person-level outcomes or impact, they should also be separately assessed for both males and females; and ● If recommendations are included, they should be supported by a specific set of findings and should be action-oriented, practical, and specific. See ADS 201, USAID Evaluation Report Requirements and the Evaluation Report Checklist and Review Template from the Evaluation Toolkit for additional guidance. OTHER REQUIREMENTS All modifications to the required elements of the SOW of the contract/agreement, whether in evaluation questions, design and methodology, deliverables and reporting, Evaluation Team composition, schedule, and/or other requirements will be agreed upon in writing by the USAID/PMI COR team will also be apprised of any changes made to the SOW. Any revisions made will be noted in the SOW annexed to the final Evaluation Report XIII. USAID CONTACTS Primary Contact Alternate Contact 1 Alternate Contact 2 Name: Lia Florey Title: Malaria Technical Advisor for USAID/PMI USAID Office/Mission PMI Email: lflorey@usaid.gov Telephone: 202-916-2117 Cell Phone: 571-242-1290 123 List other contacts who will be supporting the Requesting/Funder Team with technical support, such as reviewing SOW and final report (such as USAID/W GH EvaLS management team staff): Technical Support Contact 1 Technical Support Contact 2 Name: Title: USAID Office/Mission Email: Telephone: Cell Phone: 124 XIV. OTHER REFERENCE MATERIALS Documents and materials needed and/or useful for consultant assignment, that are not listed above. XV. ADJUSTMENTS MADE IN CARRYING OUT THIS SOW AFTER APPROVAL OF THE SOW To be completed after assignment implementation by GH EvaLS. Draft report delivery shifted to week of June 21, 2021 Final report delivery shifted to July, 2021 125 ANNEX 2: DATA COLLECTION INSTRUMENTS HQ and GLOBAL LEVEL INTERVIEW GUIDE Respondents: USAID PMI, IM Partners HQ level and WHO, RBM global level Introduction: ● My name is ___. I am a team member for the PMI Impact Malaria mid-term performance evaluation, which covers the project’s first 3 years (2018/19-2020/21). ● Thank you for agreeing to be interviewed. ● Our interview will likely take about 60 minutes. ● You may choose not to answer any question and you are free to stop your participation at any time. ● The responses you provide will be kept confidential and not ascribed to you. The results from all interviews will be pooled for analysis and we will ensure that responses cannot be traced back to any individual. All respondents’ names, titles and affiliations will be listed as an annex in the evaluation report. ● We have the option to record the interview. What do you prefer? ● Do I have your consent to begin the interview? ● May I have your name and title, please? (Interview begins here.) Date of Interview: Interviewer Name: Respondent Name: Respondent Title: Respondent Organization 126 Question Stakeholders who should be asked each question EQ: USAID/ PMI IMPACT MALARIA Partners External Respondents (WHO, RBM) 1. Please tell me about your role in the Impact Malaria Project. (title, involved for how long with project) √ √ 2. In your view, what have been the biggest successes over the Impact Malaria project’s first three years at country level? Probe: according to respondent’s role √ √ √ 1.a; 3.a 3. What have been the biggest challenges the project has faced in its first three years? Probe: according to respondent’s role √ √ 1.a 4. How well has IM been able to set up and operate in multiple countries quickly as the project has grown? √ √ 2.d 5. Across the first three project years, were core work plan deliverables largely on time? Probe: If no, what have been some of the reasons for delays? Probe: What if anything could be done to improve timeliness of deliverables? √ √ 2.a; 2.b 6. In your view, has the IM management team been clear and timely in communications with USAID/PMI about deliverables and other project management issues? Please explain your answer. √ √ 2.a; 2.b; 2.d 127 Question Stakeholders who should be asked each question EQ: USAID/ PMI IMPACT MALARIA Partners External Respondents (WHO, RBM) 7. As an Impact Malaria sub￾partner, do you have opportunities to communicate with USAID/PMI directly? Please describe. MCDI, Jhpiego, UCSF 2.b 8. I am interested in your perceptions of how well the partnership between PSI, MCDI, Jhpiego and UCSF has functioned. Probe: For example, is your organization fulfilling the role within the IMPACT MALARIA partnership that was initially agreed on? Probe: Do all four partners have current information and understand how their work aligns with and complements other partners’ activities? Probe: How if at all could the partnership be strengthened? √ √ 2.b; 2.e 9. For PSI: How well were sub partners able meet staffing needs of the award? (Ability to fill essential positions rapidly, type of staff hired, location of staff, etc.) For MCDI, Jhpiego, UCSF: How well was PSI able to meet staffing needs of the award? (Ability to fill essential positions rapidly, type of staff hired, location of staff, etc.) √ √ 2.e 10. How well has Impact Malaria performed in coordinating activities with other PMI supported Implementing Partners and projects? 128 Question Stakeholders who should be asked each question EQ: USAID/ PMI IMPACT MALARIA Partners External Respondents (WHO, RBM) 11. What are the expectations for how Data Hub data will be useful to: PMI for future planning? to Country malaria programs that may wish to collect similar data beyond the IM project? Probe: What is PMI’s expectation for standardized data across countries given that countries sometimes have different indicators? Probe: Are there types of data not currently being collected or reported through the Data Hub that PMI would like to see? 12. How well has Impact Malaria performed in coordinating activities with external global partners where needed (e.g., WHO, Roll Back Malaria, GF, other investors)? Please explain your answer. √ √ √ 2.b 13. What are your overall impressions of Impact Malaria’s successes and contributions to global malaria efforts? Probe: Do you see any problems or ways in which the project could be more effective? √ 1.b; 3.a 129 Question Stakeholders who should be asked each question EQ: USAID/ PMI IMPACT MALARIA Partners External Respondents (WHO, RBM) 14. In your view, what has IMPACT MALARIA contributed to the global body of literature for malaria prevention, diagnosis, and CM? Probe: Guidelines, manuals, M&E standards, peer reviewed literature, other publications. √ √ √ 3.a 15. Has Impact Malaria been sufficiently active in sharing lessons learned or best practices with the malaria community at large? √ √ √ 3.a 16. Please describe Impact Malaria’s participation in global technical working groups, task forces, and stakeholder meetings. Probe: How has IM helped to influence global standards and/or best practices? √ √ √ 3.a 17. Is there anything else about Impact Malaria’s progress and results at the global level you would like the evaluation team to know? √ √ √ THANK YOU. 130 PMI Impact Malaria Key Informant Interview Guide: IN-DEPTH COUNTRY REVIEWS (PMI backstops, IM field staff, NMCP stakeholders, in country partners) Introduction: ● My name is ___. I am a team member for the PMI Impact Malaria mid-term evaluation. ● USAID is conducting a midterm performance evaluation of the Impact Malaria project’s first 3 years (2018/19-2020/21). ● The purpose of this interview is to identify project strengths and weaknesses to provide recommendations for the project’s second half. ● Our interview will likely take about 60 minutes. ● If we are unable to finish today, I’ll be happy to call you again to complete the interview. ● Your participation is completely voluntary, and conducting the interview depends on your acceptance. You may choose not to answer any question and you are free to stop your participation at any time. ● The responses you provide will be kept confidential and not ascribed to you. The results from all interviews will be pooled for analysis and we will ensure that responses cannot be traced back to any individual. All respondents’ names, titles and affiliations will be listed as an annex in the evaluation report. ● We have the option to record the interview. Are you comfortable with being recorded? ● Do you have any questions? ___ Yes ___ No (If yes, note the questions.) ● Do I have your consent to begin the interview? ___ Yes ___ No ● May I have your name and title, please? (Interview begins here.) Respondent’s Name: Respondent’s Title: Date (dd/mm) Location: Country Region/district if applicable Start time: End time: Introductory Question I am going to ask you some questions about the work that has been implemented under the Impact Malaria project’s first 3 years (2018/19-2020/21). 1. To get us started, please tell me about your role in the PMI Impact Malaria project in your country (name the country/region/district). Probe: ● When did IM begin to provide support in your country? ● How many years have you worked with the project? (Previous positions?) 131 Evaluation Question 1. I have some questions about the extent to which Impact Malaria is meeting technical objectives based on the country work plans and Performance Monitoring Plans. 2. To the best of your knowledge, how has Impact Malaria contributed to improving the quality of and access to MIP interventions? Probe: ● How successfully has the implementation of MIP met coverage and quality objectives? ● To what extent has the integration of MIP into a malaria service delivery project benefitted or hindered progress on MIP coverage and uptake? ● To what extent has Impact Malaria facilitated collaboration between malaria and maternal health programs to ensure consistency in programming? (integrated planning, management, service delivery)? ● To what extent has the ANC attendance improved? ● To what extent has the support for national MIP working groups addressed challenges and bottlenecks? ● To what extent has OTSS MIP module improved provider knowledge and improved service delivery? ● What might help the country to improve MIP and ANC activities and results? 3. Globally, Impact Malaria aims for an average increase of 15% in the percentage of pregnant women receiving two or more doses of IPTp against malaria during last pregnancy. Is the country making progress toward this goal? Probe: ● Key challenges, examples of where progress has been made? ● What might help the country to reach this goal? ● Are there missed opportunities to better align or integrate malaria and ANC services? 4. Has Impact Malaria helped to improve malaria CM? Probe: ● (Diagnosing/treating MIP, diagnosis/treating severe malaria, laboratory diagnostic capacity, support for iCCM, etc.) ● What has contributed to the successes you describe? ● What have been the main challenges? ● What could help the country improve CM activities and outcomes? 5. Globally, Impact Malaria aims for an average of at least 80% of patients with suspected malaria receiving a diagnostic test. Is the country progressing toward this goal? Probe: ● What has contributed to any progress you describe? Give examples ● What have been main challenges or barriers? ● What could help the country improve malaria CM activities and outcomes? 6. Globally, Impact Malaria aims for an average of 80% of confirmed malaria cases receiving effective malaria treatment according to standard national protocols. Is the country making progress toward this goal? 132 Probe: ● What has contributed to any successes you describe? Give examples. ● What have been the main challenges or barriers? ● What could help the country to improve malaria treatment? 7. How, if at all, are MIP and CM interventions integrated? Probe: In terms of: ● Planning? ● Management? ● Service delivery? 8. Impact Malaria’s overall aim for each round of Seasonal Malaria Chemoprevention is that 80% of targeted children receive a dose of SMC. Has the country received support from Impact Malaria for SMC? If yes, probe: ● How many campaigns of SMC have been conducted? ● Have you met your SMC coverage and adherence objectives? If not, why not? ● How could the country increase the percentage of targeted children who receive a dose of SMC? ● How well has Impact Malaria handled staffing, coordinating, and managing logistics for SMC campaigns? 9. Have Seasonal Malaria Chemoprevention activities interfered with or hindered other malaria activities? If yes, please give examples. 10. For Senegal: Are you aware of Impact Malaria supported Operational Research activities in the country? Probe: ● If yes, please describe. ● Are there any lessons learned from this experience that could help other countries planning to conduct malaria-related Operational Research? (personnel, coordination, management)? 11. Are there any other key Impact Malaria activities underway in the country? Probe: ● Integrated Community CM ● Therapeutic Efficacy Studies ● Malaria slides bank ● Microscopy proficiency Probe for each: ● Are these activities progressing according to plan? ● Have there been any challenges in planning or implementing them? 133 12. Does the country take part in Impact Malaria’s Outreach Training and Supportive Supervision (OTSS+)? Probe: If so, what difference has it made in: ● Clinical quality improvement and proficiency? ● MIP clinical quality improvement? ● Diagnostic quality (RDT and/or microscopy)? ● Accurate recording and reporting of malaria data ● Other? 13. How is OTSS data collected, reported, and used in the country? Probe: ● What OTSS checklists are being used? ● Are checklists administered electronically, on paper, or both? ● How is data collected through the checklist being used? Probe: Are adjustments made to trainings, supervision, and prioritization of HFs? ● Is data submitted electronically? How is it working? 14. Are you familiar with the Impact Malaria Performance Management Plan and project indicators for your country? Probe: If yes, ● Are they useful and practical for reporting progress? ● Are there other activities or outcomes that could usefully be measured to get a more complete picture of Impact Malaria progress and results? If so, please explain. 15. Do you have any direct participation in the data and information that goes into the Impact Malaria Data Hub? If yes, please describe. Probe: ● Has the data hub improved capacity to access and use malaria data for reporting and decision making? How? ● How could it be made more useful? ● To your knowledge, what data is being collected for Case Management? For MIP? 16. What do you view as the main improvements resulting from Impact Malaria’s health systems strengthening and/or capacity building activities? Please comment on the level/s you work at or are most familiar with (central, regional, district) Probe: ● What type of HSS and CB support has the project offered? ● How are improvements measured? Probe for: ● More effective use of OTSS visits and use of OTSS checklists? ● Improvement in Health Facility and community health worker skills and/or behaviors? ● Strengthened quality assurance for MIP/IPTp, CM, and SMC activities? 134 ● Improved adherence to guidelines? ● Improved laboratory diagnostics? ● Improvements in supportive supervision/correct use of checklists and tools? ● Improvements in analysis and use of data for decision making? ● Increased private sector engagement for malaria CM? ● Improved treatment outcomes? ● support for national MIP working groups addressing challenges and bottlenecks? Probe: ● How else might Impact Malaria help to strengthen health systems or build capacity in the country for malaria prevention and CM? Evaluation Question 2. This brings us to a few questions about Impact Malaria’s management, technical support, and coordination with partners in the country. 17. Do you have any interaction with the Impact Malaria project’s HQ management and technical team? If so, please describe. Probe: ● Have you received technical support from the IMPACT MALARIA headquarters technical team? What type/s? ● Has technical support been efficient, helpful? Please describe. 18. Which Impact Malaria partners work in your country? (PSI, MCDI, Jhpiego, UCSF) Probe: ● If more than one, how are their activities coordinated? ● Do you feel that this coordination has been effective? 19. How well has Impact Malaria coordinated its activities with government and any PMI￾supported or non-PMI supported groups working on malaria in the country? Probe: ● Examples? Major successes? Gaps or shortcomings? ● How might coordination across malaria partners be strengthened, if at all? ● Does Impact Malaria collaborate with private sector malaria partners in your country? 20. Are the in-country Impact Malaria staff a good match for the country’s programmatic needs? Why or why not? Probe for: ● Numbers of staff sufficient? ● Skill sets appropriate? ● Roles appropriately defined (in terms of decision making, coordination)? ● Staffing adjustments to keep pace with any expansion (or contraction) in country’s programming within the project? 135 21. How efficiently has Impact Malaria been able to hire staff, set up offices, launch activities, and continue activities on the agreed upon timelines? Probe: ● Has Covid 19 affected timeliness of activities? If so, how? ● Have there been other reasons for any project activity delays? Please explain. ● How might project delays best be addressed? 22. Has IM supported country staff participation in global meetings like the RBM MIP WG? If yes, please describe. Are there any additional insights you would like to share? If I have additional questions as the evaluation goes forward, may I contact you again? THANK YOU FOR YOUR TIME AND THE INFORMATION YOU’VE SHARED. 136 PMI Impact Malaria Key Informant Interview Guide: SENEGAL IN-DEPTH REVIEW (PMI backstops, NMCP stakeholders, in country partners) Introduction: ● My name is ___. I am a team member for the PMI Impact Malaria mid-term evaluation. ● USAID is conducting a midterm performance evaluation of the Impact Malaria project’s Senegal cluster randomized controlled trial. ● Our interview will likely take about 45 minutes. ● If we are unable to finish today, I’ll be happy to call you again to complete the interview. ● Your participation is completely voluntary, and conducting the interview depends on your acceptance. You may choose not to answer any question and you are free to stop your participation at any time. ● The responses you provide will be kept confidential and not ascribed to you. The results from all interviews will be pooled for analysis and we will ensure that responses cannot be traced back to any individual. All respondents’ names, titles and affiliations will be listed as an annex in the evaluation report. ● We have the option to record the interview. Are you comfortable with being recorded? ● Do you have any questions? ___ Yes ___ No (If yes, note the questions.) ● Do I have your consent to begin the interview? ___ Yes ___ No ● May I have your name and title, please? (Interview begins here.) Respondent’s Name: Respondent’s Title: Date (dd/mm) Location: Country Region/district if applicable Start time: End time: General knowledge about the study/operations research To advance malaria service delivery in Senegal, Impact Malaria is working to improve drug-based prevention by conducting an operations research on the effect that time-limited mass drug administration (MDA) of two antimalarials, dihydroartemisinin-piperaquine and single low-dose primaquine, has on malaria transmission rates in a moderate-low transmission setting. This study will inform the Senegal's NMCP on the potential impact of time-limited MDA to accelerate transmission reduction and thus transition from control to elimination activities. 137 1. Are you familiar with operations research to test the effect of three rounds of MDA with DHA-PQP and low-dose PQ on confirmed malaria case incidence, compared to standard of care Seasonal Malaria Chemoprevention (SMC)? Probe: ● Have you been involved in any way in this study, either at preparation and/or implementation stages? ● If so, briefly describe your responsibilities and involvement in the study ● Please explain what is your understanding of the study, and what it is intended to accomplish? 2. Do you believe that the objective of this research, which is to provide Mass Drug Administration, responds to the current needs of the malaria control program? Probe: ● Please explain in either case, Yes or No 3. From your perspective, what progress has been made toward achieving the objectives of the operational research, which is to reduce malaria incidence and to reduce mortality? Probe: ● What evidence has led you to these conclusions? ● How does it compare with the standard of care Seasonal Malaria Chemoprevention (SMC)? ● Could you identify facilitators and barriers to the achievement of the study objectives? Evaluation Questions This brings us to a few questions about Impact Malaria’s management of the operations research, technical support, and coordination with partners in the country. 4. What has been the experience of the study to date? Please describe any challenges or successes encountered thus far in the study development and implementation? Probe for: ● Research planning ● IRB clearance ● Staffing, ● Implementation 5. Do you have any interaction with the IM Operations Research project’s management and technical team based at the University of California, San Francisco (UCFS)? If so, please describe. Probe: ● Have you received technical support from the UCFS technical team? What type /s? ● Has technical support been efficient, helpful? Please describe. 138 6. Are the in-country Impact Malaria/Operations research staff a good match for the country’s programmatic needs? Why or why not? Note to the interviewer. This is a multidisciplinary team composed of NCMP, University of Thiès and CDC. Make that distinction when probing: ● Numbers of staff sufficient? ● Skill sets appropriate. ● Roles appropriately defined (in terms of decision making, coordination)? ● Staffing adjustments to keep pace with any expansion (or contraction) in country’s programming within the project? 7. How efficiently has the OR project been able to hire staff, set up offices, launch activities, and continue activities on the agreed upon timelines? Probe: ● Has Covid 19 affected timeliness of activities? If so, how? ● Have there been other reasons for any project activity delays? Please explain. ● How might project delays best be addressed? ● Please also address the process of identifying and contracting with the local partner ● Please also speak to the interactions and communications between partners (PMI Senegal, IM HQ, NMCP, local implementing partner) 8. To the best of your knowledge and experience working at this level, does the operational research respond to the needs identified by the government and the academic community in Senegal? Probe: ● Please answer in both cases, Yes or No 9. Which Impact Malaria partners work in your country, besides UCSF? (PSI, MCDI, Jhpiego) Probe: ● If more than one, how are their activities coordinated? ● Do you feel that this coordination has been effective? 10. Has the research study affected or influenced other IM activities? If so, how? 11. What useful lessons have been learned from this research experience that might be of benefit to other countries involved with IM? 12. How well has Impact Malaria coordinated its activities with the government and academic institutions, specifically on the operations research project? Probe: ● Examples? Major successes? Gaps or shortcomings? ● How might coordination across malaria partners be strengthened, if at all? 139 Are there any additional insights you would like to share? If I have additional questions as the evaluation goes forward, may I contact you again? THANK YOU FOR YOUR TIME AND THE INFORMATION YOU’VE SHARED. 140 KII Addendum A: Additional Questions about Training and Supportive Supervision I would like to ask you a few questions about your training plan: 1. What information or data do you use to determine the training topics and audience? Probe: ● How do you determine who and how many people to train in your region or district? ● In approximately one-year, what percentage of all staff working in malaria receive training supported by the project? ● How do you select the trainees? ● How do you plan cascade training? ● Is this information recorded in the OTSS+ electronic platform? If not, where? I would like to ask you some questions about your activities in supportive supervision: 2. Explain how Supportive Supervision is linked to the training plan? Probe: ● How do you decide whom and how many to supervise? ● Is the content of the OTSS+ electronic platform already fixed? Can you make changes to respond to the training plan? ● Do you register this information somewhere else besides the OTSS+ electronic platform? If so, please explain 3. In your experience, does the complementarity between training and Supportive Supervision works well? How could it be improved? Probe: ● Does the OTSS+ electronic platform give you the information you need to improve training? If not, what are you missing, or would you like to know to make better decisions? 4. In general, or average, how many supportive supervision visits are conducted per month in your region or county? Probe: ● How many people are dedicated to supportive supervision activities in your region or county? ● What was it like before COVID-19 and during the pandemic? I would like to think back to a typical supportive supervision visit that is not limited by COVID￾19, or other external factors. 5. Please briefly describe the steps of a supportive supervision visit. Probe: ● How long does it take to conduct a visit? 141 ● How long does it take to fill in the data on the tablet? ● What do you do with the data, do you review it, how often do you send it to your head office? 6. Briefly describe to us what training is done at the service level? Probe: ● What training materials should supervisors bring with them? ● As a percentage, how much of the supportive supervision visit time is spent on training? 7. In all this process of training and Supportive Supervision, what is the role of the local NPMC staff and health authorities? Probe: ● How would you improve the process? 142 KII Addendum B: Additional Questions about the Management Information System/Data Hub The purpose of this annex is to examine the experience of the implementation level management staff on their perceptions of the strengths and weaknesses of their Management Information System, and what recommendations they would have for improving performance monitoring and decision making. These open-ended questions are directed to project staff with responsibilities on data management and reporting. To better understand the context in which you perform your duties, I would like to ask you a few general questions: 1. Please describe (approximation is fine) how many people or beneficiary population is covered by the project area under your responsibility? Probe: ● How many health services (all types) that deliver malaria interventions are in your area? ● How many health personnel with malaria CM functions are in your area? ● How many laboratory services (with or without microscopy) are in your area? 2. Do the staff working on the project with data management functions collect all the data needed? If not, what are the main challenges or limitations? Probe: ● What do you say about the data coming from the national information system? Is it usually timely and complete? ● How could these deficiencies be overcoming? Training 3. Is there a training plan which includes staff involved in data-collection and reporting at all levels in the reporting process? Probe: ● In which areas staff needs more training, e.g., the electronic platform? ● Have all staff received training on the data management processes and tools? ● What additional training would you like to receive? 4. Have you been trained in how to analyze and interpret the data? If so, how often do you do it? Probe: ● What kind of decisions do you make based on the data you collect and process? Please give us some examples 5. Do you think you have all the data and information you need to assess how your program is progressing? If not, what additional information would you like to have? 143 SURVEY QUESTIONNAIRE – Impact Malaria PROGRESS AND RESULTS AT COUNTRY LEVEL Survey for 20 IM countries Draft Survey Content: INTRO PAGE: Welcome to the online survey being conducted as part of the mid-term evaluation of the five￾year USAID/ID/PMI project PMI Impact Malaria project (2018-2023). This survey provides an opportunity for staff and people involved with the countries where PMI Impact Malaria is working to contribute to the midterm evaluation. We ask you to answer the survey questions specifically for PMI Impact Malaria in the country or regions where you work. Your contribution would help strengthen malaria control interventions. Your participation in the survey is entirely voluntary. None of these questions is compulsory and you have the option to use the ‘I don’t know’ code when you do not feel you are in a good position to answer. You have the right to stop your participation at any time. If you have questions or experience any technical difficulties with the survey, please contact Patrick Sullivan, GH EvaLS Project Manager, psullivan@engl.com. Please click the button below to continue with the survey. RESPONDENT DATA PAGE: The answers you provide will be kept confidential and will not be attributed to you or anyone. Specific identifying information will be removed from the data set before it is shared or reported. The results of this survey will be aggregated for analysis and we will ensure that responses cannot be attributed to any individual. However, as a participant in the survey, we will list your name, title, and affiliation in the survey summary information that will be included in an appendix to the final submitted report. By providing the information below, you give your consent to participate in the survey. Date Date field Name Free Response Title Free Response Organization Free Response PMI Impact Malaria Country where you work (or have oversight) Pull down menu with country names (pick all that apply to your current position) 144 At what “level” of the project do you normally work? ● HQ/global ● Regional level ● Country/national level ● Province/sub-national region level ● District/County level ● Health Facility level How long have you been working with PMI Impact Malaria in this country Pre-coded responses ● Greater than 1 year ● Less than 1 year: The number of months you have worked with PMI Impact Malaria in this country if less than 1 year (0-12 months)_____ # Question Response Notes EQ1: To what extent has PMI Impact Malaria achieved the country-level objectives? The PMI Impact Malaria project has 2 objectives that are most relevant to activities at the national level: 5-point scale for each: 1= “not at all successful” to 5 = “extremely successful” NA = Not Applicable DK = Don’t Know Relates to evaluation overall. Specifically, provides subjective assessment of achievement of objectives. 1. Objective 1: Improve quality of and access to malaria CM and MIP interventions. On a scale from 1 to 5, how would you rate the success of PMI Impact Malaria in achieving Objective 1 in your country so far? 2. Objective 2: Improve quality of and access to other malaria drug￾based approaches, and provide support to pilot/scale-up newer malaria drug-based approaches (e.g., SMC, MDA). On a scale from 1 to 5, how would you rate the success of PMI Impact Malaria in achieving Objective 2 in your country so far? Improvements in malaria diagnosis and CM in the country 145 # Question Response Notes For Questions 3 – 7 - Please answer from the perspective of the level at which you normally work (HQ/global; Regional; Country/national; Provincial/sub-national region; District/county; Health facility): On a scale of 1 to 5, to what extent would you estimate that country’s capacity has improved as a result of PMI Impact Malaria-supported activities in the following programmatic areas: 5-point scale for each: 1 = “no/minimal improvement” to 5 = “substantial improvement” NA = Not Applicable DK = Don’t Know Relates to evaluation question 1.b. 3. What is the degree of improvement in malaria diagnostic capacity in general? 4. What is the degree of improvement in MM capacity? 5. What is the degree of improvement in the use and interpretation of malaria rapid diagnostic tests? 6. What is the degree of improvement in the treatment of uncomplicated malaria cases? 7. What is the degree of improvement in the treatment of severe malaria cases? Perceptions of health provider behaviors (quality of care) 8. To what extent do you think the results of malaria diagnostic tests determine treatment decisions? 5-point scale for each: 1 = “almost never” to 5 = “almost always” NA = Not Applicable DK = Don’t Know Relates to request to consider a focus on health provider behaviors (quality of care) – 1.b. 9. How often do you think a patient who received a NEGATIVE malaria test by microscopy continues to receive treatment for malaria? 10. How often do you think a patient who received a NEGATIVE test for malaria by rapid diagnostic test (RDT) continues to receive treatment for malaria? 146 # Question Response Notes 11. How often do you think pregnant women are counseled on how to prevent malaria during ANC visits? 12. How often do you think pregnant women are receiving IPTp at every possible opportunity during ANC visits? 13. How often do you think pregnant women are being treated for malaria according to the national guidelines? Perceptions of value of checklists/other tools/Data Hub 14. In the past, have you reviewed/used tools supported by the PMI Impact Malaria, such as checklists, guidelines, training materials, job aides, etc.? 1= Yes 2 = No 3 = NA Relates to evaluation question 1.d. SurveyMonkey programming note: If Q14 is yes, continue with Q15, if no, skip to Q17. 15. To what extent do you think the tools developed by Impact Malaria (such as those listed in previous question) have improved the quality of training? 5-point scale for each: 1 = “little to no improvement” to 5 = “substantial improvement” NA = Not Applicable 16. To what extent do you think the DK = Don’t Know tools developed by Impact Malaria (such as those listed in previous question) have improved the quality of supervision? 17. In the past, have you used the PMI Impact Malaria Data Hub or used the Data Hub information for quarterly or annual progress reporting? 1= Yes 2 = No 3 = NA Relates to evaluation question 1.e. 147 # Question Response Notes 18. To what extent do you think the Data Hub data have resulted in improvements in data use? 5-point scale for each: 1 = “no/minimal improvement” to 5 = “substantial improvement” NA = Not Applicable DK = Don’t Know SurveyMonkey programming note: If yes on Q17, continue with Q18, if no, skip to Q20. 19. To what extent do you believe the data and information obtained from the Data Hub is worth the effort required to digitize OTSS+ checklists? 5-point scale for each: 1 = “Not worth the effort required” to 5 = “Completely worth the effort required” NA = Not Applicable DK = Don’t Know SMC coverage and adherence 20. Does Impact Malaria support the use of Seasonal Malaria Chemoprevention (SMC) in your country? 1= Yes 2 = No 3 = DK Relates to evaluation question 1.f. SurveyMonkey programming: If yes on Q20, continue to Q21; if other response, skip to Q25. 21. How successfully has the implementation of SMC met its coverage objectives? 5-point scale for each: 1= “not at all successful” to 5 = “extremely successful” NA = Not Applicable DK = Don’t Know 22. How successfully has the implementation of SMC met its adherence objectives? 148 # Question Response Notes 23. To what extent do you agree with the following statement: “PMI Impact Malaria has successfully staffed, coordinated and managed the logistics of seasonal malaria chemoprevention campaigns” 5-point scale for each: 1= “strongly disagree” to 5 = “strongly agree” NA = Not Applicable DK = Don’t Know Relates to evaluation question 2f 24. To what extent does PMI Impact Malaria’s support of SMC campaign activities conflict with their ability to maintain support for routine CM and MIP activities? 5-point scale for each: 1 = “no/minimal impact” to 5 = “substantial impact” NA = Not Applicable DK = Don’t Know Relates to evaluation question 2f EQ2: To what extent has PMI Impact Malaria complied with the management requirements and functions outlined in the agreement, including planning, allocation of funds, coordination among the PMI Impact Malaria partnership (PSI, MCDI, UCSF, Jhpiego), staffing requirements, and in-country support? 25. To what extent has the oversight and management of PMI Impact Malaria headquarters affected the achievement of work plan objectives at central level? 5-point scale for each: 1= “greatly hindered” to 5 = “greatly supported” NA = Not Applicable DK = Don’t Know Relates to evaluation question 2a 26. To what extent has the oversight and management of PMI Impact Malaria headquarters affected the achievement of work plan objectives at country level? 27. To what extent has coordination between PMI Impact Malaria and in-country partners affected the overall PMI Impact Malaria project in meeting the objectives of the country workplan? Relates to evaluation question 2b 149 # Question Response Notes 28. To what extent do you feel that the in-country presence of PMI Impact Malaria staff has been sufficient to meet demands of the project? (i.e., do they have enough staff?) 5-point scale for each: 1= “project is inadequately staffed” to 5 = “project is adequately staffed” NA = Not Applicable DK = Don’t Know Relates to evaluation question 2c 28b. Comments (please provide any additional information to further explain your response): Free text 29. To what extent do you think PMI Impact Malaria has had the right staff (in terms of roles and/or expertise) to meet the demands of the project? (i.e., do they have staff with the correct skills for the jobs they are being asked to do?) 5-point scale for each: 1= “staff are not appropriately skilled ” to 5 = “staff are appropriately skilled ” NA = Not Applicable DK = Don’t Know 29b. Comments (please provide any additional information to further explain your response): Free response 30. How successful has PMI Impact Malaria been in adjusting to the growth of participating countries (from 10 countries in FY 2017 to 17 countries and 2 Regional buy￾ins in FY 2019)? 5-point scale for each: 1= “not success at all” to 5 = “very successful” NA = Not Applicable DK = Don’t Know Relates to evaluation question 2d 31. To what extent do you agree with the following statement: “PMI Impact Malaria has been able to hire staff, set up offices, launch activities, and continue activities to maintain agreed￾upon timelines.” 5-point scale for each: 1= “strongly disagree” to 5 = “strongly agree” NA = Not Applicable DK = Don’t Know Relates to evaluation question 2e 150 # Question Response Notes 32. To what extent do you agree with the following statement: “PMI Impact Malaria has successfully handled staffing, coordinating, and managing logistics for Therapeutic Efficacy Study (TES) activities” 5-point scale for each: 1= “strongly disagree” to 5 = “strongly agree” NA = Not Applicable DK = Don’t Know Relates to evaluation question 2g 33. To what extent do you agree with the following statement: “PMI Impact Malaria has successfully handled staffing, coordinating, and managing logistics for Operational Research (OR) activities” 5-point scale for each: 1= “strongly disagree” to 5 = “strongly agree” NA = Not Applicable DK = Don’t Know Relates to evaluation question 2h 34. From a reporting perspective, do you consider the PMP indicators that have been agreed upon at the HQ and/or in￾country are applicable and practical (i.e., clear, easy to collect/calculate, simple to report)? 5-point scale for each: 1= “not practical at all” to 5 = “very practical” NA = Not Applicable DK = Don’t Know Relates to evaluation question 2i 35. Do you consider the PMP indicators that have been agreed upon at the HQ and/or -at country level are useful in guiding programmatic activities? 5-point scale for each: 1= “not useful at all” to 5 = “very useful” NA = Not Applicable DK = Don’t Know LOOKING FORWARD 151 # Question Response Notes 36. To improve future programming, it is important to understand key challenges in project implementation. This question is an opportunity to provide a more complete picture of the barriers PMI Impact Malaria may have been facing in your country. In your view, what have been the major challenges or barriers to PMI Impact Malaria project implementation where you work? Multiple responses may be selected: ● Communication with PMI Impact Malaria headquarters ● Communication with PMI COR team ● Coordination between PMI Impact Malaria partners in country ● Sufficient numbers and appropriate skill set of PMI Impact Malaria staff in country ● Sufficient and timely funds for project activities ● Stock-outs/low stock of key malaria supplies such as RDTs, ACT ● Stock-outs/low stock of other essential supplies such as gloves, gauze ● Staff time/motivation ● Staff turnover/absenteeism ● Problems with basic infrastructure (water, electricity, etc.) ● Coordination with government ● Coordination with other malaria stakeholders (NGOs, UNICEF, Global Fund, etc.) in country ● National policies not up to date or fully implemented ● Insufficient data for decision-making ● Insufficient technical guidance/support from PMI Impact Malaria Headquarters ● Tools such as OTSS+ checklists changing over time ● Programmatic scope changing over time (whether expansion or contraction) ● Other (specify) 37. How could PMI Impact Malaria help you to address these challenges? Open-ended 38. Can you please share up to three useful lessons learned during program years 1-3? Open-ended, but pre-numbered (1, 2, 3). 152 # Question Response Notes 39. Please feel free to add any additional comments you may have. Open-ended 153 154 ANNEX 3: DOCUMENTS REVIEWED, KEY INFORMANTS, AND SURVEY RESPONDENTS Documents provided by USAID/PMI and PSI Project wide Approved work plans: Cameroon FY18, FY20 Core Work Plans FY18, FY19, FY20, FY21 Cote d’Ivoire FY18-19, FY20, FY21 DRC FY 18-19, 20, 21 Burkina Faso FY20 Cambodia FY20 Ghana FY 19, 20, 21 Kenya FY 18-19, 20, 21 LAC FY 21 Laos FY21 Madagascar FY 19, FY20 Malawi FY21 Mali FY18, FY19, FY20, FY21 Niger FY18, FY18-19, FY20 Rwanda FY19, FY20, FY21 Sierra Leone FY18, FY18-19, FY21 Tanzania FY21 Zambia OTSS FY20, FY21 Zambia ProAct approved protocol M&E Documents PMI Impact Malaria D2A Frameworks Presentations Cameroon, Ghana, Kenya, Niger, Senegal and project wide overview presentations Gender Outputs Wednesday Webinar_Gender and Malaria Kenya IM Gender and barrier study_Protocol_Approved Cameroon ethics committee submission 4 French language documents the team was unable to read Approved Annual Reports FY 18, FY19, FY20 Quarterly Reports 155 Mali SMC Reports 2019, 2020 Success Stories and External Products PMI IM Success Stories_Complete List of Links PMI IM Malaria_SMC_Learning_Brief PMI IM Success_Story_Cameroon_Niger, CDI, Ghana, Kenya Miscellaneous Impact Malaria Technical SOW in contract Impact PMP v2 November 2020 IM Supportive Supervision Framework submission 4, May 27, 2019 Ghana HNQIS Implementation Brief Kenya Mentorship summary analyses Q3, 2020 Cameroon Assessment Report Final Draft In-Depth Country Reviews Cameroon IM Cameroon Year 1 Work Plan May 2018 – Sep 2019 Quarterly Report Jan – Mar 2019 Quarterly Report Apr – Jun 2019 IM Cameroon Year 1 Work Plan Addendum Jul 2019 – Feb 2020 Quarterly Report Oct – Dec 2019 Quarterly report Jan – Mar 2020 FY20 Work Plan Mar 2020 – Feb 2021 Quarterly Report Apr – Jun 2020 Quarterly Report Oct – Dec 2020 Rapport du Premier Round de Supervision OTSS+ Rapport Final MDRT: Rapport de Formation des Cadres de Laboratoire sur le Diagnostic de Qualite du Paludisme Additional reports produced by IM and NMCP KII and survey findings Ghana Approved Work Plans for FY19, FY20, FY21 Quarterly Reports for Jan-Mar 2019; Apr-Jun 2019; Oct-Dec 2019; Jan-Mar 2020; Apr-Jun 2020; Oct-Dec 2020; Jan – Mar 2021 Impact Malaria Data Hub National HMIS data Key Informant Interviews MIS 2014 and 2019 Ghana Health Service Institutional Care Division - Report of Laboratory Outreach Training & Supportive Supervision Round 17 and Proficiency Testing Scheme Round 5 Kenya Impact Malaria Data Hub PMI IM Kenya FY20 Work Plan October 2019 – September 2020. PMI IM Kenya Quarterly Reports (Oct-Dec 2018; Jan-Mar 2019; Apr-Jun 2019; Oct-Dec 2019; Jan-Mar 2020; Apr-Jun 2020; Oct-Dec 2020; Jan-Mar 2021). Kenya Mentorship summary analyses (Qtr 3 2020) 156 Kenya supportive supervision-mentorship Summary Analysis_OctDec2019 Kenya supportive supervision-mentorship Summary Analysis, Jan-Mar 2020 PMI Impact Malaria Annual Report, Project year 3: Fiscal Year 2020 Key Informants (5 IM staff; 3 MOH staff; 3 IM partner organizations; 1 PMI staff) Kenyan national census conducted in 2019. DHS2014 and MIS 2015 Niger IM Niger Year 1 Work Plan May 2018 – Sept 2019 IM Niger Year 1 Work Plan Addendum Feb 2019 – Mar 2020 IM Niger FY 20 Work Plan Apr 2020 – Mar 2021 Analyse de le supervision formative OTSS+ clinique de 12 Districts sanitaires des regions de Dosso et Tahoua (Niger) Niger KII and survey findings Senegal IM Senegal country brief FY20 final approved AR IMSenegal_FY20_MDA Protocol UCSF and Senegal CNERS IRB approvals UThies Year 1 Progress Report 3 HQ KIIs, 4 country KIIs and 19 survey responses Additional documents reviewed by Evaluation Team PMI 14th Annual Report 2020 CDC, Malaria Worldwide, Global Malaria Activities, PMI website Malaria Matchbox Tool. English. PMI 11th Annual Report 2017 RBM SBCC Framework 2018-2030 Zero Malaria Toolkit Final. 157 Survey Respondents38 Survey Respondents’ Affiliation Number USAID 33 CDC 10 MOH 50 MCDI 5 Other IM 13 PSI 15 Universities 4 WHO 3 Other 12 Data missing 2 Total 147 Survey Respondents’ Work Levels Number HQ/Global 31 Country/National 103 Regional Level 11 Province/Sub-Regional 3 District/county Level 8 Missing 1 Total 158 Country Where Survey Respondents Work Number Benin 18 Burkina Faso 15 Cambodia 9 Cameroon 19 Cote d’Ivoire 19 DRC 20 Ghana 24 Kenya 24 Lao 13 Madagascar 23 Malawi 17 Mali 17 Niger 21 Rwanda 13 Senegal 15 Sierra Leone 13 Tanzania 17 Zambia 21 Total 318* * Some respondents have oversight for more than one country 38 Not all respondents answered every question, therefore totals differ. 158 Survey Respondents’ Length of Affiliation with IM Number Greater than 1 year 77 Less than 1 year 30 Total 107 Key Informants Name Position Organization Ricki Orford Senior Project Director PSI Dr. Lawrence Barat Senior Technical Advisor PSI Mary Warsh Deputy Project Director PSI Natalie Hendler Country Operations Director Jhpiego Jacob Odentz Lead, Project Management PSI Leticia Isambo Senior Finance Manager PSI Gladys Tetteh Malaria Director Jhpiego Laura Skolnik Senior Director, Global Programs Jhpiego Dr. Cara Smith Gueye Associate Director UCSF Roly Gosling MEI Executive Director UCSF Lenny Kyomuhangi Senior Program Manager MCDI Tarryn Haslam Director, Malaria & WASH PSI Chris Lourenco Deputy Director Malaria PSI Andrea Bosman Team Lead WHO Grace Adeya Malaria Task Order Lead GHSC-PSM Angela Acosta Breakthrough Action Andre Marie Tchouatieu Secretariat Lead- SMC Alliance MMV Jean Yves Mukamba Chief of Party IM Cameroon Eric Tchinda Meli Technical Advisor IM Cameroon Abas Mouliom Diagnostic Technical Advisor IM Cameroon Dr Judith Hedje Resident Advisor CDC-PMI Cameroon Jessica Butts PMI HQ Country Lead CDC- PMI HQ Dr Jean Pierre Kidwang Regional Coordinator Regional Malaria Coordin. North Dr Olivier Kakesa Resident Advisor Measure Malaria Dr EWANE née EKOYOL EKOBE Germaine Chef de section Prise en charge des cas, Programme National de Lutte Contre le Paludisme Kwame Ankobea Malaria Program Specialist PMI Ghana/USAID Nana O. Wilson Resident Advisor PMI Ghana/CDC Raphael Ntumy Chief of Party IM Ghana James Sarkodie Deputy Chief of Party IM Ghana Amos Asiedu M&E Advisor IM Ghana Eva Mensah Deputy Director of Nursing Services Ghana Health Service 159 Name Position Organization Williams Addo Mills Pappoe Head, Central Laboratory Unit Ghana Health Service Prince Owusu Team lead VectorLink Ghana Dr. Kofi Amo Kodie Regional Director of Health Services Ghana Health Service Dr. Felicia Amoo- Sakyi Regional Director Dr. Fred Adomako-Boateng Regional Director Dr. Damien Punguyire Regional Director Dr. Paul Boateng NMCP Case management focal person Sylvester Segbaya Breakthrough Action COP Dr. Dickson Mwakangalu Chief of Party, Impact Malaria IM Kenya Hellen Gatakaa M&E Advisor IM Kenya Dr. Augustine Ngindu Senior Technical Advisor IM Kenya Dr. Maureen Mabiria Case Management Technical Advisor IM Kenya Dr. Victor Sumbi Technical Adisor, Supply chain Aftya Ugavi Dr. George Wadegu Malaria technical advisor Tupime Kaunti Regina Kandie Unit Manager, Case Management, NMP MOH Dr. Edwin Onyango County Malaria Coordinator MOH, Busia County Sarah-Blythe Ballard RA CDC/PMI Kenya Daniel Wacira Project Management Specialist USAID/PMI Kenya Dr. Githuka NMCP head Head DNMP Grace Miheso COP Breakthrough Action Daniel Koko Chief of Party IM Niger Elisha Sanoussi Malaria Program Director IM Niger Dr Hadiza JACKOU Head MOH/NMCP Dr Solange DIORI Chief of Neonatal and Child division MOH/Maternal and Child Health Direction Dr Philipe TACHE Country Director PSM Lungi Okoko PMI HQ Country Lead USAID-PMI HQ Dr Tassiou Health Region Director MOH Prof Jean Louis Ndiaye Head of the Research and Innovation Division University of Thies Dr. Elhadji Ba Health Program Manager Institut de Recherche pour le Développement (IRD) Dr. Fatou Ba Research and vector control NMCP focal point MOH Dr. Kathy Ramierz CDC Resident Advisor CDC/PMI Leah Moriarty Former PMI HQ Country Lead CDC-PMI HQ Meera Ventakesan Alt-COR PMI-USAID Anne Linn Child Health Task Force AFR-USAID Susan Youll MIP POC PMI-USAID Jordan Burns Severe Malaria POC PMI-USAID 160 Name Position Organization Kevin Griffith PMI-USAID Eric Halsey TES POC PMI-CDC Kim Connolly 161 ANNEX 4: EVALUATION TEAM MEMBERS Deborah McSmith, MPH, Team Lead Ms. Deborah McSmith is a monitoring and evaluation specialist with over 15 years of experience working with USAID health related programs. She has led multiple evaluations of USAID activities including the Mid-Term Performance Evaluation of the USAID/PMI Malaria Care Case Management Project. Deborah was responsible for performing all of the responsibilities of the Team Lead, including drafting and finalizing the data collection tools, leading data collection, and drafting and submitting the report. Deborah also previously served as the Team Lead of the Mid￾Term Evaluation of the USAID/Zambia Integrated Systems Strengthening Project which worked with local authorities to build capacity for strengthened malaria, HIV/AIDS, family planning, and MCH services. In addition to her evaluation work with malaria-focused programs, Deborah has led evaluations of a diverse range of health-related topics including a gap analysis for child protection sector in Moldova; a baseline assessment in South Sudan to inform integrated programming for maternal health including obstetric fistula prevention/treatment, gender-based violence including female genital cutting, and youth civil participation; and a final evaluation of USAID/South Africa Umbrella Grants Mechanisms with three managing partners and over 150 sub partners to build institutional capacity for HIV/AIDS prevention, care, treatment response. Peter Bloland, DVM, MPVM, Senior Malaria Specialist Dr. Peter Bloland is a Senior Malaria Specialist with over 30 years of experience. Dr. Bloland previously served in several notable positions at the US CDC, including as an epidemiologist at the Malaria Branch; Chief of the Malaria CM Unit; and as the CDC Team Leader to PMI during its inception, where he was responsible for providing technical and administrative leadership to all PMI activities within CDC during its first year of start-up. In addition, Dr. Bloland served as the Director at the Division of Public Health Systems and Workforce Development of the CDC ,which supported the CDC flagship Field Epidemiology and Laboratory Training Program (FELTP), the Sustainable Management Development Program, the Global Public Health Informatics Program, an Integrated Disease Surveillance and Response (IDSR) support team, and the National Public Health Institutes Development Support team. Dr. Bloland also served as the Acting Director of the Global Immunization Division at the CDC. Dr. Bloland’s malaria research is widely published, including his co-authored book, “Malaria Control During Mass Population Movements and Natural Disasters”. Marcelo Castrillo, MD, MPH, Analyst Dr. Marcelo Castrillo has more than 30 years of experience in improving health systems and program performance through monitoring, evaluation, and learning. Dr. Castrillo has vast experience in scientific methods for the design of performance and impact evaluations, and operations research. Their experience includes establishing hypotheses and evaluation questions, definition of dependent and independent variables, sampling frames, development of instruments for data collection and development of analysis plans. They are expert in complementary qualitative and quantitative methodologies to examine and analyze specific issues from different perspectives, and in identifying the weaknesses of the different methodologies, and to offer a more holistic view. 162 Dr. Castrillo served as a District Officer of the Bolivia National Malaria Eradication Service (SNEM) where he worked in Riberalta, Bolivia, the department with the highest prevalence rate, managing cases and carrying out vector control measures. He has also been a member of the Malaria Implementation Resources Team (MIRT) of the World Bank. 163 ANNEX 5: PMI IMPACT MALARIA’S RESPONSE TO COVID-19 The COVID-19 pandemic presented huge implementation challenges for Impact Malaria, particularly in terms of imposed travel restrictions that led to the cancellation of planned technical support visits. Regional ECAMMs were cancelled in DRC, Madagascar, and Zambia due to local travel restrictions. In DRC, the national ECAMM was cancelled because the facilitator was not able to travel to the country due to international travel restrictions. Malaria-focused IM activities were adapted in some countries to respond to the pandemic, including building COVID-19 training into SMC preparation and integrating COVID-19 considerations into OTSS+ checklists. COVID-19 significantly affected IM’s work on strengthening and expanding OTSS+. The pandemic limited classroom trainings; required the introduction of measures to protect supervisors, project staff, and health workers during scheduled rounds of supervisory visits; and interfered with supply chains for essential commodities. The project rapidly shifted to virtual or mixed models for training and mentoring, reducing class sizes and using larger training rooms to enable social distancing and coaching country staff through calls and video chats. IM provided face￾coverings and sanitation supplies to field staff and MOH counterparts for supervisory visits, so that many OTSS+ rounds could still be completed. Outside of the usual buy-in work plans, USAID Missions in Cameroon, DRC, and Ghana requested IM support to their Ministries of Health’s responses to the pandemic. Each country developed objectives and a six-month workplan to fill gaps and complement the work of other partners, focusing on areas of IM’s geographic coverage and scopes of work. Requested support included assisting in the development of guidelines and tools for detection and management of COVID-19 infections and infection prevention and control in clinics and laboratories, and training health workers and laboratory technicians on these new guidelines. In addition to technical and operational IM team members, three consultants were brought on board with laboratory, CM, and operational specialties to ensure that the country teams received comprehensive feedback in a short timeframe. Weekly meetings with each IM COVID-19 country team ensured that activities were closely tracked. Each country adopted a different approach to preparing for and managing the COVID-19 pandemic. Cameroon and DRC were able to draw on their experience in managing Ebola and other outbreaks and were able to adapt existing protocols and reactivate emergency communication channels. Cameroon moved quickly to develop guidelines and only required IM support to disseminate materials. In Ghana, IM supported the MOH to organize a workshop for the development of COVID-19 laboratory and CM guidelines, SOPs, training materials, and job aids. IM provided technical review of these tools and the companion training manual. In DRC, IM supported the Scientific Committee to prepare national guidelines and review the accompanying training manual. In DRC and Ghana, IM contributed to development of references and training materials and provided inputs and support to the development of COVID-19 laboratory guidelines, which were approved by the Ghana Health Service (GHS). This work reinforced country-specific guidance to align it with WHO guidelines. 164 IM also developed a COVID-19 preparedness tool to serve as a checklist for planning events, such as health training or other workshops, to minimize the risk of COVID￾19 transmission. The tool was developed in English and French, shared with the three countries, made available to other PMI countries, and annexed into the Training Manuals for CM and Laboratory COVID-19 Guidelines in DRC and Ghana. IM developed ten training modules based on WHO and CDC guidance in French and English in a user-friendly format. Modules included: ● Clinical facilitation guide ● Overview of COVID-19 ● Clinical Infection, Prevention and Control ● Clinical CM; Specimen Collection, Transport, and Testing ● Rational use of medical masks ● Lab Facilitation Guide ● Lab Microbiological Risk Assessment ● Lab Infection Prevention and Control ● Lab Biosafety ● Consultant Training Modules have been combined and developed into six e-learning modules in French and English, which will be accessible to all who need them, using an interactive yet low-bandwidth platform that can be viewed on a tablet or smartphone. All modules in both languages were to be completed by PY4 Q1, posted on the IM website, and shared with all PMI countries. IM country teams supported the three Ministries of Health to adapt country-specific training materials and implement trainings. In Cameroon, IM supported the Ministry to carry out training to health workers in the North and Far North regions. In DRC, 243 laboratory technicians in nine provinces were trained with IM support on COVID-19 and safety procedures, IM supported the establishment of a system to provide technical support and troubleshooting for participants as they implemented new procedures. An additional 216 clinicians were trained on COVID-19 CM guidelines. IM is also supporting the GHS to rollout laboratory and CM training on COVID-19 and proper safety practices, in coordination with other partners. IM supported NMCPs to integrate COVID-19 considerations into laboratory and clinic OTSS+ checklists. The first round of supervision using these updated checklists was completed in DRC in August 2020. IM Ghana worked with the GHS to update their integrated supportive supervision checklist to collect data on facility preparedness to respond to COVID￾19. A round of supervision with the updated checklist was carried out in July 2020. A rapid assessment was also conducted in Ghana in August 2020 to understand the impact of COVID-19 on the ability of health facilities to deliver essential services and the effects of COVID-19 on care￾seeking. This assessment provided a snapshot of where gaps existed in services, by region. IM country teams faced challenges with training participants not being able to access internet, or connections being of poor quality. IM DRC developed an innovative solution with the MOH by facilitating a training at national level by webinar for lab technicians across nine provinces targeting 243 lab technicians, who in each province gathered in one room with physical distancing. Following each module delivered by webinar, provincial groups then discussed the content with a live facilitator in each room. It was the first training of its kind and was considered successful; a “how to” guide was developed for replication of this approach. Participants created a WhatsApp group to maintain their connection and post questions to the group. 165 The Ministries supported by IM have expressed their appreciation for the support of specific events as well as routine activities that would have otherwise been under-supported. In DRC, IM has supported periodic national and provincial level COVID-19 committee meetings, the transportation of samples, and reporting data into the national surveillance system. IM received a “Diploma of Excellence” from the Governor of Katanga Province for its assistance. IM Cameroon’s ability to support BA with meeting logistics was also appreciated. Cameroon: After announcing the first case of COVID-19 on March 6, 2020, the MOH organized its response around eight pillars: coordination, surveillance, rapid response, laboratory, infection prevention control (IPC), CM, risk communication, and logistics. Measures to curb transmission included the closure of borders, schools, churches, mosques, and restaurants, as well as requiring face masks in public places. As of September 30, 2020, 20,838 COVID-19 cases were recorded in Cameroon, with 418 deaths. Mass testing found that 45 percent of those who tested positive were asymptomatic. During outbreaks and pandemics, resources tend to be channeled toward ending the health crisis, which prevents health systems from responding to other health priorities. IM therefore integrated COVID-19 support into the existing MoH structure in the Far North and North regions of Cameroon where the project is responsible for malaria CM. IM worked with the NMCP to integrate key COVID-19 considerations into planned SMC activities. WHO guidelines were adapted into a short COVID-19 module that included new approaches to implementing the SMC designed to prevent transmission of COVID-19. In the North and Far North regions, 5,800 and 9,450 mobilizers-distributors were briefed using the COVID-19 module respectively. Some of the key changes made to the SMC methodology to respond to COVID-19 included: maintaining social distancing, improved IPC, inclusion of COVID-19 messaging in communication, having parents/caretakers administer medicines, and limiting directly observed treatment to the first day to limit exposure between community members and distributors. The campaign was completed successfully, and no mobilizers-distributors were infected by COVID-19 during it. From July 7-24, 2020, IM supported the MOH to organize training workshops to strengthen the ability of frontline clinical staff to manage COVID-19 cases in accordance with national guidelines. Overall, 187 health professionals were trained in both regions (98 percent of the total targeted), representing all 45 health districts, including medical doctors, hygienists, nurses, and mental health nurses, based on participant lists provided by the MOH (Table 12). The workshops were facilitated by MOH staff using modules developed by the MOH. Following the training, IM’s support was commended by the MOH, as only three of Cameroon’s ten regions had staff trained from every health district, and IM supported two of these. From September 7-14, 2020, IM organized a briefing of CHWs in two districts of the North region (Ngong and Guider), and two health districts in the Far North region (Mokolo and Kaele). Districts were selected based on where IM was providing support for CHW activities and could most effectively program limited resources. The objective was to ensure that CHWs were equipped with basic COVID-19 knowledge and knew how to manage cases at their level. This included training in how they communicate to the community that COVID-19 testing was available free of charge, the location of testing and treatment centers, and what to expect from treatment. These briefings were designed and planned in collaboration with BA, and the materials and facilitators were provided by the MOH. Côte d'Ivoire: The country recorded its first cases of COVID-19 on March 11, 2020. The government took restrictive measures to limit the spread of the pandemic such as locking down the capital, Abidjan; restricting travel from Abidjan to the provinces; and limiting gatherings of 166 people. The government raised awareness among the population on the importance of social distancing and hand hygiene. These measures limited the implementation of IM activities including visits to HF, and the organization of meetings and other workshops. To mitigate this situation, IM Côte d'Ivoire implemented an e-mentoring approach. E-mentoring consisted of keeping track of malaria services in HFs through telecommunication such as phone calls and WhatsApp messages between IM OTSS officers (mentors) and HW. IM implemented e￾mentoring in April and May 2020 and helped support providers to maintain health service continuity during the pandemic, following up on availability of medicines and equipment and answering questions and concerns about infection prevention and control. Ghana: The Ghana MOH and GHS requested IM’s support to develop and roll out COVID-19 CM and laboratory guidelines, assess facility preparedness, and determine the impact of COVID￾19 on the delivery of essential services. Reporting as of September 30, 2020 indicated a total of 46,482 confirmed COVID-19 cases and 301 deaths. The most affected regions were the Greater Accra and Ashanti regions, reporting about 50% and 24.6% respectively of total cases. IM has supported the government’s COVID-19 response through: ● Support for supervision visits to assess and strengthen COVID-19 preparedness of HF and HW ● Development and implementation of a rapid HF assessment of the COVID-19 impact to deliver essential services ● Support to update clinical and laboratory guidelines and tools for the COVID-19 context ● Support for the implementation of updated guidelines on diagnosis, triage, referral, and CM of fevers ● Support of quality control for sample taking, testing, and diagnostic capabilities of testing/reference laboratories and biosafety procedures ● Support to integrate COVID-19 preparedness assessment and strengthening in HF and with HW, alongside routine ISS focused on existing health service delivery gaps. ● Support for implementation of a COVID-19 addendum checklist as part of the second round of ISS in all 16 regions of Ghana. The COVID-19 addendum addressed planning and coordination, surveillance, situation monitoring and assessment, CM, infection prevention and control, logistics security, social mobilization and risk communication, and point of entry ● Supported a rapid HF assessment of the COVID-19 impact on essential health services in Ghana from August 3-14, 2020, to provide GHS with information to better target program and behavior change activities. The assessment used a secondary analysis of HMIS data and cross- sectional review combining qualitative and quantitative approaches to assess the effect of COVID-19 on health service delivery, mitigate these effects, and enable improvement of health service performance indicators at HF level amidst the COVID-19 pandemic.. The assessment was approved as a non-human subject research by Johns Hopkins School of Public Health Institutional Review Board (JHSPH-IRB) on July 13, 2020. In Ghana, results from the national HMIS that compare the number of uncomplicated malaria suspected cases from January to June 2018, 2019 and 2020, respectively, show a reduction in suspected malaria cases during the period of COVID-19-related restrictions. According to qualitative findings, this was due to fear of contracting COVID-19 at HF, leading clients to practice self-medication and delayed care-seeking.