At the request of the United States Agency for International Development (USAID), this publication was prepared independently by International Business and Technical Consultants, Inc. (IBTCI). Photo by Michael Duff for USAID EVALUATION FINAL REPORT SECOND PERFORMANCE EVALUATION (PE2) OF USAID EBOLA PILLAR II ACTIVITIES September 2019 Second Performance Evaluation of USAID Ebola Pillar II Activities: Final Report USAID CONTRACT # AID-OAA-I-15-00022 Task Order # AID-OAA-TO-16-0040 Authors: Sara Woldehanna, Barbara De Zalduondo, Donna Espeut, Karim Sahyoun, Jackie Yiptong Avila, and Elisabeth Nolan Submitted June 30, 2019 Resubmitted: August 1, 2019 Final September 1, 2019 Prepared for: United States Agency for International Development Ronald Reagan Building and International Trade Center 1300 Pennsylvania Ave NW Washington, DC 20004 Contact Information: Annette Bongiovanni, Project Director International Business & Technical Consultants, Inc. (IBTCI) 8618 Westwood Center Drive Suite 400 Vienna, VA 22182 USA Tel: +1 (703) 749-0100 Citation: Woldehanna, S., De Zalduondo, B., Espeut, D., Sahyoun, K., Yiptong Avila, J., Nolan, E. “Second Performance Evaluation of USAID Ebola Pillar II Activities: Final Report.” Evaluation Report to USAID, September 2019, International Business & Technical Consultants, Inc., Vienna, VA. Activity Description: The Ebola Pillar II, Monitoring, Evaluation and Learning (MEL) activity is a three-year USAID-funded contract addressing USAID-coordinated efforts in mitigating the second-order impacts of the Ebola virus outbreak in Guinea, Liberia, and Sierra Leone. The activity focuses on four main components: evaluation, routine monitoring, data quality assurance, and improved knowledge management and learning. The activity is led and managed by International Business and Technical Consultants Inc. (IBTCI), with partners StatView International in Guinea, Global Research Insights, LTD (GRI) in Sierra Leone and Liberia, and Opinion Research Business (ORB) International in all three countries. Cover Photo: Sierra Leonean family outside renovated clinic. Photo by Michael Duff for USAID SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES FINAL REPORT September 2019 This publication is made possible by the generous support of the American people through the United States Agency for International Development (USAID) under the terms of contract AID-OAA-I-15-00022/AID-OAA-TO-16-00040 managed by International Business and Technical Consultants Inc. (IBTCI) and does not necessarily reflect the views of USAID or the United States Government. ACKNOWLEDGMENTS The authors would like to acknowledge the diligent involvement of the full technical team: Zhuzhi Moore, Annette Bongiovanni, Richard Columbia, Andrew Reuter, Roger Emmanuel Millimono, and Richard Ngafuan. The authors would like to thank the activity coordination staff for their continued support and commitment, including Katherine Labombarde, Maria Pastor, Yasmin Ibrahim, and Victoria Sesay. Special thanks to the editors, Irene Labombarde and Erin Dowling, and graphic designer Erin Dowling. A very special thank you goes to Latrisha Chappin and Nadine Ritcheson from USAID’s Bureau for Africa (USAID/AFR) and representatives from the bureaus for Global Health, Global Development Lab, Food Security, Democracy, Conflict, and Humanitarian Assistance, and Economic Growth, Education, and Environment, the Guinea, Liberia, and Sierra Leone missions. Various implementing partners provided valuable support and constructive comments throughout the preparation of this report, for which the authors are grateful. Finally, our gratitude goes to the many individuals who participated in the key informant interviews and focus group discussions. This evaluation would not have been possible without your contribution. n i TABLE OF CONTENTS LIST OF CASE STUDIES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ii ACRONYMS AND ABBREVIATIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii EXECUTIVE SUMMARY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . a Evaluation Purpose and Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . a Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . a Findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . a Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . b INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Pillar II Activities and Findings of Performance Evaluation 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Second Performance Evaluation Objectives and Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 METHODOLOGY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Case Study Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Data Sources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Data Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 FINDINGS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Effectiveness (EQ1): What factors contribute to the achievement of Pillar II’s objectives? . . . . . . . . . . . . . . . . . . . . . . . . 6 Health . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Governance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Agriculture and Food Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 Innovation, Communication, and Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 Self-Reliance (EQ2): What factors contribute to advancing partner country self-reliance? . . . . . . . . . . . . . . . . . . . . . . 19 Perceptions of Self-reliance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 Capacity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 Commitment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 CONCLUSIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 (EQ1) Effectiveness: What factors contributed to the achievement of Pillar II’s overarching objectives? . . . . . . . . . . . 27 (EQ1.a) What were the factors that contributed to the effectiveness of stand-alone activities and why? . . . . . . . . . . . . . . . . . . . . . . 27 (EQ1.b) What were the factors that contributed to the effectiveness of a combination of activities within each sector and why? . 28 (EQ1.c) What were the factors that contributed to the effectiveness of cross-sectoral activities and why? . . . . . . . . . . . . . . . . . . . . . 28 (EQ2) Self-Reliance: What factors contributed to advancing partner country self-reliance? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 (EQ2.a) What were the salient factors and/or determinants that positioned these activities to advance a country’s journey to self￾reliance? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 (EQ2.b) To what extent did these activities strengthen country systems to mitigate and respond to future epidemics? . . . . . . . . . 29 RECOMMENDATIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 REFERENCES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 Annexes are provided in two separate volumes: Volume 1, Annex A (Case Studies) and Volume II, Annexes B–J. ii n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES LIST OF CASE STUDIES Case Study 1. SIAPS: Developing relationships in Guinea to restore and expand the capability of the public health supply chain . . . 8 Case Study 2. CSH: Strengthening health systems in Liberia through decentralization and coordination . . . . . . . . . . . . . . . . . . . . . . . 8 Case Study 3. SIAPS: A comprehensive systems approach in Sierra Leone leads to quick gains in pharmaceutical management . . . . . 9 Case Study 4. Restoration of health services in Guinea and Liberia (Regional) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Case Study 5. CEPPS: Working to build smoother electoral and democratic processes in Guinea . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Case Study 6. CSML: Strengthening radio and civic engagement to enable inclusive community mobilization in Liberia . . . . . . . . . 13 Case Study 7. Emergency access to food security resources for vulnerable Guinean households . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Case Study 8. EREL: Cash transfers and growing trust build food security in Liberia . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Case Study 9. EFSP: Cash as a stop-gap or as a catalyst toward resiliency? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 Case Study 10. Grand Challenges: Critical new innovations quickly developed and tested in the region . . . . . . . . . . . . . . . . . . . . . . 17 Case Study 11. mHero: Innovation does not always require starting from scratch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 Case Study 12. APC/PERHS, IOM/IPC: Rehabilitating services and relationships in Sierra Leone to optimize RMNCH . . . . . . . . . . 22 Case Study 13. JHU CCP/HC3: Multi-country insights from the Health Communication Capacity Collaborative (HC3) . . . . . . . . 23 Case Study 14. IRC/PACS: Empowering communities to engage in Liberia . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 Case Study 15. Abt/HFG: Strengthening the future functionality of institutions and financing in Guinea . . . . . . . . . . . . . . . . . . . . . . 26 n iii ACRONYMS AND ABBREVIATIONS ACC Anti-Corruption Commission (Sierra Leone) AFS Agriculture and food security CDC Centers for Disease Control and Prevention CENI Independent electoral commission (Guinea) CEPPS Consortium for Elections and Political Process Strengthening (NDI) CHA Community health assistant CHT Country health teams CRS Catholic Relief Services CSH Collaborative Support for Health (MSH) CSML Civil Society and Media Leadership (IREX) CSO Civil society organization DDMS Directorate of Drugs and Medical Supplies (Sierra Leone) DHIS2 District Health Information Software 2 ECM Economic crisis mitigation EFSP Emergency Food Security Programme eLMIS Electronic Laboratory Management Information System EPHS Expanded Package of Health Services EQ Evaluation question EREL Economic Recovery from Ebola for Liberia (Mercy Corps) ESP Ebola Smart Pod ETL Education Through Listening EU European Union EVD Ebola virus disease FFP Food for Peace (USAID) FGD Focus group discussion GAO Government Accountability Office GOG Government of Guinea GOL Government of Liberia GoSL Government of Sierra Leone HC3 Health Communications Capacity Collaborative (JHU CCP) HFG Health Finance and Governance (Abt) HIS Health information systems HRIS Human resources information system IBTCI International Business & Technical Consultants, Inc. ICT Innovation, communication, and technology IOM International Organization of Migration IP Implementing partner IPC Infection prevention and control IRC International Rescue Committee JHU CCP Johns Hopkins University Center for Communications Programs JSI/APC John Snow International/Advancing Partners & Communities KI Key informant KII Key informant interview LMIS Laboratory Management Information System LSA Liberia Statistical Analysis M&E Monitoring and evaluation MCHIP Maternal and Child Health Integrated Program MCSP Maternal Child Survival Program (Jhpiego) MEL Monitoring, evaluation, and learning MOH Ministry of Health (Liberia, Guinea) MOHS Ministry of Health and Sanitation (Sierra Leone) MSH Management Sciences for Health NaCSA National Commission for Social Action (Sierra Leone) NCHSPP National Community Health Services Policy and Plan (Liberia) NDI National Democratic Institute OpenHIE Open Health Information Exchange PACS Partnership for Advancing Community Based Services (IRC) PBSL Pharmacy Board of Sierra Leone PE1 First Pillar II Performance Evaluation PE2 Second Pillar II Performance Evaluation PERHS Post-Ebola Recovery of Health Services (JSI/ APC) PHU Peripheral health units PIDS Performance Indicator Database PPP Public-private partnerships RHS Restoration of Health Services (Jhpiego) RMNCH Reproductive, maternal, newborn and child health SBCC Social and behavior change communication SIAPS Systems for Improved Access to Pharmaceuticals and Services (MSH) USAID United States Agency for International Development USG United States Government iv n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES CLOCKWISE, FROM TOP LEFT: Woman winnows rice in Sierra Leone; three Liberian health care workers outside their clinic; messages via cell phone SMS were pivotal to health communications and mobile money; women take on greater leadership roles in post-Ebola governance activities. Photo by Aliza Waxman for USAID Photo by Sam Phelps for UNICEF Photo by Michael Duff for USAID Photo by Michael Duff for USAID EXECUTIVE SUMMARY n a EXECUTIVE SUMMARY Evaluation Purpose and Questions The United States Government (USG), with the United States Agency for International Development (USAID) as the lead, mounted a multisectoral response to the 2014–2016 outbreak of Ebola virus disease (EVD). Pillar I activities sought to contain the outbreak. Pillar II activities, designed in collaboration with the governments of Guinea, Liberia, and Sierra Leone, are the focus of the present evaluation. Pillar II centered on post￾EVD recovery response activities with the goal to ensure that countries build back economic and social systems better than before the outbreak. In this way, they become better prepared for other future disease outbreaks or crises. Pillar II aimed to: (Objective 1) prevent the loss of development gains; (Objective 2) recover and strengthen existing institutions and infrastructures; and (Objective 3) build sustainable systems through public-private partnerships (PPPs), innovation, and capacity building. In October 2016, USAID awarded International Business & Technical Consultants, Inc. (IBTCI) a three-year contract to conduct monitoring, evaluation, and learning (MEL) activities in support of Ebola Pillar II work. Between December 2017 and February 2018, IBTCI conducted the first of the two Performance Evaluations (PE1) (Moore et al, 2019). This document is the second Performance Evaluation (PE2), conducted between December 2018 and June 2019. The evaluation was designed to answer the following general questions, the details of which are described in Table 1 of the full document. ■ EQ1. Which Pillar II activities contributed to the achievement of one or more of Pillar II’s three overarching objectives? What were the factors that contributed to the effectiveness of activities? ■ EQ2. What Pillar II activities facilitated partner country self-reliance? What were the salient factors and/or determinants that positioned these activities to advance a country’s journey to self-reliance? Methodology Unlike PE1, which aimed to examine whether or not Pillar II activities were effective, the goal of this evaluation is an in-depth understanding of the factors leading to or hindering success. Therefore, a multiple case study approach was taken for this evaluation. Cases spanning all three countries in four sectors were selected and explored: Health; Agriculture and Food Security; Governance and Economic Crisis Mitigation; and Innovation, Communication, and Technology. For this evaluation, the PE2 evaluation team conducted 104 key informant interviews, 41 focus group discussions, reviewed over 800 documents, and conducted site visits to service delivery areas and communities in the catchment area. Fifteen (15) activities under Pillar II are discussed in this report and the case studies appear in full in Annex A. Findings It is important to note that the findings in this report cannot be generalized to all Pillar II activities but are rather generated from the 15 case studies developed for this evaluation. Details and examples of findings are contained in the report and its annexes. With respect to the first question above (EQ1), the evaluation team observes that several factors contributed to successful outcomes in activities examined for this report: ■ Taking an evidence-based, participatory approach to activity design to ensure the needs of people are met and creating a common understanding of the priority of needs as well as ownership among all actors through a variety of mechanisms such as joint assessments; ■ In many cases, boosting ongoing projects with prevailing relationships and leveraging existing community or institutional structures and processes to enable quick and effective mobilization; ■ Addressing the interconnected parts of the health system (e.g., top/bottom and supply/demand of health system); this was especially relevant for different activities working together to address a common goal; b n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES ■ Putting processes and structures in place to build trust and relationships between implementers, their government counterparts, and communities during implementation (e.g., embedding staff in government offices); and ■ Taking different approaches for different contexts, especially in the case of governance activities, where national politics create different realities. For the second question (EQ2), the evaluation team examined the two main drivers of self-reliance—capacities and commitment—as articulated in the USAID journey to self-reliance framework. Nearly all the activities featured in the case studies sought to strengthen country capacity, to a small or large degree, in one or more areas, including: ■ Building workforce capacities and improving organizational processes to efficiently communicate, budget, procure, and deploy (government); ■ Strengthening media-related civic groups to ensure protection of civil liberties and to hold government accountable for resource allocation (civil society); ■ Enabling community members participation in health promotion or quality-of-care activities, to assure their engagement in their communities’ development solutions (citizen capacities); and ■ Reinforcing local businesses by injecting cash (rather than goods) into the local economy as well as enabling farmers to increase their harvests (economic capacities). The team found that the Pillar II activities examined leveraged the urgency for change resulting from the devastating effects of the epidemic to contribute to a foundation for enduring commitment, including: ■ Development and/or revision of policies and plans, technical standards, institutional arrangements, and/or clarified priorities, roles, and responsibilities in relation to key issues highlighted as shortcomings when EVD struck (policies, strategies, and plans); ■ In one case, a substantial increase in one government health expenditure rates (finance); and ■ Establishment of processes and structures in multiple sectors to improve accountability between government institutions and different sectors of society (mutual accountability). The evaluation also identified key challenges that prevented the full potential of some of these successes from coming to fruition, including workforce attrition, inadequate infrastructures, missed opportunities for private sector engagement, and the lack of long-term funding to sustain activities. Regardless, even though there is more to be done, the evaluation found that there has been significant progress to self-reliance with respect to withstanding future disasters in all three countries. Recommendations Based on findings from the 15 case studies in this report, the evaluation team offers the following recommendations for supporting partner countries’ response to emergencies while building self-reliance: 1. Build on and/or optimize ongoing activities and local structures and processes as program implementation “infrastructure” to expedite mobilization, ground recovery activities in local contextual understanding and leverage counterpart/partner trust and cooperation; 2. Adopt the successful Pillar II systems approach for designing activities that are complementary across sectors, geographical locations, and technical areas to maximize the potential to build sustained systems; 3. In collaboration with local and external stakeholders, develop a clear exit strategy for reducing program funding/ technical inputs and increasing long-term funding and local counterpart responsibilities that leads to sustainability and self-reliance; and 4. Create strategies to engage the local private sector in post￾crisis recovery efforts, as well as in national-level preparedness plans, to bridge implementation and infrastructure gaps often present during emergencies. INTRODUCTION n 1 INTRODUCTION Background 1. Source: USAID’s Office of Budget and Resource Management O&E spreadsheet, September 2017. The ramifications of the 2014–2016 Ebola virus disease (EVD) outbreak in West Africa extended far beyond the boundaries of public health. In addition to the tremendous strain on already weakened health systems in Guinea, Liberia, and Sierra Leone—the three countries affected the most by EVD— borders were closed, trade slowed, agricultural production suffered setbacks, schools were closed, and the population’s confidence in government response was diminished. The EVD outbreak highlighted existing institutional weaknesses in each of these countries and exposed their vulnerability to crises. While economic activity had begun to rebound and many official restrictions, including border closures, were lifted by early 2015, recovery was slowed by the depletion of household and business-sector assets, the interruption of large-scale investment plans, and the reservations of investors (United States Agency for International Development (USAID), 2015). To confront these cross-sectoral challenges, the United States Government (USG), with USAID as the lead agency, mounted a multidimensional response designed in collaboration with the governments of Guinea, Liberia, and Sierra Leone. It integrated its activities with support provided by other international partners, including the WHO, United Nations Mission for Ebola Emergency Response, World Bank, European Union (EU), the governments of the UK, Japan, Germany, China, and France, The Paul Allen Foundation, and The Bill & Melinda Gates Foundation. The response was organized in four pillars: halt the spread of the virus and provide care for those directly affected (Pillar I); mitigate the second-order social, economic, and political impacts of the epidemic (Pillar II); and strengthen partnerships for regional and global health security (Pillars III and IV) (Office of Inspector General, 2018). Pillar II Activities and Findings of Performance Evaluation 1 Pillar II activities, the focus of the present evaluation, centered on post-EVD recovery activities with the goal to ensure that countries build back economic and social systems better than before the outbreak. In this way, they become better prepared for other future disease outbreaks or crises. Pillar II aimed to: (Objective 1) prevent the loss of development gains; (Objective 2) recover and strengthen existing institutions and infrastructures; and (Objective 3) build sustainable systems through public-private partnerships (PPPs), innovation, and capacity building. In October 2016, USAID awarded International Business & Technical Consultants, Inc. (IBTCI) a three-year contract to conduct monitoring, evaluation, and learning (MEL) activities in support of Ebola Pillar II work. Between December 2017 and February 2018, IBTCI conducted the first of the two Performance Evaluations (PE1) (Moore et al, 2019). Range and breadth of activities: PE1 reported that the USG had obligated $1.866 billion to the response by 2017. Of this, $815 million went to Pillar I, $475 million to Pillar II, $29 million to Pillar III, $243 million to Pillar IV; the remaining funds were previously unprogrammed Economic Support Funds and diplomatic and consular programs. By September 2017, the USG had disbursed 46 percent of Pillar II obligations to fund 148 regional and country-specific activities. 1 These activities spanned a range of intervention types that included social protection; frontline worker support; management coordination and partnerships; information, communication, and technology; social and behavioral change communication; and institutional enhancements. While the majority of activities were in the health sector (60 percent) and accounted for more than a third of disbursements (39 percent), the greatest share of expenditure (50 percent) was in Agriculture and Food Security (AFS), largely in the form of vouchers and cash transfers. By contrast, Governance and Economic Crisis Mitigation, and Innovation, Communication, and Technology (ICT) accounted only for approximately ten percent of expenditure. Performance: As detailed in PE1 (Moore et al, 2019), Pillar II activities combined proven interventions for both short￾term relief and long-term development and provided a rare opportunity to observe support from multiple sectors directed toward common outcomes. Pillar II health activities helped to build trust in government, inspiring substantial gains in health￾seeking behaviors. Cash transfer activities allowed beneficiaries to purchase food and helped infuse money back into local economies. Assistance to market traders ensured that there were products available for purchase. In locations where Pillar II AFS activities were implemented, PE1 reported that standard measures of household food security improved considerably 2 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES in a relatively short period of time and helped alleviate hunger. Pillar II innovations have spurred partnerships between innovators and private-sector entities to bring effective innovations to the global marketplace. Sustainability: PE1 found that Pillar II activities helped lay the foundation for long-term sustainability of the outcomes described above. Health-sector activities focused on critical elements for a functional and quality health system including workforce, policy, infrastructure, financial management, and health communication. Governance activities helped build or reactivate structures and practices for an engaged citizenry and government accountability by strengthening government agencies, civil society organizations (CSOs), and the media. Finally, Pillar II activities supported decentralization of health services and a variety of community-level mechanisms, such as village savings and loan associations and community water governance that may build local resilience to future shocks. Gaps and Opportunities: While Pillar II activities helped restore and, in some cases, build stronger systems than existed prior to the outbreak, PE1 identified some missed opportunities, including interventions and inputs for agriculture and to address EVD-related stigma and discrimination that were late to launch. As the recovery effort continued, however, 2. Note that PE2 will only focus on activities selected for case studies and the evaluation question will only be answered for the selected activities. implementing partners (IPs) adapted to the various challenges through creative solutions. At the writing of the PE1 report, obstacles remained to some promising interventions, such as cash transfer programs using mobile money. In addition, the evaluation found that governance and economic issues underpinned all other systemic issues during such an outbreak—and the fact that they were poorly funded was a missed opportunity. Management: USAID set up an extensive coordination network with government and other development partners for the implementation of the Pillar II activities. PE1, which examined USAID’s emergency and long-term development mechanisms in a complex emergency, found that while funding earmarks and conditions complicated planning and implementation of Pillar II activities, USAID coordination worked well. The evaluation also noted that the commitment and passion of personnel at the agency was high, as well as that of MEL and program planning at the country level. Following the conclusion of PE1, USAID commissioned IBTCI to conduct a second Performance Evaluation (PE2) to inform the Agency’s future planning and decision-making when responding to shocks and/or serious disease outbreaks that have the potential for multi-sectoral impact (see Annex B). Second Performance Evaluation Objectives and Questions Using PE1 as a foundation, the primary purpose of PE2, conducted between December 2018 and June 2019, is to generate a nuanced understanding of the factors that contributed to the effectiveness and/or contribution to self-reliance of the Ebola Pillar II interventions. The primary audience for this evaluation is USAID staff involved in design and implementation of responses to such complex emergencies. The secondary audience is the development community that steps in to respond to such emergencies. The evaluation questions (EQs) for PE2 were presented in two domains—effectiveness and self-reliance (Table 1 below). Table 1. Evaluation questions for PE2 EQ1 Which Pillar II activities2 contributed to the achievement of one or more Pillar II’s three overarching objectives: (1) prevent the loss of development gains; (2) recover and strengthen existing institutions and infrastructure; and (3) build sustained systems through PPPs, innovation, and capacity building? 1(a). What were the factors that contributed to the effectiveness of stand-alone activities* and why (i.e. did/ would/could this activity succeed on its own merit or did it fail)? 1(b). What were the factors that contributed to the effectiveness of a combination of activities* within each sector and why? 1(c). What were the factors that contributed to the effectiveness of cross-sectoral activities* and why (i.e., how did cross-sectoral activities bolster the successful outcomes)? EQ2 What Pillar II activities facilitated new partnerships or strengthened existing partnerships (including leveraging the private sector) to advance partner country self-reliance? 2(a). What were the salient factors and/or determinants that positioned these activities to advance a country’s journey to self-reliance? 2(b). To what extent did these activities strengthen country systems to mitigate and respond to future epidemics? * Note: The reference to stand-alone and combination of activities within and across sectors refers to how activities may have worked on their own and in coordination with each other to achieve the same objectives. EVALUATION METHODOLOGY n 3 METHODOLOGY Case Study Design As the goal of this evaluation is an understanding of the factors (the “how” and “why”) leading to success (effectiveness and/ or self-reliance) within the full, real-life context in which the activities are implemented, a case study design was deemed appropriate (Yin, 1994). In order to explore the variety and range of ways these activities contribute to either effectiveness or self-reliance, IBTCI employed a multiple-case study approach to purposefully select cases that highlight different aspects of the Ebola Pillar II response. Per the U.S. Government Accountability Office (GAO) definitions, the individual case studies within this evaluation are conceived as explanatory, as their main aim is to examine the factors contributing to program effects. The overall report is then conceptualized as cumulative, as findings from the individual case studies are synthesized to answer the evaluation questions (GAO, 1990). The unit of analysis for the majority of the individual case studies are activities funded by USAID under Ebola Pillar II. USAID uses the term “activity” to identify project components that are being implemented by country partners or by other organizations that USAID has funded under a contract, cooperative agreement, grant, or other arrangement. These may have been grouped under one funding mechanism, as in the case of Fighting Ebola: A Grand Challenge for Development. The evaluation team selected cases to feature in this report if they had sufficient documentation and provided a strong learning opportunity. Selected case studies feature an activity that not only contributed to the achievement of the three Pillar II objectives, but also made progress towards achieving some or all of the 19 outcome indicators in the Pillar II Theory of Change (Moore et al, 2019). For more discussion on the selection of the case studies, refer to Annex C, Evaluation Methods. A total of 15 case studies were developed and are presented in full in Annex A. Data Sources IBTCI used both primary and secondary data collection to develop the case studies. ■ 104 key informant interviews (KIIs) with 138 persons from several stakeholder groups including government officials, other implementers, and actors from civil society and the private sector ■ 41 focus group discussions (FGDs) with 321 respondents with service providers (e.g., health workers, government officials) and beneficiaries of the selected USAID Ebola Pillar II projects (Table 2). ■ More than 800 activity documents (including quarterly and annual reports, monitoring and evaluation (M&E) plans and reports, baseline and endline studies), some collected during PE1 and others solicited especially from the IPs of the activities portrayed in the 15 case studies ■ 30 site visits to (1) service delivery sites, such as peripheral health units (PHUs) and (2) communities in catchment areas of Pillar II activities. The site visits served as an opportunity to interview both direct (e.g., reproductive, maternal, newborn, and child health (RMNCH) clients) and indirect (e.g., trained health workers) beneficiaries, as well as establish a fuller context of the case studies for the evaluators. Data collection activities, sampling procedures, as well as quality assurance procedures, are described in detail in Annex C (Evaluation Methods) and Annex E (List of KIIs, FGDs, and Sites). All primary data collection activities were guided by semi-structured interview guides and are presented in Annex D, Data Collection Tools. Figure 1. FGD Participants by Role Service Providers 32% Beneficiaries 68% 4 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES Data Analysis IBTCI used two approaches to analyze data: ■ Content Analysis to systematically code and categorize primary data for analysis to identify trends and patterns with respect to specific domains of interest for the evaluation (e.g., drivers of self-reliance) (Stemler, 2001) ■ Grounded Theory to identify new and emerging themes across all sources of data, as well as across case studies and countries (also called analytic induction) (Strauss and Corbin, 1997) IBTCI utilized a web-based qualitative data analysis software, Dedoose, to allow collaborative analysis across geographically dispersed teams of evaluators. A detailed description of the analysis process and methods including codebooks is found in Annex C, Evaluation Methods. Limitations The following are key limitations for PE2. ■ Limitations of the case study approach for conducting an evaluation should be noted. While case studies are designed to provide deeper understanding of a particular issue, findings do not generalize to all Pillar II activities. The method was chosen because it is appropriate for generating insights into what may have contributed to effectiveness. ■ Loss of institutional memory was a constant challenge during PE2. While it was important to conduct this study after a period of time has passed to judge the sustainability of these activities, by this time several activities had been closed and staff had moved. In some instances, government officials were no longer in their original post at the time of the study. Even if they were located, people often forgot key events and conditions, leading to recall bias. To mitigate against this, the evaluation team used “snowball” sampling to identify informative individuals and validated responses with a variety of other sources. ■ Inability of participants to distinguish between activities that may or may not have contributed to an outcome they observed is a result of the Pillar II design that coordinated complementing activities to support a specific goal. A further challenge to determining Pillar II contributions was that some activities were designed to layer onto ongoing programs. In addition, neither PE1 nor PE2 included an experimental design. For these several reasons, neither PE1 nor PE2 claim to attribute a certain outcome to a specific activity. In addition, the evaluation team aimed to develop a very clear and in-depth understanding of the actual activity to enable them to understand what outcomes could be plausibly be attributed to an activity. Photo by Harriet Mason for USAID EVALUATION FINDINGS n 5 FINDINGS Introduction In this report, IBTCI presents cross-cutting findings identified from across all 15 case studies. Below, IBTCI presents these findings organized per the main evaluation questions above, and then by sector/domain: ■ Effectiveness (EQ1): Health; Governance; Food Security and Agriculture; and Innovation, Communication, and Technology ■ Self-reliance (EQ2): Commitment; Capacity In each section, after the main findings synthesized across all the case studies are presented, IBTCI presents short summaries of selected case studies to better illustrate key points discussed in that section. Table 2 below provides the order of case studies summaries presented in the main report. Note that even though a case study summary is presented on one section, it likely speaks to findings in multiple sections of this report. In addition, the full case study reports (found in Annex A) describe more than the theme featured in the short summaries in this report. Table 2. List of Selected Case Studies Sector/ Dimension Country Case #/Featured Activity EQ1 Effectiveness: Factors that contributed to the achievement of Pillar II’s objectives Health Guinea 1. Systems for Improved Access to Pharmaceuticals and Services (SIAPS) Liberia 2. Collaborative Support for Health (CSH) Sierra Leone 3. SIAPS Regional 4. Restoration of Health Services (RHS) Governance Guinea 5. Consortium for Elections and Political Process Strengthening (CEPPS) Liberia 6. Civil Society and Media Leadership (CSML) Agriculture & Food Security Guinea 7. Emergency Access to Food for EVD-Affected Guineans Program Liberia 8. Economic Recovery from Ebola for Liberia (EREL) Sierra Leone 9. Food for Peace (FFP) Cash Transfer Innovation, Communication, & Technology Regional 10. Fighting Ebola: A Grand Challenge for Development Regional 11. mHero EQ2 Self-Reliance: Determinants in country’s journey to self-reliance Capacity Sierra Leone 12. Post-Ebola Recovery of Health Services (PERHS), International Organization of Migration (IOM)/Infection Prevention and Control (IPC) Regional 13. Health Communications Capacity Collaborative (HC3) Commitment Liberia 14. Partnership for Advancing Community Based Services (PACS) Guinea 15. Health Finance and Governance (HFG) 6 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES Effectiveness (EQ1): What factors contribute to the achievement of Pillar II’s objectives? Factors that have emerged as contributing to and/or detracting from the effectiveness of Pillar II activities appear in the sector in which they were most relevant or in the sector in which they were most observed across the case studies. HEALTH PE1 found that the health sector activities spanned a range of areas including health systems recovery (health worker training, health facility restoration, and updating of relevant laws, policies, and guidelines), emergency non-Ebola health services (such as reproductive, maternal and child health, immunizations, and behavior change communications), and survivor program activities. Pillar II activities helped restore health systems in each of these countries (Moore et al, 2019). In PE2, the following factors were identified as important to the success of activities examined for this report. ■ Responding to evidence-informed, felt needs positioned activities for success (see Case studies 1, 2, 3, 4, 11, and 14). In Liberia and Guinea, most activities examined for this report began with an assessment with full participation of local partners; this was seen as an indication that the local needs and views were given serious consideration, rather than IPs entering with a set, predetermined agenda (Cases 1, 2, and 4). In Sierra Leone, IPs had predefined mandates and focus; nonetheless, local partners expressed that the Pillar II health activities were participatory and demand-driven, indicating that the IPs’ plans corresponded to what the local stakeholders needed at the time of crisis (Case 3). ■ A disconnect between higher and lower levels of the health system was recognized and deliberately addressed by some activities (Cases 1, 3, 11, 12, 14, and 15). In Liberia, for instance, enabling the health ministry to be a source of timely management and technical information (not merely a paycheck) was made possible by mHero, which in turn built on efforts to update and digitize the ministry’s human resource and health information systems (see Case 11). In Guinea, when the HFG activity facilitated tours of health facilities by members of the National Assembly’s Health Commission, they were reportedly surprised by the extreme need on the ground. In Sierra Leone, health-sector IPs supported the Central Ministry of Health and Sanitation (MOHS) on discrete tasks while also assisting with local implementation and point-of-service issues at the district and health facility levels (Case 15). ■ Effective activities addressed both supply and demand sides of the universal health coverage equation. Multiple examples show that successful health activities worked on service provision while also addressing fears, attitudes, and awareness of the recipients (Cases 2, 4, 12, 13, and 14). Liberia PACS taught community health aids and volunteers the Education Through Listening (ETL) method of interacting with clients, at the community level and in clinics (see Case 14). In Sierra Leone, both PERHS and HC3 bridged the divide between communities and the providers or sites tasked with addressing their health needs (Case 12 and 13). ■ Embedding activity staff with government for long periods that stretched into years led to trust by local partners in both Guinea and Liberia (Cases 1, 2, 3, 4, 11, 14, and 15). Three elements contributed to this: willingness on the part Photo by Joshua Yospyn for JSI EVALUATION FINDINGS n 7 of individuals to work together; complementary technical skills; and having the right processes and structures to enable the embedding of external staff and ensure collaborative work (e.g., joint assessment, planning). In fact, some KIs contrasted USAID’s policy of limiting postings to two years with the Centers for Disease Control and Prevention (CDC)’s policy that permits six years and argued that the longer posting is more effective for building internal commitment among staff and supporting organizational changes. This practice has contributed to continuity of Ministry staff even if the IP left. In Sierra Leone, even without a formal embedment arrangement, KIs noted that due to trust and rapport that existed between MSH and MOHS, the SIAPS activity transitioned toward a co￾location arrangement within the MOHS over the course of implementation. ■ Finally, where possible, successful health activities (those that achieved their objectives) were implemented by building onto on-going activities (e.g. USG-funded projects) or existing community or institutional structures (e.g., health boards, facility management committees, existing information technology systems). They avoided the inertia of starting something new, leveraged well-earned reputations and relationships and enhanced community acceptance (Cases 1, 2, 4, 8, 11, 12, 13, and 14). For instance, in Sierra Leone, AFS activities built on FFP projects and SIAPS built on a pharmaceutical management project, enabling both to launch quickly and efficiently. In Liberia, PACS activated community health committees that already existed on paper, providing guidelines and logistical support to elected community members, who then transmitted health information from the clinic and passed along health events and concerns to their health facility. IPs’ commitment to coordinate with other implementers in instances when there was either geographical convergence or potential for technical overlap in Pillar II support contributed to the success of multiple activities in the health sector. In Guinea, Abt Associates, MSH, and Jhpiego worked together to create a system for data management that allows needs￾based planning at all levels and budgeting at national level (Cases 1, 4, and 15). In Sierra Leone, PERHS and HC3—both of which were tasked with contributing to the restoration of health services in the same target districts—worked in concert to ensure complementarity of effort based on their respective institutional strengths (e.g., community engagement and demand generation on the part of HC3 and health facility rehabilitation, capacity-building for health care staff and procurement of minor medical equipment, on the part of PERHS) (Cases 12 and 13). In some cases, USAID played an important role in this coordination, either by engaging with IPs to clearly define current status and plan interventions or facilitating working groups (Case 9). However, KIs did also raise concerns of either stove-piping within the missions (e.g., health team not interacting with governance) or lack of coordination between USAID, EU, and UN funded electoral observation missions. Furthermore, health programming that employed non￾traditional (according to health sector actors) cross-sectoral partners saw significant added benefits. There has been a realization in all three countries that community radio plays an integral part to the success of the health system. Radio was used for disseminating accurate and educational information on a variety of issues, including stigma and behavior change, as well as facilitating community participation (Case 6 and 13). Some Pillar II activities also engaged private sector partners for other improvements. For example, in Sierra Leone, PERHS’ work to rehabilitate some health facilities entailed contracting three private drilling firms to improve WASH infrastructure (Case 12). Finally, the following factors were seen as challenges to the sustained success of activities in the health sector. ■ Inconsistent and insufficient supplies of essential drugs is a critical factor challenging the success of health activities. In Liberia, KIs identified lack of essential commodities and drugs as one of the biggest barriers to quality health services and to the success of the newly instituted community health services strategy. In Guinea, SIAPS improved the pharmaceutical management system, allowing the accurate tracking of drug supplies, but stock-outs occurred as funding for essential drugs was insufficient (Case 1). ■ Insufficient funding has had other major implications for sustainability of outcomes. In Guinea, while health facilities were improved, they are not able to keep up with the increasing demand due to population growth (Case 4). Under-budgeting for logistics (recurrent costs, according to USAID terminology) has led to serious challenges to implementation and monitoring of activities. For example, health facilities that the evaluation team visited in Guinea had generators—but no money for fuel to run them. ■ Disparities due to social cohesion, urban or rural location, and the nature of the activity in its setting may affect activity success. This was particularly noted in Sierra Leone, where lower social cohesion was observed in urban than in rural areas targeted by HC3 and PERHS (Cases 12 and 13). Thus, in urban locations, activities that rely upon “community dialogue” for maximum impact may not be as effective as expected. Rural settings may benefit more from community aspects, but have more basic challenges, such as geographic accessibility (i.e., proximity to cities and towns, quality of roads). A primary reason given for disparities in outcomes between high- and low-performing beneficiary areas in rural Liberia was transport; in Guinea, some KIs perceived that more improvements were made in Conakry than outside. 8 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES Case Study 1. SIAPS: Developing relationships in Guinea to restore and expand the capability of the public health supply chain Before the start of the MSH SIAPS activity in Guinea, prolonged stock-outs of critical drugs were common. While a shortage of products in the country was one problem, insufficient logistics data (e.g., stock on hand, consumption) was another, negatively impacting the country’s ability to develop forecast accurately (SIAPS, 2017). The EVD crisis compounded these problems (SIAPS, 2018a) and a government KI reported that this presented a challenge to effective distribution of the required EVD medicines and materials. SIAPS was able to contribute to a number of outcomes that were central to these challenges, including the integration of the supply chain around the Central Pharmacy of Guinea, the creation of a logistics management unit, and the standardization of the Laboratory Management Information System (LMIS), which allows essential pharmaceutical data to be collected and digitized (SIAPS, 2017). Availability of monthly updates on drugs and commodities used and remaining stocks have allowed Guinea’s pharmaceutical supply chain to move to needs-based procurement planning and to identify and reallocate surpluses to avoid waste and address unforeseen crises (Republique de Guinée, 2016). Additionally, government KIs reported that capacity building among relevant stakeholders allowed them to learn and use the electronic LMIS. “ The spirit of SIAPS has enabled change – the project and its staff have quickly and well understood the local health system and have well adapted to working with it.” —Guinea KII, 2019 Several factors contributed to the activity’s effectiveness, many of which center around relationships and team building. Government KIs reported that the SIAPS team worked with local stakeholders to create a common vision toward the need to computerize the system. There was a culture of open discussion which motivated people to address the diverse needs among stakeholders and encouraged agreements and common understanding. Embedding staff within the Guinean health system helped government stakeholders recognize the attitudes and ability of the SIAPS team to understand the local health system and to contribute to the success of existing structures. KIs also valued the integration of the role of the central pharmacy into procedural systems on the regional level. “ The technical assistants who came to support us had the competencies we had wanted and required. They had the attitudes and commitment to work with us. It was a real work in partnership. They did not come to do the work in our place but work with us.” —Guinea KII, 2019 SIAPS and partners developed and implemented a comprehensive training package involving all relevant levels of the health system. KIs reported wide disparities (10 to 100 percent) among prefectures respecting reporting data. SIAPS succeeded in transforming the weakest-performing prefectures into the highest achieving. Some successful strategies reportedly included allocating time during monthly meetings for peer support at the district level (one KII respondent) and creating a culture of friendly competition among peers around reporting and discussing results (health provider FGD respondents). Case Study 2. CSH: Strengthening health systems in Liberia through decentralization and coordination The health system of Liberia was drastically affected first by 14 years of civil war (Ellis, 2006; Government of Liberia (GOL), 2015a; GOL, 2015b) and then by the EVD outbreak in 2014 and 2015. The Collaborative Support for Health (CSH) activity, a consortium led by MSH beginning in 2015, was built on MSH’s previous work in Liberia and designed to strengthen Liberia’s health system in the counties of Bong, Lofa, and Nimba. After the activity’s launch, Pillar II funding was used to expand CSH’s scope to add Grand Bassa, Margibi, and Montserrado, for a total of six counties (MSH CSH FY1/Q1 Report, 2018). KIIs and FGD respondents noted that when CSH began, there were fragmented and parallel information systems of dubious accuracy. CSH advisors worked with Ministry of Health (MOH) leadership to gather basic information that was relevant across units. CSH’s decision (with MOH support) to prioritize interoperability of systems and build on existing systems, rather than creating something new, boosted the project’s effectiveness and long-term impact. KIs contrasted the systems that CSH supported (and the mHero SMS-based communication system that used its information) with other, more complicated technological proposals, arguing that building on existing MOH systems contributes to both Ministry buy-in and the ability to be managed by Ministry counterparts. These factors both improved system performance, reporting, and contributed to the sustainability of these system upgrades. EVALUATION FINDINGS n 9 Assessments conducted at the start of CSH reported that country health teams (CHTs) were disconnected from the central MOH—yet had little autonomy to identify and respond to local health issues (MSH, 2015). Frequent drug stock￾outs had damaged the credibility of public health facilities and weaknesses in facility labs led to delays in diagnosis and treatment. CSH posted mentors in human resources, quality assurance, M&E, and supply chain management in each CHT. These CSH mentors shared office space and expertise while accompanying their CHT counterparts in their day-to-day work. In addition, they updated and rolled out a process to support and monitor community health service providers. CSH trained quality assurance staff to assess facilities on compliance with the Expanded Package of Health Services (EPHS) requirements through quarterly site visits; those were linked with updated guidelines and training on the paper-based and, later, the electronic Laboratory Management Information System (LMIS and eLMIS). Liberian MOH KIs credit CSH for moving the Ministry’s decentralization policy from principle to action in all six counties in which it worked. For instance, in Grand Bassa, Margibi, and Montserrado, the proportion of health facilities providing timely LMIS reports increased from 23 to 100 percent, from 50 to 86 percent, and from 51 to 100 percent, respectively (MSH CSH FY1/Q1 Report, 2018). KIs noted that a number of factors influenced CSH’s effectiveness, including a focus on management, embedding mentors in health system units as a long-term approach to building capacity, and a strong motivation on the part of health system actors in the wake of EVD. CSH provided support to harmonize several key health information systems so they could work together and allow for stronger data for decision-making, as well as updating the national health facility inventory which was crucial to the District Health Information System 2 (DHIS2). KIs reported that CSH’s MOH counterparts and other GOL actors involved in CSH’s areas of action (principally the Department of Public Works and Sanitation for WASH activities) were reportedly eager to collaborate. The activity also supported operational roles necessary to provide health officials with timelier workforce, supply-chain, financing, and service-delivery information needed for efficient management and quality improvement. CSH also allocated time and effort in consultation with the CHTs and with the districts and communities they served. KIIs and FGDs reported that CSH helped to activate County Health Boards in all six counties and produced a written manual that specified the board’s role and its members’ responsibilities, as well as those of the health care providers and the community members (e.g. chiefs, faith leaders, women, youth) that should be included. At the central level, CSH advisors assisted the MOH in establishing policies and guidelines and activating management tools. Reports from KIIs indicate that these products are still in place a year after the CSH project was closed. Many of CSH’s counterpart staff in the relevant units are still in place, utilizing the knowledge and skills developed in partnership with CSH advisors and mentors. Case Study 3. SIAPS: A comprehensive systems approach in Sierra Leone leads to quick gains in pharmaceutical management Prior to the EVD outbreak, health care in Sierra Leone was free for mothers and children, but fraught with challenges (e.g., poor drug availability in health facilities, low/inconsistent availability of electricity and water) (Witter, 2018). In 2015, USAID awarded MSH SIAPS, a global program that did not operate in Sierra Leone prior to the EVD outbreak, Pillar II funds to address pharmaceutical management (SIAPS, 2018c). Under the SIAPS Pillar II activity, MSH worked closely with the Ministry of Health and Sanitation (MOHS) Directorate of Drugs and Medical Supplies (DDMS), Pharmacy Board of Sierra Leone (PBSL), National Pharmaceutical Procurement Unit District Health Management Teams, and other stakeholders. The activity, which spanned all districts in Sierra Leone, over 1,300 PHUs and 24 hospitals, entailed a multi-pronged approach to: (1) strengthen DDMS ability to effectively support health facilities, (2) strengthen “last-mile” supply-chain management in districts, and (3) increase the use of information for supply decisions (SIAPS, 2018b). In a little more than two years, SIAPS helped the MOHS to create institutional structures to support effective pharmaceutical management, cultivate leadership skills among pharmacists, and strengthen the MOHS’s ability to keep quality drugs in stock and both identify and remove expired products from their shelves. As shown in Figure 2, there were major improvements in the rate of stock-outs and in the proportion of facilities remitting expired products (SIAPS, 2018b). 32% Facilities with stock-outs of MNCH drugs and commodities 66% Facilities that remitted expired products 37% 72% baseline endline Figure 2. Selected Outcomes from SIAPS 10 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES Both Pillar II implementers and Government of Sierra Leone (GoSL) counterparts noted that MSH’s deployment of respected pharmaceutical experts with local contextual knowledge of pharmaceutical management and past working relationships in Sierra Leone proved to be critical success factors in activity performance. Under the Pillar II activity, SIAPS experts drew upon its constellation of globally tested, state￾of-the-art approaches and tools related to pharmaceutical management (e.g., Treatment Register, Pharmaceutical Leadership Development Program, an evidence-driven Continuous Monitoring and Support System for supportive supervision and performance improvement) (SIAPS, 2018b and c). In reflecting on factors in the activity’s effectiveness, one KI involved in implementation underscored that the attention paid to relationships—both within the MOHS and with other stakeholders in the country’s pharmaceutical/supply-chain management landscape (e.g., UNICEF, Global Fund)—enabled SIAPS to foster buy-in and fast-track implementation of key interventions, as well as spur organizational development changes (e.g., the creation of an autonomous National Medical Supplies Agency). Case Study 4. Restoration of health services in Guinea and Liberia (Regional) Regained confidence in health services and increased quality of care have been major achievements of the Maternal Child Survival Program (MCSP) Restoration of Health Services (RHS) project in Guinea and Liberia. The EVD outbreak had a drastic impact on health service utilization and quality. Communities lost trust in the health system; delivery care (maternal and newborn health, family planning) dropped 81 percent in N’Zérékoré (Guinea) and 74 percent in Conakry within a period of 15 months. Further, health center reporting rates to districts (e.g. on stocks) had declined to only 26 percent from the health facilities (Jhpiego, 2017). In Liberia, only 44 percent of the country’s health facilities remained functioning and most essential primary care services had come to a standstill while staffing declined (Jhpiego, 2018). In both countries, the RHS projects targeted the areas most affected by EVD, covering 20 percent of the counties in Liberia and 66 percent of the population in Guinea (Jhpiego, 2018; Jhpiego 2019). Jhpiego’s MCSP had been already operating with a number of projects in Liberia and Guinea and were asked to develop an additional project to address the restoration of critical non￾Ebola health services. In both countries, upgrading of infection prevention and control (IPC) practices and restoration of RMNCH services were core objectives. The RHS project operated for 37 months in Liberia and only 18 months in Guinea. Throughout the evaluation team’s time in Guinea, improvements to the availability and quality of services in the health system were prominently mentioned in FGDs among community members, local stakeholders, and local health staff. In general, at the end of the RHS programs, there were improvements seen at most supported facilities. In Liberia, 74 of 76 supported facilities reported improvements in the availability of health workers, equipment and supplies, basic infrastructure, service provision for essential RMNCH interventions, and adherence to clinical standards to ensure quality of care (Jhpiego, 2018). A critical strategy in both countries contributing to the success of RHS was the leveraging of existing programs, (e.g., in Guinea USAID investments in IPC, the flagship Maternal and Child Health Integrated Program (MCHIP)’s strengthening of RMNCH, as well as the StopPalu malaria project). In fact, the RHS project was so well-integrated that respondents in communities, health centers, or divisions of the MOH found it difficult to distinguish from other complementary activities or determine the project’s specific impact. In both countries, baseline assessments were carried out to understand the situation and adapt the project to the specific needs of the health facilities. Staff KIs from the local health centers in Guinea stated that they valued Jhpiego staff coming to their facilities, listening, and addressing their specific needs throughout the project. A major contribution to the success of the two activities was reported to be the close integration with other Pillar II activities, especially with HC3 and SIAPS in Guinea. HC3 has been vital in reviving and/or establishing community committees to allow health centers to better engage and communicate with communities, receive feedback, and ensure participation. A challenge to the sustainability of these achievements was noted at a 2019 health center KI in Guinea: successes are starting to be “watered down” because of a steadily increasing population. Neither the infrastructure nor the available staff, drugs, or materials are sufficient to meet the needs of this growing population. In order to sustain the achieved quality improvements, continued investments are required to meet increasing demands. KIs reported a continued lack of funding for essential drugs and materials limiting services, despite a new computerized management system in place which allows an optimal use of the available goods. A further challenge is the lack of funding for meeting recurrent costs (including transport) or maintenance costs for infrastructure and technical equipment. One example of a past investment that has not been sustained at a health center in Guinea was a generator, which was still in place but could not be operated due to the lack of funding for fuel. During one site visit, it was reported that the RHS project ensured a limited availability of electricity, allowing the operation of the most critical RMNCH services by installing solar panels specifically dedicated to these services. While overall significant progress in the restoration of health services was achieved in both countries, short- and long-term success depends on factors that are beyond the scope of this project: health funding and staffing. EVALUATION FINDINGS n 11 GOVERNANCE Governance and economic crisis mitigation (ECM) activities in Pillar II were intended to support Objective 1 and Objective 3. Governance activities were primarily implemented in Guinea and Liberia and PE1 found that this sector received the least amount of funding relative to other sectors—and that despite a lack of widespread evidence of improved outcomes in trust and accountability, there have been important successes in strengthened government agencies, CSOs, the media, gender inclusiveness, and the preservation of peace (Moore et al, 2019). In both Liberia and Guinea, PE2 finds that Pillar II funding was used to create platforms enabling communication that had not happened previously (Cases 5, 6, 7, 12, 13, and 14). In Liberia, local leaders were trained and supported to convene their communities to share and validate MOH recommendations for control of EVD in order to combat stigmatization of survivors and their families and to account for the uses of EVD funds and materials. As in the health sector, successes in the governance sector depended largely on building trust with key stakeholders. Building trust in governance, however, is strongly context-dependent. For example, Liberian communities had trust in their chiefs and local leaders, but years of civil war had left a legacy of distrust for the national government and its programs. Therefore, if IPs approached local leaders (such as clan chiefs) in the correct manner, these leaders in turn were successful in getting people to support the coming intervention (Cases 6 and 14). Ethnic dynamics in governance presented a different scenario in Guinea. In order for activities to succeed, Guinea IPs recognized the importance of getting buy in from all political parties, most of which had clear ethnic or regional biases, before they could move forward strengthening local structures for governance (see Case 5). Improvements in governance are closely linked to improvements in other sectors. For example, in the health sector the creation of an improved system for tracking pharmaceuticals may enhance transparency, and therefore better governance—it is harder to “lose” drugs within the system. In Sierra Leone, activities to strengthen health systems (1, 3, 12, and, to a lesser extent, 13) centered on relationship-building, accountability and trust between different actors within and across the system. In Sierra Leone and Liberia, defining expectations and enhancing transparency and accountability with respect to official and professional roles—and defining responsibilities and procedures at every level (central, county, district, community/village)—improved the functioning of local governance mechanisms (e.g., committees, boards, meetings). These cross-cutting themes affect governance success and are essential for community mobilization and engagement not only in health arenas, but for any emergency response. A key struggle in governance was achieving the right balance between active supervision/oversight and empowerment. In Liberia, some MOH respondents reported concerns that if the supervisors could not get to the communities, due to bad road conditions or lack of funds for fuel, a community worker might just stay home and make up the data to fill in his or her reporting forms. This underscores the need to successfully communicate the value of the procedures, or the data, to the worker, and to present supervision as genuinely supportive. The danger of losing this emphasis is that people do not feel ownership or commitment to their work, resulting in disempowerment. Carol Sahley for USAID Photo by UNMEER 12 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES Case Study 5. CEPPS: Working to build smoother electoral and democratic processes in Guinea 3. CEPPS, Quarterly Report Q2 2018 reports that 24% of the candidates for the communal elections were female and 13% were under the age of 35. Data from the last communal election in 2005 was not provided. Tensions, violence, and delays throughout election processes have plagued Guinea since its return to civilian rule in 2010. Decades of authoritarian leadership were ended by a military coup in 2008. The National Democratic Institute (NDI)’s Consortium for Elections and Political Process Strengthening (CEPPS) was challenged to stabilize Guinea’s fragile political context, a critical precondition for the success of the post￾EVD recovery efforts in the country. There are currently more than 130 registered parties, most of which have clear ethnic or regional bases (Freedom House, 2019). It was feared that a repetition of the severe violence before the 2013 parliamentary elections during the 2015 presidential and long￾awaited local elections could slow or hinder the post-EVD efforts aiming at re-establishing critical citizen services (CEPPS, 2016). USAID’s Ebola Pillar II funds boosted NDI’s ongoing governance work of promoting political dialogue and consensus, strengthening the political party system, expanding civic education, enhancing citizen engagement, and oversight, and strengthening the independent electoral commission (CENI). NDI has worked in Guinea since 1998, training political party poll watchers and conducting voter education prior to the presidential election of that year (NDI, n.d.). Since the country began its democratic transition in 2008, NDI has promoted dialogue among political parties and supported the peaceful conduct of elections. Key activities of NDI have been the formation and strengthening of an Inter-Party Working Group and the development and subsequent revision of a code of conduct (Code de Bonne Conduite) which has since been adopted by 109 political parties, supported by a monitoring committee ensuring adherence to the code The effectiveness of CEPPS has been achieved largely due to the trust and wide recognition enjoyed by NDI throughout the country and the participation of key stakeholders, such as political party or civil society actors, NDI has further supported the CENI and strengthened local CSOs in monitoring elections and engaging in voter education (NDI, n.d.) as well as capacity building of local institutions and citizens to better manage their own democracy. According to a KI with a party member, CEPPS assisted parties in revising the code, the bylaws of the monitoring committee, and increasing regional party awareness of parties of the code. While initially the hope was to cover all prefectures, funds permitted coverage of only two prefectures per region, including full coverage in Conakry, according to project staff. This collaboration with CEPPS has reportedly also helped to address tensions and violent incidents post-election, taking a key role in involving critical actors to diffuse tensions. KIIs and FGD participants reported that NDI is highly regarded and accepted by all political sides in the country, allowing it to play a primary role in bringing opposing parties together, diffusing tensions, and ensuring progress in the development of democratic institutions and processes. NDI has observed that the threat of EVD has enabled different parties to unite in order to address a specific issue. Representatives of the major political parties expressed that the involvement of key stakeholders in CEPPS and its high-quality training has also been important for its success. According to one KI, NDI funding and staff limitations were the most relevant challenges to achieving more change; they have been unable to fully support the development of inter￾party committees at the prefecture level and have not been able to facilitate all required coaching and follow-up to ensure the intended capacity building. Similarly, a lack of resources was stated by several KIs, as the major challenge to sustainability, leaving changes only partially achieved; resources are lacking for critical follow-up, coaching, and especially to ensure that costs are met for meetings and visits. CEPPS supported women and youth political participation associations, helping both groups to take on leadership roles in their parties and as candidates in elections (CEPPS, 2018). This has resulted in an increase of youth and female candidates in the communal elections in February 2018, and a higher share of women elected.3 One KI noted that additional efforts are required to meet the actual target of 30 percent female representation and ensure that political parties hold to their commitments. EVALUATION FINDINGS n 13 Case Study 6. CSML: Strengthening radio and civic engagement to enable inclusive community mobilization in Liberia USAID’s Pillar II investment in the IREX/Civil Society and Media Leadership (CSML) activity was brief (five months), but it made a valuable contribution to Liberia’s recovery from the EVD crisis. The project built on a decade of prior investment and accomplishments strengthening two stakeholder groups that were crucial to a sustained and constructive response and recovery—CSOs and the media. CSML’s contribution to the EVD recovery began before they received Pillar II funds; the project had been working for four and a half years in two areas: to strengthen CSOs through training, mentoring, and small grants; and to strengthen the professionalism and reach of Liberian media and their inclusiveness and relevance to civic engagement and participation. When the EVD crisis erupted, CSML worked with the MOH, UNICEF, and IPs including JHU CCP HC3 and Jhpiego MCSP to get accurate information disseminated in six languages on EVD prevention and care and to combat stigmatization of EVD survivors. Simultaneously, the civil society team worked with the 62 CSOs in the CSML family to inform leaders in their communities on facts about EVD, to support them in convening their communities to repeat and validate the MOH’s guidelines, and to bring people on board with the EVD response (IREX, 2016). Five months before the end of the project, USAID’s Democracy, Human Rights and Governance (DRG) Strategy provided $2.2 million in Pillar II funds to IREX to enable CSML to expand its efforts toward the EVD recovery (IREX, 2016) and expand their work to all 15 counties. Leaders in 57 communities were supported by the civil society team to convene dialog sessions with their people to listen to their concerns. Complaints about corruption were escalating and CSML’s media team trained 99 journalists on how to research and report professionally on the flows and impact of EVD funds. CSML CSOs and journalists produced an “Ebola Situation Room,” a fraud hotline, and a public website where information on EVD funds—their uses and results—were synthesized and reported to the public (IREX, 2016). A number of factors appear to have contributed to CSML’s effectiveness. As noted previously in this report, KIs from IPs and civil society reported that 14 years of conflict had left deep suspicions of government at the national level, whereas people continued to trust and respond to their local leaders. By design, CSML worked with and strengthened traditional leadership structures at the community level. Grounded in CSML’s original focus on peaceful civic engagement, and on IREX’s preceding project, Search for Common Ground (SFCG), the project built the capacity of CSOs and journalists in constructive advocacy based on evidence gained through primary reporting in affected locales, where local voices could be heard. CSML had a cross-cutting focus on gender equity and human rights. Both the civil society and media components encouraged and increased participation of women (IREX, 2015; 2016) and they strove to include voices of vulnerable groups, according to KIIs with CSOs and former CSML staff. Through CSML and its predecessor, USAID invested in better governance through a “bottom up” approach. CSML created or strengthened platforms for new kinds of dialog between citizens and their leaders, enhancing accountability at the community level (e.g., Leadership Development Forums), at the district level (e.g., interactive Q&A programs on community radio) and nationally (e.g., the fraud hotline). CSML expanded the space on the national stage for the female half of the population, training and hiring women to report the news as well as to feature women’s diverse interests and concerns. They built the capacity of many of their CSO partners to win and manage donor funds, so that they would be sustained after the project’s closing. The evaluation team observed that, by building on a decade of investment in Liberia’s media network, in just five months CSML was able to help both the media and local government to fulfil their respective and linked roles in recovering from the EVD epidemic and strengthening good governance. Dara Lipton for IREX 14 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES AGRICULTURE AND FOOD SECURITY Pillar II agriculture and food security (AFS) activities aimed to support Objective 1 and Objective 2. PE1 found that these activities, receiving the largest allocation of Pillar II funding, were implemented in all countries but with the majority going to Liberia, followed by Sierra Leone. Activities in this sector included cash transfers, vouchers, and direct food support, as well as agricultural inputs, training, and small grants for traders and women’s groups to promote local markets. PE1 found that these activities supported beneficiaries with basic necessities during the EVD outbreak, but also helped restart local markets (Moore et al, 2019). Cash transfers and vouchers were effective as transitional interventions and influenced the success of other cross￾sectoral interventions. In Liberia, beneficiaries stated that cash transfers were transformational in that they came at a time when they had lost hope, and led to heightened food security in that farming inputs (e.g., seeds) were provided with the cash transfer. In Sierra Leone, Pillar II implementers of cash transfer programs were deliberate in packaging cash infusions with other interventions (e.g., health education, savings groups, farmers groups, agricultural inputs such as seeds or seed vouchers) to build the resilience of households and communities after EVD. On the other hand, in Guinea, two months after voucher distribution was completed, food security was already decreasing; beneficiaries reported that they were provided with seeds but did not have sufficient pesticides and thus lost much of their crop. One of the key factors named for success in cash transfers/ vouchers was the variety of ways that IPs tried to build trust and to avoid conflict with communities. For instance, IPs in all three Pillar II countries actively involved the community in a very transparent beneficiary-selection process whereby the community had control over who was determined a beneficiary. In addition, in both Guinea and Liberia IPs conducted needs assessments for the needed agricultural tools as well as vetted the quality and reliability of vendors that provided commodities (such as seeds), helping build community confidence in the process. Food security interventions provided excellent examples of various activities working together towards a common objective (multiple activities, one sector). The excellent coordination across IPs working under FFP was a contributing factor in the success of this effort. In Liberia and Guinea, IPs had a strong, regional community of practice that helped them respond efficiently after a humanitarian emergency. In Sierra Leone, the FFP Pillar II implementers formed a Cash Transfer Working Group, chaired by ACDI VOCA, to harmonize efforts (e.g., amount of cash transfer, numbers of cash transfers) and share learning during implementation. Conversely, inadequate infrastructure in other sectors contributed to implementation challenges, requiring adaptive management by IPs. For instance, in Liberia, where cash transfers for the most part were successfully executed, Mercy Corps had intended to provide 25 percent of its cash transfers using mobile money, but was only able to do so in one affected section of Monrovia where beneficiaries were willing to trust e-payment due to lack of electricity, cellular networks, and telecom outlets to redeem the cash transfer (see Case 8). In Sierra Leone, there were implementation challenges such as poor mobile network connectivity, especially in some remote areas. In instances when it was not feasible to implement mobile money transfers, however, IPs rapidly leveraged private sector partners such as Splash Mobile Money Ltd. to transport and disburse physical cash. Photo by Elie Gardner for CRS EVALUATION FINDINGS n 15 Case Study 7. Emergency access to food security resources for vulnerable Guinean households The combination of food voucher distribution, social behavior change communication (SBCC) sessions in nutrition, and livelihood fairs were at the core of the Ebola Pillar II funded “Emergency Access to Food for EVD-Affected Guineans Program”. The activity was implemented by Catholic Relief Services (CRS) and its local partner OCPH (Organisation Catholique pour la Promotion Humaine/Caritas Guinea) to increase short and long-term access to food for vulnerable households in EVD-affected areas (CRS, 2017). EVD affected all of Guinea, but the Forest Region was hit particularly hard. The two prefectures of N’Zérékoré and Macenta accounted for 21 percent of confirmed cases and 29 percent of the country’s total deaths (612 of 2,083) (Ebola Sitrep, 2015), while representing less than seven percent of the total Guinean population (Institut National de la Statistique, 2014). In 2015, both prefectures were considered to be stressed in terms of food security, still had EVD cases, and suffered the greatest crop decline since the outbreak began (FEWS NET, 2014). Households experienced deterioration in both food availability and their revenue. The program supported 7,008 households (35,040 beneficiaries) in these two prefectures, reducing food insecurity among them from 95 to 57 percent while households with adequate dietary diversity increased from 68 to 82 percent (CRS, 2017). Beneficiary FGD respondents reported that the support enabled them to sustain their families in terms of food and basic needs. According to beneficiaries, a major achievement has been that supported households have returned to more stable lives, which has been made possible through the combination of CRS and many other post-EVD activities in the target area (e.g., agricultural training, distribution of fertilizer and seeds, free medical treatment). CRS was able to make the most impact at the level of individual households and, to a limited extent, at the community or regional level via the injection of cash to the local economy through vouchers. The salient factors that contributed to the effectiveness of the CRS activity were the support and involvement of both the local government and the communities and a sustained focus on making sure beneficiary selection and financial management were fair and accountable. Both KII and FGD respondents stated that the availability and support of local authorities was critical, including their presence and involvement with communities and beneficiaries, traders, and vendors, helping to establish a relationship of trust. One KI stated that for the livelihood fairs, the program worked with the agricultural directorates, receiving their support in food certificates. Local authorities were part of the prefectural and local committees (CRS, 2017). Engaging the community and establishing a local committee were also key factors for success, according to both implementing staff and beneficiary FGD respondents. A first critical step has been to help communities to understand the program and its specific target households. Committees were formed at prefectural, sub-prefectural (neighborhood or village) level in each of the 22 villages or areas targeted. At both the prefectural level and the village level these focal groups included two representatives from families affected by EVD, religious leaders, and other representatives of the government and the community. These committees were not only critical in communication to communities but were also involved throughout the life of the program. A first important role was the validation of beneficiary selection criteria and procedures. In defining criteria, and later validation of beneficiary lists, committees played a critical role in ensuring that only the most vulnerable EVD-affected households were selected (CRS, 2017; Moore et al, 2019). Local committees voluntarily assisted with participant verification and crowd control for the two fairs. This collaboration was seen as particularly successful: 91 percent of beneficiaries were satisfied with the site selection and the organization of the fairs (CRS 2017). Case Study 8. EREL: Cash transfers and growing trust build food security in Liberia During the EVD outbreak in Liberia, villages were quarantined, families were affected by the disease, and many people were unable to continue their economic activities, including farming. FGD respondents in rural Montserrado and Lofa counties reported that many households were suffering from severe hunger and malnutrition. Mercy Corps, with its partners, implemented the Economic Recovery from Ebola for Liberia (EREL), one of five activities funded by USAID FFP with Pillar II resources in Liberia. EREL had two components: (1) an unconditional cash transfer component, for households with the most vulnerable EVD survivors and victims, including disabled people, the elderly, widows, and orphans; and (2) a farming training program using the Farmer Field Program, whereby “lead farmers” selected by a group of farmers are trained—then, using demonstration plots, they in turn teach their group what they have learned. EREL also provided farmers with vouchers for use toward the purchase of agricultural inputs from qualified vendors (Mercy Corps, 2016). Over the 24-month duration of the activity, Mercy Corps’ participatory approach resulted in positive outcomes. To select beneficiary households for cash transfers, EREL worked with community leaders and members to identify the most vulnerable. Community involvement also reportedly helped safeguard staff and beneficiaries from robbery on payment days and reduced fraud and corruption. Even standing in line 16 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES was an interactive process; beneficiaries were taught about nutrition while waiting in line for payment. (IDA, n.d). Mercy Corps upgraded its computer system for the registration of beneficiaries and monitoring of payments and vouchers; beneficiaries received photo ID cards and vouchers were printed abroad to reduce the risk of fraud. A high percentage of beneficiaries were women: 64 percent of the head-of￾beneficiary households, 24 percent of lead farmers. Mercy Corps explained that it made special efforts to address gender (Mercy Corps, 2017). Repeated post-distribution monitoring surveys showed that 74 percent of cash transfers were spent on food (Mercy Corps, 2015; Mercy Corps, 2016; Mercy Corps, 2017). Results of EREL monitoring indicate that supported households have increased diversity in their food diet and that there are reductions in hunger. These results were confirmed by a 2017 IBTCI population-based household survey (conducted for PE1), which found that sampled households in the three counties reported “little to no hunger” and continued improvement in dietary diversity (Moore et al, 2019). One challenge noted was that Mercy Corps had intended to provide 25 percent of its cash transfers using mobile money but was only able to do so in one area, due to a pervasive lack of electricity, cellular networks, and telecom outlets to redeem the cash transfer (Mercy Corps, 2015). The independent evaluation of EREL found that farming schools were well-received by the community and beneficiaries in PE2 focus groups reported increased knowledge on effective farming practices and increased revenues from their crops and small businesses (IDA, n.d.). KIs have also remarked on their appreciation for the support from IPs who helped them recover from the EVD crisis. Delivering large quantities of cash to vulnerable and often remote communities can be fraught with risks, from fraud to distorting the local economy. However, rigorous and intensive formative studies and implementation monitoring showed that EREL minimized those risks and was successful in delivering its intended results (Mercy Corps, 2015). Case Study 9. EFSP: Cash as a stop-gap or as a catalyst toward resiliency? Leveraging five USAID FFP activities, the Pillar II-supported cash transfer programs in the AFS sector made prudent use of existing USAID program implementation infrastructure to address acute recovery needs, such as food security, in the short term. Results described below indicate that they may also have had the positive unintended consequence of advancing self-reliance. Although there was variation in the modality for disbursing cash (e.g., mobile payment vs. physical cash payment), the five Pillar II AFS activities in Sierra Leone had common traits that contributed to self-reliance (Nestbuilders, 2017; CARE International, n.d.; CRS, n.d.; Save the Children, n.d.; World Vision, 2015). ■ Activities engaged important GoSL entities, such as the National Commission for Social Action (NaCSA; responsible for managing Sierra Leone’s national Social Safety Net Program) and Anti-Corruption Commission (ACC; mandated to create transparency and accountability), which was reported as a strength of the implementation approach by Pillar II implementers and Government counterparts; ■ Activities were gender-sensitive in their cash transfer implementation (e.g., by directing the support to the most senior female in the household, which positioned women to have more involvement and value in household decision￾making), as described by both activity beneficiaries and implementers as a key feature of beneficiary targeting; and ■ Activities implemented cash transfers as part of a package of interventions aimed at building household and community resilience, which was an aspect of activity design that was described in activity documentation and verified during FGDs with community beneficiaries. Regarding the third feature listed above, communities did not merely benefit from an infusion of cash; they were given access to other resources such as information (e.g., on nutrition and agricultural practices), agricultural inputs (e.g., seeds or seed vouchers), and community-based platforms (e.g., savings groups, farmers groups) that built resilience. FGDs with community members in Pillar II districts indicate that activity beneficiaries are still able to reap benefits in the present day. The photo (below) presents a “tree” created by FGD participants in Bombali district. The “fruit” (in green) on the tree were enduring effects of the Pillar II “seeds” (in yellow), or inputs planted by the IP. USAID EVALUATION FINDINGS n 17 In summary, the cash transfers provided by Pillar II IPs facilitated the transition point from humanitarian assistance during crisis to developmental programming. The cash infusions served as fuel while other activity inputs formed the engine that helped households and communities become more resilient. INNOVATION, COMMUNICATION, AND TECHNOLOGY Innovation, communication, and technology (ICT) Pillar II activities were designed to build sustained systems (Objective 3), especially in the areas of infrastructure, connectivity, and health information systems (HIS) to ensure regular communication and workforce compensation. USAID funded activities in this area through the Grand Challenges for Development mechanism. To date, Grand Challenges for Development has undertaken ten Grand Challenges. Prompted by the acceleration of the EVD outbreak in Guinea, Liberia, and Sierra Leone, USAID partnered with the White House Office of Science and Technology Policy, the CDC and the U.S. Department of Defense to launch Fighting Ebola: A Grand Challenge for Development in 2015. PE1 found that this Pillar II funding mechanism has brought about innovative products to the marketplace as a result of partnerships between public and private sector institutions (Moore et al, 2019). In addition to generating promising technologies for future emergencies, in the true PPP model the Grand Challenge engaged actors from multiple sectors and opened up new markets for grantees and their products. The first case study in this section highlights several examples in this area (see Case 10). Innovation includes building onto existing successes or infrastructures. The mHero case study (Case 11) details this approach, which leveraged the power of extant technology for a new purpose. While the ICT sector was successful in generating new and efficient communication technologies with special application for emergencies, respondents indicated that there has been an under-investment in necessary infrastructure (e.g., electricity, cellular networks) and technological expertise and training. Activities often proposed electronic systems, web-based systems, and systems relying on cellular networks or radio—without investing in the people and equipment required to run these platforms over time. All the information systems that were built during the response depended on cellular phone data which were often not reliable. In addition to highlighting the need for multi-stakeholder collaboration to create innovation solutions, the EREL case study (see Case 8 above) illustrates the challenges of inadequate infrastructure and workforce training to ensure effective implementation. Case Study 10. Grand Challenges: Critical new innovations quickly developed and tested in the region Fighting Ebola: A Grand Challenge for Development created a unique platform for fostering public-private partnership to solve development problems. It allowed USAID to mobilize traditional and non-traditional development partners to co-fund a variety of innovations that were used in the Ebola response and recovery. In response to the request for proposal, USAID received over 1,500 proposals and funded 14 initiatives. Grantees ranged from for-profit companies to non-profit institutions who in turn partnered with a variety of organizations including local and USG agencies (Figure 3). These initiatives designed or adapted technologies in a number of areas, including information and communication technology, personal protection equipment, decontamination products to improve safety of health care workers, health worker support tools, and behavior change strategies to prevent spread (USAID, n.d.). This funding mechanism allowed several grantees an opportunity to demonstrate the efficiency and effectiveness of their products in a large-scale humanitarian crisis. A key Figure 3. Fighting Ebola: A Grand Challenge for Development 19 10 1 3 5 13 14 51 7 5 2 Private Research/foundation/education NGO Local government INGO Government agency Grantees All Partners USAID 18 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES feature of this process was the promotion of in-country field testing, redesign, and improvement of innovative technology and products (Moore et al., 2019). After the Grand Challenge, grantees worked toward further improving and commercializing their products. Dimagi Inc. partnered with a large number of leading humanitarian response organizations to develop an OpenResponse consortium, which developed, evaluated, and shared a set of standards-based, easy-to-use, open-source mobile applications built on CommCare, its open source platform already in use. Since 2016, Dimagi has been working with several frontline responder organizations in West Africa to enable rapid deployment of future EVD and other outbreak response applications. (Tom-Aba et al, 2018) For example, in Guinea, the Earth Institute, the United Nations Population Fund, and the Guinean Ministry of Health have adapted the standardized EVD contact-tracing form to a CommCare application (Dimagi, n.d.). CommCare is now present in 60 countries where some 600 active frontline programs are being used in health and agriculture, among other sectors (Tom-Aba et al, 2018). In a different example, a team of faculty, students, and private experts led by the Resilient Africa Network (RAN) at Makerere University School of Public Health in Uganda designed the RAN “EpiTent” to enable isolation during emergencies. The new design kept the favorable features of current tent designs (simplicity, affordability, and structural safety) and re-engineered mechanisms for heat and air exchange. Several EpiTents are now in use in Northern Ugandan refugee settlements and the product was included in the WHO’s 2016–2017 Compendium of Innovative Health Technologies for Low Income Countries (WHO, 2018). To ease decontamination of surfaces and eliminate human error, another Grand Challenge team of students from Columbia University tested a powdered blue disinfectant additive. When added to disinfectant for spraying, it acquires a color (that fades over time) indicating to users when decontamination is complete. Highlight® is now a patented additive for disinfectants designed to improve the fundamental basics of frontline environmental cleaning. It is now sold by Kinnos, Inc., a company that the team of students created. In 2019. Kinnos Inc. won the Harvard President’s Innovation Challenge for Health & Life Sciences (Kinnos, n.d.). Finally, the Baylor College of Medicine developed the Ebola Smart Pod (ESP)—a novel, modular, portable, and scalable Ebola clinical facility with integrated software and patient/supply tracking systems. It can be deployed within one week and at a fraction of the cost of current facilities. Following a successfully tested prototype, the ESP was deployed in September 2017 at the Eternal Love Winning Africa Hospital in Monrovia, Liberia, as a patient isolation unit (Kim et al, 2018). Case Study 11. mHero: Innovation does not always require starting from scratch MOH KIs in all countries stated that during the EVD outbreak, responders struggled to quickly disseminate information including protocols to ensure safety, conduct contact tracing and surveillance, and carry out complete diagnostics and lab tracking. Liberia’s MOH had already rolled out DHIS2 and IntraHealth had assisted the MOH Human Resources Division to digitize human resource records and save them to their open source, integrated Human Resources Information System (iHRIS, pronounced “Iris”). UNICEF also had deployed its SMS-based communication platform, RapidPro, in Liberia and other countries. At the height of the EVD outbreak, brainstorming during a hackathon between MOH, UNICEF, and IntraHealth leaders led them to realize they could drastically improve MOH-to-health worker communication and evidence-informed management if they could connect these three existing systems. In Liberia in August 2014, IntraHealth International and UNICEF launched the global platform mHero, a two-way, mobile phone-based communication system, to provide better communications during the EVD crisis (IntraHealth, 2015). A course was quickly developed for immediate training of health workers (IntraHealth, 2016). In 2015, IntraHealth received a grant from the “Fighting Ebola: A Grand Challenge for Development” to fine-tune, scale up, and strengthen the architecture of mHero in Liberia and to plan on piloting it in Sierra Leone and Guinea. Respondents credited mHero with demonstrating that interoperability of existing systems can leverage speedier solutions to engage with frontline health workers and that data can be used from different systems to better support the health workers and monitoring systems rapidly and effectively. The development of mHero was strengthened through the active participation and commitment of key actors in multiple sectors. In Guinea, the human resources data of the MOH resided in the Ministry of Public Service (MOPS). Negotiations for data access had to take place between the ministries so that the Ministry of Health, along with IntraHealth and its technical partners Jembi Health Systems, and mPowering could obtain the data for iHRIS. One KI noted that the collaborative management structure set up to coordinate this effort was a notable achievement, given that in Guinea each ministry runs independently and with few interactions and shared responsibilities. IntraHealth worked with numerous IPs of USAID in this effort (see full case study, Annex A). While mHero was not piloted and rolled out in Guinea within the project period, it left the MOH with a road map to guide and inspire partners if they chose to continue the work. According EVALUATION FINDINGS n 19 to an MOH KI, the MOH in Guinea has included iHRIS and mHero in the Guinea National HIS Strategic Plan since, indicating its commitment to these systems (IntraHealth, 2016). In Sierra Leone, mHero was implemented with funding from the Global Development Lab (GDL); MOHS KIs associated with the program stated that its implementation underscored the importance of interoperability of all the health information systems (public and private), prompting a review of existing health systems at several GoSL ministries. Sierra Leone’s Chief Medical Officer has included interoperability among health information systems as a priority in the country’s Health Sector Recovery Plan 2015–2020 (IntraHealth, 2016). Self-Reliance (EQ2): What factors contribute to advancing partner country self￾reliance? Helping countries to achieve self-reliance—defined by USAID as “a country’s ability to plan, finance, and implement solutions to address its own development challenges” is currently the keystone of USAID’s development strategy. USAID is realigning and reorienting its policies, strategies, and program practices with an eye to transferring development capability to each country. The second evaluation question in PE2 focuses on whether and how Pillar II activities supported these countries to advance toward self-reliance. However, it is important to note that the self-reliance concept was introduced after Pillar II activities had been in place and it therefore is a perspective that was not integral to Pillar II’s design. USAID’s “journey to self-reliance” framework identifies two key drivers: Commitment (the degree to which a country has the foundation to solve its own development challenges, demonstrated through its laws, policies, and cultures and norms) and capacity (current capabilities in government, civil society, citizenry; and the economy to plan and implement). There is an important link between USAID’s current emphasis on the journey toward self-reliance and its longstanding commitment to sustainability and country ownership (Brinkerhoff, 1992). During PE1, the evaluation team created a framework to examine the presence or absence of key determinants to assess the potential for the sustainability of a specific types of intervention (Moore et al, 2019). The PE2 evaluation team developed a framework that conceptualizes sustainability, as examined in PE1, as different from self-reliance in a number of important ways—including perspective (IP/donor vs. host country), focus (interventions vs. systems), and level (multiple vs. national) (Figure 4). In addition, while determinants of sustainability are conceptualized as externalities that have an impact on the outcome of an Figure 4. Interrelationship of Sustainabililty and Self-Reliance Frameworks Sustainability determinants may also serve as indicators of country commitment Policies, Strategies, and Plans Finances External Threats and Enabling Environments Determinants Accountability Institutional Support Human Dimensions Technical Choices Perspective: IP/Donor Focus: Intervention Level: Multiple Sustainability Framework Self-Reliance Plan, Finance, Implement Perspective: Country Focus: System Level: National Drivers Increased Capacities Government, Civil Society, Citizenry, and Economy Country Commitment Laws, Policies, and Norms Self-Reliance Framework Sustained Outcomes 20 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES intervention, some of these factors are, in fact, important indicators of the commitment and ownership on the part of the country in the self-reliance framework. These two concepts are highly interrelated. Sustainability is a necessary but insufficient condition for achieving self-reliance. Cumulative and sustained outcomes in various sectors are preconditions for increased capacity at the national level to handle future development challenges. Finally, although difficult to trace from the micro level of individual actors and projects to the cumulative level of national self-reliance, ownership and political or social will (commitment) is also intertwined. Thus, some of sustainability determinants are reframed from an externality to elements within the influence of the country and its partners. In the next sections, the following key questions are examined with respect to Pillar II’s contributions: 1. Overall Self-reliance: What progress have countries made towards self-reliance in terms of prevention and/or preparedness for responding to and mitigating the impacts of emergencies? 2. Capacities: Is there any evidence of system-wide increased capacities of 1) government institutions, 2) civil society, 3) the citizenry, and 4) the economy? 3. Commitment: Is there any evidence of increased country commitment, by examining improvements in: 1) policies, strategies, and plans, 2) finances, and 3) accountability? PERCEPTIONS OF SELF-RELIANCE The following discusses overall perceptions of self-reliance with respect to the preparedness of each of these countries for emergencies and their capacities to prevent future outbreaks. “In general, as a country, you hear that the government wants to invest in education, health care…. That is the give back of the epidemic. Instead of waiting for the epidemic to occur they are talking about ways to tackle modern infrastructure.” —Sierra Leone KI, 2019 Respondents of KIIs and FGDs in all three countries generally stated that Pillar II activities had increased the countries’ preparedness with respect to disasters. The most common example provided in Guinea was a how a reported case of Lassa Fever was managed just before the evaluation (Outbreak News, 2019; WHO, 2019). KIs noted that improved health facilities and staff competencies enabled them to identify the first case and within a very short time the new labs enabled them to define the source of the fever and take the required measures. A new communication approach allowed the government to inform the community without creating fear or panic. Therefore, an outbreak was prevented. In Sierra Leone, KIs considered Sierra Leone’s multi-sectoral response to the 2017 mudslides (ReliefWeb, 2017), a demonstration of increased self-reliance. On the other hand, KIs also pointed out that, despite progress, their countries are not quite fully self-reliant, citing several challenges: sustainability of funding, workforce issues, and a lack of basic infrastructure among them. In the case of Guinea, several KIs also mentioned conflict and lack of peace also threatening this journey, as “… political trends are the major factors that affect the country’s progress toward meeting its own development challenges.” “ The system is slowly put in place, but it is still fragile and needs mentoring. Then it will be better able to contribute to the desired changes.” —Guinea KI, 2019 Across all the countries, some KIs note that for the foreseeable future, the countries are going to continue to need support to meet development challenges, especially in times of emergencies. In fact, a couple of KIs recommend the reframing of the self-reliance concept in terms of the resilience of individuals, communities, and the countries—their ability to preserve through shocks despite challenges. “ Resilience, the way it is defined, is about the capacity to predict, to manage …. whatever shock you experience, that you don’t regress. You can absorb the shock. When Ebola happened, routine health services perished. You saw that, right? People had delivery in the streets – I mean it was chaotic! People shut down facilities out of fear. That is not resilience! Resilience is – we have outbreaks, yes! But the outbreak will not shut down our facilities.” —Liberia KI, 2019 EVALUATION FINDINGS n 21 CAPACITY Nearly all the activities featured in the case studies sought to strengthen country capacity at one or more levels—from communities and health care facilities to national entities of government or civil society. In this section, the team synthesizes insights from the fifteen case studies regarding Pillar II’s contributions to the achievement of sustained changes in the capacities of: government institutions; civil society; the citizenry; and the economy. Government institutions: Pillar-II funded activities examined in this report contributed to structural improvements across different levels of government, including capacities to deliver the essential package of health services (e.g., clean water, drugs, and commodities), systems to manage and deploy human resources more efficiently, and new agencies and structures for pharmaceutical management (Cases 1–4, 11, 13, and 14). In addition, some activities helped improve government organizational processes to efficiently communicate, budget, procure, and deploy health services in all three countries. (Cases 1, 3, 4, 14, 15). Activities also worked on building workforce capacities including training, technical assistance, and/or mentoring to strengthen skills in health promotion, IPC and other competency areas (Cases 2, 11, 12, 13). Despite this, workforce development and attrition are still serious issues that threaten the sustainability of activities. For instance, in Guinea, SIAPS developed a very good system for data management and planning, but those in charge expressed that capacity since the end of the support was already diminishing due to lack of continuous training for the cadre of managers and information technology professionals, as well as a loss of staff, sometimes to the private sector. Civil society groups/organizations: Contributions to the development of civil society are much less evident across all the selected case studies. However, there are some examples of note in Liberia and Guinea, especially in the areas of holding government accountable, ensuring the protection of civil liberties, and promoting inclusive development. In Liberia, the CSML activity built the capacity of several media-related civic groups to investigate and report the allocation of EVD resources and supported 57 Community Leadership Forums in 13 counties to train local leaders to consult with their constituencies. (Case 6). Similarly, in Guinea, the NDI developed the capacities of CSOs around electoral observation. (Case 5). On a different note, the PACS activity in Liberia worked with community leaders to institute or revitalize community health committees, with guidelines that promote inclusion of women, young people, and vulnerable groups in health promotion (Case 14). Citizenry: The role of citizens is important in the self-reliance framework when they are able to stay informed and engaged in their countries’ development solutions. As in the case of civil society capacities, case study examples of Pillar II building capacity of citizens in this domain are limited; some results in this arena may be positive unintended consequences of other activities. In Sierra Leone and Liberia, for example, some activities successfully engaged community members in health promotion or quality-of-care activities through the creation/ revitalization of facility management committees) or the roll￾out of the Community Health Services policy or strategy (Case 12 - 14). In addition, in Sierra Leone, health and AFS activities included gender-sensitive targeting and engagements such as involving female community leaders in village savings and loans, empowering women farmers by supporting cooperative farms, and reaching out to the most senior female in each beneficiary household when disbursing cash transfers (Case 9). The economy/private sector: Pillar II activities examined in the case studies did not include strategies to vitalize the economy at the national level and did not include engagement of the private sector in this sphere. There were, nonetheless, mixed contributions in economic revitalization at the local level. In Guinea, the voucher approach strengthened local businesses by injecting cash (rather than goods) into the local economy (Case 7). However, local stakeholders saw the activities had missed an opportunity for ensuring long￾term benefits to supported households by not including an economic component (e.g., organizing households in groups enabling the start of small businesses). On the reverse side, some vulnerable households who received unconditional cash transfers in Liberia used a portion of their transfer to start, resume, or expand a small business, while 10,000 farmers who received seed vouchers increased their harvests, bringing stability in household income and diet diversity (Case 8). In conclusion, by contributing to the four levels described above, Pillar II activities strengthened overall capacity of the health systems in such a way as to start to build the lasting capabilities needed to manage future emergencies. 22 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES Case Study 12. APC/PERHS, IOM/IPC: Rehabilitating services and relationships in Sierra Leone to optimize RMNCH The EVD outbreak called out shortcomings in Sierra Leone’s health system, such as in IPC and the suboptimal interface between communities and health facilities (MOHS, 2017a). The Post-Ebola Recovery of Health Services (PERHS) activity, implemented by John Snow International/Advancing Partners & Communities (JSI/APC) demonstrated that rehabilitating health services cannot focus solely on service provision. From July 2015 to September 2017, PERHS implemented a comprehensive model of renovations, capacity building and community engagement. In addition to these items, the activity revitalized 84 percent of the 365 PHUs in five targeted districts in at least one of the following three ways: (1) improving water and sanitation; (2) installing solar power systems for lighting; and/or (3) providing basic medical equipment (JSI APC, n.d.). Figure 5 highlights improvements in infrastructure and equipment made between baseline and endline (JSI APC, n.d.). Reflecting upon critical factors in the activity’s effectiveness, various KIs involved in implementation noted that FMCs were important platforms for strengthening the community-facility interface and fostering community-led improvements in the quality of health services. PERHS supported the MOHS in the development and rollout of the revised community health worker policy, as well as in the development of formal FMC guidelines, training curricula, or tools that not only benefitted targeted districts, but the entire nation (JSI Research and Training Institute, n.d.; JSI APC, 2017). Various Pillar II implementers mentioned that there was also technical convergence and synergy with other actors, including but not limited to other Pillar II activities such as HC3, which also aimed to rebuild trust and restore health services (Case 13). Also, while PERHS was addressing implementation￾level issues related to IPC and other basic aspects of health facility readiness, another Pillar II activity implemented by the IOM established IPC short-course departments and mobile training at medical and allied health colleges in the country (IOM, n.d.). Pillar II KIs highlighted that contextual factors such as district health management team leadership and commitment to restoring health services also contributed to PERHS’ effectiveness. However, the PERHS implementation experience also highlighted that there are distinct differences between urban and rural settings in terms of social cohesion and how health service delivery is structured, both of which have a bearing on efforts to rehabilitate trust and services in the health sector after a crisis. PERHS’ contributions through policies, strategies, and plans, as well as bridging the divide between communities and providers, helped to progress Sierra Leone along its journey to self-reliance (APC, 2017; MOHS, 2017b). There is, however, an overarching challenge of sustainable financing to ensure that short-term achievements such as those observed under PERHS can endure for the longer-term benefit of the people of Sierra Leone. Joshua Yospyn for JSI Figure 5. Select Infrastructure Improvements by PERHS Boreholes and wells Functional toilets/latrines Electricity from solar power Waste pits and incinerators Delivery beds Delivery kits Weighing scales Steam sterilizers 0 20 40 60 80 100 baseline (Jan-Feb 2016) endline (May 2017) 58 75 46 85 100 33 86 52 76 8 55 36 55 41 67 0 EVALUATION FINDINGS n 23 Case Study 13. JHU CCP/HC3: Multi-country insights from the Health Communication Capacity Collaborative (HC3) HC3, which was implemented by The Johns Hopkins University Center for Communication Programs (JHU/CCP) between October 2012 and September 2017, was awarded Pillar II funding in 2015 to implement SBCC interventions to restore trust in health services as part of post-EVD recovery efforts in Guinea, Liberia, and Sierra Leone. This is a stand-alone activity in the Health sector with variation in activities across the three countries to contribute to Pillar II Objectives 1 and 2 (HC3, 2017a). In all three countries, the primary government counterpart for HC3 was the MOH and its sub-national health authorities, including county and district health management teams (HC3, n.d.). HC3 has worked collaboratively with the entities on not only technical implementation but also on planning and management. Pillar II implementers and government counterparts cited that HC3 has been successful at rebuilding trust and confidence in Guinea, Liberia, and Sierra Leone’s public-sector health systems by combining mutually reinforcing strategies in communities, in health facilities, with media outlets, and with national and sub-national health authorities. Conducting strategic and repeated engagement with the community and with active support from local leaders, HC3 has restored West Africans’ trust in their health services by promoting behavior change with a “Healthy Life” media campaign. In addition, there were improvements to the quality of care in health facilities through minor health infrastructure improvements with community involvement (CCP, 2017; HC3, 2017a-b; Modares and Berg, 2016). Although HC3 had a very specific scope, it did not operate in a silo. Multiple individuals involved in Pillar II implementation noted that there was an explicit focus on complementarity of effort with other implementers. Positioning the community at the center of the demand-/supply-side nexus was stated to be a key factor in the effectiveness of HC3 (HC3, 2017a-b). Though building self-reliance was not an explicit objective of HC3, implementers and government counterparts alike acknowledge that HC3 enhanced institutional capacity by transferring skills to government counterparts and civil society (e.g., community radio stations)—for example, in relation to developing materials and implementing the multiple components of a national media campaign. Nevertheless, both government counterparts and Pillar II implementers also acknowledge that there are broader issues around institutional support for community health that were beyond the scope of HC3 and bridging the divide between communities and facilities has yet to be institutionalized within the health sector. HC3 helped to develop and/or disseminate policies, strategies, and plans such as the National Health Promotion Strategy 2017 in Sierra Leone and Liberia’s revised Community Health Services Policy (HC3, 2017a-b). However, both Pillar II implementers and government counterparts frankly acknowledged that the sustainability of the HC3 initiative is hampered by the lack of financial resources to address core pillars of the public health sector; limited availability of qualified health workers; and the transferring of trained/qualified health workers who have built rapport and trust in their community to other catchment areas. As with the PERHS case study, the HC3 implementation experience also faced differences in social cohesion in urban and rural settings and underscored this as an important consideration for current and future programming that rely on repeated community engagement and action. Elie Gardner for CRS 24 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES COMMITMENT The consensus among KIs across all countries is that the EVD crisis and its devastating effects triggered an urgency for change among many actors, creating a commitment to better prepare and protect the country for the future. This section examines if and how Pillar II activities may have contributed to countries’ commitment by examining changes in three domains: policies, strategies, and plans; finances; and mutual accountability. Policies, strategies and plans: In all three countries, several of the activities examined made timely contributions to the policy landscape, including in the development and/or revision of policies and plans, clarified priorities, technical standards, institutional arrangements and/or roles and responsibilities in relation to key issues highlighted as shortcomings when EVD struck. However, the consensus among most KIs was that those policies, strategies, and plans were only a first, albeit important, executable step in a series of actions that reflect increased commitment to propel countries along a pathway to greater self-reliance; policies and plans must be backed by finances and accountability to bear fruit. Finances: Across the three countries, the lack of diversified and/or sustainable financing was a core dimension of commitment that influenced countries’ ability to leverage the time-limited, focused inputs of Pillar II implementers for longer-term benefit. For example, in Sierra Leone, the GoSL has not systematically committed the budget required to sustain Pillar II activity achievements (e.g., maintenance of equipment and/or infrastructure) (Case 12); compensation of community health workers (Cases 12 and 13); and upkeep of a monitoring system (Case 3). In addition, none of the reviewed activities focused on promoting sustainable financing. A notable exception to this is the increased commitment to sustained improvements in the health sector through support of Guinea’s National Assembly’s Health Commission (Case 15). The Commission succeeded in advocating for an increase from 2.5 percent of national expenditure pre-EVD to 8.2 percent in 2017. Mutual accountability: Several Pillar II activities described in this report made tangible contributions to processes and structures in multiple sectors to improve mutual accountability between government institutions, civil society, political parties, communities, donors, implementing partners and beneficiaries. (Cases 2, 5–9, 14, and 15). In Sierra Leone’s AFS sector, activities engaged focal government institutions to provide formal mechanisms to ensure transparency during service delivery and grievance mechanisms for beneficiaries to voice their dissatisfaction with social services. Similarly, in Liberia the EREL project built in accountability mechanisms that engaged community members to ensure only the selected households received the funds (see Case 8). In Guinea’s governance sector, Pillar II activities contributed to strengthening the Code of Conduct to hold political parties accountable and the consolidation of the CENI to ensure fair and transparent elections for citizens (see Case 5). Furthermore, the CSML activity in Liberia focused its media capacity work specifically on tracking and reporting on the utilization of the large infusion of funds from donors during and after the EVD crisis (see Case 6). Finally, across all the case studies (and beyond) there was an emphasis on improving the accuracy and timely delivery of monitoring data, which are essential for efficient use of resources as well as for documenting results. Photo by Carol Sahey for USAID EVALUATION FINDINGS n 25 Case Study 14. IRC/PACS: Empowering communities to engage in Liberia 4. The three divisions in the MOH that were supported by PACS are Community Health Services; Health Promotion; and Environmental and Occupational Health. Partnership for Advancing Community Based Services (PACS), a five-year $23 million bilateral project led by the International Rescue Committee (IRC), was designed in 2015 to support the MOH’s expanded commitment to community￾level health services (GOL 2015a; GOL MOH, 2015a). The revised National Community Health Services Policy and Plan (NCHSPP) created two new cadres of health workers: Community Health Assistants (CHAs) to serve the 28 percent of the population dwelling over 5km from a health facility; and Community Health Services Supervisors to support and oversee the CHAs (GOL MOH 2015c). PACS provided technical support to three divisions4 of the central MOH to roll out the NCHSPP (GOL MOH 2015d) in three focus counties (IRC, 2017). Late in 2015, $7 million in Ebola Emergency Funds enabled PACS to expand into three additional counties until December 31, 2017 (IRC, 2017). “ The Ebola prevention awareness that IRC did was effective to the community… They were at the front page. Through those preventive messages, the community members started to wash their hands, people were cleaning their surroundings, and people started to visit the clinic whenever they felt any symptom of sickness.” —Liberia FGD, 2019 An independent mid-term evaluation in the three base counties found that PACS-supported communities and districts significantly improved in nine out of 15 global health indicators when compared with other communities and districts. PACS reported in USAID/Liberia’s Performance Indicator Database (PIDS) that 28,208 children with suspected pneumonia received antibiotics from a PACS-trained health worker. Participants in FGDs in PACS districts expressed confidence in the health information and services brought by their CHAs and community health volunteers. Some KIs even inferred that the improved IPC and defecation behaviors in PACS communities were responsible for a drop in the frequency of diarrheal disease in their children (Liberian Statistical Analysis (LSA), 2017). An examination of the factors that contribute to these successes follows. PACS strengthened nine well-established Liberian CSOs to implement much of the community-level work in 1,442 communities in the six counties; for instance, PACS trained CSOs in health promotion and communication to enable them to train health workers in their districts to align with updated national strategies (GOL/MOH 2015d). The activity’s technical choice to leverage pre-existing relationships, and work through existing structures smoothed community entry, built trust, and also enhanced sustainability. PACS advisors sat and worked with their Ministry counterparts at central and district levels on a daily basis. The CSOs’ reactivated Community Health Committees, which were responsible for selecting candidates for the CHA role in their own communities and for mobilizing and linking their neighbors to their health facility (LSA, 2017). Stakeholder participation was built into every aspect of the work. PACS facilitated participatory self-assessments, which were the foundation of Performance Improvement Plans (IRC PACS, 2017). PACS staff taught the ETL method to service providers, CSOs, and community leaders, increasing their ability to engage with beneficiaries. The WASH component of PACS worked with “WASH entrepreneurs” to sell WaterGuard and with “natural leaders” (community members managed by the Ministry of Public Works on community-led total sanitation (IRC, 2017)). While KIs and FGD participants agree that PACS made a lasting contribution to their communities’ self-reliance, they stated that the future is still uncertain. They point out that the continuation of these improvements at the community level is contingent upon continued inflow of resources generated at the national level to fund the training, incentives, drugs, and logistics required for implementation and monitoring of the NCHSPP. “ Our team lead there was very good – they worked together, made plans and went to the field together. So communities didn’t see PACS – they saw the County and saw PACS as just giving push to get the county moving.” —Liberia KI, 2019 26 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES Case Study 15. Abt/HFG: Strengthening the future functionality of institutions and financing in Guinea Abt Associates’ Health Finance and Governance Project (HFG) was a six-year global project (2012–2018) which worked in Guinea for a duration of 24 months (June 2016–June 2018) to support both Guinea’s 2015–2020 Country Development Cooperation Strategy and the post-EVD strategy set up by the USAID Mission in Guinea under Pillar II. The overall objective was to strengthen Guinea’s health system through enhancing functionality of institutions and programs and improving their capacity to deliver quality health services, while addressing fundamental performance problems that had been exposed by the EVD crisis (HFG, 2018). One KI described how through HFG the National Assembly’s Health Commission better understands the needs of its sector, has quality data at hand, has the competencies to analyze, and discusses the national health budget, advocates, and negotiates with other ministries for a higher share of the overall national budget to be allocated for health (Moore et al, 2019, p.41). National budgeting and planning in the health sector are now based on a new bottom-up planning process based on new processes, an electronic information management system, guidance, and significant capacity building at all levels of the health system. The activity equipped the commission to successfully advocate for an increase in the health budget. In 2016, prior to the activity, the national budget allocated 4,4 percent to health (Government of Guinea (GOG) National Budget 2016, 2015). In 2017, the proposal was 5.8 percent and the Heath Commission managed to negotiate a further increase of 2.4 percent (HFG, 2018), resulting in a total budget allocation for health of 8.2 percent (GOG National Budget 2017, 2016). While the allocation fell to 6.0 percent in FY 2018 (GOG National Budget 2018, 2017), it nominally remained stable due to a significant increase of the national budget (Moore et al, 2019; HFG, 2018). In FY 2019, as the national budget grew further, the share for health also increased to 7.7 percent (GOG National Budget 2019, 2018). Whether the commission achieves its goal of 15 percent in 2020 remains open. Whether the increased competencies will be sustainable and carried over to a new Commission after general elections will stay unclear until after the next elections. In Guinea, establishment of new structures requiring both policies and funding is significant evidence of commitment. According to one KI, it likely will require an election to fully understand the sustainability of the higher level of self-reliance achieved. Government respondents reported that media coverage of critical restitution processes advanced awareness that health is a national problem and is a priority. Commitment is reflected in the development and active use/application of tools, guidance, and policies by actors at all levels in the health system. All interviewed stated that they were aware of and immediately had these at hand and that they were regularly used. Users observed a benefit in working with them. Many different respondents stated that HFG made critical contributions to increasing the country’s self-reliance and specifically in being better prepared to face a challenge such as EVD. One MOH respondent reported that the increased capacities in governance and coordination were described as critical for managing any future emergency, in combination with having the right standard documents in place. An example used by many representatives interviewed used to prove their statement was the response to a Lassa fever case identified at the end of January 2019, where communication and action were all well-implemented, illustrating that structures, processes and required capacities are now in place to respond appropriately to infectious disease threats (WHO, 2019). Michael Duff for USAID SUMMARY CONCLUSIONS n 27 CONCLUSIONS (EQ1) Effectiveness: What factors contributed to the achievement of Pillar II’s overarching objectives? This evaluation examines factors by their contribution to the success of Pillar II activities, or lack thereof, in isolation (EQ1.a, stand-alone), in combination with others in one sector (EQ1.b, multiple activities in one sector), and finally in combination with others across sectors (EQ1.c, multi-sector). Therefore, in the following sections, the findings that speak to the success of individual activities are distilled and then those that contribute to outcomes from multiple activities in one sector or across sectors. Most of the identified factors are relevant for individual activities as well as multiple activities working in concert. Activities were funded as part of a mosaic of USG support in coordination with other donors and designed and implemented to reinforce each other to achieve overall Pillar II objectives. (EQ1.A) WHAT WERE THE FACTORS THAT CONTRIBUTED TO THE EFFECTIVENESS OF STAND￾ALONE ACTIVITIES AND WHY? One of the overarching factors that led to the success of activities is a systems approach to both design and implementation. Note that this applies not only to individual, stand-alone activities, but also to multiple activities working together in one sector or across multiple sectors. Relevant elements in the systems approach include: ■ Placing people/beneficiaries and their needs in the center of the activity design (i.e., joint assessments with key stakeholders, staff, and decision makers, creating a common understanding of the priority of needs as well as ownership among all actors); ■ Recognizing and addressing the interconnected parts of the health system (e.g., top/bottom and supply/demand of health system); this was especially relevant for different activities working together to address a common goal; ■ Recognizing that systems are about people and putting processes and structures in place to build trust and relationships during implementation (e.g., embedding staff in government offices); and ■ Understanding context in all systems issues and taking different approaches for different contexts (e.g., in the governance sector, taking approaches that align with the different national political realities). Other factors that contributed to success may be self-evident but include those listed below. ■ Taking an evidence-based approach to activity design, using rigorously generated information with the active, meaningful engagement of relevant stakeholders such as community members, local health staff, or representatives of ministries facilitated ownership of the design of appropriate interventions; ■ Leveraging existing community or institutional structures and processes, rather than trying to start something new, even in the case of innovation, enabled activities to hit the ground running; and ■ Boosting ongoing projects and programs, rather than creating new activities, enabled for quick and effective mobilization and given the reputation and trust already built, ensured activity success; this re-application of previously invested effort created technically sound tools and approaches that could quickly be adapted for a recovery effort. The evaluation also identified the factors that presented challenges to activity success. For instance, contextual differences (e.g., rural/urban) had critical influences on activity implementation and therefore led to differences in outcomes. In all three countries, factors that were beyond the scope of the Pillar II activities (e.g., sustainable financing, human resource management) limited the full potential of some activities from being realized. These and other challenges will be discussed more fully in the self-reliance section. 28 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES (EQ1.B) WHAT WERE THE FACTORS THAT CONTRIBUTED TO THE EFFECTIVENESS OF A COMBINATION OF ACTIVITIES WITHIN EACH SECTOR AND WHY? 5. https://selfreliance.usaid.gov/ The nature of Pillar II’s design focused IPs on different elements or levels of the system which rely on each other to succeed. Across case studies, there were several examples of IPs collaborating with each other within a sector. In the health sector, for instance, this allowed for multiple actors to coordinate solutions across different parts of the health system—including demand-side interventions that formalized community ownership of and involvement in quality improvement in health facilities. This ensured the availability of basic medical supplies and equipment, quality of services, and attitudes of staff at the level of the health center. In the AFS sector, FFP IPs harmonized selected components of their cash transfer programs (e.g., amount), jointly explored solutions to implementation problems (e.g., poor mobile phone network coverage in some areas), and systematically engaged focal government institutions to ensure transparency in targeting and accountability in cash disbursement. Aside from the design and coordination of Pillar II activities at the country level, the other factor that contributed to the successful collaboration of multiple activities is the IPs’ own commitment to do so whenever they saw a potential for overlap, either in geography or technical coverage; USAID also facilitated some of these collaborations. In some instances, such as the FFP IPs, previous relationships and communities of practice also helped with their level of cooperation and eventual success. (EQ1.C) WHAT WERE THE FACTORS THAT CONTRIBUTED TO THE EFFECTIVENESS OF CROSS￾SECTORAL ACTIVITIES AND WHY? By engaging multiple actors in multiple sectors, Pillar II has been able to take a systems approach (top to bottom, and across); this would not have been possible with a single actor. This approach of working with multiple actors and in multiple sectors, has allowed Pillar II to address a complex situation expediently. Several examples have emerged highlighting the interconnectedness of outcomes across sectors. For instance, in the health sector, activities that reached out to non￾traditional partners (such as community radio) saw a significant boost in positive outcomes such as optimal health-seeking behaviors. In all targeted areas, improvements in infrastructure and processes in the health sector led to improvements in governance and transparency; conversely, these improvements in governance improved the health system. Cash transfers and vouchers influenced the success of other cross-sectoral interventions including agricultural recovery (e.g., through the provision of seeds) and livelihoods development (e.g., through farming groups, traders’ groups, and savings groups). Finally, the Grand Challenge was a truly successful multi-sectoral initiative in that it engaged actors from multiple sectors including the private sector—its secret ingredient was that it provided benefits to all involved. (EQ2) Self-Reliance: What factors contributed to advancing partner country self-reliance? Evaluation of Pillar II activities has evolved the understanding of the self-reliance framework itself, especially with respect to capturing contributions at the activity level. This framework lends a very useful perspective for examining where countries are in terms of their ability to manage their own challenges and how partners may contribute to this. However, the 17 self-reliance indicators that USAID has currently identified5 (e.g., government effectiveness, efficiency of tax administration, and GDP per capita), are insufficient to examine what needs to happen at the lower levels to achieve self-reliance. In many cases, as for Pillar II, programs may not be able to intervene at the level of the framework’s current indicators but will often address a specific sector or challenge. To measure contributions to a country’s self-reliance in these cases, the current indicators need to be placed in context. Therefore, the evaluation team has taken steps to delineate the self-reliance framework, including examining its link with sustainability. For instance, what would we observe within a country as it becomes stronger and more resilient? We articulate that sustained outcomes are a necessary, but not a sufficient, condition for self-reliance; a combination of sustained outcomes at multiple levels (including local) generate self￾reliance. Self-reliance needs ownership not only at the level of the national government, but also at local and individual levels. SUMMARY CONCLUSIONS n 29 (EQ2.A) WHAT WERE THE SALIENT FACTORS AND/OR DETERMINANTS THAT POSITIONED THESE ACTIVITIES TO ADVANCE A COUNTRY’S JOURNEY TO SELF-RELIANCE? Pillar-II funded activities examined in this report contributed to countries’ self-reliance in a number of areas, including to structural and process improvements across different levels of government and mechanisms for building mutual accountability between government institutions, civil society, political parties, communities, donors, implementing partners, and beneficiaries. However, funding and resources have been the most commonly raised challenge for sustainability and self￾reliance. For example, lack of funding to maintain operational aspects of processes that monitor the quality and use of essential drugs (e.g., funds for supportive supervision, internet connectivity in health facilities) limits the enduring effectiveness of successful IT-based pharmaceutical commodity management systems. Insufficient funds for salaries lead to attrition of trained staff who now have much better opportunities to get jobs due to their increased competencies. Lack of appropriate levels of investment in infrastructure (such as electricity, banking, or mobile networks) complicates cash transfer schemes. While a key strategy for USAID for mobilizing resources is engaging the private sector, case studies examined in this report did not provide any evidence that there were strategies to revitalize the national economy or to engage the private sector in this domain. While there were sporadic mentions of private sector engagement, as in the case of AFS￾sector local vendors engaged to sell food items in livelihood fairs (Case 7), there was no evidence of a concerted effort to involve the private sector in Pillar II response. It is clear that there are private hospitals and faith-based organizations that are providing service and also receive USAID funding, and that telecom companies that control cellular network coverage are critical to e-Government and e-Banking, but KIs mentioned the private sector infrequently when asked about key partners in post-Ebola recovery. A phased approach, matching a gradual growth of local commitment and increased financial resources was not observed among the cases reviewed. Without realistic assessment at the onset, failure to sustain is built in and activities to contribute to self-reliance unlikely to occur. The end of Pillar II was associated with an abrupt end of funding, except in cases where follow-up projects were put in place. This observation underscores the importance of adopting a systematic approach to institutional support (a key sustainability determinant), even under the auspices of a short￾term recovery effort. Overall, the team believes Pillar II activities were unprecedented in that large and comprehensive sets of interventions worked together well—a need was identified and backed up by significant funding—and all the levels were involved. Whether these resulted in self-reliant nations for emergencies is a different question. Despite the fact that Pillar II was not specifically designed to result in countries becoming self-reliant in the face of emergencies, this evaluation was commissioned to see what progress it may have made towards that end. Increasing a country’s preparedness for disasters requires interventions from the bottom to the top of a system (e.g., in health or agriculture) and a focus on resilience and coordination across multiple sectors. The constellation of the large number of activities under Pillar II has created at least an opportunity to contribute to greater self-reliance. (EQ2.B) TO WHAT EXTENT DID THESE ACTIVITIES STRENGTHEN COUNTRY SYSTEMS TO MITIGATE AND RESPOND TO FUTURE EPIDEMICS? The EVD crisis triggered an urgency for change and KIs did report significant progress to self-reliance. The ability to appropriately deal with incidences of Lassa fever in Guinea have been evidence of improved systems that do not require external support. In Sierra Leone, even when KIs found difficulty in articulating how country systems were strengthened, they acknowledged that the country’s response to the mudslides of 2017 was a demonstration of stronger systems to respond to emergencies. However, KIs also pointed out that despite progress, countries are not ready to go it alone—especially when it comes to break-outs of large epidemics. Achieving sustainable self-reliance at the country level is more challenging and complex than achieving sustainability for a specific service or outcome of an activity. While Pillar II activities examined have made significant contributions to strengthening country capacities and even commitment indicators, they fall short of ensuring that a complex combination of changes are achieved and sustainable. In all three countries there are cases where policies, strategies, and plans for an improvement were completed, but the processes for mobilizing resources lagged behind or were not launched at all. All elements (policies, accountability, and finance) need to be in place for a meaningful commitment. In addition, progress in each of the above domains can occur simultaneously [in parallel] or sequentially, but progress in each domain takes time. Depending upon the pre-existing capacities and bureaucratic procedures in place, designing and implementing changes in some domains can take far more time than others. 30 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES RECOMMENDATIONS The following recommendations for USAID for designing similar interventions to respond to emergent situations are based on the findings from the 15 case studies in this report. 1. Build on, reactivate, and/or optimize existing activities (USAID/USG activities/projects) and local structures and processes as program implementation ‘infrastructure’. USAID should include the identification of such existing infrastructures/processes both during the request for proposal stage, as well as at the initial design phase. Operating under the constraints of shorter-term funding and implementation periods, this will leverage pre-existing relationships and trust between implementers, government counterparts and communities to facilitate fast and effective deployment; adapt proven solutions to quickly and easily address post-crisis needs and gaps; and foster local ownership of results, contributing to a higher potential of sustainability. 2. Continue using the successful Pillar II systems approach when identifying implementers and designing activities so that there is complementarity across an entire portfolio of activities within a given sector and across sectors operating within the same geographical location or time frame, thus maximizing the potential to build sustained systems, strengthen existing institutions and infrastructures and prevent loss of development gains. To achieve this, USAID should convene advisory boards that include OUs, partner country counterparts, other donors, and implementing partners. 3. Develop a clear exit strategy for gradually reducing program funding/technical inputs from USAID and increasing long-term funding and local counterpart responsibilities, both for the level of the overall response effort as well as at the activity level. This should be developed together with key local and external stakeholders, including other development partners/donors, to ensure a coordinated approach to institutional support and capacity building that can yield enduring, positive effects on individuals, institutions and systems. A phased process of should be built in and closely monitored throughout implementation. USAID should build this requirement into the solicitation process. 4. Create strategies to engage the local private sector in post-crisis recovery efforts as well as in national level preparedness plans. Leveraging private-sector entities in recovery activities (e.g., enjoining commercial banks to have at least one branch per County/District/Prefecture; soliciting the use of various human and/or material resources such as mobile networks and skilled personnel.) can bridge the implementation and infrastructure gap that is often present in emergencies in developing nations. Engaging them more systematically for long term development is a way to build self-reliance. USAID should include local private sector representatives to the advisory boards mentioned above. USAID should also provide outreach resources to the private sector. REFERENCES n 31 REFERENCES Brinkerhoff D and Goldsmith A (1992). Promoting the sustainability of development institutions: A framework for strategy. World Development. Volume 20, Issue 3, March 1992, Pages 369-383 Care International (n.d.) Care International Partner Summary: Rapid Ebola Social Safety Net and Economic Recovery Project (RESSNER). Catholic Relief Services (CRS) (n.d.). CRS Partner Summary: Cash Based Food Security Assistance for EVD Affected Communities in Kenema. CRS (2017). Final Program Report for Emergency Access to Food for EVD Affected Guineans (EFSP) CEPPS (2016). Quarterly Report Q1 2016 CEPPS (2018). Quarterly Report Q2 2018. Retrieved from: https://www.sfcg.org/wp-content/uploads/2017/08/CEPPS-FY￾18-Quarter-4-Report-Search_for_-Common_-Ground.pdf Dimagi (n.d.). Ebola Response. Dimagi website. https://www. dimagi.com/sectors/ebola-response/ Ellis, Stephen (2006). The Mask of Anarchy: The Destruction of Liberia and the Religious Dimension of an African Civil War. Second Edition. New York: NYU Press FEWS NET (2014). Guinea, Liberia and Sierra Leone Special Report. December 31, 2014 Freedom House (2019). Democracy in Retreat - Freedom in the World 2019 - Guinea. Retrieved from: https:// freedomhouse.org/report/freedom-world/2019/guinea GAO (U.S. Government Accountability Office) (1990). Case Study Evaluations. Retrieved from: https://www.gao.gov/ products/GAO/PEMD-10.1.9 Government of Liberia (GOL) (2015a). Agenda for Transformation. Steps toward Liberia rising 2030. Monrovia, Liberia GOL Ministry of Health (MOH) (2011). Essential Package of Health Services: The Community Health System GOL MOH (2015a). Investment Plan for Building a Resilient Health System in Liberia 2015 to 2021 GOL MOH (2015b). Revised National Community Health Services Policy 2016–2021 GOL MOH (2015c). Revised National Community Health Services Strategic Plan 2016–2021 GOL MOH (2015d). National Health Communication Strategy Guinée Matin (2018). Guinée : voici le budget 2019 de l’Etat en chiffres et en rubriques. Retrieved from: https://guineematin. com/2018/12/27/guinee-voici-le-budget-2019-de-letat-en￾chiffres-et-en-rubriques/ Health Communication Capacity Collaborative (HC3) (n.d.). HC3 Responds to Ebola [Fact sheet] Baltimore: Johns Hopkins Center for Communication Programs HC3 (2017a). HC3 Sierra Leone Country Program End of Project Report: Using Social and Behavior Change Communication to Increase Trust and Address Reproductive, Maternal, Newborn and Child Health in Sierra Leone, August 2017 Baltimore: Johns Hopkins Center for Communication Programs HC3 (2017b). Health Communication Capacity Collaborative Liberia Endline Assessment. Baltimore: Johns Hopkins Center for Communication Programs. HFG (2018). HFG Guinea Final Report IDA (n.d.). Final Evaluation Report. Economic Recovery from Ebola for Liberia (EREL). Counties of Lofa, Margibi and Montserrado, Republic of Liberia Institut National de la Statistique (2014). 2014 Census Data. Retrieved from: http://www.stat-guinee.org/index.php/statisti/ donnees-structurelles/demographie/31-population-densite￾region-prefecture International Organization for Migration (IOM) (n.d.). Statement of Work, International Organization for Migration Grant (AID-GH-IO-00001), Amendment 1: IOM National Ebola Training Academy—Phase II: Establishment of Infection Prevention Control Short Course Departments and Mobile Training in Sierra Leone International Rescue Committee (2017). Partnership for Advancing Community Based Services (PACS). FY 2017 Year 3 Q4 and Annual Summary Report. Quarterly Period Covered: July 1–September 30, 2017. Annual Period Covered: October 1, 2016–September 30, 2017 IntraHealth (2015). USAID’s Fighting Ebola Grand Challenge: mHero. Quarterly Report December 11, 2015. 32 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES IntraHealth (2016). USAID’s Fighting Ebola Grand Challenge: mHero. Final report. June 6, 2015-September 5, 2016. Washington DC. November 15, 2016 IRC PACS (2017). Activity level monitoring and evaluation plan. Partnership for Advancing Community-Based Services (PACS). Revised version: May 5, 2017 IREX (2015). CSML Project Fact Sheet, October 2015 IREX (2016). Trusted Voices in Challenging Times. Civil Society Leadership Program Final Report. February 28, 2016. Civil Society and Media Report Jhpiego Corporation (2017). End of Project report: Guinea Ebola Response and Recovery: Restoration and Improvement of Reproductive, Maternal, Newborn and Child Health Services (RHS) Jhpiego Corporation (2018). Liberia Restoration of Health Services End of Project Report John Snow International (JSI) Advancing Partners & Communities (APC). (n.d.). Advancing Partners & Communities, Sierra Leone: Improving Community Health Services in the Post-Ebola Context [Infographic Factsheet] JSI APC. (2017). Post-Ebola Recovery: Strengthening Primary and Community Health Services. Arlington, VA: Advancing Partners & Communities JSI Research and Training Institute, Technical Brief. (n.d.). Rebuilding Trust in Health Services: Engaging Communities through Facility Management Committees JSU CCP. (2017). HC3 Guinea Country Program Research Report. Findings from an Endline Quantitative Phone Survey, November 2017 Kim EJ, Navarro SM, Michel S, Soyars C, Hilmers D, & Anandasabapathy S (2018). Development of a rapidly deployable mobile medical unit for emergency disaster settings. Houston: Baylor College of Medicine. Kinnos (n.d.). Kinnos Website. https://www.kinnos.us/ Liberia Statistical Analysis (LSA) (2017). USAID/Liberia Partnership for Advancing Community-based Services Midterm Performance Evaluation: Midterm Evaluation Report. September 2017 Management Sciences for Health (MSH) (2015). USAID|CSH Annual Progress Report Period: February – September 2015. Submitted to the US Agency for International Development and USAID Liberia, under the terms of Contract Number AID￾669-C-15-00001 MSH (2018). Liberia Collaborative Support for Health (CSH) Program. End-of-Project Report (February 27, 2015 – August 31, 2018) Mercy Corps (2015). Economic Recovery from Ebola for Liberia (EREL) Program Funded by USAID/ Food for Peace (FFP). Quarterly Report January 7 – March 31, 2015 Mercy Corps (2016). Economic Recovery from Ebola for Liberia (EREL) Program Funded by USAID/ Food for Peace (FFP). Annual Results Report- October 1, 2015-Sepember 30, 2016 Mercy Corps (2017). Economic Recovery from Ebola for Liberia (EREL) Program. Funded by USAID/ Food for Peace (FFP). Final Program Report (9 January 2015-31 December 2016). Submission Date: March 31, 2017 Ministère de L’Europe et des Affaires Étrangères. Republique Francaise (2015). Ebola : point de situation au 28 décembre 2015. Ebola Sitrep Ministère de la Santé (2016). Manuel de Procédures de gestion Logistique Intégrée des Produits Pharmaceutiques. Retrieved from: http://siapsprogram.org/publication/altview/manuel￾de-procedures-de-gestion-logistique-integree-des-produits￾pharmaceutiques/french/ Ministry of Health and Sanitation (MOHS) (2017a). Government of Sierra Leone. National Health Sector Strategic Plan 2017 – 2021 MOHS (2017b). Government of Sierra Leone. Standards & Guidelines for Water, Sanitation and Hygiene (WASH) Services in Healthcare Facilities in Sierra Leone. Modarres N, Berg K (2016). Qualitative assessment on health system trust and health service utilization in Liberia. Health Communication Capacity Collaborative, Baltimore: Johns Hopkins Bloomberg School of Public Health, Center for Communication Programs. Moore, Z., Bongiovanni, A., Sadaphal, S., Hernandez, O., Espeut, D., Levin, C., & Toole, M. (2019). Performance Evaluation of USAID Ebola Pillar II Activities: Final Report. Evaluation Report to USAID, March 2019. Vienna, VA: International Business & Technical Consultants, Inc. National Democratic Institute (NDI) (n.d.). Guinea – Overview. National Democratic Institute Website. Retrieved from https:// www.ndi.org/sub-saharan-africa/guinea Nestbuilders (2017). Sierra Leone Emergency Food Security Program (EFSP): Final Project Performance Evaluation Office of Inspector General, US Agency for International Development (2018). Lessons from USAID’s Ebola Response Highlight the Need for a Public Health Emergency Policy Framework. Audit Report 9-000-18-001-P. January 24, 2018. Retrieved from: https://oig.usaid.gov/sites/default/files/2018- 06/9-000-18-001-p_0.pdf REFERENCES n 33 Outbreak News Today. Lassa Fever Case in Guinea. February 4, 2019. Retrieved from: http://outbreaknewstoday.com/lassa￾fever-case-guinea-84213/ ReliefWeb (2017). Sierra Leone: Rapid Damage and Loss Assessment of August 14, 2017 Landslides and Floods in the Western Area. Retrieved from: https://reliefweb.int/sites/ reliefweb.int/files/resources/19371_Sierra_Leone_DaLA_ Web-forprinting.pdf Save the Children (n.d.). Save the Children Partner Summary: Kailahun Food for Emergency Ebola Virus Disease Support (FEEDS) Systems for Improved Access to Pharmaceuticals and Services (SIAPS) (2017). SIAPS Guinea End of Project Report. September 2017. Submitted to the U.S. Agency for International Development by the Systems for Improved Access to Pharmaceuticals and Services (SIAPS) Program. Arlington, VA: Management Sciences for Health SIAPS (2018a). SIAPS Guinea Ebola (Pillar II GHET) End of Project Report, February 2018. Submitted to the US Agency for International Development by the Systems for Improved Access to Pharmaceuticals and Services (SIAPS) Program. Arlington, VA: Management Sciences for Health SIAPS (2018b). SIAPS Sierra Leone Ebola (Pillar II GHET) End of Project Report, February 2018. Submitted to the US Agency for International Development by the Systems for Improved Access to Pharmaceuticals and Services (SIAPS) Program. Arlington, VA: Management Sciences for Health SIAPS (2018c). Toward Building Resilient Pharmaceutical Systems: SIAPS Final Report. Submitted to the U.S. Agency for International Development by the Systems for Improved Access to Pharmaceuticals and Services (SIAPS) Program. Arlington, VA: Management Sciences for Health Stemler S (2001). An overview of content analysis. Practical Assessment, Research & Evaluation. Retrieved from: https:// pareonline.net/getvn.asp?v=7&n=17 Strauss A, Corbin JM (1997). Grounded Theory in Practice. Thousand Oaks, CA: Sage Publications, Inc. Tom-Aba D, Nguku PM, Arinze CC, & Krause G. (2018). Assessing the Concepts and Designs of 58 Mobile Apps for the Management of the 2014-2015 West Africa Ebola Outbreak: Systematic Review. JMIR Public Health Surveillance 2018; 4(4): e68 Retrieved from http://publichealth.jmir.org/2018/4/e68/ USAID (n.d.). The Problem. Fighting Ebola Grand Challenge Website. http://www.ebolagrandchallenge.net/our-approach Witter S, Brikci N, Harris T, et al. (2018). The free healthcare initiative in Sierra Leone: Evaluating a health system reform, 2010–2015. Int J Health Plann Mgmt. 2018;33:434–448. Retreived from: https://doi.org/10.1002/hpm.2484 WHO (2018). WHO compendium of innovative health technologies for low-resource settings. Retrieved from: https://apps.who.int/iris/bitstream/hand le/10665/274893/9789241514699-eng.pdf?ua=1 WHO (2019). WHO supports five countries to fight lassa fever outbreaks. Accessed 28 June 2019 at: https://www.afro. who.int/news/who-supports-five-countries-fight-lassa-fever￾outbreaks Republique de Guinée World Vision. (2015). World Vision Partner Summary: Emergency Food Assistance to Port Loko District. At the request of the United States Agency for International Development (USAID), this publication was prepared independently by International Business and Technical Consultants, Inc. (IBTCI). © 2016 by Aliza Waxman for USAID ANNEX VOLUME I ANNEX A: CASE STUDIES Second Performance Evaluation of USAID Ebola Pillar II Activities September 2019 Second Performance Evaluation of USAID Ebola Pillar II Activities: ANNEX VOLUME 1 USAID CONTRACT # AID-OAA-I-15-00022 Task Order # AID-OAA-TO-16-0040 Authors: Sara Woldehanna, Barbara De Zalduondo, Donna Espeut, Karim Sahyoun, Jackie Yiptong Avila, and Elisabeth Nolan Submitted: August 1, 2019 Prepared for: United States Agency for International Development Ronald Reagan Building and International Trade Center 1300 Pennsylvania Ave NW Washington, DC 20004 Prepared by: International Business & Technical Consultants, Inc. (IBTCI) 8618 Westwood Center Drive Suite 400 Vienna, VA 22182 USA Tel: +1 (703) 749-0100 Citation: Woldehanna, S., De Zalduondo, B., Espeut, D., Sahyoun, K., Yiptong Avila, J., Nolan, E. “Second Performance Evaluation of USAID Ebola Pillar II Activities: Final Report.” Evaluation Report to USAID: Annex Volume 1, September 2019, International Business & Technical Consultants, Inc., Vienna, VA. Project Description: The Ebola Pillar II, Monitoring, Evaluation and Learning (MEL) activity is a three-year USAID-funded contract addressing USAID-coordinated efforts in mitigating the second-order impacts of the Ebola virus outbreak in Guinea, Liberia, and Sierra Leone. The activity focuses on four main components: evaluation, routine monitoring, data quality assurance, and improved knowledge management and learning. The activity is led and managed by International Business and Technical Consultants Inc. (IBTCI), with partners StatView International in Guinea, Global Research Insights, LTD (GRI) in Sierra Leone and Liberia, and Opinion Research Business (ORB) International in all three countries. Cover Photo: Liberian goat farmer shows his farm. © 2016 Aliza Waxman for USAID SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES Annex Volume 1, Annex A: Case Studies September 2019 This publication is made possible by the generous support of the American people through the United States Agency for International Development (USAID) under the terms of contract AID-OAA-I-15-00022/AID-OAA-TO-16-00040 managed by International Business and Technical Consultants Inc. (IBTCI) and does not necessarily reflect the views of USAID or the United States Government. ACKNOWLEDGMENTS The authors would like to acknowledge the diligent involvement of the full technical team: Zhuzhi Moore, Annette Bongiovanni, Richard Columbia, Andrew Reuter, Roger Emmanuel Millimono, and Richard Ngafuan. The authors would like to thank the activity coordination staff for their continued support and commitment, including Katherine Labombarde, Maria Pastor, Yasmin Ibrahim, and Victoria Sesay. Special thanks to the editors Irene Labombarde and Erin Dowling, and graphic designer Erin Dowling. A very special thank you goes to Latrisha Chappin and Nadine Ritcheson from USAID’s Bureau for Africa (USAID/AFR) and representatives from the bureaus for Global Health, Global Development Lab, Food Security, Democracy, Conflict, and Humanitarian Assistance, and Economic Growth, Education, and Environment, the Guinea, Liberia, and Sierra Leone missions, and various implementing partners provided valuable support and constructive comments throughout the preparation of this report. Finally, our gratitude goes to the many individuals who participated in the key informant interviews and focus group discussions. This evaluation would not have been possible without your contribution. n v TABLE OF CONTENTS: ANNEX A, CASE STUDIES Case Study 1: Systems for Improved Access to Pharmaceuticals and Services (SIAPS)(GUINEA) . . . . . . . . . . . . . . . . . . . A–2 Case Study 2: Collaborative Support for Health (CSH) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A–7 Case Study 3. Systems for Improved Access to Pharmaceuticals and Services (SIAPS) (SIERRA LEONE) . . . . . . . . . . . . A–15 Case Study 4. Restoration of Health Services (RHS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A–20 Case Study 5. Consortium for Elections and Political Process Strengthening (CEPPS) . . . . . . . . . . . . . . . . . . . . . . . . . . . A–23 Case Study 6: Civil Society and Media Leadership (CSML) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A–28 Case Study 7. Emergency Access to Food for EVD-Affected Guineans (ESFP) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A–34 Case Study 8: Economic Recovery from Ebola for Liberia (EREL) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A–38 Case Study 9: Food for Peace Cash Transfer (FFP/CT) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A–46 Case Study 10: Fighting Ebola – A Grand Challenge for Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A–51 Case Study 11: mHero . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A–57 Case Study 12: Post-Ebola Recovery of Health Services (PERHS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A–65 Case Study 13. Health Communications Capacity Collaborative (HC3) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A–70 Case Study 14: Partnership for Advancing Community-Based Services (PACS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A–74 Case Study 15. Health, Finance, and Governance (HFG) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A–82 vi n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME 1 BACKGROUND The United States Government (USG), with the United States Agency for International Development (USAID) as the lead, mounted a multi-sectoral response to the 2014–2016 outbreak of Ebola virus disease (EVD) in Guinea, Liberia and Sierra Leone. The response was organized around four pillars which focused on a variety of goals from containing the outbreak to strengthening partnerships for regional and global health security. Pillar II, the subject of the current evaluation, focused on post-EVD recovery response to ensure that countries rebuild economic and social systems better than before the outbreak. In October 2016, USAID awarded International Business & Technical Consultants, Inc. (IBTCI), a three-year contract to conduct monitoring, evaluation, and learning (MEL) activities in support of Ebola Pillar II work. Between December 2017 and February 2018, IBTCI conducted the first of the two Performance Evaluations (PE1) to assess the effectiveness and sustainability of Pillar II activities (Moore et al., 2019). The current evaluation, the second Performance Evaluation (PE2), sought to understand the factors that influence the effectiveness of activities funded under Pillar II. IBTCI employed a multiple case study approach for PE2. Cases spanning all three countries in four sectors were selected: Health; Agriculture and Food Security (AFS); Governance and Economic Crisis Mitigation (ECM); and Innovation, Communication, and Technology (ICT). Summaries of the case studies focusing on selected themes are included in the main report. The method and data sources used for the case studies are described in the main report and Annex C. This Annex includes all 15 case studies; they are presented in the order they are mentioned in the main report; Table A–1, however, reflects the cases in order of country and sector. Table A–1. Case Studies, Sorted by Country and Sector Case Study No. Featured Activity Sector Guinea 1 Systems for Improved Access to Pharmaceuticals and Services (SIAPS)/Guinea Health 15 Health Finance and Governance (HFG) Health 7 Emergency Access to Food for EVD-Affected Guineans Program AFS, ECM 5 Consortium for Elections and Political Process Strengthening (CEPPS) Governance Liberia 14 Partnership for Advancing Community Based Services (PACS) Health 2 Collaborative Support for Health (CSH) Health 8 Economic Recovery from Ebola for Liberia (EREL) AFS, ECM 6 Civil Society and Media Leadership (CSML) Governance Sierra Leone 12 Post-Ebola Recovery of Health Services (PERHS) Health 3 SIAPS/Sierra Leone Health 9 Food for Peace (FFP) Cash Transfer AFS, ECM, ICT Regional 13 Health Communications Capacity Collaborative (HC3) Health 4 Restoration of Health Services (RHS) Health 11 mHero ICT 10 Fighting Ebola: A Grand Challenge for Development Cross-sectoral CASE STUDIES n A–1 CASE STUDIES Photo by Mariama Keita for USAID A–2 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME 1 GUINEA CASE STUDY 1: SYSTEMS FOR IMPROVED ACCESS TO PHARMACEUTICALS AND SERVICES (SIAPS)(GUINEA) BOX A1–1. SIAPS (GUINEA) AT A GLANCE Lead Implementing Partner (IP): Management Sciences for Health (MSH) Nature of Implementation: Supplemental funding to an ongoing single sector (health), stand-alone activity Implementation Dates: April 2016–September 2017 (SIAPS started in Guinea in 2012) USAID Pillar II Budget/Obligation: USD 2.5 million1 Geographical Priority Areas: Guinea national (central, regional, district, and peripheral) Key Activity Components: Increasing pharmaceutical supply chain management capacity, strengthening logistics, improving the availability data to support transparency and decision-making, and increasing the use of information communication technology (ICT) to streamline the supply chain system’s efficiency. Key Pillar Objectives and Outcomes to which Activity Contributed: SIAPS contributed to all three Pillar II Objectives: 1 (“Prevent the loss of development gains”), 2 (“Recover and strengthen existing institutions and infrastructures”), and 3 (“Build sustained systems through public–private partnerships, innovation, and capacity-building”) with a specific focus on capacity-building. Between 2011 and 2018, Management Sciences for Health (MSH) implemented a global, USAID￾funded program called System for Improved Access to Pharmaceuticals and Services (SIAPS). Before the start of 2014, prolonged stock-outs of critical drugs in Guinea were common; the pharmaceutical system was then further negatively affected by the Ebola crisis (SIAPS, 2018), which presented a series of challenges to effectively distributing required medicines and materials. In 2015, SIAPS received additional Pillar II post￾Ebola recovery funds to implement a short-term activity to quickly improve pharmaceutical management in Guinea (see Box A1-1). As SIAPS had been active in Guinea since 2012, the case of SIAPS’s Pillar II activity highlights the benefit of building onto existing activity infrastructure to expedite rapid, targeted activities during emergency response. Additionally, the SIAPS case demonstrates the efficacy of leveraging existing relationships from within the system, rather than inserting technical expertise from outside it. 1 The objective of the Pillar II-funded activity was to radically improve and institutionalize structures to support effective 1. Source: USAID’s Office of Budget and Resource Management O&E spreadsheet, September 2017. management, leadership, and the Ministry of Health (MOH)’s ability to keep quality drugs in stock and remove expired ones from the shelf. This a critical precondition for improving the quality of health services in the country and moving from inefficient allocation-based to efficient needs-based planning and management (Defawe & Phillipset, 2015; Focus Group Discussion (FGD), 2019; SIAPS, 2018). Activities of SIAPS targeted integrating a supply chain that was operating in silos, strengthening management competencies, establishing a Logistics Management Information System (LMIS), and increasing the use of digital technology for decision-making. This activity addressed not only the shortage of products in the country, but also insufficient logistics data (e.g., stock on hand, consumption) which were negatively impacting the country’s ability to develop accurate forecasts (SIAPS, 2018). As a result, Guinea has moved away from central MOH planning—allocating predetermined amounts of drugs to certain districts and facilities—and toward a model where drugs are provided based on use and predicted need. In addition, the Guinean health system is now able to provide up-to-date data on the stocks of its medicines and health commodities. Insights from the SIAPS Program by Dr. Karim Sahyoun CASE STUDY 1: SIAPS IN GUINEA n A–3 FACTORS CONTRIBUTING TO EFFECTIVENESS 2. Average Monthly Consumption corresponds to the monthly average of the quantities consumed over three recent and typical months. Enabling the Guinean health system to have up-to-date data requires a complex supply-chain systems approach that includes components such as integrating a fragmented supply chain, moving from a paper-based management and reporting approach to one based in information technology, strengthening existing institutions and establishing new ones, defining procedures, developing manuals, improving infrastructure, and building staff competencies (SIAPS, 2018). SIAPS’s presence in Guinea prior to Ebola contributed not only to its ability to act rapidly, but also leveraged ongoing relationships and experiences of SIAPS staff within the country’s pharmaceutical system context. Positive relationships between peers, government ministries, and a basic SIAPS understanding of preexisting procedures, coupled with a positive attitude toward change, contributed to a team-like approach. As one KI put it, “They did not come to do the work in our place, but to work with us” (KII, 2019). SIAPS was thus able to contribute to a number of outcomes that were central to improving the access to and quality of essential data: ■ The supply chain has been integrated, with the Central Pharmacy of Guinea (PCG) now at its center. PCG has the monopoly for supplying all Guinean public health facilities with essential drugs on the “List of Essential Medicines” (SIAPS, 2013). PCG responsibilities include purchasing, distributing, marketing, and managing five regional depots and a central warehouse (Pharmacie Centrale de Guinée, 2019). The PCG has been enabled to perform its critical role through significantly increased capacities including infrastructure, staff competencies, and an automatization of its processes. ■ A Logistics Management Unit (LMU) has been successfully established as a specialist unit within the MOH’s National Directorate for Pharmacy and Medicines (Direction Nationale de la Pharmacie, et du Médicament), improving coordination and transparency of the supply chain system (KII, 2019; SIAPS 2018). According to SIAPS, “The LMU is a management structure that is responsible for organizing, monitoring, and supporting supply chain activities within a logistics system” (SIAPS, 2018, p. 7). Before, it was not possible to access any accurate, up-to-date data on facility stocks or their use of essential drugs and pharmaceutical materials. According to MOH staff, surplus can now be detected and redistributed, avoiding waste. ■ A Manual of Integrated Logistics Management Procedures for Pharmaceutical Products has been developed (KIIs, 2019; Ministère de la Santé, 2016) and a standardized paper-based and electronic Logistics Management Information System (LMIS) is in place and in use (KII, 2019; SIAPS, 2018; Defawse & Phillipset, 2015). Therefore, health agencies and actors in Guinea now work with distinct procedures and responsibilities. Further, essential pharmaceutical data to be collected is defined, including its use. ■ Availability of up-to-date stock data (monthly updates) has allowed the Guinean health system to move from an allocation-based planning and management approach to a needs-based approach. The LMIS captures essential data on available and usable stock, the rate of consumption,2 and losses and adjustments on a monthly basis at all levels. This data allows health centers and hospitals to base their quarterly order of medicines and health commodities on the actual stocks and needs. It also allows them to identify and reallocate surpluses to avoid waste and has a flexible component that provides for unforeseen emergencies (Ministère de la Santé, 2016). “The system covers the entire country, all 38 prefectures. Information now comes from [the prefectural level] and is accessible […]. The system helps [the MOH] to instantly have information at hand [and] allows [the MOH] to provide it to those who need it and react quickly. It helps us to better know the current stocks and distribute drugs and react to specific needs.” – KII, 2019 A–4 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME 1 Common vision All stakeholders had a common vision: the need to computerize the system while taking into consideration everyone’s needs. All were clear about what they wanted and had a common understanding, and all were very open in the dialogue enabling them to reach agreements (KIIs, 2019). “The spirit of SIAPS has enabled change. The project and its staff have quickly and well understood the local health system and have well adapted to working with it.” – KII, 2019 Attitudes and embedding of SIAPS team National stakeholders in Guinea have appreciated the attitudes and ability of the SIAPS team to understand the local health system, its critical areas and positively integrate into and support the local structures, expressing that what is done in the central pharmacy now also is done at regional level (KIIs, 2019). “The technical assistants who came to support us had the competencies we had wanted and required. They had the attitudes and commitment to work with us. It was a real work in partnership. They did not come to do the work in our place but work with us.”– KII, 2019 Comprehensive training component SIAPS developed and implemented a comprehensive training package, involving all relevant levels of the health structures providing a wide-ranging understanding of the supply chain and specific competencies. Capacitating the regional level has been seen as critical for the success (KIIs, 2019). Time constraints of health center directors and their regular absence because of training and other responsibilities has been a bottleneck in reporting data. A successful practice has been to train a second person (KII, 2019). This practice has also reduced the potentially negative risk of staff movement, which is still seen by many as a key challenge to sustaining changes. Few staff prefer to stay in the regions and especially remote locations; most seek opportunities to move to the capital, including better paid opportunities outside the governmental health service (KIIs, 2019). Targeted support to weaker prefectures and continued learning Initially, wide disparities occurred among prefectures in the amount of data reported, ranging from 10 percent to 100 percent. Targeting specifically the six weakest prefectures has allowed these to be among the best at reporting (KII, 2019). Replicating this success in other prefectures is seen as critical for the sustainability of the change and at the same time perceived as a major challenge. Allocating time during monthly district-level meetings for peer support is one strategy that has been adopted to at least partially meet this need (KII, 2019). Representatives of health centers have described that reporting and discussing results at a regional level has become like a competition in a positive sense, because they want to be stronger than their peers (FGD participants, 2019). Long-term support For this case, post-Ebola support was added as a boost to SIAPS, which had been ongoing since 2012. Further, continuity was ensured after the end of the program in 2017 through a continuation of key components as part of Chemonics’ current program (USAID Global Health Supply Chain Program), even taking over key former SIAPS staff (KIIs, 2019). Achieving comprehensive and complex change within the two-year funding phase of USAID’s post-Ebola program would not have been realistic. A full integration of the LMIS into the overall health system’s management system is still ongoing. “Currently, if we want to access information on the availability of drugs, we can get all [that] from the central pharmacy. When the integration of the IT systems has taken place, we will be able to access all information directly from here.” – KII, 2019 CASE STUDY 1: SIAPS IN GUINEA n A–5 “In summary, the system is in place, actors have been trained, and the tools are available. We have a guide that describes the entire procedure in detail. It is a system you can count on, which allows [you] to act more effectively in case of an emergency, with now only some limited external support. . . .We need to go further; we have achievements, but we need to be further coached to better perform. The system is slowly put in place, but it is still fragile and needs mentoring. Then it will be better able to contribute to the desired changes.” – KII, 2019 FACTORS CONTRIBUTING TO ADVANCING SELF-RELIANCE Interviewees from all different entities consulted during the evaluation generally agreed that SIAPS had made positive contributions to increasing the country’s self-reliance, including a better preparedness to facing a challenge such as the Ebola epidemic, while also acknowledging there is more to be done. By establishing a pharmaceutical LMIS, SIAPS has contributed to the county’s self-reliance, it has enabled planning, with input from its local- to national-level services and institutions, based on reliable data sources and a common understanding of the supply chain (KII, 2019). This contributes to improvements in essential health services by enabling the provision of essential drugs and consumables throughout the country. It allows the system to proactively respond to changing needs based on evidence and reduces wastage of scarce resources. The data provides the Health Commission with critical information to lobby for allocating sufficient funds from the national budget for essential health services (see HFG Case Study). When it comes to emergencies, the SIAPS￾supported systems and capacities allow the MOH to locate and strategically reallocate critical resources. This has increased the pharmaceutical system’s capacity to efficiently handle additional resources required to address emergencies, relying less on external management support. Commitment The urgency of the Ebola crisis, with its devastating effects for many actors, quickly demonstrated that the health system did not work and required improvement (KII, 2019). This helped stakeholders to quickly acquire a common vision about the desired changes (KII, 2019). A critical component of SIAPS has been supporting the integration of a defragmented supply chain and helping actors along the chain to understand their specific roles and responsibilities (KII, 2019; SIAPS, 2018). This was further supported with a systematic capacity building program, allowing actors at all levels to have the required training and tools to take on their newly defined roles. Quality data collection has now become a regular topic at the monthly health district meetings, including peer support (FGD, 2019; KII, 2019). Guinea’s commitment is reflected in the establishment of a LMU, the development and continued use of the pharmaceutical management manual (Ministère de la Santé, 2016), the establishment of the LMIS, and its ongoing integration with the national Health Management Information System (HMIS). Capacity As remarked earlier, leadership and management training along the supply chain has reportedly enabled individual players to understand data systems, supervise quality, conduct supervisions—in general, better fulfill their responsibilities as now defined (KII, 2019). With new tools, such as a pharmaceutical management manual and guidelines for collecting and analyzing quality data on every level of the health system, all layers of the system contribute to its utilization. In addition to directors of health centers, a second-level support training has been conducted to ensure a timely submission of data and sustainable knowledge, although this is an ongoing area of concern due to staff turnover. Institutional capacities have been developed by establishing the LMU, which is responsible for organizing monitoring and is able to facilitate supply chain activities. A–6 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME 1 Sustainability Due to the fact that support is ongoing (through Chemonics’ Global Health Supply Chain project), discrete effects related to the ending of support are not visible. However, two major challenges were identified: Human resources Despite the “mass” training provided by SIAPS, there are still significant concerns related to staff movement or retirement. Jobs outside the capital are often not sufficiently attractive to ensure that staff will stay, and jobs outside the government service are better paid (KIIs, 2019). It is challenging to hire staff with relevant competencies. Continued capacity-building for trained staff is required to sustain and further improve current achievements (KII, 2019). The loss of capacities would result in a deterioration of the quality and frequency of data provided, reducing its usefulness for planning and management. Funding for the health sector Funding for the health sector has been a major limiting factor in moving toward higher levels of self-reliance. Despite significant successes of USAID’s Health, Finance, and Governance Project (HFG) in Guinea, resulting in an increase of the share of the national budget devoted to health, funding remains a key challenge with two major implications for this case: ■ Staff salaries are not sufficiently competitive compared with non-state salaries, resulting in significant staff movement and specifically a loss of those staff in the public health system who are well-trained. ■ While the new LMIS now allows health centers to exactly define their needs, they still have stock-outs due to a lack of funding (KII, 2019; SIAPS, 2018). The availability of information does not result in funding being available. REFERENCES Defawe, O. N., & Phillipset, P. S. (2015). Diagnostic des SIGL en Guinée. Présenté à l’Agence des États-Unis pour le Développement International par le programme des systèmes pour l’amélioration de l’accès aux produits et services pharmaceutiques (SIAPS). Arlington, VA: Management Sciences for Health. Ministère de la Santé. (2016). Manuel de procédures de gestion logistique intégrée des produits pharmaceutiques. Pharmacie Centrale de Guinée. (2019). Pharmacie Centrale de Guinée: Historique. Retrieved from: http://www.pcguinee.com/ historique on 07/08/2019. SIAPS. (2013). Évaluation de la régulation du système pharmaceutique en Guinée. Mission de Juillet à Septembre 2012. Présenté à l’Agence des États-Unis pour le Développement International parle programme des systèmes pour l’amélioration de l’accès aux produits et services pharmaceutiques. Arlington, VA: Management Sciences for Health. SIAPS. (February 2018). Guinea Ebola end-of-project report. Submitted to the US Agency for International Development by the Systems for Improved Access to Pharmaceuticals and Services (SIAPS) Program. Arlington, VA: Management Sciences for Health. USAID. (April 2019). USAID policy framework: Ending the need for foreign assistance, summary. CASE STUDY 2: CSH IN LIBERIA n A–7 LIBERIA CASE STUDY 2: COLLABORATIVE SUPPORT FOR HEALTH (CSH) BOX A2–1. CSH AT A GLANCE Lead Implementing Partner (IP): Management Sciences for Health (MSH) Partners: Jhpiego, Institute for Healthcare Improvement (IHI), Results for Development (R4D), and the Development Innovations Group (DIG) Nature of Implementation: Single sector (health), stand-alone activity Implementation Dates: February 2017–February 2019 (terminated August 2018) USAID Pillar II Budget/Obligation: USD 5 million1 Geographical Priority Areas: Liberia—Bong, Lofa, Nimba, expanded to Grand Bassa, Margibi, and rural Montserrado Key Activity Components: The CSH activity was designed to strengthen the capacity of Liberia’s Ministry of Health (MOH) centrally, and in three of Liberia’s 15 counties, “to consistently and effectively deliver high-quality health services” by providing technical assistance, mentoring, and logistical support in seven technical areas: (1) leadership, management, and governance (LMG); (2) water, sanitation, and hygiene (WASH, working with the Ministry of Public Works); (3) quality assurance/quality improvement (QA/QI) of health care; (4) human resources for health (HRH); (5) supply chain management (SCM); (6) health financing; and (7) health management information systems (HMIS). For its first two years CSH was organized and reported in these seven workstreams, which together cover the essential elements of health system strengthening. In FY18, Ebola funds were added to enable CSH’s QA/QI workstream to add focus on Infection Prevention and Control (IPC), and to expand the overall program into three additional counties. An independent midterm evaluation found that CSH was spread too thin, so its WASH and LMG workstreams were deprioritized for the remainder of the activity. Key Pillar Objectives and Outcomes to which Activity Contributed: CSH enabled the MOH to strengthen and institutionalize its core technical leadership, management and accountability functions both centrally and at the county level, thus contributing directly to recovering and strengthening existing institutions and systems (Objective 2). Indirectly it contributed to stopping the loss of development gains (Objective 1), by providing implementers at the facility level and their district- and county-level supervisors with updated policies, clearer management guidelines and tools, and more accurate, comprehensive, and interoperable health information systems. Strengthening Health Systems in Liberia through Decentralization and Coordination by Dr. Barbara De Zalduondo The Collaborative Support for Health (CSH) activity, a consortium led by Management Sciences for Health (MSH) and funded by the United States Agency for International Development (USAID)/Liberia over three years with a one-year extension, was implemented between February 27, 2015, and August 2018 in Liberia. The consortium partners included Jhpiego, Institute for Healthcare Improvement (IHI), Results for Development (R4D), and the Development 1. Source: USAID’s Office of Budget and Resource Management O&E spreadsheet, September 2017. 2. The World Health Organization (WHO) specifies the six building blocks of an effective health system as: (i) service delivery, (ii) health workforce, (iii) health information systems, (iv) access to essential medicines, (v) financing, and (vi) leadership/governance (WHO, 2007). Innovations Group (DIG). CSH built on MSH’s previous work in Liberia (Rebuilding Basic Health Services, 2009–2015), and was designed to strengthen the building blocks2 of Liberia’s health system which had been decimated, first from 14 years of civil war (Ellis, 2006; Government of Liberia (GOL), 2015a; GOL, 2015b), and then by the Ebola Virus Disease (EVD) crisis in 2014–2015. Ebola took 4,810 lives in Liberia (World Health Organization (WHO), 2016a) and reversed many of A–8 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME 1 the country’s recent development gains (GOL, 2015a). CSH was aligned with the Government of Liberia (GOL) Ebola recovery strategy (GOL, 2015b) and designed to contribute to the MOH’s strategic plan (GOL/MOH 2015), USAID/ Global Health Ebola Team’s (GHET’s) results framework 3. The 20 outcomes of the CSH Results Framework fed into four Intermediate Results (IRs) under USAID/Liberia’s Development Objective (DO) 3: Improved health status of Liberians: Increased utilization of quality health services (IR3.1), more responsive services through effective health system decentralization (IR3.2), increased financial sustainability of services (IR3.3), and increased safe water supply (IR3.4). 4. Complementary to CSH were mHero (see Case Study 11), PACS (see Case Study 14), RHS (see Case Study 4), Human Resources for Health, HC3 (see Case Study 13), JSI/APC, and JSI/DELIVER. (USAID/Bureau for Global Health (BGH), n.d.), and to four intermediate results in USAID/Liberia’s Country Development Cooperation Strategy (CDCS).3 The project’s original seven objectives, and corresponding workstreams are shown in Table A2-1. Table A2–1. Objectives and Structure of the CSH Program CSH’s Initial Objective Corresponding CSH Workstreams Develop leadership, management, and governance capacity of the MOH at all levels. Leadership, management, and governance (LMG) Strengthen the Ministry of Public Works (MPW) capacity to manage water supply infrastructure improvements. Water, sanitation, and hygiene (WASH) Institutionalize Quality Assurance/Quality Improvement (QA/ QI) initiatives to improve health care service delivery. Quality Assurance/Quality Improvement (QA/QI) Strengthen human resources for health management. Human Resources (HR) Improve supply chain management. Supply Chain Management (SCM) Increase the financial sustainability of services. Finance Strengthen the health management information system. Health Management Information Systems (HMIS) At the start of Project Year 3 (September 2017) the USAID/ GHET obtained Pillar II funding and expanded CSH’s scope from the original three base counties (Bong, Lofa, and Nimba) to include three more: Grand Bassa, Margibi, and Montserrado. The latter are referred to as Ebola Expansion Funds (EEF) counties. The EEF expansion was to build upon and scale up achievements of the base project’s workstreams, focusing on three objectives (MSH, 2018): ■ Strengthen capacity of health facilities in the three counties to practice Infection Prevention and Control (IPC) standards and provide health services according to the National Health Quality Strategy and Plan. ■ Support implementation of the national health information system and accelerate improvements to health and logistics information systems. ■ Strengthen County Health Team (CHT) capacities and systems in Grand Bassa, Margibi, and Montserrado counties to strengthen the restoration of health services and strengthen core systems functions. In addition, following an external midterm evaluation, the base project was streamlined (MSH, 2015b) and the number of objectives was reduced to five. Strengthening the MPW capacity to manage water supply infrastructure improvements was removed, and development of LMG capacity was effectively mainstreamed across the remaining five objectives. Although CSH was managed as a stand-alone activity, its focus on management and governance at the central and county levels of the health system was complemented by other Pillar II-funded activities4 that worked at other levels (e.g., the International Rescue Committee [IRC] Partnership for Advancing Community-Based Services [PACS] focused on district, facility, and community levels), or on specific health system needs (e.g., JSI/ DELIVER focused on supply chain). CSH also worked across sectors, as its counterpart for advancing the GOL’s and USAID’s WASH objectives was the Ministry of Public Works (MPW). The CSH project’s overall strategy was to provide technical assistance (TA) mentoring and staffing and logistics support, working within and with the Ministry staff and systems. It illustrates and offers useful lessons learned on the benefits and risks of this approach. The addition of three EEF counties in PY3 offers insights into what can be accomplished in 10 months when extending to new counties. CASE STUDY 2: CSH IN LIBERIA n A–9 5. The USAID HICD model stresses that to improve capacity requires changes both in individuals (e.g., skills) and their environments (e.g., workplace features). “If I’m a [monitoring and evaluation (M&E)] guy, CHS M&E would sit with me and collaborate. We worked together, did visits together, collected data together, did critical thinking together—that was the kind of structure. We weren’t setting our own targets. We were supporting MOH targets.” – KII, 2019 ACCOMPLISHMENTS The CSH project followed a classic, stepwise approach to strengthen or build the systems and tools required for the Ministry to perform and sustain its core functions of guidance, service delivery, management, and quality assurance. It embedded mentors in five counterpart MOH units: Human Resources (HR); Health Monitoring, Evaluation, and Research (HMER); Supply Chain Management (SCM); QA/QI; and Planning). In each, it assisted the Ministry counterpart team to do the following: conduct assessments (including stakeholder mapping); set baselines and targets; review and update policies; refresh or develop needed guidelines and tools; establish a stakeholder engagement group; provide training; implement monitoring and evaluation (M&E); promote data review and use; and scale up and disseminate results. Each of these steps involved CSH and the Ministry in close consultation with USAID and other implementing partners such as the Global Fund for HIV/TB and Malaria, the World Bank, and U.S. Government (USG) IPs such as MEASURE Evaluation, JSI/DELIVER, and the Clinton Health Access Initiative. It involved consultation and pretesting with product users and other stakeholders as well. The effort invested in consultation was appreciated by MOH key informants (KIs, 2019). While leadership, standard setting, resource allocation, and oversight are important central MOH functions, CSH focused much of its effort at the county level, where these MOH functions were far less developed. KIs in both the central and county MOH stated that in the six CSH counties, it helped transform the MOH’s On embedded staff: “MSH integrated our staff with MOH staff. If you went to see an HRH staff, you would see them sitting next to an MOH staff. If there was one printer, it was shared; I see you every day, we are collaborating together.” – KII, 2019 decentralization policy (National Decentralization Policy [2011] and Plan [2013–2017]) from principal to action. Initial assessments found crucial gaps in each CSH workstream. The CSH project embedded staff in six MOH units, where they supported their Ministry counterparts to put in place both the conditions and staff with the skills required for the job, using the USAID Human and Institutional Capacity Development (HICD) model (USAID, 2011).5 The task was enormous, and frequently changing: The MOH itself was reorganized following hard-hitting post-EVD reviews (MOH, 2015). The Social Welfare sector was moved from the MOH to the Ministry of Gender, which created an opportunity to upgrade critical planning and management departments. For example, CSH advisors supported the consolidation of HR functions from three departments into an integrated Human Resources Unit, and CSH’s QA/QI counterparts created a home for the Ministry’s QI functions in a new Quality Management Unit (see MSH, 2018b). Implementation in the wake of the EVD crisis intensified several common challenges. The USAID Mission was programming a massive increase in resources without a matching increase in staff (Moore et al., 2019). CSH progress was delayed when staff had to decide whether to wait for USAID reviews and approvals or to proceed based on submitted drafts. The explosion of post-EVD program efforts meant that USG-funded contracts competed for the reduced pool of qualified nationals. Key informant interviews (KIIs) with former CSH staff supported MSH reports that CSH had difficulties filling some technical and management positions, especially in the more distant counties (Lofa, Nimba, and Grand Bassa). In addition, the CSH leadership had to balance being responsive to evolving GOL and USAID guidance and demands with maintaining continuity and stability in their work plan and M&E plan. For example, the Pillar II, EEF funds that enabled CSH to expand to expand from three to six counties in Project Year 3 came with new functions (such as a focus on IPC) and a significantly revised Performance Management Plan, leading to discontinuities and the need to retrain the M&E and HMIS staff and counterparts. A midterm performance evaluation of CSH (Liberia Statistical Analysis (LSA), 2017) criticized the project for being too responsive, and recommended that the project focus more on implementing proven interventions and measuring success in terms of health system performance. Following the midterm evaluation, the CSH scope of work was streamlined from seven to five objectives, and several initiatives within the remaining five were reduced or eliminated (e.g., the Capacity Collaborative Network). A–10 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME 1 Despite the challenges, the documented improvements in the MOH structure and function at the central and county levels and the views of MOH KIs indicated that the CSH project contributed to three Pillar II outcome indicators: health infrastructure strengthened, health information system developed and supported, and health financing and resource allocation improved. “We needed a Laboratory Technical Advisor but for a year went back and forth and back and forth. So finally, we hired someone, the most experienced lab person. He only stayed two months, then went to CDC. So, to find someone who will stay for the life of the project is a challenge.” – KII, 2019 FACTORS CONTRIBUTING TO EFFECTIVENESS On improved management by County Health Boards: “If you go to the county, now they have the budget committee. Every county has a budget committee. So the CHO is not spending just like that anymore. So when the funding comes in [to] the county, the procurement department comes up with the list of things to buy, which is then presented to the budget committee for allocation on what funds are to be used. And I think it is a good procedure. And because they have been helped by that, now the counties can boast of managing their own funding.” – KII, 2019 A number of factors contributed to the CSH activity’s effectiveness in helping Liberia to rebuild and strengthen its health system. Ironically, based on responses of numerous KIs in the MOH and other IPs, the strongest appears to have been the impact of observing the health system’s collapse during the EVD crisis: MOH and other GOL staff were eager to learn, collaborate, and change in order to avoid a recurrence. They recognized and agreed that substantial changes were needed. “In my opinion, there was a great deal of political will to support the initiative. There are some projects that have difficulty getting support. But CSH had high-level political will to get things started.” – KII, 2019 Given that willingness, the MSH consortium was able to suggest and support the stepwise assessment, consultation, planning, design, and implementation processes cited above (see above Table A2-1), to fill observed gaps in organizational policies, guidelines, and processes. Filling these gaps helps catalyze or fortify the structures that provide health officials with timely workforce, supply chain, financing, and service delivery information needed for efficient management and quality improvement. In each of these areas, when CSH began, there were fragmented and parallel information systems of dubious accuracy (MSH, 2015). CSH advisors worked with the MOH central leadership and with the HRM, HMER, and SCM units to gather basic information that was relevant across units. For example, CSH supported updates to the national health facility inventory, which was crucial for DHIS 2 and for planning deployment of staff and distribution of drugs and health commodities. This activity also supported the mapping of water points for WASH. CSH’s technical choice to aim for interoperability of data systems in its structure and content boosted the project’s effectiveness and long-term impact. CSH allocated time and effort in consultation not only with MOH counterparts at the central level, but also with the County Health Teams (CHTs) in the six counties—and with the districts and communities they served. CSH helped to activate the County Health Board (CHB) in the base and EEF counties. It produced a written manual that spelled out the CHB’s role and its members’ responsibilities and specified both the health care providers and the community members (e.g., chiefs, faith leaders, women, and youth) who should be included in distribution. Engaging community members in planning and oversight of their health care facilities (HCF) is a key step toward building a sense of ownership. CASE STUDY 2: CSH IN LIBERIA n A–11 The assessments conducted at the start of CSH found that CHTs were disconnected from the central MOH, yet they had little autonomy to identify and respond to local health issues. Frequent drug stock-outs had damaged the credibility of the public HCFs and weaknesses in labs at the facility level led to delays in diagnosis and treatment. CSH posted mentors (HR Management, Clinical and QA/QI, M&E, and SCM) in each CHT. Just as in the central MOH, the CSH mentors were embedded in the CHT, sharing office space and sharing their expertise while accompanying their CHT counterparts in their day-to-day work. In addition, they updated and rolled out the Joint Integrated Supportive Supervision (JISS) process. QA/ QI staff were trained to assess CSH-supported facilities on their compliance with Expanded Package of Health Services requirements through quarterly site visits, and those were linked with updated guidelines and training on the Human Resources Information System (iHRIS), and the paper-based and later electronic Laboratory Management Information System (LMIS and eLMIS). “The big one was the Quality Management Strategy, and JISS, which we helped develop in CSH. The quality strategy includes establishment of a Quality Management Unit at MOH, which still exists. And the JISS still exists, and they have continued to automate that. It [automation] started three weeks ago with CDC and Global Fund money.” – KII, 2019 At the central level, CSH advisors assisted the MOH in establishing policies and guidelines and activating management tools in all of the project’s action areas. Many of these products were observed or reported to be in place a year after the CSH project was closed. MOH and former CSH staff reported that several of CSH’s counterpart staff in the HR, HMER, Quality Management Unit (QMU), and SCM units are still in place, utilizing the knowledge and skills honed in partnership with CSH advisors and mentors. In the management information system arena, across original objectives 3, 4, 5, and 7, CSH followed MOH’s preference and prioritized interoperability of systems and building on what was there rather than creating something new (KII 2019). KIs contrasted the iHRIS, LMIS, and eLMIS systems that CSH supported—and the mHero SMS-based communication system that used them—with other “fancy” IT proposals (e.g., WHO eWIS and others not implemented). They argued that building on existing MOH systems has contributed to MOH buy-in and showed that the systems could be managed by the MOH counterparts. For example, in Grand Bassa, Margibi, and Montserrado, the proportion of HCFs providing timely LMIS reports increased from 23 percent to 100 percent, from 50 percent to 86 percent, and from 51 percent to 100 percent, respectively (MSH, 2018). These factors both improved system performance and contributed to the sustainability of these system upgrades. CSH focused on institutionalizing these system improvements at the central and county level. CHS’s results supported district- and facility-level results, because other USAID￾supported activities addressed those lower levels of services and management which also require MOH guidance from above. IRC’s PACS project and its consortium and subcontractors (Jhpiego’s RHS project and Johns Hopkins University’s (JHU’s) HC3 project) supported service delivery staff, facility improvements, and community engagement at these grassroots levels within parameters set at county and central levels. CSH quarterly and annual report data indicate that all six counties progressed on most CSH objectives, but they did not progress equally. These program reports are consistent with statement by KIs that progress was more advanced in the base counties and more challenging in the three EEF counties. For example, on the CSH indicator “Percent of health workers personnel file uploaded into iHRIS system”, data provided from USAID’s Performance Indicator Database System found the average measure reported for 2017 was 54 percent, but the counties varied from 78 percent in Bong and 84 percent in Lofa to 26 percent in Grand Bassa and 33 percent in Montserrado (see Figure A2-1). Figure A2–1. Illustrative difference in performance between base and EEF counties 0 20 40 60 80 100 Bong Lofa Nimba Grand Bassa Margibi Montserrado % of health workers with iHRIS forms uploaded into iHRIS (cumulative, life of project) % of health workers with a completed personnel file A–12 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME 1 Differences in performance were explained partly by the simple difference in implementation time: the base counties had two more years of investment from CSH. Two other factors frequently cited by KIs were geographic accessibility and the intensity of USAID’s past investment. KIs who commented on high performance in Bong attributed it to the county’s proximity to Monrovia and the paved road connecting them. There is a very strong preference for living in Monrovia, and KIs stated that when posted to Bong, many staff continue to live in Monrovia and drive to work in Kakata (capital of Bong County). In addition, nearly a third of the Liberian population dwells in the base counties (Bong, Lofa, and Nimba) that have been the focus of consistent USAID support for over a decade. Infrastructure (electricity, water, cell coverage, banks, and roads) is better, making working conditions more attractive, and there is a higher density of programs, creating larger communities of practice for technical and management personnel. KIs also cited factors that detracted from CSH performance. Geographic factors (difficult access, especially in the rainy season) were paramount and directly connected with resources. For example, with regard to the MOH’s signal QA/QI intervention—requiring regular supportive supervision visits and results tracking with the JISS tool that CSH helped develop— several KIs explained that county and district-level staff could not reach all HCFs for the mandated monthly (district-level) or even quarterly (county-level) JISS visits. One claimed that MOH used an algorithm for allocating resources that did not accommodate the dramatic variation among districts in these logistical challenges; they allocated resources equally, or according to population. Thus, a district where QA/QI visits took two days (due to distance, bad roads, and impediments such as rivers without bridges) received the same funding for fuel and per diem as districts where visits took half a day. At the central level, the midterm evaluation of CSH concluded that communication and collaboration across the seven workstreams of the program was limited, leading to some inefficiencies and missed opportunities (LSA, 2017). Two KIs knowledgeable about the case reported that, while a signal accomplishment of CSH was its development of the methods and tools for supervision of facility- and district-level MOH staff (the JISS), it had not planned and budgeted for adequate supervision of its own staff who were posted at the county level. FACTORS CONTRIBUTING TO ADVANCING SELF-RELIANCE USAID’s “journey toward self-reliance” is defined in terms of two key dimensions: capacity and commitment. Both CSH program data and KI reports affirm that CSH improved the capacity of the MOH to define, manage, and resolve its health challenges. The project strengthened capacity at the individual level through pre-service and in-service training, and through long-term (years) mentoring and collaborative problem-solving in the context of everyday work (see USAID, 2011 on the importance of this). KIs in the MOH evinced self-efficacy to sustain and use the improved systems, which they attributed to CSH’s capacity-building efforts. Per USAID’s Human and Institutional Capacity Development (HICD) model, the program simultaneously addressed key environmental dimensions of capacity, assisting the MOH to institutionalize QA/QI, HR, SC, and HMER improvements in dedicated management units. In each of these areas, CSH assisted their counterparts to define clear staff roles and responsibilities and design information systems to provide data for decision-making. In addition, the project revived the MOH scholarship program and, in consultation with a revived Scholarship Committee, the MOH directed long-term training resources toward critical gaps in nursing and laboratory personnel (MSH, 2017; MSH 2018). The second dimension of self-reliance is commitment. USAID’s standard measures of commitment are at the national and institutional levels. The stability of the QMU and HRU and the incorporation of their key staff into the MOH personnel and budget testifies to commitment at those levels. MOH financing of these institutional changes is another indicator of the sustainability of their functions. The KIIs provided additional insight on commitment at the individual level. Several described commitments in a transactional way, meaning that if a person is paid they will perform assigned duties. However a few bemoaned this approach. One central MOH staff member noted that a health care supervisor would walk miles to a market or to visit family but demanded DSA and travel allowance to come to a CHT meeting at or near their place of work. He saw this as evidence that people “don’t get it. This is for us.” (KII, 2019). “Look at the e-systems – Some were there pre-EVD but many came since then. Each of them has strengthened the system. . . . Liberia could never have an outbreak like that again. The system is stronger; the leadership is stronger; people realize these are important and [are] willing to support it with staff time and money.” – KII, 2019 Building internal commitment takes time and it involves trust, which is built and maintained through consistent and open communication (TvT Associates, 1998). The CSH’s investment in CHTs and CHBs, with guidelines that require representation in those management platforms, will contribute to building that sense of ownership. CASE STUDY 2: CSH IN LIBERIA n A–13 On the other hand, there are practical limitations to what internal commitment can do—and these practical issues were at the forefront when KIs were asked about the future. All KIs claimed that the Pillar II interventions, including CSH, have placed the country in a far stronger position to respond quickly and effectively to future epidemic threats. However, QI/QA and the continuous work of keeping the management information systems updated with timely, accurate information require funding for staff and for logistics, ranging from petrol for motorbikes to internet and cellphone access. All KIs expressed doubt that the MOH would be able to keep CSH’s improvements going without a follow-on project in place to provide the funding. The CSH project’s health care financing activities reportedly created a road map to financing universal health coverage, including studies that showed Liberians are willing to pay for reliable, quality health services (MSH, 2018b), and advanced plans for a the GOL to establish a revolving drug fund, and a Liberia Health Equity Fund (MSH, 2017; MSH 2018), but these were not operationalized before the termination of the project. All are still in draft form. “There was a gap with CSH leaving, and USAID was still trying to find the follow-on project. It would have been good if the replacement project was in place to continue the gains. But with the gap . . . things are going more slowly. Some are not sustained, and even retrogressing.” – KII, 2019 CONCLUSION The CSH project was designed to provide an unusually broad range of technical and financial support to a health system recovering from crisis. It functioned as part of USAID’s mosaic of support to strengthen Liberia’s health system and made contributions at the central and county levels that have lasted past the project’s end. Working within and in constant alignment with the MOH brought important benefits, including the ability to strengthen policies and guidance of national importance, with wide support and buy-in from MOH leadership and counterparts. It also brought challenges such as delays and adaptive management to respond to MOH requests and priorities. The greater progress in the base counties than the EEF counties against CSH’s objectives provides an interesting measure of what a comprehensive HSS program can achieve in three years versus one year, and when building on stronger versus weaker county-level infrastructure. REFERENCES Ellis, S. (2006). The mask of anarchy. The destruction of Liberia and the religious dimension of an African civil war (2nd ed.). New York: NYU Press. Government of Liberia (GOL). (2015a). Agenda for transformation. Steps toward Liberia rising 2030. Monrovia, Liberia: Government of Liberia. Government of Liberia (GOL). (2015b). Liberia economic stabilization and recovery plan (April 2015). Monrovia, Liberia: Government of Liberia. Government of Liberia Ministry of Health (GOL MOH). (2015). Investment plan for building a resilient health system. Liberia 2015–2021. Monrovia, Liberia: Government of Liberia. Government of Liberia Ministry of Health (GOL MOH). (2016a). EVD survivors care and support national policy. Monrovia, Liberia: Government of Liberia. Government of Liberia Ministry of Health (GOL MOH). (2016b). Mental health policy and strategic plan 2016–2021. Retrieved from http://www.afro.who.int/sites/default/ files/2017-06/Mental%20Health%20Policy%20and%20 Strategic%20Plan%20for%20Liberia%202016%20-%20 2021%20.pdf Government of Liberia Ministry of Health (GOL MOH). (n.d.). The national health policy (2011–2021). Monrovia, Liberia: Government of Liberia. Government of Liberia Ministry of Gender and Development. (2013). National social protection strategy and policy. Retrieved from http://fscluster.org/sites/default/files/documents/SP%20 strategy%20and%20policy_%20final%20_30Oct13.pdf Liberia Statistical Analysis (LSA). (2017). USAID/Liberia. Collaborative Support for Health (CSH): Midterm performance evaluation report. Monrovia, Liberia: Social Impact. Management Sciences for Health (MSH). (2015). CSH Annual Report FY15 and Q4 Quarterly.pdf]. Management Sciences for Health (MSH). (2016). Liberia. Collaborative Support for Health (CSH) Program. Annual Summary Report for Fiscal Year 2016 (October 1, 2015– September 30, 2016) & Quarterly Report for Quarter 4 (July 1–September 30, 2016). A–14 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME 1 Management Sciences for Health (MSH). (2017). Liberia. Collaborative Support for Health (CSH) Program Annual Report for Fiscal Year 2017 (October 1, 2016–September 30, 2017) & Quarterly Report for Quarter 4 (July 1–September 30, 2017). Boston, MA: Management Sciences for Health. [File: CSH Annual Report PY 3 submitted October 30 2017.pdf]. Management Sciences for Health (MSH). (2018a). CSH EEF Quarterly Report Q3 FY18, submitted July 2018. Management Sciences for Health (MSH). (2018b). Liberia Collaborative Support for Health (CSH) Program. End-of￾Project Report (February 27, 2015–August 31, 2018). Moore, Z., Bongiovanni, A., Sadaphal, S., Hernandez, O., Espeut, D., Levin, C., & Toole, M. (2019). Performance Evaluation of USAID Ebola Pillar II Activities: Final Report. Evaluation Report to USAID, March 2019. Vienna, VA: International Business & Technical Consultants, Inc. TvT Associates. (1998). Participation toolkit: A USAID health population and nutrition officer’s guide to using participatory approaches for managing HIV/AIDS activities. Health Technical Services Project. https://pdf.usaid.gov/pdf_docs/PNACC924.pdf USAID Bureau for Global Health (BGH). (n.d.). GHET Indicators. USAID. (2011). Human and institutional capacity development handbook: A USAID model for sustainable performance improvement. Washington, DC: USAID Bureau for Economic Growth and Trade. World Health Organization (WHO). (2007). Everybody’s business: Strengthening health systems to improve health outcomes, WHO’s framework for action. Geneva: WHO. World Health Organization (WHO). (2016). CASE STUDY 3: SIAPS IN SIERRA LEONE n A–15 SIERRA LEONE CASE STUDY 3. SYSTEMS FOR IMPROVED ACCESS TO PHARMACEUTICALS AND SERVICES (SIAPS) (SIERRA LEONE) BOX A3–1. SIAPS (SIERRA LEONE) AT A GLANCE Lead Implementing Partner (IP): Management Sciences for Health (MSH) Nature of Implementation: Single sector (health), stand-alone activity Implementation Dates: October 2015–March 2018 USAID Pillar II Budget/Obligation: USD 4.5 million1 Geographical Priority Areas: Sierra Leone national (all districts, over 1,300 peripheral health units (PHUs), and 24 public hospitals) Key Activity Components: Worked closely with the Ministry of Health’s Directorate of Drugs and Medical Supplies (DDMS), Pharmacy Board of Sierra Leone (PBSL), National Pharmaceutical Procurement Unit (which, with SIAPS support, successfully transitioned into an autonomous entity known as the National Medical Supplies Agency), District Health Management Teams, and other stakeholders. The activity entailed a multi-pronged approach to (1) strengthen DDMS’s ability to effectively support health facilities, (2) strengthen “last-mile” supply chain management in districts, and (3) increase the use of information for supply decisions. Key Pillar Objectives and Outcomes to which Activity Contributed: SIAPS contributed to Pillar II Objective 2: Recover and strengthen existing institutions and infrastructure and the following Pillar II outcomes: (1) basic health service delivery restored, (2) health infrastructure strengthened, (3) health financing and resource allocation improved, and (4) information communication technology (ICT) for health systems developed. MAIN SOURCE OF INFORMATION: SIAPS Sierra Leone Ebola (Pillar II GHET) End of Project Report, February 2018. Submitted to the US Agency for International Development by the Systems for Improved Access to Pharmaceuticals and Services (SIAPS) Program. Arlington, VA: Management Sciences for Health. A Systems Approach to Pharmaceutical Management by Dr. Donna Espeut Prior to Sierra Leone’s Ebola epidemic, health care was free for mothers and children, but fraught with challenges. These included poor drug availability in health facilities and other basic health facility readiness issues, such as low/ inconsistent availability of electricity and water (Maxmen, 2013; Witter et al., 2018).2 Between 2011 and 2018, Management Sciences for Health (MSH) implemented a global, USAID-funded program called System for Improved Access to Pharmaceuticals and Services (SIAPS), Although SIAPS did not operate in Sierra Leone before Ebola’s emergence (SIAPS, 2018a; SIAPS, 2018b), in 2015, it received Pillar II post-Ebola recovery funds to implement a 1. Source: USAID’s Office of Budget and Resource Management O&E spreadsheet, September 2017. 2. The United Kingdom’s Department for International Development (DFID) has been the primary funder of Sierra Leone’s Free Health Care Initiative (FHCI), which focuses on health care for pregnant women, lactating mothers, and children under five. short-term activity to improve pharmaceutical management in Sierra Leone (see Box A3-1). Multiple USAID Pillar II activities have demonstrated the merit of leveraging preexisting programs to address Sierra Leone’s post￾Ebola recovery needs. As described further in this case study, the SIAPS Pillar II activity highlighted the value of leveraging rapport and trust between IPs and Government counterparts— not just relying on an infusion of globally honed, state-of-the-art technical expertise, processes, software, and tools—to spur broad improvements in a limited amount of time. A–16 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME 1 FACTORS CONTRIBUTING TO EFFECTIVENESS Development experts often regard inclusive, participatory approaches that engage leaders and create an enabling environment for change as paramount to the success of grassroots or community-based health programs. The SIAPS Pillar II activity in Sierra Leone demonstrated that these approaches also apply to systems strengthening, particularly when there are relatively limited resources and time allocated for implementation. There were multiple technical components to the SIAPS activity in Sierra Leone, but at the root of the activity’s effectiveness was the people-centered approach undertaken by MSH. Deployment of respected pharmaceutical experts with local contextual knowledge of pharmaceutical management and, more broadly, the health sector in Sierra Leone—as well as working relationships in place in Sierra Leone—proved to be critical success factors in activity performance. In Sierra Leone, SIAPS experts were able to draw upon SIAPS’s constellation of globally tested, state-of-the-art approaches and tools related to pharmaceutical management, as described in this case study. The attention paid to relationships—both within the Ministry of Health and Sanitation (MOHS) and with other stakeholders in the country’s pharmaceutical/supply-chain management landscape—enabled SIAPS to foster buy-in and fast-track implementation of key interventions, as well as spur organizational development changes within the MOHS. Both activity documentation and consultations with MOHS stakeholders have highlighted SIAPS’s objectively verifiable achievements in Sierra Leone (see Table A3-1). SIAPS helped the MOHS create institutional structures to support effective pharmaceutical management, cultivated leadership skills among pharmacists, and strengthened the MOHS’ ability to keep quality drugs in stock and to identify and remove expired products from its shelves. It also introduced protocols, tools (e.g., treatment registers) and an evidence-driven system (Continuous Results Monitoring and Support System (CRMS)) for supportive supervision and performance monitoring. In addition to what was achieved, how SIAPS achieved those outcomes warrants documentation. One MOHS senior official consulted for this study noted that there were other actors in the pharmaceutical landscape, but those actors tended to have a very limited focus (e.g., reproductive health). According to that key informant (KI), “MSH was looking at things holistically to improve the entire supply chain.” SIAPS’s achievement of outcomes can be largely attributed to factors that were inherent to SIAPS’s activity design and implementation approach. SIAPS’s people-centered approach entailed leveraging relationships and trust; bolstering leadership, not just technical skills; and decentralizing technical and capacity-building support. Leveraging relationships and trust As mentioned by one MOHS key informant and one Pillar II implementation KI, rapport, trust, and mutual respect were critical success factors in SIAPS’s Pillar II performance. The SIAPS Country Director (a respected, senior-level Sierra Leonean pharmacist with both UN and private-sector experience), and the MSH SIAPS Headquarters focal point (who had Table A3–1. Selected Indicators of the SIAPS Ebola Pillar II Activity in Sierra Leone Indicator Baseline Value Endline Value % of facilities experiencing stock-outs of key maternal and child health drugs/commodities 66% 32% % of health facilities that have properly remitted expired products to the district level in the past six months 37% 72% No. of pharmaceutical management guidelines/standard operating procedures developed/ updated and submitted for adoption 0 4 No. completing training of trainers for Leadership Development Program for Pharmacists 0 17 No. of health facilities using revised tools and guidelines related to pharmaceuticals 0 1,300 Examples of other SIAPS achievements in Sierra Leone: Supported (1) development of a new organogram for the MOHS’s DDMS; (2) establishment of National Quantification Committee and seven disease-specific technical working groups on quantification; (3) creation of an autonomous Nation￾al Medical Supplies Agency, which was voted by lawmakers into existence in 2017; and (4) development of Sierra Leone’s National Antimicrobial Resistance Strategy. SOURCES: (1) SIAPS. (2018). SIAPS Sierra Leone Ebola (Pillar II GHET) End of Project Report, February 2018. Submitted to USAID by the SIAPS Program. Arlington, VA: Management Sciences for Health; (2) Systems for Improved Access to Pharmaceuticals and Services. (2018). Toward Building Resilient Pharmaceutical Systems: SIAPS Final Report. Submitted to the US Agency for International Development by the Systems for Improved Access to Pharmaceuticals and Services (SIAPS) Program. Arlington, VA: Management Sciences for Health; (3) The National Medical Supplies Agency Act of 2017. Retrieved from http://www.sierra-leone.org/Laws/2017-11.pdf. Accessed May 28, 2019. CASE STUDY 3: SIAPS IN SIERRA LEONE n A–17 previously worked in Sierra Leone), were known and trusted professionals. The close day-to-day relationships between SIAPS and DDMS, in particular, helped to fast-track the rollout of SIAPS interventions to all levels and enabled SIAPS to effect major changes in how DDMS/MOHS was structured and later operated (SIAPS, 2018b) (see quotations next page). According to one Pillar II KI, trust and collaboration were also at play in how the activity worked with the Ministry of Finance and both the pharmaceutical policy (DDMS) and regulatory (PBSL) players to identify and remove expired drugs from PHUs. It should be noted, however, that SIAPS did not pursue this “reverse logistics” endeavor alone. As a member of Sierra Leone’s Free Health Care Initiative (FHCI) Forum, SIAPS worked in concert with other members, as well as key country players in the supply-chain management arena such as UNICEF, DFID, Crown Agents-International Procurement Agency, and Sierra Leone’s Central Medical Store (SIAPS, 2018a). Bolstering leadership, not just technical skills Capacity-building and skills transfer were prominent components of SIAPS’s work. In strengthening individual capacity related to pharmaceutical management, SIAPS deemphasized didactic training and focused more on cultivating home-grown leaders in the pharmaceutical sector who would, in turn, mentor other professionals (SIAPS, 2017b). Leveraging its well-established Leadership Development Program, MSH adapted a Pharmaceutical Leadership Program (PLDP) for Sierra Leone and for two other SIAPS (non-Pillar II) countries, South Africa and Lesotho (SIAPS, 2018b). Under SIAPS Sierra Leone, 17 health care professionals completed the PLDP, which was an amalgamation of building sound pharmaceutical management skills and strengthening leadership practices (SIAPS, 2018b). Decentralized support One implementation KI mentioned that Ebola highlighted the lack of alignment between what was occurring at the central level and district pharmaceutical management realities. Hence, SIAPS adopted a bifurcated approach that entailed both setting standards and strengthening institutional capacity at the central level, and capacity building and technical support at the district level (SIAPS, 2018a). SIAPS worked with District Health Management Teams to address “last-mile” supply-chain management issues that were preventing some beneficiaries from receiving quality essential medicines when they sought care. In addition to developing and distributing user-friendly “diagnosis and dispensing treatment and summary report registers” to over 1,200 PHUs, SIAPS provided training-of-trainers support to district pharmacists and district information and monitoring and evaluation officers on the use of the new registers, with those trainers then cascading skills to 1,755 health workers (SIAPS, 2018a). SIAPS’s decentralized support was reflected in the role that it played in revitalizing Drug and Therapeutics Committees (DTCs) as quality and oversight structures in hospitals. This involved not only supporting strategic planning related to those structures, but donating computers and other office equipment, supplies and furniture, and medical reference materials to each hospital (SIAPS, 2017a). Information technology also played a role in SIAPS’s work. The activity introduced a suite of Pharmaceutical Management Information System tools to Sierra Leone under its Pillar II activity. Technology was not presented as the panacea to Sierra Leone’s supply-chain challenges, but rather as an attractive tool to support evidence-based decisions related to pharmaceutical management. At the cornerstone of that effort was the introduction of Continuous Results Monitoring and Support System (CRMS), a supportive supervision and performance￾improvement approach centered on key pharmaceutical management indicators (SIAPS, 2018b). On Advancing Pharmaceutical Management after Ebola: “MSH was here. They developed the organogram for the Directorate for Drugs and Medical Supplies . . . to come up with the TOR [Terms of Reference] for each department. Even some tools for rational use of health commodities . . . a Treatment Register, which captured the consumption rate, also quantities . . . They did some supportive monitoring and supervision. But what was so good, really good, was CRMS—Continuous Results Monitoring and Support System. Everyone talks about that. It helps us track utilization of the drugs, where they were dispensed, things like that . . . CRMS—all the DMOs [District Medical Officers] really appreciate this.” – KII, 2019 A–18 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME 1 FACTORS CONTRIBUTING TO ADVANCING SELF-RELIANCE 3. Mentioned during one KII with a Pillar II implementer. As an activity that focused on systems strengthening, SIAPS has a legacy of improved pharmaceutical management that is still observed in present-day Sierra Leone. The activity’s contributions to Sierra Leone’s journey to self-reliance relate primarily to institutional capacity (in particular, the MOHS DDMS) to lead and govern pharmaceutical management. However, SIAPS’s work also served as the impetus for increased commitment, for example, in the form of efforts to mobilize additional donor resources for DDMS that built on the Pillar II achievements of SIAPS (e.g., through The Global Fund). As already described in the preceding section on effectiveness, the people-centered approach yielded “quick wins” (i.e., achievement in a relatively short period of implementation) with respect to improved pharmaceutical management; the institutional support provided by SIAPS to the MOHS was an important driver of improved pharmaceuticals management practices that still exist today. Some technical choices such as CRMS— the evidence-driven approach to supportive supervision and performance monitoring and improvement—were well received by Government counterparts. However, issues of (1) sustained financing/resource allocation and, more broadly, a continued dependence on donor investments; (2) the human dimension (e.g., having sufficient numbers of both pharmacists and personnel who can support the data management requirements of evidence-driven pharmaceutical management); and (3) contextual factors (e.g., power supply and internet connectivity in health facilities) remain as salient sustainability determinants and thus key influences in moving the country toward self-reliance. On Recognition of SIAPS’s Legacy: “MSH delivered. We actually gave them space here. They were our partners, day￾to-day. That is why we are in a better position today.” – KII, 2019 Day-to-day institutional support was the key modality for strengthening institutional capacity—and the most prominent sustainability determinant. As mentioned by one senior-level stakeholder within the MOHS, SIAPS helped to address organizational development issues within the DDMS, influencing both organizational structure and mode of operations. According to a senior MOHS official, the Treatment Register developed under SIAPS is still in use today. Recognizing that there are other players in the pharmaceutical management landscape, SIAPS also held meetings with the DDMS and partners (e.g., World Bank, UNICEF, Global Fund, FHCI Forum) to explore an exit strategy that would entail some partners including activities in their support plans for at least the first year after SIAPS ends (SIAPS, 2018b). Additionally, before ending SIAPS supported the HIV quantification technical working group to finalize a three-year (2018–2020) quantification that the Government of Sierra Leone later included in a subsequent Global Fund proposal (SIAPS, 2018b). There is some evidence of efforts and/or effects persisting after the activity ended. In 2018, after SIAPS had ended in Sierra Leone, MSH noted that CRMS activities were ongoing in all 13 districts, covering 85 percent of PHUs (MSH, n.d.). However, in 2019, challenges related to sustainable financing have come to the fore. A senior MOHS official mentioned that when SIAPS ended, Global Fund resources assisted with maintaining CRMS practices, but that there is now a need for more support in that arena. “I know it is functioning. Just a while ago, somebody told me the rational use of medicines component of the SIAPS project has now taken higher ground. They are now using information to show that irrational use of drugs is costing the Government a lot of money with hard evidence. And that’s because there is now a unit that monitors rational use, which was not the case in the past.” – KII, 2019 Although SIAPS/MSH no longer has a presence in Sierra Leone, there is some continuity of state-of-the-art thinking and experience via the former Country Director, who is now the president of the Pharmaceutical Society of Sierra Leone and is also Chairman of the Board of Directors of the National Medical Supplies Agency (NMSA) (Pharmaceutical Society of Sierra Leone, n.d.).3 However, note that NMSA is largely donor-funded (by USAID and other development partners such as DFID, The Global Fund and The World Bank) (MOHS, 2017). A recently published journal article examining issues of fragmentation within Sierra Leone’s health system specifically raised issues related to the national medical supply and the need for further strengthening of governance and accountability mechanisms (Barr, Garrett, Marten, & Kadandale, 2019). In addition to the above factors, the human dimension is another key consideration when examining SIAPS’s contributions to Sierra Leone’s journey to self-reliance. A senior MOHS official described how there are challenges with the quantity of qualified personnel, such as pharmacists, and also the paucity of formalized mechanisms to support career progression and retain qualified staff in the public health sector. CASE STUDY 3: SIAPS IN SIERRA LEONE n A–19 One Pillar II implementer had a more positive perception of the state of human resources related to pharmaceutical management, noting that how the existing pool of human resources has improved. More specifically, rather than simply maintaining a pool of professionals at the central level, the DDMS has now assigned specific responsibilities/TOR to each professional, with expectation that each staff person will deliver vis-à-vis those responsibilities. In summary, both through capacity and commitment, SIAPS made contributions to Sierra Leone’s journey to self-reliance. However, its achievements should be regarded as “first executable steps” toward sustainability. There remains a need for sustained commitment and investment to propel the country further. “The stories, the narratives are changing, so you know something good is happening. You hardly can now hear about things like ‘Oh I went to a certain hospital and there were no drugs.’ Before it was regular, it was a regular phenomenon, but that is changing. . . . That is something worth talking about because it is working.” – KII, 2019 REFERENCES Barr, A., Garrett, L., Marten, R., & Kadandale, S. (2019). Health sector fragmentation: Three examples from Sierra Leone. Globalization and Health, 15(1), 8. doi:10.1186/s12992-018- 0447-5. Management Sciences for Health (MSH). (n.d.). Sierra Leone Country Page. Retrieved from: https://www.msh.org/our-work/ country/sierra-leone Maxmen, A. (2013). Sierra Leone’s free health-care initiative: work in progress. The Lancet, 381(9862), 191–192. Ministry of Health and Sanitation. (2017). National Medical Supplies Agency of Sierra Leone: NPPU Reform Phase II Draft Operational Plan 2017–2018 [draft]. Retrieved from https:// mohs2017.files.wordpress.com/2017/06/nmsa-operational￾plan-2017-2018-draft.docx National Medical Supplies Agency Act of 2017. Retrieved from: http://www.sierra-leone.org/Laws/2017-11.pdf. Pharmaceutical Society of Sierra Leone. (n.d.). The PSSL Gets a New Team. Retrieved from: https://salonepharmacy.org/ The%20Pharmaceutical%20Society%20of%20Sierra%20 Leone%20(PSSL)%20gets%20a%20new%20team.html Systems for Improved Access to Pharmaceuticals and Services (SIAPS). (2018a). Sierra Leone Ebola (Pillar II GHET) End of Project Report, February 2018. Submitted to the US Agency for International Development by the Systems for Improved Access to Pharmaceuticals and Services (SIAPS) Program. Arlington, VA: Management Sciences for Health. SIAPS. (2018b). Toward Building Resilient Pharmaceutical Systems: SIAPS Final Report. Submitted to the US Agency for International Development by the Systems for Improved Access to Pharmaceuticals and Services (SIAPS) Program. Arlington, VA: Management Sciences for Health. SIAPS. (2017a). Strengthening the Pharmaceutical System in Sierra Leone after Ebola, August 2017 [Factsheet]. SIAPS. (2017b). USAID-SIAPS Sierra Leone Post-Ebola Pharmaceutical Management Systems Strengthening Project Progress Update (A Presentation to USAID), April 2017. Witter S., Brikci N., Harris T., Williams, R., Keen, S., Mujica, A., … Renner, A. The free healthcare initiative in Sierra Leone: Evaluating a health system reform, 2010‐2015. Int J Health Plann Mgmt. 2018;33:434–448. https://doi.org/10.1002/ hpm.2484. A–20 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME 1 LIBERIA GUINEA CASE STUDY 4. RESTORATION OF HEALTH SERVICES (RHS) BOX A4–1. RHS AT A GLANCE Lead Implementing Partner (IP): Jhpiego Corporation Nature of Implementation: Single sector (health), activity part of a larger program (supplemental funding to the ongoing MCSP program for adding an RHS project) Implementation Dates: Guinea July 2015–December 2016, Liberia August 2015–June 2018 USAID Pillar II Budget/Obligation: USD 4 million (Guinea); USD 10.5 million (Liberia)1 Geographical Priority Areas: Regional (Guinea and Liberia), areas most affected by Ebola Key Activity Components: Guinea: Improve adherence to infection prevention and control (IPC) practices and restore quality health services (e.g., Reproductive, Maternal, Newborn, and Child Health (RMNCH) and IMNCI) in 221 focus facilities. Liberia: Improve adherence to IPC practices and restore quality health services in 77 health facilities Key Pillar Objectives and Outcomes to which Activity Contributed: In both countries, RHS contributed to two Pillar II Objectives: 1 (“Prevent the loss of development gains”), and 2 (“Recover and strengthen existing institutions and infrastructures”). Restoration of Health Services in Guinea and Liberia by Dr. Karim Sahyoun The Ebola epidemic had a drastic impact on health service utilization and quality in West Africa. Communities lost trust in health facilities, resulting in dramatic reductions in health service utilization in the affected areas for routine prevention and treatment—such as antenatal and delivery care for pregnant women, family planning, and treatment of diarrheal disease, malaria, and other communicable diseases. Delivery care dropped 81 percent in N’zérékoré (Guinea) and 74 percent in Conakry within a period of 15 months (Jhpiego, 2019). Reporting rates had dropped to only 26 percent from health facilities (Jhpiego, 2019). In Liberia, only 44 percent of the country’s health facilities remained functioning, and most essential primary care services had come to a standstill, while staffing declined (Jhpiego, 2018b). In both countries, the Maternal and Child Survival (MCSP) Restoration of Health Services (RHS) project in Guinea and Liberia targeted the areas most affected by Ebola, covering 20 percent of the counties in Liberia and 66 percent of the population in Guinea (Jhpiego, 2018b; Jhpiego, 2019). Their objective was to increase the capacity of the Ministry of Health (MOH) to consistently and 1. Source: USAID’s Office of Budget and Resource Management O&E spreadsheet, September 2017. effectively deliver high-quality health services in seven technical areas (see Box A4-1). MCSP had been already operating within Liberia and Guinea on a number of activities. In both countries, MCSP was requested to develop an additional activity to address the second-order impacts of the Ebola epidemic as part of Pillar II, specifically focusing on the restoration of critical non-Ebola health services. In both countries, the upgrading of infection prevention and control (IPC) practices for fighting Ebola and other infectious diseases and the restoration of reproductive, maternal, newborn, and child health (RMNCH) services were core objectives. Jhpiego aimed at renewing confidence in the national health systems by improving quality and accessibility. The RHS project operated for 37 months in Liberia and a much shorter period (18 months) in Guinea. Sierra Leone was not included in this study, as Jhpiego received minimal funding from USAID Pillar II for activities there (RHS was mostly implemented through a different partner, JSI Research and Training Institute). CASE STUDY 4: RHS IN GUINEA AND LIBERIA n A–21 FACTORS CONTRIBUTING TO EFFECTIVENESS Throughout the evaluation in Guinea, improvements in the health system were frequently reported in FGDs among community members, local stakeholders, and local health staff. In general, at the end of the MCSP/RHS programs, there were significant improvements reported at most supported facilities. In Liberia, those improvements included the availability of health workers, equipment and supplies, basic infrastructure (functional incinerators (57 percent to 87 percent); functional triage (41 percent to 74 percent); isolation unit (27 percent to 74 percent)); and functional water sources onsite (from 63 percent to 82 percent) (Jhpiego, 2018a). Also reportedly improved was service provision for essential RMNCH interventions and adherence to clinical standards to ensure quality of care (Jhpiego, 2018b). These improvements in turn led to an increase in confidence and utilization. In Liberia, a significant rise in the use of RMNCH services at supported facilities were observed from the baseline of the project. For example, the numbers of women receiving IPT2+ and skilled delivery services and children receiving Penta3 vaccination and diagnosis/treatment for pneumonia doubled between baseline (April–June 2015) and endline (October– December 2017). In Guinea, the share of women delivering at health facilities increased from 85 percent to 96 percent; patient utilization across all covered regions increased by nine percent, with the most prominent increases in Kindia by 62 percent (Jhpiego, 2017; Jhpiego, 2019). Leveraging existing programs A critical strategy in both countries contributing to the success of the RHS and MCSP projects was the leveraging of existing programs such as USAID and OFDA investments in IPC in Guinea, the strengthening of MNCH through MCHIP, and/ or the StopPalu malaria project. In fact, the RHS project was so well-integrated with others that respondents in communities, health centers, or divisions of the MOH sometimes had difficulty distinguishing among these, and thus trouble identifying a project’s specific impact. Complementing existing programs in this case built on the recognition of the program by local authorities, a presence in the targeted areas, and relationships with critical stakeholders. It therefore minimized the time to get a project started and increased the efficiency of its implementation. Needs-based programming In both countries, baseline assessments were carried out to understand the situation and adapt the project to the specific need of the health facilities. Staff from the local health centers in Guinea stated that they valued the closeness of Jhpiego staff who came to their facilities, listened, and addressed their specific needs throughout the project. In addition, staff from the MOH as well as local health facilities applauded the technical competence of Jhpiego staff, ensuring high-quality support. Integration with other Pillar II activities A major contribution to the success of the RHS projects has been the close integration with other Pillar II activities, especially with Health Communications Capacity Collaborative (HC3) in both countries and System for Improved Access to Pharmaceuticals and Services (SIAPS) in Guinea. HC3 has been pivotal in reviving or establishing community committees to allow health centers to better engage and communicate with their communities, receive feedback, and ensure participation of communities. In Guinea, health center staff explained that, for them, Jhpiego was better known because of its permanent presence in the region (KII, 2019). Sustainability After areas were declared Ebola-free, the percentage of MCSP￾supported hospitals and health centers that met minimum IPC standards immediately started decreasing—in the case of Guinea, to 45 percent out of a target of 75 percent (Jhpiego, 2019). The evaluation team in Guinea also observed that in several health centers and hospitals, handwashing facilities were still in place but water or soap were missing. If available, only a few seemed to use these. In the case of Liberia, a different result was observed: the median score on the Safe Quality Health Services standards for facilities assessed increased to 82 percent, demonstrating maintenance and adherence long after Ebola had ended (Jhpiego, 2018a). A challenge to the sustainability of the achievements has been voiced at a health center in Guinea: successes are starting to be watered down because of a continuously increasing population in the area. The infrastructure, available staff, drugs, and materials are not enough to meet the needs, according to KIIs with health center staff and local committees. KIIs stated that to sustain quality improvements, continued investments A–22 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME 1 would be required to meet increasing demands; numbers of staff are insufficient and the physical infrastructure is used beyond its intended limits. A continued lack of funding for essential drugs and materials continues to limit the services despite a new computerized management system being in place (allowing optimal use of the available goods, but it cannot prevent stock￾outs) (KIIs with staff at facilities and pharmacy, 2019). Staff explain that they struggle to decide how to allocate limited essential drugs and materials and whom to require to purchase these privately. A further challenge to sustainability in both Liberia and Guinea is the lack of funding for meeting recurrent costs (including transport) or maintenance costs for infrastructure, as well as technical equipment. One example of a past investment that had not been sustained at a health center in Guinea was a generator, which was still in place but could not be operated due to the lack of funding for fuel. The RHS project at least ensured a limited availability of electricity, allowing the operation of the most critical MNCH services by installing solar panels specifically dedicated to these services explained by health center staff. While overall significant progress in the restoration of health services was achieved in both countries, medium- and longer￾term success depends on factors that were beyond the scope of this project: health funding and staffing. FACTORS CONTRIBUTING TO ADVANCING SELF-RELIANCE It is challenging for a short-term intervention to contribute to a country’s self-reliance, especially when change requires intervening into a multi-layered complex health system. Three major factors can be identified for the RHS projects that significantly increased their potential for sustainability and for contributing to increased levels of self-reliance. Embedding the RHS projects into ongoing MCSP programs In both countries, Pillar II directed additional funds toward ongoing long-term interventions, increasing the potential for sustainable impact, and therefore for contributing to the countries’ self-reliance. Specifically, RHS increased the potential to provide basic health services and contributed to the countries’ ability to prevent and manage epidemics. Embedding the projects into a “mosaic” of activities The most intuitive way to view the project’s actual contribution to self-reliance is to view it as one integral part of other Pillar II health projects that were carried out in the respective countries— as well as the other projects carried out by the IP Jhpiego. RHS represented the critical last mile, connecting changes in local health infrastructures and communities to strengthen staff, processes, and capability to implement changes that had been designed at the national and regional levels (through, e.g., the SIAPS and Health, Finance, and Governance (HFG) projects). Funding of health services In the project “mosaic” of Pillar II, HFG in Guinea played a critical role in increasing commitments health funding (HFG case study) to ensure that the country budgets sufficient funds. While significant progress has been made, this has not been enough to ensure the necessary level of funding for the health sector required to sustain a number of outcomes of the RHS projects. This challenge, according to various health facility and MOH staff, results in a decrease of the project’s outcomes and therefore in its contribution to self-reliance as well. When working with Pillar II’s multi-level approach, the interdependence of different activities magnifies the importance of each individual project. The integration of different activities across different organizations and institutions at the same time creates a potential to break down barriers around established silos within the MOH and between different layers (local to national), but also calls for each intervention to fit seamlessly into its place with appropriate funding and governance. REFERENCES IBTCI. (2019). SIAPS case study. Second Performance Evaluation of USAID Ebola Pillar II Activities: Final Report, Annex Vol. 1, Annex A. Case Studies. Vienna, VA: International Business & Technical Consultants, Inc. Jhpiego Corporation. (2017). MCSP Guinea: Restoration of health services, country summary, March 2017. Jhpiego Corporation. (2018a). MCSP Liberia: Restoration of health services endline assessment findings in brief. Jhpiego Corporation. (2018b). Liberia: Restoration of health services – end of project report. Jhpiego Corporation. (2019). Guinea Ebola response and recovery: Restoration and improvement of reproductive, maternal, newborn, and child health services (RHS) – end-of￾project report. CASE STUDY 5: CEPPS IN GUINEA n A–23 GUINEA CASE STUDY 5. CONSORTIUM FOR ELECTIONS AND POLITICAL PROCESS STRENGTHENING (CEPPS) BOX A5–1. CEPPS AT A GLANCE Lead Implementing Partner (IP): National Democratic Institute (NDI) Nature of Implementation: Single sector (governance), activity supporting and integrating with the work of of the Independent Electoral Commission (CENI) and the Code of Conduct. Implementation Dates: February 2015–February 2019 (initially until December 2016) USAID Pillar II Budget/Obligation: USD 2 million (up to 2017)1 Geographical Priority Areas: Guinea national Key Activity Components: Promoting dialogue and consensus around electoral processes, strengthening inter-party dialogue, increasing civic and voter education, enhancing citizen engagement in and oversight of election systems, and strengthening CENI’s institutional and technical capacity to administer elections Key Pillar Objectives and Outcomes to which Activity Contributed: CEPPS contributed to Pillar II Objective 2 (“Contribute to the recovery and strengthening of existing institutions and infrastructure”). Working to Build Smoother Electoral and Democratic Processes in Guinea by Dr. Karim Sahyoun Guinea has been plagued by tensions,1 violence, and delays throughout election processes since its return to civilian rule in 2010. Decades of authoritarian leadership were ended by a military coup in 2008 (Freedom House, 2019). There are currently more than 130 registered political parties, most of which have clear ethnic or regional bases (Freedom House, 2019). There were fears that a repetition of the severe violence before the 2013 parliamentary elections during the 2015 presidential election and the long-awaited local elections could slow or hinder post-Ebola efforts aiming at reestablishing critical citizen services (CEPPS, 2016). Funded by the U.S. Agency for International Development (USAID) Pillar II Ebola Recovery effort, the National Democratic Institute (NDI)’s Consortium for Elections and Political Process Strengthening (CEPPS) has contributed to stabilizing Guinea’s fragile political context, a critical precondition for the success of the post-Ebola recovery efforts in the country. NDI began its activities in Guinea in 1998, training political party poll watchers and conducting voter education prior to the presidential election that year. Since the country began its democratic transition in 2008, NDI has promoted dialogue 1. Source: USAID’s Office of Budget and Resource Management O&E spreadsheet, September 2017. among political parties and supported the peaceful conduct of elections. Key vehicles have been the formation of an Inter￾Party Working Group (IPWG) and the development of a Code of Conduct for political parties, supported by monitoring committees ensuring adherence to the code (NDI, 2019). NDI has further supported the Independent Electoral Commission (CENI) and strengthened local civil society organizations (CSOs) in monitoring elections and engaging in voter education (NDI, 2019). Via Pillar II, USAID took the approach of directing funds to NDI’s ongoing governance work with an additional commitment. CEPPS significantly expanded NDI’s strategic activities. The underlying assumption was that credible, inclusive electoral processes will lead to long-term success for Guinea’s democratic transition (CEPPS, 2018). The program aimed to achieve this through its five objectives: 1. Promote political dialogue and build consensus around peaceful and legitimate electoral processes. A–24 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME 1 2. Strengthen a political party system through improving political party function and promotion of intra-party dialogue and consensus-building. 3. Expand civic and voter education. 4. Enhance citizen engagement in and oversight of election systems, processes, and management. 5. Strengthen the institutional capacity of the CENI and its technical staff to credibly and transparently administer elections. 2. CEPPS Quarterly Report Q2 2018 reports that 24 percent of the candidates for the communal elections were female and 13 percent were under the age of 35. Data from the last communal election in 2005 was not provided. In addition to CEPPS, USAID’s Ebola Pillar II also supported two complementary activities implemented by Search for Common Ground (SFCG): (1) Rebuilding Together: Community-Driven Reconciliation and Enhanced Communication in Guinea Forestière, and (2) Votons pour la Paix: Supporting Participative, Transparent, Credible, and Peaceful Elections in Guinea (SFCG, 2017). FACTORS CONTRIBUTING TO EFFECTIVENESS “We have strengthened capacities—youth and women, all members were trained. We now take a role in and have contributed to pacifying situations. Further, we organize press conferences for denouncing any violence.” –KII, 2019 CEPPS, according to all stakeholders, has managed to achieve significant positive changes. Key achievements will be highlighted before exploring factors that have enabled these changes. Promoting political dialogue CEPPS has strengthened the IPWG, which “had been established as a mechanism for constructive dialogue among political parties and the election management body, allowing all stakeholders to reach agreement and resolve critical differences on critical issues before they escalate” (CEPPS, 2018). It has allowed party members to raise and discuss concerns and seek clarification with the CENI. Strengthening critical institutions for election processes CEPPS has worked through and strengthened key national institutions, specifically the Code de Bonne Conduite (Code of Conduct) and CENI. CEPPS has supported the Consortium of Women/Girls of Political Parties in Guinea and the National Forum of Youth of Political Parties (FONAJEP), strengthening members of both structures to take on leadership roles in their parties and as candidates in elections (CEPPS, 2018). This has resulted in an increase of young and/or female candidates in the communal elections in February 2018 (CEPPS, 2018).2 Code de Bonne Conduite This structure was proposed by NDI and formed by a number of party members in 2008. Its objective is to ensure peace before, during, and after elections, and 109 parties are currently members. The Code was developed in 2008 (KII, 2019). CEPPS has allowed NDI to assist parties in revising the code, the bylaws of the Code of Conduct Monitoring Committee, and to increase the awareness of parties on the Code in the regions (KII, 2019). While initially the hope was been to cover all prefectures, funds allowed Code to cover two prefectures per region; however, Conakry was fully covered (KII, 2019). The Code’s work includes visiting member parties to remind them about the commitment they have made by signing the code (KII, 2019). CEPPS has provided support to train trainers in election observation, thus enabling parties to oversee the election process (CEPPS, 2018). Before elections, it further supported developing and distributing communiqués to institutions, parties, ministries, and departments (CEPPS, 2018). The collaboration with CEPPS through code of conduct monitoring committees, has also increased its capacity to address tensions and incidences (including outbreaks of violence) post-election, take a key role in an assessment of the situation, and involve critical actors to defuse tensions. CENI NDI has worked with and supported CENI since 2010. The collaboration has included strengthening capacities, facilitating dialogue and exchange between CENI and political actors, and increasing the visibility of CENI’s activities (KII, 2019). CEPPS has enhanced NDI’s support to strengthen CENI as a key structure for democratic processes. For example, civic education through radio spots has broadened CENI’s visibility and the understanding of electoral processes (KIIs, 2019; CEPPS, 2018). CASE STUDY 5: CEPPS IN GUINEA n A–25 CEPPS has assisted CENI in a number of areas, including training women leaders and candidates, developing posters and documents, providing financial support, and facilitating exchange between CENI and other political actors through the inter-party committee (KII, 2019). Support has been provided to evaluate elections, including during the 2015 presidential and 2018 communal elections (KIIs, 2019; CEPPS, 2018). These processes have brought together actors that initially had challenges in collaborating to learn key lessons, as well as address identified critical areas, including changes to the election law (KII, 2019). The role and support of CEPPS in the past election was seen as vital; some respondents stated that without CEPPS’s presence, the past election (2018) would not have taken place and that it has significantly increased the credibility of the election (KII, 2019). Despite CENI being an independent body, it has faced a number of critical challenges. For example, after the communal elections in 2018 the commission became so tense that most actors refused to work with CENI (KII, 2019; CEPPS, 2018). NDI has reportedly been able to reduce these tensions (KII with CENI representative, 2019). A CENI representative reported that “the head of the opposition one day refused to participate in a meeting. After NDI communicated with him, he was willing to participate and was even happy with the results” (KII, 2019). Currently, NDI KIs state that they doubt whether Guinea is able to yet organize transparent elections without external oversight. There is still a lack of trust in the local election results, “mainly linked to allegations of fraud after the closing of polling stations and a lack of transparent communication by CENI” (CEPPS, 2018). “A key area identified was gaps and weaknesses in our constitution [and] laws. We have introduced amendments to the election law trying to address insufficiencies of the election law.” – KII, 2019 Civic and voter education CEPPS collaborated with CENI on civic and voter education (KIIs, 2019; CEPPS, 2018). This has comprised training for parties and forums, including in rural areas. CEPPS’s approach has strengthened the capacities of the Code and CENI while educating communities, youth, and women (KII, 2019). Although people still demonstrate after elections, the major change is that they now turn increasingly toward the institutions to sort out the issues, indicating higher levels of trust (KII, 2019). Enabling factors for change NDI is highly regarded and accepted by all political sides in the country, which allows it to play a critical role in bringing opposing parties together, defusing tensions, and ensuring progress in the development of democratic institutions and processes (KIIs, 2019; FGD participants, 2019). NDI has observed that Ebola was a threat that enabled different parties to come together to address a specific issue (KII, 2019). Representatives of the major political parties expressed that the involvement of key stakeholders in CEPPS and high-quality training has been important for its success (FGD participants, 2019). NDI reports that its funding and staff limitations are the most relevant factors for not having achieved more change; they have reportedly not been able to support the development of inter￾party committees at the prefecture level, as partners had hoped, and have not been able to facilitate all required coaching and follow-up. Some parties and their leaders have not fulfilled commitments they had made regarding women’s participation in the elections or quotas for positions or on lists (KII, 2019). “The presence of NDI, seeing them backing us, reassures different actors and enables their involvement.” – KII, 2019 A–26 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME 1 Sustainability A major challenge to sustainability all actors reported is that most of the changes were only partially achieved; they state that ending support poses a threat to the achievements. Resources are lacking for critical follow-up and coaching and to ensure that costs are met for meetings and other visits (KIIs, 2019). A clear and visible achievement of CEPPS is that a higher number of women have been elected. For this to become a long￾term change, however, additional efforts are required to meet the target of 30 percent female representation and to ensure that political parties hold to their commitments (KII, 2019). It is reported that some critical pieces cannot be continued or further developed without further support. It is important in all areas to have a presence and to be able to act when necessary. CSOs would be able to take on that role; however, this requires the capacity to coach them. While NDI has been working with civil society actors and has strengthened them, many achievements seemed to be highly dependent on NDI, its competencies, and specific identity as an international actor. Civil society is still seen as weak (KII, 2019). For developing long-term accountability and trust into a democratic system, NDI KIIs stated that the focus has to shift from solely addressing the pre-election and election phase to actually establishing a continuous monitoring system allowing the electorate to understand whether politicians are keeping the promises they had made throughout the election. According to a CENI representative, only “the coming elections will help to understand the level of sustainability of some of the results” (KII, 2019). FACTORS CONTRIBUTING TO ADVANCING SELF-RELIANCE Self-reliance in the context of CEPPS can be seen from two different, equally important perspectives: the country being self￾reliant in its democratic processes (e.g., not requiring external support during elections); and a stable, effective democratic system as a basis for development. It has already been discussed that change has been a slow process. It involves changes of the political culture and institutions that are rooted in established traditions and culture. All stakeholders interviewed agreed that CEPPS has made positive contributions to self-reliance as an ongoing process. CEPPS has contributed to strengthening important capacities among key institutions, parties, and civil society actors. CENI and the Code have increased competencies and experience in working with and training their target groups (KIIs, 2019; CEPPS, 2018). Political parties are now able to send their own observers without external support (KIIs, 2019). Leaders are now better able to come to agreements, following the approaches on which they have been trained (KIIs, 2019). In regard to sustained capacities, however, CENI faces a challenge because of the limitation of its mandate to seven years, when it is then required to change all members (KII, 2019; CENI, 2017). “A major challenge is that problems occur after elections, usually when the support ends and all observers leave.” – KII, 2019 Funding To fulfill their critical roles, the Code, CENI, FONAJEP, and civil society actors all require financial resources to participate in meetings, facilitate trainings, run their offices, and produce informational materials. All have voiced that the funding they have does not allow them to fulfill all their roles, indicating a limitation to self-reliance (KIIs with NDI, Code, and CENI representatives, 2019; CEPPS, 2018). FONAJEP representatives expressed that they now were able to meet and plan together, but due to financial limitations were not able to implement their plans. They even had challenges in funding their meetings. The low contributions required of the parties were not fully paid (KIIs, 2019). CEPPS has contributed to strengthening institutions beyond capacity building (e.g., by helping review the Code de Bonne Conduite and bylaws of its monitoring committee). The strengthening of CENI is still ongoing (CEPPS, 2018). It is still heavily dependent on NDI support, especially in bringing and keeping different actors on board. While democratic processes have been strengthened, they are frequently challenged and threatened; full ownership by the political class has not been achieved to a level that would ensure complete self-reliance (Freedom House, 2019). Enabling Factors for Self-Reliance CEPPS has contributed to governance gains and has strengthened existing processes and institutions designed to establish self￾reliant democratic processes in Guinea. NDI has worked through and with its key local partners, creating opportunities for joint learning and capacity development. CASE STUDY 5: CEPPS IN GUINEA n A–27 REFERENCES Commission Electorale Nationale Independante (CENI). (September 19, 2017 post). https://www.electionsguinee.org/ la-ceni/, accessed on 06/12/2019. Consortium for Elections and Political Process Strengthening (CEPPS). (2016). Quarterly Report Q1 2016. CEPPS. (2018). Quarterly Report Q2 2018. Freedom House. (2019). https://freedomhouse.org/report/ freedom-world/2019/guinea, viewed on 06/08/2019. Gorodnichenko, Y., & Roland, G. (2015). Are there cultural obstacles to democratisation? VOX CEPR Policy Portal. May 14, 2015. https://voxeu.org/article/role-culture-democratisation, viewed on 06/08/2019. Moore, Z., Bongiovanni, A., Sadaphal, S., Hernandez, O., Espeut, D., Levin, C., & Toole, M. (2019). Performance Evaluation of USAID Ebola Pillar II Activities: Final Report. Evaluation Report to USAID, March 2019. Vienna, VA: International Business & Technical Consultants, Inc. National Democratic Institute, https://www.ndi.org/sub￾saharan-africa/guinea, viewed on 06/08/2019. Search for Common Ground (SFCG). (2017). Rebuilding together: Community-driven reconciliation and enhanced communication in Guinea Forestière and “Votons pour la paix”: Supporting Participative, Transparent, Credible and Peaceful Elections in Guinea, AID-675-A-12-00004 Final Report. A–28 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME 1 LIBERIA CASE STUDY 6: CIVIL SOCIETY AND MEDIA LEADERSHIP (CSML) BOX A6–1. CSML AT A GLANCE Lead Implementing Partner (IP): IREX Partners: The Carter Center, Social Impact Nature of Implementation: Single sector (governance), stand-alone activity Implementation Dates: July 2015–November 2015 (ongoing since May 2007) USAID Pillar II Budget/Obligation: USD 2.2 million1 Geographical Priority Areas: Liberia national CSML Objectives: “To improve the skills and performance of civil society organizations through training, mentoring, and small grants” and “to engender professionalism in the Liberian media industry by improving the quality of information, injecting new skills, and opening avenues for media, assisted by technology, to give citizens voice in national dialogs.” Key Activity Components: Original CSML was used a cascade capacity development strategy: the original partners strengthened four Liberian nongovernmental organizations (NGOs), which then undertook training, mentoring, and technical support with 62 CSOs and 19 community radio stations (CRSs), working in parallel Civil Society and Media workstreams. The Pillar II extension of CSML modified six of its nine objectives, focusing CSML’s civil society and media efforts on accountability for Ebola funds, and expanding technical support for CRSs in an additional eight counties. Key Pillar Objectives and Outcomes to which Activity Contributed: CSML Contributed to all three Pillar II objectives. CSML strengthened the civil society and media channels that led communities to adopt Ministry of Health (MOH) recommendations for EVD control, thus ceasing losses in development gains. It helped professionalize journalists and media outlets and local government, thereby strengthening critical infrastructure, and it introduced innovations and promoted partnerships to enhance the relevance, responsiveness and sustainability of independent media. A Decade of Investment in Community Radio and Civic Engagement Enabled Peaceful, Inclusive Mobilization during the Ebola Response and Recovery by Dr. Barbara De Zalduondo The Government of Liberia (GOL) realized early in the response to the 2014–2015 Ebola crisis that containing the disaster depended less on biomedical interventions and more on urgent, broad-scale changes in behavior (GOL, 2015; Johns Hopkins University Center for Communications Programs [JHU/CCP] 2017). Early GOL efforts to inform the population about Ebola control and prevention were met with denial and, in some cases, violence. The United States Agency for International Development (USAID) mobilized its extensive expertise in social and behavioral change communication (SBCC) through its SBCC and social mobilization 1. Source: USAID’s Office of Budget and Resource Management O&E spreadsheet, September 2017. 2. The national adult literacy rate in 2014 (measured by self-described ability to read and write in any language) was 66.7 percent. A discrepancy between urban and rural dwellers was substantial (76.0 percent compared with 50.1 percent literate), as was the gender difference, with 80.6 percent of males but only 54.8 percent of females reporting they could read and write (LISGIS, 2016). implementing partners (IPs) to help the Liberian people obtain factual information from trusted sources. The challenge was great, given pervasive distrust of national authorities that lingered from the civil war years (1989–2003), especially outside the capital, Monrovia (e.g., Abramowitz and Moran, 2012). In addition, low literacy and low access to print media, especially in rural areas2 and the need for communication in multiple languages, make community radio and community-level dialog the best method to reach affected communities through channels they could access and trust (Abramowitz et al., 2015; World Association of Newspapers, 2004). CASE STUDY 6: CSML IN LIBERIA n A–29 Since 1997 and funded by a variety of donors, IREX/Search for Common Ground (SFCG) has worked in Liberia with journalists, media companies, community leaders, and community radio stations with a focus on peace-building and giving Liberians a voice in national policies and events, from elections to uses of development funds, with a strong emphasis on gender, human rights, and sustainability (IREX, 2010). Building on SFCG, in 2010 USAID/Liberia awarded IREX the CSML project with the goal of building “sustainable institutions that represent the interests of the Liberian people.” IREX identified seven Liberian organizations as Resource Partners (RPs) and provided them with intensive organizational development training. These RPs then worked with IREX, The Carter Center and Social Impact staff in two mutually reinforcing work-streams. A civil society workstream built the capacity of 62 civil society organizations (CSOs) in sectors ranging from agriculture to education to humanitarian relief. Through 17 national and 60 county-level trainings, they strengthened the CSOs’ governance systems and built their capacity to provide leadership training for local officials They also created spaces to promote inclusion of women, young people, and marginalized groups in local dialogues and political events. Under IREX’s oversight, the RPs provided training, mentoring, and small grants to these CSOs to support follow-up actions. The second workstream provided training, mentoring, and equipment to professionalize journalists and community radio stations. Small grants to media organizations offset the costs of investigative journalism, and enabled stations to replace equipment, which was frequently being knocked out by storms. Both workstreams sought to build the management capacities and organizational systems that would make their beneficiary organizations sustainable (IREX, 2015). Training for the CSOs strengthened their management systems and skills to enable them to win and independently manage funding from the U.S. Government (USG) and other donors. Training for the media organizations and radio stations included marketing and business development to help them identify sustainable funding sources and to broaden their audiences. True to CSML’s peace-building mission, the project worked with 19 publicly owned radio stations that reached deep into the rural areas, where radio is the only means of keeping up with county and national events. CSML was its final year when the Ebola crisis occurred. With USAID’s permission, the project redirected its staff, RPs, and CSOs to contribute to the Ebola response, leveraging 4.5 years of experience and progress in civil society leadership and media development, and expanding into all 15 counties (IREX, 2014). CSML worked with the Ministry of Health, UNICEF, and IPs including JHU/CCP HC3 and Jhpiego MCSP, to prepare messages in six languages on Ebola virus disease (EVD) prevention and care, and on combatting stigmatization of EVD survivors. CSML’s media team worked with the Ministry of Information (MOI) and Ministry of Internal Affairs (MOIA), Liberian Association of Community Radio Stations, and an expanded group of 26 commercial and 16 community radio stations to transmit those messages nationwide (IREX, 2016). Simultaneously, the civil society team used its well-tested “step-down” method for translating recommendations from Monrovia into community concepts and local languages to engage community leaders in dialogue on EVD, to support them in convening their communities to hear concerns. They also explained and discussed the changes needed in traditional practices to prevent further spread. CSML RPs facilitated 57 such Community Leadership Forums (CLFs), identifying issues to address in radio broadcasts. In addition to concerns about lack of access to health facilities and shortages of food, issues cited in over 20 of the CLFs were 1) fear of an upsurge in teen pregnancy due to the closing of schools, and 2) concerns about stigmatization of people associated with Ebola (survivors and families). CSML responded to the latter by supporting community-level memorialization events and media coverage of dialogs contesting EVD stigma (IREX, 2016). In July 2015, USAID/Liberia awarded IREX Pillar II funding to extend its Ebola work (IREX, 2015). Concerns and complaints about corruption were escalating and the CSML platform, with its built-in capacity for investigative journalism and peace￾building, provided an opportunity to defuse growing tensions and suspicions at the community and national levels. CSML thus undertook a five-month scope of work to engage journalists and civil society around transparency and accountability for EVD resources. CSML trained 99 journalists on how to research and report professionally on the flows and impact of Ebola funds: what funds were provided to whom, what they were used for, and what the results were. As described in the program’s final report (IREX, 2016), CSML’s CSOs launched an “Ebola Situation Room,” where information on EVD funds and government activities were tracked and reported to the Liberian people. In addition, they launched a fraud hot line and an SMS platform at the Liberia Media Center (LMC), where citizens could report instances of suspected corruption for the journalists to pursue. The information was made broadly available on a website (https://b-m.facebook.com/trackingaidliberia/), and “Liberia is a cash-strapped country. If you think of how much capacity-building you can do, how much training you can do will all boil down to money. When one project closes, you go to the next project and focus on something else. I can tell you for the organizations where you still have some of the same [CSML trainees] who left, Liberia wins because we built their capacity so that international NGOs can see them and say wow!” – KII, 2019 A–30 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME 1 community radio stations convened call-in programs where community members could join the debate. Four thousand “lower-level chiefs” were trained during the EVD crisis. The training was able to build on prior leadership training that emphasized mutual accountability, so civil society’s demand for transparency about the amounts and uses of Ebola funds could be seen as normal rights of citizens and responsibilities of government; traditionally, questioning local leaders could have been interpreted as an affront to their authority (see Abramowitz & Moran, 2012). The CSML project, with its Pillar II extension, was applauded by all civil society and media contacts interviewed in both the first Performance Evaluation (PE1) and PE2. According to project documents, the CSML team met or exceeded its targets on 13 of the 17 indicators for the Ebola cost extension (IREX 2016b). While the activity’s outcome measures tracked output level accomplishments. such as number of grants to CSO partners, number of Ebola-related messages produced, number of news stories analyzed by the Situation Room (IREX, n.d.), former IREX staff claimed that working in CSML had transformed the media sector in the country. “Radio stations are asking citizens how they think decommissioned [Ebola Treatment Units] should be repurposed, how pick-up trucks once used during the crises should be distributed, and gauging their opinions response of the Government, INGOs, and others. . . . The recent successes of these stations point to the value of the tool in furthering dialogue at the community level on accountability and providing a platform through which radio stations can engage Government armed with data reflecting the voice of their listeners.” (IREX, 2015c p. 2) FACTORS CONTRIBUTING TO EFFECTIVENESS Leveraging prior investment A number of factors reportedly contributed to CSML’s effective use of Pillar II resources. The first and most important was the team’s ability to build on over a decade of USAID investment in building the communications sector as an instrument of peace-building and civic engagement. CSML had previously provided training on “general journalism,” including an emphasis on evidence and accuracy in reporting, so in 2015 the media team was able to recruit trained journalists and provide specialized training on health reporting.The 99 journalists trained produced 17 spots and 17 skits on EVD prevention and care, and they visited 57 medical facilities in five counties, investigating and reporting on their experiences. Key informants (KIs) said that in the past, health had been dealt with sensationally or not at all. Thanks to CSML, they learned the importance of reporting health issues in line with MOH guidance. They also observed that their audiences wanted more of that kind of content. They claimed that, due to CSML, many successful media outlets now have a health beat. “Yes, like I said before journalists were not covering health care unless it was political or they were passing the health care budget. Because of Ebola and because of our intervention health now is a major issue in terms of what the media covers. There are radio stations that have health beats, there are newspapers that have health beats.” – KII, 2019 Building capacity for constructive advocacy and civic engagement CSML’s trainings with partner CSOs and their engagement with local leaders developed skills in evidence-based dialog and advocacy. Rolling out leadership trainings and supporting community dialogs such as the Community Leadership Forums provided unprecedented opportunities for citizens to approach and talk with their leaders about issues of concern, and potential solutions. These are critical skills for effective, empowered communities (e.g., Wallerstein, 1993) with the potential to profoundly reshape governance at the local level. “Remember that CSML was civil society and media, okay. So, we strengthened our civil society to advocate for issues. The focus is not to do conflict but do research so that you can be able to articulate the message that you want to get across.” – KII, 2019 CASE STUDY 6: CSML IN LIBERIA n A–31 Consistent attention to gender equity IREX’s technical choices placed gender at the center of CSML’s planning and reporting. In the main project years, CSML emphasized including women journalists in trainings and including female journalists on all beats. In advocacy and training with editors and media outlets, CSML staff contested the trivialization of women’s issues by editors and station boards, which had limited female reporters to entertainment (e.g., beauty pageants) and homemaking. Partner CSOs ensured that women were included in district development funds and they organized women’s listening groups in the focus counties, where women could gather to discuss the radio programs on current events, including accountability for EVD resources. CSML partnered with and strengthened the Liberia Women Democracy Radio, the women’s radio station, and supported the advocacy of the Liberia Women Media Action Committee, the Female Journalists Association of Liberia, and the Press Union of Liberia to ratify a Gender Policy, which codified standards for inclusion of women in media. While gender equity remains a distant goal, the CSML saw an increase in the proportion of women journalists from 14 percent to 24 percent, enabling more women’s voices to be heard nationwide (IREX, 2016). Q: Were you involved in recovery? A: “Yes. During the Ebola [outbreak], they started the Civil Society Task Force. CSML supported them. We realized working in groups [was] much better than working in splinters. Whether Action Aid, Search for Common Ground (SFCM,) IREX, all supported it. It was really effective. It helped us to focus our expertise in different areas.”— KII, 2019 Engaging and strengthening the diverse components of the media ecosystem Enabling journalists to fulfil their roles in a democracy and to reach beyond the commercial centers to the entire population requires a number of interlocking elements: journalists trained and supported; professionalized editorial boards and outlet managers that prioritize accuracy and public service over sensationalism or political bias; platforms for collaboration and coordination across sectors and locales to provide the public with coherent information; working equipment in the newsroom and in the radio station that keeps the stories coming in and on the air; audience participation to inform the media outlets of their interests and concerns; and the capacity to obtain and manage funds to finance all of the above. Unlike many communication projects, CSML addressed all of these elements in a coherent and consistent manner over a decade. In addition to training and mentoring journalists and editors, IREX and partner staff worked with and built capacity in a wide array of partner organizations in Monrovia, where media outlets were concentrated, and nationwide, helping them progress toward sustainability in technical capacity and fundraising. CSML capacity development enabled 21 CSO partners win grants from both national and international funders over the life of the project. CSML’s media stream innovated and strengthened the technical support required to keep radio stations on the air: they funded an Emergency Equipment Pool and founded a network of trained technicians who could respond to calls to repair electrical outages and other community radio station equipment. “If you are in the media, you do not discriminate [against] anybody. We trained the journalists that when you go out to report on issues, make sure you do not leave out the disadvantaged communities. Make sure you focus on everyone. All of those were part of the training that you do not leave anyone behind. We covered everybody. Diversity is one of the things we focused on in media.” – KII, 2019 A project culture that prioritized human rights CSML trainings conveyed the importance of inclusiveness in all their activities, from reporting news to local governance. KIs stressed that giving voice to the vulnerable and promoting justice for all were core values of the project and its staff. During the Ebola crisis, CSML dispatched journalists to heavily affected areas, where they gathered and reported stories of real people, at substantial personal risk. Their reporting helped give EVD a human face, and promoted compassion instead of stigma against affected individuals and households. In the recovery period, CSML-trained journalists made 68 requests for information under the Freedom of Information Act, which had been passed by Liberia’s legislature in 2011, and they filed 221 news stories on accountability for Ebola funds and resources. CSML leadership forums explained the value of transparency, inclusiveness and mutual accountability, and continued to build skills in evidence-based advocacy and resolution of disputes. A–32 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME 1 “I can tell you that the stories are going to be different because even since Ebola we have had measles outbreak, and the way those stories are reported is sharply different from the way they were reported before Ebola hit. Because they know that they have to go and talk to different health care providers, different families, talk to people about how to prevent the virus, promoting immunization, and they are doing those kinds of stories because of the training, mentoring, and nurturing they received during the Ebola crisis.” – KII, 2019 FACTORS CONTRIBUTING TO ADVANCING SELF-RELIANCE As noted above, IREX/CSML made diverse contributions to strengthening the capacity of independent media to inform the Liberian public and to represent their interests and concerns. Institutions that reliably inform and engage the public in civic affairs are essential to Liberia’s ability to identify and resolve its own development problems, and CSML helped to build these institutions, not only in the capital, but also throughout the country. Over its five-year life of project as CSML, IREX trained journalists, editors, and media outlet managers, and strengthened the networks and organizations where they could work and transmit their values and methods to others. It strengthened the community radio organizations that for some are their only source of information about events and issues beyond the community level. Former CSML staff were realistic about the ability of independent radio stations to fund their equipment and staff over time. While they saw opportunities for private sector funding through advertising in some urban areas, stations that cover more rural and dispersed populations needed public support. Respondents mentioned instances where a rural community had mobilized to support its radio station (e.g., building a farm whose produce would support the station) but the consensus in the KIIs with former CSML staff was that most community radio stations serve low-income communities that would continue to depend on a mix of public County Development Funds and private contributions from NGOs and donors. People want their radio but don’t want to pay: “Even us, we were down for eight months. The same lightning damaged our entire equipment, so we were down for eight months. . . This station is not owned by political leaders or any person, it is owned by the residents. We submitted a request to the Commissioner (they taxed the various districts), the Commissioner to Clan chief to paramount chief. Also women contributed $1,000L, and with that they were able to send for a transmitter. – KII, 2019 Improved commitment KIs expressed enthusiasm and appreciation for the principles and skills they had acquired through the CSML training and mentoring. They expressed pride in their high professional standards as journalists, and continuing commitment to including women in their ranks. More than three years after the end of the project, former CSML staff were applying their skills and principles in a variety of institutions, sometimes in the context of new projects that are funding former CSML partner organizations (e.g., The Carter Center’s Access to Information project), and sometimes in lone outposts with minimal or no pay (e.g., KII 2019). While objective measures of commitment at the individual level are not yet established and attribution to a specific project is not possible, the IBTCI evaluation team in PE2, as in PE1, met a gender-balanced cadre of motivated, enthused, and trained civil society advocates and journalists who are aware of and actively carrying out their important roles in informing and engaging Liberians in identifying and resolving their own development challenges. CASE STUDY 6: CSML IN LIBERIA n A–33 Q: Did CSML increase Liberians’ ability to identify and respond to their development challenges? A: “We looked at the result that we had, you saw that people can change. They went from sensationalizing a health emergency to actually educating, informing, and reporting what was happening in the communities during the Ebola. We were able to empower and enlighten civil society to also advocate for their communities. To also do education awareness about health issues at that time during Ebola. They could also transfer those issues to any other coming issues that their communities may be facing.” – KII, 2019 REFERENCES Abramowitz, S.A., McLean, K.E., McKune, S.L., Bardosh, K.L., Fallah, M., Monger, J., … Omidian, P. (2015). Community￾Centered responses to Ebola in urban Liberia: The view from below. PLOS Neglected Tropical Diseases 9(4): e0003706. https:// doi.org/10.1371/journal.pntd.0003706 Abramowitz, S. A., & Moran, M.H. (2012). International human rights, gender-based violence, and local discourses of abuse in post-conflict Liberia: A problem of “culture”? African Studies Review, 119–146. Johns Hopkins University Center for Communications Programs (CCP). (2017). Health Communication Capacity Collaborative (HC3). Social mobilization lessons learned: The Ebola response in Liberia. Baltimore, Maryland: Johns Hopkins Center for Communication Programs. IREX (2010) CSML. Civil Society and Media Leadership Program. Quarterly Report April 1–June 30, 2010. Submitted to USAID/Liberia. Accessed 03032019: https://pdf.usaid.gov/ pdf_docs/PA00KXHT.pdf IREX (2014). Civil Society and Media Leadership Program. IREX (2014) Year Five Work Plan. Anticipated Activities for September and October 2014 in light of Ebola Crisis. Submitted to USAID/Liberia. September 12, 2014. IREX (2015a). SBU. A. Summary of Proposed Changes. [file: IREX CSML Cost Extension_Submitted to USAID.docx]. IREX (2015b). The CSML Quarterly. Volume 5 Issue 4; Jan￾Mar, 2015. [File: CSML Newsletter_Vol 5_Issue 4_Jan-Mar 2015_FINAL.pdf]. IREX (2015c). The SCML Quarterly, Volume 6 Issue 1, July￾Oct, 2015. IREX CSML (2016), Trusted voices in challenging times. Civil Society and Media Leadership Final Report. IREX (2016b), Final Report Annex: CSML Cost Extension: Results Framework Data Table. IREX (n.d.) Unidentified Word file in the PE2 library: Final CSML Indicators with Data.doc. Liberia Institute of Statistics & Geo-Information Services (LISGIS). (2016). Household income and expenditure survey 2014. Agenda for transformation: Baseline indicators. February 2016. USAID DRG. (2015). CSML Project Fact Sheet, October 2015. World Association of Newspapers. World Press Trends 2004. http://www.warc.com/LandingPages/Data/NewspaperTrends/ PDF/Liberia.pdf A–34 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME 1 GUINEA CASE STUDY 7. EMERGENCY ACCESS TO FOOD FOR EVD-AFFECTED GUINEANS (ESFP) BOX A7–1. ESFP AT A GLANCE Lead Implementing Partner (IP): Catholic Relief Services (CRS) with local partner Nature of Implementation: Multi-sector (health; water, sanitation, and hygiene (WASH); food security), integrated with a multi￾layer response to Ebola in the same geographic area, stand-alone activity Implementation Dates: February 2015–December 2016 USAID Pillar II Budget/Obligation: USD 3,253,1361 Geographical Priority Areas: Guinea (Forest Region), Nzérékoré, Macenta Key Activity Components: Food vouchers in Forest Region, nutrition messaging, and livelihoods fairs Key Pillar Objectives and Outcomes to which Activity Contributed: ESFP contributed to the first of the three Pillar II Objectives: 1 (“Prevent the loss of development gains”). Emergency Access to Food Security Resources for Vulnerable Guinean Households by Dr. Karim Sahyoun Although Ebola affected the entire country, Forested Guinea was hit particularly hard. The two prefectures of Nzérékoré and Macenta accounted for 21 percent of the confirmed cases and 612 of 2,083 (29 percent) of the country’s total deaths (The Ebola Sitrep, 2015) while representing less than seven percent of the total Guinean population (Institut National de la Statistique, 2014). In 2015, both prefectures were considered to be stressed in terms of food security according to the Famine Early Warning Systems Network (FEWS NET)’s Integrated Phase Classification. Both areas still had Ebola cases and suffered the greatest crop decline since the outbreak began (FEWS NET, 2014). Households 1. Source: USAID’s Office of Budget and Resource Management O&E spreadsheet, September 2017. experienced deteriorations in food availability and their revenues (CRS, 2017). A combined Pillar II approach of food vouchers, social behavior change communication activities on nutrition, and organizing livelihood fairs was at the core of the USAID-funded Emergency Access to Food for Ebola Virus Disease (EVD)-affected Guineans Program (ESFP), beginning in 2015. The program was implemented by Catholic Relief Services (CRS) and its local partner, Organisation Catholique pour la Promotion Humaine/ Caritas Guinea (OCPH). Its objective was to increase short- and long-term access to food for vulnerable households in Ebola￾affected areas. FACTORS CONTRIBUTING TO EFFECTIVENESS ESFP has shown a number of achievements throughout its lifetime: ■ The program supported 7,008 households (35,040 beneficiaries) in Nzérékoré and Macenta, reducing food insecurity among them from 95 percent to 57 percent while those with food diversity increased from 68 percent to 82 percent (CRS, 2017). Beneficiaries explained that the support enabled them to sustain their families in terms of food and basic needs (CRS, 2017). ■ Beneficiaries received eight rounds of food vouchers during the 2015 and 2016 lean seasons, getting on average a monthly allowance of USD 40 in food vouchers (70 percent of the typical Guinean breadbasket) to be used solely to purchase staple foods in local markets. In addition, vouchers with an CASE STUDY 7: ESFP IN GUINEA n A–35 average value of USD 70 were distributed during two rounds of livelihood fairs (CRS, 2017). ■ Ninety-one percent of beneficiaries were happy about the site selection and the organization of the livelihood fairs (CRS, 2017). ■ CRS held 66 sessions of Social Behavior Change Communication in Nutrition targeting 16,889 direct beneficiaries before the distributing the food vouchers (CRS, 2017). Significant improvements in household diets were seen at two levels: a reduction in food insecurity from 95 percent to 57 percent and an increased dietary diversity with households having a high level of diversity increasing from 68 percent to 82 percent (Moore et al., 2019). Improved conditions of vulnerable households, including food diversity and education, were also observed (Focus Group Discussion (FGD) participants, 2019). A major achievement of these activities has been to enable supported households to return to more stable lives through the support provided. This was often realized in combination with other post-Ebola activities (e.g., agricultural training, distribution of fertilizer and seeds, free medical treatment, training on income-generating activities), some of which were carried out by CRS but also by a variety of other actors in the target area (FGD participants, 2019). Looking back, the impact of ESFP was mostly seen at the individual household level and, to a limited extent, at the community or regional economy level through the injection of cash into the local economy (FGD participants, 2019). “Targeted households were able to regain their income by resuming their normal activities; regaining the meaning of their lives was the most important thing for them.” – KII, 2019 Supportive local government The availability and support of local authorities have been critical for implementation and a number of achievements (FGD participants, 2019; KII, 2019): Examples are authorities’ involvement in the sensitization of communities and beneficiaries concerning the eligibility to receive vouchers and the approach itself, informing traders and vendors about this activity, and ensuring the acceptability of vouchers. This helped in all instances to establish a relationship of trust while participating in the program (KII, 2019). Particularly for livelihood fairs, the program worked with the agricultural directorates and people received their support in food vouchers (KII, 2019). Local authorities were part of the prefectural- and local-level committees, which are described below (CRS, 2017). “Undoubtedly one of the most impressive aspects of the voucher component was the coordination between [nongovernmental organizations] and government.”(Moore et al., 2019, Appendix M) Community engagement/local committees Engaging the community and establishing a local committee have been key factors for success (FGD participants, 2019; KII, 2019). Helping communities to understand the program and specifically who is targeted was the first critical hurdle. Committees were formed at prefectural, sub-prefectural, neighborhood, or village level in each of the 22 villages or areas targeted. At the prefectural level, these focal points included two representatives of families affected by EVD, a priest, an imam, a prefecture representative, and a representative from each of its departments of Health and Social Action. Village￾level committees consisted of the village chief, a religious leader, one representative of women’s groups or associations, one youth representative, and two representatives of families affected by EVD. These committees were critical not only in communicating with the community, but also in assuming a number of responsibilities throughout the life of the program. A first important role was the validation of beneficiary selection criteria and procedures. In defining criteria and later the validation of beneficiary lists, committees played a critical role in ensuring that only the most vulnerable beneficiaries were selected (CRS, 2017; Moore et al., 2019). Local committees voluntarily assisted with participant verification and crowd control for the two livelihood fairs. This collaboration was seen as particularly successful: 91 percent of beneficiaries were happy about the site selection and the organization of the fairs (CRS, 2017). “The main message was to inform traders and vendors that when you collect a voucher, you will be reimbursed, though it very difficult to accept for someone doing this for the first time.” – KII, 2019 A–36 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME 1 Livelihood fairs based on Livelihood Assessment Livelihood fairs were a complementary measure to food support and intended to increase supported households’ capacity to meet their food security needs going forward. For each of the two fairs, targeted households received vouchers with a value of USD 50. To ensure that the fairs would offer relevant products, the project first conducted a livelihood assessment¬ to identify the various categories of income-generating activities undertaken by the beneficiary population. Results guided the selection of the inputs such as tools and seeds (CRS, 2017). Deliberate efforts to bring vendors on board Male and female small retailers, larger retailers, and wholesalers were all encouraged to participate to ensure the availability of all necessary goods. Those interested went through a competitive selection process. Selection criteria included restocking capacity, financial autonomy, possession of official documents provided by the Ministry of Commerce’s extension services, and means of transportation; 66 suitable vendors were selected. Local village committees validated the vendors and existing relationships with local populations (CRS, 2017). The enrolling of vendors has been a critical aspect of the program, especially because it was a new approach to the area (KII, 2019). Unaccustomed to voucher exchange activities, vendors were reluctant to hand their merchandise over to project beneficiaries. They were used to cash payments and felt insecure about accepting vouchers instead. CRS resolved this issue with the support of the local governmental authorities and the local Chamber of Commerce. These served as legal and moral supporters of the project’s exchange mechanisms. CRS included a binding road map for the trade relations between the project and its vendors in its Memoranda of Understanding with the selected vendors and with the Chamber of Commerce’s participation and leadership (CRS, 2017). Gaining the trust of vendors in the methodology—and specifically the use of the vouchers and reimbursements—was necessary to ensure vendor participation. The program used Post-distribution Monitoring to evaluate both vendor and beneficiary satisfaction. They were conducted two weeks after food voucher distributions and daily during the livelihood fairs (KII, 2019). Availability of resources and materials The availability of sufficient food, resources, and materials on local markets within the region, or brought by traders into the region, was a precondition for the program to be successful (FGD beneficiaries Nzérékoré, 2019). Despite initial restrictions in the movement of goods, this condition has been met. Protection during participant selection and food voucher distribution The EVD outbreak was ongoing during the program’s implementation period. Therefore, organizing mass meetings brought additional risks and challenges that had to be addressed. To reduce the risk of transmission of Ebola during the various fairs, CRS and OCPH ensured that proper procedures and protection measures were in place at voucher distribution sites. Critical hygiene protocols were applied at each fair site, including systematically screening temperature upon entry and installing handwashing kits containing soap and chlorine. Participants were also encouraged to avoid unnecessary physical contact. High costs associated with voucher distribution One downside observed by the evaluation team is the relatively high cost of the program. Significant effort seemed to have been necessary to ensure the voucher approach would work in the given context: ■ Vouchers had to be imported specifically from the U.S. to prevent fraud. ■ Significant efforts were necessary to bring vendors on board and to assess and validate them. ■ It took major efforts to organize the livelihood fairs, including conducting prior needs assessments. ■ Vendors had to take several steps before they could cash their vouchers, including visiting the local partner to hand in the vouchers they had collected and establish the amount they were due; then, collecting payment at a partnering financial institution. ■ The distribution of vouchers and holding the fairs created a risk that the epidemic would spread and required efforts for mitigation. CASE STUDY 7: ESFP IN GUINEA n A–37 Sustainability According to the implementing agency, EFSP was designed to meet immediate needs in an emergency situation (KII, 2019). Therefore, the focus on sustainability and self-reliance was limited. Observations on whether vulnerable and EVD-affected households have been able to sustain positive changes were contradictory. On the one hand, it was reported that many households were unable to sustain the change after the vouchers ran out, even though agricultural products from the fairs had boosted their production (FGD participants, 2019). Another observation by local stakeholders was that EVD survivors had received vouchers but were not gathered in associations or groups to benefit from training on income-generating activities. They expressed that only those who had been wise had started with petty trades and therefore were still benefiting (FGD participants, 2019). Other local beneficiaries described that households continued to benefit because they had received productive tools and knowledge (not further defined; KII, 2019). The program’s final report pointed to the issue that some of the change may not have been sustainable; 57 percent of households were food insecure when measured two months after the project had ended. CRS explained that this was due to the late timing of the measurement, which may have watered down the positive effect. The removal of benefits may have already significantly affected household hunger by this point and these results may not show the true impact of the voucher program (CRS, 2017). EVD-affected families explained that natural calamities, pests (insects), and inundation had caused low agricultural productivity and limited impact, especially as they were lacking pesticides to protect their crops. They also emphasized the continued suffering from the loss of key members of their families (FGD participants, 2019). When program funding ended, CRS had hoped to receive further funding for an additional phase to more sustainably strengthen the EVD survivors’ livelihoods by helping them build capacity in income-generating activities and agricultural production (CRS, 2017). FACTORS CONTRIBUTING TO ADVANCING SELF-RELIANCE This program was intended to meet immediate needs of EVD￾affected households, allowing them to return to a more stable level. There was no intention or deliberate effort to contribute toward self-reliance (KII, 2019). Supporting households with productive tools may have contributed to self-reliance at the level of individual households. In the context of providing relief, it may be a more relevant question to ask whether the program has done any harm to self-reliance for the target group and the state. Based on the above discussion of the program, the following observations toward self-reliance can be made: ■ CRS has closely worked with local institutions and authorities—involving them in the process of adapting the voucher concept to the local conditions—and with committees to provide services to targeted households. This approach has not weakened or disempowered institutions, but rather CRS collaborated with them and provided the opportunity to explore a new approach to providing benefits. However, it is not clear whether local actors could learn from this example in the sense of replicating this relatively complex voucher-based approach to relief. ■ In collaborating with the state’s existing services, CRS built on the state’s existing competencies (e.g., in agriculture), instead of replacing them. ■ The approach of collaborating closely with committees at different levels strengthens the ownership of communities in addressing the plight of the targeted households. ■ Introducing vouchers instead of direct food aid allowed the program to support local markets and infuse cash into the local economy instead of creating dependency and distorting markets. Based on these observations, CRS has opted for a project approach that minimized harm to self-reliance. REFERENCES Catholic Relief Services (CRS). March 2017. Final Program Report for Emergency Access to Food for EVD-Affected Guineans (EFSP). FEWS NET. (2014). Guinea, Liberia and Sierra Leone Special Report. December 31, 2014. Institut National de la Statistique. (2014). 2014 Census Data. Retrieved from: http://www.stat-guinee.org/index.php/statisti/ donnees-structurelles/demographie/31-population-densite￾region-prefecture Moore, Z., Bongiovanni, A., Sadaphal, S., Hernandez, O., Espeut, D., Levin, C., & Toole, M. (2019). Performance Evaluation of USAID Ebola Pillar II Activities: Final Report. Evaluation Report to USAID, March 2019. Vienna, VA: International Business & Technical Consultants, Inc. The Ebola Sitrep. (2015). http://www.ambafrance-gn.org/Ebola￾point-de-situation-au-28-decembre-2015 A–38 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME 1 LIBERIA CASE STUDY 8: ECONOMIC RECOVERY FROM EBOLA FOR LIBERIA (EREL) BOX A8–1. EREL AT A GLANCE Lead Implementing Partner (IP): Mercy Corps Partners: Young Men’s Christian Association of Liberia (YMCA), The Lutheran Church in Liberia (LCL); and Volunteers to Support International Efforts in Developing Africa (VOSIEDA). Subcontractors: EcoBank, Noble House/Red Rose Nature of Implementation: Single sector (agriculture and food security), stand-alone activity Implementation Dates: January 1, 2015– December 31, 2016 (LOP 24 months) USAID Pillar II Budget/Obligation: USD 13,493,176 over 2 years1 Geographical Priority Areas: Liberia, 11 selected districts in Lofa, Margibi, and Montserrado Counties EREL Objectives: To “fill minimum food basket gaps” for 30,000 households directly or indirectly affected by the Ebola outbreak; to assist 10,000 smallholder farmers to restore, maintain, and improve normal agricultural production; and to “minimize the negative impacts on child nutrition.” Key Activity Components: EREL was designed to deliver two complementary interventions: unconditional cash transfers (CTs) to selected households in the EREL-supported districts to enable vulnerable families to buy food; and provision of training and vouchers for a subset of those households, to restore and improve their farming outputs. Community engagement, promoting gender equity, and rigorous monitoring were central to the design, from identification of participating households to selection of lead farmers and vendors to tracking utilization of the CTs and fluctuations in the markets. Key Pillar Objectives and Outcomes to which Activity Contributed: Severe hunger was eliminated, dietary diversity was improved, and farming was reestablished and improved in EREL-supported households, which helped to stem the loss of development gains (Objective 1). In addition, EREL engaged and strengthened the market for private sector vendors of high-quality agricultural inputs, including seeds and tools, contributing to Objective 3. Experience, Preparation, and Intensive Monitoring Enabled Mercy Corps to Implement a High-Risk/ High Reward Intervention, Avoiding Pitfalls and Unintended Effects by Dr. Barbara De Zalduondo Mercy Corps’ Economic Recovery from Ebola for Liberia (EREL) project was one of six activities funded by USAID Food for Peace (FFP) with Pillar II resources and intended to assist Liberians in USAID priority counties to survive and recover from the Ebola crisis that had disrupted agricultural production and food supplies in 2014–2015.2 Mercy Corps (MC) has been working in Liberia since 20023 and in 2014, it conducted initial assessments of the impact of the outbreak on food security in the heavily affected counties of Lofa, Margibi, and Montserrado. They found escalating hunger, 1. Source: USAID’s Office of Budget and Resource Management O&E spreadsheet, September 2017. 2. The other Pillar II Agriculture and Food Security IPs in Liberia were ACDI/VOCA, PCI, Save the Children, World Food Program, and UNICEF (USAID, 2015:13–14). 3. Mercy Corps has worked continuously in Liberia since 2002 (Mercy Corps, 2017). fear of hunger, and severe disruption of agricultural activities, not only in villages that had been quarantined but also in these entire counties, which had among the heaviest burdens of Ebola cases and survivors in the country (Mercy Corps (MC), 2017a). EREL shared many features of the other Pillar II Agriculture and Food Security activities that USAID funds in the region. It was designed to provide short-term assistance to relieve hunger in the most vulnerable households and to help restore and improve food production and markets in these hard-hit CASE STUDY 8: EREL IN LIBERIA n A–39 areas over the longer term. These goals were achieved through two interventions: 1) unconditional cash transfers (UCTs) to households with Ebola survivors, disabled people, the elderly, widows, and orphans; and 2) for a subset of those households, provision of vouchers for seeds, tools, and other agricultural 4. EREL was the largest of the agriculture and food security (AFS) activities in Liberia; the others aimed to reach between 5,000 and 6,000 households each. inputs, conditional upon participating in training to improve the household’s farming practices. EREL operated in eight of the 15 districts in Margibi, Montserrado, and Lofa counties, with other IPs serving in other highly affected districts and counties. UNCONDITIONAL CASH TRANSFER INTERVENTION The selected households received a monthly UCT equivalent to USD 42 per month for six months. Beneficiary households were identified through “community-based targeting,” in which, after consultation with local authorities, each of 512 communities in the eight districts organized a registration committee and a verification committee. The registration committee, made up entirely of community members, nominated the households that met MC’s vulnerability criteria. Seventy percent of the beneficiaries selected were female (MC, 2016a; 2017a). Overall, the validation committee made sure that the registration list was fair and that the people who came to receive the UCTs were those on the list. MC provided oversight, ensuring that the EREL criteria for selection were followed. Community leaders were active participants, demonstrating trust in the program and were responsible for resolving disputes. The USD 42 per month for vulnerable families was intended to purchase food and other products from their neighbors, thereby contributing to the revival of local markets. Nutrition information, including the importance of dietary diversity and the value of supplemental foods for older breastfed infants, was provided informally to beneficiaries while they waited in line for the cash allowance. AGRICULTURAL INPUTS Groups of smallholder farmers (usually 25 to 50) who expressed interest in farming nominated two lead farmers who received intensive training in basic agronomic best practices. The lead farmers, in turn, taught what they learned to the group by cultivating demonstration plots that had been bestowed by the group members. The lead farmers shared their newly acquired knowledge on land and soil selection, seed testing, plot preparation, planting and transplanting techniques, composting, weeding, and reducing post-harvesting crop losses. In addition, at three intervals aligned with the farming calendar, each farmer received vouchers for a total amount of USD 60 for the purchase of seeds, tools, and other agricultural inputs at agricultural fairs that were organized by EREL in each district. The vendors invited to participate in these fairs were pre-screened by MC and qualified based on the cost and quality of their offerings. MC contracted Noble House/Red Rose, a UK and Turkish company, to upgrade the software and hardware used for the registration and production of vouchers. Each beneficiary was provided with a photo ID card with a bar code. To prevent fraud and corruption, vouchers were printed abroad and flown to Liberia. The data were made widely available through a web￾based dashboard (MC, 2017a). EREL was originally funded for nine months, but in September 2015 funding was extended to December 2016 and EREL’s scope and targets were increased to reach 30,000 households with UCTs and provide agricultural inputs to 10,000 smallholder farmers.4 In 24 months, EREL’s 12 indicators (MC 2016c; 2017b) were met or exceeded (see Table A8-1 and Table A8-2). More detailed outcome indicators found decreases in household hunger and an increase in diversity of the household dietary habits; in September 2015, MC’s baseline study found that 27 percent of 290 sampled households in the three counties were reduced to eating only one meal per day (MC, 2017e). EREL’s performance monitoring plan (PMP) was revised when the project was extended, switching to standard FFP indicators of hunger and food security. Program monitoring surveys in March, June, September, and December of 2016 showed a decrease in households that experienced severe hunger (Household Hunger Score (HHS) score of 4–6) from 3.6 percent to 0, and moderate hunger (HHS score of 2–3) declined from 54.4 percent to 7.7 percent (MC, 2016a). Household dietary diversity also had increased significantly. These improvements coincide with the findings of the IBTCI population-based household survey conducted in 2017, which found that sampled households in the three counties reported “little to no hunger” (HHS score of 0–2), and continued improvement in dietary diversity (see Table A8-3). A–40 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME 1 MANAGING RISKS Attempting to provide UCTs and agricultural input vouchers on such a large-scale under crisis conditions and in hard-to-reach areas was a high-risk undertaking. MC was able to draw on its extensive experience in Liberia; guidance from the Ministry of Gender, Children, and Social Welfare; and guidance and information-sharing from USAID/Liberia’s Food for Peace (FFP) working groups (MC, 2015a). The trust and established relationships of EREL’s implementing organizations enabled MC to mitigate and manage these risks (see Table A8-4, next page). Table A8–1. Total cash recipients, percent receiving all tranches, and percent female-headed County Target Total cash recipients % of target % receiving all 6 tranches % female￾headed Montserrado 12,000 11,428 95.2 90.6 73 Margibi 10,000 10,181 101.8 99.2 72 Lofa 8,000 8,468 105.8 85.7 65.5 Total 30,000 30,077 100.2 92.1 70.5 Source: Mercy Corps, 2017 (final report), p. 16 Table A8–2. Cumulative agricultural voucher recipients County Original target Revised target Total voucher recipients % of target Undistributed Montserrado 2,500 1,962 1,910 97 52 Margibi 2,500 3,038 3,038 100 0 Lofa 5,000 5,000 4,983 99.6 17 Total 10,000 10,000 9,931 99.31 69 Source: Mercy Corps, 2017 (final report), p. 16 Table A8–3. Household Dietary Diversity Score (HDDS) and Household Hunger Score (HHS), by region, Liberia 2017 Characteristic HDDS Number of Households HHS Number of Households Montserrado 6.0 312 1.2 402 Margibi 5.7 383 1.6 426 Lofa 5.3 303 1.1 405 Source: Moore et al., 2019, Annex L: Table HHS L17 “After Ebola, the first thing we received in this community was the money YMCA gave to us. That money really helped me. At that time I was here with nothing in my hands, even food. I never had it and I’ve got children and nobody to help them. The day we received that money it was like we were entering heaven.” – FGD participant, 2019 CASE STUDY 8: EREL IN LIBERIA n A–41 Table A8–4. Risk mitigation strategies employed Risk of Large UCT Distribution Practices that Minimized the Risk Security: EREL staff and community committees handled large amounts of cash Community members participated in providing security entering villages and during distri￾bution. Rather than armed guards, they were the visible faces protecting participants, whom they had selected. Conflict among community mem￾bers and/or among or with vendors Selection of beneficiaries was done by trusted community members (beneficiary selection committees). A free telephone hotline was available for people to report complaints or con￾cerns about fraud. A complaint box was prominent at each UCT distribution and agricultur￾al fair, and concerns were referred to the town chief for resolution. Market distortion MC conducted baseline and quarterly follow-up surveys of prices of tracer foods in local markets and did not detect any significant changes or trends in prices. Interruption of CTs due to external￾ities, from roads blocked by weather to the banking system MC worked with USAID to find backup solutions when EcoBank proved an unreliable part￾ner and the rainy season made delivering cash by road impassable in some areas. Biased selection of beneficiaries, overlooking vulnerable households MC and its partner civil society organizations had clear criteria for inclusion, and discussed and agreed on these with community leadership from the outset, so there was no confusion or room for bargaining. Frivolous uses of the cash A vast body of research finds that, when properly scaled and administered, UCTs in low￾and middle-income countries are used to improve the recipients’ health, education, or livelihood. These findings are especially true when recipients are women. EREL’s vulnerability criteria prioritized women, and post-distribution monitoring (PDM) surveys confirmed that the funds were used to support family welfare. 5. The PDM surveys were conducted quarterly, except in the period when the extension of the program was under negotiation (e.g., MC 2015e; 2016f; 2017f; 2017h; 2017i). Performance Evaluation (PE1) interviews and focus group discussions (FGDs) in PE2 confirmed that these efforts resulted in an effective and highly regarded contribution to EVD recovery in these counties. Rigorous Post-Distribution Monitoring (PDM) surveys5 conducted by EREL confirmed that the UCTs were being used largely to purchase food (Figure A8-1). Health and education were the second and third most common use of the money received from the UCTs, and a small fraction of the funds were used by individuals to start or expand their farms or small businesses. 6% Education 9% Health 4% Livelihoods 1% Debt 2% Savings 4% Other 74% Food “When we received that money, we went and bought food because YMCA said that the money was for us to buy food. After distributing the money, YMCA also brought seed, which helped us to plant pepper, cucumber, bitter boil, rice, and other smaller things. . . . For me, I went to the [Ebola Treatment Unit] when I was sick during the Ebola. When I returned from the ETU, things were not easy for me and when that money and seed came, it was like my savior.” – FGD participant, 2019 Figure A8–1. Consolidated household expenditure of cash for FY2016. (Source: Mercy Corps, 2016a, cited also in IDA, n.d.) A–42 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME 1 FACTORS CONTRIBUTING TO EFFECTIVENESS A number of factors contributed to EREL’s effectiveness in both the short and longer term. First, as noted above, MC drew on extensive experience and evidence and guidance from USAID and partners to propose an effective combination of interventions. Providing the monthly infusions of cash saved beneficiaries from immediate hunger, helped revive local markets, and enabled some to revive or start small businesses. Providing vouchers enabled farmers to restock seed for planting and to replace the farming tools they had sold or lost, and encouraged some, especially women, to take up farming for food and income. Making the agricultural vouchers contingent upon participation in the training and “learning by doing” from the lead farmers they had nominated from their community increased the likelihood that the inputs would be used effectively and bring optimal returns to the farmers. “The project was very effective because it helped us to increase our desire for farming, and [we sell] the products we get from our farm to get money to buy our basic needs. We also use that money to send our children to school.” “The farming has been very helpful to us. We thank God that today we can afford for our children through the farming we do.” – FGD participants, 2019 Community participation Community participation was built into every stage of EREL implementation, from the baseline assessments that used participatory rural appraisal methods (e.g., MC 2015b), to working with community leaders to form the committees that selected the beneficiaries, to the selection and support of the lead farmers. Community participation created community ownership and community vigilance over the resources provided. This helped ensure that recipients were able to safely carry their cash home without being robbed, and that the EREL staff were able to transport large quantities of cash with minimal threat and fear of robbery. Given the infusions of money and goods, and the fact that not all members of any community were eligible to receive the benefits, EREL was intentionally designed to be “conflict-aware” (MC, 2017a). In addition to the registration and validation committees, EREL implementing staff kept local leaders closely involved in the activities, and when concerns or conflicts arose, they were referred to the town chief for resolution, thereby relying on and reinforcing authentic local social control mechanisms. “I was there and I monitored everything from the start to the end of the projects. Our people were involved with the project; YMCA did not impose on them. We had two committees: one was responsible for registration and the other was to scrutinize the listing to make sure that people that were to benefit from the program names are in place. . . . The money helped to restore our people’s dignity and confidence because our people lost hope. They thought that everything was gone; they compared Ebola to the civil war.” – KII, 2019 A focus on women EREL was designed to be gender-responsive (MC 2017a). The definition and criteria of vulnerability, which set the criteria for selecting EREL beneficiaries, prioritized women, widows, and female-headed households. EREL monitored and reported on the gender distribution in all project activities. MC implementers encouraged community leaders to include women on the registration and validation committees. Between 64 percent and 73 percent of UCT recipients in the three counties were female-headed households (MC 2016e, p. 3). Sixty-nine percent of trainee farmers and 27 percent of lead farmers were women (MC, 2016a). While there is no indication that men misused UCTs or agricultural inputs, research elsewhere suggests that the predominance of female beneficiaries would have contributed to success (e.g., Baily, 2013; Food and Agricultural Organization, n.d.). CASE STUDY 8: EREL IN LIBERIA n A–43 Updated data services Implemented by Noble House/Red Rose, the updated data services provided EREL staff with sophisticated software that provided ID cards for each beneficiary, and printouts for each Registration and Validation committee, with photographs and barcodes associated with each name on their list. Use of the lists and barcodes also enabled Red Rose to provide MC with real￾time data on the distributions and a dashboard for tracking their progress. USAID support EREL encountered challenges that could have derailed the project. MC had contracted with EcoBank to provide and transport the cash required for the UCTs in the three counties. Disbursement events were planned and announced to beneficiaries in advance, and some recipients had to pay for transportation to get to their distribution point. Thus, it was critical that the EREL teams be in place with the required cash on the designated dates. EcoBank was not able to sustain this commitment. EcoBank was the only financial institution that had outlets in the three counties, but the USAID Mission stepped in and negotiated with AfriTel to supply the needed cash (MC, 2017a). Another externality was the weather; during the rainy season, the road linking the capital and the distribution center in Lofa County was washed out, intensifying the deprivation in Lofa County. MC consulted with USAID and they approved airlifting the required cash by helicopter so that beneficiaries were able to receive their cash and vouchers, even in the rainy season (MC, 2017i). Coordination and shared knowledge Throughout the recovery period, USAID FFP staff convened all AFS implementers to ensure coordination and to share experiences and solutions to common challenges, and MC participated in several working groups: the Food Security Cluster, the Cash Technical Working Group, the Early Recovery Cluster, and the Cash Transfer Learning Partnership (a regional group). In these coordination and knowledge-sharing settings, MC interacted with the Ministry of Agriculture, the Ministry of Gender, Children and Social Protection, UNICEF, the Food and Agricultural Organization, and other IPs working on food security, including ACDI/VOCA and the World Food Programme. Key informants in the Ministry of Agriculture reported that their collaboration on testing and validating seeds during the Ebola crisis has been extended through the Food and Enterprise Development program. These and other KIIs, including those conducted during PE1, indicated that cordial relationships and collaboration among these diverse AFS actors were the norm. MC acknowledged incorporating insights from these working groups in the EREL agricultural strategy (e.g., MC, 2015a). Shared memory of the EVD crisis Cutting across explanations of the effectiveness of the EREL project was the view, expressed by KI and FGD participants, that it met a desperate need; people were highly motivated to participate and were willing to follow EREL’s requirements. FACTORS CONTRIBUTING TO ADVANCING SELF-RELIANCE The EREL program was designed in a time of crisis with a dual purpose: to meet short-term needs of the humanitarian emergency in three counties heavily affected by Ebola (the UCTs), and to build capacity for lasting improvements in smallholder farming among the beneficiaries of the UCT intervention. USAID established its Journey to Self-Reliance after the EREL program was designed and concluded, so EREL monitoring and evaluation focused on traditional food security indicators (hunger, dietary diversity, farming practices, etc. as reported above). Nevertheless, EREL program reports, the household survey conducted by IBTCI in 2018, and the interviews and FGDs conducted for PE2 suggest that, at the household level, both EREL’s UCT and agricultural input strategies have contributed to building the two factors that comprise USAID’s vision of self-reliance: capacity and commitment (USAID, 2019). Improved capacity EREL program reports and the independent evaluation of EREL (IDA, n.d.) found direct benefits of the EREL interventions to improve farming among the 10,000 recipients of agricultural inputs. The benefits were in terms of both the knowledge and skills of good farming practices and beneficiaries’ access to viable seeds and tools that were essential for beneficiaries to restart, and/ or expand their small holder farms. The UCTs clearly increased the capacity of recipient households to feed their families, but also had longer-range indirect benefits. For example, they gave beneficiary households the capacity to return their children to school, thereby reducing the long-range impact of the EVD crisis on children’s education and welfare. As noted above, during the distribution of UCTs, beneficiaries (the majority, women) were exposed to information on nutrition and health, to increase their A–44 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME 1 capacity to improve their children’s well-being. In addition, EREL post-distribution monitoring found that overall, 4 percent of UCT benefits were allocated to livelihood improvements: they gave beneficiaries the capacity to restart or expand a small business other than farming. “There is a lady in this town who is a widow and she has two children. When she received that US$42 she went and bought oil, rice, and some items and began to sell them. Today her children are still going to school from that same money. There were some men also in this town who were not doing anything at all. When some of them received those seeds, I now see them making farm for themselves.” – KII, 2019 Increased commitment Indicators for commitment at the individual or household level have not been established for USAID’s self-reliance framework, but the construct has been recognized for decades in other frameworks for development assistance. For example, in the 1990s, USAID’s framework for participatory methods identified stakeholder commitment as “the practical benefit that is claimed for participatory approaches to the design, implementation, and evaluation of development programs” (TvT Associates, 1994:3). The evidence for EREL’s impact on commitment at the individual level is limited because it was not sought. However, FGDs with community members in areas supported by EREL illustrated that the farming beneficiaries are committed to farming because they rely on the outputs of their farms to meet their household’s basic needs. Also indirect are the indications that EREL built trust in the interventions and in the community committees, the trainers and the vendors involved, through working with known local partner organizations, by ensuring community participation and local leadership involvement in every aspect of the program, and by establishing consistency and quality in the inputs provided. Maintaining trust was critical to retaining community support, and, according to IBTCI’s framework (Moore et al., 2019), it would have contributed to the sustainability of EREL’s benefits. CONCLUSION Timely delivery of the EREL program’s cash and agricultural inputs in the assigned places to 30,000 and 10,000 specific persons, respectively, was a logistical tour de force. Using its extensive background knowledge and experience in Liberia and knowledge from the FFP community of practice, along with extensive, rapid formative research, MC designed a combination of inputs that met immediate needs for food and other essentials for 30,077 households in their three assigned counties, and set the stage for longer term improvements in food security in their communities. EREL did not include all the strategies used by FFP IPs in Liberia—for example, it did not include support to savings and loans associations or cash for work. However, through rigorous and constant monitoring, by providing mechanisms for stakeholders to express and resolve concerns, by making monitoring data widely accessible, and by working in partnership with USAID/Liberia, the EREL program successfully carried out a complex and high-risk program as planned. REFERENCES Baily, S. (2013). The impact of cash transfers on food consumption in humanitarian settings: A review of evidence. Retrieved 06/20/2019 on CaLP website: http://www. cashlearning.org/downloads/cfgb---impact-of-cash-transfers￾on-food-consumption-may-2013-final-clean-(3).pdf Food and Agricultural Organization (FAO). (n.d.). Women: The key to food security. Women and Population Division, Sustainable Development Department, FAO. Rome, Italy. Retrieved 07/03/2019 at: http://www.fao.org/3/x0171e/ x0171e00.htm#TopOfPage Government of Liberia. (2015). Agenda for Transformation. Steps toward Liberia rising 2030. Monrovia. Government of Liberia/Ministry of Health. (2015). Investment plan for building a resilient health system in Liberia 2015 to 2021. Monrovia. IDA. (n.d.) Final Evaluation Report. Economic Recovery from Ebola for Liberia (EREL). Counties of Lofa, Margibi, and Montserrado, Republic of Liberia. [File: Final Report Annex 5 – IDA Final Evaluation EREL program.docx]. Mercy Corps. (2015a). Economic Recovery from Ebola for Liberia (EREL) Program. FY15 Q2 [File: EREL Mercy Corps Liberia Quarterly Report Jan–March 2015.pdf]. CASE STUDY 8: EREL IN LIBERIA n A–45 Mercy Corps. (2015b). Lofa County Assessment. Economic Recovery from Ebola for Liberia (EREL) Program. 18 May 2015. Mercy Corps. (2015c). Economic Recovery from Ebola for Liberia (EREL). FY15 Q3 Annex IV –Agricultural Inputs Supply Chain Assessment Report.docx. Mercy Corps. (2015d). Economic Recovery from Ebola for Liberia (EREL). FY15 Q3 Annex V –Agricultural Inputs Vendor Assessment Report.docx. Mercy Corps. (2015e). Economic Recovery from Ebola for Liberia (EREL). FY15 Q3 Annex VI – Draft Market Monitoring Report.docx. Mercy Corps. (2016a). Economic Recovery from Ebola for Liberia (EREL). Annual Results Review (ARR) November 14, 2016. [File: EREL_ARR_MCLiberia_14Nov2016.pdf]. Mercy Corps. (2016b). Economic Recovery from Ebola for Liberia (EREL). FY16 ARR Annex 2 – Success story from Montserrado.docx. Mercy Corps. (2016c). Economic Recovery from Ebola for Liberia (EREL). FY16 ARR Annex 3 – Indicators List Report. doc. Mercy Corps. (2016d). Economic Recovery from Ebola for Liberia (EREL) Program. FY2016 Q1 Report. Annex 3.6.docx. Mercy Corps. (2016e). Economic Recovery from Ebola for Liberia (EREL) Program. FY2016 Q3 Report. Mercy Corps. (2016f). Monitoring Report on Household Agriculture Post Distribution (PDM). FY16 Quarter 1 – November–December 2015. Economy Recovery from Ebola for Liberia (EREL). Mercy Corps. (2017a). Economic Recovery from Ebola for Liberia (EREL) Program. Final Program Report (9 January 2015–31 December 2016). Submitted March 31, 2017. Mercy Corps. Washington, DC. [File: Mercy Corps EREL Final Program Report March 2017.pdf]. Mercy Corps. (2017b). Economic Recovery from Ebola for Liberia (EREL). Final Report Annex 1. Logical Framework and Results Framework. Mercy Corps. (2017c). Economic Recovery from Ebola for Liberia (EREL). Final Report Annex 2. Case Study: Adopting Innovative Farming in Rural Montserrado County. Mercy Corps. (2017d). Economic Recovery from Ebola for Liberia (EREL). Final Report Annex 2. Case Study: Adopting Innovative Farming in Rural Montserrado County. Mercy Corps. (2017e). Economic Recovery from Ebola for Liberia (EREL). FY17 Q1 Report Annex I – Endline HDDS and HHS Report.docx. Mercy Corps. (2017f). Economic Recovery from Ebola for Liberia (EREL). FY17 Q1 Report Annex II – PDM–Margibi County.docx. Mercy Corps. (2017g). Economic Recovery from Ebola for Liberia (EREL). FY17 Q1 Report Annex Ill – Larkayta Market. doc. Mercy Corps. (2017h). Economic Recovery from Ebola for Liberia (EREL). FY17 Q1 Report Annex IV – Agricultural Inputs PDM–Montserrado County.docx. Mercy Corps. (2017i). Economic Recovery from Ebola for Liberia (EREL). FY17 Q1 Report Annex V – Agricultural Inputs PDM–Lofa County.docx. Moore Z, Bongiovanni A, Sadaphal S, Hernandez O, Espeut D, Levin C, and Toole M (2019). Performance Evaluation of USAID Ebola Pillar II Activities: Final Report. Evaluation Report to USAID, February 2019, International Business & Technical Consultants, Inc., Vienna, VA. Rogers, B. L., & Coates, J. (2015). Sustaining development: A synthesis of results from a four-country study of sustainability and exit strategies among development food assistance projects. Washington, DC: FHI 360/Food and Nutrition Technical Assistance III Project (FANTA). TvT Associates, 1994:3 USAID. (2011). Human and institutional capacity development handbook: A USAID model for sustainable performance improvement. Washington, DC: USAID. USAID. (2015). Use of international disaster assistance funds for local and regional procurement, cash, and food vouchers under the emergency food security program report to Congress. FY 2015. Washington, DC: USAID. USAID. (2019). Policy Framework. Ending the need for foreign assistance. Washington, DC: USAID. Retrieved 06/19/2019 at: https://www.usaid.gov/sites/default/files/documents/1870/ WEB_PF_Full_Report_FINAL_10Apr2019.pdf A–46 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME 1 SIERRA LEONE CASE STUDY 9: FOOD FOR PEACE CASH TRANSFER (FFP/CT) BOX A9–1. FFP/CT AT A GLANCE Lead Implementing Partners (IP)s: 1. ACDI/VOCA for the “Sustainable Nutrition and Agriculture Promotion Plus (SNAP+)” activity 2. Care International (Care) for the “Rapid Ebola Social Safety Net and Economic Recovery (RESSNER)” activity 3. Catholic Relief Services (CRS) for the “Cash Based Food Security Assistance project for EVD Affected Communities in Kenema” activity 4. Save the Children (StC) for the “Kailahun Food for Emergency Ebola Virus Disease Support (FEEDS)” activity 5. World Vision for the “Emergency Food Assistance to Port Loko District” activity Nature of Implementation: Single sector (agriculture and food security), none implemented as a stand-alone activity Implementation Dates: Varied by IP, between April 2015 and December 2017 USAID Pillar II Budget/Obligation: USD 13,740,816 (ACDI/VOCA); USD 4,550,110 (CARE); USD 4,868,711 (CRS); USD 9,947,904 (STC); USD 6,584,958 (World Vision)1 Geographical Priority Areas: Sierra Leone districts Kailahun, Bombali, Tonkolili, Kenema, Port Loko Key Activity Components: All five activities highlighted in this case study were implemented by FFP IPs and included at least an unconditional cash transfer (UCT) component. As described in the case study, the UCTs were not implemented as stand-alone interventions; IPs implemented complementary interventions (e.g., linkages to/formation of savings groups, trainings/sensitization on nutrition and hygiene practices, and provision of agricultural inputs such as seeds). There were also differences across the activities in how cash disbursements were made (e.g., mobile money transfer or physical cash payout). Key Pillar Objectives and Outcomes to which Activities Contributed: The above Pillar II agriculture and food security activities were designed to contribute to Pillar II Objective 1 (“Prevent the loss of development gains”). They contributed to the following Pillar II Theory of Change outcomes: social protection provided, household food security increased, and school attendance restored. FFP/CT: Cash as a Stop-gap or as a Catalyst toward Resiliency? by Dr. Donna Espeut Food security was an acute need during and after Sierra Leone’s Ebola epidemic. However, food alone cannot fill the void of lost lives, income, and livelihoods resulting from a devastating crisis. Leveraging five United States Agency for International Development (USAID) Food for Peace 1. Source: USAID’s Office of Budget and Resource Management O&E spreadsheet, September 2017. (FFP) activities to implement short-term, Pillar II-supported cash transfer (CT) programs was an effective use of existing program implementation infrastructure within the country. Box 9-1 provides an overview of the five Pillar II activities that implemented CT schemes in the recovery phase. FACTORS CONTRIBUTING TO EFFECTIVENESS Across the five activities, activity effectiveness can be attributed to factors both inherent to the Pillar II activities and contextual factors that created an enabling environment for their post￾Ebola efforts. Because implementing partners (IPs) were already implementing FFP projects in the targeted Pillar II districts, they understood the local context, as well as key drivers and manifestations of vulnerability and need. They also had existing relationships CASE STUDY 9: FFP/CT IN SIERRA LEONE n A–47 with communities, local gatekeepers, and decision makers. Also, Pillar II CT activities were implemented in two distinct phases that enabled the IPs to apply learning from one phase to the next. The creation of a Cash Transfer Working Group, which was initiated by ACDI/VOCA, served as a useful platform for learning, harmonization (e.g., in CT disbursement amount), and exchange across the five Pillar II activities. Using adaptive management practices, IPs were able to employ viable solutions when they encountered implementation challenges (e.g., poor mobile network coverage in highly remote areas). IPs also made intervention linkages to non-CT interventions— whether implemented under the same Pillar II activities or through other mechanisms or platforms. Two Government of Sierra Leone (GOSL) entities, in particular—the National Commission for Social Action (NaCSA), which is the focal institution for the National Social Safety Net Program, and the Anti-Corruption Commission (ACC)—were committed and effective co-actors in the Pillar II CT activities, which helped to foster transparency and accountability in implementation. The Ministry of Social Welfare Gender and Children Affairs (MSWGCA) also actively participated as a collaborator in selected districts. Across the five CT activities, Pillar II support reached over 67,000 households (364,000 individuals) via CT interventions (Moore et al., 2019). Phased implementation contributed to activity effectiveness. Pillar II CT activities were implemented in two distinct phases that did not just entail further expansion of CT support to more beneficiaries; the phased approach allowed the IPs to also apply learning from one phase to the next, modifying program approaches to extend reach and optimize effectiveness (ACDI/VOCA, 2017; Care International, 2018; Catholic Relief Services (CRS), 2018; MSD Consulting, 2018a and 2018b; Save the Children, 2016). IPs encountered budget savings later in the implementation cycle due to the depreciation of the Leone (local currency of Sierra Leone) against the US dollar (CRS, 2018). As described by key informants for all five CT activities, as well as by a GOSL key informant, this resulted in additional CT payments, as agreed by the IPs via the Cash Transfer Working Group and USAID. The aforementioned Cash Transfer Working Group was a critical platform to exchange information, harmonize efforts, and jointly solve problems across the CT activities (MSD Consulting, 2018a and 2018b; Radice, 2017; Save the Children, 2016). Notably, Pillar II CTs were positioned as necessary but insufficient contributions to post-Ebola recovery. More specifically, for all the Pillar II FFP CT activities, CTs were not stand-alone interventions (see Table A9-1). This feature was raised in Key Informant Interviews (KIIs) with each of the five Pillar II IPs and was also described in at least five Focus Group Discussions (FGDs) with Pillar II beneficiaries in the activities’ catchment communities. As will be described later in this case study, this packaging of CTs with other recovery and resilience￾building interventions was a key contributor to advancing self￾reliance at a local level. Implementation was not without challenges. However, the adaptive management practices of the IPs were a critical success factor in activity effectiveness. One tangible illustration relates to the modality employed for cash disbursement. As described during four IP KIIs, some activities encountered major constraints in employing digital funds transfer, particularly in highly remote settings. Alternative arrangements were made with Splash Mobile Money, whose brand is rooted in digital cash transfer but who was equipped to ultimately support physical cash disbursement (“direct cash”) in locations with poor mobile network coverage (CRS, 2018; MSD Consulting, 2018a and 2018b; Save the Children, 2016). The above troubleshooting example not only illustrates adaptive management on the part of the IPs, but also highlights the important role of external partners. More specifically, the capacity and strategic involvement of private-sector mobile service providers was a factor in activity effectiveness. As described in five KIIs with IPs and three KIIs with private-sector stakeholders, private-sector mobile service provider partners Orange and Africell had been involved in Ebola response activities, using their mobile phone networks for mobile payment of health worker salaries during the epidemic. Thus, the private￾sector partners understood the practicalities of managing mobile fund disbursement when that was a feasible option for the Pillar II IPs. External engagement also extended to the GOSL and IP commitment to and added value of engaging GOSL entities are common observations across the five CT activities. More specifically, the grassroots involvement of ACC, NaCSA, and MSWGCA staff in Pillar II implementation was a facilitating factor, as underscored in three KIIs with GOSL officials. GOSL entities have field offices either in the Pillar II districts or in nearby regional headquarter towns, enabling the GOSL entities to do much more than just provide oversight at arm’s length. This created both transparency and accountability in the identification of beneficiaries and the disbursement of CT funds. It also cultivated ownership on the part of those GOSL entities. Two GOSL key informants (KIs) mentioned that the CT IPs were proactive, collegial, and willing to be coordinated by Government when implementing their Pillar II activities. A–48 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME 1 Table A9–1. Interventions that complemented the Pillar II CTs, Sierra Leone Pillar II IP Examples of Complementary Interventions Implemented Alongside CTs ACDI/VOCA ■ Village savings and loan groups (VSLAs) ■ Trainings on nutrition and household sanitation ■ Agricultural trainings (e.g., on crop production) ■ Household economic planning CARE ■ Seed voucher distribution ■ Sensitization on good nutrition and hygiene and sanitation practices ■ Geographic Information System (GIS) Capacity Building of Social Protection Secretariat of NaCSA ■ Linkages to VSLAs (non-Pillar II) CRS ■ Trainings on nutrition and hygiene, financial management, and improved agronomic practices ■ Support in the formation of farmer/savings groups, with formalized linkages to District Agricultural Office (Ministry of Agriculture, Forestry, and Food Security [MAFFS]) Save the Children ■ Beneficiary orientation sessions on nutrition, health, education, WASH, and child protection ■ Support to small-scale traders with capacity-building in business skills ■ Support to Kailahun district MAFFS with logistical support (seed distribution and fertilizers to targeted communities) World Vision ■ Conditional seed vouchers ■ Agricultural trainings ■ Savings for Transformation SOURCES: ACDI/VOCA, 2017; Care International, 2018; CRS, 2018; Save the Children (2016); MSD Consulting, 2018a and 2018b. FACTORS CONTRIBUTING TO ADVANCING SELF-RELIANCE CTs are never intended to be a sustainable endeavor. Enduring effects observed long after Pillar II implementation ended can largely be attributed to technical choices made by the IPs (e.g., implementing a complementary package of interventions, not just disbursing cash). As described in this section, the human dimension also features prominently in advancing micro-level self-reliance under the Pillar II CT activities, with Community Identification Committees, VSLAs, and farmers groups helping to bolster social capital. In addition, the Pillar II CT programs helped to advance women’s involvement and value in decision￾making—an important human dimension in sustaining results. The journey to self-reliance is often described as a macro￾level/country-level phenomenon. However, based on the implementation experiences and documented enduring effects of the five Pillar II CT activities, contributions of those activities to advancing self-reliance are most palpable at a micro level. In reality, the CTs were the fuel while other activity inputs formed the engine that is continuing to propel targeted households and communities on their journey to self-reliance. The use of CTs as part of a package of interventions—a technical choice that, as described earlier, was an important factor in activity effectiveness—was also key contributor to micro-level self-reliance. As one former staff member of a Pillar II IP stated: “The project was implemented in two phases. . . . There was a lot of improvement and lessons learned during the implementation . . . to strengthen communities to build a kind of resilience. . . . We had a lot of components that were included in the program, . . . organizing beneficiaries into farming groups as well as village savings—VSLAs…. of course, for long-term sustainability and support. In the second phase, agriculture was key because it was seen as a form of long-term sustainability.” Many households that were CT recipients joined VSLAs and remained independent after the CTs ended by relying on their savings, accessing credit and other social funds, and expanding their safety net through this group savings approach (ACDI/ VOCA, 2017; CRS, 2018). For example, the evaluation of the World Vision activity in Port Loko acknowledged that the only sustainability measures taken were those embedded in the complementary activities and that beneficiary households applied newly acquired knowledge in their farming practices— CASE STUDY 9: FFP/CT IN SIERRA LEONE n A–49 constructed their own rice seed banks and participated in savings groups to finance their petty trading and agricultural activities (MSD Consulting, 2018b). Similar resilience outcomes were noted in the CRS activity, whose end-of-activity evaluation documented significantly increased agricultural production of targeted households, resulting from increased household expenditure on agricultural livelihoods (MSD Consulting, 2018a). Gender equality in targeting and implementation was a contributor to self-reliance. For example, ACDI/VOCA deliberately targeted women as CT beneficiaries (ACDI/VOCA, 2017), which is consistent with the GOSL’s social protection principles. This feature of targeting was also described in detail during two separate FGDs with CARE and Save the Children Pillar II beneficiaries, as well as during three KIIs. Gender￾sensitive targeting was not limited to the CT component of the activities; it also applied to the complementary interventions. For example, under the World Vision activity in Port Loko District, of the 60 percent of targeted smallholder farmers for whom there was a documented increase in agricultural production from improved technologies or practices as a result of USG assistance, 1,209 were female and 807 were male (World Vision, 2015). CRS’s final evaluation also documented that gender sensitization contributed to joint decision-making and resulted in low levels of conflict (e.g., between husband and wife) in participant households (CRS, 2018). The Pillar II IPs also fostered community ownership in a number of ways, and stakeholders regard this as an important factor in advancing self-reliance.The IPs mentored and supported Community Identification Committees (CICs)/ Cash Transfer Committees consisting of members of targeted communities to pre-list all households on rosters, then select the most-needy households for the Pillar II CTs (ACDI/VOCA, 2017; MSD Consulting, 2018a and 2018b; Save the Children, 2016). This identification, vetting, and selection was done in a transparent, participatory manner. For example, in Kailahun District, Save the Children validated its final beneficiary list for CTs through community hearings held in collaboration with ACC and NaCSA (Save the Children, 2016). ACDI/VODA further fostered community ownership through its mentoring of small number of community members as Private Service Providers (PSPs). As discussed during two KIIs with IPs and reinforced by community members participating in an FGD in Tonkolii, the PSPs not only supported VSLAs established under ACDI/VOCA but they also proliferated new VSLAs after the activity ended. Lastly, IPs were mindful of exit strategies vis-à-vis any assistance provided to Government counterparts. For example, CARE handed over a Geographical Information (GIS) system, which it used to support geo-targeting of CT beneficiaries to NaCSA, for use in the GOSL’s Social Safety Net Program (CRS, 2018; CARE International, 2018). However, the overall approach undertaken by the Pillar II IPs vis-à-vis the GOSL resonates with some stakeholders, even in the present day. Gender equality was reflected in the implementation approach, and this design feature contributed to self-reliance at a micro level. “In the case of beneficiaries, we made sure that all the support was received by the women. . . . We made sure that the most senior wife in the household was the recipient on behalf of the household, for the simple reason that they are the one doing the nutrition, the feeding; they take the responsibility of the household. They manage the household economy better. . . . They are the ones that are not consulted very much in terms of decision-making. So, by making sure that they receive the money, we could be sure that they’ll be involved in the decision-making.” – KII, 2019 Possible shifts in GOSL-NGO partnership that can be leveraged in the future: “Stronger GOSL-NGO interface . . . Through the interface into these activities, this program, we strengthened the relationship between the two of us, and now as I’m talking, that has built some confidence that we can really work together. It does not matter if we’re government and if they’re NGOs. Normally, NGOs and government are two things that don’t work together at all. But we’re able to showcase that we can work together, we have common interest, [and] as long as we follow the due procedures, we can work together.” – KII, 2019 A–50 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME 1 REFERENCES ACDI/VOCA. (2017). Emergency Food Security Program (EFSP) FY 17 Quarterly Report: January–March 2017. Submitted April 30, 2017. CARE International (n.d.). Care International Partner Summary: Rapid Ebola Social Safety Net and Economic Recovery Project (RESSNER). CARE International. (2018). RESNNER Final Program Report, revised June 26, 2018. Catholic Relief Services. (n.d.). CRS Partner Summary: Cash￾Based Food Security Assistance for EVD Affected Communities in Kenema. Catholic Relief Services. (2018). Cash based food security assistance for Ebola virus disease (EVD)- affected Sierra Leoneans in Kenema district. Final results report: September 9, 2015 – February 28, 2018. Moore, Z., Bongiovanni, A., Sadaphal, S., Hernandez, O., Espeut, D., Levin, C., & Toole, M. (2019). Performance Evaluation of USAID Ebola Pillar II Activities: Final Report. 10.13140/RG.2.2.30980.78722. MSD Consulting Limited – Sierra Leone. (2018a). Cash based food security assistance for Ebola virus disease (EVD)-affected Sierra Leoneans in Kenema district, implemented by Catholic Relief Services (CRS) Sierra Leone, final evaluation report. MSD Consulting Limited – Sierra Leone. (2018b). Cash based emergency food security program (EFSP) implemented by World Vision in Port Loko district, Sierra Leone: Final evaluation report, January 2018. Nestbuilders. (2017). Sierra Leone Emergency Food Security Program (EFSP): Final project performance evaluation. Radice, H.W. (2017). Cash transfers for food security in epidemics: A review of the USAID Food for Peace response to the Ebola crisis in Liberia and Sierra Leone. Retrieved from: http://www.cashlearning.org/downloads/calp-ebola￾documentation-report-web.pdf Save the Children. (n.d.). Save the Children partner summary: Kailahun food for emergency Ebola virus disease support (FEEDS). Save the Children. (2016). Sierra Leone: Kailahun food for emergency Ebola virus disease support (FEEDS) 16 (AID￾FFP-G-15-00024) FY16 annual results report. November 14, 2016. World Vision. (2015). World Vision partner summary: Emergency food assistance to Port Loko district. World Vision. (2015). Sierra Leone EFSP Logframe. CASE STUDY 10: A GRAND CHALLENGE n A–51 SIERRA LEONE LIBERIA GUINEA CASE STUDY 10: FIGHTING EBOLA – A GRAND CHALLENGE FOR DEVELOPMENT BOX A10–1. GRAND CHALLENGES FOR DEVELOPMENT AT A GLANCE Lead Implementing Partners (IP)s: Johns Hopkins University Center for Bioengineering Innovation and Design; IntraHealth; SPR Advanced Technologies, Inc.; Columbia University – Kinnos; Scripps Health; Baylor College of Med; TOMI Environmental Solutions; IBM Research Africa; Modula S Inc.; Makerere University; International Personal Protection, Inc.; Shift Labs; Dimagi Inc.; 3D Family Productions Nature of Implementation: Multi-sector activity USAID Pillar II Budget/Obligation: USD 793,635 (Johns Hopkins University); USD 700,000 (IntraHealth International ); USD 655,788 (SPR Advanced Technologies, Inc.); USD 649,342 (Colombia - Kinnos); USD 632,058 (Scripps Health); USD 613,927 (Baylor College of Medicine); USD 559003 (TOMI Environmental Solutions, Inc.); USD 526,355 (IBM Research); USD 500,000 (Modula S Inc.); USD 482,231 (Makerere University (MAK)); USD 323,845 (International Personnel Protection ); USD 318,682 (Shift Labs, Inc.); USD 298,996 (DIMAGI, INC.); USD 268,455 (3D Family Production)1 Grand Challenge Objectives: Develop new practical and cost-effective solutions to improve infection treatment and control that can be rapidly deployed to help health care workers (HCWs) provide better care and to transform our ability to combat Ebola. Key Activity Component: Public-private partnerships Key Pillar Objectives and Outcomes to which Activities Contributed: The Grand Challenge was initiated to contribute to Pillar II Objective 3 (“build sustained systems”) especially in the areas of infrastructure, connectivity, and health information systems (HIS) to ensure regular communication and workforce compensation. It contributed to Pillar II outcomes: (1) information communication technology (ICT) systems for health, gender equity, and civic participation improved, and (2) power, water, roads, and communication infrastructure restored and expanded. Key Message: The Grand Challenge created a unique platform for fostering public–private partnership to solve development problems. It allowed USAID to mobilize traditional and nontraditional development partners to co-fund a variety of innovations that were used in the Ebola response and recovery in these countries as well as other epidemics and ongoing Ebola outbreaks elsewhere. This funding mechanism created mutual benefits by providing grantees an opportunity to demonstrate the efficiency and effectiveness of their products in a large-scale humanitarian crisis, some going further to commercialize their products. Grand Challenges: Critical New Innovations Quickly Developed and Tested in the Region by Dr. Jackie Yiptong Avila The Grand Challenges for Development mobilized governments, companies, non-governmental organizations (NGOs) and foundations to solve development problems. In 2015, the U.S. Agency for International Development (USAID) partnered with the White House Office of Science and Technology Policy, the Centers for Disease Control and Prevention, and the U.S. Department of Defense to launch Fighting Ebola: A Grand Challenge for 1. Source: USAID’s Office of Budget and Resource Management O&E spreadsheet, September 2017. Development. A variety of innovations were funded by and used in Ebola Pillar II during the response and recovery of the Ebola virus disease (EVD) outbreak. Over 1,500 proposals were received, and 14 initiatives were funded. These Grand Challenge initiatives addressed the various types of issues encountered by the health system and health care workers when attempting to provide timely care to patients during the outbreak as illustrated in Figure A10-1. Grantees ranged from for-profit companies A–52 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME 1 to nonprofit institutions, who in turn partnered with a variety of organizations including local and U.S. Government (USG) agencies (see Figure A10-2). This evaluation found that the Grand Challenge has funded innovations that were relevant to fighting the Ebola outbreak. Furthermore, these Grand Challenge Initiatives, in several cases, have resulted in products that have been used in other epidemics and are in application in the ongoing EVD outbreaks in the Democratic Republic of Congo (DRC). Some of these initiatives have shown the importance of rapid communication strategies in an emergency context and prompted the governments of Liberia, Guinea, and Sierra Leone to improve their Health Information. Several Grantees have been able to improve and commercialize their products after demonstrating through the Grand Challenge Program the efficiency and effectiveness of their products in combatting EVD. Below is a brief description of each of the 14 initiatives organized by the five types of themes that they addressed. 1. INFORMATION COMMUNICATION TECHNOLOGY Information Communication Technology (ICT) funded by the Grand Challenge has allowed health authorities and workforce and the local communities to communicate important information, which permitted the monitoring of adherence to screening and triage protocols, data dissemination, surveillance and contact tracing, diagnostics and lab tracking, and stock tracking, as well as patient self-reporting. The mHero and CommCare Grand Challenge initiatives have both had long￾term effects on the health information system of Liberia, Guinea, and Sierra Leone. mHero, implemented by IntraHealth, integrates data from several sources and uses a two-way, mobile phone-based communication system to connect ministries of health (MOHs) and health workers. mHero is now built in the Health Information System of the Liberia MOH and has been included in the strategic plan of Health Information System (HIS) of Guinea and Sierra Leone. (Refer to Case Study 11: mHero for an in-depth description.) Dimagi Inc. partnered with a large number of leading humanitarian response organizations to develop an Open Response consortium, which developed, evaluated, and shared a set of standards-based, easy-to-use, open-source mobile applications built on CommCare, its open source platform already in use. A Dimagi Inc. spokesperson who spoke to the evaluation team credits the Grand Challenge for the opportunity to break into the humanitarian crisis environment and obtain exposure with an aid organization. The company took the lessons Figure A10–1. Types of activities supported Figure A10–2. Grantees and partners 19 10 1 3 5 13 14 51 7 5 2 Private Research/foundation/education NGO Local government INGO Government agency Grantees All Partners CASE STUDY 10: A GRAND CHALLENGE n A–53 learned in Guinea, Sierra Leone, and Liberia and has leveraged its best application design and knowledge, which are based on the programmatic realities shared by its partners, to develop a series of Ebola starter applications that can quickly be deployed during a crisis. The company has free, downloadable software on its website that can easily be adapted. Since 2016, Dimagi has been working with several frontline responder organizations in West Africa to enable rapid deployment of future EVD and other outbreak response applications. (Tom-Aba, Nguku, Arinze, & Krause, 2018). For example, in Guinea, the Earth Institute, the United Nations Population Fund, and the Guinean Ministry of Health have adapted the standardized EVD contact-tracing form to a CommCare application (Tom-Aba et al., 2018). In a study conducted in 2018, CommCare was among the only three tools that contain all three key functionalities of outbreak management for EVD, namely surveillance capability, contact tracing, and case management (Tom-Aba et al., 2018). CommCare is now present in 60 countries where some 600 active programs and 30,000 frontline workers are using the application in health as well as in other sectors such as agriculture. In India alone, approximately 240,000 Anganwadi nutrition health workers are also using this system (Tom-Aba et al., 2018). 2. PERSONAL PROTECTIVE EQUIPMENT (PPE) The Grand Challenge has contributed to the design and/ or reengineering of Personal Protective Equipment (PPE) for HCWs. Johns Hopkins University Center for Bioengineering Innovation and Design (JHU CBID) and its partner Jhpiego designed the PPE ensemble, which aimed to improve HCWs’ visibility and comfort. The intent of this initiative was to test if the PPE offered increased efficiency in the doffing process thus reducing the risk of the HCWs being exposed to Ebola and other virus infection. Given the positive results, DuPont, one of the largest PPE manufacturers, signed a licensing agreement with JHU in 2015. The hoods and coveralls designed and tested during the Grand Challenge are now available as per manufacturability requirements. In 2018, Kappler, a Grand Challenge grantee, has offered a timely response to the DRC Ebola outbreak with affordable National Fire Protection Association certified single-use protective garments. The Grand Challenge has also allowed the testing of SPR Advanced Technology’s barrier technology which is a long￾lasting, spray-on barrier that creates an electrostatic field that kills and repels microbes on treated surfaces. It can potentially be used as a component in the design of effective medical PPE materials that are reusable, require fewer removals, and generate less infectious waste. This evaluation has not found evidence that this technology has been used since the Grand Challenge. 3. ENHANCE CARE, INCREASE ACCESS, AND IMPROVE WORKER SAFETY IN EBOLA TREATMENT UNITS In most emergency humanitarian service delivery operations, tents are used to house victims. The Grand Challenge awarded a grant to a team of faculty, students, and private experts led by the Resilient Africa Network (RAN) at Makerere University School of Public Health in Uganda to design a next-generation tent to support emergency humanitarian service delivery. Known as the RAN EpiTent, the redesigned Emergency Service Delivery Unit has provided a substantially improved operating environment for HCWs and patients through an innovative design that keeps the favorable features of current tent designs (simplicity, cheapness, and structural safety) with reengineered mechanisms for heat and air exchange. EpiTent is now the name of the flagship product of EpiTent Limited, a nonprofit company founded in 2017 and registered in Uganda. Several EpiTents are now in use in Northern Ugandan refugee settlements, and the product was included in the WHO’s 2016–2017 Compendium of Innovative Health Technologies for Low-Income Countries (WHO, 2018). The Baylor College of Medicine was awarded a grant to develop the Ebola Smart Pod (ESP), a novel, low-cost mobile Ebola clinical facility with integrated software and patient/supply tracking system. The ESP was built as an expandable aluminum￾based box deployable via ships and trucks. A full simulation was initiated in Houston, Texas, with simulated patients and HCWs in full personal protective equipment prior to shipping the unit to Liberia for infectious disease care. The ESP was successfully deployed in September 2017 at the Eternal Love Winning Africa Hospital in Monrovia, Liberia, as a patient isolation unit (Kim, Navarro, Michel, Soyars, Hilmers, & Anandasabapathy, 2018). The Grand Challenge provided a grant to Modula S. Inc., a specialist in the construction of transportable, ultra-efficient buildings for the Department of Defense. The funds were used to design an infection isolation clinic that is easy to disinfect, clean, maintain, and operate to keep the HCWs safe and comfortable while improving patient care and maximizing HCW efficiency. It was designed to overcome the challenges that traditional soft￾shell facilities (tents) face. This hard-shell unit features improved thermal efficiency and super-insulation, with interior surfaces and fixtures constructed from “direct-contact kill” antibacterial and/or antiviral materials. At the end of the project, the reports show that the clinic had been scientifically validated during this A–54 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME 1 experiment, but the evaluation did not find evidence of plans for further development or for commercialization. The Grand Challenge grant to TOMI Environmental Solutions, Inc., included funding to design and fabricate three versions of a mobile decontamination chamber featuring SteraMist BIT decontamination technology. The goal of this innovative chamber and technology was to reduce the time HCWs spend in full PPE and reducing the chance of virus transmission during the PPE doffing process. At the end of the project, TOMI reported that it had submitted two proposals to the Liberian Ministry of Health and Social Welfare to deploy SteraMist technology at proposed regional Serious Infection Units and to decommission/decontaminate an Ebola Treatment Unit and that it had received a great response. The firm was also continuing its internally funded research to prove the efficacy of SteraMist against EVD and other harmful organisms that affect population health. It appears that SteraMist is currently used in hospital settings, but this evaluation has not uncovered its use in epidemic settings. Another Grand Challenge grant was given to Columbia University to make the powdered additive Highlight® developed by students at Columbia University widely available for use in West Africa. This additive improves visualization of surface coverage of the disinfectant to which it is added for spraying by giving it a color that fades over time indicating to users when decontamination is complete. This additive also prevents staining. The product was successfully applied for decontamination during the Ebola outbreak and the first prototype of Highlight® was born from there. Highlight® is now a patented additive for disinfectants designed to improve the fundamental basics of frontline environmental cleaning and to eliminate human error. In 2017, the team of the Columbia’s Ebola Design Grand Challenge who helped fight the Ebola crisis in West Africa, founded the company Kinnos Inc., which commercializes Highlight®. This product has been recognized for its potential to revolutionize the standard of health care. In 2019, Kinnos won the Harvard President’s Innovation Challenge for Health & Life Sciences. After being tried in Liberia and Guinea, Highlight® has been field-tested by International Medical Corps in Haiti for the cholera outbreak and is stockpiled by large medical NGOs for hemorrhagic fever scenarios. It has been sent to DRC for the current Ebola outbreak and has been implemented in wide-scale epidemic preparedness training programs in Uganda. Kinnos has begun pilot tests of Highlight® in US hospitals to tackle health care-associated infections and antimicrobial resistance. 4. HEALTH CARE WORKERS’ TOOLS The purpose of the Grand Challenge grant to Scripps Health was to test the use of a wearable, wireless vital sign monitoring device with a sensor suite capable of continuous measurements of multiple key physiological parameters. The test was performed on three sub-groups of patients throughout their continuum of Ebola care: 1) those with suspected exposure; 2) infected patients; and 3) those in the recovery phase/post-infection. The goal of this innovation was to allow for improved, automated patient oversight, enabling increased timeliness of care for acutely ill individuals. At the same time, the automated device increases the safety of HCWs by minimizing unnecessary exposure risks and improves their productivity by allowing them to concentrate on care activities that truly require a human touch. At the end of the project, Scripps Health presented the results of this scientific study. This evaluation has not found evidence of follow-up studies or use and commercialization of the tool. The DripAssist developed by Shift Labs, Inc., is a handheld, battery-operated gravity IV infusion monitor and alarm that monitors gravity intravenous infusion rates. It was created to work in a wide variety of care settings and passed extensive technical testing in the lab as well as numerous rounds of user testing in the United States and abroad. The purpose of this grant was to test DripAssist to determine whether the device was ready for deployment in a low-resource setting such as an Ebola Treatment Unit. Shift Labs believed this device can improve the treatment of patients with Ebola by enabling more effective and efficient IV fluid and drug delivery. In fact, the tests carried out during the Grand Challenge showed that the device improved the accuracy and efficiency of patient hydration, which resulted in reduced workloads for HCWs. The monitor delivered IV fluids with precision to patients, a factor that is particularly important when treating children or elderly patients because it eliminates the risk of fluid overload. The DripAssist has become a product staple for the WHO and Médecins Sans Frontiers (MSF). ZMapp, the life-saving medication developed to treat Ebola patients, is now in use in 20 developing countries. 5. BEHAVIOR CHANGE TO ELIMINATE EBOLA The persistence of infections throughout 2014 and 2015 had several causes. Two were especially important. The first was a lethal, tenacious, and unforgiving virus. The second was the fear and misunderstanding that fueled high-risk behaviors. As long as these high-risk beliefs and behaviors continued, the virus had an endless source of opportunities to spread (WHO, 2015). Social and behavior change communication (SBCC) uses communication to change behaviors by positively influencing CASE STUDY 10: A GRAND CHALLENGE n A–55 knowledge, attitudes, and social norms. The Grand Challenge recognized that SBCC plays a crucial role in stemming the spread by calming fears, dispelling rumors, providing answers, and pulling together a coordinated response. It funded two projects at the community level. The Grand Challenge funding to IBM was for the continuation of a research initiative by this firm to develop and implement an epidemic surveillance and management tool that enables an MOH to identify outbreaks, better understand the health services on the ground, verify the situation through citizen feedback, and plan remedial actions with the insights obtained. The system developed by IBM comprised two main components: a citizen engagement module and a context awareness module. Using the data collected during the Ebola outbreak, additional research was undertaken to improve the automatic categorization of community messages and use the language and context to understand the underlying meaning behind the words and messages. This could then be used to better uncover emerging topics and trends and develop hypotheses for their origination. Overall the project showed that technology developed through this Grand Challenge could be very valuable for a Ministry of Health or District Health Management Team in preparing for and managing an epidemic situation, as well as generally managing a health system outside an epidemic situation. However, IBM found that more funding was necessary for further research and the next steps for this project, which included scaling within country and to other countries, and effectively collecting and collating enough data to be able to combine the two modules into a single model. The evaluation has not found evidence that IBM has pursued this project further (IBM, 2017). AFRICA STOP EBOLA was a music and media behavior change communication intervention in the form of a singing contest. It was inspired by the song “Africa Stop Ebola” a song recorded by some of West Africa’s most famous artists. The song was released in October 2014 to international acclaim and it quickly became a highly popular song in West Africa. MSF, a strong health partner deeply involved in the Ebola response in affected countries; was a strong supporter of this project. 3D Family Productions submitted a proposal to Ebola Grand Challenge to extend the impact of the song in Guinea. The project was designed to engage local artists, HCWs, and the media to change attitudes and behaviors about Ebola by making and disseminating songs themed around prevention, stigmatization, and support for Ebola survivors. The project developed a promotional campaign over three months. It engaged local artists to disseminate information about how to stop the spread of Ebola by writing and performing Ebola￾themed songs that appealed to local audiences in their own languages and terms. The program concluded with a televised song contest shown across Guinea with 14 participants and three award winners. In 2015, USAID commissioned a Performance Evaluation of the AFRICA STOP EBOLA initiative. This study was performed by an independent evaluator (Chirinos, 2015). The survey results indicated that on average the participants in the song contest had a good level of knowledge about how to stop Ebola. Around 90 percent of MSF health workers surveyed responded that the AFRICA STOP EBOLA initiative had been an effective way to engage communities in raising awareness about Ebola, and on average 90 percent of the general public surveyed considered that this initiative was an effective way to engage communities in Ebola prevention. The results of the evaluation suggest that musicians and celebrities can play an important role in influencing community leaders to drive social mobilization for Ebola prevention. TV and Radio. The main mode of information for the general public was effectively used in the AFRICA STOP EBOLA initiative. CONCLUSION This study concludes that the Fighting Ebola: Grand Challenge for Development has funded innovations that were relevant to handling the Ebola outbreak. It has provided the opportunity to test, redesign, and improve innovative technology and products in a real-time humanitarian crisis. For several grantees, this has meant product visibility and a new market environment. But above all, the products and services are sustainable and effective in Ebola detection and outbreaks. The evaluation recommends that USAID continue to monitor the advancement of the products and their application, especially in the current Ebola outbreak in the DRC to identify further progress made by the innovations that were funded and how they are saving lives in health crisis. A–56 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME 1 REFERENCES Chirinos, C. (2015). Africa stop Ebola: Music and media campaign to raise public health awareness, final report. Vienna, VA: USAID. IBM (March 2017). Report to USAID. Kim, E. J., Navarro, S. M., Michel, S., Soyars, C., Hilmers, S., & Anandasabapathy, S. (2018). Development of a rapidly deployable mobile medical unit for emergency disaster settings. Houston, TX: Baylor College of Medicine. Published online March 15, 2018. Tom-Aba, D., Nguku, P. M., Arinze, C. C., & Krause, G. (2018). Assessing the concepts and designs of 58 mobile apps for the management of the 2014–2015 West Africa Ebola outbreak: Systematic review. JMIR Public Health Surveillance, 4(4): e68. http://publichealth.jmir.org/2018/4/e68/ United States Agency for International Development (USAID). (2015). The Problem—Fighting Ebola: A Grand Challenge for Development. http://www.ebolagrandchallenge.net/our￾approach World Health Organization (WHO) (2015). Factors that contributed to undetected spread of the Ebola virus and impeded rapid containment. Retrieved from https://www.who.int/csr/ disease/ebola/one-year-report/factors/en/ World Health Organization (WHO). (2018). Compendium of innovative health technologies for low-resource settings, 2016–2017. Geneva: World Health Organization. License: CC BY-NC-SA 3.0 IGO. CASE STUDY 11: mHERO IN LIBERIA, GUINEA, AND SIERRA LEONE n A–57 SIERRA LEONE LIBERIA GUINEA CASE STUDY 11: mHERO BOX A11–1. mHERO AT A GLANCE Lead Implementing Partner (IP): IntraHealth Partners: Jembi Health Systems; mPowering Frontline Health Workers Nature of Implementation: Single sector (health), stand-alone activity Implementation Dates: Regional, varied by country between April 2015 and December 2017 USAID Pillar II Budget/Obligation: USD 700,000 (regional); USD 250,000 (Sierra Leone)1 Geographical Priority Areas: Libera, Guinea, and Sierra Leone mHero Objectives: An SMS-based communication platform, mHero enables two-way communication between the Ministry of Health (MOH) and the health workforce by integrating information from the Ministry’s human resources information system (iHRIS) and other health information systems (HIS). mHero’s objectives under the Ebola Grand Challenge were to strengthen the communications platform, to scale up mHero in Liberia, and to expand and pilot mHero in Guinea and Sierra Leone. Behind these objectives was the aim to overcome fragmentation in the HIS ecosystem, demonstrating the value of interoperability among the several management systems. Key Activity Components: At the central level, IntraHealth’s technology partners strengthened mHero’s web-based platform, and added online training and tools to enable users to work with and adapt the platform globally for their data collection and mass communication needs. mHero activities supported by the Ebola Grand Challenge in Liberia, Guinea, and Sierra Leone differed according to the level of data contained within iHRIS and the condition of each nation’s HIS. In all three countries, IntraHealth embedded staff and provided technical assistance (TA), mentoring, and training, building MOH staff capacity to administer, maintain, and use mHero. By enhancing and connecting existing MOH systems and by promoting interoperability and the open health information exchange (OpenHIE) architecture with government and non-governmental partners, mHero was designed to build ownership and use of the national health information system over the long term, thereby enhancing its sustainability. Key Pillar Objectives and Outcomes to which Activity Contributed: mHero’s main contribution has been to strengthen the health system in each country, thereby contributing to Pillar II restoring and strengthening existing institutions and systems (Objective 2). It has accomplished this through innovative investment in ICT interoperability, and through engaging with private telecom companies, thereby building sustainable systems through public–private partnerships, innovation, and capacity-building (Objective 3). Innovation Does Not Always Require Starting from Scratch: mHero Improved and Integrated Existing Systems and Made Them More Powerful and Sustainable by Dr. Barbara De Zalduondo The Ebola outbreak in West Africa (2014–2015) dramatically heightened interest information tools such as the mHero platform among ministries of health (MOH) of Guinea, Liberia, and Sierra Leone. They realized that a lack of communication between the MOH and health workers throughout their countries had contributed to the breakdown of 1. Source: USAID’s Office of Budget and Resource Management O&E spreadsheet, September 2017. services and had slowed their recovery (Government of Liberia (GOL), 2015; GOL/MOH 2015). In previous years, the Liberia MOH had rolled out the District Health Information System 2 (DHIS 2) (DHIS 2, 2016), and IntraHealth had assisted the MOH Human Resources Division to A–58 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME 1 digitize human resource records and store them in an integrated Human Resources Information System (iHRIS). UNICEF also had deployed its SMS-based, open source communication system, RapidPro, in Liberia and many other countries. At the height of the EVD outbreak, brainstorming at a hackathon between MOH, UNICEF, and IntraHealth leaders led them to realize they could drastically improve communication between the Ministry and its health workers, as well as improve evidence-informed management if they could connect these three existing, systems. iHRIS provided the list of health workers, their training and skills, posts, and cell phone numbers; DHIS2 provided information on and from health facilities; and with RapidPro, the MOH could send text messages to and/or receive messages from its entire workforce listed in iHRIS or some segment of it (see Figure A11-1). IntraHealth piloted mHero in Liberia in 2014, before the USAID Fighting Ebola: A Grand Challenge for Development (GC). For the GC, IntraHealth proposed to strengthen the architecture of the mHero platform (Objective 1), take mHero to national scale in Liberia (Objective 2), and “expand and pilot” it in Guinea and Sierra Leone (Objectives 3 and 4). Work on mHero in Sierra Leone began nearly a year after Guinea and Liberia started. By design, and in keeping with the scale of the Ebola GC resources, IntraHealth did not have project offices or expatriate staff posted in any of the three countries. At the global level, IntraHealth strengthened the mHero website architecture. A four-week, self-paced online course was 2. IntraHealth reported that of the 99 individuals who enrolled in the online course, only 14 completed it. No follow-up explanation was offered for this low completion rate. However, among those who finished, 91 percent rated it “useful” or “very useful” (IntraHealth, 2016). developed, with input from partners (UNICEF, Jembi Health Systems, and mPowering) covering mHero’s background, uses, and workflows. Separate tracks were provided for users, system administrators, and stakeholders, with appropriate focus and technical details for each group.2 A wide variety of HIS tools was made available on the website, from a simple calendar of international days and tips on data use, to a “technical deep dive on interoperability” (IntraHealth, 2016, p. 16). At the country level, activities varied so much that former mHero staff felt that each country experience was unique and needed attention separately (Key informant interview (KII), former mHero staff, USA). LIBERIA In Liberia, mHero was implemented and led by the HMIS unit of the Ministry of Health, with collaboration from the MOH ICT support team. IntraHealth embedded two staff members in the HMIS unit, where they served as Data Managers, while the heads of HMIS and Human Resources (HR) led the advocacy, planning, and decision-making about the development and uses of mHero. mHero KIIs for Performance Evaluation (PE) 2 stressed that building uptake of mHero by health workers required and created a culture change because there was no tradition of MOH communicating with its workforce at the county, district, or facility levels. One early use case for mHero was simply to validate the contact information and employment status of individuals on the MOH payroll. Certain departments, including Reproductive Health, saw how mHero could help them stay abreast of needs, including needs for contraceptive supplies, at the facility level. The Laboratory department saw the advantages of being able to send lab test results to remote health facilities. “We assumed the Ministry would have a million messages for [community health workers (CHWs)], but they were not used to communicating with their staff in the field. We changed the culture. At the start, people didn’t know what to do with the tech once they had it.” – KII, 2019 Champions of DHIS2 saw an enormous range of potential uses for mHero and they developed 35 use cases over the life of the project, from simple information blasts reaching thousands of health workers to narrowly targeted uses as a data collection tool (e.g., BenDor, 2015). KIs stated that the team aspired to add links to mHero to be able to draw information from the supply chain management system, the Laboratory Information System, and others (KIIs, 2019). Thus, parallel to building the technical platform, a key contribution of the mHero project was Messages developed, sent, and received RapidPro Interlinked Registry DHIS 2 Facility Data Health Worker (HW) Data Interoperability Layer Data flows from iHRIS and DHIS2 to RapidPro to enable messages to be sent Data flows from the HWs through RapidPro into iHRIS and DHIS2 iHRIS mHero HWs Figure A11–1. The mHero platform links existing systems (Source: Nicholson, 2016) CASE STUDY 11: mHERO IN LIBERIA, GUINEA, AND SIERRA LEONE n A–59 in pointing to the kinds of technical and management decisions that could be improved by being able to bridge and access the MOH’s six main databases in an integrated way. “We assumed people were accustomed to respond to TXT, but they were not used to it... If people are literally hanging cell phones from trees to get connectivity, you can’t assume a text service will be easy.” – KII, 2019 IntraHealth assisted mHero’s Champions in the HMIS Unit and the Division of Reproductive Health to establish a Technical Working Group that brought together all the main actors in the health sector. They were encouraged to contribute to and use mHero. mHero offered, and the MOH required, increasing interoperability under the OpenHIE architecture (OpenHIE, 2019), through adoption of international standards. mHero also promised to increase coherence and efficiency through a centralized approach to communicating with health workers: messages had to be cleared by the MOH and were sent by the MOH rather than by an IP. IntraHealth quarterly and final reports and KII respondents reported a high degree of interest and cooperation from partners and IPs, especially during the Ebola virus disease (EVD) outbreak. IntraHealth conducted a survey in October-November 2015 with 266 randomly selected health workers around the country to establish baselines for CHW awareness and use of mHero. At that time, 8 percent of health workers were aware of it. On this basis, the life of project (LOP) target was set at 20 percent awareness. An endline assessment nine months later found that 19 percent of health workers were aware of mHero. Most of those reported having received an SMS from the MOH (IntraHealth, 2016a, p. 18). While the 19 percent represents a large increase over the baseline, it begged the question, why were the other 81 percent still not aware of mHero and not responding to it? Four KII respondents provided two explanations: mHero sends messages to health workers’ personal cell phones. If they have lost their phones or changed phone numbers, they will miss messages. Secondly, KIs commented that health workers might have ignored incoming messages because they were receiving many spam messages from mobile network operators and mistook the mHero message for spam. However, the workflow management structure was revised so that messages sent to a cadre of health workers were also sent to the supervisors of these individuals, which has increased health worker motivation to take mHero seriously (KII former mHero staff). GUINEA With funding from Johnson & Johnson, IntraHealth had begun work converting Guinea’s health workforce information into an iHRIS before the GC (IntraHealth 2016). There was strong support from the Government of Guinea and the USAID Mission in Conakry for IntraHealth’s proposal to strengthen the iHRIS and mHero was a welcomed Ebola GC initiative. “We didn’t feel any animosity in Guinea, but the technological environment—they were just a little behind. Liberia had had DHIS 2 for a couple of years, and Guinea had not. So Guinea was just wrapping heads around what this HIS system meant, and in Liberia they already knew. Also I felt there was less bureaucracy in Liberia, for better or worse. USAID was equally supportive in both places.” – KII, 2019 As in Liberia, IntraHealth embedded mentors in the MOH to move the project forward. However, the context in Guinea was very different. First, the MOH’s HR data were being stored in a 2012 Microsoft Access database with limited capacity (IntraHealth, 2015b). IntraHealth supported the consultations required to establish standards and definitions of health worker cadres, positions, and other classes of information to be included in the iHRIS. In addition, when mHero began, the Human Resources Directorate was in the Ministry of Public Service, not the MOH, making it difficult for the MOH to access HR information (KII, former mHero staff, USA). The formality of the government culture meant that meetings and discussions had to be scheduled formally, ruling out the kind of ad hoc, “hallway” conversation among experts that had launched mHero in Liberia in 2014 (KIIs, former mHero staff). Eventually, a technical working group (Groupe Technique Elargie) was established for vetting and setting direction on mHero. In addition, a human resource unit was created in the MOH, and a management structure was set up to allow the MOH’s Bureau of Strategy and Development to serve as “service providers” of mHero to other units. The iHRIS is the lynchpin of the Ministry and its workforce, so IntraHealth adapted to the setting and focused on helping the Ministry first to obtain management authority over its human resources, and then to digitize its Human Resources for Health A–60 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME 1 (HRH) records. IntraHealth partnered with UNICEF, which had vastly more resources. Due to its “limited bandwidth” (IntraHealth, 2016a), the MOH authorized UNICEF to obtain contacts and employment details from its health workers and to house the resulting database at UNICEF instead of inside the MOH. The number of staff in the MOH with the required IT expertise was small, and they were also working with other IPs on the DHIS 2 rollout, which was a priority for the Ministry. As a result, the development of mHero was often delayed. “In Guinea, UNICEF was very strong— stronger than us—so they could go to people they wanted to get information from. So UNICEF went to health workers directly and got phone numbers directly. This was not a parallel system, which we fought against because there was no system.” – KII, 2019 IntraHealth provided technical assistance along with the mentoring and training of MOH staff and participated in the iHRIS technical working group. They also put in place the framework, guidelines and standard operating procedures, training modules, and tools to enable linking HR, facility, and health service and health outcome data, for evidence-informed decision-making at central and decentralized levels of the health system. KIs reported that the MOH included iHRIS and mHero in the Guinea National HIS Strategic Plan, indicating commitment to these system at the highest level (see also Odusote, 2015). IntraHealth consulted and included partners in the HIS space in refinements of the iHRIS and in ongoing efforts to establish and retain interoperability among HIS sources and systems. IntraHealth worked with USAID- supported IPs such as eHealth Africa, Health Financing and Governance project, Jhpiego, MEASURE, Catholic Relief Services, Research Triangle International (RTI), and IMC, as well as Deutsche Gesellschaft für Internationale Zusammenarbeit, Projet d’Appui de la Sante, and the EU PASSP project (IntraHealth 2016). While mHero was not piloted and decentralized in Guinea within the project period, it left the MOH with a road map to help guide and inspire partners to continue the work, institutionalizing interoperable, and user-focused development of Guinea’s HIS. “The MOH of Guinea is fairly new to HIS, which led to more resources and time being put towards orientation and sensitization on the functionality of iHRIS and mHero, and how this functionality can improve the ability of the MOH to effectively provide health services to the country” (IntraHealth, 2016a, p. 30) SIERRA LEONE Supplemental funding from the Global Development Lab enabled IntraHealth to assist the Ministry of Health and Sanitation (MOHS) in Sierra Leone to undertake mHero (IntraHealth 2016), beginning in August 2015. IntraHealth first collaborated with WHO, which was providing data entry support to the Ministry to digitize its health system information (IntraHealth, 2106b). IntraHealth created a “fresh instance” of iHRIS comprising 9,767 personnel records for health workers, and a separate database that included over 2,000 health volunteers for the first time (IntraHealth, 2016a). The project provided mHero training for 27 MOH staff from the Directorate of Human Resources as well as payroll and ICT staff, covering the management of mHero, development of use cases and RapidPro workflows, and practical applications of the system to aid in management decision-making (IntraHealth, 2015a and 2015b). Selected trainees were provided with Training of Trainers training as well. IntraHealth led an “interoperability review” of all the health information systems (public and private) in the health sector. It revealed that none of the users of these systems across several key management units (including Directorates of Policy, Planning, and Information; Reproductive and Child Health; Ministry of Information and Communication) knew whether their databases were interoperable. This created a stark baseline for the work needed to increase interoperability among the health system’s databases (IntraHealth, 2015b). In 2016 IntraHealth partnered with the MOHS and World Health Organization (WHO) to offer an “Interoperability Academy,” which encouraged all stakeholders to adopt a common understanding of the value of interoperability, and to commit to pursuing it. The academy concluded with the Bintumani Declaration (WHO, 2016), in which participants committed to improving the strength and interoperability of their respective information systems. Sierra Leone’s Chief Medical Officer saw the potential of interoperability to operationalize the mandate of the Health Sector Recovery Plan (2015–2020) to strengthen HMIS (GOSL/MOHS, 2015). At the Chief Medical Officer’s request, IntraHealth developers completed an InterLinked Registry that connects iHRIS and DHIS2 using a newly produced, standardized list of health facilities (IntraHealth, 2016a). With that in place, use-cases for mHero could be developed to improve health workforce management and a myriad of other applications CASE STUDY 11: mHERO IN LIBERIA, GUINEA, AND SIERRA LEONE n A–61 in health system strengthening. In addition, IntraHealth applied to Airtel (a cellular communications company) for a new short code for the MOHS, without which mHero users would have to pay out of pocket to send messages. IntraHealth supported the development of a sustainability plan, which included recommendations for incorporating mHero into MOHS training and recruitment plans, and for strengthening the training of the Ministry’s IT staff so that glitches do not disrupt use of the system as time goes on. FACTORS CONTRIBUTING TO EFFECTIVENESS mHero is one of the seven Ebola Grand Challenge projects that is still in use today. While mHero is only in use in Liberia, roadmaps were created for Guinea and Sierra Leone to use the system in the future. The concept for mHero emerged from informal discussions between IntraHealth and UNICEF experts who saw the value of improving and bridging existing HRH and HIS systems, and from strong leadership within the MOH at the Division/Directorate level. In Liberia it was anchored in national policy with high-level political support because visionary leaders in the HMIS Unit championed it, and MOH at the highest level experienced its benefits firsthand. USAID included mHero in the mHealth Compendium, 2016 one of ten innovations that had been brought to scale (K4Health, 2016). A number of factors contributed to its effectiveness. “[The Minister of Health] was at the airport en route to a meeting, when a case popped up on her cellphone. She was in Beijing—and got text messages informing her of the health situation! She was very pleased. This incident is still mentioned in meetings.” – KII, 2019 Building on existing systems The most widely acknowledged success factor of mHero was that, by design, it built upon existing systems (iHRIS and DHIS2) rather than attempting to start something entirely new. The iHRIS system and DHIS2 were in different degrees of progress in the three countries, but in all three countries, a proposal to strengthen them and to gain added value by linking them appealed to the system owners in the health ministries. In contrast, when an innovation clearly foreshadows the dismantling or abandonment of an existing system, that system’s caretakers may balk, especially if the innovator comes from the outside. “Yes, well, we didn’t really set new accountability mechanisms, we just worked within it. They set an HIS strategic plan [and] we integrated into that. We integrated into everything that already existed. We did not set up anything NEW …. [Senior MOH official] is an advocate of that—he didn’t want to see anything new. He said, mHero is new but it’s not new: it’s taking pieces and plugging them together to make this work. I think that’s what got buy-in.” – KII, 2019 Keeping it simple IntraHealth and UNICEF were striving to help the health system meet an urgent need, designing during the Ebola crisis. They examined the systems and options that were in place and used a “pieces and parts approach” to enable the MOH to communicate with frontline health workers as rapidly as possible. They identified common fields that could connect the iHRIS database with facility and health service information in DHIS2 and reached out to specific facilities or to cadres of health workers (e.g., Officers in Charge). They utilized the existing RapidPro system, relying on health workers to use their own cell phones. They did not have time or funding to conceive and build bells and whistles, and this turned out to be a great advantage. Both MOH and IP respondents cited contrasting examples of data systems that had been introduced and had collapsed under their own weight, such as the EDEWS from WHO. In contrast, mHero kept it simple, with the necessary and sufficient components to meet the Ministry’s needs (see quote next page). A–62 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME 1 Q: I haven’t heard about that! What is [electronic disease early warning system] (eDEWS)? A: It’s a WHO system meant for crisis. But it required an app, and a lot of gigabytes of data, and needed smart phones and servers, and all that. mHero was offering a solution that integrated with DHIS 2 and did not require all of that. . . . These [WHO] guys were working day and night, but I could see it would fall apart. Q: Why? A: You cannot have a system that requires two international people working day and night to keep going. It was clearly too much infrastructure; too much customization. Just too much. – KII, 2019 Built-in guardrails to ensure coherent communication with health workers A system providing access to the entire health workforce was very attractive to many IPs in the health sector. The mHero design enabled many partners to use the system, but they did not have direct access to health workers or their contact information. Intrahealth and mHero’s Champions in Liberia were aware of the chaos in the mHealth sector in Uganda that occurred in 2012 and instituted methods to prevent a free-for-all: partners submitted proposed workflows to the mHero team. If approved, the MOH sent the message out to health workers in the relevant cadres and locations on behalf of the IP, received their responses (if any), and conveyed the responses to the originating partner. Standards-based interoperability mHero’s immediate objective was to link iHRIS and DHIS2 systems to RapidPro. By adhering to the OpenHIE architecture and requiring compliance with global data exchange standards from all users, these components are linkable to others that respect the same standards. Thus, in the future, if the Ministry chooses, the mHero platform will be able to link additional information systems, such as the supply chain information system, laboratory information system, and others. Adaptability The mHero system could be adapted for an infinite variety of uses. Two counties in Liberia (Margibi and Grand Kru) adapted it for use as a disease surveillance system. Indeed, in the last six months of the award, the mHero team designed and implemented a disease surveillance application that is still in use nationally today. FACTORS CONTRIBUTING TO ADVANCING SELF-RELIANCE “mHero in Liberia was truly a multi-donor and multi-partner effort.” – (IntraHealth, 2016a, p. 19) In all three countries, IntraHealth’s methodology was to provide technical advice, training, and mentoring to the HMIS and IT staff in the MOH to build their capacity to manage, scale up, and innovate the mHero platform and/or its iHRIS components. The IntraHealth team began with a thorough review of existing health information systems, and the units and organizations involved, both in the capitals and in decentralized levels of the health system. They obtained a detailed knowledge of the varied stakeholders, including their interests and capacities, and invested effort in convening and participating in collaboration, building buy-in beyond the MOH. During the Ebola crisis, mHero was endorsed and used by a wide variety of stakeholders. The system is in full use today in Liberia, led and operated by an empowered and self-reliant MOH staff. “Partnerships with IPs were critical for encouraging resource sustainability for mHero. Several partners, such as Johns Hopkins University Center for Communication Programs (JHUCCP), MEASURE, mStar, and UNICEF, utilized mHero during the project period.” – (IntraHealth, 2016a, p. 21). In Guinea, the HIS enterprise was focused on establishing a digitized iHRIS and on rolling out DHIS2. IntraHealth’s technical assistance, training, and mentoring contributed to those objectives, which are precursors to a functioning mHero platform. In Sierra Leone, the pieces for an active mHero system were in place at the end of 2016. In all three countries, iHRIS and mHero are included in the national HIS strategy (IntraHealth 2016a). However, rolling out the strategy has not CASE STUDY 11: mHERO IN LIBERIA, GUINEA, AND SIERRA LEONE n A–63 been prioritized for either Guinea or Sierra Leone as of the time of this evaluation. Former members of the IntraHealth team and high-level MOH staff in Liberia stressed mHero’s broader impact in strengthening health systems. Before the EVD crisis and before mHero, frontline health workers were not accustomed to hearing from the MOH, and the MOH was not accustomed to reaching out to frontline workers or to receiving comments or request from them. MOH staff in Liberia stated that, by providing a vehicle to change these norms, “mHero pushed a larger conversation than anyone ever imagined.” In addition, the mHero team saw the need for an overarching management authority that could require IPs and partners to conform and make their individual systems interoperable parts of the larger HIS ecosystem, regardless of which USAID unit or donor funded them. One KI noted that, without the necessary contractual requirement from their donor and the MOH, IPs “didn’t really have a business incentive to do this work.” (KII, MOH, Monrovia). “You can’t look at an initiative as a silo, but [you] have to look at it as the sum of its parts. The parts become something better. It’s not just sending SMS to people, it’s what does that all mean? It means governance, data quality, [and] interoperability. It means technical content, what is core supervision, what is good disease surveillance. It’s actually bigger than SMS.” – KII, 2019 All three countries also identified a common threat to the sustainability of the mHero system: covering the cost of the SMS messages. While the project was underway, users of mHero were sheltered from fees for their mHero messages because UNICEF paid for them, having negotiated rates for SMS messages sent with a “short code” through RapidPro. Aside from a one-time donation of one million SMS messages free of charge, the private sector cellular network companies in each country charge for every SMS message. There was universal consensus that if health workers were asked to pay those charges, their participation in mHero would collapse. HMIS staff in the Liberia MOH acknowledged this, and cited examples of MOH staff who have proposed to “bring in the phone companies” and convince them to contribute SMS messaging to the MOH free of charge. But so far, no Telecom has agreed to come forward, nor has this cost been considered in the Ministry’s health budget. “mHero funding ends this December. But with the time we have [a county￾level Senator] who sits on the Health Committee, said, ‘Look, why don’t you bring the phone companies and sell this as their Social Corporate Responsibility?!’ Good idea. But initial negotiations were not between GOL and companies. They were USAID-provider, talking about customer pricing.” – KII, 2019 Working closely with the MOH in each country and building on the HIS systems they have in place, the mHero teams have shown that powerful innovation can come from connecting existing systems; it need not always involve inventing something new. In the HIS arena, the tendency for organizations to propose, and donors to fund, new systems and pieces of software needs to be managed, so that new pieces contribute to the interoperable, eHealth ecosystem, and so that the MOH remains a unifying source of coherent communication with its workforce. In each country, progress has been made in readiness for self￾reliance, but each of them has farther to go to reach that goal. A–64 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME 1 REFERENCES DHIS 2. (2016). DHIS 2 User Manual. Version 2.22 2016- 11-23 11:33:56. Accessed 07/04/2019 at: https://docs.dhis2. org/2.22/en/user/html/dhis2_user_manual_en.html Government of Liberia. (2015). Agenda for Transformation. Steps toward Liberia rising 2030. Monrovia. Government of Liberia Ministry of Health. (2015). Investment Plan for Building a Resilient Health System in Liberia 2015 to 2021. Government of Sierra Leone Ministry of Health and Sanitation (GOSL/MOHS). (2015). Health Sector Recovery Plan (2015–2020). Government of Sierra Leone. Freetown. Accessed 06/22/2019 at: https://mohs2017.files.wordpress.com/.../health￾sector-recovery-plan-2015-2020.docx IntraHealth. (2015a). USAID’s Fighting Ebola Grand Challenge: mHero. Quarterly Report. June 6, 2015– September 5, 2015. [file https://drive.google.com/drive/ folders/1L5HlL46KHukth6UfEKgBji02jbGrIn4E] IntraHealth. (2015b). USAID’s Fighting Ebola Grand Challenge: mHero. Quarterly Report. December 11, 2015. IntraHealth. (2016a). USAID’s Fighting Ebola Grand Challenge: mHero. Final report. June 6, 2015–September 5, 2016. Washington, D.C. November 15, 2016. K4Health. (2016). mHealth Compendium Database. https:// www.mhealthknowledge.org/resources/mhealth-compendium￾database?appSession=9U6358TSZ0RQE9DC1AZ01648T PCV77MJ29DMW4B60GJ3AJ2R86587L50N4S5IMK44O 5S6EH7W0169FANZ5866XB1G6B99ME9AM76MVOU 2C5J3581ASH4S6YF495Y21OW, accessed June 4, 2019. Odusote, K. (2015). As Ebola resurges in Guinea, so do government efforts to quash it. IntraHealth VITAL, April 2015. Accessed 06/15/2019 at: https://www.intrahealth.org/vital/ ebola-resurges-guinea-so-do-government-efforts-quash-it OpenHIE. (2019). Open HIE in Action. https://ohie.org/demo/ Nicholson, E. (2016). mHero Liberia. Endline Analysis. Presentation, September 2016. BenDor, P. (2015). Spotlight on mHero: Using mHero as a data collection tool for HIS strengthening. IntraHealth. December 2015. USAID. (2015). mHero Award. Attachment B. Program Description. File: mHero Award_AID-OAA-A-15-00025_ IntraHealth_International_Award_Cooperative_Agreement_ FE.pdf World Health Organization (WHO). (2016). Sierra Leone Health Information Systems Interoperability Workshop 2–4 August 2016. Bintumani Hotel. Accessed 06/22/2019 at: https://www.healthdatacollaborative.org/fileadmin/uploads/ hdc/Documents/SL_HIS_Interoperability_Meeting_Report_ Final__2_.pdf CASE STUDY 12: PERHS IN SIERRA LEONE n A–65 SIERRA LEONE CASE STUDY 12: POST-EBOLA RECOVERY OF HEALTH SERVICES (PERHS) BOX A12–1. PERHS AT A GLANCE Lead Implementing Partner (IP): JSI Research and Training Institute, Advancing Partners & Communities Nature of Implementation: Single sector (health), stand-alone activity Implementation Dates: July 2015–September 2017 USAID Pillar II Budget/Obligation: USD 17 million1 Geographical Priority Areas: Sierra Leone districts Bombali, Port Loko, Tonkolili, Western Area Urban, and Western Area Rural Activity Objectives: (1) Improve regulatory and policy environment to enable increased service delivery access, focusing on peripheral health units (PHUs) and the community by (a) revising the national community health worker (CHW) policy and its subsequent implementation, (b) reviewing and updating water, sanitation, and hygiene (WASH) standards and guidelines for health care facilities, and (c) developing operational guidelines for facility management committees (FMCs) and health management committees (HMCs); (2) Increase the capacity and effectiveness of the health workforce and community platforms to provide high-quality RMNCH services in line with IPC and WASH guidelines by (a) training health post staff, state-enrolled community health nurses, maternal and child health aides, and CHWs, and (b) improving facility-to-community engagement by strengthening FMCs; (3) Improve physical and operational conditions of CHPs and MCHPs in project districts, including community health units (CHUs) to enhance the quality, safety, and access to health services by (a) renovating health posts—both major (expanded) and minor (limited) renovations, with focus on WASH and IPC improvements, and (b) procuring minor medical equipment to support RMNCH services, and installing solar-powered lighting systems. Key Pillar Objectives and Outcomes to which Activity Contributed: Pillar II Objectives 1 (“Prevent the loss of development gains”) and 2 (“Recover and strengthen existing institutions and infrastructures”), and three particular Pillar II outcomes: (1) RMNCH services restored, (2) health worker capacity to deliver non-Ebola services increased, and (3) health infrastructure strengthened at community and district levels. SOURCES: Alva, S., Davis, N., Son, Y. & Stan, L. (2018). Post-Ebola Recovery: Strengthening Community Health Services: Endline Facility Assessment Report. Arlington, VA: Advancing Partners & Communities; APC Quarterly Financial Report for the Quarter Ending September 30, 2017 Rehabilitating Services and Relationships in Sierra Leone to Optimize RMNCH by Dr. Donna Espeut Prior to Ebola’s emergence in 2014, Sierra Leone’s health sector had been on a slow but steady trajectory of reconstruction and rehabilitation after decades of war (Desia, 2010; Government of Sierra Leone (GOSL) Ministry of Health and Sanitation (MOHS), 2017). The Ebola epidemic shone a light on shortcomings within the health system such as infection prevention and control (IPC) and a suboptimal 1. Source: USAID’s Office of Budget and Resource Management O&E spreadsheet, September 2017. interface between communities and health facilities (GOSL/ MOHS, 2017). Since 2012, JSI Research & Training Institute, Inc. (JSI), in partnership with FHI 360, has been implementing Advancing Partners & Communities (APC), a cooperative agreement awarded by the United States Agency for International Development’s (USAID’s) Office of Population and A–66 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME 1 Reproductive Health to implement community-based programs, with a particular focus on family planning (FP), in multiple countries (Advancing Partners & Communities (APC), 2019). In 2015, APC received Pillar II funds to implement Post-Ebola Recovery of Health Services (PERHS) in Sierra Leone (see Box 12-1). PERHS focused on improving quality of reproductive, maternal, newborn, and child health (RMNCH) service delivery at peripheral health units (PHUs), namely, Maternal and Child Health Posts (MCHPs), Community Health Posts (CHPs), and Community Health Centers (CHCs) (APC, 2017a). Rehabilitating health services to make tangible contributions to RMNCH cannot focus solely on service provision. The PERHS Pillar II activity demonstrated that restoring basic health services required a comprehensive model of renovations, capacity building, and community engagement to help Sierra Leone’s health system recover from the shocks of the Ebola crisis. In its targeted districts, PERHS was effective in breathing life into Facility Management Committees (FMCs)—a concept that was introduced years prior with the launch of Free Health Care in 2010 but had not been fully realized or supported. PERHS supported the MOHS to develop formal FMC guidelines, training curricula, and tools to optimize FMCs as structures to rebuild trust between communities and health facilities, as well as facilitate community involvement in health facility quality improvement. PERHS also made a deliberate effort to create and leverage technical convergence and synergies with other actors (e.g., UNICEF on CHW issues, and another Pillar II activity, the Health Communication Capacity Collaborative (HC3), which is also involved in post-Ebola health sector recovery). External factors such as District Health Management Team (DHMT) leadership and commitment to rehabilitate the formal health system and its relationship with communities—even in the face of broader operational challenges such as MOHS budgetary constraints and nonpayment of salaries—was also a critical success factor in PERHS’s success in its targeted districts. Lastly, the PERHS implementation experience highlighted the distinct dynamics of Western Area Urban relative to rural areas—both in terms of social cohesion issues and how health service delivery is structured (few health posts, better staffed and equipped facilities in urban settings)—an issue that will be an important consideration in future programming of a similar nature. FACTORS CONTRIBUTING TO EFFECTIVENESS During the two-year implementation period, PERHS supported 84 percent of the 365 PHUs in the five target districts (APC, n.d. Improving). The activity revitalized 305 PHUs in at least one of the following three ways: 1) improving water and sanitation; 2) installing solar power systems for lighting; and/or 3) providing basic medical equipment (APC, 2017a). A comparison of data from the activity’s baseline and endline health facility assessments (conducted in January–February 2016 and May 2017, respectively) indicates major improvements in the availability and functionality of basic infrastructure and basic medical equipment (Figure A12-1) (Alva, Cunningham, & Davis, 2016; APC, n.d. Improving). In addition, MOHS Health Management Information System/District Health Information System 2 (DHIS 2) data show incremental improvements in RMNCH care seeking over the course of implementation in all five districts; for example, the rate of facility-based deliveries at USAID-supported facilities was 2.3 percent in 2014 (height of the epidemic), 4.6 percent in 2015, and 8.7 percent in 2016. PERHS trained 950 clinical and nonclinical staff spanning 243 PHUs; for example, 575 health workers were trained on IPC and RMNCH and 81 were trained on the integrated management of child and newborn illnesses (IMNCI) (APC, n.d. Improving). Women constituted 80 percent of all PHU staff trained by PERHS (JSI/APC, 2017). PERHS staff also conducted 1,937 mentorship and supportive supervision visits (APC, n.d. Improving). Knowledge scores on key topics such as IPC, maternal health, newborn health, and child health all increased between baseline and endline (APC, 2017a). Figure A12–1. Percentage of Sampled PHUs with Infrastructure and Medical Equipment (SOURCE: Advancing Partners & Communities. (n.d.) Sierra Leone: Improving Community Health Services in the Post-Ebola Context [Infographic Factsheet]) Boreholes and wells Functional toilets/latrines Electricity from solar power Waste pits and incinerators Delivery beds Delivery kits Weighing scales Steam sterilizers 0 20 40 60 80 100 baseline (Jan-Feb 2016) endline (May 2017) 58 75 46 85 100 33 86 52 76 8 55 36 55 41 67 0 CASE STUDY 12: PERHS IN SIERRA LEONE n A–67 The above elements—all of which can be classified as supply-side enhancements—are expected features of restoring basic health services, with a focus on RMNCH. However, there was also an explicit focus on participation and inclusion as part of community engagement (see quotation next page). Two IP key informants also noted that whether in training of health workers or engaging various segments of the community (traditional leaders, mothers, youth), PERHS emphasized gender issues in planning and monitoring. FMCs consisted of persons who were either selected or elected through community-led processes, contributed to the rebuilding of trust between communities and health facilities/health service providers (JSI, n.d. Rebuilding). FMC members received training on how to elicit feedback from their fellow community members on facility-based health services, explore solutions, and communicate with DHMTs on issues that require DHMT attention (JSI, n.d. Rebuilding). A key component of the FMC’s work was the development of Facility Improvement Action Plans to address facility maintenance needs. In total, PERHS trained 2,539 FMC members, of whom 64 percent were male and 36 percent were female (JSI, n.d. Rebuilding). The activity strengthened a total 214 FMCs across its five districts and, by the end of implementation, 97 percent of supported health facilities met with their FMCs at least once every three months, up from 76 percent at baseline (APC, 2017a). The activity then worked with the Ministry of Health and Sanitation (MOHS) to adapt the FMC strategy and toolkit for national scale-up. In addition to its work related to FMCs, PERHS made other national contributions through the following policy, standards, and guideline development: ■ Support in the development of 2017 MOHS Standards and Guidelines for WASH Services in Health Facilities in Sierra Leone, with JSI being mentioned in the acknowledgments section of the national document for its contribution (GOSL/ MOHS, 2017b). ■ Development of a Community Engagement (CE) Strategy and Toolkit that guided the engagement of local communities through the FMCs. ■ Support in the revision of the National Community Health Worker Policy, 2016–2020. After the official launch of the 2016–2020 Community Health Worker (CHW) Policy in February 2017, PERHS supported the rollout of training to 1,491 CHWs in Bombali (PERHS effort led by GOAL), Western Area Urban (PERHS effort led by Action Against Hunger), and Western Area Rural (PERHS effort led by Save the Children) (JSI, n.d. Supporting).2 2. Using MOHS Health Management Information System/District Health Information System 2 (DHIS 2) data, PERHS observed incremental improvements in RMNCH care-seeking over the course of implementation in all five districts; for example, the rate of facility-based deliveries at USAID-supported facilities was 2.3% in 2014 (height of the epidemic), 4.6% in 2015, and 8.7% in 2016. Engaging women as catalysts for change, not as RMNCH clients: “We deliberately reached the women who were hard to reach. They were very important drivers in the community engagement process. They could speak from the perspective of being women, being mothers, going through a delivery process where they didn’t have anything and now we go to facilities and receive some quality of care.” – KII, 2019 Collaboration and inclusion were the cornerstone of PERHS’s entire implementation approach, and not just in its community￾engagement work. As mentioned by two senior-level MOHS officials and two IP key informants, PERHS worked with various MOHS departments and the Water Directorate of the Ministry of Water Resources at the central level, and DHMTs and district councils at the district level. There were also technical convergence and deliberate synergies with other implementers. For example, to further bolster health sector capacity in relation to IPC, USAID awarded Pillar II funds to the International Organization on Migration (IOM), with the mandate to establish IPC short-course departments and mobile training at the College of Medicine and Allied Health Services (COMAHS) in Freetown and at Tonkolili College of Health Sciences (IOM, n.d.). That Pillar II activity entailed the following: 1) establishment of IPC departments at COMAHS and Tonkolili College of Health Sciences; 2) mobile IPC Certification Training; and 3) renovation and equipment of COMAHS and Tonkolili College of Health Sciences IPC departments (IOM, n.d.). While there was no implementation overlap between PERHS and the IOM activity, the two activities each addressed institutionalization of IPC within their own scopes, with the IOM activity achieving a milestone on October 5, 2017, when COMAHS and Njala University graduated the inaugural class of IPC trainers, who will in turn cascade that training to university students throughout the country (USAID, 2017). PERHS also collaborated with other parties (e.g., UNICEF) that were also engaging the Government of Sierra Leone (GOSL) on the CHW issue (as noted by one IP key informant). There was also complementarity of effort with other Pillar II health sector implementers such as Health Communication Capacity Collaborative (HC3), as mentioned in KIIs with three different IP stakeholders. For example, as mentioned during a group interview with three IP KIs, HC3 also addressed trust issues between communities and PHUs by using community dialogues and communication interventions as the modality to foment trust and better relations in catchment areas not addressed by PERHS. A–68 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME 1 Implementation was not without its challenges, however. According to one key informant and PERHS quarterly progress reports, there was mixed success in engaging private sector players in rehabilitating WASH infrastructure in PHUs.Three commercial vendors—World Hope International, EDAL Drilling Ltd., and MAG Energy—served as commercial vendors for the solar electricity, water, sanitation, and hygiene rehabilitation work pursued under the auspices of PERHS. The differing degrees of success were largely attributed to institutional capacity issues that posed a challenge in performing the technical work within a limited time frame. External factors that affected activity performance Two Pillar II implementers and one MOHS KI mentioned that the leadership and commitment of DHMTs—even in the face of broader operational challenges such as MOHS budgetary constraints and nonpayment of salaries—was a critical success factor in PERHS’s success. Other contextual factors resulted in internal variation within PERHS. Three stakeholders for this case study mentioned that the distinct dynamics of Western Area Urban—both in terms of social cohesion issues and how health service delivery is structured—had a bearing on the activity. One IP KI described how the resourcing of health service delivery differs in Western Area Urban (more staff, more equipment but fewer health posts) than in rural settings, and subsequently the DHMT arrangement was altered to create two separate DHMTs—one for Western Area Rural and the other for Western Area Urban. FACTORS CONTRIBUTING TO ADVANCING SELF-RELIANCE PERHS’s contributions in the form of policies, strategies, and plans (e.g., in relation to FMCs) helped to advance Sierra Leone to progress in its journey to self-reliance. There are, however, overarching issues of the absorptive capacity and institutional capacity of the MOHS to leverage discrete, time-limited investments and infusions of technical support such as PERHS for longer-term gains. Notably, JSI APC has been providing further institutional support via a follow-on activity to PERHS, making it difficult to assess the enduring effects of PERHS at this time. However, both Governmental and nongovernmental stakeholders consulted for this study remarked on the GOSL’s continued dependence on donor support and, more broadly, the lack of sustained financing for the health system. Anecdotally, this has been reflected in the inconsistent/less-than-ideal maintenance and/or resupply (e.g., bulbs) of institutional enhancements provided under PERHS. Two IP KIs raised the issue of limited time as a major constraint in contributing to self-reliance, with a clear mandate from USAID to meet recovery objectives, not emphasize sustainability. However, there are identifiable factors and determinants that relate to PERHS’s ability to help advance Sierra Leone’s journey to self-reliance. For example, four KIs consulted for this case study (one Government stakeholder and three Pillar II implementers) suggested that there are persistent effects with respect to open and accountable governance via FMCs. This observation is consistent with the accountability and human dimension as key sustainability determinants. PERHS also made tangible contributions to self-reliance in the form of policies, plans, and guidelines that are still in use in today. However, there are issues related to the implementation/ operationalization of the above. Finances—more specifically, both the mobilization of sufficient resources and rational use of existing budgetary resources in the public sector—were mentioned by one MOHS informant and three Pillar II implementers as a bottleneck in Sierra Leone’s journey to self￾reliance. Two IP and two GOSL KIs spoke to the culture of external/ donor dependence, which has had negative implications on aspects of health service delivery that were addressed by PERHS, such as minor repairs and equipment maintenance. One IP KI expounded on the development paradox that exists in countries such as Sierra Leone, whereby an infusion of external resources can effect positive changes in a limited period of time but once that external support is removed, the Government is unable to allocate resources. The challenge of leveraging a single activity such as PERHS to advance self-reliance: “I know the government cannot do it alone and I’ve always said that. Governments in developing countries cannot do it alone without the help of their partners. Though government has increased its budgetary allocations, there are areas where we also need both the technical and financial support from the partners in order for us to actually have a better health service delivery. We always need partners, despite the increase in budgetary allocation, to support the health sector.” – KII, 2019 CASE STUDY 12: PERHS IN SIERRA LEONE n A–69 The remarks from one MOHS stakeholder encapsulate sentiments regarding the capacity aspect of self-reliance. One IP KI mentioned that there has been some replication of PERHS strategies and intervention models. For example, DFID (United Kingdom Department for International Development) has adopted PERHS’s model of community engagement for its new RMNCH program in Sierra Leone. Another IP key informant noted that since PERHS ended, attempts have been made on the part of PERHS’s national partners to mobilize their own donor resources to replicate aspects of the PERHS approach, particularly in relation to community engagement; however, those strategies are usually “pruned” from program budgets and not implemented. The insights gleaned from implementation of PERHS as a post￾Ebola recovery effort are relevant when considering the role of institutional support as a sustainability determinant. Over the life of the activity, attempts were made on the part of the implementer to engage the Central MOHS, DHMTs, and District Councils (the governance entities within Sierra Leone’s districts) in dialogue “to develop, implement, review, and adapt district plans . . . to support institutionalization of project interventions” (JSI/APC, 2017, p. 9). There will be a need to revisit issues of sustainability and self￾reliance when JSI’s follow-on activity to PERHS, which focuses on a subset of PERHS interventions for a subset of PERHS-supported health facilities, comes to an end (JSI personal communication, February 26, 2019). “Well I think that was why probably USAID through JSI supported the renovation, refurbishment . . . to reduce the number of Maternal and Child deaths at community levels. Those were some of the things that they took into consideration during the recovery, . . . critical issues that I believe were factors that drive the resilience of the health sector and once they are addressed I’m sure we will be able to withstand whatever infectious diseases that may come.” – KII, 2019 REFERENCES Advancing Partners & Communities. (2019). Retrieved from: https://www.advancingpartners.org/sites/default/files/apc_one_ page_write_up_final.pdf Advancing Partners & Communities. (2017a). Post-Ebola recovery: Strengthening primary and community health services. Arlington, VA: Advancing Partners & Communities. Advancing Partners & Communities (2017b). Quarterly Financial Report for the Quarter Ending September 30, 2017. Advancing Partners & Communities. (n.d.). Sierra Leone: Improving community health services in the post-Ebola context [Infographic Factsheet]. Alva, S., Cunningham, M., & Davis, N. (2016). Post-Ebola health services strengthening: Baseline facility assessment report. Arlington, VA: Advancing Partners & Communities. Alva, S., Davis, N., Son, Y. & Stan, L. (2018). Post-Ebola recovery: Strengthening community health services: Endline facility assessment report. Arlington, VA: Advancing Partners & Communities. Desia, A. (2010). Bulletin of World Health Organization, 88:725– 726. Government of Sierra Leone, Ministry of Health and Sanitation. (2017a). National Health Sector Strategic Plan 2017–2021. Government of Sierra Leone, Ministry of Health and Sanitation. (2017b). Standards & guidelines for water, sanitation, and hygiene (WASH) services in healthcare facilities in Sierra Leone. IOM. (n.d.). Statement of Work, International Organization for Migration Grant (AID-GH-IO-00001), Amendment 1: IOM National Ebola Training Academy—Phase II: Establishment of Infection Prevention Control Short Course Departments and Mobile Training in Sierra Leone. JSI/APC. (2017). Advancing Partners & Communities Post￾Ebola health services strengthening: Sierra Leone quarterly report, quarter 3 (April 1–June 30, 2017). JSI Research & Training Institute, Inc. (n.d.). Technical Brief. Supporting Sierra Leone’s community health using MOHS Health Management Information System/District Health Information System 2 (DHIS 2). JSI Research and Training Institute. (n.d.). Technical Brief. Rebuilding trust in health services: Engaging communities through facility management committees. USAID. (2017). USAID supports landmark achievement in infection prevention control. Retrieved from: https:// sl.usembassy.gov/usaid-supports-landmark-achievement￾infection-prevention-control/ A–70 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME 1 SIERRA LEONE LIBERIA GUINEA CASE STUDY 13. HEALTH COMMUNICATIONS CAPACITY COLLABORATIVE (HC3) BOX A13–1. HC3 AT A GLANCE Lead Implementing Partner (IP): Johns Hopkins University Center for Communication Programs (JHU CCP) Health Communication Capacity Collaborative (HC3) Nature of Implementation: Single sector (health), stand-alone activity Implementation Dates: July 2015–August 2017 USAID Pillar II Budget/Obligation: USD 5.5 million (Guinea); USD 2.6 million (Liberia); USD 5.485 million (Sierra Leone)1 Geographical Priority Areas: Regional Key Activity Components: Although there was variation across the three countries in the level of effort dedicated to each component, Pillar II support entailed a combination of the following: (1) demand generation (through nationwide mass media campaigns and targeted Community Dialogues in focus geographies), (2) health facility rehabilitation, and (3) SBCC technical support and capacity strengthening of Ministry of Health counterparts. Key Pillar Objectives and Outcomes to which Activity Contributed: HC3 was designed to contribute to Pillar II Objectives 1 (“Prevent the loss of development gains”) and 2 (“Recover and strengthen existing institutions and infrastructures”). In reference to the Pillar II Theory of Change, there are three particular outcomes of relevance: (1) basic health service delivery restored, (2) health behavior gains enhanced and reinforced, and (3) health infrastructure (including community involvement) strengthened. Multi-Country Insights from the Health Communication Capacity Collaborative by Dr. Donna Espeut The Health Communication Capacity Collaborative (HC3),1 which was implemented by The Johns Hopkins University Center for Communication Programs (JHU CCP) between October 2012 and September 2017, was USAID’s flagship global social and behavior change 1. Source: USAID’s Office of Budget and Resource Management O&E spreadsheet, September 2017. communication (SBCC) program (HC3, 2017d). In 2015, USAID’s Global Health Ebola Team awarded Pillar II funding to HC3 to implement SBCC interventions to restore trust and health services as part of post-Ebola recovery efforts in Guinea, Liberia, and Sierra Leone (see Box A13-1). FACTORS CONTRIBUTING TO EFFECTIVENESS After the Ebola epidemic, how did the three Ebola-affected West African countries build trust and confidence in health systems that did not garner widespread public trust even before the crisis? HC3 rebuilt trust and confidence in Guinea, Liberia, and Sierra Leone’s public-sector health systems by combining mutually reinforcing strategies in communities, in health facilities, with media outlets, and with national and sub-national health authorities. By conducting strategic and repeated engagement with the community with active support from local leaders, HC3 promoted behavior change with a Healthy Life media campaign and through improving the quality of care in health facilities with minor health infrastructure improvements engaging community involvement. However, factors such as limited social cohesion issues in urban versus rural settings emerged as important considerations for current and future SBCC programming that rely on repeated community engagement and community action. CASE STUDY 13: HC3 IN LIBERIA, GUINEA, AND SIERRA LEONE n A–71 HC3’s Pillar II activities entailed mutually reinforcing program components (SBCC support to government health authorities, mass media campaigns, community dialogues, and health facility “makeovers”) to create an enabling environment for meaningful community buy-in and participation in the restoration of trust and health services in the post-Ebola era. Engagement of and technical support to Government health authorities In all three countries, the primary government counterpart was the Ministry of Health (MOH) and its sub-national health authorities (e.g., county or district health management teams). HC3 worked collaboratively with the aforementioned entities on SBCC planning and management, not just technical implementation (HC3, n.d.). As stated in two separate key informant interviews (KIIs) with Pillar II implementers, HC3 pursued advocacy meetings with sub-national health officials, supported the development of SBCC action plans, and supported the development of national strategies and plans (e.g., Sierra Leone’s National Health Promotion Strategy). These comprised efforts to institutionalize some of the best practices introduced under the auspices of the Pillar II activity (HC3, 2017a; HC3, 2017d). Raising awareness, rebuilding trust, and sparking community involvement via Community Dialogues and nationwide mass-media campaigns HC3-supported Community Dialogues (for community-based discussion about the quality of services and community-driven changes in facilities to enhance quality of care) that included key decision-makers and/or “gatekeepers” of behavior change and health access such as representatives of women’s groups, traditional and religious leaders, health workers, and elected local officials (JHU CCP, 2017). Mass media and distance-learning radio modules were nationwide in scope but also included grassroots reinforcement mechanisms such as radio-listening groups to elevate knowledge, attitudes, and self-efficacy related to health-care seeking (HC3, 2017c). According to an endline assessment conducted by HC3 in Liberia, 86 percent of sampled women could recall or were exposed to at least one message from HC3’s mass media campaign (Healthy Life) and 47 percent of all women had high exposure to or recall of three or more Healthy Life messages (HC3, 2017c). Another qualitative assessment in HC3 catchment areas documented that community members reported a notable improvement in the quality of health service delivery post-Ebola (Modarres & Berg, 2016). Positioning the community at the center of the demand-side/supply-side nexus This was a key factor in effectiveness. Facility enhancements were born out of community engagement efforts and HC3 promoted mutual accountability between communities and health facility staff through Makeover Committees or Facility Management Committees (FMCs), which provided oversight (HC3 2017d; HC3 2017f; USAID, Dalan Consulting, & CCP, 2016). In Guinea, demand generation entailed both rehabilitation of some health facilities and the marketing and promotion of enhanced services under the Gold Star (Etoile d’Or) brand, which existed before the Ebola outbreak and was revitalized by HC3 to convey a high-quality standard to the public. It also established accountability on the part of service providers to meet Gold Star’s quality standards for service provision (JHU CCP, 2017). When exploring key drivers of activity performance, Pillar II implementers in four different KIIs highlighted differences between urban and rural settings. More specifically, stakeholders mentioned that, in retrospect, activity design should be responsive to differences in the nature and extent of social cohesion in urban settings (which has implications for community-centered, community-driven models such as those adopted by HC3) and the extent to which health service delivery is structured (e.g., fewer first-level health facilities such as health posts), and/or resourced (more health workers, better-equipped sites to accommodate higher client volume, and flow in urban areas). Link between demand-side (community engagement) efforts and quality improvement in health facilities: “You had a series of those [community] engagements and at the end of the day, there was a makeover plan. The communities will say, ‘These are the changes we want to see and this is what we’ll be able to contribute to make this thing happen.’ . . . People are able to say, ‘I will give my time, even if I am not able to donate.’ They will clean the facility, fence the facility, divide the work among catchment areas, and identify skilled men to assist with renovation.” – KII, 2019 A–72 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME 1 Explicit focus on synergies and complementarity of effort HC3 had a very specific scope but did not operate in a silo. As raised in five KIIs with Pillar II implementers, throughout HC3 implementation there was an explicit focus on synergies and complementarity of effort with other implementers. For example, in Guinea, the Maternal and Child Survival Program (MCSP), which was implemented by Jhpiego, had supported health facilities to achieve quality standards to meet Gold Star criteria since 2012 (JHU CCP, 2017). To minimize duplication of effort, HC3 committed a lower level of effort to facility enhancements and a higher level of effort to demand￾generation efforts. In Liberia, HC3 invested a high level of effort in a national mass-media campaign called Healthy Life, an MOH brand for health messages and quality services. Where geographical convergence existed with other Pillar II IPs, HC3 pursued complementary activities that were within its Pillar II scope of work but were not duplicative of the work of others. For example, in three counties in Liberia, HC3 pursued community engagement activities in the catchment areas of most health facilities that were enhanced/supported by the MCSP/Restoring Health Services activity (HC3, 2017b). Similarly, in response to the fact that a Pillar II activity implemented by Partnership for Advancing Community-Based Services (PACS) had an extensive community engagement component, HC3 deemphasized community engagement activities in PACS areas and redoubled efforts related to other aspects of HC3 such as the mass-media campaign (HC3, 2017b). Similar implementation arrangements existed in Sierra Leone. According to three Sierra Leone Pillar II IPs, the Advancing Partners & Communities (APC) activity had a much larger scope related to facility enhancements than HC3, and where there was geographical convergence between the two activities, HC3 focused on community engagement and demand-generation activities to create synergy with the facility rehabilitation being performed by APC. Cooperation and collaboration were not limited by country borders South-to-south cooperation between HC3 IPs in the three Pillar II countries was another contributor to activity effectiveness. In both Guinea and Liberia, CCP had a presence in the country prior to Ebola’s emergence (HC3, 2017b; HC3, 2017c; HC3, 2017d). The Ebola Communication Preparedness Kit was developed in Liberia, the first country of implementation under Pillar II, but later adapted and used in Guinea and Sierra Leone (HC3, n.d.). FACTORS CONTRIBUTING TO ADVANCING SELF-RELIANCE The supply-side enhancements were a demand generator, and they are still evident in the present day. “Because of the PHU we are now healthier than before. There is room for more people to come and be treated in the PHU.” “The PHU is now comfortable for us to come and have our necessary treatment.” “After the training of the PHU staff, they are doing their job well, with care and in order.” – Feedback from different clinic attendees at a first-level health facility supported by HC3 in urban Sierra Leone Although building self-reliance was not an explicit objective of HC3, the activity advanced the three West African countries’ journeys. First, HC3 enhanced institutional capacity by transferring skills, for example, for developing materials and implementing the multiple components of a national media campaign to the MOH at the central and district levels in all three countries. Second, HC3 helped develop or disseminate policies, strategies, and plans such as the National Health Promotion Strategy 2017 in Sierra Leone and Liberia’s revised Community Health Services Policy. Third, HC3 expanded civil society and media capacity, for example, through the engagement of local radio stations in the Healthy Life campaign in Liberia, the Get Kol Art Pik Welbodi Reproductive, Maternal, Newborn, and Child Health (RMNCH) campaign in Sierra Leone, and the pursuit of a branding campaign (Gold Star) in Guinea to identify health facilities with high standards for quality of care. Fourth, HC3 engaged civil society in HC3’s community engagement efforts such as Community Dialogues on RMNCH and community-facilitated radio discussion groups. HC3’s community-centered approaches helped build trust and accountability between health care providers and communities. However, without complementary efforts that systematically address other factors that can erode trust and confidence in the formal health sector (e.g., health workforce issues, financial access barriers to health care seeking, and drug availability at service delivery sites) the enduring impact of an activity such as HC3 will be limited. Although HC3 ended in 2017, JHU CCP still has a presence in Sierra Leone under the auspices of Breakthrough ACTION, whose scope and mandate differ from HC3’s but still reinforce some of the social and behavior change features of JHU CCP’s Pillar II HC3 activity (JHU CCP, n.d.). Two KIs (one involved in Pillar II implementation and another a government official) noted that there was a vision for community health workers (CHWs) who were engaged in community- CASE STUDY 13: HC3 IN LIBERIA, GUINEA, AND SIERRA LEONE n A–73 based activities under HC3 to play a formal role in facilitating Community Dialogues after the activity ended. However, financing of CHWs is still evolving and there is a need to explore other means of incentivizing CHWs to continue their facilitation role. There are broader issues around institutional support for community health that were beyond the scope of HC3— bridging the divide between communities and facilities is yet to be institutionalized within the health sector. HC3 contributed to the development and/or dissemination of strategies, plans, and tools such as the National Health Promotion Strategy 2017 and National RMNCH Message Guide in Sierra Leone; dissemination activities related to the Liberia’s revised Community Health Services Policy (HC3, 2017b; HC3, 2017d). Those resources are being used in different ways by SBCC implementers. One HC3 sub-implementer and one senior government official in Sierra Leone mentioned the replication of HC3 strategies in other projects. There was also an investment in strengthening institutional capacity in messages and materials development. This was an explicit focus in Liberia, and HCP sponsored an MOH staff person to attend a regional Leadership in Strategic Health Communication workshop in Nigeria in 2016 (HC3, 2017e). There is evidence of supply-side enhancements persisting after HC3 has ended. However, in one FGD with community beneficiaries in a former HC3 district in Guinea, respondents mentioned how the realities of limited availability of qualified health workers and the transferring of trained/qualified health workers with whom they built rapport and trust to other catchment areas, counteract the inputs made under HC3. Meaningful community engagement is necessary but not sufficient. Without continued investment in contextual factors that erode trust in formal health services—for example, financial and nonfinancial access barriers and quality-of-care gaps—the sustained impact potential of community ownership of and involvement in health may not be fully realized (Modarres, 2016; USAID et al., 2016). REFERENCES HC3 in Action. (2017a). Strengthening the SBCC System. Health Communication Capacity Collaboration. (2017b). Liberia End of Project Report, May 2017 Health Communication Capacity Collaborative. (2017c). Liberia Endline Assessment. Baltimore: Johns Hopkins Center for Communication Programs. Health Communication Capacity Collaborative. (2017d). Sierra Leone Country Program End of Project Report: Using Social and Behavior Change Communication to Increase Trust and Address Reproductive, Maternal, Newborn and Child Health in Sierra Leone, August 2017. Health Communication Capacity Collaboration. (2017e). Year 5 Semi-Annual Report October 1, 2016–March 31, 2017. Submitted June 21, 2017. Health Communication Capacity Collaborative (HC3). (2017f). Social Mobilization Lessons Learned: The Ebola Response in Liberia. Baltimore, Maryland: Johns Hopkins Center for Communication Programs. Retrieved from: http://pdf.usaid.gov/ pdf_docs/pa00mvks.pdf Health Communication Capacity Collaborative. (n.d.). HC3 Responds to Ebola [Factsheet]. Johns Hopkins University Center for Communication Programs. (2017). HC3 Guinea Country Program Research Report: Findings from an Endline Quantitative Phone Survey, November 2017. Johns Hopkins University Center for Communication Programs. (n.d.). Where we work. Retrieved from: https:// breakthroughactionandresearch.org/where-we-work/ Modarres, N., Berg, K. (2016). Qualitative assessment on health system trust and health service utilization in Liberia. Health Communication Capacity Collaborative, Baltimore: Johns Hopkins Bloomberg School of Public Health, Center for Communication Programs. USAID, Dalan Consulting and CCP. (2016). Ebola Response and use of Health Services in Sierra Leone: Qualitative Study: Final Version December 2016. A–74 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME 1 Empowering Communities to Engage in Liberia by Dr. Barbara De Zalduondo 1. Source: USAID’s Office of Budget and Resource Management O&E spreadsheet, September 2017. LIBERIA CASE STUDY 14: PARTNERSHIP FOR ADVANCING COMMUNITY￾BASED SERVICES (PACS) BOX A14–1. PACS AT A GLANCE Lead Implementing Partner (IP): International Rescue Committee (IRC) Partners: Population Services International (PSI) (focus: health promotion); Global Communities (GC) (focus: Water, Sanitation, and Hygiene [WASH]); Jhpiego (focus: clinical standards); and YMCA and Planned Parenthood of Liberia (focus: strengthening civil society organizations [CSOs]) Nature of Implementation: Single sector (health), stand-alone activity Implementation Dates: October 2015–December 2017 (ongoing since February 2015; continuing until September 2019) USAID Pillar II Budget/Obligation: USD 7 million1 Geographical Priority Areas: Liberia national Activity Objectives: 1) Broadened capacity of Ministry of Health (MOH), Community Health Teams (CHTs), non￾governmental organizations (NGOs), and community organizations to implement and manage community services; (2) increased availability of community-based health and social welfare services; (3) improved health-seeking behavior and practice; and (4) improved access to safe WASH services. Key Activity Components: PACS was designed to yield sustainable improvements at the district and community levels of the health system, through technical support and resources to three government ministries (Ministry of Health (MOH), Ministry of Gender, Children, and Social Welfare [MGCSW], and the Ministry of Public Works [MPW]), and by building the capacity of Civil Society Organizations (CSOs) that were known and trusted at the community level. PACS’s work with Government and nine CSOs in the six counties mobilized and supported sustainable increase in community engagement with health promotion, as well as with the health facilities and the community-level administrative structures such as Community Health Committees (CHCs) and the Health Facility Development Committees (HFDCs) that serve them. The addition of Ebola funds enabled IRC to expand its geographic reach, and also added Infection Prevention and Control (IPC) to its key results. Key Pillar Objectives and Outcomes to which Activity Contributed: PACS was designed to contribute to Pillar II Objectives 1 (“Prevent the loss of development gains”) and 2 (“Recover and strengthen existing institutions and infrastructures”). It contributed to Pillar II outcomes: (1) basic health service delivery restored, (2) health behavior gains enhanced and reinforced, and (3) health infrastructure (including community involvement) strengthened. Key Message: Working in teams and with the Government of Liberia (GOL), IPs, and CSO partners, PACS’s health service, health promotion, community engagement, and infrastructure activities leveraged synergies between individuals, sectors, health system levels, and community priorities to improve health behavior and to strengthen sustainable access to Liberia’s Essential Package of Health Services at the community level (Government of Liberia Ministry of Health [GOL MOH], 2015a). An independent midterm evaluation (in the three base counties) found that PACS-supported communities and districts improved more significantly on 9 out of 15 global health indicators than comparison communities and districts (Liberia Statistical Analysis [LSA], 2017, p. 15, 64). However, continued success is contingent upon continuity in resources for the training, incentives, drugs, and logistics required for implementation and monitoring of the National Community Health Services Plan (NCHSP). CASE STUDY 14: PACS IN LIBERIA n A–75 PACS was launched at a time of profound change in Liberia’s health system (GOL 2015a; GOL MOH, 2015a). Lessons learned from the Ebola Virus Disease (EVD) crisis led the MOH to revise its National Community Health Services Policy (GOL MOH, 2011), expanding the country’s commitment to delivering health services at the community level and creating two new cadres of health workers (HWs): Community Health Assistants (CHAs) and Community Health Services Supervisors (CHSS) (GOL MOH, 2015c). PACS provided technical support to three divisions 2 of the central MOH to roll out the revised National Community Health Services Plan (NCHSP), the Community-Based Information System (CBIS), and the National Health Communication Strategy, including the training, supervision, data collection, and reporting work required. With the Ministry of Gender, Children, and Social Welfare (MGCSW) PACS refined and validated the Kinship Care, Foster Care, and Supported Independent Living Guidelines (GOL MGCSW, 2017). 2. The three divisions in the MOH that were supported by PACS are Community Health Services, Health Promotion, and Environmental and Occupational Health. PACS designed and implemented sub-awards with nine CSOs to map and strengthen links between the communities and health facilities and the health system actors in their catchment areas. In the first year, PACS established offices in all six counties and mapped the 91 health facilities, 1,442 Community Health Committees (CHCs), and 91 Health Facility Development Committees (HFDCs) in the 18 districts in those six counties where PACS was mandated to operate (see Table A14-1). In addition, PACS’s WASH team (led by GC) strengthened the MPW and the capacities of “natural leaders” to catalyze and support community engagement in WASH, from digging wells to supporting behavior changes to reduce open defecation and move toward Community-Led Total Sanitation (CLTS) certification. PACS also provided some support to the MGCSW on quality standards for orphanages and policies related to fostering. Table A14–1. Health Facilities and Community Structures in PAC’s 18 Districts County Health Districts Health Facilities Communities CHC HFDC Lofa 3 (of 6) 22 210 210 22 Bong 4 (of 8) 16 451 451 16 Nimba 2 (of 6) 18 140 140 18 Marbigi 2 (of 4) 11 159 159 11 Grand Bassa 3 (of 6) 8 147 147 8 Rural Montserrado 4 (of 5) 16 335 335 16 PACS Total 18 (of 35) 91 1,442 1,442 91 SOURCE: IRC/PACS, 2017b, p. 3 FACTORS CONTRIBUTING TO EFFECTIVENESS Stakeholder participation “There are many reasons [for high performance in County X]. We had a very strong counterpart. . . . Our team lead there was very good; they worked together, made plans, and went to the field together. So communities didn’t see PACS, they saw the county and saw PACS as just giving a push to get the county moving.” –KII, 2019 PACS’s greatest success factor was baked into the project in each component of its design and carried through in implementation. In its work with MOH divisions and at the county level, PACS advisors sat and worked with their counterparts, and advised, supported, and mentored them in everyday work. PACS tailored its support to organizations according to the specific steps in the Performance Improvement Plans (PIP) they developed together. PIPs were developed through a participatory self-assessment that was then validated by PACS. PIPs were formulated for Ministry Units, CSO partner organizations, and CHCs and HFDCs at the community A–76 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME 1 level. The PIPs of the community partners mapped out an evidence-based, organizational development pathway that the organizations had helped create for themselves, including formulating a strategic plan, monitoring and evaluation (M&E) plan, standard operating procedures, and developing explicit policies for administrative functions such as communications, 3. Prior to the revised CHSP, health facilities had trained and worked with General Community Health Volunteers, or gCHVs. In the new strategy they were renamed CHVs. 4. The majority of those communities were in the PACS base counties (Bong, Lofa, and Nimba). Only 234 of the 953 from PACS original and 234 from the close PACS modification counties achieved the ODF certification. human resources (HR), procurement and supply chain, finance, and grant-making. PACS staff helped and mentored their counterparts’ management of these processes; achieving milestones on these PIPs were key indicators that PACS reported annually to USAID (IRC, 2017a, pp. 19, 23). Strengthening the CHCs The principal community-level structures supported by PACS (the CHCs and HFDCs) had existed on paper, but the Year 1 mapping found less than half were active. Through its CSO partners, PACS strengthened or rebooted the CHCs in 1,492 communities, providing training on their mandates and responsibilities and standard operating procedures for their activities. It worked with community leaders, building their skills to mobilize people around the community’s health needs. Each CHC had the important role of managing recruitment of individuals from and dwelling in their community to serve in the new cadres (CHAs and CHSSs) ensuring that these envoys for health both met explicit criteria (literacy, numeracy, etc.) and were embraced by the community. Posted in communities located more than 5 kilometers from a health facility, CHAs were trained and tasked with delivering integrated clinical case management (iCCM) of common childhood illnesses (e.g., malaria, respiratory infections, diarrhea) and with promoting healthy behaviors using the Education Through Listening (ETL) method. The focus of health promotion ranged from IPC to antenatal care and facility-based delivery, dispensing family planning, and referring cases beyond their capacity to the local health facility. CHSSs were charged with visiting their 10 CHAs twice each month to provide support and to gather reports for entry into the Community-Based Information System (CBIS), which was integrated into the District Health Information System 2 (DHIS 2) and reviewed quarterly at the county level. General community health volunteers (gCHVs), a preexisting cadre3 with less training and fewer qualifications, continued to serve in their catchment areas within 5 km of a facility, and work largely on health promotion and referral, under supervision by the facility’s Officer in Charge (OIC). “The Ebola prevention awareness that IRC did was effective to the community… They continued their supports to the community till the Ebola was over and also during the recovery period. They were at the front page. Through those preventive messages, the community members started to wash their hands, people were cleaning their surroundings, and people started to visit the clinic whenever they felt any symptom of sickness.” – FGD participants, 2019 Participation in WASH component “We did the Community-Based Information System (CBIS) with [EEF] Mod funds. There were other partners helping develop the system, but with GHET funds we helped train the data clerks in all six counties. That is a big step forward for the country, in terms of being able to collect and manage community-level data, and having it feed into DHIS 2. – KII, 2019 PACS also took a participatory approach for the WASH component. It worked with “WASH entrepreneurs” and “natural leaders” and community members managed by DPW, who received training on IPC, well and latrine construction and maintenance, and Community-Led Total Sanitation (CLTS). Their job was to “trigger” community mobilization for WASH goals and to provide technical support on the activities (especially latrine construction, well-digging, safe burial spaces, and household IPC) needed for community certification as CLTS. As of project Y4 Q2, 1,186 communities had been certified Open Defecation Free (ODF).4 In addition, 280 “WASH entrepreneurs” were trained to market and distribute Waterguard—a chlorine-based product to make drinking water safe—through community radio call-in shows, as well community-based events (IRC/PACS, 2018). CASE STUDY 14: PACS IN LIBERIA n A–77 In all these activities, PACS staff led from behind, promoting genuine understanding of the rationale and value of the 5. CHAs’ iCCM training included diagnosing and providing ORS for diarrhea, Artemisinin for malaria, antibiotics for acute respiratory infections, and Acetaminophen for fever. recommended changes, and enhancing beneficiaries’ ownership of the methods and results. Working at the community level PACS’s technical choice to work at the community level through established CSOs that had preexisting relationships in the relevant communities smoothed its entry into communities, built trust, and enhanced sustainability. The selection of CSOs was based on a comprehensive institutional assessment, where the strengths, weaknesses, and comparative advantages of each CSO were identified (IRC, 2015b). It is axiomatic that working through trusted channels increases the effectiveness of community development efforts (e.g., Chambers, 1994). In addition, PACS trained its partner CSOs in the Education through Listening (ETL) approach to health promotion, which helped them to promote IPC, discourage home births, and restore trust in and use of the health facility. The ETL skill is highly transferable to other health and development topics. External factors On accessibility and receptivity: “For some areas, we can look at [location X in Grand Bassa county] was more performing in general. It was easier, smoother due to accessibility. [location Y] was challenging—the road, internet access, infrastructure—yet we still worked. Because everybody cared about EVD. There were several organizations, [and] they were eager to learn anything they could. – KII, 2019 A number of external factors, including geography and the memory of the EVD crisis, also contributed to PACS’s results. One that was mentioned often by KIs and focus group participants was the receptivity of government, IPs, and community members to assistance and advice after the trauma of the EVD crisis. Another factor mentioned repeatedly was geographic accessibility. When asked to explain differences between higher- and lower-performing PACS-supported sites, the most frequently mentioned factor was accessibility—in terms of distance from the county or district seat, road conditions, and cellular communication coverage. Some GOL managers noted that community proximity to HCFs was a success factor, relating this to the greater ease of access by supervisors, which avoided decays in CHA and CHVs’ services and reporting. The receptivity of implementers and community stakeholders also resulted from the harmony between the NCHSP’s mandates and stakeholder needs. Health workers at the health facilities noted that the CHAs community-level service lessened the strain/demand on them, and community members expressed confidence that they could obtain care closer to home—as long as their CHAs’ supply of basic drugs5 held out. KIs and FGDs with community leaders aligned with PACS’s midterm evaluation’s finding that communities appreciated the Natural Leaders’ advocacy and assistance for ending open defecation practices and digging and protecting clean water sources (LSA, 2017). One leader in Lofa attributed a perceived decline in rates of child illness to improved sanitation and IPC by community members. Several county level MOH staff and Officers in Charge (OICs) appreciated that PACS’s work with CHCs, and their support for community participation in the HFDCs, which opened communication and collaboration between the clinics and the communities they served. Thus, PACS strengthened or created new platforms for dialog across political and health system levels Some OICs and County staff had been posted previously in sites where they had never been introduced to local leadership at all. On HCP appreciation of CHAs: “I have experienced over the period that working with them at the community [level] has been able to ease our tension while working in the health facility. People don’t wake us up at night for treatment because the CHAs live in the community with the people and they already know what to do to help their people.” – KII, 2019 Mission-critical risk from drug stock-outs Virtually all respondents at the county and community levels complained that drug stock-outs threatened the effectiveness of both HCFs and CHAs/CHVs. At the start of the CHSP, MOH staff were reportedly reluctant to allocate a portion of their drug allotments to CHAs and CHVs because their allotments were then insufficient to meet the needs at the facilities. The county level Community Health Division Directors, with PACS support, advocated at central and county levels with the A–78 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME 1 Supply Chain Management Division and OICs, and convinced them to dedicate 20 percent of facility allotments to their CHAs and CHVs. Nevertheless, PACS’s CBIS data in Year 3 showed no progress in remedying the under-supply of essential medications at the facility level (see Figure A14-1), with knock￾on consequences for the CHAs and CHVs. “Stock-outs are one of the biggest challenges facing not only PACS but the viability of the NCHA program as a whole” (IRC/PACS 2017c. p. 8). Adaptive management A number of additional inputs and events led PACS to change focus and activities over the course of the project. The most dramatic was the expiration of the Ebola Emergency Fund 6. The MOH intended to bring the CHSS onto its payroll in all six counties, but as of early 2019, this had not yet happened. 7. From the outset, PACS was intended to build “sustainable country ownership for community-based health, social welfare, and WASH services” (IRC 2016b, p. 7). (EEF) funding in December 2017, three years before the end of the project. In Year 3, PACS was providing the MOH with the funding to cover the basic payments to 1,101 CHAs (monthly incentives, reimbursement for transportation) in the six counties; as of December 2017, support for these implementers in the three “modification” counties (Grand Bassa, Margibi, and rural Montserrado) was terminated and IRC refocused its efforts on the original three counties.6 According to a KI, another disruption occurred when USAID’s support and improvement of the MOH’s supply chain was interrupted by a long delay in the transition between USAID’s supply chain support IPs (from JSI to Chemonics). In Year 3, PACS decided to change its allocation of resources to districts, from equal allotments to allotments proportional to population (IRC, 2017c). All these features contributed to PACS’s mission to strengthen the links between communities and health care facilities through community involvement. The midterm evaluation of PACS (LSA, 2017) conducted a quantitative difference-in-differences analysis comparing PACS and non-PACS districts. The PACS districts were significantly more improved than the comparison districts on 9 of 15 indicators: increases for measles coverage, Penta3, Oral Polio Vaccination third dose, malaria cases treated with artemisinin-based combination therapy, Depo-Provera users, antenatal care visits, coverage of intermittent preventive treatment for malaria for pregnant women second dose, and coverage of tetanus toxoid second dose and a decrease in Acute Respiratory Infection (ARI) treated with antibiotics. This means that performance for 60 percent of the indicators assessed is better than in comparable districts without the PACS interventions. FACTORS CONTRIBUTING TO ADVANCING SELF-RELIANCE As noted earlier, PACS was designed with the aim of contributing to Liberia’s capacity to implement and manage its community health services (IRC/PACS (2016b),7 which is an important component of a self-reliant and effective national health system. PACS functioned as part of a mosaic of U.S. Government (USG) support to other levels and functions of the health system: PACS’s partner organization Global Communities (GC) supported the MPW; Management Sciences for Health (MSH) Collaborative Support for Health (CSH) focused on the central and county-level MOH; JHU CCP HC3 focused on the central and county level health promotion teams, and JSI/ DELIVER (and then Chemonics) supported the MOH Supply Chain Management Division. This made it possible for PACS to not only contribute to technical working groups in all of these areas, but also focus effort on the facility and community levels, where CHVs, CHAs, and their supervisors are managed. 0 20 40 60 80 100 Q1 Q2 Q3 Q4 81% 81% 80% 19% 19% 20% (Data not available) Experienced stock-outs Did not experience stock-outs Figure A14–1. Persistent stock-outs (Source: PACS Y3 Q4 and Annual Summary and Donor Submission, p. 8) CASE STUDY 14: PACS IN LIBERIA n A–79 Synergies From its niche supporting community health structures, PACS staff worked up and down the chain of command and with the several interlinked systems that were required to make community health services work. At the central level, PACS staff built the capacity of the Community Health Services Division; advocated with the Supply Chain Management Division to supply the Community Health (CH) program with drugs; supported the MPW in its responsibilities to improve access to clean water (wells) and water improvement (including sale of its water purification product, Waterguard); and championed the CBIS to add a new level of granularity to the DHIS 2 to support decision-making based on community-level health conditions and needs. At the community level, PACS added medical waste management and water safety to the natural leaders’ training package, and promoted teamwork between them, the CHVs or CHAs, the CHCs and the HFDCs. The midterm evaluation of PACS found suggestive evidence that this integration of strategies and functions had spillover benefits for community health. “In PACS districts, the achievement of ODF status is significantly correlated with 10 of 15 key health facility service indicators. For example, an increase in the cumulative number of PACS ODF communities in a district is associated with increased vaccination coverage, treatment of childhood illnesses, antenatal care, short-acting family planning use, and post-partum health facility visits.” – (LSA, 2017 page 15) Contributions to capacity “For us as health workers, the training that we underwent is still live in us till now. I can tell you for real, in case there is an Ebola outbreak, we as health workers already know what to do.” “Because we were not trained before the Ebola outbreak, many health workers died. Those training about preventions that we learned, and those awarenesses that were created, those are things we can never forget in case of any health crisis.” “The GCHVs still operate within the community; we call on them at times. Things have improved. If there is an outbreak people already the steps to take. I can say that we have made significant progress.” – FGD participants, 2019 Despite IRC’s aim to position PACS behind rather than in front of the MOH, the IP was widely credited in KIIs and FGDs with building the capacity of community members to be effective members of the health system, through trainings and provision of tools and job aids. PACS-supported training of 1,490 women and men as CHAs prepared half of the national CHA workforce, instilling skill in iCCM and in health promotion (see quotations next page). The MTE highlighted a serious gender imbalance in the recruitment of CHVs and CHAs (LSA, 2017) and PACS attempted to redress it, although the lower literacy of women in PACS’s communities has made that difficult. In addition to training CH personnel, community chiefs received training in leadership and ETL. CHCs in all 1,442 communities received templates for creating meeting agendas and meeting minutes, with action items to be followed up at the next meeting. PACS trained data clerks and posted them in each district to improve the flow of health information from the CHAs and CHVs into the CBIS. Between Q3 and Q4 in Year 3, sub-grantee timely and adequate reporting increased from 35 percent to 50 percent (IRC, 2017c). These skills and tools prepared leaders and community members to engage effectively with their health care providers, and to do their parts in supporting the health promotion, iCCM and referral work of the CHVs, CHAs, and natural leaders. The rollout of the CBIS further strengthened the capacity of the MOH to monitor health needs at the community level, with implications for improving procurement and distribution of essential drugs and commodities. “Local community health structures including TTMs and CHCs are demanding incentives in order to be on par with gCHVs/ CHAs” – (IRC, 2017b, p. 9) Part of the skills required of PACS staff and CSO partners at all levels was to deal with “procurement culture clash” (IRC, 2017c, p. 15). PACS’s mandate to build the capacity of its CSO partners to receive and directly manage funds according to USG regulations required building both the skills and the will to follow strict procurement and reporting guidelines, which sometimes clashed with local expectations. For example, with USAID’s approval, PACS acceded to MOH’s request that it A–80 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME 1 control the vehicles that were assigned to the project, but only on the condition that MOH would maintain them and cover fuel costs, which it did not do. The result was vehicles under MOH control but out of service. PACS reported that an all-day meeting of senior MOH officials came to a halt when the officials were informed that the price of their lunch had to be deducted from the per diem payment for the day (KII with Monrovia PACS staff). Disseminating administrative and finance guidelines had to be followed up with mentoring and negotiation in order to genuinely build local capacity and the will to manage USG funds in accordance with regulations. “Another scenario: When we started, we were working with CHVs with no incentives or compensation. Now we are giving compensation to those working beyond 5 km from a health care facility—that’s National policy, not PACS. And CHVs within 5 km have no incentive so don’t expect as much performance as when attached to monetary value. – KII, 2019 Contributions to commitment The combination of skills building, mentoring and supervising, and providing essential drugs and commodities, all in the context of a role that was recognized and valued by community members, has built a strong sense of commitment among the CH providers. In Year 4, 99 percent of the CHAs and CHVs who were trained with PACS support were still active in CH service delivery in their assigned communities. However, in KIIs and FGDs, views on commitment were closely tied with issues of remuneration. Several respondents, including KIs in the MOH and health workers who participated in FGDs, commented on tensions between the CHA and CHV cadres because (in PACS’s original three counties) the former receive a monthly incentive payment (USD 50) in addition to transportation costs, whereas the CHVs, some of whom began serving their communities as CHVs long before, only receive the latter. The issue extends to other key actors in the health system, as noted earlier. Self-reliance requires capacity, commitment and resources USAID’s framework recognizes that the resources required for self-reliance include people with the needed training, mandates and supportive systems, and the funding to pay those people and to provide them with the materials they need to do their jobs. The issue of resources was a constant concern that was expressed by nearly all sources used in this case study. The concern about drug stock-outs was discussed earlier. Liberia’s CHSP faces broader funding challenges. Numerous respondents commented that the health system has been and continues to be too dependent on USAID and NGOs (see below). Only a few respondents expressed ideas of how to change this. Two IPs and several government officials (MOH and Ministry of Internal Affairs) suggested that Liberia should shift from its commitment to free health care to one where people pay for service, noting that people value what they pay for. A District Superintendent in Lofa County said that his County Health Team (CHT) wants to introduce a modest fee for health services to contribute to a revolving drug fund (KII with GOL). However, most depicted the problem with desperation: a wall into which the country was inevitably going to crash. This is particularly concerning in light of the cost-effectiveness analysis conducted by LSA as part of the midterm evaluation. LSA compared three scenarios: ceasing payment of CHAs; continuing incentives but without provisioning drugs and supplies; and continuing incentives and provision of drugs and supplies. They found that “paying stipends to CHAs without provision of iCCM drugs (scenario 2) does not yield an economic return. By contrast, CHA services with drugs, commodities and stipends produce a 2:1 return on investment over a five-year horizon.” (LSA, 2017, p. 22). In other words, when strengthening the community health system, more investment will bring more results, and less investment will bring a great deal less—and possibly jeopardize the substantial progress made. “I’m concerned; there’s a lot of concern [about] how the government will continue the community health program once PACS, UNICEF, [and] Last Mile Health—who are now paying the bills—[leave]. How will the government keep it going? PACS is in its last year, so we are having discussions at the national level and with the county health teams: How are you going to continue paying CHAs, CHSS, giving CHSS fuel to do supervision, and repair and service their motor bikes? And there is no answer. I think we’re looking at what we’ve built coming to an end. When we did a transition meeting with the MOH and they saw the tremendous costs to maintain the program, they said, ‘Wow, where is that money?’” – KII, 2019 CASE STUDY 14: PACS IN LIBERIA n A–81 CONCLUSION PACS has worked successfully with MOH and MPW counterparts, with other IPs, and with communities themselves to collaboratively heighten community interest and engagement in health, to expand access to the essential package of health services mandated in Liberia’s national health policy, and to improve the central and distal systems through which the CHSP is managed. This progress has been boosted by substantial USG investment, through various mechanisms. USAID is working with the GOL to break out of the mind-set that looks largely to outside donors to finance these essential government services (KIIs with IPs, Monrovia), as increasing domestic sources of funding for the CHSP (and other key health policies) is central to the country’s journey toward self-reliance. REFERENCES Chambers, R. (1994). The origins and practice of participatory rural appraisal. World Development 22(7): 953–969. CiteSeerX 10.1.1.454.4672. doi:10.1016/0305-750X(94)90141-4 Ellis, S. (2006). The mask of anarchy: The destruction of Liberia and the religious dimension of an African civil war (2nd ed.). New York: NYU Press. Government of Liberia (GOL). (2015a). Agenda for transformation: Steps toward Liberia rising 2030. Monrovia: Government of Liberia. GOL Ministry of Gender, Children, and Social Welfare (MGCSW). (2017). Kinship care, foster care, and supported independent living guidelines. Cited in Liberia Statistical Analysis (2017). Government of Liberia (GOL) Ministry of Health (MOH). (2011). Essential package of health services: The community health system. GOL MOH. (2015a). Investment plan for building a resilient health system in Liberia 2015 to 2021. GOL MOH. (2015b). Revised national community health services policy 2016–2021. GOL MOH. (2015c). Revised national community health services strategic plan 2016–2021. International Rescue Committee (IRC). (2015). Partnership for Advancing Community-based Services (PACS) annual report. Time Period: February 28, 2015–September 30, 2015 [File: PACS Annual Report FY 15 Y1_Final version.docx]. IRC. (2016a). Partnership for Advancing Community-based Services (PACS) Year 2 annual results summary. Annual Period Covered: October 1, 2015–September 30, 2016. [File: PACS Year 2 FY16 Annual Results Summary Report.docx]. IRC. (2016b). Partnership for Advancing Community-based Services (PACS). Institutional self-assessment report for the Community Health Services Division, Ministry of Health. May 2016. IRC. (2017a). Activity-level monitoring and evaluation plan. Partnership for Advancing Community-Based Services (PACS). Revised version: May 5, 2017. IRC. (2017b). Partnership for Advancing Community-Based Services (PACS), FY 2017 Year 3 First Quarterly Report. Quarterly Period Covered: Oct 1, 2016–December 31, 2016. IRC. (2017c). Partnership for Advancing Community-Based Services (PACS), FY 2017 Year 3 Q4 and annual summary report. Quarterly Period Covered: July 1–September 30, 2017, Annual Period Covered: October 1, 2016–September 30, 2017. IRC. (2018). Partnership for Advancing Community-Based Services (PACS). FY 2018, Year 4 Q2 Report, January 1–March 31, 2018. Liberia Statistical Analysis (LSA). (2017). Midterm performance evaluation report: USAID/Liberia, Partnership for Advancing Community-Based Services (PACS). September 2017. Moore, Z., Bongiovanni, A., Sadaphal, S., Hernandez, O., Espeut, D., Levin, C., & Toole, M. (2019). Performance Evaluation of USAID Ebola Pillar II Activities: Final Report. Evaluation Report to USAID, February 2019. Vienna, VA: International Business & Technical Consultants, Inc. TvT Associates. (1994). The participation toolkit: Health technical services project. Accessed 07/05/2019 at https://pdf. usaid.gov/pdf_docs/PNACC924.pdf USAID. (2011). Human and institutional capacity development handbook: A USAID model for sustainable performance improvement. Washington, DC: USAID Bureau for Economic Growth and Trade. A–82 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME 1 GUINEA CASE STUDY 15. HEALTH, FINANCE, AND GOVERNANCE (HFG) BOX A15–1. HFG AT A GLANCE Lead Implementing Partner (IP): Abt Associates Nature of Implementation: Supplemental funding to an ongoing global project to expand to Guinea, multi-sector (health and governance), stand-alone activity Implementation Dates: June 2016–June 2018 USAID Pillar II Budget/Obligation: USD 6 million1 Geographical Priority Areas: Guinea national (with the majority of activities at central level) Key Activity Components: Auditing the Ministry of Health (MOH) to inform the restructuring of the MOH; strengthening the managerial capacity of the MOH; developing policies, strategic plans, and procedural manuals; improving management of Human Resources for Health (HRH); and improving Health Governance Key Pillar Objectives and Outcomes to which Activity Contributed: HFG mainly contributed to the second Pillar II Objective 2 (“Recover and strengthen existing institutions and infrastructures”). Restoration and Expanding the Capability of the Public Health Supply Chain by Dr. Karim Sahyoun Abt Associates’ Health, Finance, and Governance Project (HFG) was a six-year global project (2012–2018) that worked in Guinea for a duration of 24 months (June 2016–June 2018) to support both Guinea’s 2015–2020 Country Development Cooperation Strategy and the post-EVD strategy set up by the USAID Mission in Guinea under Pillar II. The two year funding for the project’s Guinea activities targeted two sectors: the health sector, which was funded by 1. Source: USAID’s Office of Budget and Resource Management O&E spreadsheet, September 2017. the USAID/Washington Global Health Ebola Team (GHET) and the democracy and governance sector, which was funded by USAID/Guinea with Democracy and Governance Ebola Recovery funding. Abt Associates, the project’s lead implementer, worked with a number of consortium members: Avenir Health, Broad Branch Associates, Development Alternatives Inc., the Johns Hopkins Bloomberg School of Public Health, Results for Development Institute, RTI International, and Training Resources Group, Inc. OBJECTIVE OF ACTIVITY The overall objective was to strengthen Guinea’s health system through enhancing functionality of institutions and programs and improving their capacity to deliver quality health services. The project aimed at addressing fundamental performance problems of the health system that had been exposed by the Ebola crisis (HFG, 2018). The project therefore focused on strengthening the institutional capacity, accountability, and transparency of the Ministry of Health (MOH) to improve the utilization and quality of health services. A further objective was the strengthening of the oversight of the health sector by the Health Commission of the National Assembly, allowing it to develop and advocate for a realistic health budget (IBTCI, 2018, p. 41). The sustainability of the project’s achievements is critical for a long-term increase in Guinea’s self-reliance in better managing the health sector and responding to health emergencies such as the Ebola crisis. Although HFG was a stand-alone activity, it was closely linked to other health sector activities of USAID’s post-Ebola CASE STUDY 15: HFG IN GUINEA n A–83 commitment to Guinea, with activities reinforcing each other (e.g., Health Communication Capacity Collaborative (HC3) aiming at increased local funding, Systems for Improved Access to Pharmaceutical Services (SIAPS) ensuring that data is systematically collected and available for planning, and Jhpiego Corporation with several Pillar II-funded programs developing planning guides and supporting the required capacity-building). Several key informant interviews (KIIs) with representatives from 2. Data Sources all from http://www.mbudget.gov.gn/index.php/lois-de-finances/ Specifically: 2015: http://mbudget.gov.gn//wp-content/uploads/2017/10/loi-2015.pdf 2016: http://mbudget.gov.gn//wp-content/uploads/2017/10/LFI201611-4.pdf 2017: http://mbudget.gov.gn//wp-content/uploads/2017/10/LOIF2017-1.pdf 2018: http://mbudget.gov.gn//wp-content/uploads/2017/10/LFI2018.pdf 2019: http://www.mbudget.gov.gn/wp-content/uploads/2018/11/Texte-de-loi-du-projet-de-loi-de-finances-2019.pdf IPs, MOH, and the Government of Guinea (GOG) underscored this complementarity. HFG demonstrated success in increasing health outlays as a percentage of government funding as reported in PE1. The case study focuses on this accomplishment, which was identified as a potential “transformative intervention” (IBTCI, 2019, p. 59), while elaborating other areas that have contributed to it. ACCOMPLISHMENTS HFG focused primarily on challenges within the health sector that had either been identified prior to the Ebola epidemic or exposed through it, as representatives of the MOH and the National Assembly’s Health Commission explained. HFG’s key results over two years include the following: ■ A stronger central MOH, better able to carry out core functions; ■ Improved capacity in governance of the health sector; and ■ Improved human resources management and training capacity for health (HFG, 2018). Positive results in governance, specifically strengthening the oversight of the health sector and resulting in increased health funding, has been identified as a major success to the project by all KIs consulted on HFG, including representatives from implementing partners, the GOG, and the MOH. The Health Commission, an oversight body formed by members of parliament, has been in place since 2013; it was, however, facing a number of challenges in fulfilling its role. Its members were from very diverse contexts, differently equipped for their roles with very different capacities relevant to the health sector. While this commission should have actively participated in discussions around the health budget, according to KIIs with the IP and Health Commission there was, due to a lack of understanding of budgeting and the exact role of the commission, typically no interaction at all. In 2016, prior to the activity, the national budget allocated 4.4 percent to health. In 2017, the proposal was 5.8 percent and the Heath Commission managed to negotiate a further increase of 2.4 percent (HFG, 2018), resulting in a total budget allocation for health of 8.2 percent (Table A15-1). While the allocation fell to 6.0 percent in financial year (FY) 2018, it nominally remained stable at 1.3 trillion GFN, the drop in the percentage was due to a significant increase of the national budget (Moore et al., 2019; HFG, 2018). In FY 2019, the national budget grew further, the share for health also increased percentage wise to 7.7 percent. This is recognized by KIs as one of the most notable successes of HFG’s involvement; it excelled in assisting the commission to advocate for increases in the health budget. Table A15–1. Guinea’s health budget as a share of its total national budget (2015–2019)2 Budget 2015 2016 2017 2018 2019 Annual Budget (in trillion GNF) 15. 1 14.2 16.2 21.1 21.8 Health Budget (share of annual budget in %) 4.2 4.4 8.2 6.0 7.7 Health Budget (in trillion GNF) 0.6 0.6 1.3 1.3 1.7 “HFG started the training of deputies of Health Commission at the National Assembly. . . . This improved our capacities because there is no school that teaches people to be deputies. So we benefited of theoretical training on economy to know how to control government’s actions, how to study a budget, . . . how to make a proposal of law, and finally how to build relations with international institutions.” – KII, 2019 A–84 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME 1 The introduction of a timely, participatory, and decentralized operational planning process was critical for the health commission to engage in negotiations and advocate for an increased budget. The Health Commission achieved a better understanding of the actual steps required to influence budgeting within the system and were provided with quality information required to negotiate and the data through which to convince other ministries of the relevance of their request. KIs from Abt Associates and the MOH reported that collaboration with Jhpiego assisted in this effort. The collaboration with Jhpiego included strengthening health structures in operational planning at prefectural and regional levels based on revised or newly developed planning tools and guides. The successes in increasing the health budget and the capacity building at critical levels in the health system in turn contributed to the recovery and strengthening of existing health services and infrastructure, according to KIs from the MOH, GOG, and Abt Associates. However, according to representatives from local health centers, the MOH, and the GOG, these increases have not been sufficient to meet the growing needs in the context of population growth. While there is evidence of progress, the level of health funding is still far from the Health Commission’s commitment to achieve 15 percent by 2020 (Moore et al., 2019). Only 7.2 percent of the 8.2 percent allocated for health was provided in FY 2017. Figures for FY 2018 were not available. This raises the question whether sufficient capacities are in place to assure proper spending. All actors interviewed, as well the majority of HFG’s reports, stated that the time for implementation was too short to achieve the program’s ambitious objectives. One KI of Abt Associates said, “The project has been very short, too short for the objectives it aimed to achieve.” Abt Associates representatives and a representative of the Health Commission added that, at least in some critical areas, continued support either through other donors or follow-up projects has been received. “However, it [USAID] left us in the good hands of FHI360 that has taken the torch and is up to the task. Today, although we are not glad for the departure of HFG, we are not weaned either. We have no regret because FHI360 is up to the task and going even further than HFG. Because through FHI360, we were able to make a second round in the country to assess the improvement in the health facilities upon the recommendations that were made six months back.” – KII, 2019 FACTORS CONTRIBUTING TO EFFECTIVENESS The successes of the activity were based on a number of contributing and often interrelated factors. The project used a learning approach, training health commission members in listening, analyzing, and working on weaknesses and priorities that had been identified. An analysis of the health sector was a critical starting point for the activity. Nothing was predefined and HFG focused on priority areas, as KIs of Abt Associates and the Health Commission at the National Assembly explained. Starting the activity with a joint analysis process contributed to capacity building and awareness of critical stakeholders. “And it was through HFG that we were able to make the first round of the country in order to assess health facilities. HFG supported us materially, financially and with logistics. We were able to visit almost all the prefectural hospitals . . . and the hospitals in the capital. . . . When we went for a survey in 2017, the notice was very catastrophic. Everybody had given up and local authorities including prefectural health directors, hospital directors and health centers/posts had all given up. [We encountered] dirty health facilities [and] premises, rooms [with] defective beds, poorly maintained materials, unusable toilets, [and] absent medical staff. Spider webs and wasp nests etc. Anyway, it was desolation. But we did many recommendations.” – KII, 2019 Combining participation of key stakeholders with targeted capacity building A high level of participation of key stakeholders in the assessment has created momentum for action and ownership. It has equipped decision-makers with the needed understanding of the issues and for which they are advocating, and the necessary procedural steps they need to follow. KIs of Abt Associates and the Health Commission at the National Assembly have affirmed that HFG has enabled them to participate in setting relevant priorities and taking joint responsibility for their implementation. CASE STUDY 15: HFG IN GUINEA n A–85 A specific example of the increased participation of the Health Commission has been the equal participation of all major political parties, enabling the commission to move away from partisanship and make budgeting a joint effort, according to a commission KI. Needs-based HFG avoided coming with predefined priorities and activities; rather it facilitated the environment in which they could be developed; therefore, stakeholders reported that HFG supported the Guinean health sector and the different participating entities to expose and seek solutions to critical needs. Systems approach to linking activities USAID Ebola Pillar II activities addressed many interwoven layers of the health system. This has resulted in different activities enabling and supporting the success of other activities. One example is the complementary roles of Jhpiego and HFG as described above. According to one MOH KI, Jhpiego’s ability to support the generation of robust data was critical to HFG’s work with the Health Commission, giving them additional tools to demonstrate the challenges and needs of the sector—and thus, to more effectively advocate for increased funding. Quality of collaboration/relationships and attitudes of local and external partners MOH and IP representatives both reported that they valued the quality of their relationship; both sides described the external staff as highly dedicated and focused on working with their national counterparts. KIs of Abt Associates described that embedding staff in the MOH has been pivotal to achieving the results. Again, the shortage of time for implementing the program was seen by Abt Associates’ KIs as a major hindering factor for fully achieving changes, as the actual period left for implementation was only 18 months. Changes to governmental processes frequently take much longer. Contradictory opinions on the effectiveness “How to boost health funding and the government’s contribution is the greatest challenge. A child needs to be on solid feet; if it is not supposed to fall, accompaniment is necessary.” – KII, 2019 Some MOH KIs challenged the program’s approach in providing them with planning and analysis skills but not with funding. They reported that HFG helped them to do a good analysis, understand the needs, develop plans and have strategies in place in addition to advocacy skills for raising support. However, they were now facing the challenge that these plans were far beyond the resources that were allocated. They felt that the project should have taken them further toward providing the required funding. Other MOH and GOG KIs had quite a different opinion; they highlighted that increased planning competencies had enabled them to access an increasing number of donors to support implementing their plans, for example the construction of the new Central Pharmacy. FACTORS CONTRIBUTING TO ADVANCING SELF-RELIANCE In general, KIs from all different entities agreed that HFG had made critical contributions to increasing the country’s self-reliance, specifically with respect to being better prepared to face a challenge such as the Ebola outbreak. An increased commitment to government funding for the health sector can be observed, health services and their governance have been strengthened by HFG and other USAID Pillar II-funded programs reviewed. Interestingly, on this specific question KIs frequently looked beyond the specific activity and named achievements from different Pillar II activities that had contributed to increasing the country’s ability to manage a similar health crisis as Ebola. One MOH KI described the increased capacities in governance and coordination as critical for managing any future emergency, in combination with having the right standard documents and procedures in place. Another example many KIs (MOH, GOG, local health center representatives) used to prove their statement was a Lassa fever case identified at the end of January 2019, when communication and action were all implemented in an exemplary way. This case illustrated a vivid example of structures, processes, and capacities that were in place. The increased capacity to act without external support was specifically identified in the area of prevention. Equally, interviewees all acknowledged that there was more to do to be able to manage a similar crisis without external support, especially when managing a significant health crisis after its outbreak. A–86 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME 1 Key factors for commitment The Ebola crisis and its devastating effects triggered an urgency for change for many actors, including MOH and its divisions, GOG, and civil society, creating a commitment to better prepare and protect the country for the future. The crisis provided urgency to the need to act. The participatory learning approach involved including stakeholders in the assessment stage, allowing them to jointly identify priorities and issues; this approach significantly contributed to an increase in engagement, and thus, commitment. Media coverage of critical recovery needs following the Ebola crisis has helped to widely communicate that health is a national problem and thus a priority (KII with Health Commission, 2019). Training in relevant competencies (e.g., analysis, management, and governance) has further equipped and allowed actors to feel as if they can now make a meaningful contribution (KII with Health Commission, 2019). Commitment is reflected in the development and use/ application of tools, guidance, and policies by actors at all levels in the health system. All KIs and FGDs were aware of these and immediately had these at hand and stated that these were regularly used. Users saw a benefit in working with them, improving the process and results of their work. One interesting example concerning the guidance and tools for the operational budgeting was reported by an MOH KI (see quotation below). The Health Commission, according to KIIs with IP representatives, has developed new and improved relationships with UNFPA and UNICEF. Other donors have begun to discuss supporting the national health sector with the Commission. Some relationships with and involvement of civil society are also emerging. According to an Abt Associates KI, however, this is still rare; it has been facilitated by the Open Society Initiative for West Africa. “Budgeting for the health sector is an extensive activity, since it is a decentralized multi￾layered process, taking place in all health districts. In the past, this planning and the resulting overall budget had only been prepared so late that it could not inform the first draft and negotiations for the national budget. The planning for the 2017 budget, for example, was prepared in June 2016. In 2018, the MOH took the initiative to plan earlier, in March for the 2019 [budget]. Since this was still too late to feed into the first draft of the national budget, the planning in 2019 for the 2020 operational plan was already facilitated in January 2019. As a result, the budget was available at the most strategic time for the Health Commission to proactively influence the first draft of the national budget. Further evidence for commitment and ownership is that the BSD has been working on further adjustments of the budgeting guide, based on learning from past planning processes.” – KII, 2019 Key factors for capacities “For supporting these planning processes long-term, we do not have sufficient staff yet, it even has declined in numbers and in competence. We need to train some of our staff to have the capacity to pilot and lead process. Currently, two or three staff can do a good job on coordination; if they leave, it will be a challenge for sustainability. One among them will retire in December. We require training of the leadership.” – KII, 2019 Self-driven change in the health sector requires capacity development at multiple. This is illustrated by the challenge of strengthening a broad spectrum of relevant competencies of Health Commission members to be able to actively fulfill their role. Another example is building the capacity for operational planning, with the training of leaders and managers in the MOH through HFG. One concern about sustainability is whether these newly developed capacities are tied to specific individuals. This concern, raised by several KIs from the GOG and MOH, reflects the challenge of staff turnover (e.g., retirement and staff moving to the private sector or from the different regions to Conakry). As one MOH KI illustrated, HFG had trained a pool of 10 MOH trainers to ensure the continuation of capacity building. One of these was about to retire and the trained staff in other departments had already left to work as independent consultants in other countries. Even without staff turnover, acquired capacities may be lost, according to another MOH KI. Capacity CASE STUDY 15: HFG IN GUINEA n A–87 building is one step, but whether this learning has been applied and become part of new standard practices or culture is another. HFG had intended to ensure the sustainability of capacities by collaborating with training institutions, adding to their existing courses critical competencies and developing a system of continuous learning for staff. From the evidence reviewed for this case, it is not clear whether continued training will be available to allow a sustainable, increased level of competencies to ensure self-reliance. The case for the Health Commission is different. Each legislative election may lead to a significant change in the composition of the committee. HFG, as explained by a former program staff, has put a strategy in place to ensure continuity by including assistants in the training—who are not elected and remain in their positions to support the newly elected members of the health commission. While currently continued support is provided to the Health Commission through a follow-up project, an Abt Associates KI suggested that it would require an election to fully understand whether the increased level of capability would be sustainable. Only then would it become visible whether impact would be carried over to a new health commission. “The Health Commission is formed by parliamentarians, who are regularly elected, therefore members may be replaced. The next election is expected in 2019, [so] there is an imminent potential [for] the replacement of members. However, sustainability is ensured by a different group of people who were strategically involved: parliamentary assistants. They document all [and] they inform the new commission how things were done. We included assistants in the training, helped them to access and have relevant information and knowledge. However, there is a limitation: many assistants have very limited technical capacities. They were not chosen for competency, but due to relationships. For the parliamentarians of the Commission, some members will remain, some will be new. Capacity-building needs to be continuous and needs to evolve.” – KII, 2019 REFERENCES Guinée Matin (2018). Guinée : voici le budget 2019 de l’Etat en chiffres et en rubriques. Retrieved from: https://guineematin. com/2018/12/27/guinee-voici-le-budget-2019-de-letat-en￾chiffres-et-en-rubriques/ Health, Finance and Governance (HFG) (2018). HGF Guinea. Final report. Moore, Z., Bongiovanni, A., Sadaphal, S., Hernandez, O., Espeut, D., Levin, C., & Toole, M. (2019). Performance Evaluation of USAID Ebola Pillar II Activities: Final Report. Evaluation Report to USAID, February 2019. Vienna, VA: International Business & Technical Consultants, Inc. USAID. (April 2019). USAID policy framework: Ending the need for foreign assistance, summary. At the request of the United States Agency for International Development (USAID), this publication was prepared independently by International Business and Technical Consultants, Inc. (IBTCI). Photo by Sam Phelps for UNICEF ANNEX VOLUME II ANNEXES B–J Second Performance Evaluation of USAID Ebola Pillar II Activities September 2019 Second Performance Evaluation of USAID Ebola Pillar II Activities: ANNEX DOCUMENT II: Annexes B–J USAID CONTRACT # AID-OAA-I-15-00022 Task Order # AID-OAA-TO-16-0040 Authors: Sara Woldehanna, Barbara de Zalduondo, Donna Espeut, Karim Sahyoun, Jackie Yiptong Avila, and Elisabeth Nolan Submitted August 1, 2019 Final September 30, 2019 Prepared for: United States Agency for International Development Ronald Reagan Building and International Trade Center 1300 Pennsylvania Ave NW Washington, DC 20004 Prepared by: International Business & Technical Consultants, Inc. (IBTCI) 8618 Westwood Center Drive Suite 400 Vienna, VA 22182 USA Tel: +1 (703) 749-0100 Citation: Woldehanna S et al (2019). “Second Performance Evaluation of USAID Ebola Pillar II Activities: Final Report.” Evaluation Report to USAID Annex Volume II, September 2019, International Business & Technical Consultants, Inc.: Vienna, VA. Project Description: The Ebola Pillar II, Monitoring, Evaluation and Learning (MEL) activity is a three-year USAID-funded contract addressing USAID-coordinated efforts in mitigating the second-order impacts of the Ebola virus outbreak in Guinea, Liberia, and Sierra Leone. The activity focuses on four main components: evaluation, routine monitoring, data quality assurance, and improved knowledge management and learning. The activity is led and managed by International Business and Technical Consultants Inc. (IBTCI), with partners StatView International in Guinea, Global Research Insights, LTD (GRI) in Sierra Leone and Liberia, and Opinion Research Business (ORB) International in all three countries. Cover Photo: Community Health Worker, Sierra Leone. Photo by Sam Phelps for UNICEF SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES ANNEX DOCUMENT II ANNEXES B–J September 2019 This publication is made possible by the generous support of the American people through the United States Agency for International Development (USAID) under the terms of contract AID-OAA-I-15-00022/AID-OAA-TO-16-00040 managed by International Business and Technical Consultants Inc. (IBTCI) and does not necessarily reflect the views of USAID or the United States Government. ACKNOWLEDGMENTS The authors would like to acknowledge the diligent involvement of the full technical team: Zhuzhi Moore, Annette Bongiovanni, Richard Columbia, Andrew Reuter, Roger Emmanuel Millimono, and Richard Ngafuan. The authors would like to thank the activity coordination staff for their continued support and commitment, including Katherine Labombarde, Maria Pastor, Yasmin Ibrahim, and Victoria Sesay. Special thanks to the editors Irene Labombarde and Erin Dowling, and graphic designer Erin Dowling. A very special thank you goes to Latrisha Chappin and Nadine Ritcheson from USAID’s Bureau for Africa (USAID/AFR) and representatives from the bureaus for Global Health, Global Development Lab, Food Security, Democracy, Conflict, and Humanitarian Assistance, and Economic Growth, Education, and Environment, the Guinea, Liberia, and Sierra Leone missions, and various implementing partners provided valuable support and constructive comments throughout the preparation of this report. Finally, our gratitude goes to the many individuals who participated in the key informant interviews and focus group discussions. This evaluation would not have been possible without your contribution. n 05 ANNEXES VOLUME II ANNEX B. Scope of Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B–1 ANNEX C. Evaluation Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C–1 ANNEX D. Data Collection Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D–1 ANNEX E. List of Persons Interviewed, Focus Group Discussions, and Site Visits . . . . . . . . . . . . . . . . . . . . . . . . . . . . E–1 ANNEX F. Disclosures of Conflicts of Interest and NDAs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . F–1 ANNEX G. Evaluation Team Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . G–1 ANNEX H. Statements of Difference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . H–1 ANNEX I. Pillar II Theory of Change . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I–1 ANNEX J.   Additional Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . J–1 06 n PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES ANNEX B. SCOPE OF WORK n B–1 ANNEX B. SCOPE OF WORK Second Performance Evaluation of USAID Ebola Pillar II Activities / Final Report Annex II Document DESCRIPTION/SPECIFICATIONS/STATEMENT OF WORK 1. TITLE Pillar II Ebola Recovery Efforts to Mitigate Second Order Impacts of the Epidemic in Guinea, Sierra Leone, & Liberia 2. BACKGROUND Since the acceleration of the Ebola Virus Disease (EVD) outbreak began in Guinea, Liberia and Sierra Leone in mid-2014, the United States has mounted a whole-of-government response based on four pillars: I) Controlling the epidemic at its source in West Africa; II) Mitigating second-order impacts, including blunting the economic, social, and political consequences in the region; III) Engaging and coordinating with a broader global audience; and IV) Fortifying health security infrastructure in the region and beyond. This background section summarizes the US Government (USG) strategy for Pillar II: Mitigating Second￾Order Impacts. The 2014 Ebola virus disease (EVD) outbreak was first of that scale and the first to affect large urban populations. While historical precedents like HIV/AIDS and SARS are informative about channels of second-order impact, there are no precedents in recent experience for a large-scale epidemic that combines EVD’s deadliness, immediate and long-term impact, and rapid spread. The West African Ebola outbreak accelerated in mid-2014 and swiftly disrupted development progress in the most affected countries—Guinea, Liberia, and Sierra Leone. The epidemic has brought the human and institutional weaknesses in each of these countries to the fore, while creating extraordinary demands on political leaders and global responders. Public institutions in these countries were under acute stress, with risks of institutional failure and increased vulnerability to similar crises in the future. Due to fears associated with EVD, the health, agriculture, democracy and governance, trade and markets, and education sectors, as well as the overall economy, have been heavily impacted by government restrictions, reduced international investments and operational transitions and activity suspension and withdrawal, as well as border closures. In addition, the scientific community is now learning a great deal about the psychological and physical consequences of EVD, EVD viral persistence in survivors of the disease, risk of EVD relapse in survivors, and the potential for survivors to transmit the virus to others. There have been significant second-order impacts of the epidemic on household welfare, human development, and investment. While focusing primarily on containing the outbreak, governments and development partners sought from the outset to limit or mitigate these second-order impacts by avoiding excessive restrictions on movement and providing humanitarian relief to areas with high infection rates. However, with the global efforts focusing on containing EVD, many people have failed to receive treatment for other diseases, such as malaria and measles, and this has led to even more deaths. The main impacts of the crisis on private incomes and public revenues have resulted from policy-induced and voluntary reductions on normal economic activity in the form of transport restrictions, border closings and local quarantines, furloughs of non-essential public-sector workers, departure of expatriate personnel, and reduced engagement in production and trade by citizens. While economic activity has begun to rebound in the region, where many official restrictions have been lifted, the pace of recovery is weakened in all three most-affected countries by the depletion of household and business-sector assets (including in the financial sector) during the crisis, the interruption of large￾scale investment plans, and the ongoing caution of investors in a situation where there are elevated uncertainties and health risks. The USAID Ebola strategic framework laid out an EVD recovery approach that includes measures to revive economic activity, restore inclusive growth, and strengthen the capacities and resilience of public institutions. Pillar II programs targeted second￾order impacts of EVD with a focus on restoring development gains lost due to the effects of EVD including strengthening key institutions and infrastructure further weakened by the impact of the disease; addressing decreased levels of citizen trust in government; rebuilding a reduced willingness of people to accept social messaging on EVD; and ensuring that countries are recovering in a way that leaves them better prepared for and resilient to crises like EVD moving forward. B–2 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME II 3. RELATIONSHIP TO USG GLOBAL PRIORITIES USAID’s Ebola recovery efforts fall within the development priorities of the U.S. Government (USG), particularly in Africa, which focus on 1) strengthening democratic institutions, 2) spurring economic growth, trade, and investment, 3) advancing peace and security, and 4) promoting opportunity and development. USAID, as the USG’s premier development agency, leads the development and is leading the implementation of the USG’s strategy on Ebola. Pillar II of the Strategy focuses on recovery, post-Ebola, and ensuring that countries better prepare for and to respond to EVD and other future crises. There are five sub-components of Pillar II: food security, health systems recovery and non-Ebola health services, governance and economic crisis mitigation, basic education, and innovation, technology and partnership. 3.1. PILLAR II SUB-COMPONENT M&E OBJECTIVES United States Missions in West Africa (e.g., USAID/Liberia, USAID/Guinea, USAID/Sierra Leone), and USAID Operating Units (OUs) are responsible for developing, implementing, monitoring, evaluating, and reporting on sector-specific activities such as food security, health, democracy and governance, etc. They are directly responsible for carrying out the field activities and are expected to achieve the intended activities-level results in one or more of the affected countries where they have program activities (see list below). Each Bureau or Office in Washington that is supporting Ebola recovery efforts under Pillar II of the USG Strategy adheres to its own M&E framework with established objectives and activities and quarterly updates to the AEU. Below are illustrative objectives each Bureau or Office is working toward. Food Security Increased availability of, and accessibility to, food by vulnerable households reduced poverty and under nutrition, and trade and market systems restored: ■ Household food access increase: increase food access of the most vulnerable groups impacted directly and indirectly by Ebola in both urban and rural areas • Households received cash transfers/food vouchers for food • Children received hot meals in schools ■ Market and agricultural recovery: increase food availability through the recovery of local food production and market function • Households received agricultural input vouchers • Small-scale traders received cash grants Health Systems Recovery and non-Ebola Emergency Health Services USAID, working with host governments and other partners, will address the longer-term recovery of health systems with the goal of improving their ability to respond to the EVD outbreak; increase demand for health services within communities in Ebola-affected countries; build capacity to manage health crises; and improve service delivery. The following objectives summarize the goals under this subcomponent: ■ Restoration of primary health care in public facilities: restart health services including immunizations, materials and child health, and family planning to pre-EVD levels ■ Social mobilization, behavior change risk communication ■ Supply chain and commodities: improve service delivery and logistics ■ Human resources for health: improve management of health crises through triage protocols; integrate capabilities created during the response into the health care system ■ Health governance and management: address healthcare workforce issues. Governance and Economic Crisis Mitigation (GECM) Programs focused on recovery will target those individuals and communities hardest hit by EVD, supporting activities that empower individuals, including survivors, and community groups to foster greater accountability of government-provided services most critical to respond to and recover from EVD; improve social￾protection services for vulnerable populations; enhance water and sanitation services in EVD-affected communities; and provide healthy and safe learning environments for the return of students to more than 4,400 primary schools. The intended results for this subcomponent are: ■ Civil society: empower individuals and communities to improve accountability in public sector services ■ Elections process: peaceful and transparent elections ■ Governance ■ Water and sanitation: reestablish clean water provision in Voinjama, Liberia ■ Safe learning environments ■ Social protection systems ■ Private-sector engagement Innovation, Technology and Partnership Funds for this subcomponent will support collaborative efforts between the USG, local governments, and the private sector in the three countries and the West Africa region to improve the quality and sharing of health information and decision-making, and ANNEX B. SCOPE OF WORK n B–3 strengthen communications and digital financial systems necessary to prevent, detect, and respond to epidemics. These objectives will be attained by pursuing the following indicators: ■ Grand challenge: attract innovations, new technologies and partnerships to address response and recovery challenges ■ Information communications technology: improve health information sharing, quality, and decision making; strengthen communications and digital financial systems to improve prevention, detection and response to future epidemics Tailored to individual country context and needs, these activities will help strengthen critical institutions, including governance and infrastructure required to keep countries on a path toward long￾term development progress. USAID will also support efforts that address the limited capacity of government institutions to respond to this shock through initiatives to increase the effectiveness, trans￾parency, accountability, and responsiveness of governance struc￾tures, as well as maintain momentum on key governance reforms. Funding will support critical infrastructure rehabilitation and investments that address key barriers to trade and access to basic services including power, water supply, transportation, and telecommunications infrastructure. Funding will also support education programs aimed at re-opening and improving the conditions of schools in order to keep children in school and protect and advance educational gains in literacy and numeracy. Activities will focus on ensuring that schools are equipped to prevent EVD transmission—for example, through the installation of hygiene and handwashing stations—and that school personnel are trained in appropriate screening and response protocols. Funds will support programs to assist vulnerable populations in meeting their basic needs as well. To meet the challenges of lack of local capacity, infrastructure deterioration, and the government’s difficulty in carrying out the activities or provide adequate supervision, USAID entered into partnerships with contractors and subcontractors knowledgeable about and experienced in implementing activities in Guinea, Sierra Leone, and Liberia. Through these efforts, USAID worked collaboratively with their partners to develop activities implementation plans and M&E plans. Those plans delineated the roles and responsibilities of each party. Implementing partners are in charge of executing the tasks in the field and provide regular monthly and/or quarterly reports to USAID. Additionally, they conduct routine monitoring of their activities. USAID undertakes routine monitoring, usually on a quarterly basis, and procures an evaluation of their programs in the field. 4. SCOPE The contractor will conduct performance evaluations of USAID’s Ebola recovery efforts in Guinea, Liberia, and Sierra Leone during the timeframe of March 2015 through December 2019. The purpose of these activities will involve capturing, aggregating, and reporting on all recovery activities’ performance and impact toward achieving established Pillar II macro-level objectives as stated in the AEU Strategic Framework and the monitoring and evaluation (M&E) Results Framework. More than a year since the start of the EVD outbreak, the affected countries have reduced the number of cases to zero by identifying and breaking new chains of transmission with the support of the international community. However, given the precarious state of the current situation in Guinea, Sierra Leone, and Liberia, the EVD response highlighted seven significant challenges to recovery (See Annex II: Challenge to Recovery). The evaluations shall examine the extent to which the activities undertaken in the field and the implementation approaches effectively mitigated second-order impacts of EVD and assess the sustainability of these interventions post-EVD outbreak. The evaluations will capture stories related to outputs- and outcomes￾level changes, as well as stories regarding challenges and successes regarding implementation of projects’ activities. These evaluations will be instrumental in determining how successful the USAID recovery strategy has been in meeting the objectives and providing recommendations for improving the USG design and coordination efforts to contain outbreaks around the world. In addition to analyzing component-specific information on indicators during this recovery phase, the evaluations team will also provide in-depth evaluations of and lessons learned from the activities. 5. CONTRACT COMPONENTS The present contract contains four major components, as outlined below. Throughout all of these components, the Contractor shall ensure that Pillar II goals, objectives, and assumptions are in line with the USAID Country Development Cooperation Strategy and host country priorities: ■ Component I – Evaluation ■ Component II – Routine Monitoring ■ Component III – Data Quality Assurance ■ Component IV – Support Improved Knowledge Management and Learning B–4 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME II 5.1. COMPONENT 1: EVALUATION Under this component, the Contractor shall carry out two performance evaluations in each of the three countries during the period of performance of this contract. The Contractor shall design the performance/process evaluations that focuses on descriptive and normative questions, as defined in ADS 203.3.1.1, capturing what a particular activity/project or program has achieved; how it is being implemented on the ground; whether the activities are on track to attaining the expected results; how it is perceived and valued by host governments and benefiting target populations with regard to questions pertaining to program design, management, operational, and decision-making. Performance evaluations involve before and after comparisons but lack in general a rigorously defined counterfactual. The Contractor shall also design and implement each of the performance evaluations in close collaboration with USAID/Washington, based on the circumstances in each targeted country. The design will include variables measuring key outcomes, carrying out data collection and analysis, and advising each Mission on how to develop precise approaches to develop measurable indicators, key evaluation questions, and a learning plan/agenda. 5.2. EVALUATIONS STRATEGY/DESIGN Given the nature and extent of the EVD outbreak as well as the illustrative evaluation questions, we anticipate that the design shall use a mixed method approach to collecting data, i.e., include a combination of rigorous qualitative data gathering as a main research approach with time-bound descriptive quantitative data to capture such quantitative information. In order to ensure the maximum value for learning and use, the proposed methodology should include: 1. A detailed evaluation work plan, including activities, timeline, and level of effort. 2. A detailed evaluation design, including a description of the methodology outlining the methodological strengths and limitations identified or anticipated – The contractor shall submit a detailed evaluation design and methodology. However, it is anticipated that the final methodology will be developed collaboratively by the proposed Contractor and USAID/AEU M&E team and the USAID M&E Reference Group (EMERG) in Washington, DC. Within 15 days of approval of the evaluation work plan, the Contractor must submit to the Task Order Contracting Officer’s Representative/Task Order Agreement Officer’s Representative (TOCOR/TOAOR) an evaluation design which will become an annex to the Evaluation report. The evaluation design will include: 1) a detailed evaluation design matrix that links the Evaluation Questions in the SOW to data sources, methods, and the data analysis plan; 2) the list of potential interviewees and sites to be visited and proposed selection criteria and/or sampling plan (must include calculations and a justification of sample size, plans on how the sampling frame will be developed, and the sampling methodology); 3) known limitations to the evaluation design; and 4) a dissemination plan. Note that the draft questionnaires and other data collection instruments or their main features will be prepared while USAID is reviewing the Evaluation Design and delivered at a later date to be mutually agreed. USAID Washington Bureaus, Mission Offices, and relevant stakeholders will be asked to take no more than 15 business days to evaluate the proposed design and consolidate comments through the (TOCOR/ TOAOR). Once the Contractor receives the consolidated comments on the initial evaluation design and work plan, it is expected to respond with a revised evaluation design and work plan within six business days. The final Evaluation Design will include the final evaluation questions; the methodology and data collection approaches and instruments; the involvement of Missions’ staff, local partners, and other national stakeholders; data analysis, data quality assurance, and dissemination plan as well as a plan for communication and effective use of evaluations results. The following elements shall be taken into consideration in planning, executing, and reporting on the task: a. Evaluation Briefings: The Contractor shall hold an in-briefing and out-briefing with the Mission in each country to discuss plans and present major and/or preliminary findings to USAID and other stakeholders. It is anticipated that the briefings shall take place in each Mission’s facilities. b. Draft Evaluation Report: The draft evaluation report should be consistent with the guidance provided in Section C.5.3 c. Final Evaluation Report: The final report shall address each of the questions identified in the SOW and any other issues the Contractor considers to have a bearing on the objectives of the evaluations. Any such issues may only be included in the report after consultation with USAID. The submission date for the draft evaluation report shall be determined in the evaluation work plan. Once the initial draft evaluation report is submitted, the AEU shall have 10 business days in which to evaluate and comment on the initial draft, after which the TOCOR/TOAOR shall submit the consolidated comments to the Contractor. The Contractor shall then be asked to submit a revised final draft report seven business days hence, and the AEU will evaluate and submit comments on this final draft report within five business days of its submission. The main body of the Final Evaluation Report shall not exceed 30 pages (exclusive of annexes) and must include a title page, an executive summary, introduction, background of the project being evaluated, the main evaluation questions, the methodology or methodologies, the limitations to the evaluation, findings, conclusions, and recommendations and lessons learned. The executive summary must be between three and five pages in length and summarize the purpose, background of the intervention being evaluated, main evaluation questions, methods, findings, ANNEX B. SCOPE OF WORK n B–5 conclusions, and recommendations and lessons learned. The executive summary shall not contain any information not included in the main body of the Report. The evaluation methodology shall be explained in the report in detail. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (e.g., selection bias, recall bias, etc.). The report must include the evaluation design as an annex. All tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides must be included as annexes. Sources of information must be properly identified in an annex. All quantitative data collected by the evaluation team must be provided in an electronic file in easily readable format per USAID data policy and requirements (e.g., MS Excel). The data must be organized and fully documented for use by those not familiar with the intervention or the evaluation, so that an independent evaluator could use it and arrive at the same findings and conclusions.The annexes to the report shall include: ■ The Evaluation SOW; ■ Any statements of difference regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team; ■ All tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides; ■ Sources of information, properly identified and listed; and ■ Disclosure of conflict of interest forms for all evaluation team members, either attesting to a lack of conflicts of interest or describing existing conflicts of interest. In accordance with AIDAR 752.7005, the Contractor shall make the final evaluation reports publicly available through the Development Experience Clearinghouse within 30 calendar days of final approval of the formatted report. 5.3. CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT Per the USAID Evaluation Policy and USAID ADS 203, draft and final evaluation reports will be evaluated against the following criteria to ensure the quality of the evaluation report. ■ The evaluation report should represent a thoughtful, well￾researched, and well-organized effort to objectively evaluate what worked in the project, what did not, and why. ■ Evaluation reports shall address all evaluation questions included in the SOW. ■ The evaluation report shall include the SOW as an annex. All modifications to the SOW—whether in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline—need to be agreed upon in writing by the (TOCOR/TOAOR). ■ The evaluation methodology shall be explained in detail. All tools used in conducting the evaluation—such as questionnaires, checklists, and discussion guides—will be included in an annex in the final report. ■ Evaluation findings will assess outcomes and their impact on males and females. ■ Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups if applicable, etc.). ■ Evaluation findings shall be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or the compilation of people’s opinions. Findings shall be specific, concise, and supported by strong quantitative and/or qualitative evidence. ■ Sources of information need to be properly identified and listed in an annex. ■ Recommendations need to be supported by a specific set of findings. ■ Recommendations shall be action-oriented, practical, and specific, with defined responsibility for the action. 5.4. OTHER REQUIREMENTS All quantitative data collected by the evaluation team must be provided in machine- readable, non- proprietary formats as required by USAID’s Open Data policy (see ADS 579). The data shall be organized and fully documented for use by those not fully familiar with the project or the evaluation. USAID will retain ownership of the survey and all datasets developed. B–6 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME II ANNEX C. EVALUATION METHODS n C–1 ANNEX C. EVALUATION METHODS Second Performance Evaluation of USAID Ebola Pillar II Activities / Final Report Annex II Document C.1 EVALUATION TEAMS In addition to the home office staff, IBTCI recruited three country teams to support in the design and conduct the evaluation. Each country team was led by a Senior Evaluation Advisor, who had a high level of experience and expertise in evaluation and especially in the various sectors relevant to Pillar II: health, governance, and agriculture (See Annex G). Each country team also included a local study coordinator who was an experienced data collector and knew the local context well. The Sr. Evaluation Advisors worked with the home office team in the design of the overall evaluation, and eventually, in writing individual case studies and the final report. Each country team also met in country to collect data. See C.10 Roles and Responsibility for a detailed description the complete evaluation team and roles and responsibilities. C.2 CASE STUDY DESIGN Each of the three countries implemented a different array of activities within the Pillar II program. In PE1, the Evaluation Team (the “Team”) examined activities by country and different sectors (thematic areas) that USAID supported. PE1, conducted between December 2017 and February 2018, showed that USAID’s Ebola Pillar II projects were effective and achieved the overall Pillar II objectives in all three countries. The primary purpose of the current performance evaluation activity (PE2), then, is to inform the knowledge base for the Agency’s future decision-making so it may improve its planning and preparedness for responding to shocks and/or serious disease outbreaks—especially those that have the potential for multi-sectoral impact. Therefore, PE2 employed a different method from that employed for PE1 (mixed methods including cross-sectional surveys). As the goal of PE2 is an understanding of the factors (the “how” and “why”) leading to success (effectiveness and/or self-reliance) within the full, real-life context in which the activities are implemented, a case study design was deemed appropriate (Yin, 1994). In order to explore the variety and range of ways these activities contribute to either effectiveness or self-reliance, IBTCI employed a multiple-case study approach to purposefully select cases that highlight different aspects of the Ebola Pillar II response. Per the U.S. Government Accountability Office (GAO) definitions, the individual case studies within this evaluation are conceived as explanatory, as their main aim is to examine the factors contributing to program effects. The overall report is then conceptualized as cumulative, as findings from the individual case studies are synthesized to answer the evaluation questions (GAO, 1990). C.3 CASE STUDY SELECTION A total of 15 case studies were selected spanning the three countries in different sectors as well as key outcome indicators (see Table C-1). The unit of analysis for the majority of the individual case studies are activities funded by USAID under Ebola Pillar II. USAID uses the term “activity” to identify project components that are being implemented by country partners or by other organizations that USAID has funded under a contract, cooperative agreement, grant, or other arrangement. These may have been grouped under one funding mechanism, as in the case of Fighting Ebola: A Grand Challenge for Development. Case studies were selected purposively to answer the PE2 evaluation questions on effectiveness and self-reliance. The evaluation team selected activities to feature in the case studies if they met the following criteria: ■ Contributed to the achievement of the three Pillar II objectives, but also made progress towards achieving some or all of the 19 outcome indicators in the Pillar II Theory of Change (See Annex I). C–2 SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME II ■ At least three-quarters of the project documents over the life of the project were available . In addition, activities were prioritized, especially to ensure an excellent learning opportunity for either the evaluation question related to effectiveness or journey to self-reliance. Finally, the PE2 Team ensured representation of activities across key variables: ■ Up to four case studies for each country ■ At least one case study for each sector (thematic area) ■ Multi-country (regional) case studies to understand lessons learned across all countries Table C-1 lists the activities selected for case study development and highlights some of the key selection decision-making variables. Table C–1. Case Study Selection Activity Potential Learning Sector Relevant Indicators Guinea Systems for Improved Access to Pharmaceuticals and Services (SIAPS) ■ Effectiveness ■ Self-reliance Health ■ Capability of the public health supply chain restored and expanded Health Finance and Governance (HFG) ■ Effectiveness ■ Self-reliance Health ■ Health financing and resource allocation improved; Increase of health outlays as a percentage of government spending Emergency Access to Food for EVD-Affected Guineans Program ■ Effectiveness ■ Self-reliance Agriculture and food security Economic crisis mitigation ■ Cash Transfers/ Seeds/ Training ■ Agriculture and import markets’ function restored and improved; Social protection provided to vulnerable households Consortium for Elections and Political Process Strengthening (CEPPS) ■ Effectiveness ■ Self-reliance Governance ■ Election participation increased Liberia Partnership for Advancing Community Based Services (PACS) ■ Effectiveness Health ■ Health behavior gains enhanced and reinforced ■ Health infrastructure (including community involvement) strengthened Collaborative Support for Health (CSH) ■ Effectiveness ■ Self-reliance Health ■ Improved facilities; infection prevention and control (IPC) Economic Recovery from Ebola for Liberia (EREL) Agriculture and food security Economic crisis mitigation ■ Cash Transfers/ Seeds/ Training ■ Agriculture and import markets’ function restored and improved; Social protection provided to vulnerable households Civil Society and Media Leadership (CSML) ■ Effectiveness ■ Self-reliance Governance ■ Management systems and accountability improved at government levels, ministries, and stakeholders Sierra Leone Post-Ebola Recovery of Health Services (PERHS) ■ Effectiveness ■ Self-reliance Health ■ Improved facilities ■ IPC SIAPS ■ Effectiveness ■ Self-reliance Health ■ Capability of the public health supply chain restored and expanded ANNEX C. EVALUATION METHODS n C–3 Table C–1. Case Study Selection Activity Potential Learning Sector Relevant Indicators Food for Peace Cash Transfer (FFP/CT) Agriculture and food security Economic crisis mitigation ICT ■ Cash Transfers/ Seeds/ Training ■ Agriculture and import markets’ function restored and improved; Social protection provided to vulnerable households ■ Connectivity; support and expand infrastructure of mobile money Regional Health Communications Collaborative (HC3) ■ Effectiveness Health ■ Health behavior gains enhanced and reinforced Restoration of Health Services (RHS) ■ Effectiveness ■ Self-reliance Health ■ Improved facilities ■ IPC mHero ■ Effectiveness ■ Self-reliance ICT ■ Expanded communications Fighting Ebola: A Grand Challenge for Development ■ Effectiveness ■ Self-reliance Cross -sectoral ■ Innovations C.4 DATA COLLECTION METHODS To develop the case studies, the Team utilized both primary and secondary sources of data, including: 1. Semi-structured key informant interviews (KIIs): The Team conducted 104 KIIs with 138 persons from several stakeholder groups including government officials, other implementers, and actors from civil society and the private sector (Table C-2, Table C-3). 2. Focus groups discussions (FGDs): The Team facilitated 41 focus group discussions (FGDs) with 321 respondents with service providers (e.g., health workers, government officials) and beneficiaries of the selected USAID Ebola Pillar II projects (Table C-4, Table C-5). 3. Site visits: The Team visited (30) service delivery sites, such as peripheral health units (PHUs) and communities in catchment areas of Pillar II activities, to interview both direct (e.g., reproductive, maternal, newborn, and child health (RMNCH) client) and indirect (e.g., trained health worker) beneficiaries, as well as to establish a fuller context of the case studies for the evaluators. 4. PE1 report: The Team utilized analysis and relevant information from the PE1 report. 5. Activity documents and IP reports: The Team reviewed more than 800 activity documents, (including quarterly and annual reports, monitoring and evaluation (M&E) plans and reports, baseline and endline studies). Some of these documents were collected during PE1 and others were specifically solicited from the IPs of the activities portrayed in the 15 case studies. 6. Other sources of data: The Team also consulted a variety of other secondary data sources and literature during the development of the case studies. All primary data collection activities were guided by semi￾structured KII guides and FGD guides (see Annex D). Data collection tools were designed by the Team. Table C–2. Key Informant Interviews (KIIs) and Key Informants (KIs) Country Number of KIIs Number of KIs Guinea 35 43 Liberia 36 49 Sierra Leone 33 46 Total 104 138 C–4 SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME II Table C–3. KIIs by Stakeholder Group Country Type of KI Number of KIs Guinea (43) Partner country government staff 16 Service providers 8 Implementing partner staff 19 Liberia (49) Partner country government staff 12 Community service organization staff 11 Implementing partner staff 19 Other partners 3 Sierra Leone (46) Partner country government officials 17 Implementing partner staff 25 Community service organization staff 4 Table C–4. Focus Group Discussions (FGDs) Country Beneficiaries Service Providers Total FGDs Guinea 8 6 14 Liberia 13 9 22 Sierra Leone 3 2 5 Total 24 17 41 Table C–5. Participants of FGDs Country Beneficiaries Service Providers Number of Participants Guinea 47 35 82 Liberia 129 59 188 Sierra Leone 37 14 51 Total 213 108 321 ANNEX C. EVALUATION METHODS n C–5 C.5 SAMPLING AND RECRUITMENT STRATEGY To gather primary data (KIIs and FGDs), the Team selected respondents purposively based on their stakeholder group and capacity to speak knowledgeably about the PE2 Evaluation Questions (EQs) and/or EVD activities. Because Pillar II activities varied widely in geographical coverage, time period, and duration and they involved such a wide range of stakeholders, it was necessary to prioritize and recruit respondents from the stakeholder groups relevant for each PE2 EQ (government staff, implementing partners, community-based organizations, geography, etc.). The list differed for each country. A sampling matrix for the KIIs and FGDs was prepared by the Team to help with selection and recruitment of KIIs in each country, distributed as evenly as possible across the identified activities. To maximize practical insights and lessons learned, the sampling for key informants was designed to capture insights and lessons learned from as broad a range of knowledgeable and experienced people as possible. KII respondents were selected from the prioritized respondent groups that were identified in PE1. Finally, “snowball” sampling was used to select additional key informants; interviewed persons were asked to identify others who might have good information to share. To ensure adherence to the recruitment specification and any quotas, recruitment for FGDs was conducted by local evaluation specialists supervised by the Senior Evaluation Advisors. The local evaluation specialist personally calls all respondents prior to the focus group taking place to ensure that they have been correctly recruited and that they fit the agreed recruitment specifications. Any respondent who is ineligible is then replaced. A summary list of the KIIs and FGDs for each country is provided in Annex E. C.6 DATA COLLECTION PROCEDURES The active data collection phase included procedures to assure data quality processes and procedures at the field and home office levels during both collection and management. 1. Putting participants at ease In order to elicit information that is both accurate and reliable, every effort was made to ensure that participants were comfortable and at ease. If at all possible, interviews were conducted in the primary language of the participant. All participants were informed of the purpose of the study, of the protections in place to protect their identity, and of their right to withdraw from the study at any point. 2. Accurately capturing information All FGDs and KIIs were audiotaped if permission was given by the participant(s). Moderators/interviewers then transcribed recordings and translated into English. The evaluation advisor reviewed each transcript against the audio files, as well as notes, to verify that all dialogue had been captured within the transcripts. If any problems or omissions were noted, the transcript was returned to the transcriber who was then required to listen again to the entire audio file and provide further information. When transcription became too time-intensive and threatened the ability to meet deadline, the audio tapes were sent to a professional transcription service for verbatim transcriptions. 3. Data management, data back-ups and storage The data was stored on a single machine (IBTCI HO server, “Primary Data’’ subfolder under Pillar II Folder), with incremental backup locations, and periodically reviewed to check for authenticity, integrity, and quality. Access to data was restricted to only those who signed the non-disclosure agreement. C.7 DATA ANALYSIS Data analysis was a three-phase process: ■ Phase I involved organizing data for efficient coding and retrieval, both for the current analysis effort and for future queries. ■ Phase II involved coding all data with predetermined coding framework by trained qualitative coders. ■ Phase III involved the Team examining coded data to identify themes and patterns to answer the EQs for each case study, as well as ultimately across case studies. The Team used a web-based qualitative data analysis software – Dedoose (https://www.dedoose.com/) – to organize, code, and analyze the qualitative data. The use of Dedoose allowed for collaborative work and use of coded data by all coders, as well as by evaluation advisors. C–6 SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME II PHASE I – ORGANIZE DATA Three different types of data were imported into Dedoose. ■ Participant information (structured data), which included relevant information about the participants in the study including gender, age, whether or not the person was present in country during the Ebola epidemic, and other salient factors. ■ Data information (structured data), which included information to describe the transcripts, including the data collection method (e.g., FGD, KII), the interviewer, and the assigned coder. ■ Transcripts (qualitative data) were the primary source of data. After importing into Dedoose, participant information and data information were linked to each transcript. PHASE II - CODE The goal of coding was to organize the qualitative data in such a way that the final phase (III) of analysis would be able to be performed powerfully. A hierarchical code book was created to ensure consistent coding by several coders. The following large domains of code were created (see Annex J.2 Codebook for a complete listing of the codes): 1. Activities are various interventions and/or sets of one or more interventions implemented by UN agencies, private firms, and NGOs funded by USAID under Ebola Pillar II to bring about a variety of changes to address the second order effects of the 2014–2016 EVD outbreak in West Africa. Sub-codes under Activity name each of the interventions that are targeted for case study analysis. 2. Actors are various persons, organizations, and entities who participated in addressing issues relevant to each country’s people, economy, and social systems. 3. Sectors are various thematic areas in which development activities are programmed, including Agriculture and Food Security, Economic Crisis Mitigation, Education, Governance, Health, and Innovation, Technology, and Partnerships. 4. Factors are various circumstances which affect the success or failure of activities against their objectives, which implementing partners (IPs) may or may not have control over. Factors may describe the context in which an activity may succeed or the determinants that must be present in order for an activity to be sustainable. 5. Strategies are intentional actions made by development actors to achieve intended objectives. Types of strategies coded for are those defined in the codebook and include activity design, approach to implementation, partnerships, and others. 6. Outcomes targeted by strategies and activities: a. With reference to Effectiveness (EQ1), codes include effectiveness, success, failure, high performer, and low performer. b. With reference to Self-Reliance (EQ2), codes include capacity, commitment, sustainability, and attitudes, behaviors, and choices. 7. Objectives are the overall objectives of an entire development program, in this case the Ebola Pillar II recovery program. 8. Target Populations are key populations or vulnerable groups which may be the targeted beneficiaries of the activities, such as women, older adults, and youth. 9. Sentiments are strong views or attitudes toward a situation or event, including positive and negative. 10.Good quotation are quotes that may be useful to highlight certain issues in the case studies and reports. 11. New idea are any important or relevant ideas that were mentioned that cannot be described by any of the codes above. Coders were trained in coding. Each coder was assigned a certain number of transcripts to code. Weekly meetings were held to assess progress and document any issues or lessons learned during coding. In general, coders were instructed on the following general principles: ■ A specific item may be coded by several codes simultaneously. ■ As often as possible, the most specific code should be applied; for instance, if a strategy is being discussed, a code describing a specific strategy should be applied rather than the general Strategy code. ■ If a new type of a certain code is being discussed, this can be applied using the Other code. For instance, if a strategy is being discussed that does not fall under any of the pre-coded sub￾codes, it should be coded by Strategy: other. ■ A code should be applied to a large enough piece of text to give context to what is being described. In other words, a whole paragraph was coded unless the paragraph was talking about two different concepts. ANNEX C. EVALUATION METHODS n C–7 PHASE III - IDENTIFY THEMES AND PATTERNS Once data was coded, Sr. Evaluation Advisors performed a variety of analyses on the coded data in Dedoose. These analyses were done in two stages: 1. Develop each case study. 2. Analyze case data across all data to examine for cross-cutting findings. Each evaluation advisor used one or both of the following two approaches to analyze data and build the case study: ■ Content Analysis to systematically code and categorize primary data for analysis to identify trends and patterns with respect to specific domains of interest for the evaluation (e.g., drivers of self-reliance) (Stemler, 2001) ■ Grounded Theory to identify new and emerging themes across all sources of data, as well as across case studies and countries (also called analytic induction) (Strauss and Corbin, 1997) The IBTCI home office scheduled group meetings between PE2 Team members at regular intervals (often weekly) to discuss findings from individual case studies and countries, solve problems, and generate cross-cutting findings to write the final report. The Team gave special attention to differences in the experiences and perspectives between beneficiary groups, sectors, and countries. C.7.1 Effectiveness (EQ1) The first EQ explores how Pillar II activities achieved their success (Table C-6). Table C-6. EQ1: Effectiveness Overall Which Pillar II activities contributed to the achievement of one or more Pillar II’s three overarching objectives: (1) prevent the loss of development gains; (2) recover and strengthen existing institutions and infrastructure; and (3) build sustained systems through PPPs, innovation and capacity building? 1.a What were the factors that contributed to the effectiveness of stand-alone activities and why (i.e. did/would/could this activity succeed on its own merit or did it fail)? 1.b What were the factors that contributed to the effectiveness of a combination of activities within each sector and why? 1.c What were the factors that contributed to the effectiveness of cross-sectoral activities and why (i.e. how did cross￾sectoral activities bolster the successful outcomes)? For these questions, the Team focused on identifying factors (noted by key informants and the respondents) as contributing to an activity’s successes or lack thereof, especially with respect to outcomes to the three objectives as described in the overall Pillar II Results Framework. To answer the sub-questions on factors that contributed to successful outcomes of “stand-alone”, “multiple activities in one sector”, and “multiple activities in multiple sectors”, the Team sought examples of such collaborations within a case study as well as across all cases. C.7.2 Self-Reliance (EQ2) The second EQ focuses on Pillar II activities’ contributions to a country’s journey to self-reliance (Table C-7). Table C-7. EQ2: Self-Reliance Overall What Pillar II activities facilitated new partnerships or strengthened existing partnerships (including leveraging the private sector) to advance partner country self-reliance? 2.a What were the salient factors and/or determinants that positioned these activities to advance a country’s journey to self-reliance? 2.b To what extent did these activities strengthen country systems to mitigate and respond to future epidemics? To address the PE1 EQ regarding sustainability of Pillar II activities, the PE1 Team created a Sustainability Framework to examine determinant(s) for both promising interventions and those with a lesser chance of positive outcome, based on a review of the literature. Key determinants that contribute to sustainability include policies, financial resources allocated from C–8 SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME II national or private sector accounts, the anticipation and mitigation of operational issues (such as supply and logistics, maintenance, and provision of power, internet, and water), accountability, institutional support, human resources (availability, capacity, and supportive supervision), and community support at the local and national levels. Which determinants are missing may help explain why an intervention may not achieve maximum sustainability. In PE1, the team used this Framework to examine the presence or absence of these determinants as a method to assess the potential for the sustainability of a specific types of intervention. IBTCI posits that results may be sustained over time provided that a number of enabling and influencing determinants are also supported during program efforts (Moore, 2019). Helping countries to achieve self-reliance—defined by USAID as “a country’s ability to plan, finance, and implement solutions to address its own development challenges” is currently the keystone of USAID’s development strategy. USAID is realigning and reorienting its policies, strategies, and program practices with an eye to transferring development capability to each country. The EQ2 in PE2 focuses on whether and how Pillar II activities supported these countries to advance toward self-reliance. USAID’s “journey to self-reliance” framework identifies two key drivers: ■ Country commitment: This is the degree to which a country has the processes and structures necessary foundation to solve its own development challenges. Key parameters that demonstrate a country’s commitment include its laws (including open and accountable governance), social and micro-/macro- economic policies, and cultures and norms (including inclusive development across gender, social groups, and geography). ■ Country capacity: This relates to current capabilities across the dimensions of political, social, and economic development. Capacities that are important for advancing a country to self￾reliance are in a number of areas comprising of government (including the quality of government workforce and services), civil society (including free media a means to hold government accountable and to provide mechanisms beyond elections through which citizens can be heard), citizenry (the level of to which they are informed and engaged), and the economy (including the extent to which it is functioning and productive). There is an important link between USAID’s current emphasis on the journey toward self-reliance and its longstanding commitment to sustainability (Brinkerhoff, 1992). As discussed above, during PE1, the Team created a Framework to examine the potential for the sustainability of a specific types of intervention (Moore et al, 2019). For PE2, the Team created a new Framework to conceptualize self-reliance as different from but related to sustainability in a number of important ways (Figure C-1). Sustainability as examined in PE1 is different from self-reliance in a number of important ways: ■ Perspective: While sustainability examines the lasting effects of interventions supported by implementing partners/donors, self-reliance examines outcomes through the lens of the host country. Figure C–1. Interrelationship of Sustainability and Self-Reliance Frameworks Sustainability determinants may also serve as indicators of country commitment Policies, Strategies, and Plans Finances External Threats and Enabling Environments Determinants Accountability Institutional Support Human Dimensions Technical Choices Perspective: IP/Donor Focus: Intervention Level: Multiple Sustainability Framework Self-Reliance Plan, Finance, Implement Perspective: Country Focus: System Level: National Drivers Increased Capacities Government, Civil Society, Citizenry, and Economy Country Commitment Laws, Policies, and Norms Self-Reliance Framework Sustained Outcomes ANNEX C. EVALUATION METHODS n C–9 ■ Focus: While sustainability is examining the context that influences the intervention, self-reliance takes on a much broader, systems focus. ■ Level: While sustainability examines outcomes from multiple levels (including individual, local, regional, national), self￾reliance starts examination at the national level. Despite these differences, sustainability and self-reliance are highly interrelated. The Team recognizes that some determinants of sustainability—originally conceptualized as externalities that have an impact on the outcome of an intervention—are, in fact, important indicators of the commitment and ownership on the part of the country in the self-reliance framework. These determinants include: ■ Policies, strategies and plans, which lay the ground work for the sustainability of intervention, are also a key sign of a country’s commitment. ■ Allocation of financing is a critical factor that determines the sustainability of outcomes; it is also a key indicator of a country’s commitment. ■ Accountability (including financial, political, social, and administrative) is an important determinant of sustainability— and also an indicator of both a country’s commitment and its government capacities. The other determinants of sustainability, are not, at first glance, as clearly relevant for a country’s journey to self-reliance as they are more narrowly focused on the intervention and its outcomes. ■ Technical Choices: Examines the appropriateness and relevance of the solution (intervention) to the problem ■ Human Dimension: Explores cultural and institutional norms, but is also narrowly focused on whether and how they affect the intervention outcomes. ■ Institutional Support: Examines the systems that are in place to support implementation of an intervention. ■ External Threats & Enabling Environments explores the factors that are outside the control of the activity but non-the-less may affect the intervention. Sustainability is a necessary but insufficient condition for achieving self-reliance. Cumulative and sustained outcomes in various sectors are preconditions for increased capacity at the national level to handle future development challenges. Finally, although difficult to trace from the micro level of individual actors and projects to the broader concept of national self-reliance, ownership and political/social will (commitment) is also intertwined in a theory of change. While on one hand, self-reliance and the sustainability frameworks are examining similar dimensions, on the other self-reliance shines light on a very important issue that is missing for sustained and long-term development—country ownership (commitment). In doing so, some of the sustainability determinants transform from an externality to something that is within control of the country and its development partners. Thus, to answer PE2’s self-reliance EQs, the Team used the new Framework above to guide the analysis of: 1. Overall Self-reliance: What progress have countries made towards self-reliance in terms of prevention and/or preparedness for responding to and mitigating the impacts of emergencies? 2. Capacities: Is there any evidence of system-wide increased capacities of 1) government institutions, 2) civil society, 3) the citizenry, and 4) the economy? 3. Commitment: Is there any evidence of increased country commitment, by examining improvements in: 1) policies, strategies, and plans, 2) finances, and 3) accountability? C.8 DATA MANAGEMENT AND QUALITY ASSURANCE Qualitative data arrived as a continuous stream and were stored in raw form as they were received from the field. All Team members transmitted KII and FGD transcripts in a timely manner (within 48 hours of collection) to the home office from IBTCI email only. The Team used Dedoose to store, code, and analyze qualitative data from KIIs and FGDs. After data collection and prior to analysis, data were converted to a single format to assure uniformity for storage and export. The Data Manager was responsible for tracking the data collection with the date of interview, date of transcript received, and date of upload to the Dedoose project library, ensuring that if data is lost during transmission, field-based Evaluation Teams resent it. Assignments and document grouping were built in under a Master Data Analysis Dedoose file (e.g., KII from IPs coded by Person A, FGDs from communities Person B). The Final Evaluation Report underwent two USAID reviews before its finalization. The Final Evaluation Report was submitted to IPs referenced in the report for fact-checking. C–10 SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME II C.9 ETHICAL CONSIDERATIONS, CONFIDENTIALITY, AND DATA SECURITY This evaluation, as in all IBTCI research, followed IBTCI’s policy and guidelines regarding ethics in research, which are aligned with USAID and the U.S. Department of Health and Human Services’ policies. The evaluation also received local institutional review board approval in Guinea, Liberia, and Sierra Leone. The IBTCI policy includes measures to protect human subjects (including informed consent and confirmation of no penalty for non-participation) and to balance the costs to respondents (time, transport, psychosocial costs when addressing sensitive subjects) with the resultant benefits that are expected to spring from the research. Knowledge gleaned from PE2 should contribute to improving on-going activities and informing USAID’s future programming, which constitute a benefit to the respondents. IBTCI will make every effort to minimize the burden and maximize the benefits to respondents and beneficiaries. Confidentiality of information. All data gathered in relation to the Pillar II PE2 were collected and held in electronic files on a computer separate from the shared server to which only the immediate research principals (IBTCI key staff and consultants, USAID Contracting Officer Representative, Agreement Officer Representative, and M&E point of contact) have access. IBTCI has ensured that all staff involved in the Pillar II PE2 provide written acceptance the non-disclosure agreement as a condition to work on PE2, and IBTCI’s ethics policy as a condition of employment (See Annex F). Adverse events protocol. Due to the continuing sensitivity of the EVD crisis, especially to individuals who faced personal losses due to the outbreak, especial care was taken to ensure that the interviewers/leaders of FGDs understood how to cope with adverse reactions to participation in the study. The routine provisions in the informed consent procedures stressed all respondents’ rights to skip questions or to stop data collection at any point if they were uncomfortable. Ethical issues mitigation. Appropriate steps were taken to anonymize all data after completion of analysis, prior to submitting collected data to USAID. No issues were identified during data collection and analysis that involved ethically problematic issues, including harmful, illegal, or wrongful behavior or issues of coercion, violence, corruption, fraud, or other forms of malpractice. C.10 DISSEMINATION PLAN This is a utilization-focused evaluation in goals and methodology (Patton, 2012). As such, dissemination of results is an integral part of IBTCI’s planning and budgeting for the evaluation. IBTCI frames dissemination as an interactive and iterative process of consultation, reflection, and mutual learning. Presentation of the preliminary results from PE2 was delivered to the USAID country missions and USAID Operating Unites (OUs) on July 22, 2019; comments on the report were solicited and discussions on lessons learned were held. The selected sections of final evaluation reports will be disseminated at a learning forum to be held in Washington DC in September 4, 2019, planned in close collaboration with the USAID/Activity Final Forum Evaluation Team. Following USAID approval of the Final PE2 Report, presentations will be made for the Missions and OUs in August or September 2019. The approved report will be uploaded in the USAID Development Experience Clearinghouse. C.11 LIMITATIONS The Team sought complete documentation for the selected PE2 selected Pillar II activities. Complete documentation, by USAID standards, includes the IP’s SOW and any significant modifications to its contracts or cooperative agreements, the M&E plan, and reports of progress against indicator targets, baseline studies or assessments, annual and quarterly reports, midline and endline evaluations, and any research or special studies conducted. Perhaps due to the multi-sectoral nature of Pillar II activities and multiple USAID funding streams, it has been difficult to obtain access to activity data and detailed financial reporting. Additionally, IBTCI found inconsistencies in data between sources: information on the start date and end date was often incomplete and indicator data for most activities were not received. Another critical challenge has been that some Pillar II funding is channeled through ongoing cooperative agreements that do not disaggregate Pillar II funding from other funding sources. USAID has noted this issue and some IPs now disaggregate Pillar II expenditures in quarterly reports. Reporting is inconsistent, however, and cooperative agreements do not require that the IPs provide disaggregated figures. Another limitation was that the loss of institutional memory that was a constant challenge during PE2. While it was important to conduct this study after a period of time has passed, to judge the sustainability of these activities, at the time of data collection several activities had been closed and staff had moved. In some instances, government officials were no longer in their original post at the time of the study. Even if they were located, some people forgot key events and conditions, leading to recall bias. IBTCI has ANNEX C. EVALUATION METHODS n C–11 tried to mitigate this through triangulation of a variety of methods and sources of data. The Team also used probing and other methods during interviews to try to draw respondents out. Finally, a key limitation to making attributions or causality was the inability of participants to distinguish between activities that may or may not have contributed to an outcome they observed. This was a result of the Pillar II design that coordinated complementing activities to support a specific goal. A further challenge to determining Pillar II contributions was that some activities were designed to layer onto ongoing programs. In addition, neither PE1 nor PE2 included an experimental design. For these several reasons, neither PE1 nor PE2 claim to attribute a certain outcome to a specific activity. C.12 ROLES AND RESPONSIBILITY The following table outlines roles and responsibilities of key staff on PE2. Table C–8. Roles and Responsibilities of Key Staff PE2 Task Responsible Staff Role Technical oversight Annette Bongiovanni Project Director and Technical Oversight Management and quality control Richard Columbia Chief of Party (COP) Evaluation study design Annette Bongiovanni, Zhuzhi Moore, Barbara de Zalduondo, Donna Espeut, Karim Sahyoun, and Betsy Nolan, Project Director, Team Lead, Senior Evaluation Advisors, Research Associate KII and FGD questionnaire design, pre-testing review and version control Zhuzhi Moore, Betsy Nolan, Barbara de Zalduondo, Donna Espeut, and Karim Sahyoun Data Collection Lead, Research Associate, Sr. Evaluation Advisors Country field teams: KII and FGD facilitation, production and submission of written transcripts Guinea: Karim Sahyoun and Roger Emmanuel Millimono, Liberia: Barbara de Zalduondo and Richard Ngafuan Sierra Leone: Donna Espeut and Samba Kamara Sr. Evaluation Advisors, Field Coordinators Data collection oversight Zhuzhi Moore Data Collection Lead Data quality assurance and analysis Sara Woldehanna Data Analyst/DQA Lead Dedoose Data Administrator Betsy Nolan and Yasmin Ibrahim Data Administrators Desk Review Manager Betsy Nolan Data Administrator Dedoose Data Coders Betsy Nolan (lead), Katherine Labombarde, Yasmin Ibrahim, Victoria Sesay, Maria Pastor Data Coders Case study development, analysis, and writing Sara Woldehanna, Barbara de Zalduondo, Donna Espeut, Karim Sahyoun, Jackie Yiptong Avila, Betsy Nolan PE2 team lead, Sr. Evaluation Advisors, Research Associate C.13 REFERENCES Brinkerhoff, D. & Goldsmith, A. (1992). Promoting the sustainability of development institutions: A framework for strategy. World Development. Volume 20, Issue 3, March 1992, Pages 369-383 Moore, Z. et al. (2019). Performance Evaluation of USAID Ebola Pillar II Activities: Final Report. Evaluation Report to USAID, February 2019, International Business & Technical Consultants, Inc., Vienna, VA. Patton, M.Q. (2008). Utilization-Focused Evaluation, 4th ed, Michael Quinn Patton Sage Publications, 2008. Stemler, S. (2001). An overview of content analysis. Practical Assessment, Research & Evaluation. Retrieved from: https:// pareonline.net/getvn.asp?v=7&n=17 Strauss, A. & Corbin, J.M. (1997). Grounded Theory in Practice. Thousand Oaks, CA: Sage Publications, Inc. C–12 SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME II ANNEX D. DATA COLLECTION TOOLS n D–1 ANNEX D. DATA COLLECTION TOOLS Second Performance Evaluation of USAID Ebola Pillar II Activities / Final Report Annex II Document TOOLS LISTING D.1 Informed Consent for Key Informant Interviews (KIIs) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D–2 D.2 Implementing Partner Guide (kii) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D–3 D.3 Country Government Guide (KII) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D–10 D.4 NGO, CSO, and Other Partners’ Guide (KII) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D–17 D.5 Facilitator Instructions and Information Sheet (FGD) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D–24 D.6 Facilitator’s Guide (FGD) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D–27 D–2 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME II USAID/Pillar II Ebola Recovery Efforts to Mitigate Second Order Impacts of the Epidemic in Guinea, Sierra Leone, and Liberia Performance Evaluation 2 Oral Informed Consent Good morning/Good afternoon/Good evening: My name is __________________, [and my colleague is __________________.] We are part of the IBTCI team contracted by USAID Africa Bureau to conduct the second performance evaluation of USAID’s activities in support of [Country’s] recovery from the Ebola outbreak in 2014-2015. USAID has programmed resources to support the country’s recovery from the indirect effects of the Ebola crisis beyond the actual illness, such as the effects of the Ebola crisis on health services for conditions other than Ebola, or the effects on the economic, agricultural, or education sectors. This is the second of two performance evaluations. IBTCI conducted the first performance evaluation (PE1) from November 2017 to February 2018. Overall, PE1 found that that USAID’s Ebola Pillar II activities were effective and achieved the Pillar II overall objectives in all three countries. The second performance evaluation (PE2), which is currently underway, will focus in-depth on a limited number of Ebola recovery activities (case studies) to inform the knowledge base for the Agency’s future decision-making so it may improve its planning and preparedness for responding to shocks and/or serious disease outbreaks, especially those that have the potential for multi-sectoral impact. We are conducting in-person key informant interviews (KIIs) with stakeholders such as yourself, whose views are essential for answering the key questions that frame the evaluation. The Evaluation Questions (EQs) address effectiveness of USAID’s Pillar II activities (i.e., recovery activities) and how these activities have advanced country self-reliance, or the ability of countries to address their own development challenges. If you agree to the interview, I will ask your views on a number of questions and issues, and I will take notes of our discussion. You are very welcome to stop me to ask questions during the interview. This interview will take approximately one hour. The information you provide will remain confidential. To protect your privacy, we will detach your name and organization from your responses. We might include quotes to emphasize a point made by many respondents; however, any quote used in the report will be identified only by the participant’s gender and stakeholder group. It will not be linked directly to you or to the organization where you work. If you do not wish to have anything you mention quoted, please let us know now. Your participation in this discussion is completely voluntary. You may choose not to participate, or not to answer some questions, or to stop the interview at any time. If you chose not to participate, there will be no negative consequences. You will not be receiving any payment or allowances for your participation. If you have any questions about your rights as a study participant, please contact [NAME] from the [Country] Institutional Review Board at Tel No. [ ]. Do you have any objection to participating in this interview, or do you have any questions before you can decide? Are you willing to be interviewed? [InfConsent]_______________ Yes/No I would like to tape record the interview so that we can be sure that we captured your views correctly. The recording will not be shared with anyone outside of the research team. May I record the interview? [OKtoRecord]____________________Yes/No [INSERT NAME AND CONTACT OF LOCAL POINT OF CONTACT INCLUDING IRB IN CASE OF QUESTIONS AS APPLICABLE TO COUNTRY] Interviewer: Please sign and date_________________________________________________ Participant ID: (transcript file document name) ___________________________________ Thank you very much. D.1 INFORMED CONSENT FOR KEY INFORMANT INTERVIEWS (KIIS) ANNEX D. DATA COLLECTION TOOLS n D–3 USAID/Pillar II Ebola Recovery Efforts to Mitigate Second Order Impacts of the Epidemic in Guinea, Sierra Leone & Liberia Performance Evaluation 2 Key Informant Interview Guide for Implementing Partners (revised 02/23/2019) BACKGROUND & INFORMED CONSENT 1. Date: 2. Country: 3. Time Start: 4. Time Finish: 5. Participant ID (transcript file document name) Names of KII team members (interviewers): 6._____________________________________________ 7._____________________________________________ 8. Name of Préfecture/County/District: 9. Town/village 10. Venue of the interview 11. Recording file ID # 12. Recording Quality Ok? Circle Best Answer 0 1 2 3 13. Privacy of the setting I don’t know Public Non￾respondents (i.e., family, neighbors, or others) within earshot Private 14. Comfort of the Interviewee I don’t know Anxious Ok At Ease 15. Other people present (if any, non-respondents): ________________________________ Respondent Profile: Name Name of the Position (Formal or Informal) Organization Contact information (telephone/email?) D.2 IMPLEMENTING PARTNER GUIDE (KII) D–4 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME II 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. THIS PAGE WILL BE REMOVED AND STORED SECURELY. THE INTERVIEW CONTENTS WILL BE ASSOCIATED WITH THE SOCIODEMOGRAPHIC BACKGROUND INFORMATION ONLY THROUGH THE PE2 ID CODE (Q.5). WARM UP AND INDIVIDUAL INFORMATION The three over-arching objectives of USAID’s funding for recovery activities were: (1) to prevent the loss of development gains; (2) the recovery and strengthening of existing institutions and infrastructure; and (3) to build sustained systems through PPPs, innovation and capacity building. As explained in the Informed Consent document, this evaluation takes a case-study approach which focuses on individual activities (or groups of activities) to determine what factors influenced their effectiveness and their ability to sustain long-term outcomes. One of the case studies in [COUNTRY] focuses on [NAME OF ACTIVITY]. 40 But to get started, please tell us more about your current role and responsibilities within [NAME OF ORGANIZATION FROM Q18]? 40 [IndFocus] Narrative: 41 Were you in-country in 2014-2015 during the Ebola epidemic? 41 Yes/No: 42 EQ1.1a. USAID’s engagement with Ebola recovery was very broad: Agriculture and food security; Health; Education; Governance; Economic crisis mitigation; and Information, communication and public-private partnerships. Have you [as individual] been involved in any of those sectors? I will read them off again, and would appreciate it if you would tell us yes/no: 01 Agriculture; Food Security 02 03 04 Education 05 Governa nce 06 07 ANNEX D. DATA COLLECTION TOOLS n D–5 Health Services & Systems EVD survivor support Economics, trade, investment Innovations, Technology, and Partnerships 43 Now we would like to talk about the Ebola recovery Activity that is the focus of our case study. We are specifically interested in [NAME OF ACTIVITY]. What was your role in [ACTIVITY]? 43 [Involvement] Narrative D–6 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME II EQ 1 – Effectiveness EQ 1 – Effectiveness EQ1.1a Did [NAME OF ACTIVITY] focus on a specific target population? EQ1.1b What was the main level of implementation of [Activity/ies] – was it mainly focused on actors at the national level, or sub-national? [if sub-national, probe for specific levels] From reports we have read that [ACTIVITY] was implemented in [LIST THE PRIORITY AREAS]. Was it delivered throughout these areas, or concentrated in specific locations within those areas? [Probe – what was the coverage of the Activity?] EQ1.2a Were there geographic areas where [NAME OF ACTIVITY] was implemented that were higher performing/more successful than others? What were the factors that contributed to the greater effectiveness of [NAME OF ACTIVITY] in these areas? Why? EQ1.2b. Were there geographic areas where [NAME OF ACTIVITY] was implemented that were lower performing /less successful than others? What were the factors that hindered effectiveness of [NAME OF ACTIVITY] in these areas? Why? [Probe: Did this activity succeed on its own merit or did it face challenges that made it difficult to implement?] EQ1.3 Were there other actors within your sector that were critical to [NAME OF ATIVITY] achieving success? - If so, which other actors? - What were their critical contributions? [Probe: List sectors one at a time if there are multiple sectors] EQ1.4. Were there actors in other sectors that were critical to [ACTIVITY] achieving its objectives? - If so, which other actors? ANNEX D. DATA COLLECTION TOOLS n D–7 - Why were their contributions critical/essential? What did those other sectors add/contribute? [Probe: How did cross-sectoral activities bolster the successful outcomes? [Probe: Were there additional sectors that should have been involved that weren’t? Were there gaps in cross-sectoral collaboration?] EQ1.5 [ASK IF NOT ALREADY COVERED]. How did [NAME OF ACTIVITY] include women, young people, the elderly and other vulnerable groups? Were there any specific successes or challenges with these groups? [Probe: ask for more details on inclusion and participation in the activity’s design, and/or implementation] EQ 1.6. Most of the investments in recovery from Ebola were made in 2015, 2016 and 2017. [ACTIVITY] closed in [MONTH AND YEAR THAT THE ACTIVITY CLOSED]. Can you still see any of the results or achievements of the project in effect today? [Be prepared to cite the main objectives and results of the ACTIVITY – to probe/ask if any are still in place] If so, which of the project’s results are still evident today? How or why did those results persist after the project funding was finished? If not, why did some results fade after the project closed down? EQ 2 – Self-Reliance This evaluation also focuses on how Ebola recovery activities have contributed towards the country’s capacity and commitment and to plan, finance, and manage its own development, and to address future challenges. EQ2.1 [IF TIME PERMITS] Did [NAME OF ACTIVITY] contribute towards increasing [COUNTRY’S] ability to address its development challenges? If yes, how did it do this? [PROBE: Some dimensions of commitment and capacity are: open, enabling trade and the business environment, habitat protection, safety and security, quality education and health care, ICT/phone and internet communications.] EQ2.2 Looking at the way things are now, has [the ACTIVITY/your department] increased [COUNTRY]’s capacity to solve its own development challenges? We are referring to individual capacity (skills, knowledge; relationships, expertise); And institutional capacity (human resource planning, SOPs, governance, resourcing and resource D–8 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME II mobilization; and other organizational development issues [Probe: Refers to capacity of the government, civil society, citizens, the strength of the health system, and the productivity and functioning of the economy] EQ2.3. Were there ways that [NAME OF ACTIVITY] contributed to the [COUNTRY] people’s commitment to solve their own development challenges? We are referring to changes in attitudes and behaviors, and informal governance mechanisms (such as for example broader social inclusion, including women and young people, new partnerships, We are also referring to changes in Structures – such as laws, policies, taxes or other ways to generate resources for national priorities. [Probe: Has it facilitated new partnerships or strengthened existing partnerships, including with the private sector?] [Probe: Discuss sustainability determinants: policies, strategies, and plans; finances; accountability; institutional support; technical choices; human dimension] EQ2.4 What were the key factors in the context of [THE COUNTRY OR THE GEOGRAPHIC AREAS WHERE THE ACTIVITY WORKED] that affected [ACTIVITY]’s ability to advance [COUNTRY]’s progress toward meeting its own development challenges? [Probe: Refers to sustainability determinant: economic situation or trends; the political situation or trends; social and cultural factors; external threats and enabling environments] EQ2.5 OVERALL, to what extent did [ACTIVITY] strengthen [COUNTRY]’s ability to mitigate and respond to future epidemics? 3.1 Is there a topic that we did not cover, or a question or issue that you would like to raise? Thank you very much for your time and insights. We will combine your responses with those of many ANNEX D. DATA COLLECTION TOOLS n D–9 other respondents from throughout [COUNTRY] as well as those from [OTHER 2 COUNTRIES], to present USAID and the countries with a balanced performance evaluation of the resources USAID provided to projects and interventions aimed at limiting the indirect effects of the Ebola crisis. End: Record the time: _____________ INTERVIEWERS AND SUPERVISORS: AFTER INTERVIEW: ● PLEASE ADD THE PE2 ID CODE TO THE HEADER. ● Note the gender and age group of the interviewee only when the age is not provided for Question 3.2 above. For the respondent identified in Gender Estimated Age Group (if 3.2 not asked/not answered) <30; 30-45; 46-60; >60 Comment (if any) Question 16 28. 29. Question 20 30. 31. Question 24 32. 33. ● THEN DETACH THE PAGES WITH PERSONAL IDENTIFICATION FROM THE REST OF THE TRANSCRIPT ● STORE THE TRANSCRIPT SEPARATELY FROM THE IDENTIFYING INFORMATION AND INFORMED CONSENT FORMS. D–10 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME II USAID/Pillar II Ebola Recovery Efforts to Mitigate Second Order Impacts of the Epidemic in Guinea, Sierra Leone & Liberia Performance Evaluation 2 Key Informant Interview Guide for the Government (revised 02/24/2019) TO CUSTOMIZE FOR THE SECTOR OF THE GOVERNMENT RESPONDENT BACKGROUND & INFORMED CONSENT 1. Date: 2. Country: 3. Time Start: 4. Time Finish: 5. Participant ID (transcript file document name) Names of KII team members (interviewers): 6._____________________________________________ 7._____________________________________________ 8. Name of Préfecture/County/District: 9. Town/village 10. Venue of the interview 11. Recording file ID # 12. Recording Quality Ok? Circle Best Answer 0 1 2 3 13. Privacy of the setting I don’t know Public Non￾respondents within earshot Private 14. Comfort of the Interviewee I don’t know Anxious Ok At Ease 15. Other people present (if any, non-respondents): ________________________________ Respondent Profile: Name Title (Formal or Informal Position) Organization Contact information (telephone/email?) 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. THIS PAGE WILL BE REMOVED AND STORED SECURELY. THE INTERVIEW CONTENTS WILL BE ASSOCIATED WITH THE SOCIODEMOGRAPHIC BACKGROUND INFORMATION ONLY THROUGH THE PE2 ID CODE (Q.5). D.3 COUNTRY GOVERNMENT GUIDE (KII) ANNEX D. DATA COLLECTION TOOLS n D–11 WARM UP AND INDIVIDUAL INFORMATION As you may have heard, the three over-arching objectives of USAID’s funding for Ebola recovery activities in Guinea, Liberia and Sierra Leone were: (1) to prevent the loss of development gains; (2) recovery and strengthening of existing institutions and infrastructure; and (3) to build sustained systems through PPPs, innovation and capacity building. As explained in the Informed Consent document, this evaluation takes a case￾study approach which focuses on individual activities (or groups of activities) to determine what factors influenced their effectiveness and their ability to sustain long-term outcomes. The case studies in [COUNTRY] in the [NAME RESPONDENT’S SECTOR/S] are [NAME/S OF ACTIVITIES]. 40 But to get started, please tell us more about your current role and responsibilities within [NAME OF ORGANIZATION FROM Q18]? 40 [IndFocus] Narrative: 41 Were you in-country in 2014-2015 during the Ebola epidemic? 41 Yes/No 42 EQ1.1a. USAID’s engagement with Ebola recovery was very broad: Agriculture and food security; Health; Education; Governance; Economic crisis mitigation; and Information, communication and public-private partnerships. Have you [as individual] been involved in any of those sectors? I will read them off again, and would appreciate it if you would tell us yes/no: 42 01 Agriculture; Food Security 02 Health Services & Systems 03 EVD survivor support 04 Education 05 Governa nce 06 Economics, trade, investment 07 Innovations, Technology, and Partnerships 42 43 Now we would like to talk about the Ebola recovery Activity/ies that is/are the focus of our case study. We are specifically interested in [NAME OF ACTIVITY]. What was your role in [ACTIVITY/These Activities]? D–12 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME II 43 [Involvement] Narrative EQ 1 – Effectiveness For all of the following questions, if the respondent did not work directly with the USAID activities we have chosen as case studies, ask about their knowledge of these activities as well as their work with other recovery activities or in the sector as a whole. EQ1.1a Do you recall [NAMES OF ACTIVITIES FROM Q42], focusing on a specific target population? EQ1.1b What was the main level of implementation of [Activity/ies] – was it/were they mainly focused on actors at the national level, or sub-national? [if sub-national, probe for specific levels] EQ1.1c. We are reviewing the ACTIVITY or SECTOR’s results in [LIST THE PRIORITY AREAS]. Were these activities delivered throughout these areas, or concentrated in specific locations within those areas? [Probe – what was the coverage of the Activity, within the priority areas?] EQ1.2a Were there geographic areas where [NAME OF ACTIVITY or Ebola recovery activities in the sector] was implemented that were higher performing/more successful than others? What were the factors that contributed to the greater effectiveness of [NAME OF ACTIVITY] in these areas? Why? EQ1.2b. Were there geographic areas where [NAME OF ACTIVITY or activities in the sector] was implemented that were lower performing/less successful than others? What were the factors that hindered effectiveness of [NAME OF ACTIVITY] in these areas? Why? [Probe: Did this activity succeed on its own merit or did it face challenges that made it difficult to implement?] EQ1.3 Were there other actors within your sector that were critical to [NAME OF ATIVITY] achieving ANNEX D. DATA COLLECTION TOOLS n D–13 the sector CLOSED]. Can you still see any of the results or achievements of those investments in effect today? [Be prepared to cite the main objectives and results of the ACTIVITY or Activities in the Sector – from the Checklists- to ask if any are still in place] If so, which of the results of [USAID-funded] recovery efforts are still evident today? How or why did those results persist after the project funding was finished? If not, why did the results of [the Activity/s or Ebola recovery efforts in the sector] ]fade after the project closed down? EQ 2 – Self-Reliance This evaluation also focuses on how Ebola recovery activities have contributed towards the country’s capacity and commitment and to plan, finance, and manage its own development, and to address future challenges. EQ2.1 [IF TIME PERMITS] Did [NAME OF ACTIVITY] contribute towards increasing [COUNTRY’S] ability to address its development challenges? If yes, how did it do this? [PROBE: Some dimensions of commitment and capacity are: open, enabling trade and the business environment, habitat protection, safety and security, quality education and health care, ICT/phone and internet communications.] EQ2.2 Looking at the way things are now, has [the ACTIVITY/your department] increased [COUNTRY]’s capacity to solve its own development challenges? We are referring to individual capacity (skills, knowledge; relationships, expertise); And Institutional capacity (human resource planning, SOPs, governance, resourcing and resource mobilization; and other organizational development issues [Probe: Refers to capacity of the government, civil society, citizens, the strength of the health system, and the productivity and functioning of the economy] EQ2.3. Were there ways that [NAME OF ACTIVITY] contributed to the [COUNTRY] people’s commitment to solve their own development challenges? We are referring to changes in attitudes and behaviors, and informal governance mechanisms (such as for example broader social inclusion, including women and young people, new partnerships, D–14 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME II the sector CLOSED]. Can you still see any of the results or achievements of those investments in effect today? [Be prepared to cite the main objectives and results of the ACTIVITY or Activities in the Sector – from the Checklists- to ask if any are still in place] If so, which of the results of [USAID-funded] recovery efforts are still evident today? How or why did those results persist after the project funding was finished? If not, why did the results of [the Activity/s or Ebola recovery efforts in the sector] ]fade after the project closed down? EQ 2 – Self-Reliance This evaluation also focuses on how Ebola recovery activities have contributed towards the country’s capacity and commitment and to plan, finance, and manage its own development, and to address future challenges. EQ2.1 [IF TIME PERMITS] Did [NAME OF ACTIVITY] contribute towards increasing [COUNTRY’S] ability to address its development challenges? If yes, how did it do this? [PROBE: Some dimensions of commitment and capacity are: open, enabling trade and the business environment, habitat protection, safety and security, quality education and health care, ICT/phone and internet communications.] EQ2.2 Looking at the way things are now, has [the ACTIVITY/your department] increased [COUNTRY]’s capacity to solve its own development challenges? We are referring to individual capacity (skills, knowledge; relationships, expertise); And Institutional capacity (human resource planning, SOPs, governance, resourcing and resource mobilization; and other organizational development issues [Probe: Refers to capacity of the government, civil society, citizens, the strength of the health system, and the productivity and functioning of the economy] EQ2.3. Were there ways that [NAME OF ACTIVITY] contributed to the [COUNTRY] people’s commitment to solve their own development challenges? We are referring to changes in attitudes and behaviors, and informal governance mechanisms (such as for example broader social inclusion, including women and young people, new partnerships, ANNEX D. DATA COLLECTION TOOLS n D–15 We are also referring to changes in Structures – such as laws, policies, taxes or other ways to generate resources for national priorities. [Probe: Has it facilitated new partnerships or strengthen existing partnerships, including with the private sector? [Probe: Discuss sustainability determinants: policies, strategies, & plans; finances; accountability; institutional support; technical choices; human dimension] EQ2.4 What were the key factors in the context of [THE COUNTRY OR THE GEOGRAPHIC AREAS WHERE THE ACTIVITY WORKED] that affected [ACTIVITY]’s ability to advance [COUNTRY]’s progress toward meeting its own development challenges? [Probe: Refers to sustainability determinant: economic situation or trends; the political situation or trends; social and cultural factors; external threats and enabling environments] EQ2.5 OVERALL, to what extent did [ACTIVITY] strengthen [COUNTRY]’s ability to mitigate and respond to future epidemics? 3.1 Is there a topic that we did not cover, or a question or issue that you would like to raise? Thank you very much for your time and insights. We will combine your responses with those of many other respondents from throughout [COUNTRY] as well as those from [OTHER 2 COUNTRIES], to present USAID and the countries with a balanced performance evaluation of the resources USAID provided to projects and interventions aimed at limiting the indirect effects of the Ebola crisis. End: Record the time: _____________ INTERVIEWERS AND SUPERVISORS: AFTER INTERVIEW: ● PLEASE ADD THE PE2 ID CODE TO THE HEADER. ● Note the gender and age group of the interviewee only when the age is not provided for Question 3.2 above. D–16 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME II For the respondent identified in Gender Estimated Age Group (if 3.2 not asked/not answered) <30; 30-45; 46-60; >60 Comment (if any) Question 16 28. 29. Question 20 30. 31. Question 24 32. 33. ● THEN DETACH THE PAGES WITH PERSONAL IDENTIFICATION FROM THE REST OF THE TRANSCRIPT ● STORE THE TRANSCRIPT SEPARATELY FROM THE IDENTIFYING INFORMATION AND INFORMED CONSENT FORMS. ANNEX D. DATA COLLECTION TOOLS n D–17 USAID/Pillar II Ebola Recovery Efforts to Mitigate Second Order Impacts of the Epidemic in Guinea, Sierra Leone & Liberia Performance Evaluation 2 Key Informant Interview Guide for NGOs, Civil Society Organizations and Other Partners (revised 02/14/2019) BACKGROUND & INFORMED CONSENT 1. Date: 2. Country: 3. Time Start: 4. Time Finish: 5. Participant ID (transcript file document name) Names of KII team members (interviewers): 6._____________________________________________ 7._____________________________________________ 8. Name of Préfecture/County/District: 9. Town/village 10. Venue of the interview 11. Recording file ID # 12. Recording Quality Ok? Circle Best Answer 0 1 2 3 13. Privacy of the setting I don’t know Public Non￾respondents (within earshot) Private 14. Comfort of the Interviewee I don’t know Anxious Ok At Ease 15. Other people present (if any, non-respondents): ________________________________ Respondent Profile: Name Title (Formal or Informal Position) Organization Contact information (telephone/email?) 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. THIS PAGE WILL BE REMOVED AND STORED SECURELY. THE INTERVIEW CONTENTS WILL BE ASSOCIATED WITH THE SOCIODEMOGRAPHIC BACKGROUND INFORMATION ONLY THROUGH THE PE2 ID CODE (Q.5). D.4 NGO, CSO, AND OTHER PARTNERS’ GUIDE (KII) D–18 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME II WARM UP AND INDIVIDUAL INFORMATION As mentioned before, this evaluation takes a case-study approach which focuses on individual USAID￾funded activities (or groups of activities) to determine what factors influenced the effectiveness and ability of activities to sustain long-term outcomes. The case studies in [COUNTRY] are: [READ LIST OF ACTIVITIES, INCLUDING MULTI-COUNTRY ACTIVITIES: Jhpiego, Intrahealth mHero, and Johns Hopkins CCP HC3 activities]. 40 But first, please tell us more about your current role and responsibilities within [NAME OF ORGANIZATION FROM Q18]? 40 [IndFocus] Narrative: 41 Were you in-country in 2014-2015 during the Ebola epidemic? 41 Y/N 42 EQ1.1a. USAID’s engagement with Ebola recovery was very broad: Agriculture and food security; Health; Education; Governance; Economic crisis mitigation; and Information, communication and public-private partnerships. Have you [as individual] been involved in any of those sectors? I will read them off again, and would appreciate it if you would tell us yes/no: 01 Agriculture; Food Security 02 Health Services & Systems 03 EVD survivor support 04 Education 05 Governa nce 06 Economics, trade, investment 07 Innovations, Technology, and Partnerships 43 [IF INTERVIEW IS AT THE SUB-NATIONAL/COMMUNITY LEVEL, ask:] What kinds of Ebola Recovery activities took place in this district/ community, in the years after the spread of EVD was controlled? [After asking the open-ended question, probe for the specific interventions of the ACTIVITIES THAT WERE IMPLEMENTED IN THIS COMMUNITY] 43 ANNEX D. DATA COLLECTION TOOLS n D–19 EQ 1 – Effectiveness [For all of the following questions, if the respondent did not work directly with the USAID Activities we have chosen as case studies, ask about their knowledge of these Activities. If they profess no knowledge, decide whether to proceed with a limited KII focusing on EVD recovery, or to request help in finding a more appropriate respondent. ] Now we would like to talk about the Ebola recovery Activity/ies that is/are the focus of our case study. We are specifically interested in [NAME OF ACTIVITY/ACTIVITIES] Q1.1a Do you recall [NAMES OF ACTIVITIES FROM Q42],? If yes: What stands out in your mind about [ACTIVITY/IES]? [PROBE: What were its main accomplishments, or contributions to EVD recovery? What was it known for? [If no, outline the key interventions and the IP/s involved and ask if the respondent is familiar with those, and shift away from Activity name, to refer to the interventions (e.g. Cash Transfer activities; IPC training...] Q1.1b IF Q1.1a= YES: How did you interact or work with [ACTIVITIY/IES]? [PROBE: how did [ACTIVITY] coordinate or collaborate with you and other Activities’ stakeholders?] Q1.2a Were there places where [NAME OF ACTIVITY or Ebola recovery activities in the sector] was implemented that were higher performing/more successful than others? What were the factors that contributed to the greater effectiveness of [NAME OF ACTIVITY] in these areas? Why? Q1.2b. Were there places where [NAME OF ACTIVITY or activitiesin the sector] was implemented that were lower performing/less successful than others? What were the factors that hindered effectiveness of [NAME OF ACTIVITY] in these areas? Why? [Probe: Did this activity succeed/fail on its own merit or did it face challenges that made it difficult to implement?] D–20 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME II Q1.3. Were there other actors in your sector or other sectors that were critical to [ACTIVITY achieving its objectives? - If so, which other actors? Which other sectors? - Why were their contributions critical/essential? What did those other actors add/contribute? [Probe: How did cross-sectoral activities bolster the successful outcomes? [Probe: Were there additional sectors that should have been involved that weren’t? Were there gaps in cross-sectoral collaboration?] Q1.4 Was the community involved in design, implementation, or management of ACTIVITY/IES? If so, in what ways? [Whether it was or was not] How did this affect the Activity’s results? [PROBES] Did community volunteers play a role? Was the activity tailored/adapted to different community conditions? How did the community participate? Was this a true collaboration, or did the activity or community control the interaction?] Was the community more, or less involved in [ACTIVITY/IES] than in other programs at the time? Were any groups missed out/left out? Why/How did that happen? Q1.5 Were there any external factors [conditions or events outside the Activity’s control that enabled or restricted [ACTIVITY/IES]’s effectiveness or impact? [PROBE FOR sustainability determinants: policies, strategies, & plans; finances; accountability; institutional support; technical choices; human dimension What were these factors and why did they contribute to or restrict effectiveness? [PROBE: Any issues with citizen engagement and trust in government? Gender culture? Economic conditions?] ANNEX D. DATA COLLECTION TOOLS n D–21 [Probe: if not previously discussed, ask for more details on effectiveness with reference to gender, youth, or other vulnerable groups. Were there any specific successes or challenges for these groups?] Q1.6 Most of the investments in recovery from Ebola were made in 2015, 2016 and 2017. [ACTIVITIES] closed in [MONTH AND YEAR THAT THE ACTIVITY OR USAID Ebola Recovery Activities in the sector CLOSED]. Can you still see any of the results or achievements of those investments in effect today? [Be prepared to cite the main objectives and results of the ACTIVITY or Activities in the Sector – from the Checklistts- to ask if any are still in place.] If so, which of the results of [USAID-funded] recovery efforts are still evident today? How or why did those results persist after the project funding was finished? If not, why did the results of [the Activity/s or Ebola recovery efforts in the sector] fade after the project closed down? EQ 2 – Self-Reliance Q2.1 This evaluation also focuses on how Ebola recovery activities contribute towards a country’s ability to address its own development challenges, including future epidemics. We are specifically focusing on the activities you said you are familiar with [LIST ACTIVITIES LISTED in Q1.1 AGAIN HERE] Did these activities contribute towards [COUNTRY’S] ability to address its own development challenges? If yes, which ones and how? [Probe: Some dimensions of commitment and capacity are: open, enabling trade and the business environment, habitat protection, safety and security, quality education and health care, ICT/phone and internet communications.] Q2.2 Looking at the way things are today, has [the ACTIVITY/IES increased [COUNTRY]’s capacity to solve its own development challenges? We are referring to individual capacity (skills, knowledge; relationships, expertise); And Institutional capacity (human resource planning, SOPs, governance, resourcing and resource mobilization; and other organizational development issues D–22 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME II [Probe: Refers to capacity of the government, civil society, citizens, the strength of the health system, and the productivity and functioning of the economy] Q2.3. Do you observe ways that [NAME OF ACTIVITY] has contributed to the [Liberian] people’s commitment to solve their own development challenges? We are referring to changes in attitudes and behaviors, and informal governance mechanisms (such as for example broader social inclusion, including women and young people, new partnerships, willingness to pay taxes, and so on], We are also referring to changes in Structures – such as laws, policies, taxes or other ways to generate resources for national priorities. [Probe: Has it facilitated new partnerships or strengthen existing partnerships, including with the private sector?] [Probe: Discuss sustainability determinants: policies, strategies, & plans; finances; accountability; institutional support; technical choices; human dimension] EQ2.4 What were the key factors in the context of [THE COUNTRY OR THE GEOGRAPHIC AREAS WHERE THE ACTIVITY WORKED] that affected [ACTIVITY]’s ability to advance [COUNTRY]’s progress toward meeting its own development challenges? [Probe: Refers to sustainability determinant: economic situation or trends; the political situation or trends; social and cultural factors; external threats and enabling environments] EQ2.5 OVERALL, to what extent did [ACTIVITY] strengthen [COUNTRY]’s ability to mitigate and respond to future epidemics? 3.1 Is there a topic that we did not cover, or a question or issue that you would like to raise? ANNEX D. DATA COLLECTION TOOLS n D–23 Thank you very much for your time and insights. We will combine your responses with those of many other respondents from throughout [COUNTRY] as well as those from [OTHER 2 COUNTRIES], to present USAID and the countries with a balanced performance evaluation of the resources USAID provided to projects and interventions aimed at limiting the indirect effects of the Ebola crisis. End: Record the time: _____________ INTERVIEWERS AND SUPERVISORS: AFTER INTERVIEW: ● PLEASE ADD THE PE2 ID CODE TO THE HEADER. ● Note the gender and age group of the interviewee only when the age is not provided for Question 3.2 above. For the respondent identified in Gender Estimated Age Group (if 3.2 not asked/not answered) <30; 30-45; 46-60; >60 Comment (if any) Question 16 28. 29. Question 20 30. 31. Question 24 32. 33. ● THEN DETACH THE PAGES WITH PERSONAL IDENTIFICATION FROM THE REST OF THE TRANSCRIPT ● STORE THE TRANSCRIPT SEPARATELY FROM THE IDENTIFYING INFORMATION AND INFORMED CONSENT FORMS. D–24 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME II IBTCI Evaluation of USAID Ebola Recovery (Pillar II) Activities – Performance Evaluation 2 (PE2) Focus Group Discussions Guide Template: Stakeholder Group:_Pillar II Service Beneficiaries_________ Sub-Group (if applicable)_________________________ FG Discussion date: DD MM YY FGD Session Code: ______________ Objectives: USAID wants to hear views on its Ebola recovery (Pillar II) activities from the men, women and adolescents who are, or were, direct and indirect beneficiaries of those programs. FGDs with beneficiaries will be conducted in urban and rural areas, and will seek the opinions of community members about: a) their experiences with the focus Activities [or the sector]; b) their value in helping them or their community to get back on their feet after the EVD crisis; c) what effects (positive and negative) of those 2015-2017 projects were sustained after the project closed; and d) what has changed in the community’s, and/or the country’s capacity and their commitment to solve their own development challenges. Recruitment: In consultation with the IP (and local authority if appropriate), the evaluation team will identify and recruit groups of diverse community leaders for the FGD. Since time does not permit separate FGDs with men, women and young people, we will recruit one large, diverse group (max 12), comprised of 3-4 adult male leaders, 3-4 adult female leaders, and 3-4 young people (age 15-24). Participants should not be known to each other, but are known to be a) beneficiaries of one or more Pillar II programs in the selected thematic areas; and b) able to express their views in a group discussion. Potential recruits should be read the informed consent statement before they come to the FGD, so only individuals who understand and give their consent will come to the event. In addition, the informed consent statement must be summarized by the Facilitator/Note-taker at the start of the event, and the Agreement List must be passed around so each can write/sign their name confirming their informed consent. Tailoring: This guide must be tailored to the place and respondent group involved. Please be sure the Facilitator and Note-taker know and use names of the relevant Ebola recovery activity(ies), the year each one closed, and the names of the USAID Implementing Partners and/or local CSOs involved. Please also have reviewed the objectives and interventions that comprised the Case Study Activity/ies that were implemented in this locale/facility. Logistics: Ensure that the note-taker has all the verbal informed consent statements, and the list of recruits, with gender, approximate age, and name. Transfer the names to the transport reimbursement receipt book, for reimbursement after the FGD. D.5 FACILITATOR INSTRUCTIONS AND INFORMATION SHEET (FGD) ANNEX D. DATA COLLECTION TOOLS n D–25 Instructions to Facilitator: Please follow the Introduction and Warm-up steps (1) Introducing yourself and the evaluation project; 2) Ensuring everyone has signed in and has received a FGD recruitment and informed consent page; 3) Introducing the topic; 4) Explaining FGDs; 5) Stating the ground rules; 6) Explaining confidentiality and what will be done with the data; 7) Reviewing the informed consent (IC) essential points; 8) Requesting their details and their decisions on IC to be indicated on the final tear-off page of the flier; and 9) Collecting the informed consent forms and saying a respectful thank you and good-bye to any who do not feel comfortable staying. Only after those steps should you begin with the FGD discussion questions listed below. FGD Participants Name Transport $ Paid Signature D–26 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME II IBTCI Evaluation of USAID Ebola Recovery (Pillar II) Activities – Performance Evaluation 2 (PE2) Focus Group Discussions Guide Template: Stakeholder Group:_Pillar II Service Beneficiaries_________ Sub-Group (if applicable)_________________________ FG Discussion date: DD MM YY ______________ SECTION A: ADMINITRATIVE INFORMATION – to be completed before the FGD in collaboration with the recruiter FGD ID PRE-DISCUSSION INFORMATION Name of Recruiter: **Where were participants recruited?: (What organization/s or contact mobilized the participants?) # Participants consented and recruited: M F Ado No-shows (Age/sex/source) M F Ado FGD INFORMATION Time of discussion: (24 hour clock): Start Time: End Time: Name of Facilitator: Name of Note-taker: Place of Discussion County/Prefecture/District: Town or Community: Venue/Facility Comments: Instructions to Supervisor: ** Information to be reviewed against the planning guide and edited for clarity and standardized place/organization names ANNEX D. DATA COLLECTION TOOLS n D–27 FGD Questionnaire Number: Location of FGD (town or facility) Date of FGD: RELEVANT QUESTIONS FROM GUIDE RECORD NOTES IN THE FORMAT BELOW Q2 from FGD Guide: We would like to talk to you about the time just after the Ebola crisis – when we learned that the epidemic was under control (that is, since [the end of 2015]). Can you remember any activities or projects that have helped your [community/workplace] to recover from the crisis? Could you name these? We’re going to talk about a story, about a huge forest, and the trees in the forest, and what happened to them in 2014, when a wildfire came through. It was a beautiful forest, with many kinds of trees. Some parts were burned very badly! Other parts of the forest were not burned so badly, but EVERYWHERE in the forest was affected by the smoke. A lot of people came to help replant and strengthen the forest. In some they brought water or fertilizer for the trees, and some planted seeds, and the trees bore fruit. In our story, the Forest is [Liberia / Sierra Leone / Guinea]. The trees in the forest are our communities. The wildfire was Ebola. The smoke is the indirect effects that Ebola had on our health care system, on our farms and jobs and trade, on our trust. And the water and fertilizer and seeds are the things that you did, and the Ministries and projects did to help the community get back on its feet. And the fruit are the things that got better. Key points in the story: - The huge forest – lots of different trees, bearing many different kinds of fruit - A wildfire broke out and swept through the forest - Some parts of the forest were hit harder than others by the fire, but the whole forest was hurt by the smoke. - The forest is Liberia; the trees are our many different communities. - The wildfire was Ebola - The smoke was the many ways the outbreak affected our whole country – the economy, jobs, farming, roads, health services, schools, and trust in government and even our neighbors. - Some parts of the forest were more affected than others. - The Forest is [COUNTRY]; the trees are our communities; the fire was Ebola, and the smoke was the ways the outbreak affected everything, including regular health services, farming, jobs, markets, schools, community meetings, everything – even friendships and relationships. 1. How much was THIS [community/clinic or hospital] affected by the wildfire? 2. How was it affected by the smoke? Back to our story: After the fire was OUT, some people came to help replant and strengthen the forest. Some brought water or fertilizer for the trees, and some planted seeds, to help our trees revive after the wildfire. D.6 FACILITATOR’S GUIDE (FGD) D–28 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME II RELEVANT QUESTIONS FROM GUIDE RECORD NOTES IN THE FORMAT BELOW 3. Can you remember any activities or projects that have helped this [community/workplace] to recover from the crisis? Could you name these? What water was brought or seeds were planted in your community to help people recover from the Ebola crisis? (POSSIBLE PROBE: Who planted those seeds? What groups or projects were involved?) Q4 from FGD Guide: Can you tell us about the positive effects (fruit) that [activity x] has had in your community? What were the fruit of this activity in your community? 4. Thank you for describing some of the water and fertilizer and seeds that were brought to help your part of the forest recover. Looking at our tree, what kind of fruit did those seeds produce after the epidemic? What changed in the [community/organization]? ANNEX D. DATA COLLECTION TOOLS n D–29 RELEVANT QUESTIONS FROM GUIDE RECORD NOTES IN THE FORMAT BELOW Q9 from FGD Guide If needs of women and children have not yet been discussed, ask:] ? (PROBE ON: How different members (women, men, youth, etc.) of the community benefited from the fruit on the tree.) Have women and children benefited from the project/s as well as men? What are some of the ways that women and girls have benefited? Q5 from FGD Guide What were the factors that have allowed that [activity x] to have positive effects / fruit in your community? 5a. There are some things that help trees produce fruit. They are like water, fertilizer or good soil for the tree. What things helped this tree grow fruit? 5b. There are also things that can make it difficult to produce a lot of fruit. What things made it difficult for this tree to produce more fruit for your community? What could have made those helping projects more effective? Q6 from FGD Guide What were the factors that have prevented [activity x] to have more positive effects in the community? Q7 from FGD Guide Have the positive effects/ fruit of that [activity x] lasted? Please discuss and tell us for each whether the positive effects a) further increased b) have remained the same c) have reduced d) have completely vanished Why? 6. Let’s take another look at our tree. What fruit do you still see on the tree today? (PROBE: Have the fruit gotten bigger, smaller, or have they been the same size since the post-Ebola recovery activity/services ended?) Q8 from FGD Guide How could this/these interventions be improved so that they bring more benefits, and so that the benefits last longer [in your community/organization]? 7. Let’s continue to look at the fruit that remained on this tree. What can be done to keep this tree producing fruit for a long time? D–30 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME II A N N E X E . L I S T O F I N T E RV I E W S , D I S C U S S I O N S , A N D V I n E–1 ANNEX E. LIST OF PERSONS INTERVIEWED, FOCUS GROUP DISCUSSIONS, AND SITE VISITS Second Performance Evaluation of USAID Ebola Pillar II Activities / Final Report Annex II Document E-1. KEY INFORMANT INTERVIEWS Type of Stakeholder Organization No. of KIIs Guinea Service Provider Commercial Health Center 1 District/Local Health Center 6 Regional Hospital 1 Government of Guinea Agriculture Prefectural Directorate (DPA) 2 Civil Society Organization 1 Code of Conduct Monitoring Committee (Commission de Suivi du Code de Bonne Conduite des partis politiques de Guinée - CCMC) 1 Commission Électorale Nationale Indépendante (CENI) 1 Ministry of Reconciliation 2 Ministry of Health (MOH) 6 National Assembly 1 National Youth Forum of Political Parties (FONAJEP) 1 Office of the National Mediator (ONM) 1 Implementing Partners Abt Associates 3 CRS 1 Jhpiego 4 MSH 2 NDI 2 OCPH 1 Other stakeholders 1 RTI International 1 SFCG 2 Chemonics 2 Liberia CSOs/NGOs Community Radio Station/Media Center 5 Faith Based Organizations 4 Other 2 Service Provider County/local Health Department 4 Key informants were given assurances of confidentiality and anonymity. Therefore, in E-1 we provide number of key informants by type of respondent stratified by country instead of listing them by name. E–2 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME II E-1. KEY INFORMANT INTERVIEWS Type of Stakeholder Organization No. of KIIs Government of Liberia Ministry of Health (MOH) 7 Ministry of Information, Cultural Affairs & Tourism 1 Ministry of Internal Affairs (GoL) 2 Ministry of Agriculture 2 Implementing Partners JHU CCP 1 Chemonix 2 IntraHealth 2 IRC 1 IREX 1 Mercy Corps 2 MSH 2 Other 8 Other Global/Local partners Center for Disease Control (CDC) 1 Other 2 Sierra Leone Government of Sierra Leone Anti-Corruption Commission (ACC) 3 Ministry of Agriculture, Forestry, and Food Security 1 Ministry of Health and Sanitation (MOHS) 11 Ministry of Social Welfare, Gender and Children's Affairs 1 National Commission for National Action (NaCSA) 1 Implementing Partners ACDI/VOCA 2 Care 2 MSH 1 GOAL 1 IOM 1 JHU CCP 3 John Snow, Inc. 2 Rofutha Development Association 1 Save the Children 3 Search for Common Ground (SFCG) 1 True Care Organization 1 UNICEF 1 World Vision 6 Other partners Private sector communication partners 4 A N N E X E . L I S T O F I N T E RV I E W S , D I S C U S S I O N S , A N D V I n E–3 E-1. KEY INFORMANT INTERVIEWS Type of Stakeholder Organization No. of KIIs Guinea Service Provider Commercial Health Center 1 District/Local Health Center 6 Regional Hospital 1 Government of Guinea Agriculture Prefectural Directorate (DPA) 2 Civil Society Organization 1 Code of Conduct Monitoring Committee (Commission de Suivi du Code de Bonne Conduite des partis politiques de Guinée - CCMC) 1 Commission Électorale Nationale Indépendante (CENI) 1 Ministry of Reconciliation 2 Ministry of Health (MOH) 6 National Assembly 1 National Youth Forum of Political Parties (FONAJEP) 1 Office of the National Mediator (ONM) 1 Implementing Partners Abt Associates 3 CRS 1 Jhpiego 4 MSH 2 NDI 2 OCPH 1 Other stakeholders 1 RTI International 1 SFCG 2 Chemonics 2 Liberia CSOs/NGOs Community Radio Station/Media Center 5 Faith Based Organizations 4 Other 2 Service Provider County/local Health Department 4 Government of Liberia Ministry of Health (MOH) 7 Ministry of Information, Cultural Affairs & Tourism 1 Ministry of Internal Affairs (GoL) 2 Ministry of Agriculture 2 E–4 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME II E-1. KEY INFORMANT INTERVIEWS Type of Stakeholder Organization No. of KIIs Implementing Partners JHU CCP 1 Chemonix 2 IntraHealth 2 IRC 1 IREX 1 Mercy Corps 2 MSH 2 Other 8 Other Global/Local partners Center for Disease Control (CDC) 1 Other 2 Sierra Leone Government of Sierra Leone Anti-Corruption Commission (ACC) 3 Ministry of Agriculture, Forestry, and Food Security 1 Ministry of Health and Sanitation (MOHS) 11 Ministry of Social Welfare, Gender and Children's Affairs 1 National Commission for National Action (NaCSA) 1 Implementing Partners ACDI/VOCA 2 Care 2 MSH 1 GOAL 1 IOM 1 JHU CCP 3 John Snow, Inc. 2 Rofutha Development Association 1 Save the Children 3 Search for Common Ground (SFCG) 1 True Care Organization 1 UNICEF 1 World Vision 6 Other partners Private sector communication partners 4 A N N E X E . L I S T O F I N T E RV I E W S , D I S C U S S I O N S , A N D V I n E–5 E-2. FOCUS GROUP DISCUSSIONS Type of Stakeholder Group Respondent Demographics No. of FGDs Guinea Recipients Community Beneficiaries 4 IGA Beneficiaries 1 CSO members 2 Media Partners 1 Service Providers Health Workers 4 Field Supervisors 1 Health Agents 2 Women’s Association, Community Health Committee Members 1 Liberia Recipients Community Beneficiaries 10 Community Elders 1 Community Youth 1 Male Community Beneficiaries 1 Service Providers Health Workers 9 Sierra Leone Recipients Community Beneficiaries 3 Service Providers Health Service Providers 1 Service Providers 1 Site visits for PE2 included informal observations by evaluation team members to help them establish a fuller context for the case studies they were developing; there were no structured observation guides. Site visits were also used as an opportunity to interview key informants. E-3. SITE VISITS Type of Site Visited No. of Sites Guinea Pharmacy/medical store 4 National/local government office 1 Regional/local health centers 7 Liberia Community/town 3 IP office 1 Media Center 1 National/local government office 1 Regional/local health centers 9 E–6 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME II E-3. SITE VISITS Type of Site Visited No. of Sites Sierra Leone Community/town 2 Regional/local health centers 1 A N N E X F. D I S C L O S U R E S A N D N O N - D I S C L O S U R E AG R E E M E N T S n F–1 ANNEX F. DISCLOSURES OF CONFLICTS OF INTEREST AND NDAS Second Performance Evaluation of USAID Ebola Pillar II Activities / Final Report Annex II Document Sara Woldehanna, Senior Evaluation Specialist, Team Leader Karim Sahyoun, Senior Evaluation Advisor F–2 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME II Donna Espeut , Senior Evaluation Advisor, Liberia Karim Sahyoun, continued A N N E X F. D I S C L O S U R E S A N D N O N - D I S C L O S U R E AG R E E M E N T S n F–3 Donna Espeut, continued Barbara De Zalduondo, continued Barbara De Zalduondo, Sr. Evaluation Advisor F–4 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME II Jacqueline Yiptong-Avila, Sr. Evaluation Advisor Elisabeth Nolan, Evaluator A N N E X F. D I S C L O S U R E S A N D N O N - D I S C L O S U R E AG R E E M E N T S n F–5 Elisabeth Nolan, continued F–6 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME II ANNEX G. EVALUATION TEAM INFORMATION n G–1 ANNEX G. EVALUATION TEAM INFORMATION Second Performance Evaluation of USAID Ebola Pillar II Activities / Final Report Annex II Document Sara Woldehanna – Senior Evaluation Specialist, Team Leader Sara Woldehanna is responsible for writing the final PE2 report in collaboration with the senior Evaluation Team Advisors. In more than 20 years of experience in research and evaluation, Ms. Woldehanna has planned and led the implementation of scientifically rigorous studies of varying scopes, complexities and methods, spanning a number of public health and development topics from emerging pandemic threats and HIV & AIDS to education. Ms. Woldehanna earned a Master’s degree in Applied Anthropology, with a concentration in Community Health from the University of Maryland, a Masters in Mechanical Engineering with a focus in pollution control from Lehigh University, and a B.A. in Physics from Randolph Macon Woman’s College. Barbara de Zalduondo – Senior Evaluation Advisor Barbara de Zalduondo is a Senior Evaluation Advisor for the Ebola Pillar II project. As a senior health policy and programming expert with over 25 years of experience, Dr. Zalduondo has led several projects in policy development, technical assistance, HIV program planning, research, monitoring and evaluation in low- and middle￾income countries, with special skills in mixed, qualitative and quantitative methods and on social and cultural determinants of health. Ms. Zalduondo holds a PhD in biological anthropology from Harvard University, a Master of Science in Behavior Sciences from the Harvard School of Public Health, and a Bachelor of Art in biological anthropology from Harvard University. Karim Sahyoun – Senior Evaluation Advisor Karim Sahyoun is a Senior Evaluation Advisor for the Ebola Pillar II project. Dr. Sahyoun’s nearly 20 years of experience in using qualitative and quantitative data collection and analysis tools have led to expertise in evaluation in multiple areas including economic livelihoods, Civil Society Organization assessments, food security, disaster risk reduction, education, health, HIV/ AIDS, organizational development of community structures, and economic empowerment. Dr. Sahyoun holds a PhD in agricultural economics and social sciences from Humboldt University, a Master of Science in tropical agricultural development, economics and planning from the University of Reading, and a Bachelor of Science from Kassel University. Donna Espeut – Senior Evaluation Advisor Donna Espeut is a Senior Evaluation Advisor for the Ebola Pillar II project. Dr. Espeut is a senior public health specialist who has led USG funded international health project evaluations for over ten years and over twenty countries, honing in on her skills as an evaluation specialist and program manager through a range of health program elements including HIV/AIDs, family planning, child survival, and nutrition. Dr. Espeut holds a PhD in reproductive health and family planning from John Hopkins University, a Master of Science in maternal and child health from Johns Hopkins University, and a Bachelor of Arts in human biology from Stanford University. EVALUATION TEAM G–2 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME II Jacqueline Yiptong-Avila – Senior Evaluation Advisor Jackie Yiptong Avila was a Survey Statistician and Methodologist for over 30 years at Statistics Canada. She is a trained Program Evaluator and in the last ten years, she has become very familiar with USAID and USDA Evaluation Policy and Guidelines after conducting the evaluation of Feed the Future Food security and Nutrition Programs and Food for Progress Agriculture Value Chain Programs in West Africa. She knows the Ebola Pillar II project quite well since she was the Data Quality Assurance Director (Interim) in 2017 and she has been on the team of trainers for this project on several occasions. Elisabeth Nolan - Evaluator Elisabeth Nolan is the Ebola Pillar II MEL’s Research Associate. Ms. Nolan has 11 years’ experience in the coordination of data management, statistical analysis, evaluation and survey research, both domestically and abroad. She has participated in data collection, management and analysis for over 15 research studies, and most recently worked to coordinate qualitative research efforts, technical support, and activity mapping for the Ebola Pillar II MEL activity’s two performance evaluations. She completed her Masters in International Development Economics from the American University School of International Service, Washington, DC and her Bachelors in Community Development and Spanish Language at Portland State University. ANNEX H.. STATEMENTS OF DIFFERENCE n H–1 ANNEX H. STATEMENTS OF DIFFERENCE Second Performance Evaluation of USAID Ebola Pillar II Activities / Final Report Annex II Document H–2 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME II ANNEX I. THEORY OF CHANGE n I–1 ANNEX I. PILLAR II THEORY OF CHANGE Second Performance Evaluation of USAID Ebola Pillar II Activities / Final Report Annex II Document Ebola Pillar II Activities (in six interacting thematic areas) Activities & Input Types (Used across several thematic areas) Basic health service delivery restored Health worker capacity to deliver EVD and non-EVD services Comprehensive services developed and provided for EVD survivors Health behavior gains enhanced and reinforced Health infrastructure (including management systems) strengthened Health information system developed and supported Health financing & resource allocation improved School attendance restored and gender balance improved Literacy & numeracy improved Household food security increased Nutritional status improved Household standard of living improved Enabling gvt policies developed & harmful policies & practices revised Community service organizations strengthened to provide advocacy, transparency, accountability Management systems & accountability of ministries & stakeholders improved at all levels Agriculture & import market function restored & improved Social protection provided for EVD-affected groups ICT systems for health, gender equity, & civic participation improved Power, water, roads, & communica￾tion infrastructure restored & expanded Outcomes Ebola Pillar II Objectives Loss of development gains prevented Institutions and infrastructure recovered and strengthened Sustained systems built through partnerships, innovation, and capacity building Technical Assistance Training (leadership, infection protection & control, management, gender equity) Clinical Care M&E Systems Development Consultation with Community Leaders Education Refurbishing/Restocking Facilities Nutritional Support Income Support (cash transfers, vouchers, food for work) Agricultural Inputs Social and Behavior Change Communication Staffing Support Restoring/Improving Infrastructure (roads, power) Procurement, Logistics, & Distribution Support Partnership Development ICT Inputs Innovation, Technology, & Partnerships Economic Crisis Mitigation Governance Agriculture & Food Security Basic Education Non-EVD Health Services, Health Systems Recovery, & Survivor Programs I–2 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME II ANNEX J. ADDITIONAL TABLES n J–1 ANNEX J. ADDITIONAL TABLES Second Performance Evaluation of USAID Ebola Pillar II Activities / Final Report Annex II Document J-1. CASE STUDY VS FINDINGS MATRIX Findings SIAPS (G) CHS SIAPS (SL) RHS CEPPS CSML Emerg Food EREL CT/ FFP GC mHero PERHS HC3 PACS HFG Factors related to effectiveness (EQ!) Responding to evidence-informed, felt needs positioned activities for success X X X X X X A disconnect between higher & lower levels of health system was recognized and deliberately addressed by some activities X X X X X X Successful activities addressed both supply and demand sides of universal health coverage equation X X X X X Embedding activity staff with government for long periods that stretched into years led to trust by local partners X X X X X X X Successful health activities were implemented by building on on￾going activities or existing com-munity or institutional structures X X X X X X X X IPs commitment to coordinate with other implementers X X X X X Health programming that employed non￾traditional cross￾sectoral partners X X X J.1 CASE STUDY vs FINDINGS MATRIX The following table maps the findings in the main report against specific case studies. J–2 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME II J-1. CASE STUDY VS FINDINGS MATRIX Findings SIAPS (G) CHS SIAPS (SL) RHS CEPPS CSML Emerg Food EREL CT/ FFP GC mHero PERHS HC3 PACS HFG Challenges: Inconsistent & insufficient supplies of essential drugs X Challenges: Insufficient funding has had other major implications for activity success X Challenges: Disparities due to social cohesion, urban or rural location, and the nature of the activity in its setting may affect activity success X X Pillar II funding creating platforms enabling communication X X X X X X Building trust in governance strongly context-dependent X X Ethnic dynamics in governance X Improvements in governance are closely linked to improvements in other sectors X X X X Promoted dialogue among political parties and supported the peaceful conduct of elections X Worked with and strengthened traditional leadership structures at community level X Constructive advocacy based on evidence gained through primary reporting in affected locales X Cross-cutting focus on gender equity and human rights X ANNEX J. ADDITIONAL TABLES n J–3 J-1. CASE STUDY VS FINDINGS MATRIX Findings SIAPS (G) CHS SIAPS (SL) RHS CEPPS CSML Emerg Food EREL CT/ FFP GC mHero PERHS HC3 PACS HFG Cash transfers and vouchers were effective as transitional interventions and influenced the success of other cross-sectoral interventions X X X Excellent coordination across IPs working under FFP X Challenges: Inadequate infrastructure in other sectors contributed to implementation challenges, requiring adaptive management by IPs X Making sure beneficiary selection and financial management were fair and accountable X Activities engaged important local government entities X Activities were gender-sensitive in their cash transfer implementation X Activities implemented cash transfers as part of a package of interventions aimed at building household & community resilience X Innovation includes building onto existing successes or infrastructures X Under-investment in necessary infrastructure (e.g., electricity, cellular networks) and technological expertise & training X J–4 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME II J-1. CASE STUDY VS FINDINGS MATRIX Findings SIAPS (G) CHS SIAPS (SL) RHS CEPPS CSML Emerg Food EREL CT/ FFP GC mHero PERHS HC3 PACS HFG Fostering public-private partnership to solve development problems X Grantees ranged from for-profit companies to non-profit institutions who in turn partnered with a variety of organizations including local & USG agencies X Drastically improve MOH￾to-health worker communication & evidence-informed management if they could connect these three existing systems X Interoperability of existing systems can leverage speedier solutions X Active participation & commitment of key actors in multiple sectors X Contributions to Self-reliance Organizational processes to efficiently communicate, budget, procure, & deploy health services X X X X X X X Activities also worked on building workforce capacities X X X X Workforce development & attrition are still serious issues that threaten the sustainability of activities Holding government accountable, ensuring the protection of civil liberties, & promoting inclusive development X X X ANNEX J. ADDITIONAL TABLES n J–5 J-1. CASE STUDY VS FINDINGS MATRIX Findings SIAPS (G) CHS SIAPS (SL) RHS CEPPS CSML Emerg Food EREL CT/ FFP GC mHero PERHS HC3 PACS HFG Successfully engaged community members in health promotion or quality-of-care activities X X X Gender-sensitive targeting & engagements X Mixed contributions in economic revitalization at the local level X Expand small businesses and increased harvests X Strengthening the community-facility interface & fostering community-led improvements in quality of health services X Technical convergence & synergy with other actors X X Distinct differences between urban & rural settings X Combining mutually reinforcing strategies in communities, in health facilities, with media outlets, & with national & sub-national health authorities X Lack of diversified and/or sustainable financing X X X X Mutual accountability between government institutions, civil society, political parties, communities, donors, implementing partners & beneficiaries X X X X X X X X PACS strengthened nine well-established Liberian CSOs to implement much of the community-level work X J–6 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME II J-1. CASE STUDY VS FINDINGS MATRIX Findings SIAPS (G) CHS SIAPS (SL) RHS CEPPS CSML Emerg Food EREL CT/ FFP GC mHero PERHS HC3 PACS HFG Stakeholder participation was built into every aspect of the work X Active use/ application of tools, guidance, & policies X ANNEX J. ADDITIONAL TABLES n J–7 J.2 CODEBOOK J-2. TABLE OF CODES ID CODE DESCRIPTION 1 *Q Question number 2 *FGD Q2 Activities or projects that helped your community/workplace recover from Ebola 3 *FGD Q4 Positive effects from the activity 4 *FGD Q5 What factors allowed activity to have positive effects 5 *FGD Q6 What factors that prevented activity from having positive effects 6 *FGD Q7 Have the positive effects lasted? 7 *FGD Q8 How could interventions be improved to have more benefits and longer lasting benefits? 8 *FGD Q9 Have women and children benefited? How? 9 *Q: 1.1a Did activity focus on a specific target population? 10 *Q: 1.1b What was the level of implementation of the activity? 11 *Q: 1.1c What was the coverage of the activities in their priority areas? 12 *Q: 1.2a Were there areas where the activity was implemented that were higher performing? 13 *Q: 1.2b Were there areas where the activity was implemented that were lower performing? 14 *Q: 1.3 Were their other actors within the activity's sector that were critical to activity success? 15 *Q: 1.4 Were their actors in other sectors that were critical to activity success? 16 *Q: 1.5a Did activity design include community in design, implementation or management? If so, who and what did they do? 17 *Q: 1.5b Did activity include women, young people, the elderly or other vulnerable groups? If so, what were the successes or challenges? 18 *Q: 1.6 Can you still see any of the results or achievements of activities that were funded in 2015-2017 today in the community? If so, which ones and how or why were they persistent? If not, why? 19 *Q: 2.1 Did the activity contribute towards increasing the country's ability to address its development challenges? 20 *Q: 2.2 Did the activity increase the country's capacity (individual, institutional) to solve its own development challenges? 21 *Q: 2.3 Did the activity increase the country's commitment (attitudes, behaviors, informal governance, laws, policies, etc) to solve their own development challenges? 22 *Q: 2.4 What are the key contextual factors that affected the activity's ability to advance the country's progress towards meeting development challenges? 23 *Q: 2.5 Overall, to what extent did the activity strengthen the country's ability to mitigate and respond to future epidemics 24 *Q: 3.1 Is there a topic we did not cover or something else you would like to raise? 25 *Q: 40 Role & Responsibilities 26 *Q: 41 In-country during Ebola 27 *Q: 42 Involvement in different sectors 28 *Q: 43 Role in case study activity 29 *Other question J–8 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME II J-2. TABLE OF CODES ID CODE DESCRIPTION 30 ACTIVITY ACTIVITY: This code should be used when a USAID activity/project is mentioned that does not fall into the specific activity codes, or an activity is mentioned within a sector, but it isn't clear which activity is being referenced. 31 Abt HFG ACTIVITY: This code should be used when Abt Associates, Abt, HFG, or Health Finance and Governance in Guinea are mentioned, or it's clear in the text that this is the activity being discussed. 32 ACDI/VOCA ACTIVITY: This code should be used when ACDI/VOCA, SNAP, or SNAP+ in Sierra Leone are mentioned. This code should not be used when only EFSP is mentioned, as this program relates to multiple projects under multiple IPs. 33 CARE RESSNER ACTIVITY: This code should be used when the text mentions CARE or RESSNER in Sierra Leone. This code should not be used when only EFSP is mentioned, as EFSP relates to multiple projects under multiple IPs. 34 CEPPS-NDI ACTIVITY: This code should be used when the text mentions CEPPS or NDI in Guinea or it's clear this is the activity being discussed from the text. Not to be confused with activities under SFCG. 35 CRS ACTIVITY: This code should be used when Catholic Relief Services or CRS are mentioned. This code should not be used in Liberia or Sierra Leone when only EFSP is mentioned, as this program relates to multiple projects under multiple IPs in those countries. The code can be used to refer to EFSP in Guinea, as this was the only activity under the program there. 36 Fighting Ebola Grand Challenges ACTIVITY: This code should be used when the text mentions the Fighting Ebola Grand Challenge or any of its components. This could refer to: 3D Family Stop Ebola Campaign in Guinea, Baylor College Smart Pod, Columbia University Kinnos Highlight, Dimagi CommCare, IBM Epic Platform, International Personnel Protection PPE, Intrahealth mHero, Johns Hopkins Improved PPE, Makerere U Epi-tent, Modula S rapidly deployable emergency treatment unit, Scripps STAMP sensor, Shift Labs Drip Assist, SPR Improved antiviral coating, or Tomi SteraMist. These were innovations funded to be tested in countries, but may or may not be in use now. 37 IOM ACTIVITY: This code should be used when the text mentions IOM, Institute on Migration or their Infection Prevention Control activities in Sierra Leone, or it's clear this is the activity being discussed from the text. Not to be used when IPC or Infection Prevention Control are mentioned by themselves, as there were many other activities in that category. 38 IRC PACS ACTIVITY: This code should be used when the text mentions IRC, International Rescue Committee, PACS, or Partnership for Advancing Community-based Services in Liberia, or if it is clear from the text this is the activity being discussed. 39 IREX CSML ACTIVITY: This code should be used when the text mentions IREX, the International Research & Exchanges Board, CSML, or the Civil Society and Media Leadership activity in Liberia, or if it is clear this is the activity being discussed. 40 Jhpiego ACTIVITY: This code should be used when the text mentions Jhpiego, MCSP Restoration of Health Services, RHS, or if it is clear from the text this is the activity being discussed. 41 JHU HC3 ACTIVITY: This code should be used when Johns Hopkins, JHU, CCP, Center for Communications Programs, HC3, Health Communication Capacity Collaborative, or any of these in combination with SBCC/SBC in any country are mentioned, or it is clear these are the activities being discussed. This code should not be used for the Johns Hopkins Fighting Ebola Grand Challenge project, which focused on a personal protective suit (PPE). 42 JSI APC ACTIVITY: This code should be used when the text mentions JSI, John Snow, APC, or Advancing Partners and Communities in Sierra Leone. This code should not be used for JSI APC activities referring to survivors in any country. For JSI Survivor activity, code this with generic "ACTIVITY" code. 43 Mercy Corps EREL ACTIVITY: This code should be used when the text mentions Mercy Corps or EREL in Liberia or it is clear this is the activity being discussed. This code should not be used when only EFSP is mentioned, as EFSP relates to multiple projects under multiple IPs. 44 mHero ACTIVITY: This code should be used when Intrahealth or mHero are mentioned in any country or it is clear this is the activity being discussed. ANNEX J. ADDITIONAL TABLES n J–9 J-2. TABLE OF CODES ID CODE DESCRIPTION 45 MSH Collaborative Support for Health ACTIVITY: This code should be used when the text mentions MSH, Management Sciences for Health, CSH or Collaborative Support for Health in Liberia or it is clear this is the activities being discussed. This code should not be used for MSH activities in Sierra Leone or Guinea. Occasionally misspelled to CHS in transcripts. 46 MSH SIAPS ACTIVITY: This code should be used when the text mentions MSH, Management Sciences for Health, or SIAPS in Guinea and Sierra Leone or its clear these are the activities being discussed. This code should not be used for MSH activities in Liberia. 47 Save the Children ACTIVITY: This code should be used when the text mentions Save the Children or FEEDS in Sierra Leone or it is clear this is the activity being discussed. This code should not be used when only EFSP is mentioned, as EFSP relates to multiple projects under multiple IPs. 48 Search for Common Ground ACTIVITY: This code should be used when the text mentions Search for Common Ground or SFCG in Guinea, or it's clear this is the activity being discussed from the text. Not to be confused with activities under CEPPS-NDI. 49 World Vision ACTIVITY: This code should be used when the text mentions World Vision in Sierra Leone or it is clear this is the activity being discussed. This code should not be used when only EFSP is mentioned, as EFSP relates to multiple projects under multiple IPs. 50 Other Activity ACTIVITY: is USAID activity/project that is being evaluated. 51 ACTORS ACTOR: This code should be used when the text mentions various persons, organizations, and entities who participate in addressing issues relevant to each country’s people, economy, and social systems. 52 Citizens/ Beneficiaries ACTOR: This code should be used when the text mentions activity beneficiaries, people who received some benefit from the USAID activity, or citizens who are involved in or affect development work in general. 53 Civil Society ACTOR: This code should be used when the text mentions actors from civil society: civil society organizations, NGOs, and non-profit entities, social groups and organizers. This code should not be used for community media: radio, news, or television. For these see Actors: Media and Communication Channels. 54 Cross-Sector ACTOR: This code should be used when the text mentions actors from more than one sector working together to address development issues, or one actor working across sectors. 55 Donor, Other ACTOR: This code should be used when the text mentions donors not including USAID. This could be the Gates Foundation, DfID, Other bilateral donors from other countries, The Global Fund, MCC, or the Paul Allen Foundation. 56 Government ACTOR: This code should be used when the text mentions actors from the government, including the Ministry of Health or other ministries, government officials, public officials from various sectors, public health facilities, schools, or infrastructure. 57 Implementing Partner, USAID ACTOR: This code should be used when the text mentions USAID implementing partners. 58 Media & Communication Channels ACTOR: This code should be used when the text mentions actors from the media industry, including community radio, public and private actors involved in journalism in print, radio, or television. 59 Private Sector ACTOR: This code should be used when the text mentions actors from the private sector. This includes small, medium, and large businesses and the people who work for them. Can be used in combination with other types of actors. 60 Within Sector ACTOR: This code should be used for two or more actors working together to address development issues in the same sector. For this code to be used, both actors should be from the same sector. 61 USAID ACTOR: This code should be used when USAID is mentioned. This code should not be used for specific USAID contractors. 62 Other ACTOR: This code should be used when the text mentions various persons, organizations, and entities who participate in addressing issues relevant to each country’s people, economy, and social systems but are not part of a specific actor code. J–10 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME II J-2. TABLE OF CODES ID CODE DESCRIPTION 63 FACTOR FACTOR: Factors are those items mentioned by respondents to be contributing to the success or failure of activities. 64 Accountability (Sustainability determinant) FACTOR: This code should be used when accountability is mentioned as a factor in success or failure, including financial, administrative, political, and social accountability. Accountability comprises the mechanisms and procedures through which decision-makers, policy-makers, and implementers justify and assume responsibility for the actions, decisions, services, and products that they undertake. Accountability is best proven through written documentation. 65 Challenges FACTOR: This code should be used if a specific identified factor code does not apply, but respondent mentions factors that are challenges. 66 Conflicts & Tensions FACTOR: This code should be used when the respondent mentions conflicts and tensions, including violence, interpersonal conflict, war, tensions between political parties, social groups, or other persons or groups. 67 Context FACTOR: This code should be used if a specific identified factor code does not apply, but respondent mentions factors that are contextual. 68 Dependency FACTOR: This code should be used if dependency on other citizens, donors, or the government is mentioned. 69 Enabling Environments (Sustainability determinant) FACTOR: This code should be used when the context or circumstance of a particular environment in the country, sector, or geographic area is helpful to the achievement of success or sustainability of results by the activity. Enabling environments have a direct or indirect influence on the ability of an intervention to be sustained, even when all other determinants are supportive of a sustainable outcome. 70 External Threats (Sustainability determinant) FACTOR: This code should be used when the context or circumstance of a particular environment in the country, sector, or geographic area is detrimental to the achievement of success or sustainability of results by the activity. External threats have a direct or indirect influence on the ability of an intervention to be sustained, even when all other determinants are supportive of a sustainable outcome. 71 Facilitator FACTOR: This code should be used if a specific identified factor is not mentioned, but respondent mentions factors that facilitate success. 72 Family FACTOR: This code should be used if the respondent mentions family as an enabling or detrimental factor to achieving outcomes. 73 Finances (Sustainability determinant) FACTOR: This code should be used if the respondent mentions financing, finances, budgets, allocation of financial resources for implementation of plans, determination of how much funding isneeded, the source of those funds, or how the funds will be apportioned and used. 74 Gaps FACTOR: This code should be used if the respondent mentions gaps in understanding, implementation, inclusion, or programming, or missed opportunities or missing pieces in strategies. 75 Human Dimension (Sustainability determinant) FACTOR: This code should be used for any situation whereby a human being’s input is needed to sustain the intervention. Identify the human dimension—whether there are enough staff may be part of it, but also whether the culture and norms of humans involved help achieve sustainable results. 76 Individual Capacity FACTOR: This code should be used if a respondent mentions increased or decreased capacity of individuals that affects the success or sustainability of an activity or group of activities. 77 Institutional Capacity FACTOR: This code should be used if a respondent mentions increased or decreased capacity of the institutions of the country, including the capacity of systems, groups, or processes within the country. 78 Institutional Support (Sustainability determinant) FACTOR: This code should be used for organizational arrangements and/or mechanisms are in place that a) operationalize agreed-upon inputs, roles and responsibilities for implementing policies and interventions and b) produce exit and transition strategies when feasible and appropriate. This code should be used when institutions support the success, failure, or sustainability of activities achieving outcomes. 79 Leadership FACTOR: This code should be used when leadership affects activity success or sustainability. ANNEX J. ADDITIONAL TABLES n J–11 J-2. TABLE OF CODES ID CODE DESCRIPTION 80 Management FACTOR: This code should be used when management affects activity success or sustainability. 81 Policies, Plans, and Strategies (Sustainability determinant) FACTOR: This code should be used if there is a law, regulation, directive, or guidance in place that outlines priorities and standards to guide the efforts of an institution, sector, or the country as a whole. This code should not be used for an individual's plans to do something in the future. 82 Relationships FACTOR: This code should be used when relationships between people or organizations affect activity success or sustainability. 83 Social Cohesion FACTOR: This code should be used when social cohesion between people or organizations affect activity success or sustainability. 84 Spiritual Beliefs/ Practices FACTOR: This code should be used when spiritual beliefs/practices among people or organizations affect activity success or sustainability. 85 Technical Choices (Sustainability determinant) FACTOR: This code should be used when a technical approach is mentioned that is selected for implementation or decision-making. "Good" technical choices are fit for purpose, effective in addressing the problem, feasible given time or resource constraints, and technically relevant. This code can be used for technical choices which are considered "good" or "bad" choices for the problem at hand. 86 Trust FACTOR: This code should be used when trust between people or organizations affect activity success or sustainability. 87 Urban-Rural Differences FACTOR: This code should be used when urban-rural differences affect activity success or sustainability. 88 Other Factor This code should be used when a specific Factor or Determinant code does not apply but the text mentions things that contribute to activity success or failure. 88 Other Factor This code should be used when a specific Factor or Determinant code does not apply but the text mentions things that contribute to activity success or failure. 89 GOOD QUOTE GOOD QUOTE: This code should be used when something someone says in an interview would make a good quote to illustrate an idea which may be reported on in the main report(s). 90 OBJECTIVE OBJECTIVE: This code describes overall objectives of an entire development program that are addressed by activities, interventions or contextual factors 91 Build sustained systems OBJECTIVE: This code should be used when building sustained systems in the country is an overall objective addressed by activities, interventions or context described in the text. 92 Mitigate/respond to future epidemics OBJECTIVE: This code should be used when mitigating or responding to future epidemics is an overall objective addressed by activities, interventions, or contextual factors mentioned in the text. 93 Prevent loss of development gains OBJECTIVE: This code should be used when preventing the loss of development gains is an overall objective addressed by activities, interventions, or contextual factors mentioned in the text. 94 Recover/ strengthen existing institutions/ infrastructure OBJECTIVE: This code should be used when recovering or strengthening existing institutions or infrastructure in the country is an overall objective addressed by activities, interventions or contextual factors described in the text. 95 Self-Reliance OBJECTIVE: This code should be used when achieving country self-reliance, or a country's ability to address their own development challenges is an overall objective addressed by activities, interventions, or contextual factors described in the text. 96 Strengthen country systems OBJECTIVE: This code should be used when strengthening country systems in the country is an overall objective addressed by activities, interventions or context described in the text. 97 Other Objective OBJECTIVE: This code should be used for overall objectives of an entire development program that are addressed by activities, interventions or contextual factors in the text that do not fit into the specific objective codes already defined. J–12 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME II J-2. TABLE OF CODES ID CODE DESCRIPTION 98 OUTCOME OUTCOME: Outcomes are results of strategies and activities 99 Attitudes OUTCOME: This code should be used when the outcome or results of activities and strategies are changes in attitudes of beneficiaries, groups, or the society as a whole. 100 Behavior OUTCOME: This code should be used when the outcome or results of activities and strategies are changes in behaviors of beneficiaries, groups, or the society as a whole. 101 Choices OUTCOME: This code should be used when the outcome or results of activities and strategies are changes in choices of beneficiaries, groups, or the society as a whole. 102 Capacity OUTCOME: This code should be used when the outcome or results of activities and strategies are changes in the capacity of the country to address their own development challenges, including the capacity to plan, resource, and manage, the capacity of the government, civil society including free media, the country's citizens, and the productivity and functioning of the economy. This code should not be used for the strategy of capacity building. 103 Commitment OUTCOME: This code should be used when the outcome or results of activities and strategies are changes in the commitment of country actors including changes to laws, policies, actions, behaviors, and informal governance mechanisms—such as cultures and norms— that enable the country to create and strengthen the institutions in order to solve its own development challenges. 104 Effectiveness OUTCOME: This code should be used when the outcome or results of activities and strategies is increased effectiveness in addressing the development objectives of Pillar II. 105 Failure OUTCOME: This code should be used when the results of activities are described as a failure or lack of effectiveness to achieve their goals 106 High Performer OUTCOME: This code should be used when high performing areas or activities are discussed in the text. 107 Low Performer OUTCOME: This code should be used when low performing areas or activities are discussed in the text. 108 Persistent Effects OUTCOME: This code should be used when the results of activities are described as lasting longer than the activity's implementation. 109 Success OUTCOME: This code should be used when the results of activities or strategies are described as a success or having achieved their goals. 110 Sustainability OUTCOME: This code should be used when the results of activities and strategies are described as sustainable, lasting beyond the life of the project, or achieving long-lasting effects. 111 Other Outcome OUTCOME: This code should be used for outcomes or results of strategies and activities described in the text that are not already described by one of the specific outcome codes. 112 SECTOR SECTOR: This code should be used for specific thematic areas of activities that cannot be coded with a specific sector code that already exists. This should be used for the sector of an activity or for things hap- pening in the sector as a whole, including but not limited to USAID activities. 113 Agriculture and Food Security SECTOR: This code should be used for the context, activities, and circumstances of the agriculture and food security sector, including farming, hunger, and food availability and affordability. 114 Economic Crisis Mitigation SECTOR: This code should be used for the context, activities, and circumstances of the economic crisis mitigation sector. This could include macroeconomic factors such as currency, commodities, trade, or investment, or microeconomic factors such as cash transfers for beneficiaries to prop up collapsing markets or make sure beneficiaries can afford basic necessities. 115 Education SECTOR: This code should be used for the context, activities, and circumstances of the education sector, including education policy, schools, teachers, and students. 116 Governance SECTOR: This code should be used for the context, activities, and circumstances of the governance sector, including government policy, officials, management, elections, democracy, conflicts, and accountability. ANNEX J. ADDITIONAL TABLES n J–13 J-2. TABLE OF CODES ID CODE DESCRIPTION 117 Health SECTOR: This code should be used for the context, activities, and circumstances of the health sector, including health policy, workers, management, systems, supply chain, pharmaceuticals, facilities, and medicine, supply and demand for health services, disease surveillance, mental health, health-seeking behaviors, and the general health status of citizens. 118 Innovation, Technology, and Partnerships SECTOR: This code should be used for the context, activities, and circumstances of the Innovation, Technology, and Partnerships sector, which may include technology infrastructure, innovative ways to approach development issues, especially in terms of technology, information and communication technology (ICT), mobile money, payments or apps, mobile or digital solutions addressing development issues, and public private partnerships (PPPs). 119 Other Sector SECTOR: This should be used for another thematic area sector that is not already covered by another specific code. This should not be used for private sector. Instead use Actors: private sector. 120 SENTIMENT SENTIMENT: This code should be used when an attitude, opinion, or feeling about something being said is clear, but does not fit into the negative or positive specific sentiment codes. 121 Negative SENTIMENT: This code should be used when the attitude, opinion, or feeling of a respondent is clearly negative. 122 Positive SENTIMENT: This code should be used when the attitude, opinion, or feeling of a respondent is clearly positive. 123 STRATEGY STRATEGY: Strategies are intentional activities performed by the IP to achieve success or sustainability of program effects, which are different from factors, which the IP may or may not have any effect on. 124 Activity Design STRATEGY: This code should be used when the design of the USAID activity or other activities are described, especially as a strategic decision to design for higher probability of effectiveness or sustainability. 125 Capacity Building/ Training STRATEGY: This code should be used when capacity building or training is discussed. 126 Cash Transfers STRATEGY: This code should be used when cash transfers is discussed. 127 Collaboration STRATEGY: This code should be used when collaboration between different actors is discussed. 128 Coordination STRATEGY: This code should be used when coordination between different actors is discussed. 129 Empowerment STRATEGY: This code should be used when empowerment, citizens or actors being given the authority, power, or confidence to make decisions central to activities is mentioned. 130 Human Resources/Staffing STRATEGY: This code should be used when human resources or staffing is discussed, including making sure there are enough staff, that they have the correct skills, or are managed, paid, and tracked well in country systems. 131 Implementation Approach STRATEGY: This code should be used when the approach of implementing the USAID activity or other activities are described, especially as a strategic decision made for higher probability of effectiveness or sustainability. 132 Participation & Inclusion STRATEGY: This code should be used when participation or inclusion in an activity or decision￾making process is discussed, including the inclusion of citizens, USAID, the government or other actors. 133 Partnerships - new STRATEGY: This code should be used when new partnerships between actors are mentioned. This code should not be used for strengthening existing partnerships that were in place before the epidemic. 134 Partnerships - strengthen existing STRATEGY: This code should be used when strengthening existing partnerships between actors is discussed. This code should not be used when new partnerships are discussed. 135 PPPs STRATEGY: This code should be used when PPPs or public-private partnerships between different actors is mentioned, especially as a strategic decision made to improve the probability of success or sustainability. J–14 n SECOND PERFORMANCE EVALUATION OF USAID EBOLA PILLAR II ACTIVITIES • ANNEX VOLUME II J-2. TABLE OF CODES ID CODE DESCRIPTION 136 Other Strategy STRATEGY: This code should be used for strategies which do not fit into a specific strategy code. 137 TARGET POPULATION TARGET POPULATION: Target populations are key populations or vulnerable groups which may be target￾ed beneficiaries of activities. 138 Vulnerable Groups TARGET POPULATION: This code should be used when vulnerable groups are discussed, especially as the target beneficiaries of a USAID, other donor, or government activity. This code should not be used for children/youth, women, or older adults, or Ebola survivors. 139 Women and Gender TARGET POPULATION: This code should be used when women or gender is discussed, especially as the target beneficiaries of a USAID, other donor, or government activity. 140 Youth TARGET POPULATION: This code should be used when children, adolescents, or youth are discussed, especially as the target beneficiaries of a USAID, other donor, or government activity. 141 Older Adults TARGET POPULATION: This code should be used when older adults, seniors, old people, or elders are discussed, especially as the target of a USAID, other donor, or government activity. 142 EVD Survivors TARGET POPULATION: This code should be used when Ebola survivors or EVD survivors are discussed, especially as the target of a USAID, other donor, or government activity 143 Health Workers TARGET POPULATION: This code should be used when health workers are discussed, in the private and public sectors, especially as the target of a USAID, other donor, or government activity 144 Ministry or Government Staff TARGET POPULATION: This code should be used when government workers or ministry staff are discussed, especially as the target of a USAID, other donor, or government activity 145 Other Target Population TARGET POPULATION: This code should be used when none of the specific codes for a target population can be applied. 146 New Idea This code should be used when none of the other codes in this list capture the idea